Speaker 1: All right, cool. What a way to start an episode,
Speaker 1: just leading off with our technical prowess and expertise.
Speaker 2: Hey Warren, Yeah, I really set that up nicely because
Speaker 2: I've for sure been having any issues. But maybe I'll
Speaker 2: just ignore that for a second. Jump straight to the point.
Speaker 2: We have a survey going for the podcast. It's of
Speaker 2: an Adventures in DevOps dot com slash survey, but you
Speaker 2: can see it everywhere. Please submit responses because it's a
Speaker 2: good feedback for us, something critical and thoughtful. If you'll
Speaker 2: win some a Tobos credits. And if it's not, well,
Speaker 2: there's not a lot that I can do about it
Speaker 2: at that point.
Speaker 1: If it's not helpful feedback, then just put some contact
Speaker 1: details in there and I will contact you and we'll
Speaker 1: have a good chat about it if you're so interested.
Speaker 1: I'm pretty excited today because we've got Paul Marston here
Speaker 1: with this as well. Paul, Welcome to the show.
Speaker 3: Hello, thank you for affording me the time. Yeah, So
Speaker 3: just to introduce myself, So, I work for Anchor slash
Speaker 3: Web three. I head up the note operations, so basically
Speaker 3: twenty four to seven running of blockchain nodes for integration
Speaker 3: partners that we've brought on board and also running you know,
Speaker 3: other nodes that we run specifically for our enterprise customers
Speaker 3: and other customers to access other blockchains of the networks. So, yeah,
Speaker 3: thank you for having me. Nice to be here, nice
Speaker 3: to meet you all.
Speaker 1: Yeah, for sure, I'm looking forward to this conversation because, well,
Speaker 1: for many reasons. One, I think it's super interesting to
Speaker 1: understand or just like to get some insight into what
Speaker 1: running infrastructure for Web three looks like when you have
Speaker 1: like a Web two background, because there are some different challenges.
Speaker 1: And I'm particularly excited to have you on the show
Speaker 1: because I'm very familiar with your work because a Polygon,
Speaker 1: we're a customer of yours, and you run a pretty
Speaker 1: significant amount of infrastructure on our behalf and do a
Speaker 1: fantastic job of it. I'll be upfront with that, like
Speaker 1: we Yeah, we don't even give a second thought to
Speaker 1: the infrastructure that y'all run for us, because it's just
Speaker 1: always there. And so I think that was one of
Speaker 1: the reasons I was excited to have you on this show,
Speaker 1: because you've clearly put the time and effort into the
Speaker 1: project to figure out how to make these things run
Speaker 1: at scale. So there we go. So give us a
Speaker 1: little bit about your background before you got into Web three,
Speaker 1: what like, what choices you make in life that brought
Speaker 1: you to this moment.
Speaker 3: Yeah, that's a great question. So yeah, similar to us.
Speaker 3: I've been in ANKA for a year now and similar
Speaker 3: to as I mentioned in my interview before I joined,
Speaker 3: so I've had quite a different ten years doing one
Speaker 3: thing ten years doing another. So yeah, I started out
Speaker 3: as an underwriter of all things, working in a call center,
Speaker 3: working on mainframe based you know, black and green screen
Speaker 3: terminals effectively for a retail company in the UK, taking
Speaker 3: applications and then deciding if people got loans or not
Speaker 3: based on some score that was spat out of you know,
Speaker 3: let's call it an arbitrary scoring algorithm for people's credit scores.
Speaker 3: And then then sort of slowly moved on from there
Speaker 3: doing more in depth and more technical elements. It was
Speaker 3: mainly an operational role that was to start with, but
Speaker 3: more technical elements in the let's call it Web two
Speaker 3: but specifically sort of financial services, moving on to doing
Speaker 3: testing and analytics, you know, requirements gathering for software builds,
Speaker 3: et cetera. And then I think that the main the
Speaker 3: main takeaway from like about three or four years of
Speaker 3: that period was I started working on migrations for customers.
Speaker 3: So they would come to where I was working at
Speaker 3: the time, first data, and we we would effectiveably do
Speaker 3: gap analysis between what they ran today for credit card processing,
Speaker 3: loan processing, et cetera, and how different our system was
Speaker 3: and what we needed to do in order to uplift
Speaker 3: our system or change their processes to bring them on board.
Speaker 3: And I think one thing I will say is even
Speaker 3: said this to my wife at the weekend. Working in
Speaker 3: a fast paced environment like you know, financial services migrations
Speaker 3: and i'm sure other migrations for other software, it really
Speaker 3: gives you a great background in dealing with things that
Speaker 3: you just cannot plan for. Every single event that we had,
Speaker 3: there was always something that came up that we've forgotten
Speaker 3: about or hadn't been captured as a requirement, et cetera.
Speaker 3: And being able to think quickly and because you have
Speaker 3: a set period where you need to migrate these things right,
Speaker 3: people want to use their credit cards to pay for things,
Speaker 3: you know, being able to being in that position having
Speaker 3: to think quickly, act quickly, resolve issues and move forward.
Speaker 3: It was always fixed forward. You know, there was never
Speaker 3: any going back. Yeah, I think it sets you up
Speaker 3: for for any role that you do in the future.
Speaker 3: So yeah, that's that was like migrations, and then I
Speaker 3: moved on to working on more digital focused products. That
Speaker 3: was the later part of my financial services background. I
Speaker 3: was working at Visa, I was looking enough to work
Speaker 3: on the Apple payroll out in the UK when they
Speaker 3: came to the UK, working with Apple and various other
Speaker 3: tokenization providers after that as well, So that was good.
Speaker 3: And then yeah, on the on the web free side,
Speaker 3: So it's it's almost I shouldn't say this, but I'll
Speaker 3: say it anyway, but it's almost like a love story
Speaker 3: between me and this guy I met in Discord and
Speaker 3: he's gonna hate that I've said that, but yeah, so
Speaker 3: so me and me and Pische or Peter. You may
Speaker 3: see him in Telegram and he's all over web free.
Speaker 3: But we we met. I think it's probably twenty seventeen,
Speaker 3: back end of twenty seventeen or twenty eighteen. We met
Speaker 3: in Discord for Horizon, you know, the crypto, and from
Speaker 3: there just hit it off, you know. We we had
Speaker 3: similar opinions, not the Peter would say necessarily that my
Speaker 3: opinion would be correct against his, but we had similar
Speaker 3: opinions in how we should be setting up servers, provisioning
Speaker 3: Linux running nodes, and you know that back then it
Speaker 3: was it was the secure node that Horizon had launched
Speaker 3: at that time, and yeah, we just we just had
Speaker 3: you know, very similar approaches to things, and naturally that
Speaker 3: came together and we actually ran a small startup for
Speaker 3: us a short time and then you know, i'd life
Speaker 3: got in the way of that and that kind of
Speaker 3: dwindled out. Peter moved on and continued to work in
Speaker 3: web Free, came to Anchor. I was at the time,
Speaker 3: I was finishing golf a contract for a UK based acquirer,
Speaker 3: so still doing some financial services work. And then I
Speaker 3: did another startup just a year prior to coming to Anchor,
Speaker 3: and then Peter got in touch and he was like, hey,
Speaker 3: there's a great opening here. It's very similar to what
Speaker 3: we've done previously. Why don't you come and join me?
Speaker 3: And yeah, and that's how I ended up here. And
Speaker 3: I'm just coming up to a year being in to
Speaker 3: Anchor right on.
Speaker 1: Very cool. So for someone who's not spent a lot
Speaker 1: of time working in web three specific to like the
Speaker 1: infrastructure that powers it. What would you say are the
Speaker 1: most significant holly shit factors that they experience.
Speaker 3: So from an infrastructure perspective, probably the scale. You know,
Speaker 3: certainly where we are, the number of bed and you
Speaker 3: know we're our selling point. Our USP is run it
Speaker 3: on bear metal, get the best performance possible, have many
Speaker 3: independently working systems, so you're not going to worry about
Speaker 3: what're losing one, three, five, whatever, your services stormline and yeah,
Speaker 3: you know, do it at a cost, do it at
Speaker 3: at the right cost to me all those factors, and yeah,
Speaker 3: so really the scale at which we have we provisioned
Speaker 3: hardware or anchor versus any of the organizations I've worked
Speaker 3: for in the past, with the exception of probably Visa,
Speaker 3: I would say, yeah, the scale of the hardware, the
Speaker 3: the not so much the effort in managing it, but
Speaker 3: ensuring that you have you know, the right rollouts, the
Speaker 3: right answer will playbooks to configure this host, that host,
Speaker 3: et cetera. On the right infrastructure's code in place. You know,
Speaker 3: it's significant, And yeah, I think that's the main difference.
Speaker 3: I think the other element to it as well is
Speaker 3: just the sheer overhead of managing that on a twenty
Speaker 3: four to seven three sixty five basis. We don't have
Speaker 3: bank holidays in Web three. We did, we did in
Speaker 3: financial services, and we used to have this think called
Speaker 3: we used to have this think call the weekend. That
Speaker 3: doesn't seem to make sist in Web three. So yeah,
Speaker 3: scale maintenance.
Speaker 1: And I didn't realize that was specific to Web three.
Speaker 1: I thought that got killed by COVID. That's that's my
Speaker 1: misunderstanding there.
Speaker 3: Now.
Speaker 1: I think when it talked, when you're talking about scale, like,
Speaker 1: one of the things that really sync in for me
Speaker 1: was each of the nodes. You know, because when you're
Speaker 1: talking about high availability, fault tolerance, redundant see that sort
Speaker 1: of stuff in a decentralized environment, what you're actually talking
Speaker 1: about there is every single node has to have a complete,
Speaker 1: accurate copy of the entire blockchain.
Speaker 3: To the extent that it needs to to serve its requirement. Right,
Speaker 3: So so we run different types of nodes. We have
Speaker 3: which mainly mainly come down as free categories, full node,
Speaker 3: which is you know, limited states at the you know,
Speaker 3: generally serves requests at the tip of the chain. And
Speaker 3: then we have the archive nodes, which, as you point
Speaker 3: out there, yeah, all required to store states all the
Speaker 3: way back to the genesis block and be able to
Speaker 3: serve that on you know, all day, every day twenty
Speaker 3: you know, three sixty five.
Speaker 1: Yeah, for sure.
Speaker 3: Just to add on to that as well, that the
Speaker 3: amount of data you have to store, you know, for
Speaker 3: an archive node, we're talking terabytes and terabytes, which for
Speaker 3: an independent service would be unheard of in the Web
Speaker 3: two world, right, you would probably have many services consuming
Speaker 3: from an enormous database as opposed to that note over
Speaker 3: there having five terabytes, that note over there having five terabytes,
Speaker 3: et cetera. So that's another good difference.
Speaker 2: So having transition from global payments infrastructure working with like
Speaker 2: Visa and some other payments companies into the Web three world, like,
Speaker 2: what sort of environment where you met with like is
Speaker 2: it common for people to transition from Web two? I mean,
Speaker 2: I see there's a lot of overlap in payments and
Speaker 2: financial structures within the Web three ecosystem, so you know,
Speaker 2: maybe there's some alignment there or was it completely different,
Speaker 2: you know, unexpected things coming up left and right that
Speaker 2: you just hadn't had experience with having been onside before.
Speaker 3: Yeah, So a great, great question. I think there's like two.
Speaker 3: There's probably two areas where it differs quite a bit.
Speaker 3: So like in the Web two space, a lot of
Speaker 3: the time there are there are lockins with vendors who
Speaker 3: sold your product, and with that comes additional bolt on
Speaker 3: products that you take as an organization because because you
Speaker 3: know you can, you can build a cohesive and hopefully
Speaker 3: you know, coherent platform to serve credit card traffic, loan traffic,
Speaker 3: you know, whatever you need to debit card traffic, et cetera.
Speaker 3: And in doing that, you almost make a decision about
Speaker 3: a lot of your infrastructure based on one key product
Speaker 3: that you need, and then as a result of taking
Speaker 3: that key product, you get all these other things as well.
Speaker 3: So you kind of mold yourself to working within those
Speaker 3: constraints of what those products have and what you've got
Speaker 3: off the shelf, and what is the key product that
Speaker 3: you've usually bought, whereas in Web three I would say
Speaker 3: there's there's there's less of that obviously, you know from
Speaker 3: we're not buying it in the Web three space, we're
Speaker 3: not buying in products that we run as such, or
Speaker 3: let's say a billing platform that we need to implement
Speaker 3: and run as part of some kind of processing platform.
Speaker 3: It's more that we're you know, our vendors primarily are
Speaker 3: our bare metal hosting providers who we deal with and
Speaker 3: you know, maintain good relationships with. So I would say
Speaker 3: there's more, there's more freedom to choose in the web
Speaker 3: free space, and we can dictate our own path more.
Speaker 3: I think the main differences between the approach to let's
Speaker 3: say development, release management and let's say notification of changes
Speaker 3: and version increments, et cetera in the Web two space
Speaker 3: versus Web three is web Web three. It happens instantly, right,
Speaker 3: you know, we can we can come in on a
Speaker 3: morning and a blockchain team that we've worked with can
Speaker 3: announce there's a hard falk tomorrow, and we've got to
Speaker 3: be ready. It doesn't matter if you're on in ten nodes,
Speaker 3: forty eight nodes, however many nodes. You know, the expectation
Speaker 3: is that you're ready. To Contrast that with what would
Speaker 3: have happened in Web two, we would have tested that
Speaker 3: for weeks. You know. I was just joking on a
Speaker 3: call with some guys the other day. They they were
Speaker 3: talking to someone loosely links to banking. Let's say, I
Speaker 3: can't say more than that, and they were saying, oh, yeah,
Speaker 3: so they said it's a six month project. I was like, whoa, whoa,
Speaker 3: I don't know you, you've not worked in this industry before.
Speaker 3: Then every finance project starts out as a six month project,
Speaker 3: and then three years later we're still working on implement
Speaker 3: to get VP and and and a lot of that is,
Speaker 3: you know, obviously there's there's decision making and you know,
Speaker 3: changes attack et cetera, and general evolution of what it
Speaker 3: is that you want to deploy and offer to your customers.
Speaker 3: But in some cases, you know your your timeline for development.
Speaker 3: In let's say the Web two space, the finance space
Speaker 3: will be shortened and you will have this huge chunk
Speaker 3: of testing that happens before things go into production. And
Speaker 3: I'm talking you know, business system testing, user acceptance testing,
Speaker 3: operational acceptance testing, et cetera, et cetera. So I think, yeah,
Speaker 3: I think there's more flexibility. We have more say over
Speaker 3: where we want to drive things in the web free space.
Speaker 3: In the Web two, I would say it's more rigid,
Speaker 3: but I wouldn't necessarily say that's all bad. I think
Speaker 3: there are good practices and processes that we can overlay from.
Speaker 3: Certainly my experience in Web two slash finance with what
Speaker 3: we do and how we approach the web free space.
Speaker 2: Makes sense. So just to get a better understanding of
Speaker 2: what it is that you're sort of doing. Would it
Speaker 2: be accurate to say you're like a cloud provider for
Speaker 2: web three companies that are running their own chains and
Speaker 2: you're hosting the nodes for them, or there's a gross
Speaker 2: oversimplification of what I'm sure you're actually doing.
Speaker 3: I mean, that doesn't that's not half bad in terms
Speaker 3: of the description. I think, you know, if you coming
Speaker 3: in cold and taking that away, I think I think
Speaker 3: that's pretty accurate. My only aversion is the cloud thing.
Speaker 3: So yes, it's cloud because it's not at your home,
Speaker 3: but we explicitly, don't you know, we have some critical
Speaker 3: services running in cloud infrastructure, AWS, GCP, etc. That's generally
Speaker 3: what I think people think of as cloud. But we're
Speaker 3: all about bare metal, least latency and highest performance.
Speaker 2: If so, now this is my own for my own understanding,
Speaker 2: Like what's the benefit of using a provider that offers
Speaker 2: dedicated nodes for the chain that you're developing, Like don't
Speaker 2: you have some sort of consensus protocol and so all
Speaker 2: of your users are going to be well, some set
Speaker 2: of the users are going to be running their own
Speaker 2: nodes anyway, how does that work? Like, what's the benefit?
Speaker 2: And I've seen, like some of the cloud writers a
Speaker 2: ws et cetera, have pretty bad versions of managed blockchain things,
Speaker 2: and I have yet to understand why they do that.
Speaker 2: I'm sure they're not good. I'm sure you have some
Speaker 2: opinions about that as well, but like I just don't
Speaker 2: understand what the point is.
Speaker 3: Yeah, So to come back to the point that was
Speaker 3: made earlier on actually, you know, the if you take
Speaker 3: a source, the fact that let's say for a developer,
Speaker 3: they have to firstly understand whatever this git repository is
Speaker 3: selling them in terms of how to set up a node.
Speaker 3: You know, I think there are there are a lot
Speaker 3: of good developers out there who have that skill set
Speaker 3: who can go from you know, like bootstrapping a Docker environment,
Speaker 3: running something in kumerinettes, et cetera, to then actually doing
Speaker 3: the code they want to do. But I think certainly
Speaker 3: there's there's a number of developers and I don't want
Speaker 3: to generalize anyone here, but there are certainly a number
Speaker 3: of developers who simply want to hid an API, and
Speaker 3: you know, let's say you need an Ethereum archive node
Speaker 3: or a Polygon archive node, you're talking about downloading for
Speaker 3: many days terabytes worth of data before your node is
Speaker 3: even ready to serve those requests. So yeah, So the
Speaker 3: simple answer is from from a developer standpoint, it's it's
Speaker 3: just plug and play. You know, you come up, you
Speaker 3: come up to our anchor website, you register, you start
Speaker 3: consuming traffic. If you want a higher rate limit, you
Speaker 3: pay a little bit of money. If you want lots
Speaker 3: of traffic, then you know, you come and talk to
Speaker 3: us about having an enterprise contract. So I think I
Speaker 3: think that's the main benefit for a developer. The other
Speaker 3: the other benefit, I guess is that ongoing maintenance version upgrades.
Speaker 3: You know, the redundancy that we offer and the latency
Speaker 3: that we offer, you know that that leads itself to
Speaker 3: an easier, let's say development and testing time frame for
Speaker 3: that specific development.
Speaker 2: So just you know something that I'll basically there's a
Speaker 2: lot of parts that go from doing this, like from
Speaker 2: the software development to actually running the nodes, like for
Speaker 2: the network, for the for the chain that's out there,
Speaker 2: and that's a lot of infrastructure, a lot of process,
Speaker 2: and that's what's being automated, either helping with the development
Speaker 2: side or the testing or rolling out for what the
Speaker 2: new versions are that users should pull down and start
Speaker 2: running within their Whoever the node hosting providers are, you know,
Speaker 2: whether they're general population or whether they're independent companies or whatever. Okay,
Speaker 2: so there's a lot of work that actually needs to
Speaker 2: be done there make this work effectively, and you're solving
Speaker 2: everything in that space.
Speaker 3: Absolutely, we are. Yeah, I like to think so. Yeah.
Speaker 1: I think really good perspective to put on that is
Speaker 1: if you are, like, if you're a developer who wants
Speaker 1: to create the like a cryptopunk n FT, you know,
Speaker 1: the only thing you're interested in is selling cool little
Speaker 1: eight bit graphics, and the barrier to entry to that
Speaker 1: without having access to hosted RPC nodes, the barrier to
Speaker 1: entry is you've got to set up a node that
Speaker 1: has terabytes and terabytes of data on it just to
Speaker 1: generate your your NFTs. And that's where services like anchor
Speaker 1: come in and take take that barrier away from you
Speaker 1: or remove that barrier for you. One question I want
Speaker 1: to ask, just to elaborate on it a little bit.
Speaker 1: You mentioned hard for earlier. Can you elaborate a little
Speaker 1: bit on what a hard fork is.
Speaker 3: Yeah, So, in the simplest terms, it's a breaking change
Speaker 3: which requires a version upgrade. So essentially, a blockchain will
Speaker 3: run to a particular block, and at that block time,
Speaker 3: the protocol is in the network, how it talks, how
Speaker 3: the notes talk to one another. They will undergo a
Speaker 3: breaking change, which means unless you're running this new version
Speaker 3: of software, your node is basically going to sit there
Speaker 3: stalled and isn't able to communicate with the others here
Speaker 3: with them and then you know, propagate blocks and receive transactions,
Speaker 3: et cetera.
Speaker 2: So I've always looked at this as imagine the blockchain
Speaker 2: is your database, and so of hard work is where
Speaker 2: you make a non back whards compatible change to the
Speaker 2: schema of the database, and all of the clients of
Speaker 2: the world need to decide how they're going to interpret
Speaker 2: that new new schema of the database. I don't know
Speaker 2: that's always worked for me. I don't know if that's accurate, but.
Speaker 3: No, there's no harm in that description. Yeah, absolutely. I
Speaker 3: think the key element you've got there, right, is that
Speaker 3: it's a breaking change, and yeah, that's the main thing.
Speaker 1: Yeah, and I think the unique constraint to it is
Speaker 1: that the decentralized aspect of it, because when you hit
Speaker 1: that block, you're now dependent on everyone who operates a
Speaker 1: node on that network, or not everyone, but a majority
Speaker 1: of the node operators on that network to adopt that
Speaker 1: change and implement it. Otherwise you end up with a
Speaker 1: network that now has multiple people claiming that they have
Speaker 1: the latest block.
Speaker 3: Yeah, yeah, absolutely.
Speaker 2: I mean that's an interesting thing here because let's say
Speaker 2: the Web two world, if you've exposed your internal database
Speaker 2: to your customers and they're you know, making I mean,
Speaker 2: no one would ever do this, right, No one ever
Speaker 2: did this history of the world, for sure. But you know,
Speaker 2: your customers have access to the schema, to the underlying
Speaker 2: data that you have in your database, maybe with their
Speaker 2: own special user name and password, and you make a
Speaker 2: schema change there. I mean, in the Web two world,
Speaker 2: I just can't ever fathom a story where you like
Speaker 2: go around to each of your customers are like, Okay,
Speaker 2: we're going to make a change. I need you to,
Speaker 2: you know, change your software to actually support this. And
Speaker 2: having worked in a bunch of these companies where we
Speaker 2: even had an API, you know, rest or you know
Speaker 2: something else, Getting your customers to actually make a change
Speaker 2: and start using the latest version is I mean, that's
Speaker 2: a gargratuan task that I don't think I ever saw
Speaker 2: a company actually do successfully. Oh yeah, we'll just make
Speaker 2: a breaking change and get all our customers to update.
Speaker 2: Was easier said than done. And I feel like in
Speaker 2: the blockchain world you actually end up in this state
Speaker 2: where you have customers I'll call them customers, not really customers. Right,
Speaker 2: the end users who are using utilizing the chain will
Speaker 2: have still been using the previous software version, which doesn't
Speaker 2: understand what happens after the fork, and that's sort of
Speaker 2: what creates this maybe two future chains or whatever you're utilizing.
Speaker 3: Yeah, it's kind of a leader follower mentality, isn't it.
Speaker 3: I think? And ultimately the developers and the foundations of
Speaker 3: these networks are free to choose how they want to
Speaker 3: how they want to push the protocol, and really who
Speaker 3: is it is their choice, whereas in the web two
Speaker 3: space and a financial setting, you know you possibly have
Speaker 3: shareholders and who are big customers and all this kind
Speaker 3: of stuff. So it is Yeah, that's another good point.
Speaker 3: Change management and management of external customers who consume your service.
Speaker 3: It's much more it was much more complex to manage
Speaker 3: a much more bigger a task to get them ready
Speaker 3: for those new releases, as you've alluded to, than in
Speaker 3: the Web three space. You know, you're either on the
Speaker 3: train or you're on the train unfortunately.
Speaker 2: I mean there's like a huge I feel like there's
Speaker 2: actually quite a difference in perspective here because in Web
Speaker 2: two the customers don't really necessarily influence each other, right,
Speaker 2: Like what one customer is hitting your version one API
Speaker 2: and different cstomers hitting your version two API. There's not
Speaker 2: motivation for them to switch other than the value of
Speaker 2: using a later version. But in the Web three space,
Speaker 2: they likely want to be on the same network or
Speaker 2: within the same network on the latest version because there
Speaker 2: is cross communication between different users of the nose. I mean,
Speaker 2: they're all contributing to the same ledger or chain in
Speaker 2: a way, so they're not doing it in isolation. And
Speaker 2: that pretty much means as a blockchain company that's or
Speaker 2: creating anyone who's creating a chain. You're just writing the
Speaker 2: software that you think a majority of your customers want,
Speaker 2: which I hope you're doing. If you're not in the
Speaker 2: blockchain world. But you almost have to be doing it
Speaker 2: because if you do it and they and the majority
Speaker 2: don't accept the whatever happens after the hard fork, you
Speaker 2: did all that work for sure, now that you can't
Speaker 2: force them to migrate.
Speaker 3: Yeah, yeah, indeed, I think I think the main the
Speaker 3: main difference to call out here is that generally when
Speaker 3: the when the breaking changes come, they will impact an
Speaker 3: element of the network or the way a certain way
Speaker 3: that you could figure a node, or that you can
Speaker 3: how you stake tokens for that particular network or that
Speaker 3: kind of thing, whereas something customer facing is much less
Speaker 3: likely to happen. And what I mean is, you know
Speaker 3: it's there are far there are fewer changes for the
Speaker 3: API that faces the customers that they're consuming, you know, individually, Generally,
Speaker 3: you know, if it's going to be EVM compatible, it's
Speaker 3: always going to be VM compatible. And all those methods
Speaker 3: available in various name spaces generally perpetuate, you know, and
Speaker 3: they don't change what you will sometimes see. And again,
Speaker 3: as you mentioned, you may see you may see a
Speaker 3: new method or a new name space open which gives
Speaker 3: you some other a yeah that you can play with,
Speaker 3: you know, that can happen. But yeah, I think generally
Speaker 3: the hard fault changes aren't things that our customers would
Speaker 3: see or be aware of unless we missed one, and
Speaker 3: our you know knows will stop runt. Yeah.
Speaker 1: I think another way to think about that is that
Speaker 1: it's much more community driven for Web three. You know,
Speaker 1: as a Web two business. It's probably a horrible example,
Speaker 1: but like Visa could say, you know what, we're not
Speaker 1: going to support the US dollar anymore on April first,
Speaker 1: will deny all transactions priced in US dollars, And as
Speaker 1: a customer of Visa, you can go damn okay, and
Speaker 1: you can look for someone else, you can train take
Speaker 1: your business somewhere else, but that's your only option. And
Speaker 1: in a Web three world, you can propose a hard
Speaker 1: fork saying hey, we're dropping support for this, and the
Speaker 1: Web three community can look at that hard fork, can
Speaker 1: either adopt it or say, you know, I don't think
Speaker 1: we're going to go that direction, and the community just
Speaker 1: doesn't adopt the chain the change and goes off and
Speaker 1: operates on their own.
Speaker 3: I think I think I can see I can see
Speaker 3: the argument there. I think the contrast of what you're
Speaker 3: suggesting in Web two versus what you're suggesting webs free
Speaker 3: is significantly different, Like we're not going to do us
Speaker 3: already more of visa. I mean I actually want.
Speaker 2: Yeah, I actually do. Wonder if you have any like
Speaker 2: statistics or no information about this, because I mean, I
Speaker 2: think the world has seen enough public hard forks and
Speaker 2: the way most of them as far as my experience
Speaker 2: is gone, I guess the hard works I know of
Speaker 2: primarily are the ones in ethereum. The world pretty much
Speaker 2: adopts the majority of the change. And I wonder how
Speaker 2: many companies that are running chains end up doing some
Speaker 2: sort of hard fork where it's got rejected by the community,
Speaker 2: Like does that? Does that ever happen that the majority
Speaker 2: and we just don't hear about it.
Speaker 3: I don't know, I don't think so. I mean, isn't
Speaker 3: that the isn't that the background of ethereum versus ethereum
Speaker 3: classic of sorts? Obviously, I don't know. That was so
Speaker 3: long ago. I can't remember exactly how that happened.
Speaker 2: But that's definitely the community saying you know, what happened
Speaker 2: here is not okay and just rejecting out and arguably
Speaker 2: it's the same thing that happens with any hard work
Speaker 2: though that goes through. There's also still the proof of
Speaker 2: work Ethereum chain that's out there that you people are
Speaker 2: performing work to mine the coins and get value out.
Speaker 2: But it's such a small part of the majority of
Speaker 2: compared to the size the number of nodes that are
Speaker 2: being added to the current Ethereum network.
Speaker 3: Yeah, indeed, so, I mean we had the pectoralup grade
Speaker 3: recently on Aleski, one of the Ethereum test nets. Sorry
Speaker 3: I should mention if if people don't know, so, I
Speaker 3: mean that was problematic because there were there are multiple
Speaker 3: different client options you can run. This is another rabbit
Speaker 3: hole that we can run down and just give you
Speaker 3: a bit of background there. So primarily we run Aragon
Speaker 3: clients what's called Aragon to run our archive nodes, and
Speaker 3: then full notes we generally run you know, geth which
Speaker 3: is as you're probably aware that the main Ethereum client
Speaker 3: or Maine. I guess it's debatable based on what side
Speaker 3: of the fence you're on, but yeah, there there were
Speaker 3: there are essentially three. There were. There were a number
Speaker 3: of changes that happened for Pectra which were implemented correctly
Speaker 3: in some clients and not in others. Now, the deep
Speaker 3: technical understanding of it all, I couldn't go into and
Speaker 3: tell you, but but we ended up in a position where,
Speaker 3: you know, for a couple of weeks, probably slightly longer,
Speaker 3: we had notes that were going off in one direction
Speaker 3: and others were going in another direction. And I think
Speaker 3: it was a difficult time put it that way, And
Speaker 3: so I think it's I think the most important thing
Speaker 3: for these foundations is that they have, you know, all
Speaker 3: of the distributed or decentralized developers, even if they're generating,
Speaker 3: you know, even if they're developing clients, not just running nodes,
Speaker 3: you know, singing from the same hymnsheet and you know,
Speaker 3: taking the same root forward if you will. So yeah,
Speaker 3: I think that's a good example of where we've seen
Speaker 3: problems on a hard walk. And there are other examples where,
Speaker 3: you know, some other chains are like the Cosmos SDK
Speaker 3: based chains where you can't actually you can't upgrade them
Speaker 3: ahead of time. It's like another thing. Contrast it to
Speaker 3: Web two. Right, So in Web two you've got like
Speaker 3: eight weeks, I don't know something to test your new
Speaker 3: client version. Cosmos SDK, it's like on this block you upgrade,
Speaker 3: so I can't do it two blocks before, ten blocks before,
Speaker 3: a week before. No, on this block you upgrade. So
Speaker 3: it's you know, you don't get the benefit of testing
Speaker 3: and seeing how resilient that latest version of code is.
Speaker 2: Do you you know the thing that keeps going around
Speaker 2: in the back of my mind right now is security.
Speaker 2: H No, I think I am that's sort of my specialty,
Speaker 2: so I tend to get in this area very quickly.
Speaker 2: But I know that there's a lot of madness in
Speaker 2: the world right now, with things like s bombs and
Speaker 2: supply chain attacks. But so whether or not or how
Speaker 2: bad it is is a separate question. But I'm sort
Speaker 2: of curious like the comparison, like, are do you feel
Speaker 2: like malicious attackers coming in through a supply chain attack
Speaker 2: on the tools and technology that you're utilizing to you know,
Speaker 2: run your platform is significantly worse or you know, similar
Speaker 2: to what would happen if you weren't you know, while
Speaker 2: we're working at Visa. Although that's payment, so it's you
Speaker 2: know that that's sort of bad in a different angle.
Speaker 2: But you know, compared to privacy data that a web
Speaker 2: a web two app may be.
Speaker 3: Storing, that's a good question which I probably don't have
Speaker 3: an end depth answer for but I think you know
Speaker 3: you're talking to supply chain attacks, and as much of
Speaker 3: I consume or I buy some either I bring in
Speaker 3: some open source software or I buy a product from someone,
Speaker 3: and there is some kind of attack back to embedded
Speaker 3: within that.
Speaker 2: So it's whether or not you see higher supply chain
Speaker 2: attacks through say like dependencies or open source technologies that
Speaker 2: you're utilizing to manage or monitor your data center compared
Speaker 2: to the ones that would be utilized in a non
Speaker 2: Web three world. So you know, if I run my
Speaker 2: own data center, if I'm AWS or whatever and I'm
Speaker 2: using Grafana, Nagios or whatever, someone you know, no one
Speaker 2: wants to talk about using. Are they the same technologies
Speaker 2: since you have the same concerns or are they modeled differently?
Speaker 2: Are they targeted for web web three? And so do
Speaker 2: you find that the processes that you would put in
Speaker 2: place in your company would be different from the ones
Speaker 2: that would be say up and running and say a
Speaker 2: visa or another large organization.
Speaker 3: Yeah, gotch So I would say reassuringly, that's one of
Speaker 3: the areas where Web two and Web three ten to
Speaker 3: not differ all that much. I think approaches to security
Speaker 3: a fairly standard, and thankfully we've got you know, well
Speaker 3: defined external best practices that influence how we should, you know,
Speaker 3: go about doing business, both in both in Web two
Speaker 3: and Web three. You know, we've recently done Stock two
Speaker 3: audit as well, so we're fully stock too compliant and
Speaker 3: you know we're ongoing we're being audited to that. But
Speaker 3: I think there's another tick in the box that we're
Speaker 3: doing the right things, taking the right approaches, et cetera.
Speaker 3: But in terms of the tooling, I would say there's
Speaker 3: many similarities. You know, we use Graffana to monitor our nodes,
Speaker 3: report on their status, that height, how many requests per
Speaker 3: second they're managing, et cetera. So from that perspective, I
Speaker 3: think a lot of the tooling is similar. I think
Speaker 3: basically the answer to that question.
Speaker 2: Is no, there's nothing specially you're doing compared to what
Speaker 2: you would be doing if you were in any other
Speaker 2: vertical or using any different technology stack. Well, so I
Speaker 2: hitted at this before, Actually, how good are the cloud
Speaker 2: like public cloud? I hate this term public cloud supported
Speaker 2: hyperscalars uh manage blockchain solutions?
Speaker 3: You know, I've never used one, so it would be
Speaker 3: unfair of me to comment in terms of, let's say,
Speaker 3: giving them, giving you a bad impression of them. I
Speaker 3: would expect that they have good documentation based on the
Speaker 3: providers who you know, who develop them. I suspect they've
Speaker 3: got you know, good developer support and run relatively stably
Speaker 3: and were I able to you know, if I could
Speaker 3: say that about all of the blockchains and the nodes
Speaker 3: and the networks we ran, and it would be a
Speaker 3: wonderful thing. But I couldn't say that. So from that perspective,
Speaker 3: I think as an organization, if you're if you're looking
Speaker 3: to do something on blockchain that doesn't necessarily need to
Speaker 3: be public and decentralized, you know they possibly they're a
Speaker 3: good option.
Speaker 2: I'm just bringing up here in my comparison of when
Speaker 2: someone in the UK says something versus the English that
Speaker 2: Americans are supposed to understand. Ah, because there's there's quite
Speaker 2: a nice comparison chart there, But no, I think I
Speaker 2: think that's a good point, and so maybe I'll extend
Speaker 2: it a little bit. Uh. Do you find that the
Speaker 2: conversation of using hyperscalar nodes comes up or it's just
Speaker 2: like not something that's frequently talked about because I don't
Speaker 2: personally understand the use case for what they're providing, and
Speaker 2: I'm just interested that there's like a whole world that
Speaker 2: I've just never been exposed to.
Speaker 3: Well, I think, you know again, I think they're targeted
Speaker 3: toward someone who, let's you know, take any financial services company.
Speaker 3: They don't want to buy, you know, consumer hardware and
Speaker 3: run it in their data center, right. They want to
Speaker 3: go to to IBM or too Oracle, or to some
Speaker 3: you know, some big blue chip organization who's going to say, yes,
Speaker 3: you can have that bit of kit, and with that
Speaker 3: bit of kit, I'm going to give you three years
Speaker 3: unlimited on site support, free replacements, et cetera. You know,
Speaker 3: you can't there isn't There isn't there isn't a blockchain
Speaker 3: STA you could take off the shelf necessarily that has
Speaker 3: that sort of service layer wrapped around it, ready to go.
Speaker 3: And I think that's why, you know, a big enterprise
Speaker 3: type customer may look to approach an implementation with one
Speaker 3: of those managed services or not necessarily managed, but you know,
Speaker 3: broadly supported unknown stacks. I guess now I have in
Speaker 3: terms of conversations I generally get involved in integrating new
Speaker 3: blockchains and ensuring that the ones that we case two
Speaker 3: now are scaled correctly in the right locations, et cetera.
Speaker 3: So personally, I haven't been involved in those conversations. I'm
Speaker 3: sure they will happen, you know. I'm sure our sales
Speaker 3: team are covering all manner of things that I couldn't
Speaker 3: even dream up that are being discussed in the web
Speaker 3: three space. But no, not, not specifically, I haven't an
Speaker 3: approached to integrating one of those.
Speaker 2: What always scale scares me. My sales team is on
Speaker 2: the innovative side, you know that mean things that you
Speaker 2: you know, you clearly haven't developed yet, because are things
Speaker 2: that are going to be coming on the pipeline.
Speaker 3: Could we do this? Or now we've got X and Y,
Speaker 3: can't we make I don't know Z not necessarily right?
Speaker 2: Well, you know, I think this goes both ways. I
Speaker 2: think there's the I think we talk a lot about
Speaker 2: this on a ventures and DevOps, So the audience is
Speaker 2: probably sick of hearing about it. If you haven't pulled
Speaker 2: your customers in to ask them where to innovate, then
Speaker 2: you're probably building things that they don't care about. But
Speaker 2: if they're innovating, then they should be ahead of you.
Speaker 2: And so I think what the important thing is is
Speaker 2: being able to move quickly once you've identified. So a
Speaker 2: sales team doing a good job would mean that they're
Speaker 2: able to figure out what they can promise that hasn't
Speaker 2: been built yet, because I mean, if they're promising things
Speaker 2: that can't be built, that's that's where the issue is.
Speaker 3: Yes, yeah, indeed, indeed maybe, I mean there's an element
Speaker 3: the Websuo space was full of that. I would say
Speaker 3: we used to always pull out this diagram when we
Speaker 3: first met with clients, and it was I think it's
Speaker 3: a typical accenter diagram, you know, the rope swing, where
Speaker 3: you know they this is what they wanted, this is
Speaker 3: how their requirements were gathered, this is this is what
Speaker 3: the developers built, and this was the MVP that got
Speaker 3: on it. You know, it had all the elements of
Speaker 3: a rope swing, but it ain't a rope swing.
Speaker 1: That's such a great meme if you if you haven't
Speaker 1: seen that, maybe we can put it in the show
Speaker 1: notes for you because it's just it's just so fantastic. So, Paul,
Speaker 1: one of the things I wanted to ask you is
Speaker 1: that how many different chains are you supporting at anchor?
Speaker 3: If I think about my most recent table, I think
Speaker 3: it was over one hundred cells. Now that there may
Speaker 3: be sorry, a hundred rows. Still I still make my
Speaker 3: own notes manually, right, even in the words three digital space,
Speaker 3: I'm still still making a confidence table just to keep
Speaker 3: just to keep things organized. But yeah, it's well, well
Speaker 3: over one hundred. I couldn't give you an exact number
Speaker 3: because I think even today one of my team has
Speaker 3: been finishing golf implementing one. So yeah, it's constantly evolving.
Speaker 1: I know. Wa it looks that is it's one hundred
Speaker 1: independent products that your team is supporting. And then we've
Speaker 1: already talked about the issues with hard forks and making
Speaker 1: sure that they're staying in sync and operating correctly. So
Speaker 1: you're doing that across one hundred different products, which seems
Speaker 1: like a lot to take on.
Speaker 3: It is that now there are there are various ways
Speaker 3: let's say one approaches that, and you know, let's say
Speaker 3: teams might approach that, but for the for the most part,
Speaker 3: there are there are there are only a handful, let's say,
Speaker 3: of unique clients that blockchain teams use. And so what
Speaker 3: I mean by that is, you know, Aragon is a
Speaker 3: perfect example there. We can use that for Polygon, we
Speaker 3: can use it for BMB smart chain, we can use
Speaker 3: it for ethereum, et CEPA and separate, and you know
Speaker 3: even some teams have taken Aragon and made and early
Speaker 3: I don't know. I mean it's testing prod right, So
Speaker 3: it's like an Aragon that we can use on the
Speaker 3: optimistic roll ups, et cetera, et cetera. So so there
Speaker 3: are obviously a good word used to using the web
Speaker 3: two space synergies and ways we can converge. It's lovely buzzwords.
Speaker 3: You know, how you approach things, right, So let's say,
Speaker 3: for example, you need to host an op roll up
Speaker 3: or even one of these most recent Eragon three, even
Speaker 3: the Aragon free client they've recently released. You could write
Speaker 3: a doc com post file and if you've had enough
Speaker 3: environment environment variables in that doc com post file, you
Speaker 3: could take it and run one chain with it, and
Speaker 3: then you just create another environment variable file and run
Speaker 3: a second chain and run a third chain, et cetera. So, yes,
Speaker 3: it is, it is. It is an overhead. I won't
Speaker 3: lie that. You know, obviously we're monitoring twenty four to
Speaker 3: seven we get alerts, and from that perspective, it does
Speaker 3: seem like a lot, But there are learnings from running
Speaker 3: one chain that we can overlay on another. There are
Speaker 3: ways to make certain clients behave better when they're pairing
Speaker 3: and sinking and you know, being able to stay at
Speaker 3: the tip. There are ways that we can configure clients
Speaker 3: that make them better in terms of responsiveness and latency
Speaker 3: for various requests, you know, by adjacent orbc on. All
Speaker 3: of those learnings you then roll out when you do
Speaker 3: the next integration of a similar client, et cetera, et cetera. So, yeah,
Speaker 3: it is a lot, but you know, we have to
Speaker 3: take from that what we can and reuse wherever we can.
Speaker 2: When you when you have a hundreds of one hundred
Speaker 2: products that you're supporting here, it's not like you have
Speaker 2: a multi tenant solution where you have one hundred customers
Speaker 2: and they're all utilizing your product in a consistent way.
Speaker 2: I assume you're having you have to build integrations into
Speaker 2: each of those products to be able to understand how
Speaker 2: they're working, and not all of them are using the
Speaker 2: same protocols to do the management like it's like if
Speaker 2: one company had gRPC and another company was using you know,
Speaker 2: and weird other proto buff format, and then there's h GDP,
Speaker 2: and then there's someone's using rest or some other like
Speaker 2: each each company is basically conceiving of their own protocol.
Speaker 3: Yeah, yeah, it's it's a it's a good call out.
Speaker 3: So actually generally the conversation we we've had has been around,
Speaker 3: you know, the operations of blockchain nodes. Obviously, the other
Speaker 3: elements of what we do here are anchor is we
Speaker 3: you know, we we've built our own cloud native you know,
Speaker 3: many multi protocols supporting load balancer, which is you know
Speaker 3: again it's a distributed load balancer in all the locations
Speaker 3: we want to be and we're we're building out our
Speaker 3: own global network as well, and so we can there.
Speaker 3: As I mentioned, there are similarities in how we run
Speaker 3: nodes because they're similar clients. Generally it's configuration and best
Speaker 3: practice approach to making those run the right way. But
Speaker 3: in in the load balancer from the load balance aside,
Speaker 3: because obviously that serves the customer request that are coming in.
Speaker 3: We we do have to do differentiation there as well.
Speaker 3: So as you mentioned, you know, you have like an
Speaker 3: ethereum like client, so it talks roughly we call that, Oh,
Speaker 3: it's an EVM client, right, or you get some nu
Speaker 3: answers to that it's EVM light, or you know, it's
Speaker 3: EVM but only up to a certain point, so it
Speaker 3: doesn't have these other new name spaces and new mesic calls,
Speaker 3: et cetera. And we're able to, you know, we abstract
Speaker 3: from that detail of the node, you know, effectively like
Speaker 3: a schema into the load balances. So the load balancer,
Speaker 3: you know, knows what that node can speak in terms
Speaker 3: of protocols and what it can speak in terms of
Speaker 3: you know, the APIs and the actual messages within that protocol. Yeah,
Speaker 3: there's a lot that we need to have in place
Speaker 3: and maintain, and but thankfully we're at the point now
Speaker 3: where generally the changes in the load balancer level are
Speaker 3: incremental and generally the changes that we make in terms
Speaker 3: of the approach to running the nodes is a g
Speaker 3: and it's incremental, and it's built upon, you know, the
Speaker 3: knowledge that we built up out the last of years
Speaker 3: just running these nodes all the time.
Speaker 1: From my understanding, like load balancing is not load balancing
Speaker 1: as we think about it. In a Web two world, right,
Speaker 1: in a Web two world, you know it's the health check.
Speaker 2: Cool.
Speaker 1: Yeah, the health check is cool. We're pretty much okay.
Speaker 1: It's the route traffic to it. But in the Web
Speaker 1: three world, it's more than that, because you have your
Speaker 1: health check. Yeah, the service is up and available and
Speaker 1: respond it's responsive and able to take requests. But then
Speaker 1: you've got a follow up question. This node is operating,
Speaker 1: but is this node fully synchronized with the network, and
Speaker 1: if it's not, you can't route traffic to it. And
Speaker 1: then the third aspect of it is what's the request
Speaker 1: that's coming into the load balancer? For example, is this
Speaker 1: is it asking about a recent block or is it
Speaker 1: asking about a really old block that's only going to
Speaker 1: exist on archive nodes. And so now with that information,
Speaker 1: you know that there's only certain nodes that you can
Speaker 1: route this request to to give the caller the correct
Speaker 1: information back. And so that adds a whole new layer
Speaker 1: of complexity to load balancing.
Speaker 2: I've got the analogy, I think. So if we go
Speaker 2: on the databases, like could you imagine running Reddus and
Speaker 2: Cassandra and my squl and an Aurora database and elastic
Speaker 2: search and having like whatever consensus protocol you had for
Speaker 2: figuring out like which is the primary nodes, which ones
Speaker 2: are secondary, and where to route requests automatically to the
Speaker 2: appropriate shards and also manage the infrastructure for that with
Speaker 2: only using a single piece of technology rather than using
Speaker 2: you know, dedicated pieces like separate pieces. Like that's that's
Speaker 2: as I see the problem.
Speaker 3: Yeah, and so maybe it would be I have many
Speaker 3: retics is what's redici red eye?
Speaker 2: You get decording the term. I think you are the
Speaker 2: first person to have.
Speaker 3: So so yeah, maybe you would have many of a
Speaker 3: particular database instance type let's call it, and then you know,
Speaker 3: many of another one and many of another one. But
Speaker 3: but yeah, I mean it's challenging. There's not a day
Speaker 3: goes by where we're not discussing what the algorithm should
Speaker 3: be to load balance requests, right, And I don't mean
Speaker 3: that in a way that you know, we're immature and
Speaker 3: we're trying to build the algorithm. It's more every day
Speaker 3: we learn something new. Every day we learn, you know,
Speaker 3: or we get a new customer who interacts with our
Speaker 3: service in a slightly different way and they do a
Speaker 3: you know, maybe they do a slightly different sequence of
Speaker 3: calls to ultimately maybe achieve the same output as a
Speaker 3: different customer, and you know what would that mean in
Speaker 3: terms of how their requests are handle coming in and
Speaker 3: how they're rooted.
Speaker 1: And I think.
Speaker 3: You know this shouldn't be taken as a failure. It's
Speaker 3: like it's an acknowledgement of effectively we don't believe there's
Speaker 3: a perfect load balancing algorithm. I mean I I don't.
Speaker 3: I'm sure that you know, some of the younger members
Speaker 3: of the team would have their own subjective opinion about that, right,
Speaker 3: But I think you know, you get to the you
Speaker 3: get to the point given the volume of requests. You know,
Speaker 3: we're talking tens and tens of thousands a second right
Speaker 3: of requests that are coming in to be at ninety
Speaker 3: nine point nine nine percent. You know that's that's achievable,
Speaker 3: perfect divaspirational. You know, we as we discuss internally, and
Speaker 3: so we have anyway to get back to the question. So, yes,
Speaker 3: yes it's complex, and yes there is. Let's say you
Speaker 3: need to have a more fine grain control of where
Speaker 3: you root a request based on a better underlying knowledge
Speaker 3: of what that request is and what it's you know,
Speaker 3: what it's intended outcome is the perfect example is the
Speaker 3: one you gave. You know, a request comes in, it's
Speaker 3: for a I don't know, let's say block one thousand
Speaker 3: nine ethereum, that's got to go to an archive. Know
Speaker 3: that's not that block is you know the state for
Speaker 3: that block and all the information is not going to
Speaker 3: be on a full node, and we have to know
Speaker 3: when it's coming in. We have to interpret that and
Speaker 3: root it to a particular archive known. So we have
Speaker 3: you know, we have ruled in the load balance of
Speaker 3: that the gear towards rooting those requests correctly.
Speaker 2: I think some people may be cheering, and I think
Speaker 2: others are gonna regret that I asked this question. Has
Speaker 2: AI had any impact on the web? Threw world for you?
Speaker 1: Where we go? There's the magic word?
Speaker 3: So you know what. I listened to the last podcast
Speaker 3: and it's interesting how you you almost tentatively or sheepishly
Speaker 3: approach having this question, you know, bringing this AI question in.
Speaker 3: So I'll be honest, right, So from a first of
Speaker 3: all standpoint, had I embraced the last oh and the
Speaker 3: last to be honest and the last start of I
Speaker 3: was in, if I'd embraced AI, I would have done
Speaker 3: stuff in half the time, and I didn't at that point,
Speaker 3: and more than anything that was, So this is just
Speaker 3: a person I'll say, I'll go on, I'll go on
Speaker 3: to bag stuff in a moment, more than anything that was,
Speaker 3: because I would needed to make sure I knew how
Speaker 3: it worked right. I was the only guy, or you know,
Speaker 3: one of a very small team doing development, and I
Speaker 3: felt like if I had I could say myself, I no,
Speaker 3: three weeks, six weeks here, But what's that going to
Speaker 3: cost me when I try and maintain this and operate
Speaker 3: it in a production environment, you know, for customers. So
Speaker 3: I shout away from it a bit there. Now at Anchor,
Speaker 3: we have embraced AI at the moment, we're learning and
Speaker 3: it is learning from us. You know, it understands blockchain,
Speaker 3: It understands that we've got a load balancer. It understands
Speaker 3: the end goal of what we're trying to do. But
Speaker 3: there are still some instances where we're asking it questions
Speaker 3: and we get a confident answer that we know isn't right.
Speaker 3: So yeah, we're just working to try and feed it
Speaker 3: more and take effectively take away some of those you know,
Speaker 3: easy to answer questions or easy to diagnose problems, and
Speaker 3: we allow the moniker do that for us, and we
Speaker 3: can ask a questions to try and assisters troubleshooting and
Speaker 3: assistance with helping the customer. But you know, it's early days,
Speaker 3: and I can see it rapid. It's already evolved massively,
Speaker 3: and I can see it only you know, the curve
Speaker 3: is exponential with these things, right. I can see it
Speaker 3: getting much more capable as the months go on, and
Speaker 3: I think we'll be levering leveraging it more and more.
Speaker 3: We like it, and I think used for the right thing,
Speaker 3: it's a great fol.
Speaker 1: When you make statements like that, it seriously makes me
Speaker 1: question if I might be an AI, because I'm always
Speaker 1: super confident and also at the same time usually super wrong.
Speaker 3: You're just another person on the internet, right, So what
Speaker 3: do you use.
Speaker 1: For what's your tool stack from an opside look like?
Speaker 1: Are you guys using a lot of terror form answerable?
Speaker 1: Do you do Subernetes stuff?
Speaker 3: What's that world look like? Yeah? Absolutely so, we we do.
Speaker 3: We use terraform. We on the node op side generally,
Speaker 3: we've we've we've created our own template if you will,
Speaker 3: that we consume and create terraform from that we then
Speaker 3: go off and manage the state on the hardware through
Speaker 3: if you see what I mean. So it's similar to
Speaker 3: what I mentioned before about if you have enough, if
Speaker 3: you have a dot com post file with enough environment variables,
Speaker 3: you can spin up any number of networks. And we've
Speaker 3: done a similar thing in terms of how we interact
Speaker 3: with how we'd launch nodes based on using terraform to
Speaker 3: do that launcher, but having this template in place to
Speaker 3: simplify things effectively, and yeah, we use answerable, we use
Speaker 3: all day long. We use a w X you know,
Speaker 3: to schedule playbooks running across hosts et cetera, you know,
Speaker 3: to push out security updates and that kind of thing.
Speaker 3: But yeah, we we split it. So we have some
Speaker 3: nodes that are you know, fully if not, if not
Speaker 3: close to fully automated, and we have others where our
Speaker 3: preferences to effectively approach the bear metal manually and set
Speaker 3: up the node in such a way because it's so
Speaker 3: different from the other ones, you know, we set it
Speaker 3: up in such a way that we can you know,
Speaker 3: do it right for a start, and then obviously create
Speaker 3: our own documentation off the back of that about how
Speaker 3: to maintain that so so yeah, we do. We do
Speaker 3: use quite a bit of Hashi cup stuff. I think
Speaker 3: the last podcast I listened to I was I was
Speaker 3: reading about open Tofu after that because someone was talking
Speaker 3: about open Tofu quite a bit, so I was actually
Speaker 3: reading about that this morning. So yeah, Hashi Cup stuff.
Speaker 3: And then in terms of other automation, generally it's running
Speaker 3: oh sorry, kubernettes. You mentioned we do run kubernettes, not
Speaker 3: not for our not for the RPC nodes. I would say,
Speaker 3: we use it in other area of the areas of
Speaker 3: the business. So where we want you know, bear metal
Speaker 3: close to bear metal performance, least latency, high high capability
Speaker 3: in terms of serving requests, will generally, you know, either
Speaker 3: do it via the auto deployer, which is which is
Speaker 3: powered by which we use Nomad behind another Hashi Court product,
Speaker 3: or will do them manually. And then if it's a
Speaker 3: lighter client and you know, recently launch like a small
Speaker 3: test net that we don't expect to run for ages,
Speaker 3: we'll use Kubernetes. So and you know, there are other
Speaker 3: areas of the business that also use Kubernetes outside of
Speaker 3: the note operations. So so yeah, we use quite a
Speaker 3: few of those orchestration tools and Hashi Court products.
Speaker 1: And it seems like there's probably an adoption phase for
Speaker 1: those as well. Whenever you bring on a new chain
Speaker 1: doing like doing things manually, kind of hand curating it
Speaker 1: to figure out what the best the best set of
Speaker 1: configs and the right things to be monitoring and tuning
Speaker 1: for this are. And then as you learn that in
Speaker 1: from you start turning that into turning it over to
Speaker 1: your automation.
Speaker 3: Yeah. Absolutely that. Yeah, so one of the things actually,
Speaker 3: that was my surprise this morning. So I've been working
Speaker 3: on new blockchain integration and so usually when I do
Speaker 3: the manual one, I've deliberately set it up so system
Speaker 3: D or Docker doesn't restart it automatically, because I want
Speaker 3: to know when that from the point I started. I
Speaker 3: want to see it fail and you know, see what
Speaker 3: that longevity is if you will. And yeah, that was
Speaker 3: my surprise this morning, Like four nodes were down and
Speaker 3: I was like, oh, why are these down? Surely I've
Speaker 3: set these to restart, And it's because I'd set them
Speaker 3: to not restart so I could see the point at
Speaker 3: which they fell over. But yeah, it is an element
Speaker 3: of that what we generally do as part of the
Speaker 3: integrations now and this is you know, something I've put
Speaker 3: in place since I started. So we want to make
Speaker 3: sure that we hit if we're going to do manual nodes,
Speaker 3: and we agree that that's going to be, you know
Speaker 3: how we run most of the nodes, we always have
Speaker 3: at least one or two automated nodes as well. And
Speaker 3: that's simply because as long as we have the template
Speaker 3: in place to do the automation, we can scale at will.
Speaker 3: You know, we can just come up and put another
Speaker 3: template and with various variables and we can scale some others.
Speaker 3: So yeah, we don't like to leave ourselves in a
Speaker 3: position where we've kind of got nowhere to go, or
Speaker 3: we have to set up another host manually in a
Speaker 3: specific way to run this particular blockchain note. So yeah, yeah,
Speaker 3: it's it's exactly that approach.
Speaker 2: So where we left off last time with web how
Speaker 2: Web three is going. We had n f T s
Speaker 2: and I think the general population failed to understand n
Speaker 2: fts in any capacity. So that went well, But I
Speaker 2: am curious, like what the what, what the what? Where
Speaker 2: the innovation is at today? Like you know, where where
Speaker 2: you see that either where it's currently at or you
Speaker 2: know what's coming next that is super interesting for you.
Speaker 3: Yeah, yeah, that's that's a great question. And you know,
Speaker 3: similar to the break, the break that I had when
Speaker 3: you know, when I wasn't working with Peter and he
Speaker 3: came to web three. When I asked him, or what's that?
Speaker 3: What's happened? You know, has anything changed? Actually his responsible
Speaker 3: were they're just Linux services, right, And I was like, surely,
Speaker 3: there's clearly there's something more than that. But of course,
Speaker 3: the you know, the the L two space had evolved
Speaker 3: in that period where I, you know, hadn't been working
Speaker 3: explicitly in blockchain. But I think I think while that's
Speaker 3: while that might not necessarily be where we see the
Speaker 3: greatest innovation, I mean I might be wrong. I see
Speaker 3: that for me and my team, I see that as
Speaker 3: as a great learning opportunity because there are lots of
Speaker 3: you know, new toolkits that are available and new ways
Speaker 3: of running L two's and you know, soon to be
Speaker 3: L three's, and who knows how how many layers that
Speaker 3: we're going to see and so from so from my
Speaker 3: perspective as well as let's call them the the general
Speaker 3: integrations we do for blockchain networks, it's it's good for
Speaker 3: me and my team to have access and be exposed
Speaker 3: to these layer twos because you know, it's a good
Speaker 3: learning opportunity. They also, you know, we run this role
Speaker 3: up as a service where you can literally rock up,
Speaker 3: define new shame parameters, and launch your own blockchain if
Speaker 3: you want to, as an L two. And yeah, I
Speaker 3: think that that makes it so accessible for people. So
Speaker 3: I think that is where we're going to see a
Speaker 3: lot of innovation, certainly, you know, for me and my team,
Speaker 3: I think it's where we're going to learn a lot
Speaker 3: more and have access to these other technologies, other development stacks,
Speaker 3: et cetera.
Speaker 2: Okay, so you've you've you've gotten to the point where
Speaker 2: you're actually outside of my area of understanding when you
Speaker 2: say like layer two or layer three, we're talking about
Speaker 2: like lightning network, Like what does that actually mean in
Speaker 2: the context of a blockchain?
Speaker 3: And and L two allows you to run blockchain which
Speaker 3: at various intervals generates and stores state in a transaction
Speaker 3: on an L one. So a lot of these are
Speaker 3: used ethereum as there L one, which is effectively the
Speaker 3: the data availability layer. So if you needed to, let's say,
Speaker 3: bootstrap another L two node, you could based on the
Speaker 3: state that's been stored on the L one. I'll be honest,
Speaker 3: this is going past some of my understandings. But effectively,
Speaker 3: at L two allows you to sort of, you know,
Speaker 3: branch off and have transactions between wallets et cetera, et cetera,
Speaker 3: and all of that happens on the L two and
Speaker 3: periodically it then records that state on the L one. Now,
Speaker 3: the differences, the reason for this is, you know, obviously
Speaker 3: there are you can only do a certain number of
Speaker 3: transactions per second based on the block time and the
Speaker 3: size of the blocks on each of the blockchains. Now,
Speaker 3: if you separate all that transaction activity onto an L
Speaker 3: two and only at various intervals record that state onto
Speaker 3: the L one, it means that you've got less going
Speaker 3: on on the L one, and therefore, in theory, you
Speaker 3: can have many more transactions with fewer confirmations and the
Speaker 3: full transactions on the L one. So people call it
Speaker 3: block space, block space, lots more block space and lots
Speaker 3: more transactions. But I shouldn't say well, because someone might
Speaker 3: say you don't know what you're talking about, so don't
Speaker 3: go on these podcasts again.
Speaker 2: Well, I think we'll definitely let you back on. But
Speaker 2: I think we're not the official keepers of that, but
Speaker 2: I will I will ask you here. I always had
Speaker 2: this fear that it's sort of like you have your
Speaker 2: enterprise service bus and now you're building some micro services
Speaker 2: on top of that, and they're storing intermediary state in memory.
Speaker 2: Like for these higher layers, there must be some risk
Speaker 2: with the layer collapsing in some way or creating a
Speaker 2: conflict between different isolated parts that are on the same
Speaker 2: layer that would cause a conflict on that based chain,
Speaker 2: And like, how does that get resolved or is that
Speaker 2: even a problem that is concerned.
Speaker 3: That's a deeply technical question which I'm trying to think
Speaker 3: the best way to dodge.
Speaker 2: So you can say anything, because I honestly still to
Speaker 2: this day, don't have the answer to this question.
Speaker 3: But yeah, so I should do the AI trick, right,
Speaker 3: I should just I should have just responded confidently. Yeah, yeah, so,
Speaker 3: I mean, I'll be honest, I haven't seen instances of
Speaker 3: that problem. I'm certain that they must exist, but if
Speaker 3: you will, I could almost theorize how in my work,
Speaker 3: But maybe that's a bit dangerous. But you know, even
Speaker 3: on an l one you can get to a point
Speaker 3: where a longer chain is submitted with a great number
Speaker 3: of blocks, and therefore it unwinds some of the other
Speaker 3: blocks in the chain, and you know that everyone follows
Speaker 3: the longer chain, right, So my understanding is it would
Speaker 3: work the same way with an L two. You know,
Speaker 3: there could be a point at which the state hadn't
Speaker 3: been recorded, and therefore it kind of reflects back to
Speaker 3: the state prior to that, and then you build back
Speaker 3: up to what is this next state that it needs
Speaker 3: to be in terms of the transactions that need to occur,
Speaker 3: et cetera. But a note to self, I'm going to
Speaker 3: do research on that one.
Speaker 2: Well, I mean, there's the canonical and you worked in payments,
Speaker 2: so maybe there's some insight here. Like you don't want
Speaker 2: to have to have a consistent state amongst all customer
Speaker 2: all users in the world that have a visa credit card,
Speaker 2: you know, if they're making a transaction, you want to
Speaker 2: bucket them. So like if you're in bank processing world,
Speaker 2: you want to allow people to send money to each other.
Speaker 2: Maybe there's a regionality, so your L two's only exist
Speaker 2: in like in one country, in the likelihood of cross
Speaker 2: country transactions is low. And when that happens, then you
Speaker 2: have to ensure that you have a consistent understanding of
Speaker 2: what the l one chain is. But other than that,
Speaker 2: you just, you know, you risk it because you're blocking
Speaker 2: transactions in some way that are happening to that chain
Speaker 2: outside and you can't fully trust that because someone could
Speaker 2: be doing something in one of those other you know,
Speaker 2: independent same layer. But I see that there are some
Speaker 2: opportunities there. But like as you said, you know, we
Speaker 2: can theorize, but we're not the experts on this. That's fine,
Speaker 2: I got it.
Speaker 1: It's a it's a deep, deep, deep rabbit hole.
Speaker 3: Oh yeah, well indeed, And you know, I think I
Speaker 3: think abstracting from the technical complexities of building a blockchain
Speaker 3: client and a you know, consensus mechanism, that's what those
Speaker 3: technical people are for, I would say. But yeah, right,
Speaker 3: it's a great question.
Speaker 1: I think that's a way of saying, it's someone else's problem.
Speaker 3: I hire someone to solve that for me, you know,
Speaker 3: so I don't have to know I see the problems
Speaker 3: when they're not necessarily how they intended to solve them.
Speaker 1: You know, you've got deep experience both in Web two
Speaker 1: and Web three, So if you could share one piece
Speaker 1: of a Web two learning with the Web three crowd.
Speaker 1: What would that be?
Speaker 3: So less haste, more speed. I think that that kind
Speaker 3: of comes back to the point I made earlier on
Speaker 3: about rigor around processes, testing, you know, time to adopt
Speaker 3: new versions, et cetera. That kind of thing. I think.
Speaker 3: You know, there are there are great innovations and you know,
Speaker 3: great inventions in the Web three space. But I think
Speaker 3: I think, yeah, that we won and even organizationally, we
Speaker 3: should afford ourselves the time to do them properly. And yeah,
Speaker 3: you know, I mean that's that's kind of what I've
Speaker 3: been doing and saying since I since I came into anchor,
Speaker 3: you know, for me, right, So imagine for note operations. Look,
Speaker 3: I don't care that you can update sixteen nodes in
Speaker 3: five seconds. That is not That is not what I'm
Speaker 3: here to do. I want to know that there's at
Speaker 3: least X number of nodes online at all times in
Speaker 3: these locations. So you do them one at a time, slowly, sequentially,
Speaker 3: but make sure you do it right and it all happens. So, yeah,
Speaker 3: that that would be my learning from Web two to
Speaker 3: work three, less hate, more space.
Speaker 2: I think there's a corollary here because one of the
Speaker 2: I think you mentioned this earlier on in the episode,
Speaker 2: that one of the ideas with Web three is let's
Speaker 2: forget everything we did with Web two and like start
Speaker 2: all over again. But there have been innovations I think
Speaker 2: in Web two in the last twenty years that even
Speaker 2: before that that were discarded that would benefit people to
Speaker 2: sort of pay attention with. And I think there's this
Speaker 2: aspect of experience from cross industry. And I see that
Speaker 2: for like a lot of blockchain companies are like only hiring,
Speaker 2: you know, you must have blockchain experience, must have you know,
Speaker 2: Web three app development experience. And I'm like, okay, but
Speaker 2: for sure experience outside of that would be really useful.
Speaker 2: And I think especially around the processes, it's in a
Speaker 2: lot of ways, it's still software development, it's still product management,
Speaker 2: it's still you know, business intelligence. And I hate that
Speaker 2: term BI, but you know, I'll use it here as
Speaker 2: an example what you can pull from other companies. So
Speaker 2: I really like that, and I am I'm dying to
Speaker 2: hear the corollary to Will's question.
Speaker 1: Yeah, that flip the question around what is one thing
Speaker 1: that Web two could take away from Web three?
Speaker 3: Yeah, I mean there was, Yeah, I mean do you
Speaker 3: mean specifically when we're like a particular sector or it's
Speaker 3: or generally I suppose, I mean, you know, so the
Speaker 3: flip the flip. The flip to the answer is why
Speaker 3: why wouldn't you just test everything in production? Right? What
Speaker 3: do you mean? What do you mean less speed? You know,
Speaker 3: I remember, I remember even in the Web two I mean,
Speaker 3: you know we did. We did in various roles in
Speaker 3: the past. I won't going into which ones they were.
Speaker 3: You know, we've we've we've done tests in production which
Speaker 3: where we literally dragged the business over the line and
Speaker 3: got them to let us put this thing in production
Speaker 3: and do the test. So I think I think on
Speaker 3: both sides. You know, Web three can learn from Web
Speaker 3: two in terms of more rigor around pro is, you know,
Speaker 3: affording themselves the time to do things correctly and write.
Speaker 3: And similarly Web two can learn from Web three in
Speaker 3: that regard, you know, be more open to innovation, embrace
Speaker 3: it more and and you know, take and take more risks.
Speaker 3: In some cases.
Speaker 1: It goes back to the joke I made a long
Speaker 1: time ago that true ci c D is VIM on
Speaker 1: the Pride server.
Speaker 3: You know, my brother would love that. He's he always
Speaker 3: goes on and on still about them. And I was like,
Speaker 3: have you tried visual Studio and he was like, no,
Speaker 3: you know vs coast. I remember the first time I
Speaker 3: saw VIM and I was watching over some someone's shoulder.
Speaker 3: It was a chef screen and I was looking at him, going,
Speaker 3: what is he doing? He's going to delete this slide?
Speaker 3: And because I was used to using NANO, I thought
Speaker 3: it was going to be an absolute disaster, But so
Speaker 3: there actually is.
Speaker 2: Maybe a slightly a different tangent. I I ser to
Speaker 2: see this pattern where malicious attackers are utilizing public APIs
Speaker 2: associated with blockchain companies technology to store smuggle out encrypted
Speaker 2: data from their victims. So they'll take data from a
Speaker 2: victim's environment, they'll encrypt it with the public version of
Speaker 2: a key that they have, and then upload it to
Speaker 2: a public blockchain. And I'm curious if you've heard about this,
Speaker 2: if you've seen something like this and it's a known problem, Like,
Speaker 2: I don't know what there's a solution of this, honestly,
Speaker 2: and just like what can be done? Because I feel
Speaker 2: like it's sort of these things where in the past
Speaker 2: we had lots of these data sharing sites where you
Speaker 2: can drop a file or someone hirates a movie or
Speaker 2: something or some music and puts it on there, and
Speaker 2: then they always end up getting shut down and because
Speaker 2: they're a source for this illegal content. And I don't
Speaker 2: know that there's something obvious that can be done with
Speaker 2: watchain technologies out there, which you have a public ledger
Speaker 2: in a lot of cases.
Speaker 3: Yeah, yeah, that's a good that's a good one. And
Speaker 3: we see, in fact, we see more and more blockchain
Speaker 3: based storage solutions coming, you know, as the month and
Speaker 3: years go by. Honestly, I mean, we need to sort
Speaker 3: the users out, don't we. That's the problem. That's someone
Speaker 3: you know, you're not gonna be able to find me tomorrow, Howards, right,
Speaker 3: because someone's going to have hacked my computer and the
Speaker 3: encrypted my data and put it on a blockchain network,
Speaker 3: can't they now have said that? But yeah, I think
Speaker 3: I think the problem it's a pepcac right. The problem
Speaker 3: exists between the keyboard and the chair in that instance. Now,
Speaker 3: if we could stop users being if we could stop
Speaker 3: users being taken advantage of and having this information stored,
Speaker 3: and that's the root cause for me, and that's what
Speaker 3: we should go off and fix. I think in terms
Speaker 3: of because I would say I would champion you know,
Speaker 3: the blockchain space doing these you know, public accessible data
Speaker 3: availability layers and hosting solutions. I think it's I think
Speaker 3: it's a good thing.
Speaker 1: There was one and I can't remember the exact details
Speaker 1: of it, but you'll, I think you'll appreciate this one.
Speaker 1: There was a similar strategy, but instead of taking the
Speaker 1: data and uploading it to the blockchain, they would take
Speaker 1: it and then submit it as a transaction to the blockchain.
Speaker 1: But intentionally price is so low that it would never
Speaker 1: get mined. So now the transaction exists on the chain,
Speaker 1: but it never gets written to a block so you
Speaker 1: can't really track it down that way, and you read it.
Speaker 2: Out you read the data out there because the transactions
Speaker 2: are being traced and monitored in the network. And then
Speaker 2: when it gets dumped, it's not on the chain. You
Speaker 2: know it's gone, and so you have to actually look
Speaker 2: at the analytics data of what had happened. It's not
Speaker 2: even you know, available for all that is ingenious. I
Speaker 2: hope there aren't any you know, people with malicious intent
Speaker 2: that are watching this podcast.
Speaker 1: Oh, entirely unlikely.
Speaker 3: You never know, You never know yeah, I don't know.
Speaker 1: It went way over my head.
Speaker 2: Yeah, I mean it's probably if you do enough times,
Speaker 2: it will be sitting there that someone will like eventually
Speaker 2: it's on the it's in the queue to actually get
Speaker 2: turned into a block. I mean, there was a it
Speaker 2: was a great post a while ago about making databases
Speaker 2: out of things that do not that should not be
Speaker 2: a database. So one of them was TC using icmp
Speaker 2: pings and so you put some data in the ic
Speaker 2: ic m P header and you send it to a
Speaker 2: random IP on the internet and then you know, you'll
Speaker 2: get the ping back with the payload and you can
Speaker 2: you know, use it and so like in femoral database
Speaker 2: at that point, and I like, that's super unreliable. But
Speaker 2: this is this is like an extension of this is
Speaker 2: like a whole system, you know, built in design to
Speaker 2: actually utilize this data in a reliable way.
Speaker 3: Yeah.
Speaker 1: Could you just need it to hold the data until
Speaker 1: you get out of the building or whatever.
Speaker 2: Oh, I mean in a lot of these cases, it's
Speaker 2: not a you haven't trespassed like physically to right, Yeah, yeah,
Speaker 2: you just basically you know, you've deployed some malicious tool
Speaker 2: to through MPM or pipe or anything else someone downloads
Speaker 2: that they lost their credentials, or you infiltrate an organization
Speaker 2: and you have some keys to the database and you
Speaker 2: upload those automatically. You just sit there querying all of
Speaker 2: the block scanners that you know report transactions for any
Speaker 2: single chain, and you're just like, oh, there it is,
Speaker 2: there's my there's my transaction. You don't even have to
Speaker 2: pay for anything, right, I mean, you could put like
Speaker 2: a miniscool amount of money associated with the particular chains,
Speaker 2: you at least get the transaction started, and other than that,
Speaker 2: you know it will never complete and you don't have
Speaker 2: to worry about it if you just pull it off
Speaker 2: and you just scan and get the data.
Speaker 1: That seems like a good time to move on to picks.
Speaker 1: Now that we've shared that information with everyone.
Speaker 2: Well, if it's pick time, then I guess I'm I'm
Speaker 2: up first bringing on. Okay, so a few episodesis sodes ago.
Speaker 2: I started sharing my keyboard and I think I got
Speaker 2: some questions about what the heck I'm actually using. So
Speaker 2: first of all, let this be the official pick for me.
Speaker 2: I use a Divorac keyboard programmer Divorac on Linux, where
Speaker 2: I've remapped all the keys as well, So I highly
Speaker 2: recommend doing this, Like if you work in multiple currencies,
Speaker 2: change like the dollar on the on the keyboard layout
Speaker 2: to also do the euro sign and maybe the yen
Speaker 2: or one like whatever you're utilizing. Like it's just so
Speaker 2: much easier every single time you want one of those,
Speaker 2: Like imagine if you had a magic emoji button on
Speaker 2: your keyboard. I've basically done that, but I also want
Speaker 2: to share the keyboard because I absolutely love this. So
Speaker 2: what I have is, uh, it's a logic Logitech K
Speaker 2: two ninety five silent keyboard, and this thing is so quiet,
Speaker 2: like I can type on it while I'm on the
Speaker 2: podcast and no one will ever know. That's how quiet
Speaker 2: this is. And that's important because if I'm too loud, well,
Speaker 2: other people that I live with will find out exactly
Speaker 2: what I'm doing whenever I'm doing because I am an
Speaker 2: angry typer.
Speaker 1: Yes, cool deal, Well what'd you bring for a pick?
Speaker 3: So I inherited a number of records recently. You know,
Speaker 3: if there's kids listening these big black discs twelve thirty
Speaker 3: three because because my father passed away and so we
Speaker 3: sold his stereo system. You know, an old an old
Speaker 3: silver techniques thing it was, and so I've been I've
Speaker 3: been looking because I wanted to get myself a record player.
Speaker 3: So that that's my pick for today. So I think
Speaker 3: for two reasons. Number one is like we need to
Speaker 3: not lose the tactility, right. I know a lot of
Speaker 3: people everyone's all about digital and web three and all
Speaker 3: this wonderful thing, but I think we're slowly losing possessions
Speaker 3: as people and and I don't think that's that In
Speaker 3: some cases that's great and others I don't think it's
Speaker 3: that great a thing. So yeah, I want to I
Speaker 3: want to get a record player, which is to play
Speaker 3: these old records that I have and I may increase
Speaker 3: my collection. Now the particular record player is the interesting bit.
Speaker 3: So I thought, I try and look for something that
Speaker 3: was quirky, you know, new different, and you don't get
Speaker 3: that a lot with record players, right, it's usually and
Speaker 3: a stylus. Now I found this record player by a
Speaker 3: company in the Netherlands, and I think they're called mini
Speaker 3: Ot or Miniot, and they have a record player which
Speaker 3: can you can have vertical or horizontal and it actually
Speaker 3: plays the backside of the disc. So if you imagine
Speaker 3: it plays it counterclockwise, and the need thing is it
Speaker 3: scans the entire record, so you can almost use it
Speaker 3: as a CD. Once it's done a scan of the record,
Speaker 3: you can skip tracks, you know, forward, backwards, et cetera. So, yeah,
Speaker 3: my pick is this quirky record player that I'm going
Speaker 3: to treat myself to hopefully in the next couple of months.
Speaker 3: That's wild.
Speaker 1: So will it still let you play the Beatles White
Speaker 1: Album backwards so you can get the satanic messages?
Speaker 3: That's a good question. Well, it has this feature where
Speaker 3: you can push it to place, and maybe it allows
Speaker 3: you to drag it backwards. I don't know, that's a
Speaker 3: good question.
Speaker 1: I'll let you know.
Speaker 2: I'll let you know, right. I wonder are there turntables
Speaker 2: that let you determine the direction of the wheel? I
Speaker 2: wonder if that's the thing.
Speaker 3: I don't know.
Speaker 1: It was just a it was a rumor I heard
Speaker 1: back when I was a kid about the Beatles Wide Album.
Speaker 1: All right, cool. So for my pick, it's funny because
Speaker 1: we didn't plan this, but I'm actually picking a tool
Speaker 1: called super Whisper super Whisper dot com. It's audio input,
Speaker 1: so you don't have to type at all, and you
Speaker 1: don't have to learn a new keyboard layout or deal
Speaker 1: with RSI. But it's actually surprisingly accurate, you know, because
Speaker 1: like every phone and computer now has some sort of
Speaker 1: voice transcription thing, and never in my life have I
Speaker 1: used the word doc. But super Whisper knows exactly what
Speaker 1: I'm trying to say, and it doesn't. Really It's done
Speaker 1: a really good job and you don't have to, at
Speaker 1: least in my experience with it. You don't have to
Speaker 1: slow down or just give it bites at a time,
Speaker 1: you know, you can just have a full fledged streaming
Speaker 1: thought process and it captures it very, very accurately. So
Speaker 1: it's been fun to play with and also will work
Speaker 1: locally so you can set it up so that it
Speaker 1: doesn't send whatever you're telling it back to their servers
Speaker 1: for training or storage or ransomware or whatever they want
Speaker 1: to hould you hostage for cool all right, Paul, thank
Speaker 1: you so much, man, this has been a super cool conversation.
Speaker 1: I'm super excited that you came on and chatted with
Speaker 1: us about it.
Speaker 3: Yeah, my pleasure. Thanks very much for the time, and
Speaker 3: apologies for the ins and issues, you know.
Speaker 1: Yeah for sure. Well, yeah, you got to come back
Speaker 1: on and let us know how the record player works.
Speaker 1: So we've got to do that part anyway. Yeah, it
Speaker 1: sounds right cool, right, I'm born. Thank you so much,
Speaker 1: appreciate everything, and for all of our listeners, thank you
Speaker 1: for listening. And we'll see y'all next week.