Speaker 1: Hello everyone, and welcome back to another episode of Adventures
Speaker 1: in DevOps. Today, I have a slight bit of news though.
Speaker 1: We have an upgrade to our podcast, as will is
Speaker 1: a way for a few episodes.
Speaker 2: So I've asked Amy.
Speaker 1: Knight, are expert on reliability architecture, to step in and Amy,
Speaker 1: are you ready for today's episode?
Speaker 3: I am excited to be here.
Speaker 2: That's good.
Speaker 1: I think I put her on the spot with calling
Speaker 1: her the expert. I don't think she was prepared for that.
Speaker 4: No, not ever expert she is.
Speaker 1: She's very modest, I will say that. So today's guest
Speaker 1: is Brian POTRELLI welcome and thank you for coming. He's
Speaker 1: a founder of Fusion auth and some other products as well.
Speaker 1: It's nice to have a fellow of expert on the show.
Speaker 4: I have to say, yeah, thanks for having me us.
Speaker 3: I was interested in how this was going to go
Speaker 3: with two experts.
Speaker 1: You know, I get this question like how did you
Speaker 1: become an expert? And usually I say something like well,
Speaker 1: I started investing in learning about security a lot of
Speaker 1: years ago, and after a lot of years, then I
Speaker 1: get up on stage and someone asked me how did
Speaker 1: you become an off expert? I don't think there's like
Speaker 1: a dedicated path, Brian, how did you end up in
Speaker 1: this area?
Speaker 4: Kind of dumb?
Speaker 2: Luck.
Speaker 4: So, we we are working on a sort of a
Speaker 4: niche product, and after we realized that it wouldn't scale,
Speaker 4: we actually wanted to start building a couple of other products.
Speaker 4: And so one of the products ideas that we came
Speaker 4: up with was a forum, and so we actually built
Speaker 4: that out completely. And when we built it, we decided
Speaker 4: we weren't going to add a log in and registration
Speaker 4: component to the forum itself. Instead, we were always going
Speaker 4: to delegate that to whatever the company already had, and
Speaker 4: so that required some type of authentication system, right, And
Speaker 4: in order for us to test this, we had build
Speaker 4: our own authentication system, and so we did, right. So
Speaker 4: we built it, we integrated with our forum. Everything worked,
Speaker 4: it was magical, and then we were like, hey, this
Speaker 4: afe thing's pretty cool. Maybe we'll just use it for
Speaker 4: like some of our other apps, and so we started
Speaker 4: integrating it with everything. Well, the forum didn't work out,
Speaker 4: and we're all kind of sitting in a room and
Speaker 4: I'm like, guys, this off things really neat, Like let's
Speaker 4: do that instead. So we like basically said a one
Speaker 4: to eighty dropped the forum started focusing on off and
Speaker 4: then had to go learn a bunch of standards and
Speaker 4: read a bunch of specifications and you know learn, you know,
Speaker 4: learn a lot about security in a very short period
Speaker 4: of time. So but totally luck, No.
Speaker 2: I totally got it.
Speaker 1: Actually, we ended up in a similar spot with a
Speaker 1: previous product that, uh, there was a lot of complexity
Speaker 1: and what we were doing, and we found that our
Speaker 1: customers were actually more interested in how we solve our
Speaker 1: technical problems than the product that we were offering. At
Speaker 1: that time, this was like a for COVID, and we
Speaker 1: were trying to sell like leaders of Leadership SaaS and
Speaker 1: it turns out a lot of companies wanted to say
Speaker 1: that they cared about leaders and building their leaders up,
Speaker 1: but they didn't actually want to pay for it. Maybe
Speaker 1: that's a little bit of a shocker.
Speaker 4: Yeah, not shocking, shocking, shocking, right.
Speaker 1: The main topic of today is I think it's going
Speaker 1: to be a little bit of a controversial episode. This
Speaker 1: may be our most controversial episode yet multi tenant versus
Speaker 1: single tenant architecture. Yeah, I guess that probably a lot
Speaker 1: of our audience already has a strong opinion one way
Speaker 1: or the other.
Speaker 2: Amy any thoughts.
Speaker 3: I guess my thought is just like alarm bells start
Speaker 3: to go off in my head. But there we go
Speaker 3: back with like the reliability stuck.
Speaker 1: Like you're like, if someone says we're going to go
Speaker 1: for multi tenant architecture.
Speaker 3: Like that, hold up, yeah, exactly exactly, Like I want
Speaker 3: to note the details.
Speaker 1: At this point, yees, so you think it's like inherently
Speaker 1: dangerous to have multi tenant architecture.
Speaker 3: I was just at the point where like the I
Speaker 3: don't know if the more separation the better with that
Speaker 3: sort of thing, But prove me wrong.
Speaker 1: Well, Brian's built a company on top of it.
Speaker 4: Yes, we're building this this forum, and then we built
Speaker 4: this sort of standalone, you know, off thing. And our
Speaker 4: other product was also you know, downloadable and single tenant
Speaker 4: and standalone, and we did we built the original product,
Speaker 4: which is called clean Speak. We built that that way
Speaker 4: because it was high performance, right, so like it filters
Speaker 4: profanity and chat so wow, a couple way of saying it.
Speaker 4: And so we're talking billions, tens of billions, hundreds of
Speaker 4: billions of chat messages a month, and so in order
Speaker 4: to do that in a multi tenant way is pretty challenging,
Speaker 4: especially when you're talking about super low latency with something
Speaker 4: like chat and so what happens is like you're in
Speaker 4: a game, you send a chat message, it goes across
Speaker 4: the wire into this, you know, the chat server. Okay,
Speaker 4: so that's in a bank of servers that's sitting in
Speaker 4: AWS or you know, somewhere like that. There's no reason
Speaker 4: you should have to jump out across the open Internet
Speaker 4: again to go filter that and see if there's any
Speaker 4: issues with it and then come all the way back.
Speaker 4: You know, that could introduce one hundred milliseconds of latency.
Speaker 4: It'd be way better if it just jumps across the
Speaker 4: backplane and essentially goes right to you know, the filtering service.
Speaker 4: And we could do that in like under a millisecond.
Speaker 4: And even with you know, in internetwork latency, it's like
Speaker 4: one or two milliseconds, so we can really shave down
Speaker 4: milliseconds here. And then we took all of this sort
Speaker 4: of like common code and deployment models and bundling and
Speaker 4: all this stuff we had and we just like copy
Speaker 4: and pasted it.
Speaker 1: So the history sort of put you in this direction.
Speaker 1: It was sort of like a sign from the universe
Speaker 1: that maybe this was the path for you to take.
Speaker 4: We kind of made a bold assumption that that was
Speaker 4: a good idea. It's like, hey, let's make this downloadable,
Speaker 4: let's make it single tenant, let's make it run anywhere
Speaker 4: in the world. And then we actually started having all
Speaker 4: these companies come to us, some of them quite large,
Speaker 4: and they're like, hey, we can't use your competitors. We
Speaker 4: can't use like A zero because it's in the cloud
Speaker 4: on it, we can't use you know, like pain because
Speaker 4: they're just pushing everybody to cloud, and we can't use
Speaker 4: all these tools, and we really want something that we
Speaker 4: can run in our own data center. And I was like, well,
Speaker 4: we got that, so yeah, let's totally do this right.
Speaker 2: So that's really interesting.
Speaker 1: So taking the architecture as the value being provided by
Speaker 1: the product as sort of the competitive advantage, yep.
Speaker 4: Yeah, we continue to do that. So, like I think
Speaker 4: our one of our taglines we've been messing around with
Speaker 4: is like aughts so modern you can download it right,
Speaker 4: and it's it's really flipping everything on its head because
Speaker 4: for the last gosh fifteen years, people have been like, yep,
Speaker 4: it's got to be multi tenant, it's got to be
Speaker 4: you know, SaaS, it's got to be cloud, and that's
Speaker 4: the only way to build an effective company, profitable company,
Speaker 4: scalable company. And I'm like, no.
Speaker 3: I'll be honest too, what you're saying. It kind of
Speaker 3: shocked me. Conversations that I've been having more and more
Speaker 3: people are actually people are considering moving off cloud to
Speaker 3: save costs, which is nothing I would have ever thought
Speaker 3: of before I had this initial conversation like two years
Speaker 3: ago with someone.
Speaker 4: Yeah, if if you have let's say, like a cloud
Speaker 4: native Lamba driven, you know, lots of like io into
Speaker 4: like something like Dynamo or something like that, Like sure,
Speaker 4: when you start and you're small, can be very cost effective.
Speaker 4: When you scale, it can get so expensive these services
Speaker 4: are like outrageous.
Speaker 3: The other thing I would say too, is depending on
Speaker 3: your architecture and how long it's been around. I feel
Speaker 3: like moving to the cloud, you're sort of forced into
Speaker 3: certain boxes of what they can scale most efficiently. So
Speaker 3: if you have something that was not built to run
Speaker 3: on a certain machine type, you're going to run into
Speaker 3: issue that scale. Would that be something that you've kind
Speaker 3: of experienced.
Speaker 4: Too, Absolutely, Yeah, The the cloud providers really love to
Speaker 4: push you in the directions that they feel the most
Speaker 4: confident and comfortable, and they where they have the biggest profit.
Speaker 4: Right if you look at just you know, like Cognito
Speaker 4: as an example in our space, like that's not a
Speaker 4: cost effective solution, like by anyone's anyone's you know view. Now,
Speaker 4: when you first start, it's very cost effective because it's free,
Speaker 4: and you know, it's pretty easy to integrate. But the
Speaker 4: second that you like start scaling and start enabling any
Speaker 4: of the features, it's cost prohibitive. And so they know
Speaker 4: how to scale that and run that really well, and
Speaker 4: then they know how to monetize the crap out of it.
Speaker 1: So I love that you, you know, you tried to
Speaker 1: pull back on like, well, you know, maybe it's really
Speaker 1: easy to integrate with Cognito, although I feel like, I see,
Speaker 1: I'm in a bunch of tech communities and I just
Speaker 1: see complaints every all day long about oh I tried
Speaker 1: to make this work and it didn't happen, or you know,
Speaker 1: I looked at the documentation and start saying like amplify everywhere,
Speaker 1: like I don't want to use amplify. I'm like, well,
Speaker 1: you know, welcome to Welcome to the mess that that
Speaker 1: is Cognito in AWS. So I started I stopped saying that, Yeah,
Speaker 1: it may be cost effective, you know, a small scale,
Speaker 1: but the overhead, the total cost of ownership, yeah, definitely
Speaker 1: high with with Cognito, for sure.
Speaker 4: It was super painful. And Amazon also does this really
Speaker 4: funny thing where they force you to use like you
Speaker 4: can't just use Cognito, like you said, you have to
Speaker 4: pull in fifteen other services and you got to use
Speaker 4: platformation and you got to do this, and it's just like, guys,
Speaker 4: this is getting insane, right, Like I don't want to
Speaker 4: do this. I just I just want to lock my users.
Speaker 3: I may have cheated and listened to some of your
Speaker 3: episodes on competitor podcasts to kind of get a better
Speaker 3: understanding before this one. Is it still true also that
Speaker 3: the customization is just not there.
Speaker 4: Cognito actually made a really big update, so I have
Speaker 4: to credit they. I think they restructured their product team
Speaker 4: and then they put some dedicated engineers on Cognito. For
Speaker 4: the last like eighteen months, they kind of up their game,
Speaker 4: Like you can quasi customize their log in. You can
Speaker 4: customize like the email templates and some of the messaging templates,
Speaker 4: and I think they even like support localization stuff. So
Speaker 4: they did a pretty decent job of like leveling themselves
Speaker 4: up to your point, it's like it's still a pain
Speaker 4: in about to integrate with and it's just feature crippled.
Speaker 4: I mean, they just like have it's a very limited platform.
Speaker 1: I think the biggest challenge here is something that a
Speaker 1: lot of people dismiss when they're selecting a product out
Speaker 1: there in the world, is a the alignment with the
Speaker 1: company that you're going with, Like what is the core
Speaker 1: value that the company really focus on and cares about? Right,
Speaker 1: you were saying that the single tendency is the core
Speaker 1: value that we're you know, one of the core values
Speaker 1: that we're offering here, you know, is that aligned, Like
Speaker 1: don't just use the product based off of the features,
Speaker 1: but you know, look to see about the team that's
Speaker 1: building in and what their you know, long term direction is.
Speaker 1: And you know, I think it's really important because when
Speaker 1: we look at things like cogn it's not really clear
Speaker 1: you know, what they're really going after. And if you
Speaker 1: start comparing the products and get really deep, you can
Speaker 1: see things like they're like tendency in Cognito itself is
Speaker 1: not really there as a solution. If you have customers
Speaker 1: that are in the business space, it doesn't really fit
Speaker 1: well with that, and it's detaining your point, you know,
Speaker 1: on the refresh with Cognito. Yeah, I mean it's configurable
Speaker 1: now for sure, But there are nine different kinds of
Speaker 1: Lambda functions that you can integrate with Cognito. Like it's
Speaker 1: not a simple strategy, Like it's not just like configuration
Speaker 1: driven a lot of times, and this is one of
Speaker 1: the huge types with AWO services in general. It's that
Speaker 1: if you want to configure it, you write your own
Speaker 1: LAMB to function with your own configuration with whatever, non
Speaker 1: documented payloads and responses, and then figure out how to
Speaker 1: integrate that into your solution and deployed and whatever to
Speaker 1: the cloud to get it to work right, and if
Speaker 1: there's a problem then you know, good luck. So some
Speaker 1: people love this, Oh yeah, all of my infrastructure everything
Speaker 1: is in AWS and you know, good on you, for sure.
Speaker 2: But I don't wish Cognito on anyone.
Speaker 4: Yeah, I mean that's a good point, right, because they
Speaker 4: AWS has this sort of disease I guess you could say,
Speaker 4: where they constantly want you to use their services to
Speaker 4: do things right. So like, you know, we use like
Speaker 4: cloud front, and in order to do all the redirects
Speaker 4: that we need to do, we have to use like
Speaker 4: an edge function, and then that edge functions go loads
Speaker 4: something out of like an S three bucket in order
Speaker 4: to like download it, and then it caches it, and
Speaker 4: you know, it's like it's ridiculous. It's like all I
Speaker 4: want to do is set up redirects, and I have
Speaker 4: to jump through like fifteen hoops just to get something
Speaker 4: so simple done because of the way that Amazon has
Speaker 4: designed everything, and they're like, well just stick a lamb
Speaker 4: on it, you know that don't work, and don't document.
Speaker 4: We'll document it later, you know.
Speaker 1: I mean, I do appreciate as a company, like comparing
Speaker 1: them to the other cloud providers, And I didn't know
Speaker 1: this was the direction we were going to go with
Speaker 1: this with this episode, but I do appreciate a WS
Speaker 1: more than the other ones. I do think they suffer
Speaker 1: from a very focused mindset as sort of the two
Speaker 1: piece of teams that they have. They're really focused on delivering,
Speaker 1: you know, exactly the thing that maybe that customer wants,
Speaker 1: but that may mean over time, they're missing some other
Speaker 1: critical features. I think what you're talking about here. A
Speaker 1: good example is returning security headers on outfront responses. So
Speaker 1: you've got some data stort inn s Rebucket, and you're
Speaker 1: hosting a website or your API, and you're like, I
Speaker 1: just want to add a header that removes the x
Speaker 1: frame options, like you can embed this as a as
Speaker 1: an iframe in a website, And you'd be like, I
Speaker 1: just want to add a header, and I don't want
Speaker 1: to muck with the underlying service, which I may not
Speaker 1: even own. Right, Maybe I've deployed Fusion ofth or some
Speaker 1: other service in a container in my infrastructure and I
Speaker 1: don't even control that product, but I want to add
Speaker 1: some headers to it. Well, good luck, honestly, because up
Speaker 1: until about three years ago, you couldn't do this without
Speaker 1: throwing a huge lamb to function at it. Now they
Speaker 1: do have something called the response policy headers that you
Speaker 1: can set in clautform, But it's taken probably like almost
Speaker 1: ten years for this to roll around and even be
Speaker 1: added in this feature, while other CDNs have been providing
Speaker 1: this baseline thing for quite a long time.
Speaker 4: Yeah, no, totally agree. They there. I think their product
Speaker 4: teams are so isolated and they don't have a lot
Speaker 4: of input from you know, just standard devs like us
Speaker 4: right that are just like, hey, I just want to
Speaker 4: get this thing done. And they're they're really listening, you know,
Speaker 4: to their largest customers, which we all know. I mean,
Speaker 4: you know, they make tens of millions of dollars off
Speaker 4: some of these customers every so, so yeah, it's it's
Speaker 4: pretty painful and they takes a long time for them
Speaker 4: to get actual stuff done. But at least they have
Speaker 4: the lambdas and the things and so like you have
Speaker 4: to jump into some code and you got to do
Speaker 4: these things. But at least there are workarounds where like
Speaker 4: you said, you know, there have been times where you know,
Speaker 4: in the past where we use services and you literally
Speaker 4: just can't do it. So it's like, well, crap, Okay,
Speaker 4: I'll stand up another service to proxy requests through and
Speaker 4: then I'll manipulate the request in my proxy service and
Speaker 4: then ask it to clout right, and it's like, oh
Speaker 4: my gosh, why am I having to manage all this
Speaker 4: junk just to add ahead or business? And I totally
Speaker 4: agree with that assessment of AWS, But again, they're one
Speaker 4: of the better out there, right. So here's the other
Speaker 4: thing that's interesting AWS. I think some of their product
Speaker 4: teams are actually starting to evaluate letting people download some
Speaker 4: of their services and run them locally, right, Like we
Speaker 4: see that a little bit with like Dynamo dB. You
Speaker 4: can run a like kind of a scraped down version
Speaker 4: of it. I think there's some local lambda execution things
Speaker 4: that you can do, and they're looking at more and
Speaker 4: so you know, maybe they'll do that with Cognito at
Speaker 4: some point too. But like, one of the hiccups that
Speaker 4: I see with all the cloud providers is the ability
Speaker 4: for a developer to test these services without having to
Speaker 4: like fire up an entire org, you know, run a
Speaker 4: bunch of terraform or CloudFormation or whatever you need to
Speaker 4: get everything set up, and then you can run your
Speaker 4: tests against it and good luck having twenty developers try
Speaker 4: to run tests at the same time where you're constantly
Speaker 4: tearing things down and recreating them and so moving that
Speaker 4: stuff back local I think is actually something that the
Speaker 4: industry has been at, which means that every developer can
Speaker 4: isolate themselves at dev time from every other one and
Speaker 4: that's a huge benefit.
Speaker 1: Yeah, I think the developer experience story with aws like
Speaker 1: offline is not the best so much that there's an
Speaker 1: entire company dedicated actually out of Switzerland called local Stack
Speaker 1: to emulate the emulate it offline, and you know, it's great.
Speaker 1: And the joke is like when is Agbo's going to
Speaker 1: buy local Stack because there's local Stack in the documentation,
Speaker 1: there's local Stack replacements for a DYNAMOITYB local which doesn't work.
Speaker 1: There's great integrations for Lambda and for SAM, the servilist
Speaker 1: application Transform for Information.
Speaker 2: It's really surprising.
Speaker 1: It's like one of the things like as you mentioned,
Speaker 1: like even though our product is focused one hundred percent SaaS,
Speaker 1: we offer a shim clone of our API for companies
Speaker 1: around locally because of course, like you're a developer and
Speaker 1: you're like, you don't care about maybe the off part,
Speaker 1: but your services depend on it, and so you need
Speaker 1: to have an answer here. And it's ridiculous how SaaS
Speaker 1: providers everywhere. I figured this out, but the cloud providers
Speaker 1: haven't done it yet.
Speaker 4: Yeah, you know, it's wild. We sort of like to
Speaker 4: take that to the nth degree, right, which is like
Speaker 4: mocking is dangerous, and so we'll market and then we'll
Speaker 4: just simulate the responses we know we get from production,
Speaker 4: so like maybe they record it and then they grab
Speaker 4: some you know, some stuff out of production, and then
Speaker 4: they can start simulating, well, what happens if like a
Speaker 4: lambda changes and then the responses start changing, or like
Speaker 4: AWS modifies something and it's coming back slightly differently.
Speaker 1: We definitely see that more sophisticated customers care about it,
Speaker 1: and end testing or integration testing and having a story
Speaker 1: dedicated to that is an important aspect, especially when you're
Speaker 1: offering something that's essentially infrastructure for your your customers. I
Speaker 1: mean different when it's just like some third party service
Speaker 1: which is solving some edge case or some CRM, but
Speaker 1: when it becomes a critical piece of infrastructure for your customers,
Speaker 1: it's a question that's going to come up pretty frequently.
Speaker 3: Absolutely, I kind of had a question I was going
Speaker 3: to ask, kind of been like doing how to ask
Speaker 3: this in the most intelligent way possible? How much of
Speaker 3: your product would you say? In most customers use cases,
Speaker 3: it probably simplifies things like performance and load testing. My
Speaker 3: experience and the source of like massive bought attack and
Speaker 3: at the same time my experience performance testing in a
Speaker 3: multi tenant environment can be tricky because while you would
Speaker 3: assume that you can just go off to the races,
Speaker 3: that is a false assumption. And if you have to
Speaker 3: performance test load test at a certain scale, it requires
Speaker 3: coordinating with the company and making sure that other customers
Speaker 3: aren't doing it at the exact same time. So I
Speaker 3: guess long story short, I kind of maybe know the answer,
Speaker 3: but maybe someone to speak to.
Speaker 4: Yeah, it's a phenomenal question. So there's sort of two
Speaker 4: aspects of this, and the first one is I'm a
Speaker 4: developer and I just want to like sort of load
Speaker 4: test locally, and I want to I just want to
Speaker 4: see what sort of throughput looks like with different configuration
Speaker 4: options on my laptop. And you download, you get it
Speaker 4: set up, you integrate, and then you just literally use
Speaker 4: a hammer, hammer the crap out of it and see
Speaker 4: see what happens, and then you can reset everything and
Speaker 4: then you can try it in the cloud. When you
Speaker 4: have a multi tenant provider, they even like a lot
Speaker 4: of times say you load tests are just not allowed,
Speaker 4: right and like period hard stop. You pay them enough money,
Speaker 4: they'll probably let you load test, but they're going to
Speaker 4: have to figure out how to get that traffic off
Speaker 4: of the main servers because they don't want to impact
Speaker 4: the you know, ten thousand other customers that are on
Speaker 4: the same servers when we deploy fusion to the cloud,
Speaker 4: especially single tenant cloud, right, so like every customer gets
Speaker 4: dedicated compute, dedicated database, dedicated iout and you can load
Speaker 4: test your cloud and not affect a single other customer
Speaker 4: because your compute is again completely isolated from everyone else's
Speaker 4: and your database is also, you know, because we use rds,
Speaker 4: and so your database has a specific number of eyeops
Speaker 4: and it's you know, we're we're using Amazon, trusting Amazon
Speaker 4: to basically do that. That isolation of all those things
Speaker 4: because everything's true, you know, shared underneath the hoods. But
Speaker 4: but AWS is even really good at time boxing things
Speaker 4: and like limiting io and so that's another benefit to
Speaker 4: our customers where they don't have to call us up
Speaker 4: and say, hey, we're going to go test. We're like,
Speaker 4: go ahead and mow test it. It's your box. You're
Speaker 4: not going to affect the customers over here because they're
Speaker 4: on their own hardware, but you're going to crash your
Speaker 4: own stuff, but you know, go for it, you know,
Speaker 4: So yeah.
Speaker 3: I break it up too, because these are just things
Speaker 3: I think that sometimes people don't necessarily realize could potentially
Speaker 3: be issues that are larking.
Speaker 1: I'll be I'll be the dissenting opinion. So I like,
Speaker 1: I totally agree that you you sort of get some
Speaker 1: of these aspects for free when you change your architecture
Speaker 1: paradigm from from one to another. For instance, for us,
Speaker 1: we've had to go in a different direction because we
Speaker 1: want to be able to have a single point of
Speaker 1: reference for a lot of our architecture.
Speaker 2: And as we.
Speaker 1: Focus primarily in the SaaS version, that simplifies a lot
Speaker 1: of a support request or triaging or logging, et cetera,
Speaker 1: because everything is just rolled out into the same stack.
Speaker 1: But that means we've had to invest a lot in
Speaker 1: how do we deal with increased scale. I will say
Speaker 1: something like load test away against our service, Like we've
Speaker 1: had to put so much effort into understanding how to
Speaker 1: increase scale that no, like ten companies one hundred companies
Speaker 1: coming at us at the same moment, it's just not
Speaker 1: going to matter. I will say that that we'll hit
Speaker 1: a bunch of rate limiting stuff that we have in place,
Speaker 1: like you will start getting blocked, So make sure you
Speaker 1: have a second account ready to go, because if you
Speaker 1: start running this on your primary account, like you will
Speaker 1: probably have a production downtime when you get rate limited
Speaker 1: from doing something. But we've had to split rate limiting
Speaker 1: in a lot of different ways, like per user, per application,
Speaker 1: per individual tenant, per service client. So I mean you
Speaker 1: are making a trade off from one to another. And
Speaker 1: if you're going down one like a different path here
Speaker 1: where you're going down multi tenancy, they have to be
Speaker 1: problems that you want to solve. We were interested in
Speaker 1: solving these problems, and we were cognizant of like our
Speaker 1: team having worked in areas where there was a lot
Speaker 1: of historical challenges that they've experienced and they know their
Speaker 1: way around building large multi tenant systems. But I think
Speaker 1: it's a really great point where if you don't have
Speaker 1: that expertise, that you're going to get yourself in trouble,
Speaker 1: especially when you're providing infrastructure level products for your customers.
Speaker 4: But aren't they then just testing your rate limits? So
Speaker 4: like the counterpoint of that is that, I mean we
Speaker 4: all have rate limits right because especially at like the
Speaker 4: WAFT level and the infrastructure level, because you have to
Speaker 4: like not be flooded, but you can take those down
Speaker 4: like we can. We can basically say like okay, we're
Speaker 4: going to take we're going to isolate you and take
Speaker 4: you out of rate limits, and it's like go for it.
Speaker 4: Literally just bang on it until the servers fall over.
Speaker 1: My response is always like what do you want it
Speaker 1: to be?
Speaker 4: Like?
Speaker 1: You don't want this to be like actually no rate limits,
Speaker 1: like you want there to be something here to happen,
Speaker 1: and so yeah, for sure. I mean, if they're testing
Speaker 1: their own software, I think this is where the mistake is.
Speaker 1: They believe they have a need to sort of validate
Speaker 1: how our software is going to respond to their needs.
Speaker 1: And I think that's the fundamental flaw here is that
Speaker 1: they like, either you're paying us, so you trust us
Speaker 1: with this product and we give you assurances there and
Speaker 1: if you're not willing to trust those, like you may
Speaker 1: think about why that is, like why is it that
Speaker 1: you actually want to take these extra steps. We do
Speaker 1: get questions like oh yeah, how much can we have?
Speaker 1: And I'm like how much do you how much do
Speaker 1: you want? Because you can have that much, it's not
Speaker 1: a problem. I assure you, you're not going to find
Speaker 1: out where our service is going to fall over for
Speaker 1: you because it's it's going to scale automatically to handle
Speaker 1: whatever you throw at it, and you can for sure test.
Speaker 2: That if you want.
Speaker 1: But often I find the bigger problem is when rate
Speaker 1: limits come into play, is that they're usually at a
Speaker 1: moment where your customers are not necessarily compaired to handle
Speaker 1: the rate limiting. So even in a single tenant architecture,
Speaker 1: you know, what do you want to happen there? Do
Speaker 1: you want just one user to get kicked out of
Speaker 1: their flow? Do you want to use Like I think
Speaker 1: this is like an unsolveable problem because our customers will
Speaker 1: say we want the right thing to happen. I'm like,
Speaker 1: I don't know what the right thing is here, Like
Speaker 1: why don't you tell me what you think the right
Speaker 1: thing is? And that's probably how the service works, and
Speaker 1: then they usually get stuck because it's very difficult to
Speaker 1: correctly answer, like what is actually the thing that's supposed
Speaker 1: to happen?
Speaker 4: Yeah, and it's going to vary for each customer, right,
Speaker 4: Oh yeah, So it's semantics, So no, I totally agree.
Speaker 4: So I guess there's you know, it's it's just more
Speaker 4: of a statement. But there are benefits to both. One
Speaker 4: of which is like, if you have that level of
Speaker 4: scale and a multi tenant system, yeah, you're gonna have
Speaker 4: to really think about a lot of these constraints and
Speaker 4: how you change your rate limits and how you can
Speaker 4: get all that that much data through AWS versus if
Speaker 4: you have a single tenant system, then you're like, okay,
Speaker 4: well I'm just going to focus on singularly getting that
Speaker 4: through one little window just for that customer and not
Speaker 4: worry about everybody else because they're on their own, right,
Speaker 4: They're they're doing a different type of throughput.
Speaker 1: So I got I got something that you know, I'm
Speaker 1: sure is going to come across as part of the controversy.
Speaker 1: It's so much easier to deploy upgrades to a multi
Speaker 1: tenant system than it is to n repeated instances of
Speaker 1: a of a single tenant system.
Speaker 4: Right, That's that's just that sound dangerous. And they're like,
Speaker 4: well yeah, but off zero and every Cognito does and
Speaker 4: everyone else does, and I'm like, well sure they do,
Speaker 4: and they will break you, Like if they change something,
Speaker 4: you will be broken and you won't even know it,
Speaker 4: and like just go look at I mean, like there's
Speaker 4: just hit Reddit and say, ah zero upgrade broken, and
Speaker 4: you'll find so many people complaining about some change that
Speaker 4: off Zero or Cognito or Microsoft or somebody did that
Speaker 4: broke their entire application. And so our meth are sort
Speaker 4: of like theory on this whole thing, because you really
Speaker 4: actually don't want a multi tenant upgrade, right, you want
Speaker 4: a single tenant upgrade. You just want the ability to
Speaker 4: do it very easily and seamlessly. And so what we
Speaker 4: do is we say, we release new version of software,
Speaker 4: Please bring it back to dev, run it locally, run
Speaker 4: all your tests against it, make sure it's completely perfect,
Speaker 4: then schedule an upgrade. And then you basically we just
Speaker 4: we allow them to schedule an upgrade, click a button,
Speaker 4: we upgrade their services, and they're they're off to the races, right,
Speaker 4: And so we build processes and tools that allow this
Speaker 4: to have, you know, our customers to do this very easily,
Speaker 4: but it's very important, and we talk to every single
Speaker 4: one about it, is like, please take this back to
Speaker 4: dev and run all your tests against this new version
Speaker 4: before you move into production and make sure nothing's going
Speaker 4: to break up. Auto upgrades are just dangerous inherently in
Speaker 4: the industry, and like build tools, dependencies anytime you're automatically
Speaker 4: upgrading something in your stack without fully testing it. Beware.
Speaker 2: Yeah, I mean, I think you're absolutely right.
Speaker 1: I've been on this particular horn for quite a while
Speaker 1: about companies or software dev teams that automatically upgrade dependencies
Speaker 1: in their requirements text file or package chase on or
Speaker 1: whatever have you, with the argument of automatically getting whatever
Speaker 1: security upgrades come with us. And I think what I
Speaker 1: really see here is a responsibility model, like who is
Speaker 1: going to take full responsibility for their being a breaking
Speaker 1: change somewhere? And it sounds like, you know, with these
Speaker 1: other competitors out there in the market, they don't take
Speaker 1: responsibility for the breaking changes that they make. You've made
Speaker 1: it transparent and we promise no breaking changes. So I mean, there,
Speaker 1: it's really ridiculous that you can be in this state.
Speaker 1: I mean, I would be totally okay with Amazon taking
Speaker 1: full responsibility for no breaking changes if they are going
Speaker 1: to upgrade my RDS instance or dining with tob et cetera,
Speaker 1: other models, But it really can't be the case that
Speaker 1: you're using a managed provider and they roll out features
Speaker 1: that can break your production software like that, that's not
Speaker 1: a real that's not a real solution.
Speaker 4: In my mind, breaking changes is a very hard thing
Speaker 4: to define.
Speaker 2: Yeah.
Speaker 4: Right, So, like there's three levels of bugs in software. Right,
Speaker 4: There's a top level bug that is simply something that
Speaker 4: was inherently unfunctional and then becomes functional again. There's a
Speaker 4: semantic bug at the top level, which is I changed
Speaker 4: the semantic of something because it was not correct previously
Speaker 4: according to our docks or whatever, and now it is.
Speaker 4: And then there's nested semantic changes, which means I've called
Speaker 4: through service through service. Through service through service, the one
Speaker 4: way down at the bottom changed and it revealed a
Speaker 4: bug all the way up to the top. So I
Speaker 4: go fix the service way down at the bottom, and
Speaker 4: then the one at the top gets magically fixed. Okay,
Speaker 4: so when you have a patch release that's fixing a bug,
Speaker 4: it's still possible that someone's depending on the broken nature
Speaker 4: of that and you blow them out of the water.
Speaker 4: And this can it's literally just a dot release, right,
Speaker 4: So I could have thrown an exception and now I
Speaker 4: no longer do that. Well, the smart developer is like, well,
Speaker 4: I'm going to catch your exception and I'm just going
Speaker 4: to handle the path where like it's fine, Well, now
Speaker 4: you're returning the status code that I don't expect. Oh crap,
Speaker 4: I was expecting a five hundred. Now you're returning to
Speaker 4: four or one. Dude, like you're killing me your smalls,
Speaker 4: Like I can't. So, yes, you say you have zero
Speaker 4: incompatible changes, but that is too hard for a standard
Speaker 4: developer to reason through. So the only way to truly
Speaker 4: figure this problem out would be for the build tools,
Speaker 4: the testing tools, and the development time tools to basically
Speaker 4: certify that the entire landscape of all public things in
Speaker 4: our API, our code, whatever it might be, that people
Speaker 4: can't consume, here are the breaking changes, and here are
Speaker 4: the non breaking changes, right, because there's every release has
Speaker 4: breaking changes depending on how you're using the tool.
Speaker 3: You bring up such a good point. We're in a
Speaker 3: certain place we were discussing before.
Speaker 1: The call is famous for this.
Speaker 3: I seated at multiple places, but there was a lot.
Speaker 4: At this place. These are insanely hard problems to solve.
Speaker 4: And what's happened in the software development industry is that
Speaker 4: everyone got so excited about new languages and frameworks and
Speaker 4: building apps fast, and now there's like, you know, you
Speaker 4: know whatever Jive coding or whatever it's called I don't
Speaker 4: know what yes, and like bobob coding. So like there's
Speaker 4: all this stuff that we're just like we're just throwing
Speaker 4: it at the top end. We're like, oh, this is amazing.
Speaker 4: Look at all the stuff we can do and these
Speaker 4: cool frameworks and you know, React and all these things,
Speaker 4: and we forgot to go fundamentally solve software engineering problems
Speaker 4: at the core level, which is like, how do I
Speaker 4: certify that this version and this version are unquote compatible
Speaker 4: at the binary level, compatible at the public API level,
Speaker 4: compatible at the consumption level, compatible at the run time level.
Speaker 4: We don't have tools for that. There are literally no
Speaker 4: tools in any language that certify those things because we
Speaker 4: just as an industry forgot about them and we assumed
Speaker 4: that the developer was smart enough to do them, and
Speaker 4: they're not. Like, no one is. No one can know
Speaker 4: the entire you know, code base and certify that it's compatible.
Speaker 4: It's impossible. I made fun of some of these new terms.
Speaker 4: My son's a software engineer, and so he's like, Oh,
Speaker 4: I'm going to vibe code this thing tonight.
Speaker 3: I'm like, god, dude, vibe like an authentication just like,
Speaker 3: oh my god.
Speaker 1: There was actually a post about using Claude to generate
Speaker 1: two compatible integration and how much it failed basically even
Speaker 1: with the driver being a very experienced senior engineer in
Speaker 1: developing some like helping develop some of the standards. Like,
Speaker 1: that's how ridiculously not safe it is to do that.
Speaker 1: I do want to call out, like, this is for
Speaker 1: sure a nearly impossible problem. And I don't think you've
Speaker 1: you've even sold it enough here, so like let me,
Speaker 1: I just want to share that. It's like if you
Speaker 1: haven't if your if your service like returns an enom
Speaker 1: like a value you know, zero, one or two, and
Speaker 1: you add in the ability for it to return a four,
Speaker 1: like is that like does that break someone? And it's
Speaker 1: not a breaking change technically, but it is it is
Speaker 1: it for sure will break someone because in most software
Speaker 1: languages there is no code to say, like I expect
Speaker 1: only these results and if I get a different one,
Speaker 1: what to do in that scenario? And so you will
Speaker 1: be putting your customer in a scenario where their system
Speaker 1: will break in some unexpected way.
Speaker 2: So I think if.
Speaker 1: You're ready to go down this approach, if you if
Speaker 1: you run managed software like a cloud provider, or you've
Speaker 1: done something ridiculous like we have. You have to really
Speaker 1: understand the system thinking approach, like what based off of
Speaker 1: our current API, what did our customers write, Like what
Speaker 1: magical thing happened in their head that they wrote down
Speaker 1: that was was correct at the time, and now after
Speaker 1: this change is no longer correct. And so very often
Speaker 1: when I say no breaking changes, I mean that means
Speaker 1: like you can add fields, and even that's a little
Speaker 1: bit on the edge, but like renaming things or adding
Speaker 1: error codes, you know, we're very careful about. I think
Speaker 1: a standard one is like don't don't, like you have
Speaker 1: to be so careful not to over engineer anything, because
Speaker 1: that for sure means later you're going to be like, wow,
Speaker 1: I wish I hadn't put that in the API, because
Speaker 1: now someone could be depending on it, and it's impossible
Speaker 1: to know what fields someone's depending on with a particular
Speaker 1: get Now, there are some tricks here for anyone who
Speaker 1: actually does care and does decide to do this. You
Speaker 1: can embed the assumptions in your SDKs that you roll
Speaker 1: it to your customers, and then you can track which
Speaker 1: SDK versions they're using to understand what sort of things
Speaker 1: that they'll run into. And by making sure that the
Speaker 1: requests that coming from their service are all using an
Speaker 1: upgraded version of the SDK. You can be sure that
Speaker 1: any dependency that rolls out for that customer would not
Speaker 1: have a problem. Which does mean that in our own
Speaker 1: code base we do have feature flags for certain customers
Speaker 1: to potentially take certain dangerous upgrades. But fundamentally we do
Speaker 1: have to segregate by customer and understand the SDK. And
Speaker 1: that's still not sufficient because customers will delegate out integrations
Speaker 1: to all services to backstage and whatever other internal developer
Speaker 1: tooling or other client that you don't even control, and
Speaker 1: so getting that that integration to work correctly is just
Speaker 1: another huge thing. Like it's not always like you can
Speaker 1: get your customer on the phone and promise them a
Speaker 1: huge discount to make to make a change, or you know,
Speaker 1: threaten them with a huge increase if they are still
Speaker 1: on an old version of Kubernetes. And I mean, I'm
Speaker 1: your ofth provider in the cloud because you know, oh
Speaker 1: the cloud friders are doing that now too. So I
Speaker 1: do think that there's a whole spectrum here of problems
Speaker 1: that you're going to run into, and you have to
Speaker 1: be conscious of how you're going to tackle every side
Speaker 1: of it.
Speaker 4: Yeah, it's tricky. The software engineers have to think so
Speaker 4: hard about the architecture for their unges, their APIs. It's like,
Speaker 4: are we going to do? Are we going to version them?
Speaker 4: So like you know, slash V one, slash V two,
Speaker 4: slash v three, And then when we make a change,
Speaker 4: when do we upgrade the version number? And then is
Speaker 4: that version number tied to the SDK and the SDK
Speaker 4: only calls into this when it's this version or cause
Speaker 4: the old version? And can you make a compatibility translation
Speaker 4: between V one and V two and B three? I mean,
Speaker 4: it's like it's a lot of mental overhead just to
Speaker 4: make a simple change, Like, ah, dude, I just want
Speaker 4: to like return this extra field in the API. You're like, well,
Speaker 4: is another field dependent on that field now? Because if
Speaker 4: you have dependent fields, you can't make that change unless
Speaker 4: you version the API and then version of the SEK,
Speaker 4: and then you have to make sure you have a
Speaker 4: compatibility layer between those two versions. And their brains explode
Speaker 4: and they're like, dude, all I just wanted to make
Speaker 4: a feel change.
Speaker 3: Sorry, this is why I don't such faith.
Speaker 4: And a terrified for this specific reason.
Speaker 3: It really really does. And like at the stuff that
Speaker 3: I've been seeing playing with different products that are coming,
Speaker 3: it's learning it's hallucinations, and like, I just don't see
Speaker 3: how this stopped. Like I know there's obviously balances, and
Speaker 3: you know it's people obviously like test these before it
Speaker 3: comes out, but once it's like once it's deployed and
Speaker 3: it's learning its own hallucinations, Like how do you stop it?
Speaker 3: I don't.
Speaker 1: I love I love how you said obviously people test
Speaker 1: these because you know I.
Speaker 3: Once you haven't like deployed in your environment. It's what
Speaker 3: is it the Schroeder's cat, like if it if it's
Speaker 3: but I forget how that goes and if I've even
Speaker 3: pronounced that correctly. But Schroder's cat where it's like, if
Speaker 3: it happens in another world, does it then become reality?
Speaker 3: Like if it's hallucinating on something that's false, but now
Speaker 3: it has become reality, Like is it reality?
Speaker 4: I don't think it's a phenomenally existential question.
Speaker 1: What I was going to ask is you know how
Speaker 1: the S and MCP stands for security? Where are you
Speaker 1: on the spectrum of AI is terrible and going to
Speaker 1: ruin the world? And is it the best thing ever
Speaker 1: created by humans? And I guess we know where Amy
Speaker 1: stands on that particular point.
Speaker 3: I mean, I'm sure it's going to excel at certain things,
Speaker 3: but solving the types of things that people think it's
Speaker 3: going to solve, I just don't. I don't see how
Speaker 3: that can happen, because I mean that we're fandomically, like
Speaker 3: we're at the limb of ourselves and we fail.
Speaker 4: So how it's really hard to engineer a prompt that
Speaker 4: tells them about your entire software development life cycle and
Speaker 4: the way that you've architected your entire system, right, So,
Speaker 4: like your prop would have to be like generate me
Speaker 4: a new API and blah blah blah. But keep in
Speaker 4: mind that we version our APIs this way and these
Speaker 4: are breaking changes. We use this SLDC process to get
Speaker 4: our SDK updated. You have to link the SDK to
Speaker 4: this and that and that. There's the AI is. You
Speaker 4: just can't do that, right, So, like AI is great
Speaker 4: for helping me code complete a four loop, right, and
Speaker 4: it'll guess based on the things that's seen in the
Speaker 4: code and where I'm at and logically what I want
Speaker 4: to kind of do, and I can make a prompt.
Speaker 4: It's like, hey, make me a map reduce on this list,
Speaker 4: and okay, I got you. Great for that stuff. Generating
Speaker 4: full code bases and let alone adding to existing code
Speaker 4: bases large chunks of code freaks me out. And I
Speaker 4: always tell people. I'm like, don't let that anywhere near
Speaker 4: your security layer, Like anything that has to do with security, please, please, please,
Speaker 4: please please do not let AI generate it and just
Speaker 4: ship that like, let it help you, but review your code,
Speaker 4: test it, do a security audit, do a pen test,
Speaker 4: do a load test. Like you still have to do
Speaker 4: all the things that we're doing which require knowledgeable software
Speaker 4: engineers to do them right. You can't just let AI
Speaker 4: do that either. So like the doomsdayers are like, okay,
Speaker 4: well you know engineering is dead, you know, no, no
Speaker 4: more engineers coming out of college. No, we're stopping. We'll
Speaker 4: just take that off to college curriculum. And I'm like,
Speaker 4: we're so far away from that, so so so so
Speaker 4: far away. Like you guys, keep going to college, let
Speaker 4: go get your degrees.
Speaker 3: That's going to create more jobs.
Speaker 4: Yeah, it will, it will, It's like.
Speaker 3: It will.
Speaker 1: So I think the biggest problem here isn't that You're
Speaker 1: absolutely right. It's that people believe that it's going to
Speaker 1: take over stuff, and so it's already affecting things like
Speaker 1: universities and whatnot. I mean, there is something to be
Speaker 1: said for specifications. I find that if you have a
Speaker 1: very well written out spec using an LM to generate
Speaker 1: on a transformation so a translation or getting it written down,
Speaker 1: let's say, open API specification into something else that's programmatic,
Speaker 1: so an SDK. But at that point, why are you
Speaker 1: just not using a rules generator? But maybe the thing
Speaker 1: that generates the generator, that thing could be l M based.
Speaker 1: So you know, I see, I see the hesitation here.
Speaker 1: You know, I'm totally totally on the same side for
Speaker 1: the most part.
Speaker 2: Yeah, for sure.
Speaker 4: Yeah, it's again, it's a tool. Use it effectively, you're good.
Speaker 4: Don't use it for everything like that. Just that sounds dangerous.
Speaker 1: I like the argument that people have been using Oh
Speaker 1: but there were a lot of naysayers about the Internet
Speaker 1: when it came out, So can't you envision at some
Speaker 1: point AI also being great and the people that are
Speaker 1: jumping up and down right now like they're just on
Speaker 1: the forefront of innovation.
Speaker 4: And I say, remember the Internet was never designed to
Speaker 4: host applications.
Speaker 1: And I think you've established some converts probably today after.
Speaker 1: You know, it's interesting because from my experience it's been
Speaker 1: that a full multi tenant solution is never the right answer.
Speaker 1: And I also am not a fan of full single
Speaker 1: tenant solutions. Like we end up usually somewhere in between.
Speaker 1: We don't have millions and millions of database instances running
Speaker 1: one for every customer, but there are things like dedicated
Speaker 1: tables per customer, or dedicated certificates per customer, or dedicated
Speaker 1: you know, CDNs, et cetera. But there are also some
Speaker 1: shared components as well. And I feel like understanding where
Speaker 1: the direction the business is going and where the value
Speaker 1: is that you're providing helps to pick the right part
Speaker 1: of the spectrum and not assume it's like a pure
Speaker 1: extreme case, like it's only A or B and nothing
Speaker 1: in between.
Speaker 4: Exactly. Yeah, I mean we're you can consider us fybrid too, right, Yeah,
Speaker 4: when we run in the cloud, we I mean we're
Speaker 4: even looking at like starting to share a single database
Speaker 4: instance across like lots of small customers because the cost
Speaker 4: of scale there is is way better and it's way
Speaker 4: easier to manage and multitude of reasons, right, So we're
Speaker 4: we're already looking at starting to do some of the
Speaker 4: we would do like a database slice per customer. We
Speaker 4: would probably put them all on the same database and
Speaker 4: just like our tables or anything. But but yeah, I
Speaker 4: totally agree with you, right, So, like we we want
Speaker 4: everybody to have the best of both worlds, and it's like,
Speaker 4: how do we do that most effectively? Though? So the
Speaker 4: coolest use case that we have, which kind of is
Speaker 4: only usable and you know, downloadable, single tenant and they
Speaker 4: deploy it in their own special way, is that we
Speaker 4: work with a satellite company and they actually have pushed
Speaker 4: fusee not up to these low Earth orbit satellites, and
Speaker 4: so we can say that fusion oth runs in space.
Speaker 4: It's pretty sweet. And so we're like, we like to
Speaker 4: say we're on all all all the continents and space,
Speaker 4: although Antarctica we faked that one. We just you know,
Speaker 4: had somebody that was doing a tour down there, like
Speaker 4: fire a FUSEE near it, and then we said, okay,
Speaker 4: we're on Antarctica now.
Speaker 3: But yeah, for.
Speaker 1: Sure, amy have have we converted you any if I
Speaker 1: managed to at least pull you back a little bit
Speaker 1: away from multi tenant is always wrong and security vulnerabilities,
Speaker 1: I definitely.
Speaker 3: I mean, if you's in a perfect world, there's one scenario,
Speaker 3: but in a practical world, I do agree, like it
Speaker 3: depends it has to be crimination.
Speaker 1: Yeah, I see some uh past trauma there really starting
Speaker 1: to show.
Speaker 4: I think we've had trauma on both sides. We're likeable.
Speaker 4: It was expensive and so hard to manage, and how
Speaker 4: do I get this into production? And then it's like, well,
Speaker 4: we're using this service and it just crashed, you know.
Speaker 4: I mean I think we all have a decent amount
Speaker 4: of stories from both sides.
Speaker 1: Yeah, I mean I think my my single tenant one
Speaker 1: was definitely we were running Jira, and uh, I mean
Speaker 1: that that's that's already the beginning and the end of
Speaker 1: the whole trauma right there. Yeah. But it turns out
Speaker 1: that when you're running it and they tell you that
Speaker 1: there's a major change, you would hope that the provider
Speaker 1: gave you the capability to automatically migrate the database and
Speaker 1: all the backwards incompatible stuff, But very frequently there would
Speaker 1: be database crashes and you would lose all of your data,
Speaker 1: and it's just like that's a thing that happened, and
Speaker 1: I think the pendulum swung really hard to the other
Speaker 1: side was like, we don't ever want to deal with
Speaker 1: this ever again, because we don't trust companies to provide
Speaker 1: us the tools to.
Speaker 2: Actually do the upgrades.
Speaker 1: And now we're back going I think, really coming the
Speaker 1: other way, which is, yes, we don't trust companies to
Speaker 1: you know, still be alive or you know, running their
Speaker 1: APIs in a non backwards compatible way, because now they
Speaker 1: just release new stuff all the time and it breaks.
Speaker 2: And I think this is just a story of it.
Speaker 1: It's no matter what solution you pick, it it's wrong
Speaker 1: or bad in some way.
Speaker 3: Everything everything is very much it depends and then it
Speaker 3: changes constantly. So you know what you have one year,
Speaker 3: the decision you make you're one is going to be
Speaker 3: a very different problems the decision you make year two, three, four, hopefully.
Speaker 4: Yeah, I totally agree with all things said.
Speaker 1: Everyone's camp okay, Yeah, that's that's a good, good camp
Speaker 1: to be in.
Speaker 4: I think we've we've hit the AI topic, which everyone
Speaker 4: is obligatory. Now everyone has to hit an AI topic.
Speaker 1: And the truth is, we actually have quite a few
Speaker 1: episodes recently in the past few months that have heavily
Speaker 1: delved into ai.
Speaker 2: UH and LMS and anything on the topic.
Speaker 1: So if there you are, if anyone is interested in
Speaker 1: reviewing those, there's a there's definitely a plethora of not
Speaker 1: not a limited amount of information podcast episodes, so you
Speaker 1: can go into hours and hours where we debate one
Speaker 1: way or the other. So then let's uh, let's move
Speaker 1: on to picks. Amy is okay if I put you on
Speaker 1: the spot?
Speaker 3: Sure, I you know, since I'm like new here, I
Speaker 3: have a slew in my head. But I'm going to
Speaker 3: pick walking since that's what I'm doing today. Morning is
Speaker 3: usually when I have the most energy. So I think
Speaker 3: I've gotten I'm looking on my tunel here almost two.
Speaker 4: Miles treadmill do you have I've been like condubating getting one.
Speaker 3: I am a Peloton fan girl. That could be my
Speaker 3: second pick, but in all reality, I was looking so
Speaker 3: I have the first version of the Peloton thread. I
Speaker 3: like the slated one. However, this one I think is
Speaker 3: slightly smaller and so it works perfectly for my face.
Speaker 3: I have a desk up here. You can actually I
Speaker 3: don't have it hooked up right now, but it's the
Speaker 3: external monitor. You can actually hook up your laptop to
Speaker 3: the external monitor too, so it works really good for
Speaker 3: like a little setup.
Speaker 4: Nice, all right, I'll look into it.
Speaker 1: Brian Feelfred, I guess why I'll go.
Speaker 4: So two. I have two things. Actually one of them
Speaker 4: is just totally random because it just popped into my head.
Speaker 4: But like I like, like, you know, snakes on a
Speaker 4: plane style movies, right, So, like I watched one recently
Speaker 4: which I thought was was really hilarious and it was
Speaker 4: just like sort of like way over the top in
Speaker 4: terms of like the the way that they did it,
Speaker 4: And so that was that was one of them. Now
Speaker 4: I'm just Fight or Flight and other kind of like
Speaker 4: stakes not a plane movie anyway, if you're into that
Speaker 4: kind of like heavy gore, like crazy comedy, like ridiculousness,
Speaker 4: that one's a fun movie. But the one that I'm
Speaker 4: really interested in kind of because it's you know, kind
Speaker 4: of goes along with what we're doing here and talking about,
Speaker 4: is there's this cool tool called search Craft, and I'm
Speaker 4: like on like this huge kick with it because it's
Speaker 4: it's a completely new search engine. They've rewritten, you know,
Speaker 4: from scratch. It's not built on Lucine, it's not built
Speaker 4: you know on Elastic. It's completely different. It's built in rust.
Speaker 4: It's like super efficient, highly performance, requires like you know,
Speaker 4: seventy percent less compute. You can run it locally, you
Speaker 4: can run it in their cloud, and you can run
Speaker 4: it in your cloud. It's just a really really cool
Speaker 4: evolution of something that you know a lot of apps
Speaker 4: in need, which is just a simple search service. And
Speaker 4: and I think they really have something that's gonna change
Speaker 4: the way that we think about search. So anyway, that's
Speaker 4: that's the thing that I've really been gone how about lately.
Speaker 4: So anybody wants to check it out. It's just I
Speaker 4: think it's searchcraft dot. I owe searchcraft dot I owe.
Speaker 1: I think is it Well, we'll get the link and
Speaker 1: put it in the in the show notes. Is this
Speaker 1: just like every company I should be using it? Or
Speaker 1: is like if I'm just developing something myself, it's there's
Speaker 1: a great opportunity to also pull it in.
Speaker 4: At some point every company should be using it. It's
Speaker 4: because like you know, you know Elastic, right, it requires
Speaker 4: like seven servers and they all have to have like
Speaker 4: full pi ops and they have to have ten gigs
Speaker 4: a RAM each and it's just a it's a hawk,
Speaker 4: and is it impossible to run and manage. Search Craft
Speaker 4: is not quite there yet. Their clustering is coming along,
Speaker 4: and they're really they're really heavy in development right now.
Speaker 4: But like, just as a nice search search is for
Speaker 4: an app that you're building, you should totally be using it.
Speaker 4: Don't even think about open search, elastic search, just skip
Speaker 4: that junk and go straight to search Craft.
Speaker 1: Yeah, I'm I'm totally with you in general here. I
Speaker 1: find that, first of all, i'm really suspicious of any
Speaker 1: application that's built in Java.
Speaker 2: That's that's that's my first thing.
Speaker 1: But the second one is is that even the promised
Speaker 1: managed services in a WS with the other cloud providers,
Speaker 1: they're they're not really fully managed. It's just pretty much
Speaker 1: like you don't get access to the easy two machines,
Speaker 1: but you still have to pretty much manage it yourself.
Speaker 2: So there's a there's a huge allure of.
Speaker 1: Just not having to understand the complexity of index management
Speaker 1: UH and document management and just using an API that
Speaker 1: runs with low compute or memory impact.
Speaker 4: So it's huge and it's hybrid again, so you can
Speaker 4: not locally run in the cloud.
Speaker 1: Okay, thank you, Okay, So my pick today is going
Speaker 1: to be uh these uh Scarpus shoes. I actually really
Speaker 1: liked them. I found them randomly one day when I
Speaker 1: was walking in in Zurich. They're wide enough if you
Speaker 1: if you want wide shoes, and they're must in Switzerland
Speaker 1: if you just hop on some walking trails.
Speaker 2: They have vibrum soles.
Speaker 1: I just really like them, and I never heard of
Speaker 1: them before. Apparently they were made in Italy, and I
Speaker 1: guess today I'm just one of the lucky ten thousand.
Speaker 4: Scarpa correct me if I'm wrong. But they started making
Speaker 4: mount rock cling shoes right so then.
Speaker 1: Oh yeah, so in Switzerland it's it's been a huge
Speaker 1: path down here. So really realistically these are just regular
Speaker 1: walking shoes.
Speaker 2: But you can get all.
Speaker 1: There's a six ratings for hiking in Switzerland T one two,
Speaker 1: T six and T one's like flat ground.
Speaker 2: You don't there's no danger.
Speaker 1: All the way to T six you're totally exposed, risk
Speaker 1: of death everything. And they off they have dedicated shoes
Speaker 1: for every single type of hike you could possibly go on.
Speaker 1: Ones that support crampons, dedicated hiking shoes, you know all
Speaker 1: they go all the way up to your like above
Speaker 1: your ankles for support dedicated like climbing shoes, so you know,
Speaker 1: the Will band in case you're you're into rock climbing
Speaker 1: and whatnot. They're really nice, Like I think they really
Speaker 1: high quality, really soft h They're some of the best
Speaker 1: ones that I've seen. They don't work for me for hiking, unfortunately,
Speaker 1: but for walking around everywhere. I absolutely love them so far.
Speaker 2: They're great.
Speaker 4: That's awesome. What was the model on those?
Speaker 1: I think they're the Planet Mohido Suede.
Speaker 4: Okay, because they've got like a bunch of different because
Speaker 4: I used to do rock climbing and stuff, and they've
Speaker 4: got like their whole climbing thing. They've got trailing, They've
Speaker 4: got like a bunch of different styles. So I'll have
Speaker 4: to go check those out.
Speaker 1: Yeah, I mean these these actually don't even have the
Speaker 1: virum soul, but these are.
Speaker 2: They're they're pretty.
Speaker 1: They're pretty nice, thick thick soles that don't get worn out.
Speaker 2: I have a lot of traction.
Speaker 1: But yeah, I mean, if the scarp is fit, I
Speaker 1: definitely recommend them to to anyone. Uh, it's actually really interesting.
Speaker 1: Like I've been I've really gotten to shoes lately because
Speaker 1: I've just realized I have no idea what.
Speaker 2: I've been buying my whole life.
Speaker 1: Binding Appropriately fitting shoes actually a real challenge. Don't just
Speaker 1: put anything random on your feet. And so like I've
Speaker 1: been watching like what do people wear out there in
Speaker 1: the world, and a lot of people are just wearing crap, honestly,
Speaker 1: Like you go to one of these regular shoe stores
Speaker 1: and just buy stuff, and there's such a huge difference
Speaker 1: from buying shoes like this versus just picking up the
Speaker 1: one shoe that will you'll your foot will go through
Speaker 1: the sole after a year or two years, and something
Speaker 1: like this that just keeps on lasting.
Speaker 4: Honestly, Yeah, yeah, I totally agree. So I was like
Speaker 4: at a conference and I just my shoes was shot.
Speaker 4: So I jumped into like an H and M and
Speaker 4: I'm like, I'll just grab anything that's in there. And
Speaker 4: I grabbed them and put them on my feet, and
Speaker 4: I walked like another mile that day, and I'm like,
Speaker 4: oh my god, my feet hurt so bad, and like
Speaker 4: by the end of the show, like the heel was
Speaker 4: worn out, the bottom is falling apart. I'm like, oh, well,
Speaker 4: that's why they when they cost like forty dollars, you know,
Speaker 4: thirty five bucks or whatever.
Speaker 2: It was crazy.
Speaker 1: Yeah, I mean that's pretty cheap, even for a shoe. Okay, well,
Speaker 1: I guess that's a good point, probably at the episode
Speaker 1: before we get into into too much into into shoes
Speaker 1: and hiking. So thanks Amy as our temporary guest expert
Speaker 1: here and reliability, And thank you Brian so much for
Speaker 1: coming on the show.
Speaker 2: I hope we see you again.
Speaker 4: Yeah, this is awesome. Happy to come back anytime.
Speaker 1: Thanks for everyone listening, and we'll see you all next time.