WEBVTT 1 00:00:07.759 --> 00:00:10.800 Hello everyone, and welcome back to another episode of Adventures 2 00:00:10.800 --> 00:00:13.839 in DevOps. Today, I have a slight bit of news though. 3 00:00:14.160 --> 00:00:16.679 We have an upgrade to our podcast, as will is 4 00:00:16.719 --> 00:00:18.039 a way for a few episodes. 5 00:00:18.039 --> 00:00:19.440 So I've asked Amy. 6 00:00:19.199 --> 00:00:23.600 Knight, are expert on reliability architecture, to step in and Amy, 7 00:00:23.640 --> 00:00:25.000 are you ready for today's episode? 8 00:00:25.519 --> 00:00:26.920 I am excited to be here. 9 00:00:27.559 --> 00:00:28.000 That's good. 10 00:00:28.239 --> 00:00:30.199 I think I put her on the spot with calling 11 00:00:30.199 --> 00:00:33.439 her the expert. I don't think she was prepared for that. 12 00:00:34.719 --> 00:00:38.000 No, not ever expert she is. 13 00:00:38.159 --> 00:00:42.759 She's very modest, I will say that. So today's guest 14 00:00:42.840 --> 00:00:46.280 is Brian POTRELLI welcome and thank you for coming. He's 15 00:00:46.280 --> 00:00:49.200 a founder of Fusion auth and some other products as well. 16 00:00:49.359 --> 00:00:51.600 It's nice to have a fellow of expert on the show. 17 00:00:51.600 --> 00:00:53.479 I have to say, yeah, thanks for having me us. 18 00:00:54.520 --> 00:00:56.640 I was interested in how this was going to go 19 00:00:56.880 --> 00:01:00.359 with two experts. 20 00:01:02.079 --> 00:01:03.679 You know, I get this question like how did you 21 00:01:03.719 --> 00:01:06.599 become an expert? And usually I say something like well, 22 00:01:06.760 --> 00:01:10.359 I started investing in learning about security a lot of 23 00:01:10.400 --> 00:01:12.480 years ago, and after a lot of years, then I 24 00:01:12.480 --> 00:01:14.079 get up on stage and someone asked me how did 25 00:01:14.079 --> 00:01:17.400 you become an off expert? I don't think there's like 26 00:01:17.799 --> 00:01:19.840 a dedicated path, Brian, how did you end up in 27 00:01:19.879 --> 00:01:20.359 this area? 28 00:01:20.640 --> 00:01:21.319 Kind of dumb? 29 00:01:21.400 --> 00:01:21.640 Luck. 30 00:01:23.000 --> 00:01:27.040 So, we we are working on a sort of a 31 00:01:27.159 --> 00:01:30.480 niche product, and after we realized that it wouldn't scale, 32 00:01:31.439 --> 00:01:34.640 we actually wanted to start building a couple of other products. 33 00:01:34.719 --> 00:01:37.079 And so one of the products ideas that we came 34 00:01:37.159 --> 00:01:39.719 up with was a forum, and so we actually built 35 00:01:39.760 --> 00:01:43.519 that out completely. And when we built it, we decided 36 00:01:43.560 --> 00:01:45.680 we weren't going to add a log in and registration 37 00:01:45.760 --> 00:01:49.640 component to the forum itself. Instead, we were always going 38 00:01:49.680 --> 00:01:53.319 to delegate that to whatever the company already had, and 39 00:01:53.400 --> 00:01:58.120 so that required some type of authentication system, right, And 40 00:01:58.159 --> 00:02:00.319 in order for us to test this, we had build 41 00:02:00.319 --> 00:02:03.519 our own authentication system, and so we did, right. So 42 00:02:03.560 --> 00:02:06.799 we built it, we integrated with our forum. Everything worked, 43 00:02:06.799 --> 00:02:09.719 it was magical, and then we were like, hey, this 44 00:02:09.800 --> 00:02:11.680 afe thing's pretty cool. Maybe we'll just use it for 45 00:02:12.080 --> 00:02:13.800 like some of our other apps, and so we started 46 00:02:13.800 --> 00:02:16.599 integrating it with everything. Well, the forum didn't work out, 47 00:02:17.319 --> 00:02:19.199 and we're all kind of sitting in a room and 48 00:02:19.240 --> 00:02:22.479 I'm like, guys, this off things really neat, Like let's 49 00:02:22.479 --> 00:02:26.319 do that instead. So we like basically said a one 50 00:02:26.360 --> 00:02:30.039 to eighty dropped the forum started focusing on off and 51 00:02:30.080 --> 00:02:32.960 then had to go learn a bunch of standards and 52 00:02:33.000 --> 00:02:36.520 read a bunch of specifications and you know learn, you know, 53 00:02:36.879 --> 00:02:39.199 learn a lot about security in a very short period 54 00:02:39.240 --> 00:02:43.439 of time. So but totally luck, No. 55 00:02:43.479 --> 00:02:44.120 I totally got it. 56 00:02:44.120 --> 00:02:46.080 Actually, we ended up in a similar spot with a 57 00:02:46.199 --> 00:02:49.560 previous product that, uh, there was a lot of complexity 58 00:02:49.599 --> 00:02:51.680 and what we were doing, and we found that our 59 00:02:51.719 --> 00:02:54.639 customers were actually more interested in how we solve our 60 00:02:54.680 --> 00:02:58.199 technical problems than the product that we were offering. At 61 00:02:58.199 --> 00:03:01.639 that time, this was like a for COVID, and we 62 00:03:01.639 --> 00:03:04.240 were trying to sell like leaders of Leadership SaaS and 63 00:03:04.319 --> 00:03:06.719 it turns out a lot of companies wanted to say 64 00:03:06.840 --> 00:03:09.719 that they cared about leaders and building their leaders up, 65 00:03:09.719 --> 00:03:11.919 but they didn't actually want to pay for it. Maybe 66 00:03:11.919 --> 00:03:13.000 that's a little bit of a shocker. 67 00:03:13.199 --> 00:03:15.960 Yeah, not shocking, shocking, shocking, right. 68 00:03:16.680 --> 00:03:19.039 The main topic of today is I think it's going 69 00:03:19.080 --> 00:03:20.879 to be a little bit of a controversial episode. This 70 00:03:20.919 --> 00:03:23.840 may be our most controversial episode yet multi tenant versus 71 00:03:23.879 --> 00:03:28.879 single tenant architecture. Yeah, I guess that probably a lot 72 00:03:28.879 --> 00:03:31.159 of our audience already has a strong opinion one way 73 00:03:31.240 --> 00:03:31.599 or the other. 74 00:03:32.039 --> 00:03:32.840 Amy any thoughts. 75 00:03:33.080 --> 00:03:35.319 I guess my thought is just like alarm bells start 76 00:03:35.360 --> 00:03:36.960 to go off in my head. But there we go 77 00:03:37.039 --> 00:03:38.719 back with like the reliability stuck. 78 00:03:38.919 --> 00:03:40.520 Like you're like, if someone says we're going to go 79 00:03:40.599 --> 00:03:43.120 for multi tenant architecture. 80 00:03:42.599 --> 00:03:47.400 Like that, hold up, yeah, exactly exactly, Like I want 81 00:03:47.439 --> 00:03:48.680 to note the details. 82 00:03:48.240 --> 00:03:50.879 At this point, yees, so you think it's like inherently 83 00:03:50.960 --> 00:03:53.400 dangerous to have multi tenant architecture. 84 00:03:53.520 --> 00:03:56.639 I was just at the point where like the I 85 00:03:56.639 --> 00:03:59.479 don't know if the more separation the better with that 86 00:03:59.759 --> 00:04:02.479 sort of thing, But prove me wrong. 87 00:04:03.479 --> 00:04:05.560 Well, Brian's built a company on top of it. 88 00:04:06.039 --> 00:04:10.639 Yes, we're building this this forum, and then we built 89 00:04:10.639 --> 00:04:14.520 this sort of standalone, you know, off thing. And our 90 00:04:14.560 --> 00:04:18.720 other product was also you know, downloadable and single tenant 91 00:04:18.759 --> 00:04:23.040 and standalone, and we did we built the original product, 92 00:04:23.040 --> 00:04:26.040 which is called clean Speak. We built that that way 93 00:04:26.079 --> 00:04:28.920 because it was high performance, right, so like it filters 94 00:04:29.040 --> 00:04:32.160 profanity and chat so wow, a couple way of saying it. 95 00:04:32.600 --> 00:04:37.439 And so we're talking billions, tens of billions, hundreds of 96 00:04:37.439 --> 00:04:40.240 billions of chat messages a month, and so in order 97 00:04:40.279 --> 00:04:44.879 to do that in a multi tenant way is pretty challenging, 98 00:04:45.120 --> 00:04:48.199 especially when you're talking about super low latency with something 99 00:04:48.240 --> 00:04:51.360 like chat and so what happens is like you're in 100 00:04:51.399 --> 00:04:53.639 a game, you send a chat message, it goes across 101 00:04:53.639 --> 00:04:56.800 the wire into this, you know, the chat server. Okay, 102 00:04:56.800 --> 00:04:58.399 so that's in a bank of servers that's sitting in 103 00:04:58.439 --> 00:05:01.279 AWS or you know, somewhere like that. There's no reason 104 00:05:01.279 --> 00:05:03.319 you should have to jump out across the open Internet 105 00:05:03.360 --> 00:05:06.000 again to go filter that and see if there's any 106 00:05:06.040 --> 00:05:07.800 issues with it and then come all the way back. 107 00:05:07.839 --> 00:05:11.040 You know, that could introduce one hundred milliseconds of latency. 108 00:05:11.240 --> 00:05:13.639 It'd be way better if it just jumps across the 109 00:05:13.720 --> 00:05:18.120 backplane and essentially goes right to you know, the filtering service. 110 00:05:18.439 --> 00:05:20.959 And we could do that in like under a millisecond. 111 00:05:21.199 --> 00:05:23.639 And even with you know, in internetwork latency, it's like 112 00:05:23.639 --> 00:05:25.959 one or two milliseconds, so we can really shave down 113 00:05:26.079 --> 00:05:29.319 milliseconds here. And then we took all of this sort 114 00:05:29.360 --> 00:05:32.240 of like common code and deployment models and bundling and 115 00:05:32.279 --> 00:05:34.319 all this stuff we had and we just like copy 116 00:05:34.360 --> 00:05:35.040 and pasted it. 117 00:05:35.480 --> 00:05:38.160 So the history sort of put you in this direction. 118 00:05:38.319 --> 00:05:41.040 It was sort of like a sign from the universe 119 00:05:41.079 --> 00:05:43.120 that maybe this was the path for you to take. 120 00:05:43.399 --> 00:05:45.319 We kind of made a bold assumption that that was 121 00:05:45.319 --> 00:05:47.920 a good idea. It's like, hey, let's make this downloadable, 122 00:05:48.720 --> 00:05:51.279 let's make it single tenant, let's make it run anywhere 123 00:05:51.279 --> 00:05:54.680 in the world. And then we actually started having all 124 00:05:54.720 --> 00:05:57.759 these companies come to us, some of them quite large, 125 00:05:58.160 --> 00:06:01.240 and they're like, hey, we can't use your competitors. We 126 00:06:01.279 --> 00:06:03.519 can't use like A zero because it's in the cloud 127 00:06:03.560 --> 00:06:05.240 on it, we can't use you know, like pain because 128 00:06:05.240 --> 00:06:07.480 they're just pushing everybody to cloud, and we can't use 129 00:06:07.519 --> 00:06:10.319 all these tools, and we really want something that we 130 00:06:10.319 --> 00:06:12.519 can run in our own data center. And I was like, well, 131 00:06:12.560 --> 00:06:17.199 we got that, so yeah, let's totally do this right. 132 00:06:17.319 --> 00:06:19.360 So that's really interesting. 133 00:06:19.399 --> 00:06:23.800 So taking the architecture as the value being provided by 134 00:06:23.839 --> 00:06:26.800 the product as sort of the competitive advantage, yep. 135 00:06:27.000 --> 00:06:29.040 Yeah, we continue to do that. So, like I think 136 00:06:29.079 --> 00:06:31.800 our one of our taglines we've been messing around with 137 00:06:31.959 --> 00:06:35.040 is like aughts so modern you can download it right, 138 00:06:35.680 --> 00:06:38.839 and it's it's really flipping everything on its head because 139 00:06:39.600 --> 00:06:43.480 for the last gosh fifteen years, people have been like, yep, 140 00:06:43.680 --> 00:06:45.519 it's got to be multi tenant, it's got to be 141 00:06:45.759 --> 00:06:49.000 you know, SaaS, it's got to be cloud, and that's 142 00:06:49.000 --> 00:06:52.240 the only way to build an effective company, profitable company, 143 00:06:52.319 --> 00:06:55.399 scalable company. And I'm like, no. 144 00:06:58.160 --> 00:07:02.360 I'll be honest too, what you're saying. It kind of 145 00:07:02.399 --> 00:07:05.720 shocked me. Conversations that I've been having more and more 146 00:07:05.759 --> 00:07:09.000 people are actually people are considering moving off cloud to 147 00:07:09.079 --> 00:07:12.160 save costs, which is nothing I would have ever thought 148 00:07:12.160 --> 00:07:15.040 of before I had this initial conversation like two years 149 00:07:15.079 --> 00:07:16.240 ago with someone. 150 00:07:16.600 --> 00:07:20.279 Yeah, if if you have let's say, like a cloud 151 00:07:20.360 --> 00:07:25.480 native Lamba driven, you know, lots of like io into 152 00:07:25.560 --> 00:07:28.639 like something like Dynamo or something like that, Like sure, 153 00:07:28.680 --> 00:07:31.800 when you start and you're small, can be very cost effective. 154 00:07:32.160 --> 00:07:36.399 When you scale, it can get so expensive these services 155 00:07:36.439 --> 00:07:38.079 are like outrageous. 156 00:07:38.560 --> 00:07:42.240 The other thing I would say too, is depending on 157 00:07:42.600 --> 00:07:46.000 your architecture and how long it's been around. I feel 158 00:07:46.000 --> 00:07:48.399 like moving to the cloud, you're sort of forced into 159 00:07:48.480 --> 00:07:53.079 certain boxes of what they can scale most efficiently. So 160 00:07:53.360 --> 00:07:55.720 if you have something that was not built to run 161 00:07:55.759 --> 00:07:58.000 on a certain machine type, you're going to run into 162 00:07:58.079 --> 00:08:01.000 issue that scale. Would that be something that you've kind 163 00:08:01.040 --> 00:08:02.279 of experienced. 164 00:08:01.759 --> 00:08:07.240 Too, Absolutely, Yeah, The the cloud providers really love to 165 00:08:07.319 --> 00:08:10.399 push you in the directions that they feel the most 166 00:08:10.439 --> 00:08:13.600 confident and comfortable, and they where they have the biggest profit. 167 00:08:13.759 --> 00:08:16.959 Right if you look at just you know, like Cognito 168 00:08:17.000 --> 00:08:19.920 as an example in our space, like that's not a 169 00:08:19.920 --> 00:08:25.920 cost effective solution, like by anyone's anyone's you know view. Now, 170 00:08:25.959 --> 00:08:29.040 when you first start, it's very cost effective because it's free, 171 00:08:29.120 --> 00:08:31.920 and you know, it's pretty easy to integrate. But the 172 00:08:31.959 --> 00:08:34.600 second that you like start scaling and start enabling any 173 00:08:34.639 --> 00:08:39.159 of the features, it's cost prohibitive. And so they know 174 00:08:39.240 --> 00:08:41.080 how to scale that and run that really well, and 175 00:08:41.120 --> 00:08:43.360 then they know how to monetize the crap out of it. 176 00:08:43.480 --> 00:08:46.200 So I love that you, you know, you tried to 177 00:08:46.200 --> 00:08:48.159 pull back on like, well, you know, maybe it's really 178 00:08:48.159 --> 00:08:51.320 easy to integrate with Cognito, although I feel like, I see, 179 00:08:51.320 --> 00:08:53.279 I'm in a bunch of tech communities and I just 180 00:08:53.279 --> 00:08:56.840 see complaints every all day long about oh I tried 181 00:08:56.879 --> 00:08:59.600 to make this work and it didn't happen, or you know, 182 00:08:59.639 --> 00:09:02.679 I looked at the documentation and start saying like amplify everywhere, 183 00:09:02.759 --> 00:09:04.600 like I don't want to use amplify. I'm like, well, 184 00:09:04.639 --> 00:09:07.200 you know, welcome to Welcome to the mess that that 185 00:09:07.320 --> 00:09:11.320 is Cognito in AWS. So I started I stopped saying that, Yeah, 186 00:09:11.320 --> 00:09:14.360 it may be cost effective, you know, a small scale, 187 00:09:14.360 --> 00:09:17.399 but the overhead, the total cost of ownership, yeah, definitely 188 00:09:17.480 --> 00:09:19.480 high with with Cognito, for sure. 189 00:09:19.600 --> 00:09:22.559 It was super painful. And Amazon also does this really 190 00:09:22.559 --> 00:09:25.559 funny thing where they force you to use like you 191 00:09:25.600 --> 00:09:28.039 can't just use Cognito, like you said, you have to 192 00:09:28.200 --> 00:09:30.679 pull in fifteen other services and you got to use 193 00:09:30.679 --> 00:09:33.639 platformation and you got to do this, and it's just like, guys, 194 00:09:33.679 --> 00:09:36.919 this is getting insane, right, Like I don't want to 195 00:09:36.919 --> 00:09:38.720 do this. I just I just want to lock my users. 196 00:09:39.159 --> 00:09:42.039 I may have cheated and listened to some of your 197 00:09:42.039 --> 00:09:45.679 episodes on competitor podcasts to kind of get a better 198 00:09:45.720 --> 00:09:49.320 understanding before this one. Is it still true also that 199 00:09:49.360 --> 00:09:51.480 the customization is just not there. 200 00:09:52.159 --> 00:09:55.120 Cognito actually made a really big update, so I have 201 00:09:55.200 --> 00:09:59.240 to credit they. I think they restructured their product team 202 00:09:59.440 --> 00:10:03.000 and then they put some dedicated engineers on Cognito. For 203 00:10:03.039 --> 00:10:05.840 the last like eighteen months, they kind of up their game, 204 00:10:06.039 --> 00:10:10.159 Like you can quasi customize their log in. You can 205 00:10:10.200 --> 00:10:13.600 customize like the email templates and some of the messaging templates, 206 00:10:13.600 --> 00:10:17.519 and I think they even like support localization stuff. So 207 00:10:17.639 --> 00:10:20.879 they did a pretty decent job of like leveling themselves 208 00:10:21.000 --> 00:10:23.639 up to your point, it's like it's still a pain 209 00:10:23.679 --> 00:10:26.639 in about to integrate with and it's just feature crippled. 210 00:10:26.679 --> 00:10:29.879 I mean, they just like have it's a very limited platform. 211 00:10:30.120 --> 00:10:32.759 I think the biggest challenge here is something that a 212 00:10:32.799 --> 00:10:35.039 lot of people dismiss when they're selecting a product out 213 00:10:35.039 --> 00:10:37.639 there in the world, is a the alignment with the 214 00:10:37.679 --> 00:10:39.519 company that you're going with, Like what is the core 215 00:10:39.639 --> 00:10:42.159 value that the company really focus on and cares about? Right, 216 00:10:42.320 --> 00:10:46.080 you were saying that the single tendency is the core 217 00:10:46.240 --> 00:10:47.840 value that we're you know, one of the core values 218 00:10:47.840 --> 00:10:50.039 that we're offering here, you know, is that aligned, Like 219 00:10:50.039 --> 00:10:51.799 don't just use the product based off of the features, 220 00:10:51.840 --> 00:10:53.799 but you know, look to see about the team that's 221 00:10:53.799 --> 00:10:56.000 building in and what their you know, long term direction is. 222 00:10:56.480 --> 00:10:59.039 And you know, I think it's really important because when 223 00:10:59.039 --> 00:11:01.159 we look at things like cogn it's not really clear 224 00:11:01.320 --> 00:11:02.919 you know, what they're really going after. And if you 225 00:11:02.919 --> 00:11:05.120 start comparing the products and get really deep, you can 226 00:11:05.159 --> 00:11:08.639 see things like they're like tendency in Cognito itself is 227 00:11:08.679 --> 00:11:11.399 not really there as a solution. If you have customers 228 00:11:11.399 --> 00:11:14.279 that are in the business space, it doesn't really fit 229 00:11:14.320 --> 00:11:16.960 well with that, and it's detaining your point, you know, 230 00:11:17.200 --> 00:11:20.200 on the refresh with Cognito. Yeah, I mean it's configurable 231 00:11:20.200 --> 00:11:22.399 now for sure, But there are nine different kinds of 232 00:11:22.480 --> 00:11:25.720 Lambda functions that you can integrate with Cognito. Like it's 233 00:11:25.759 --> 00:11:28.480 not a simple strategy, Like it's not just like configuration 234 00:11:28.600 --> 00:11:30.240 driven a lot of times, and this is one of 235 00:11:30.320 --> 00:11:33.279 the huge types with AWO services in general. It's that 236 00:11:33.320 --> 00:11:35.240 if you want to configure it, you write your own 237 00:11:35.320 --> 00:11:38.399 LAMB to function with your own configuration with whatever, non 238 00:11:38.440 --> 00:11:42.360 documented payloads and responses, and then figure out how to 239 00:11:42.360 --> 00:11:44.600 integrate that into your solution and deployed and whatever to 240 00:11:45.039 --> 00:11:46.360 the cloud to get it to work right, and if 241 00:11:46.399 --> 00:11:48.840 there's a problem then you know, good luck. So some 242 00:11:48.840 --> 00:11:51.399 people love this, Oh yeah, all of my infrastructure everything 243 00:11:51.440 --> 00:11:55.320 is in AWS and you know, good on you, for sure. 244 00:11:55.360 --> 00:11:57.440 But I don't wish Cognito on anyone. 245 00:11:57.759 --> 00:12:00.399 Yeah, I mean that's a good point, right, because they 246 00:12:01.240 --> 00:12:06.799 AWS has this sort of disease I guess you could say, 247 00:12:06.799 --> 00:12:10.639 where they constantly want you to use their services to 248 00:12:10.679 --> 00:12:12.919 do things right. So like, you know, we use like 249 00:12:13.240 --> 00:12:16.879 cloud front, and in order to do all the redirects 250 00:12:16.919 --> 00:12:18.600 that we need to do, we have to use like 251 00:12:18.600 --> 00:12:21.639 an edge function, and then that edge functions go loads 252 00:12:21.679 --> 00:12:23.559 something out of like an S three bucket in order 253 00:12:23.600 --> 00:12:25.519 to like download it, and then it caches it, and 254 00:12:25.799 --> 00:12:27.960 you know, it's like it's ridiculous. It's like all I 255 00:12:28.000 --> 00:12:29.320 want to do is set up redirects, and I have 256 00:12:29.360 --> 00:12:32.240 to jump through like fifteen hoops just to get something 257 00:12:32.320 --> 00:12:35.759 so simple done because of the way that Amazon has 258 00:12:35.799 --> 00:12:37.879 designed everything, and they're like, well just stick a lamb 259 00:12:38.000 --> 00:12:40.840 on it, you know that don't work, and don't document. 260 00:12:41.159 --> 00:12:42.320 We'll document it later, you know. 261 00:12:43.600 --> 00:12:45.840 I mean, I do appreciate as a company, like comparing 262 00:12:45.879 --> 00:12:47.960 them to the other cloud providers, And I didn't know 263 00:12:47.960 --> 00:12:49.519 this was the direction we were going to go with 264 00:12:49.320 --> 00:12:52.360 this with this episode, but I do appreciate a WS 265 00:12:52.440 --> 00:12:55.600 more than the other ones. I do think they suffer 266 00:12:55.679 --> 00:12:58.960 from a very focused mindset as sort of the two 267 00:12:58.919 --> 00:13:01.559 piece of teams that they have. They're really focused on delivering, 268 00:13:01.720 --> 00:13:04.080 you know, exactly the thing that maybe that customer wants, 269 00:13:04.080 --> 00:13:06.480 but that may mean over time, they're missing some other 270 00:13:06.519 --> 00:13:08.919 critical features. I think what you're talking about here. A 271 00:13:08.960 --> 00:13:13.480 good example is returning security headers on outfront responses. So 272 00:13:13.720 --> 00:13:15.559 you've got some data stort inn s Rebucket, and you're 273 00:13:15.559 --> 00:13:17.919 hosting a website or your API, and you're like, I 274 00:13:18.000 --> 00:13:21.159 just want to add a header that removes the x 275 00:13:21.200 --> 00:13:23.360 frame options, like you can embed this as a as 276 00:13:23.399 --> 00:13:25.759 an iframe in a website, And you'd be like, I 277 00:13:25.799 --> 00:13:27.120 just want to add a header, and I don't want 278 00:13:27.120 --> 00:13:28.960 to muck with the underlying service, which I may not 279 00:13:29.039 --> 00:13:31.879 even own. Right, Maybe I've deployed Fusion ofth or some 280 00:13:31.919 --> 00:13:34.440 other service in a container in my infrastructure and I 281 00:13:34.480 --> 00:13:36.200 don't even control that product, but I want to add 282 00:13:36.200 --> 00:13:38.799 some headers to it. Well, good luck, honestly, because up 283 00:13:38.879 --> 00:13:41.240 until about three years ago, you couldn't do this without 284 00:13:41.240 --> 00:13:43.559 throwing a huge lamb to function at it. Now they 285 00:13:43.639 --> 00:13:46.600 do have something called the response policy headers that you 286 00:13:46.600 --> 00:13:48.960 can set in clautform, But it's taken probably like almost 287 00:13:48.960 --> 00:13:52.120 ten years for this to roll around and even be 288 00:13:52.159 --> 00:13:55.279 added in this feature, while other CDNs have been providing 289 00:13:55.480 --> 00:13:57.840 this baseline thing for quite a long time. 290 00:13:58.080 --> 00:14:01.360 Yeah, no, totally agree. They there. I think their product 291 00:14:01.360 --> 00:14:04.320 teams are so isolated and they don't have a lot 292 00:14:04.360 --> 00:14:07.600 of input from you know, just standard devs like us 293 00:14:07.679 --> 00:14:08.879 right that are just like, hey, I just want to 294 00:14:08.879 --> 00:14:12.159 get this thing done. And they're they're really listening, you know, 295 00:14:12.320 --> 00:14:14.519 to their largest customers, which we all know. I mean, 296 00:14:14.559 --> 00:14:16.639 you know, they make tens of millions of dollars off 297 00:14:16.639 --> 00:14:20.519 some of these customers every so, so yeah, it's it's 298 00:14:20.519 --> 00:14:23.840 pretty painful and they takes a long time for them 299 00:14:23.840 --> 00:14:26.919 to get actual stuff done. But at least they have 300 00:14:27.200 --> 00:14:29.279 the lambdas and the things and so like you have 301 00:14:29.279 --> 00:14:30.720 to jump into some code and you got to do 302 00:14:30.759 --> 00:14:34.039 these things. But at least there are workarounds where like 303 00:14:34.080 --> 00:14:36.440 you said, you know, there have been times where you know, 304 00:14:36.480 --> 00:14:39.000 in the past where we use services and you literally 305 00:14:39.039 --> 00:14:41.080 just can't do it. So it's like, well, crap, Okay, 306 00:14:41.120 --> 00:14:43.960 I'll stand up another service to proxy requests through and 307 00:14:43.960 --> 00:14:46.720 then I'll manipulate the request in my proxy service and 308 00:14:46.759 --> 00:14:50.039 then ask it to clout right, and it's like, oh 309 00:14:50.080 --> 00:14:52.600 my gosh, why am I having to manage all this 310 00:14:52.720 --> 00:14:55.080 junk just to add ahead or business? And I totally 311 00:14:55.159 --> 00:14:59.519 agree with that assessment of AWS, But again, they're one 312 00:14:59.559 --> 00:15:02.840 of the better out there, right. So here's the other 313 00:15:02.840 --> 00:15:06.799 thing that's interesting AWS. I think some of their product 314 00:15:06.840 --> 00:15:11.840 teams are actually starting to evaluate letting people download some 315 00:15:11.960 --> 00:15:13.960 of their services and run them locally, right, Like we 316 00:15:14.039 --> 00:15:16.519 see that a little bit with like Dynamo dB. You 317 00:15:16.519 --> 00:15:18.200 can run a like kind of a scraped down version 318 00:15:18.200 --> 00:15:22.960 of it. I think there's some local lambda execution things 319 00:15:23.000 --> 00:15:27.080 that you can do, and they're looking at more and 320 00:15:27.120 --> 00:15:29.399 so you know, maybe they'll do that with Cognito at 321 00:15:29.440 --> 00:15:31.440 some point too. But like, one of the hiccups that 322 00:15:31.480 --> 00:15:34.840 I see with all the cloud providers is the ability 323 00:15:34.919 --> 00:15:38.919 for a developer to test these services without having to 324 00:15:39.120 --> 00:15:42.320 like fire up an entire org, you know, run a 325 00:15:42.320 --> 00:15:44.840 bunch of terraform or CloudFormation or whatever you need to 326 00:15:44.879 --> 00:15:46.919 get everything set up, and then you can run your 327 00:15:46.919 --> 00:15:51.320 tests against it and good luck having twenty developers try 328 00:15:51.360 --> 00:15:53.159 to run tests at the same time where you're constantly 329 00:15:53.159 --> 00:15:56.279 tearing things down and recreating them and so moving that 330 00:15:56.320 --> 00:15:59.320 stuff back local I think is actually something that the 331 00:15:59.320 --> 00:16:02.720 industry has been at, which means that every developer can 332 00:16:02.759 --> 00:16:05.960 isolate themselves at dev time from every other one and 333 00:16:06.000 --> 00:16:07.080 that's a huge benefit. 334 00:16:07.279 --> 00:16:10.559 Yeah, I think the developer experience story with aws like 335 00:16:10.679 --> 00:16:13.519 offline is not the best so much that there's an 336 00:16:13.639 --> 00:16:17.759 entire company dedicated actually out of Switzerland called local Stack 337 00:16:18.080 --> 00:16:22.200 to emulate the emulate it offline, and you know, it's great. 338 00:16:22.200 --> 00:16:23.960 And the joke is like when is Agbo's going to 339 00:16:23.960 --> 00:16:27.840 buy local Stack because there's local Stack in the documentation, 340 00:16:28.039 --> 00:16:31.639 there's local Stack replacements for a DYNAMOITYB local which doesn't work. 341 00:16:31.679 --> 00:16:35.679 There's great integrations for Lambda and for SAM, the servilist 342 00:16:35.679 --> 00:16:38.360 application Transform for Information. 343 00:16:38.480 --> 00:16:39.360 It's really surprising. 344 00:16:39.399 --> 00:16:40.919 It's like one of the things like as you mentioned, 345 00:16:41.000 --> 00:16:44.039 like even though our product is focused one hundred percent SaaS, 346 00:16:44.519 --> 00:16:48.679 we offer a shim clone of our API for companies 347 00:16:48.679 --> 00:16:51.480 around locally because of course, like you're a developer and 348 00:16:51.519 --> 00:16:54.360 you're like, you don't care about maybe the off part, 349 00:16:54.440 --> 00:16:56.799 but your services depend on it, and so you need 350 00:16:56.840 --> 00:16:59.600 to have an answer here. And it's ridiculous how SaaS 351 00:16:59.600 --> 00:17:02.720 providers everywhere. I figured this out, but the cloud providers 352 00:17:03.399 --> 00:17:05.000 haven't done it yet. 353 00:17:05.200 --> 00:17:07.519 Yeah, you know, it's wild. We sort of like to 354 00:17:07.720 --> 00:17:10.279 take that to the nth degree, right, which is like 355 00:17:10.680 --> 00:17:13.200 mocking is dangerous, and so we'll market and then we'll 356 00:17:13.240 --> 00:17:15.400 just simulate the responses we know we get from production, 357 00:17:15.519 --> 00:17:18.039 so like maybe they record it and then they grab 358 00:17:18.119 --> 00:17:19.880 some you know, some stuff out of production, and then 359 00:17:19.920 --> 00:17:22.759 they can start simulating, well, what happens if like a 360 00:17:22.839 --> 00:17:25.440 lambda changes and then the responses start changing, or like 361 00:17:25.680 --> 00:17:29.480 AWS modifies something and it's coming back slightly differently. 362 00:17:29.759 --> 00:17:34.960 We definitely see that more sophisticated customers care about it, 363 00:17:35.359 --> 00:17:38.759 and end testing or integration testing and having a story 364 00:17:38.799 --> 00:17:41.799 dedicated to that is an important aspect, especially when you're 365 00:17:41.799 --> 00:17:46.039 offering something that's essentially infrastructure for your your customers. I 366 00:17:46.039 --> 00:17:48.359 mean different when it's just like some third party service 367 00:17:48.359 --> 00:17:51.400 which is solving some edge case or some CRM, but 368 00:17:51.440 --> 00:17:54.000 when it becomes a critical piece of infrastructure for your customers, 369 00:17:54.119 --> 00:17:56.160 it's a question that's going to come up pretty frequently. 370 00:17:56.319 --> 00:17:58.640 Absolutely, I kind of had a question I was going 371 00:17:58.680 --> 00:18:01.319 to ask, kind of been like doing how to ask 372 00:18:01.359 --> 00:18:05.240 this in the most intelligent way possible? How much of 373 00:18:05.319 --> 00:18:08.519 your product would you say? In most customers use cases, 374 00:18:08.559 --> 00:18:12.440 it probably simplifies things like performance and load testing. My 375 00:18:12.519 --> 00:18:15.160 experience and the source of like massive bought attack and 376 00:18:15.240 --> 00:18:18.680 at the same time my experience performance testing in a 377 00:18:18.759 --> 00:18:22.559 multi tenant environment can be tricky because while you would 378 00:18:22.599 --> 00:18:25.359 assume that you can just go off to the races, 379 00:18:25.440 --> 00:18:30.119 that is a false assumption. And if you have to 380 00:18:30.319 --> 00:18:33.640 performance test load test at a certain scale, it requires 381 00:18:33.680 --> 00:18:36.160 coordinating with the company and making sure that other customers 382 00:18:36.200 --> 00:18:38.319 aren't doing it at the exact same time. So I 383 00:18:38.319 --> 00:18:41.279 guess long story short, I kind of maybe know the answer, 384 00:18:41.319 --> 00:18:42.359 but maybe someone to speak to. 385 00:18:42.599 --> 00:18:46.119 Yeah, it's a phenomenal question. So there's sort of two 386 00:18:46.240 --> 00:18:49.559 aspects of this, and the first one is I'm a 387 00:18:49.599 --> 00:18:51.720 developer and I just want to like sort of load 388 00:18:51.759 --> 00:18:53.759 test locally, and I want to I just want to 389 00:18:53.759 --> 00:18:56.440 see what sort of throughput looks like with different configuration 390 00:18:56.519 --> 00:18:58.720 options on my laptop. And you download, you get it 391 00:18:58.759 --> 00:19:01.400 set up, you integrate, and then you just literally use 392 00:19:01.400 --> 00:19:03.519 a hammer, hammer the crap out of it and see 393 00:19:03.880 --> 00:19:06.920 see what happens, and then you can reset everything and 394 00:19:06.960 --> 00:19:08.640 then you can try it in the cloud. When you 395 00:19:08.640 --> 00:19:11.319 have a multi tenant provider, they even like a lot 396 00:19:11.319 --> 00:19:13.759 of times say you load tests are just not allowed, 397 00:19:14.440 --> 00:19:18.599 right and like period hard stop. You pay them enough money, 398 00:19:18.599 --> 00:19:20.240 they'll probably let you load test, but they're going to 399 00:19:20.279 --> 00:19:22.319 have to figure out how to get that traffic off 400 00:19:22.359 --> 00:19:25.160 of the main servers because they don't want to impact 401 00:19:25.359 --> 00:19:27.519