WEBVTT 1 00:00:05.440 --> 00:00:08.400 Hey, growl. Welcome to another episode of Ruby Ropes. I'm 2 00:00:08.480 --> 00:00:10.960 David Kumia and today on our panel we have Luke 3 00:00:11.000 --> 00:00:15.839 Sutter's Stutters, and we have John Epperson everybody, and today 4 00:00:15.919 --> 00:00:18.120 we have a special guest, Jesse Speedback. 5 00:00:18.320 --> 00:00:19.079 Great to be here. 6 00:00:19.280 --> 00:00:21.679 So, Jesse, would you mind explaining just a bit about 7 00:00:21.719 --> 00:00:23.559 who you are, some of the things that you're doing, 8 00:00:24.000 --> 00:00:25.839 who you work for, and what are you're famous and 9 00:00:25.879 --> 00:00:26.640 all that good stuff. 10 00:00:27.879 --> 00:00:31.120 Yeah. Absolutely, so. My name is Jessin Speedback. I'm a 11 00:00:31.239 --> 00:00:34.520 senior engineer at a company called Ibata, which is a 12 00:00:34.520 --> 00:00:38.200 cash back for shopping app based in Denver, Colorado. I've 13 00:00:38.200 --> 00:00:40.600 been working there for about three and a half years. 14 00:00:40.960 --> 00:00:42.880 We are to do some hiring, so check out our 15 00:00:42.920 --> 00:00:47.479 cruise page. I guess I'm famous as it were, because 16 00:00:47.960 --> 00:00:51.520 I gave a talk at the first Remote Rails comp 17 00:00:51.640 --> 00:00:54.920 this past May, and I talked about kind of how 18 00:00:54.960 --> 00:00:56.960 crummy of a developer I am. 19 00:00:57.119 --> 00:00:59.560 Yeah, I think we can all relate to that on 20 00:00:59.640 --> 00:01:03.439 the daily basis sometimes, So would you might have given 21 00:01:03.479 --> 00:01:07.200 a bit of a you know, highlight talk about what 22 00:01:07.599 --> 00:01:09.920 you covered the conference and stuff, so we can just 23 00:01:10.000 --> 00:01:11.040 kind of pick it up from there. 24 00:01:11.239 --> 00:01:13.560 We'll link to the in the show notes a link 25 00:01:13.599 --> 00:01:16.599 to the conference, but just for those who maybe didn't 26 00:01:16.640 --> 00:01:17.079 see it. 27 00:01:17.200 --> 00:01:21.359 Sure, and there's no substitute for actually watching this fantastic talk. 28 00:01:23.200 --> 00:01:28.760 But more more seriously, the talk is really about my 29 00:01:28.879 --> 00:01:31.599 experience as a tech lead and I bought a working 30 00:01:31.599 --> 00:01:34.719 on a pretty critical project over the course of about 31 00:01:35.599 --> 00:01:40.040 six six months or so, and over that time I 32 00:01:40.120 --> 00:01:45.079 made four very big mistakes that put the project in jeopardy. 33 00:01:45.599 --> 00:01:48.719 And hopefully there are mistakes that I will learn from 34 00:01:48.760 --> 00:01:51.159 and not make again as I continue to lead projects 35 00:01:51.159 --> 00:01:55.120 that I bought in the future. And my hope is 36 00:01:55.200 --> 00:02:00.760 that by sort of articulating these mistakes and what I 37 00:02:00.840 --> 00:02:03.760 learned from them, other folks can benefit. And so the 38 00:02:03.799 --> 00:02:06.400 four mistakes that I made are first, we picked the 39 00:02:06.439 --> 00:02:09.800 wrong technology. Can get more into that. We also, as 40 00:02:09.840 --> 00:02:13.319 a team we siloed work, so work was divided up 41 00:02:13.360 --> 00:02:17.599 in not the best way. We fell into the premature 42 00:02:17.680 --> 00:02:22.159 optimization trap. And then maybe worst of all, we made 43 00:02:22.680 --> 00:02:25.639 way too many changes at one time. So I can 44 00:02:25.680 --> 00:02:26.960 go into detail on any of those. 45 00:02:27.319 --> 00:02:30.719 Yeah, and I think it's fair to say that our 46 00:02:30.759 --> 00:02:34.599 listeners other members on the panel have been there before. 47 00:02:35.240 --> 00:02:38.280 Yeah, I've worked with loads of people who made those mistakes. 48 00:02:38.360 --> 00:02:41.199 Obviously I've never made them myself, but oh my word. 49 00:02:43.159 --> 00:02:45.199 Luke is the perfect developer. Let's be fair. 50 00:02:45.680 --> 00:02:48.039 No, I was actually going to jump in and say that, like, 51 00:02:48.719 --> 00:02:51.360 I feel like I don't feel like I'm alone, but 52 00:02:51.560 --> 00:02:55.759 usually when you make a mistake, it cascades into more 53 00:02:55.840 --> 00:02:59.479 for whatever reason. I generally feel like you you either 54 00:02:59.599 --> 00:03:01.719 kind of a rolling along and you feel pretty good 55 00:03:01.719 --> 00:03:05.719 about stuff, or you're just like in a bottomless pit 56 00:03:07.039 --> 00:03:09.479 and there's very little in between. Time you just suddenly 57 00:03:09.520 --> 00:03:13.080 notice that you're at the bottom. So yeah, just saying. 58 00:03:13.159 --> 00:03:15.680 And absolutely, and I just want to kind of call 59 00:03:15.719 --> 00:03:19.280 out that there's like a certain amount of privilege that 60 00:03:19.319 --> 00:03:22.240 comes with being able to talk about our mistakes. Right, 61 00:03:22.719 --> 00:03:25.639 I'm not worried that my boss is going to fire me, 62 00:03:26.000 --> 00:03:29.120 and I'm also not worried that folks won't take me 63 00:03:29.159 --> 00:03:32.560 seriously for getting this talk. If anything, it probably improves 64 00:03:32.560 --> 00:03:35.520 my reputation as evidence by getting to talk to three 65 00:03:35.560 --> 00:03:37.960 of you gentlemen. So I just want to call that 66 00:03:38.000 --> 00:03:40.840 out because I think it's important. 67 00:03:41.000 --> 00:03:43.680 I think that, Okay, So speaking of that, I think 68 00:03:43.680 --> 00:03:45.840 there's a give and take there too, So I think 69 00:03:45.879 --> 00:03:48.680 this is one of this will get into a thing 70 00:03:48.719 --> 00:03:52.039 that I care a lot about this particular subject because 71 00:03:52.319 --> 00:03:55.159 for most of my shoot, I still feel it today 72 00:03:55.240 --> 00:03:57.919 or whatever, it doesn't really matter. I still always feel 73 00:03:57.919 --> 00:04:01.599 this pressure to not let people know about the mistakes 74 00:04:01.599 --> 00:04:03.960 that I made, right, because letting people know about the 75 00:04:03.960 --> 00:04:07.159 mistakes that I made makes me more vulnerable to them 76 00:04:07.240 --> 00:04:09.520 not wanting me to work for them, right, like fire 77 00:04:09.599 --> 00:04:13.159 me whatever. The bad thing is that it's a boogeyman, 78 00:04:13.240 --> 00:04:16.360 it's not real. It's just a pressure, right, And so 79 00:04:17.399 --> 00:04:18.959 I guess what I'm trying to get at is like 80 00:04:19.240 --> 00:04:22.040 I kind of feel like one of the things that 81 00:04:22.120 --> 00:04:26.040 I always care a lot about, what mistakes is telling 82 00:04:26.040 --> 00:04:30.800 people about your mistakes, I actually have discovered in reality 83 00:04:31.920 --> 00:04:35.879 actually makes people respect you more, but it doesn't change 84 00:04:35.920 --> 00:04:38.360 the fact that it's like really hard, and I still 85 00:04:38.360 --> 00:04:39.680 feel that pressure to this day. 86 00:04:39.879 --> 00:04:42.360 Yeah, John, Yeah, I agree with what you're saying. I 87 00:04:42.360 --> 00:04:46.800 think that for a lot of us, talking about mistakes 88 00:04:46.839 --> 00:04:49.279 openly and honestly and what we learn from them actually 89 00:04:49.319 --> 00:04:53.040 builds our credibility. But that's not always the case, depending 90 00:04:53.079 --> 00:04:55.920 on sort of how you present what you look like. 91 00:04:56.639 --> 00:05:00.279 I think that you guys recently did an episode. I 92 00:05:00.279 --> 00:05:05.360 think you talked about issues of hiring and getting equality 93 00:05:05.360 --> 00:05:07.120 in a workplace, and I think that that kind of 94 00:05:07.120 --> 00:05:08.240 plays in here for sure. 95 00:05:08.680 --> 00:05:11.160 And I would go as far to say that it 96 00:05:11.160 --> 00:05:14.160 depends on what the mistake is, what kind of mistake 97 00:05:14.199 --> 00:05:18.279 it is. If it is just utter and complete negligence, 98 00:05:19.120 --> 00:05:21.959 then those aren't really the kind of mistakes I would 99 00:05:21.959 --> 00:05:25.399 really want to be forthcoming and outright about, you know. 100 00:05:25.560 --> 00:05:29.759 But like if I went in and always had a 101 00:05:29.879 --> 00:05:33.319 VP and tunnel into my production environment and then we 102 00:05:33.399 --> 00:05:37.720 got malware in our local work environment and that transferred 103 00:05:37.759 --> 00:05:41.360 over to production and encrypted all of our production data, 104 00:05:41.759 --> 00:05:43.720 I don't know if I would really say like, oh, yeah, 105 00:05:43.800 --> 00:05:46.199 you know, that was a silly mistake of mine. No, 106 00:05:46.480 --> 00:05:50.000 it's like, okay, not only of our customers effective, but 107 00:05:50.079 --> 00:05:54.399 now you know, my jobs in jeopardy because I decided 108 00:05:54.439 --> 00:05:58.040 to always take shortcuts. But something like and what I'm 109 00:05:58.079 --> 00:06:03.879 really interested in is your technology framework or the use 110 00:06:04.000 --> 00:06:07.120 the wrong technology? Can you explain a bit more about 111 00:06:07.160 --> 00:06:07.920 the scenario? 112 00:06:08.560 --> 00:06:12.439 Sure? Absolutely, so. This is the high level we had 113 00:06:12.480 --> 00:06:17.720 ibought it. We have just a wonderful majestic monolith rails application. Actually, 114 00:06:17.720 --> 00:06:20.680 originally the application we've written in Scala, and then after 115 00:06:20.720 --> 00:06:22.720 about a month of that, they switched it over and 116 00:06:23.040 --> 00:06:25.680 rebuilt the whole thing in Rails and it's been Rails 117 00:06:25.720 --> 00:06:29.360 ever since. So it's going on close almost ten years 118 00:06:29.360 --> 00:06:31.600 at this point. So it's a it's a large application, 119 00:06:31.680 --> 00:06:36.279 it serves millions of users. Very cool, and about three 120 00:06:36.360 --> 00:06:38.959 years ago when I joined, we started thinking. We started 121 00:06:39.000 --> 00:06:42.199 growing the team and thinking about how we could, like 122 00:06:42.639 --> 00:06:45.839 many people have pull pieces out of the monolith into 123 00:06:46.360 --> 00:06:49.959 micro services. So this project in particular was about taking 124 00:06:50.000 --> 00:06:52.800 a piece of billing logic from the system, from the 125 00:06:52.800 --> 00:06:55.959 monolith and pulling it out into a micro service. The 126 00:06:56.040 --> 00:06:59.519 hope was to make it better, encapsulated, easier to iterate on, 127 00:07:00.120 --> 00:07:03.560 not you know, isolate dependencies, every reason that you'd think 128 00:07:03.600 --> 00:07:07.639 to build a micro service. So we chose Actually, before 129 00:07:07.639 --> 00:07:11.319 I say the technology, because before I get trolled by 130 00:07:11.360 --> 00:07:15.040 all the lovers of this technology, I'm going to preface 131 00:07:15.079 --> 00:07:18.199 this by saying I don't think that this technology is wrong, 132 00:07:18.360 --> 00:07:21.120 and I don't think it's bad in and of itself. 133 00:07:21.560 --> 00:07:23.800 I just think it was not the right technology for 134 00:07:23.920 --> 00:07:24.560 the problem. 135 00:07:25.160 --> 00:07:28.399 And the team hold on a second hold on the 136 00:07:28.439 --> 00:07:32.319 second Jesse because the slide I'm looking at says big mistakes. 137 00:07:32.639 --> 00:07:36.759 It doesn't say small mistakes. It said big mistakes. So 138 00:07:36.879 --> 00:07:38.759 let's just keep this in mind, beful, what we reveal, 139 00:07:38.839 --> 00:07:41.560 what this technology is, What a big mistake. 140 00:07:41.399 --> 00:07:46.360 Was absolutely this. Whoever uses this technology is definitely making 141 00:07:46.399 --> 00:07:49.920 a big mistake. So you spoiler. We weren't building like 142 00:07:50.079 --> 00:07:53.759 us a side rails app micro service, which would probably 143 00:07:53.800 --> 00:07:56.519 would not have been as big a mistake. But the 144 00:07:56.639 --> 00:08:00.519 issue really was that our team, the small team of 145 00:08:00.560 --> 00:08:03.920 developers and then a larger team of engineers in the company, 146 00:08:04.360 --> 00:08:07.199 really did not have a ton of experience with the 147 00:08:07.240 --> 00:08:11.959 framework that we chose, and as a result, we ended 148 00:08:12.040 --> 00:08:15.160 up having to do a lot of plumbing and reinventing 149 00:08:15.199 --> 00:08:19.480 the wheel and just not benefiting from the institutional experience 150 00:08:19.519 --> 00:08:24.399 that exists at Ibottom. And unfortunately, right this this could 151 00:08:24.480 --> 00:08:26.680 work if you're doing kind of like a proof concept, 152 00:08:26.800 --> 00:08:29.959 like let's show what this technology can do. Let's pick 153 00:08:30.040 --> 00:08:34.039 a pretty isolated use case. But the building logic that 154 00:08:34.080 --> 00:08:38.559 we were pulling out about Monolith was basically like do 155 00:08:38.759 --> 00:08:42.080 or die. Right, if it did not work, it costs 156 00:08:42.120 --> 00:08:45.000 millions of dollars to fix, or you know, it ends 157 00:08:45.080 --> 00:08:47.919 up costing company millions of dollars. So it really we 158 00:08:47.919 --> 00:08:49.960 were walking on a tide rope and there was no 159 00:08:50.000 --> 00:08:53.360 net underneath us, and unfortunately we decided to I guess 160 00:08:53.360 --> 00:08:56.799 like walk on our hands instead of go across normally. 161 00:08:57.399 --> 00:09:01.480 So the framework that we used is called and I 162 00:09:01.519 --> 00:09:04.720 think for a team that knows AKA, this probably would 163 00:09:04.759 --> 00:09:07.559 have been really a perfect tool for the job. But 164 00:09:07.840 --> 00:09:09.840 our team and our company really did not have a 165 00:09:09.879 --> 00:09:13.240 ton of experience with Akka, and so unfortunately we weren't 166 00:09:13.279 --> 00:09:15.799 able to sort of take advantage of it and use 167 00:09:15.840 --> 00:09:19.240 it in a way that sort of professional AKA developers 168 00:09:19.960 --> 00:09:20.480 likely can. 169 00:09:20.960 --> 00:09:24.039 So this is kind of like a functional programming thing, right. 170 00:09:24.120 --> 00:09:28.840 It deals with data streams and passing data along in 171 00:09:28.879 --> 00:09:33.519 a functional paradigm, and it's meant to accommodate high volume 172 00:09:33.639 --> 00:09:39.279 data across highly parallelized system. So you know, at the 173 00:09:39.360 --> 00:09:42.840 time we went there, well I'll talk about when we 174 00:09:42.879 --> 00:09:45.799 went there in a second, but in retrospect, it was 175 00:09:46.200 --> 00:09:49.919 it was something that could handle basically ten thousand x 176 00:09:50.360 --> 00:09:52.159 really what we needed in terms of what it was 177 00:09:52.200 --> 00:09:55.960 designed to handle. So just sort of on paper, it 178 00:09:55.960 --> 00:09:59.200 probably wasn't the best the best move in that respect, 179 00:09:59.279 --> 00:10:00.879 but I could also talk about the team as well 180 00:10:00.879 --> 00:10:02.399 and why it wasn't a great a great bit. 181 00:10:02.799 --> 00:10:05.279 Yeah, can you can you talk for just a minute 182 00:10:05.320 --> 00:10:07.519 about like what was the problem that you were trying 183 00:10:07.559 --> 00:10:09.240 to solve, why did you choose? 184 00:10:09.360 --> 00:10:11.720 Like what did you need akup for sure? 185 00:10:11.840 --> 00:10:15.440 Yeah? Perfect. So the problem that we're trying to solve, 186 00:10:15.679 --> 00:10:17.480 and you have to know a little bit about the 187 00:10:17.519 --> 00:10:19.759 Ibota app. So I assume all of you, all of 188 00:10:19.759 --> 00:10:22.600 you guys have downloaded it and are actively using it. 189 00:10:23.000 --> 00:10:25.000 Absolutely it's the best Apple that they used. 190 00:10:25.240 --> 00:10:25.440 Good. 191 00:10:25.919 --> 00:10:31.759 I've definitely looked at it and I was like, nope. 192 00:10:30.840 --> 00:10:33.039 That's funny. We did a cert. Well I'll get to 193 00:10:33.039 --> 00:10:36.559 that later, But basically, I bought it as a way 194 00:10:36.600 --> 00:10:39.960 for you to get digital coupons. You can get brands 195 00:10:40.000 --> 00:10:42.720 put offers in the app. You click on the offer, 196 00:10:43.240 --> 00:10:45.840 you show us evidence that you bought the thing that 197 00:10:45.960 --> 00:10:47.679 is on offer, and then I bought it. Will pay 198 00:10:47.720 --> 00:10:50.399 you cash back, they'll put me send it to your 199 00:10:50.399 --> 00:10:52.600 PayPal account, give you a gift card to Amazon, whatever 200 00:10:52.600 --> 00:10:55.519 you want. So the problem that we're trying to solve 201 00:10:55.639 --> 00:10:57.440 is how do we make sure that the offers in 202 00:10:57.519 --> 00:11:01.399 the app don't exceed the budget that is allocated to 203 00:11:01.480 --> 00:11:03.600 them by the brands that put those offers in the air. 204 00:11:03.759 --> 00:11:05.919 And that sounds maybe like an easy problem, like there's 205 00:11:05.960 --> 00:11:08.840 an easy way to just say, Okay, there's five hundred 206 00:11:08.840 --> 00:11:13.720 thousand dollars in budget for Oreo coupons. Just divide five 207 00:11:13.759 --> 00:11:16.639 hundred thousand dollars by how many how much money we're 208 00:11:16.679 --> 00:11:20.080 giving out per coupon, and then you know, there's actually 209 00:11:20.159 --> 00:11:23.919 much Like it's obviously much harder than that, And in 210 00:11:24.000 --> 00:11:26.679 order to preserve a good user experience, we need to 211 00:11:26.720 --> 00:11:30.679 make sure that we're not yanking content and surprising our users. 212 00:11:30.720 --> 00:11:32.720 Like you would be really upset if you went to 213 00:11:32.720 --> 00:11:35.519 the store specifically to buy Oreos to get the coupon 214 00:11:36.120 --> 00:11:38.840 and then by the time you checked out, the coupon 215 00:11:38.960 --> 00:11:41.200 was no longer in your application. So we have to 216 00:11:41.279 --> 00:11:46.039 run some predictive algorithms to basically guess when we're going 217 00:11:46.120 --> 00:11:48.799 to run out of money and kind of slow the coupon, 218 00:11:48.919 --> 00:11:52.440 slow the velocity down as we approach that point. 219 00:11:52.840 --> 00:11:58.000 Oh dear, that's that's that's that's a dangerous recipe, I 220 00:11:58.080 --> 00:12:00.279 must add to think about. It is a vague in 221 00:12:00.279 --> 00:12:03.200 the UK at the moment. Is it available in Canada. 222 00:12:03.919 --> 00:12:06.840 We are only in the United States right now. Sorry. 223 00:12:07.080 --> 00:12:11.080 Yeah, So in my defense, this looks like the perfect 224 00:12:11.080 --> 00:12:13.200 storm for me because you've both got the kind of 225 00:12:13.240 --> 00:12:15.519 strong financial components. So if you get it wrong, you 226 00:12:15.600 --> 00:12:19.759 lose money. And if you get it long, then people 227 00:12:19.799 --> 00:12:22.639 have wasted their time going to the shop. You know, 228 00:12:23.279 --> 00:12:25.960 in Britain it's so small. I can walk to the shop. 229 00:12:26.360 --> 00:12:28.799 You know, sometimes I just open the window and shout 230 00:12:28.840 --> 00:12:30.799 at them and tell them what I want. But you know, 231 00:12:30.960 --> 00:12:32.679 I know when I was living in the States and 232 00:12:32.720 --> 00:12:35.279 people maybe drive for two hours to go to Walmart. 233 00:12:35.879 --> 00:12:39.519 So this is this is this is quite a quite 234 00:12:39.559 --> 00:12:40.679 a problem you've got there. 235 00:12:41.120 --> 00:12:43.600 Yeah, it was. It's interesting that you say that, because 236 00:12:43.720 --> 00:12:45.919 right when I joined the company, I was kind of 237 00:12:46.200 --> 00:12:48.919 put on the team. I was actually giving you my 238 00:12:49.559 --> 00:12:52.200 tech body, my mentor. When I joined the company, this 239 00:12:52.360 --> 00:12:55.840 was his project, and as someone knew in the company, 240 00:12:55.840 --> 00:12:58.559 it was it was very it was really overwhelming problem 241 00:12:58.639 --> 00:13:04.600 space because basically, these campaigns and application almost have like a 242 00:13:04.600 --> 00:13:08.080 physical momentum to them. So if you imagine trying to 243 00:13:08.120 --> 00:13:11.480 stop a moving train, you can't just hit the brakes 244 00:13:11.480 --> 00:13:13.320 and expect it to stop on a dime. You have 245 00:13:13.399 --> 00:13:17.399 to apply the brakes over some distance to slow the 246 00:13:17.440 --> 00:13:19.519 train down. And that's really how the content in the 247 00:13:19.559 --> 00:13:23.639 application is modeled. And it's I'm not a physics person, 248 00:13:24.039 --> 00:13:25.679 and so it's really confusing. 249 00:13:26.759 --> 00:13:30.279 So all right, so this seems a slightly I mean, 250 00:13:30.320 --> 00:13:34.240 it's a it's a twist right on your classical inventory problem, right, 251 00:13:34.480 --> 00:13:38.120 is what it kind of sounds like. So the way 252 00:13:38.159 --> 00:13:42.720 that AKA came into this is is what you were taking, 253 00:13:42.759 --> 00:13:46.279 like what all these requests from users and trying to 254 00:13:46.360 --> 00:13:48.600 decide whether or not you're going to I guess give 255 00:13:48.600 --> 00:13:50.039 them the coupon or not or something. 256 00:13:50.600 --> 00:13:53.879 Yeah, that's that's that's sort of right. So we have 257 00:13:54.159 --> 00:13:58.120 an event based architecture where our system and for the 258 00:13:58.200 --> 00:14:00.120 for folks who are familiar with that, I mean is 259 00:14:00.120 --> 00:14:04.799 that basically your system publishes events, which is data that 260 00:14:04.960 --> 00:14:08.240 signified that something has happened of interest in the system. 261 00:14:08.320 --> 00:14:12.559 So maybe, like you know, shopping cart loaded is an 262 00:14:12.559 --> 00:14:14.840 event that you might have in like a typical inventory 263 00:14:14.840 --> 00:14:18.039 space or something like that. So we have events that 264 00:14:18.120 --> 00:14:22.039 we're interested in, like content awarded, which means that John 265 00:14:22.240 --> 00:14:24.639 went to the store, submitted or received through the app, 266 00:14:25.120 --> 00:14:28.080 and got cash back. So the content has been awarded, 267 00:14:28.600 --> 00:14:32.679 So we listen for those events in order to keep 268 00:14:32.720 --> 00:14:35.759 track in real time of how much budget is being used, 269 00:14:36.279 --> 00:14:40.039 and we basically track that over time to make a 270 00:14:40.159 --> 00:14:44.120 rough prediction about how fast things are moving. And so AKA, sorry, John, 271 00:14:44.159 --> 00:14:47.399 to get back to your original question, AKA is really 272 00:14:47.440 --> 00:14:53.799 good at streaming data, at streaming large amounts, high volume data. 273 00:14:54.399 --> 00:14:58.080 And I bought a we'll get on the order of 274 00:14:58.159 --> 00:15:01.679 several hundred thousand of these content aword events per day, 275 00:15:02.480 --> 00:15:04.600 which seems like a lot, but it's actually much lower 276 00:15:04.600 --> 00:15:06.919 than I think what AKA can kind of deal with 277 00:15:07.080 --> 00:15:08.720 or is designed to deal with out of the box. 278 00:15:09.200 --> 00:15:10.240 Yeah did you so? 279 00:15:11.080 --> 00:15:15.480 Did you consider alternatives? Did you reject them for various reasons? 280 00:15:16.480 --> 00:15:19.320 And and I guess so. Obviously AKA didn't work out 281 00:15:19.320 --> 00:15:21.840 for you, so you must have picked something else that 282 00:15:22.200 --> 00:15:25.200 you like better. And how did you arrive at that 283 00:15:25.320 --> 00:15:27.720 new thing? And what was that? If that makes some sense? 284 00:15:28.080 --> 00:15:36.080 Yeah? Perfect, So yeah we we Basically this comes down 285 00:15:36.120 --> 00:15:40.360 to some team issues again and not not an issue 286 00:15:40.360 --> 00:15:45.919 with OCCA. So the team issue was basically that at 287 00:15:45.960 --> 00:15:47.799 the start of this project, we scaled up our team, 288 00:15:47.960 --> 00:15:50.320 like this is a lot of work we need, we 289 00:15:50.360 --> 00:15:53.320 need to bring in some artillery, and we brought in 290 00:15:53.360 --> 00:15:56.679 a new developer from outside of the company who is awesome. 291 00:15:57.519 --> 00:16:00.360 She's a rock star and had it was from the 292 00:16:00.399 --> 00:16:06.159 ad product space and with dealing with volume, much volume 293 00:16:06.200 --> 00:16:09.159 of streaming data at a scale much higher than than 294 00:16:09.200 --> 00:16:11.600 what we needed or we're going to be dealing with 295 00:16:11.639 --> 00:16:17.360 in it in any near near future. And you know, 296 00:16:17.480 --> 00:16:19.960 she was coming from I believe in AKA shop and 297 00:16:20.000 --> 00:16:22.919 so she joins the team. We're excited to have her. 298 00:16:22.960 --> 00:16:24.879 We think she's a rock star. I mean, she's a 299 00:16:24.960 --> 00:16:28.320 rock star. And she's like, this is a perfect use 300 00:16:28.360 --> 00:16:31.159 case for ACCA. We're like, okay, never heard of that, 301 00:16:31.200 --> 00:16:33.960 but you know, we trust you. You crushed our interview 302 00:16:34.240 --> 00:16:38.000 and we think you're amazing. So yeah, that's that sounds 303 00:16:38.000 --> 00:16:42.879 pretty good. And again this isn't a knock against Akka. 304 00:16:42.960 --> 00:16:46.080 I think this is just that in our company, we 305 00:16:46.159 --> 00:16:49.480 have a ton of infrastructure setup to support the tools 306 00:16:49.480 --> 00:16:53.279 that our company has sort of blessed that that are 307 00:16:53.360 --> 00:16:55.639 kind of frequently in use. In fact, we have like 308 00:16:55.639 --> 00:16:57.840 a name for that. We call it the paved road, 309 00:16:58.480 --> 00:17:00.679 which is like, if you're an engineer, you have a 310 00:17:00.720 --> 00:17:03.159 lot of autonomy and abada belt what you use but 311 00:17:03.240 --> 00:17:05.079 if you stay on the paved road, it should be 312 00:17:05.119 --> 00:17:08.599 an easy path. And akka, which is a jam you know, 313 00:17:08.640 --> 00:17:13.720 which used with Java or Scala typically is not on. 314 00:17:13.920 --> 00:17:16.799 I bought a paved road. So we had to kind 315 00:17:16.799 --> 00:17:20.880 of have a contentious meeting, a contentious conversation with the 316 00:17:21.079 --> 00:17:24.359 architects at Abada and say, no, we really we believe 317 00:17:24.440 --> 00:17:27.279 in in our developer and we think that she knows 318 00:17:27.319 --> 00:17:30.039 what she's doing, and we're ready to sort of make 319 00:17:30.079 --> 00:17:33.240 our bed and lay in it. And they were like, well, 320 00:17:33.519 --> 00:17:37.039 you know, as long as you know that, so they 321 00:17:37.160 --> 00:17:39.039 let us kind of they gave us just enough rope 322 00:17:39.079 --> 00:17:41.759 to forget how the rest of that goes to hang 323 00:17:41.839 --> 00:17:42.400 ourselves with. 324 00:17:43.000 --> 00:17:46.599 Okay, So so it sounds like you had an advocate 325 00:17:46.640 --> 00:17:49.400 and that's kind of how you landed on it. Where 326 00:17:49.440 --> 00:17:51.599 did you go after you decided that ACA was the 327 00:17:51.599 --> 00:17:53.400 poor choice? Like what did you end up with? 328 00:17:53.920 --> 00:17:56.200 Yeah? So this and this kind of gets into the 329 00:17:56.200 --> 00:18:00.960 next problem, which was the siloed work. So we're starting 330 00:18:01.000 --> 00:18:04.960 to work on building this AKA powered race car of 331 00:18:05.000 --> 00:18:08.039 a micro service to pull biling logic out of our 332 00:18:08.160 --> 00:18:11.519 rails monolith. And there's sort of two streams of work. 333 00:18:11.759 --> 00:18:15.599 There's the development of the ACA micro service, which is 334 00:18:15.960 --> 00:18:18.799 we were writing the reason Cotland, which is the JBM. 335 00:18:19.240 --> 00:18:22.079 It's kind of a nicer job. It's what Android apps 336 00:18:22.079 --> 00:18:24.799 are written in. It's actually pretty nice. So we have 337 00:18:24.920 --> 00:18:27.000 that work going on, and then we have to integrate 338 00:18:28.200 --> 00:18:33.079 that micro service sorry into the Rails monolith. So I 339 00:18:33.160 --> 00:18:36.359 ended up taking on a lot of the integration work 340 00:18:36.920 --> 00:18:40.319 and the other developer took on most of the Akka 341 00:18:40.759 --> 00:18:43.440 and Cotland work. So we have these sort of two 342 00:18:43.519 --> 00:18:47.480 very isolated pieces of work that are siloed. And then 343 00:18:47.920 --> 00:18:51.440 something terrible happened. And I don't blame this person at all, 344 00:18:51.519 --> 00:18:54.480 because I wouldn't want to be on my team anyway. 345 00:18:54.920 --> 00:18:57.799 She decided that she was much more interested in data 346 00:18:57.839 --> 00:19:01.240 engineering and she moved to a different team in the company. 347 00:19:01.759 --> 00:19:04.359 So and that's actually, you know a great thing about 348 00:19:04.400 --> 00:19:07.759 working at Ibada, Like we've got tons of really cool problems, 349 00:19:07.759 --> 00:19:11.319 different spaces, and people are almost encouraged to move around 350 00:19:11.359 --> 00:19:13.039 to find things that they're that they like and are 351 00:19:13.039 --> 00:19:17.400 good at. So she made this move, and now now 352 00:19:17.559 --> 00:19:20.960 the the problem of using a framework that none of 353 00:19:21.039 --> 00:19:24.119 us had experience with and that the general company didn't 354 00:19:24.160 --> 00:19:26.519 have a ton of experience with really became self evident 355 00:19:26.880 --> 00:19:29.039 because now there was there was still work left to 356 00:19:29.079 --> 00:19:33.079 do on the micro service. And I'm just I'm just 357 00:19:33.440 --> 00:19:36.400 a rails developer, Like I had to go into there 358 00:19:36.680 --> 00:19:40.079 and start reading Cotlan, started reading the ACA documentation and 359 00:19:40.160 --> 00:19:42.799 try to wrap my head around what what this whole 360 00:19:42.920 --> 00:19:48.319 you know, actor system meant. And it was. It was tough, 361 00:19:48.519 --> 00:19:53.119 and that's why I call it a big mistake. So sorry, Dave. 362 00:19:53.519 --> 00:19:56.839 You know from my experience too, is that, yeah, Ruby 363 00:19:56.920 --> 00:19:59.480 is slow, you know, there's no getting around that when 364 00:19:59.519 --> 00:20:03.720 you compare to a compiled language. But holy crab, is 365 00:20:03.759 --> 00:20:07.599 it fast too. You know, I have a production application 366 00:20:08.119 --> 00:20:13.680 which processes over five hundred thousand active jobs every single day, 367 00:20:14.279 --> 00:20:17.359 and that it does it extremely quick. I don't need 368 00:20:17.400 --> 00:20:20.279 it to be faster. I mean that's plenty fast on 369 00:20:20.359 --> 00:20:23.839 the current setup that it's on, which is two cores 370 00:20:24.079 --> 00:20:27.440 and four gigs of RAM, and we have two servers 371 00:20:27.720 --> 00:20:31.279 dedicated to the background job, so two vms. It's able 372 00:20:31.279 --> 00:20:33.440 to handle that load and we don't have to worry 373 00:20:33.480 --> 00:20:36.599 about it crashing or anything like that. So I mean 374 00:20:36.640 --> 00:20:40.319 that's good enough for us. Now I imagine that it 375 00:20:40.359 --> 00:20:43.720 would be able to double that workload before we ever 376 00:20:43.839 --> 00:20:46.720 ran into any kind of performance issues where we needed 377 00:20:46.759 --> 00:20:47.920 to start scaling up. 378 00:20:48.440 --> 00:20:52.240 Absolutely, Davi, and we were originally in the monolith. The 379 00:20:52.279 --> 00:20:55.640 way the system worked was it ran on a background 380 00:20:55.640 --> 00:21:00.559 scheduled job using rescue, and the scheduled job based took 381 00:21:00.680 --> 00:21:04.000 roughly forty five minutes to run, So it was running 382 00:21:04.720 --> 00:21:06.559 like kind of like a waterfall, like every ten minutes, 383 00:21:06.599 --> 00:21:08.319 we'd start a new job that would take forty five 384 00:21:08.319 --> 00:21:12.319 minutes to run. And so going from that to basically 385 00:21:12.440 --> 00:21:16.839 completely real time is a huge improvement. And if you know, 386 00:21:16.920 --> 00:21:19.880 real time for five hundred thousand events per day versus 387 00:21:19.880 --> 00:21:23.319 five million events per day are different things, but if 388 00:21:23.359 --> 00:21:26.440 you're at real time, it's already this just enormous improvement 389 00:21:27.000 --> 00:21:28.759 over what we were working with, which is like this 390 00:21:28.839 --> 00:21:32.799 forty five minute loop. So at that point, and I 391 00:21:32.799 --> 00:21:35.559 guess I'll say another thing before I get into like 392 00:21:35.559 --> 00:21:38.599 how we fixed how we fixed my mistake, This is 393 00:21:38.640 --> 00:21:41.759 actually getting to what Dave said, this is a case 394 00:21:41.759 --> 00:21:44.920 of premature optimization. We sort of didn't do the back 395 00:21:44.960 --> 00:21:48.839 of the envelope math well enough or didn't have a 396 00:21:48.839 --> 00:21:51.519 clear picture of like, okay, this is really like designed 397 00:21:51.559 --> 00:21:55.519 to handle like literally a thousand times more traffic than 398 00:21:55.599 --> 00:21:59.640 our best day. So you know, we didn't that was 399 00:21:59.720 --> 00:22:01.559 that was definitely a mistake, like we should have asked 400 00:22:01.599 --> 00:22:04.839 that question and realized, Okay, maybe maybe this is a 401 00:22:04.880 --> 00:22:07.200