WEBVTT 1 00:00:04.599 --> 00:00:07.679 Hey everybody, Welcome back to another episode of the Ruby 2 00:00:07.759 --> 00:00:12.080 Rogues podcast. This week, on our panel we have yush Nuwatya. 3 00:00:13.519 --> 00:00:13.720 Hello. 4 00:00:13.919 --> 00:00:18.239 No, I'm Charles max Wood from top End Devs. And 5 00:00:18.320 --> 00:00:22.640 this week we have a special guest. We have Mohammed Hassan. Mohammad, 6 00:00:23.320 --> 00:00:26.079 you built light stack, which is pretty cool. What else 7 00:00:26.120 --> 00:00:27.000 doul people know about you? 8 00:00:28.480 --> 00:00:32.119 Well, I've been doing Ruby for for a long one now, 9 00:00:32.159 --> 00:00:34.280 I guess, I guess two thousand and five or something. 10 00:00:35.560 --> 00:00:38.079 I've been been in this industry for for a long while. 11 00:00:38.119 --> 00:00:40.439 I go by the handle old Mo for a reason. 12 00:00:40.799 --> 00:00:47.119 I think, yeah, it's it's literally old Mo, and I've 13 00:00:47.119 --> 00:00:53.119 been I've been doing computers since nineteen ninety something, so yeah, 14 00:00:53.399 --> 00:00:55.759 I have have a lot of work that has been 15 00:00:55.759 --> 00:00:59.439 done in the Ruby land, especially in concurrency. I've released 16 00:00:59.439 --> 00:01:02.479 the library long ago if anyone remembers, was called never 17 00:01:02.520 --> 00:01:08.040 block once fibers started, and uh yeah, I got fascinated 18 00:01:08.120 --> 00:01:11.959 by by this idea of like high concurrency and and 19 00:01:12.359 --> 00:01:16.280 doing high performance stuff in Ruby. And then afterwards I 20 00:01:17.200 --> 00:01:21.920 also got into got in love with with embedded databases 21 00:01:22.120 --> 00:01:26.079 Siquo light to be specific. So basically I'm marrying the two. 22 00:01:26.280 --> 00:01:32.280 So like high performance Ruby, high performance embedded database siqual 23 00:01:32.359 --> 00:01:35.799 Light in particular, and I've been I've been doing that 24 00:01:36.120 --> 00:01:41.159 with my own work in my own startups and looking 25 00:01:41.560 --> 00:01:44.959 forward now to actually deliver something to the community that 26 00:01:45.079 --> 00:01:50.879 will you know, spread this, uh this value to everyone 27 00:01:51.079 --> 00:01:52.760 that is using both Ruby and siqual Light. 28 00:01:54.879 --> 00:01:59.719 Cool. So I looked at it and it and you 29 00:01:59.719 --> 00:02:01.280 can reckon me where I'm wrong. I didn't have a 30 00:02:01.359 --> 00:02:04.280 chance to try it because my life is absolutely nuts. 31 00:02:05.120 --> 00:02:06.719 You think the kids get out of school, your life 32 00:02:06.760 --> 00:02:11.960 gets lets the nuts, It doesn't. So yeah, I'm looking 33 00:02:12.000 --> 00:02:13.919 at it, and I'm sitting here thinking well, because I 34 00:02:13.960 --> 00:02:15.840 remember back in the day, sequel Light was kind of 35 00:02:15.879 --> 00:02:19.960 the not serious database. You know, you use it development 36 00:02:20.039 --> 00:02:23.400 and then you'd use a real database like post ris ql. 37 00:02:25.599 --> 00:02:29.800 But with light stack it looks like you provide the database, 38 00:02:29.840 --> 00:02:31.960 which is kind of obvious, but then you're also pulling 39 00:02:32.000 --> 00:02:37.639 in stuff like cashing and jobs, and yeah, do you 40 00:02:37.680 --> 00:02:40.400 want to just talk about all of this because yes, 41 00:02:40.439 --> 00:02:42.879 you pull in just one gem and it does all 42 00:02:42.919 --> 00:02:43.479 this stuff. 43 00:02:44.360 --> 00:02:47.479 Yes, I think. I think one one the biggest issue 44 00:02:47.479 --> 00:02:50.919 here was was the amount of dependencies you need when 45 00:02:51.000 --> 00:02:54.199 you start a new rails app like you need, you 46 00:02:54.280 --> 00:02:58.879 need like a database naturally, yeah, and then you need 47 00:02:58.919 --> 00:03:02.639 some some cash sover and and and then things go 48 00:03:03.120 --> 00:03:06.439 a little bit crazy and you make a cluster of 49 00:03:06.479 --> 00:03:10.280 those cash servers. And then you need a job processor, 50 00:03:10.960 --> 00:03:12.840 and you need a queue for that job processor. So 51 00:03:12.879 --> 00:03:16.039 you have read this for example, and then uh a 52 00:03:16.159 --> 00:03:19.719 process for sidekick or something like that. And then you 53 00:03:19.759 --> 00:03:26.159 need full tech search, so you use something like elastic 54 00:03:26.400 --> 00:03:32.319 or open search. And then you need like uh cable 55 00:03:32.560 --> 00:03:36.680 uh capabilities, so you need a pop sub server and 56 00:03:36.719 --> 00:03:39.479 you have usually read this for that as well, or 57 00:03:39.520 --> 00:03:42.360 postographs and and and you end up with so many 58 00:03:42.400 --> 00:03:45.800 dependencies and so many moving parts in your application. And 59 00:03:45.840 --> 00:03:48.960 I was I was thinking of like, yeah, like making 60 00:03:49.080 --> 00:03:53.840 making Siquo light your your database is nice, but like 61 00:03:53.960 --> 00:03:57.520 limiting it to just database means it's only limited to 62 00:03:57.560 --> 00:04:01.879 smallish applications that only require database. Actually, sequel Light, due 63 00:04:01.879 --> 00:04:05.479 to its latency characteristics, is capable of a lot more. 64 00:04:06.120 --> 00:04:08.800 And I went out to try and see if I 65 00:04:08.879 --> 00:04:11.879 can actually prove that it is capable of a lot more. 66 00:04:12.479 --> 00:04:19.360 And so yeah, database, cash, pops up, que, job processor, 67 00:04:20.759 --> 00:04:25.480 full tex search, everything you'd need for not everything, but 68 00:04:25.560 --> 00:04:29.600 almost everything. You'd need for your application, and currently in 69 00:04:29.720 --> 00:04:32.639 the rapport but not released is like ID which is 70 00:04:33.120 --> 00:04:37.399 greatest basically reported to sequel light, so you won't need 71 00:04:37.920 --> 00:04:42.279 for your key data another separate process running somewhere in 72 00:04:42.319 --> 00:04:43.120 your infrastructure. 73 00:04:45.120 --> 00:04:50.040 Very cool, I have to say, like I switched over 74 00:04:50.120 --> 00:04:58.480 to what is it the caching and queueing that? Yeah, 75 00:04:58.600 --> 00:05:02.199 solid cash and solid cute that came out with. Sorry, folks, 76 00:05:02.240 --> 00:05:04.439 I'm a little slow. I am under the weather big time, 77 00:05:05.040 --> 00:05:07.759 but this looks so cool. I just wanted to talk 78 00:05:07.759 --> 00:05:12.240 through it, and so I was super excited to get 79 00:05:12.319 --> 00:05:16.040 rid of rettis right, because I haven't been using action 80 00:05:16.160 --> 00:05:21.519 Cable a ton, so yeah, I kind of like the 81 00:05:21.560 --> 00:05:24.040 idea of And I've also been using Melee search and 82 00:05:24.079 --> 00:05:26.480 I've been trying to move that into my Postgress database 83 00:05:26.480 --> 00:05:31.480 as well, just use Postgress full tech search. So so 84 00:05:31.600 --> 00:05:33.800 this appeals to me a lot, because I, you know, 85 00:05:33.879 --> 00:05:35.399 my stack gets a lot simpler. 86 00:05:36.319 --> 00:05:39.920 Yes, definitely. I think I think there are two dimensions here. 87 00:05:40.079 --> 00:05:42.560 One of them is simplifying the stack, and one of 88 00:05:42.600 --> 00:05:46.120 them is actually realizing the use potential in having an 89 00:05:46.160 --> 00:05:49.399 embitted database. So one way to simplify the stack is 90 00:05:49.439 --> 00:05:52.240 to move everything to posgress posts is very capable, it's 91 00:05:52.319 --> 00:05:56.759 very solid, and it's been there since forever and you 92 00:05:56.839 --> 00:06:00.279 can rely on it. But what I'm my thing is 93 00:06:00.680 --> 00:06:05.519 there is a new dimension of performance when you go embedded, 94 00:06:05.879 --> 00:06:10.240 like a complete a completely different thing. I think in Brighton, 95 00:06:10.279 --> 00:06:12.879 ruge I presented what it takes for a query to 96 00:06:12.959 --> 00:06:18.920 run if you go through like client server stack versus 97 00:06:19.000 --> 00:06:22.279 an embedded stack, and what do that is? Basically you're 98 00:06:22.319 --> 00:06:24.800 you're doing for most treats, you're doing a memory call 99 00:06:25.759 --> 00:06:32.360 and and and that's that's a huge advancement in performance 100 00:06:32.439 --> 00:06:35.160 that you'll only appreciate it if you see it. Once 101 00:06:35.240 --> 00:06:37.839 you see that your Rail's application is able to able 102 00:06:37.879 --> 00:06:43.759 to deal with like potentially thousands of concurrent connections and 103 00:06:43.759 --> 00:06:48.279 then responding to them very quickly, you'll you'll, you'll you'll 104 00:06:48.279 --> 00:06:49.120 have to see it to believe it. 105 00:06:49.160 --> 00:06:52.319 Basically very cool. Are usually you played with this? 106 00:06:53.079 --> 00:06:55.560 I haven't actually had a chance to play with it. 107 00:06:56.079 --> 00:06:59.720 Sequel Light and Rails has been something that's been on 108 00:06:59.800 --> 00:07:03.040 my do list for a while, but well, life gets 109 00:07:03.040 --> 00:07:07.439 in the way. I think my next project is almost 110 00:07:07.439 --> 00:07:10.279 definitely going to be my next side project is definitely 111 00:07:10.279 --> 00:07:15.399 going to be a sequel light rather than postgress. But yes, 112 00:07:15.439 --> 00:07:21.000 I just I just want to get very clear in 113 00:07:21.040 --> 00:07:24.519 my head about on the gotcha's with sequel light. And 114 00:07:25.639 --> 00:07:29.759 let's start with just what light stock actually does in 115 00:07:29.879 --> 00:07:34.759 certain things like with light job and sorry, with light 116 00:07:34.839 --> 00:07:39.600 cable and like like Katie he said, like like like 117 00:07:39.800 --> 00:07:43.519 cable used like pub sub it's like a drop in replacement, 118 00:07:43.600 --> 00:07:46.920 right and like Katie replicates credits that are these building 119 00:07:46.959 --> 00:07:51.519 on like native sequel light features off like pub sub 120 00:07:51.519 --> 00:07:52.839 in a key value store or. 121 00:07:54.720 --> 00:07:58.360 Is a little blend then than that, so it doesn't 122 00:07:58.399 --> 00:08:04.120 have like something like notify in in postgress. And I 123 00:08:04.240 --> 00:08:06.959 like I had to build this functionality into Syqul light 124 00:08:07.800 --> 00:08:10.079 on top of Syqul itself, but on top of native 125 00:08:10.079 --> 00:08:12.839 sequel light. So by the way, the solid family of 126 00:08:13.439 --> 00:08:16.040 products like Solid cash, slid C they have they do 127 00:08:16.120 --> 00:08:19.279 have sequel light support, but the problem is they do 128 00:08:19.399 --> 00:08:24.800 deliver that on through active record, which is like an 129 00:08:24.879 --> 00:08:31.519 order of man nude more abstracted then going native with 130 00:08:31.639 --> 00:08:35.919 sycuolight like directly and hence the siqal light performance. If 131 00:08:35.960 --> 00:08:38.000 if you go directly to the metal, it's it's a 132 00:08:38.080 --> 00:08:42.200 much much faster and the nice thing about about what 133 00:08:42.279 --> 00:08:44.960 we have in in light stack is that we have 134 00:08:45.200 --> 00:08:50.159 the seql code completely the coupled from the from the 135 00:08:50.240 --> 00:08:55.480 Ruby logic, so you can optimize that independently and work 136 00:08:55.519 --> 00:08:59.639 on ensuring gas every call is through an index. You're 137 00:08:59.679 --> 00:09:05.080 not working, uh, you're not walking tables or doing any 138 00:09:05.080 --> 00:09:10.279 table scans things of that sort. So it's uh, I guess, 139 00:09:10.320 --> 00:09:12.799 I guess it's. It's it's implemented, but the functionality is 140 00:09:12.840 --> 00:09:17.559 not that hard. I guess it's it's really straightforward getting 141 00:09:17.600 --> 00:09:21.240 a cable and getting it to perform the problem that 142 00:09:21.320 --> 00:09:24.039 your core issue would be to get it to perform 143 00:09:24.120 --> 00:09:28.600 correctly under like concurrent operation, which is which is tricky 144 00:09:28.759 --> 00:09:31.559 and because you don't know whether the application would run 145 00:09:31.600 --> 00:09:35.879 in in threads, run fibers, multiple processes or not. And 146 00:09:35.919 --> 00:09:41.039 this is I think the core contribution of light stack, 147 00:09:41.080 --> 00:09:44.279 which it abstracts all this and and make sure any 148 00:09:44.279 --> 00:09:47.200 component can be brought up very quickly, like I built 149 00:09:47.240 --> 00:09:50.919 like cable in one hour. Oh wow, okay, because because 150 00:09:51.039 --> 00:09:57.279 all the machinery there for for like making sure that 151 00:09:59.480 --> 00:10:05.919 like uh, pulling of connections, threads don't go into these 152 00:10:06.000 --> 00:10:09.080 conditions and and proceeds are not affecting each other that 153 00:10:09.279 --> 00:10:13.519 was there already, so you only get to implement the 154 00:10:13.559 --> 00:10:17.240 logic of the higher level of course one hour with bugs. 155 00:10:17.279 --> 00:10:20.799 But but then I had a working for the type 156 00:10:21.279 --> 00:10:25.720 and and and unlikely you was not much harder. It 157 00:10:25.799 --> 00:10:29.480 was done in two days, so and then it was 158 00:10:29.519 --> 00:10:31.919 mostly the functionality, a lot of functionality, a lot of 159 00:10:32.000 --> 00:10:36.320 logic on top of the connection. So I guess, I 160 00:10:36.320 --> 00:10:39.679 guess that's that's what light stack is able to do, 161 00:10:39.799 --> 00:10:44.000 bring up those features quickly, and I implement as much 162 00:10:44.240 --> 00:10:46.399 of them as possible in sequel land rather than in 163 00:10:46.519 --> 00:10:49.840 rugby land, for for performance reasons from one side, and 164 00:10:49.840 --> 00:10:54.679 and also for portability and separation of concerns as well. 165 00:10:55.600 --> 00:10:58.240 Yeah, I think that that's a great approach. I've kind 166 00:10:58.279 --> 00:11:02.919 of been uh, blissfully unaware of sequel up to a 167 00:11:03.000 --> 00:11:06.360 point because Actor record kind of handles it for me. 168 00:11:06.440 --> 00:11:08.399 But for the client I'm working with at the moment, 169 00:11:09.279 --> 00:11:11.200 we've had to do some really advanced stuff and I 170 00:11:11.240 --> 00:11:14.080 have been getting my hands dirty with seql and the 171 00:11:14.120 --> 00:11:17.519 power that it has has blown me away a little bit. 172 00:11:18.000 --> 00:11:22.960 So yeah, I completely get your approach there. So yeah, 173 00:11:23.000 --> 00:11:25.519 just for personal curiosity, could you talk about how you 174 00:11:26.720 --> 00:11:29.799 how you did pops up with just like SEQL and 175 00:11:29.919 --> 00:11:34.120 database features without having something like notify at your disposal. 176 00:11:34.519 --> 00:11:37.799 Yes. So the idea is when when you do a 177 00:11:37.879 --> 00:11:43.399 database calls it in the traditional sense it goes through 178 00:11:43.679 --> 00:11:46.600 the network to the server and back, but in in 179 00:11:46.600 --> 00:11:50.720 in the sequel like sense, it's just a mess of coal, 180 00:11:51.039 --> 00:11:53.840 a mess a call to a local memory address, especially 181 00:11:53.840 --> 00:11:57.600 if you if you're looking at some page that is 182 00:11:57.639 --> 00:12:02.039 definitely in the cash because it is like access, it's 183 00:12:02.120 --> 00:12:06.159 hot in the hot section of your data. So basically 184 00:12:06.240 --> 00:12:09.440 it's a pulling it. It's pulling happening from from Ruby. 185 00:12:10.200 --> 00:12:16.759 Like each thread that is that is subscribed to the 186 00:12:16.799 --> 00:12:21.679 pops up would constantly pull the database and ask it 187 00:12:21.759 --> 00:12:26.600 for for data. And and in benchmarking like four proceeds 188 00:12:26.840 --> 00:12:30.600 constantly writing and constantly pulling. I'm able to get around 189 00:12:30.759 --> 00:12:34.120 like one hundred and twenty thousand messages retired per second. 190 00:12:34.919 --> 00:12:39.120 So yeah, this is this and that's not on very 191 00:12:39.159 --> 00:12:42.720 high end hardware, so it is capable. And now I 192 00:12:42.759 --> 00:12:45.840 was even able to do faster than some some IPC 193 00:12:47.480 --> 00:12:51.960 cross process solutions that are tried at first as an optimization, 194 00:12:52.000 --> 00:12:54.759 but then it ended up like siquolites faster. 195 00:12:54.799 --> 00:12:59.039 Still, Yeah, I think when you eliminate the network from 196 00:12:59.080 --> 00:13:02.720 the equation of the performance speed up you get is 197 00:13:02.879 --> 00:13:06.480 just hard to comprehend because you're any AUTHO removing so 198 00:13:06.600 --> 00:13:09.759 much complexity from the call. Right, You're not opening a 199 00:13:09.799 --> 00:13:12.720 network connection, you're not sending anything over a wire. 200 00:13:13.759 --> 00:13:18.120 You're not data from from buffers to like the network 201 00:13:18.159 --> 00:13:21.120 stack in the OS. There are so many, so many 202 00:13:21.159 --> 00:13:24.759 things that are eliminated, and your your life becomes a 203 00:13:24.799 --> 00:13:25.399 lot easier. 204 00:13:26.159 --> 00:13:29.360 I'm in some cases, I think I kind of feel this, 205 00:13:29.600 --> 00:13:33.039 you know, because I've used some of the cloud databases, right, 206 00:13:33.240 --> 00:13:37.240 so you know, even if I'm pulling a you know, 207 00:13:37.399 --> 00:13:40.320 a database on Linode and a server on Linode or 208 00:13:40.360 --> 00:13:45.080 Digital Ocean or whatever, you know, pick a poison like those. 209 00:13:45.200 --> 00:13:49.159 I feel like I felt that more than when I 210 00:13:49.320 --> 00:13:52.480 have you know, to virtual servers that I've set up 211 00:13:52.480 --> 00:13:56.519 in the same network, you know, on Digital Otion and stuff, 212 00:13:56.519 --> 00:14:01.279 because all of that stuff is it's I think, it's 213 00:14:01.279 --> 00:14:04.600 all software and it's fast, right, and. 214 00:14:04.960 --> 00:14:06.799 It is fast, but it's not free. 215 00:14:07.720 --> 00:14:09.960 No, that's fair, and it will never be free. You 216 00:14:10.200 --> 00:14:12.840 don't get that for free. But yeah, I think I 217 00:14:12.840 --> 00:14:15.120 think there are different levels to this, right, I don't 218 00:14:15.120 --> 00:14:19.080 know that, but completely convinced that you know, it's gonna 219 00:14:19.120 --> 00:14:23.279 be that noticeable depending on what your setup is. Yeah, 220 00:14:23.399 --> 00:14:25.240 you should be under the circumstances. 221 00:14:25.480 --> 00:14:28.600 Yeah, yeah, definitely you should. You should try actually to 222 00:14:28.679 --> 00:14:32.720 go through the Like I'm all the benchmarks that they 223 00:14:32.720 --> 00:14:35.159 have for light stack. By the way, on on the 224 00:14:35.200 --> 00:14:39.480 GitHub ripple, these are run against postglass on the same machine. 225 00:14:40.240 --> 00:14:42.159 Okay, so you put mostgress on the same machine and 226 00:14:42.200 --> 00:14:44.559 run not even on a versus I think. 227 00:14:44.639 --> 00:14:48.440 Okay, Like all these are run through against postgress on 228 00:14:48.480 --> 00:14:52.279 the same very same machine, on the same space, same everything. 229 00:14:52.440 --> 00:14:54.120 Well that changes my argument a little bit. 230 00:14:54.840 --> 00:15:00.639 Yeah, so I like the point is how how much 231 00:15:00.759 --> 00:15:03.759 of the overhead? How much is the overhead compared to 232 00:15:04.120 --> 00:15:08.440 the actual work that you're doing, and it there is 233 00:15:08.480 --> 00:15:10.799 a spectrum here, not not all queries would be much 234 00:15:10.840 --> 00:15:13.360 faster and sequal lighte Like, if you're doing heavy io, 235 00:15:13.480 --> 00:15:15.919 if you hit the desk, then they will both be 236 00:15:16.360 --> 00:15:20.559 both be a lot slower. Then in that case, the 237 00:15:21.759 --> 00:15:24.759 savings that you had by not going through the network 238 00:15:24.799 --> 00:15:28.759 stack will be very un noticeable. They will not exist. 239 00:15:28.799 --> 00:15:32.240 Basically because reading a single block from disk is much 240 00:15:32.320 --> 00:15:36.399 much more expensive, like orders of magnitude more expensive. So like, 241 00:15:36.720 --> 00:15:39.519 if you're talking about quis that are fulfilled from the 242 00:15:39.559 --> 00:15:45.159 page cash, then the difference would be really not If 243 00:15:45.159 --> 00:15:51.399 you have complex queries that will require disk access, then 244 00:15:51.440 --> 00:15:55.600 the difference will be less. But in no situation will 245 00:15:55.639 --> 00:15:59.080 siquo light bill will be slower than post aggress in 246 00:15:59.159 --> 00:16:03.559 reed scenarios, given, of course, the quity plan that was 247 00:16:03.600 --> 00:16:08.240 provided by both databases. Both have very advanced quity planners, 248 00:16:08.399 --> 00:16:12.120 and they should produce for most in the most case, 249 00:16:12.519 --> 00:16:16.360 similar plans for the quities. But in case they're producing 250 00:16:16.360 --> 00:16:20.200 the similar plan, definitely Sequolite will be either as fast 251 00:16:20.279 --> 00:16:23.799 or much faster than like in a spectrum between fast 252 00:16:23.840 --> 00:16:28.919 and as fast and much faster than prosrogress. That's three equities, right, 253 00:16:29.039 --> 00:16:30.840 It is a completely different story. 254 00:16:31.960 --> 00:16:34.519 Right. So my question then is, because yeah, I'm looking 255 00:16:34.519 --> 00:16:38.360 at these benchmarks and it's pretty impressive, what if you 256 00:16:38.440 --> 00:16:40.240 have to reach it over the network? Right, So, what 257 00:16:40.279 --> 00:16:43.879 if you've got enough traffic to where you've spread your 258 00:16:43.919 --> 00:16:47.000 application servers out over multiple servers, and then I'll have 259 00:16:47.000 --> 00:16:49.360 to hit the same source of truth, right, the same 260 00:16:49.440 --> 00:16:52.279 database that way, How does that change the equation. 261 00:16:52.440 --> 00:16:55.720 So siquolite is not designed for this scenario like scaling 262 00:16:55.720 --> 00:17:00.000 out and connecting from multiple application servers to the same 263 00:17:00.159 --> 00:17:03.840 node database node is not what sequal light is designed for. 264 00:17:04.000 --> 00:17:07.680 Sickle Light is designed generally for virtical scaling rather than 265 00:17:07.720 --> 00:17:12.920 horizontal scaling, so you get a better, bigger machine or 266 00:17:12.960 --> 00:17:16.359 a VM basically, and in that case you can serve 267 00:17:16.839 --> 00:17:21.799 more from the same resources. But there's many attempts to 268 00:17:23.200 --> 00:17:29.200 actually enable enable this level of scaling. But but then 269 00:17:29.920 --> 00:17:35.400 the point would be a little bit moved, like why 270 00:17:35.440 --> 00:17:39.480 not use postal glass. I'm working myself on a solution 271 00:17:39.680 --> 00:17:44.799 that that is midway, so it only transmits rights, but 272 00:17:44.839 --> 00:17:47.720 all reads are local, so you have multiple nodes, each 273 00:17:47.759 --> 00:17:51.160 one has a complete copy of the database and rights 274 00:17:51.160 --> 00:17:55.680 are distributed. I've written about that on my blog and 275 00:17:55.920 --> 00:17:58.559 still didn't open source it yet. But I'm not surely 276 00:17:58.559 --> 00:18:00.200 if I'm going to open source it, or maybe you 277 00:18:00.240 --> 00:18:03.519 just try to build the service around it. But but 278 00:18:03.599 --> 00:18:05.640 you get a copy of the database of full copy 279 00:18:05.640 --> 00:18:08.599 of database locally and you can do all reads locally, 280 00:18:08.799 --> 00:18:11.039 so you get those benefits for reds and you get 281 00:18:11.279 --> 00:18:15.039 slight extralatency for rights. 282 00:18:16.200 --> 00:18:19.319 Yeah, but yeah, I mean I have to admit most 283 00:18:19.359 --> 00:18:22.480 of the time when I'm scaling, Yeah, I just go 284 00:18:22.519 --> 00:18:25.640 into lin ode or Digital Lotion or wherever I'm hosting, 285 00:18:25.680 --> 00:18:28.799 and I say I want more RAM and a little 286 00:18:28.799 --> 00:18:31.160 more disk space, and it goes okay, and then my 287 00:18:31.200 --> 00:18:34.319 website's down for a minute. So what you're talking about 288 00:18:34.440 --> 00:18:35.200 that makes sense to me. 289 00:18:36.079 --> 00:18:42.359 Yeah, yeah, because it simplifies yourself, like because once you 290 00:18:42.440 --> 00:18:45.160 go out of that, you you have to consider, okay, 291 00:18:45.440 --> 00:18:50.160 what about like a little balancing, what if it note fails, 292 00:18:50.359 --> 00:18:52.480 what happens here, what happened there? Then then you have 293 00:18:52.640 --> 00:18:55.880 so many moving parts and they keep just growing exponentially 294 00:18:56.000 --> 00:18:57.960 as as you add complexity to resist them. 295 00:18:58.160 --> 00:18:58.480 Right. 296 00:19:00.319 --> 00:19:03.960 I think in twenty twenty four, the amount of traffic 297 00:19:04.000 --> 00:19:08.400 you can serve with one single beefy sell. Yeah, it's 298 00:19:08.480 --> 00:19:12.720 just bloody unreal. I think horizontal scaling, If you need 299 00:19:12.799 --> 00:19:16.559 horizontal scaling, you will have enough money to pay people 300 00:19:16.559 --> 00:19:20.319 to get you off sequel light and on the postgress exactly. 301 00:19:20.440 --> 00:19:24.000 That's I think. I think that the point is, or 302 00:19:24.119 --> 00:19:29.599 or the point that I'm trying to to spread, is 303 00:19:30.759 --> 00:19:35.680 it is unwise to scale more than what you need 304 00:19:35.880 --> 00:19:40.200 initially in your small or starting projects. You can get 305 00:19:40.240 --> 00:19:43.759 a lot more a lot more value in like eliminating 306 00:19:44.279 --> 00:19:49.319 overheads and increasing performance and eliminating like moving parts by 307 00:19:49.680 --> 00:19:53.039 moving to an embedded solution. And sycolitis is one heck 308 00:19:53.079 --> 00:19:57.440 of an embedded solution. It's an amazing solution. It just works. 309 00:19:57.720 --> 00:20:03.359 And Yeah, my advice would be, don't focus on your infrastructure, 310 00:20:03.440 --> 00:20:08.680 don't have DevOps, focus on your application, and it will 311 00:20:08.720 --> 00:20:11.640 carry you a long way. And once you're in a 312 00:20:11.680 --> 00:20:14.039 position where you need to, you really need to move 313 00:20:14.079 --> 00:20:17.920 out of this exactly what you said, Like, you have 314 00:20:17.960 --> 00:20:20.359 the money and the funds to move out of this. 315 00:20:20.599 --> 00:20:23.920 Either either you're getting a lot of money or you 316 00:20:24.000 --> 00:20:29.039 have enough external funding to get out of this given 317 00:20:29.079 --> 00:20:31.720 that you have that amount of user activity. 318 00:20:32.640 --> 00:20:35.480 Yeah, DHH actually said I'm trying to find the tweet, 319 00:20:36.200 --> 00:20:38.920 but he said in a tweet that if you bought 320 00:20:38.960 --> 00:20:42.039 like I think it was thirty two or sixty four 321 00:20:42.079 --> 00:20:49.279 gigabyte Hetzner level thing, that you at this point could 322 00:20:49.279 --> 00:20:54.039 have run the entirety of base camp a few years ago. Right, 323 00:20:54.440 --> 00:21:00.359 And so yeah, yeah, I'm really kind of yeah, I'm 324 00:21:00.400 --> 00:21:02.559 liking the way that this looks. 325 00:21:03.039 --> 00:21:06.319 And then look at at things now like this camp 326 00:21:06.359 --> 00:21:09.119 now is a beast. It's a huge application and so 327 00:21:09.240 --> 00:21:13.480 many subscriptions. But and they had to go through a 328 00:21:13.519 --> 00:21:17.559 lot of pain to scale that. But if you're starting 329 00:21:17.839 --> 00:21:21.119 something like that, now you've got like at least five 330 00:21:21.160 --> 00:21:23.799 to six years of growth within a single box. 331 00:21:25.400 --> 00:21:29.799 Yeah, I think, I said when they first started back 332 00:21:29.839 --> 00:21:32.559 in like the mid two thousands or whatever, for a 333 00:21:32.759 --> 00:21:36.720 number of years they were running on one physical server. Yeah, 334 00:21:36.799 --> 00:21:39.559 not even virtualization back then, because this is two thousand 335 00:21:39.599 --> 00:21:43.759 and four. So yeah, I'm fully on board with this. 336 00:21:44.079 --> 00:21:47.240 Just rent the VM approach. 337 00:21:48.519 --> 00:21:50.400 Yeah, he said, you're right here. He said, you can 338 00:21:50.440 --> 00:21:52.880 rent a dedicated forty eight cores two hundred and fifty 339 00:21:52.880 --> 00:21:56.880 six gig two terror by AMD epic server from Heatzner 340 00:21:57.400 --> 00:21:59.759 at two hundred and thirty six dollars a month or 341 00:22:00.079 --> 00:22:04.759 undred and thirty six euros a month. Yeah. I mean, 342 00:22:04.799 --> 00:22:07.480 I've I've never had an application that got so much 343 00:22:07.519 --> 00:22:09.480 traffic that that wasn't enough. 344 00:22:10.839 --> 00:22:12.920 Yeah, and those four the eight courses like translate to 345 00:22:13.119 --> 00:22:17.880 ninety six course in cloud speak. So yeah, that's that's 346 00:22:17.920 --> 00:22:21.759 a lot more than many many applications and and a 347 00:22:21.759 --> 00:22:25.400 lot cheaper, h Yeah, than than what what deployments we 348 00:22:25.400 --> 00:22:26.079 see on the cloud. 349 00:22:27.839 --> 00:22:30.119 Yeah, it's crazy how much money you can save when 350 00:22:30.119 --> 00:22:36.440 you cut out complexity. Well, let's let's dig into a 351 00:22:36.480 --> 00:22:39.759 little bit just about sequel Light and its use and 352 00:22:39.960 --> 00:22:43.240 in a web application. So putting light stack and all 353 00:22:43.240 --> 00:22:46.119 the amazing stuff that does, let's put that to one side. 354 00:22:46.160 --> 00:22:50.240 And if I wanted to use sequel light directly with rails, 355 00:22:50.640 --> 00:22:53.440 I know there are some gotchas backups being one main 356 00:22:53.480 --> 00:22:55.680 thing is how do I back up and not lose 357 00:22:55.720 --> 00:22:59.160 my file? And the second thing is I've been looking 358 00:22:59.200 --> 00:23:03.160 through the campfire a code base which is obviously sequel 359 00:23:03.200 --> 00:23:07.680 light ones product from Petty Sound Signals, And the only 360 00:23:07.839 --> 00:23:11.200 thing I could find in there that was like specifically 361 00:23:11.920 --> 00:23:14.440 to do with sequel light in terms of like initialization 362 00:23:14.519 --> 00:23:16.880 and set up with something called a busy handler. So 363 00:23:16.960 --> 00:23:19.400 could you talk what that is and why that's necessary. 364 00:23:20.000 --> 00:23:26.759 Okay, So, originally sequal Light is configured for like really 365 00:23:26.799 --> 00:23:30.079 really embedded use cases, So it is for usage on 366 00:23:30.119 --> 00:23:33.079 a phone and probably between the three of us, like 367 00:23:33.160 --> 00:23:36.960 we have like maybe at least one hundred sequel Light 368 00:23:37.000 --> 00:23:40.680 instances running at this very moment on our phones and computers. 369 00:23:41.079 --> 00:23:45.400 But but this use cases is completely different from the 370 00:23:45.440 --> 00:23:51.519 web application that requires like multiple users accessing it at 371 00:23:51.519 --> 00:23:53.880 the same time as much as possibly as fast as possible. 372 00:23:54.400 --> 00:23:58.279 So there are a few caveats and a few knobs 373 00:23:58.319 --> 00:24:01.759 that he that should be set. What light stack does 374 00:24:01.799 --> 00:24:06.880 it It completely does that by default when it's publishing 375 00:24:06.920 --> 00:24:12.359 its own like Siicula driver to active record and what 376 00:24:13.559 --> 00:24:18.000 there are other gems like fact un minded with Stephen Mark, 377 00:24:18.599 --> 00:24:22.279 he goes, yeah, he has a lot of effort done 378 00:24:22.319 --> 00:24:25.640 in that in that area, and and I think I 379 00:24:25.680 --> 00:24:28.599 think he is helping a lot like the Vanilla's equal 380 00:24:28.640 --> 00:24:32.680 light experience for rails. By the way, one one thing 381 00:24:32.680 --> 00:24:35.079 about let's stack light Steck is not just trailed. It's 382 00:24:35.319 --> 00:24:38.000 basically you can use light stack with Anami and use 383 00:24:38.039 --> 00:24:43.039 it with with Sinatra whatever bring more cute like. But 384 00:24:43.119 --> 00:24:47.400 let us get back to the configuration. So basically what 385 00:24:47.559 --> 00:24:49.799 you need is the following. You need to ensure that 386 00:24:50.559 --> 00:24:53.039 you can do rights while you can do reads. The 387 00:24:53.079 --> 00:24:58.119 default configuration can only either do rights or reads. So 388 00:24:58.240 --> 00:25:00.920 you change the journal in mode from the default which 389 00:25:00.960 --> 00:25:04.160 is delete, to well which is right ahead log. So 390 00:25:04.480 --> 00:25:07.920 in that case you're you can have a writer and 391 00:25:08.160 --> 00:25:11.000 either at the same time, actually an unlimited number of 392 00:25:11.000 --> 00:25:13.559 readers and a writer at the same time. That's the 393 00:25:13.599 --> 00:25:18.480 first configuration. The second thing is which is a little 394 00:25:18.559 --> 00:25:23.119 bit uh tricky, like when when when when connections are 395 00:25:23.119 --> 00:25:26.480 trying to capture the lock on the database, they will 396 00:25:27.000 --> 00:25:29.559 either succeed in getting the lock or fail because another 397 00:25:29.640 --> 00:25:33.680 process capture the lock. And if it fails, then it 398 00:25:33.759 --> 00:25:36.599 will return to you, like ah, siqual Light busy error, 399 00:25:37.480 --> 00:25:39.400 and hence we need to have a busy handler, and 400 00:25:39.440 --> 00:25:41.880 that busy handler what will will do It will capture 401 00:25:41.920 --> 00:25:45.400