WEBVTT 1 00:00:04.599 --> 00:00:07.759 Ah, Hi everybody, welcome to another episode of the Ruby Rogues Podcast. 2 00:00:08.080 --> 00:00:12.279 I am your host today, Valentino Soul, and I'm joined by a very 3 00:00:12.320 --> 00:00:18.199 special guest today. Hey, I am you, Guinny. Yes, yeah, 4 00:00:19.079 --> 00:00:27.079 sorry, Yeah, welcome. I appreciate you taking the time to come 5 00:00:27.120 --> 00:00:32.759 on today. I'm gonna just introduce yourself and tell everybody why you've gotten so 6 00:00:32.840 --> 00:00:37.759 famous here on the show today. I'm not sure about famous, but I'm 7 00:00:37.880 --> 00:00:42.439 very happy to be here. And yes, I'm your Guinny. Longtime Ruby 8 00:00:42.479 --> 00:00:47.600 Rogues podcast listener, thank you for reminding me. I'm a software developer who 9 00:00:47.679 --> 00:00:55.479 worked for many years using Ruby as my primary programming language and created some open 10 00:00:55.520 --> 00:01:02.320 source libraries and gems like buch loader for exam. Recently, I started my 11 00:01:02.439 --> 00:01:10.280 own company called Demi, and this company helps other companies track data changes in 12 00:01:10.319 --> 00:01:15.599 their databases. And I recently wrote a blog post about different audit trail approaches 13 00:01:15.439 --> 00:01:19.439 that can be used in Ruby. And yeah, today I'll be happy to 14 00:01:19.480 --> 00:01:26.079 discuss it and share my knowledge. Awesome, thank you so much. And 15 00:01:26.280 --> 00:01:30.560 yeah, I appreciate bashloader. I definitely use that every day. So very 16 00:01:30.560 --> 00:01:38.760 cool. Yeah, So, I mean audologging such a popular topic to be 17 00:01:38.799 --> 00:01:42.760 honest, like one of those like kind of things that you over time will 18 00:01:42.799 --> 00:01:48.159 just end up using, right, it's something that you want to just also 19 00:01:48.239 --> 00:01:52.239 drop in and kind of hope that it works right, like and just like 20 00:01:52.480 --> 00:01:55.680 don't have to think about it again. I think for the most part, 21 00:01:57.079 --> 00:02:05.359 what kind of you started down this path of like you know, really really 22 00:02:05.400 --> 00:02:09.360 diving into this. Yeah, it's funny that. Yeah, you mentioned that 23 00:02:09.599 --> 00:02:15.400 everyone is familiar with this problem or maybe at some point heard about gems like 24 00:02:15.439 --> 00:02:19.759 paper trail and things like that. But I noticed that it's mostly in the 25 00:02:19.840 --> 00:02:24.919 Rugby community. When you step aside and for example, go into like JavaScript 26 00:02:24.960 --> 00:02:30.319 world or Golang world, like people have not heard about tools like that, 27 00:02:30.400 --> 00:02:36.439 they would use completely different approaches. And because Ruby has rails and active record 28 00:02:36.800 --> 00:02:42.199 as a like main or library, there are some lots of unified tools. 29 00:02:42.759 --> 00:02:46.840 And how I first encountered like, first of all, I used all these 30 00:02:46.879 --> 00:02:50.520 gems maybe for many years. For example, paper trail is one of the 31 00:02:50.520 --> 00:02:53.319 most famous ones. What it does is basically, at a high level, 32 00:02:53.400 --> 00:02:59.960 just connects to like your model. In your model, you have some callback 33 00:03:00.199 --> 00:03:07.080 and these callbacks record changes in a separate table. And recently I worked at 34 00:03:07.080 --> 00:03:12.879 a company called Angelist, angelust Venture and angelist us Ruby, and one of 35 00:03:12.919 --> 00:03:16.599 the main problems that I tried to solve there was related to tracking. And 36 00:03:16.639 --> 00:03:21.639 there are many different reasons why people would want to track their data changes. 37 00:03:21.960 --> 00:03:25.000 At this company, specifically, it was related to compliance and regulation, so 38 00:03:25.159 --> 00:03:30.400 the company operated in the financial space. It was important. There are some 39 00:03:30.639 --> 00:03:38.280 certain compliance requirements for example, soque socks and many different weird acronyms similar to 40 00:03:38.319 --> 00:03:42.400 SoC two which is mostly about security, but those are related to like financially 41 00:03:42.400 --> 00:03:46.840 reporting and things like that. And it was super critical to be able to 42 00:03:46.960 --> 00:03:51.879 record what happens with data in the company, as well as be able to 43 00:03:52.639 --> 00:03:58.240 audit it and trace back like why this change was made. But I'm noticing 44 00:03:58.280 --> 00:04:02.879 that the full disclaim I'm working at a company called Demi that we try to 45 00:04:02.919 --> 00:04:09.759 build a productized solution something like that. And there are many different other reasons, 46 00:04:10.000 --> 00:04:14.800 like we're noticing why people care about audit trail in general. The most 47 00:04:14.960 --> 00:04:20.199 maybe straightforward one is rollbacks, like maybe you made some changes by accident or 48 00:04:20.240 --> 00:04:26.199 maybe by design, and you want to allow people to revert the changes they 49 00:04:26.240 --> 00:04:33.079 made recently. That's one of the use cases troubleshooting is another one. So 50 00:04:33.399 --> 00:04:38.720 there are lots of companies that want to understand what happened. Maybe some companies 51 00:04:38.759 --> 00:04:43.959 implement on calls for their engineers and they do something manually, go in and 52 00:04:44.000 --> 00:04:46.720 do something for their customers, and that needs to be tracked as well. 53 00:04:47.199 --> 00:04:53.199 Regulation as I mentioned, in certain like financial sectors, banking and things like 54 00:04:53.240 --> 00:04:58.480 that, it's kind of like required. Some companies build activity feed as a 55 00:04:58.519 --> 00:05:01.879 security feature or table stay features. For example, if you use like notion, 56 00:05:02.199 --> 00:05:06.120 maybe enterprise plant would have like a FTIVT feed for oddmans to see what 57 00:05:06.240 --> 00:05:13.800 happens within the organization. And people can also use these data for fraud detection 58 00:05:14.000 --> 00:05:23.160 analytics and things like that. Yeah, you make a couple of great kind 59 00:05:23.160 --> 00:05:28.839 of crucial points here, primarily like there's activity right like that people like the 60 00:05:28.920 --> 00:05:32.000 trap, which I feel like often is like separated. And it's funny that 61 00:05:32.040 --> 00:05:38.399 you mentioned that this is like mostly like rails and Ruby specific, and I 62 00:05:38.439 --> 00:05:44.079 think maybe the other realms like go toward like the log capture approach. We're 63 00:05:44.120 --> 00:05:47.480 just like blogs are getting blasted and you try and like resolve all of the 64 00:05:47.560 --> 00:05:53.800 the time stamps as as they come, which I think is crazy Yeah, 65 00:05:54.079 --> 00:05:57.399 there are many different views on how to use, for example, data and 66 00:05:57.600 --> 00:06:01.519 interact with the database. In their Ruby community, like convention over configuration, 67 00:06:01.680 --> 00:06:08.079 we have active record and many other ms data mappers for example. If you 68 00:06:08.240 --> 00:06:12.600 take a look at in the Go community, it comes from simplicity of the 69 00:06:12.680 --> 00:06:17.759 language. That is why many people try to stick with like a low level 70 00:06:17.800 --> 00:06:24.240 adapters and write sco queries themselves, and that is why it's even like hard 71 00:06:24.279 --> 00:06:29.800 to build a unified solution like that. In the Jazz community, for example, 72 00:06:29.879 --> 00:06:33.160 there are lots of many ORMs that I don't know, work on edge 73 00:06:33.560 --> 00:06:39.319 and like work in serverless environments and things like that, So there are lots 74 00:06:39.319 --> 00:06:43.399 of lots of crazy technologies, which is also hard to build a unified solution. 75 00:06:44.199 --> 00:06:46.959 But yeah, communities like Ruby, Python, I'm noticing that. Yeah, 76 00:06:46.959 --> 00:06:51.199 they're more mature and they have already lots of great tools around that. 77 00:06:53.839 --> 00:06:58.319 Yeah, that's that's really interesting. I mean definitely we could take some learnings 78 00:06:58.360 --> 00:07:03.800 there. But I would like to just think about, like, you know, 79 00:07:03.839 --> 00:07:08.439 why people even like you mentioned a lot of like you know, mostly 80 00:07:08.639 --> 00:07:14.920 it seems like reporting and like trying to you know, align with you know, 81 00:07:15.040 --> 00:07:19.319 requirements, regulatory requirements, or things like that, even internal you know 82 00:07:19.399 --> 00:07:25.639 regulation where you want to be able to confirm that things are happening, why 83 00:07:25.680 --> 00:07:30.240 they changed, right, And I feel circling back to this like notion of 84 00:07:30.319 --> 00:07:34.759 like activity is happening on your app and you want to like be able to 85 00:07:34.800 --> 00:07:41.319 isolate how the activity affected your data. Is that kind of like the approach 86 00:07:41.360 --> 00:07:46.000 that you're taking with themi of trying to visualize how those two things match up 87 00:07:46.319 --> 00:07:51.959 or you know, where do you focus on? Yeah, we take a 88 00:07:53.000 --> 00:07:56.319 slightly different approach. So there are many different approaches. So for example, 89 00:07:56.360 --> 00:08:00.240 I wrote this blog post article that describe some of the approaches. At a 90 00:08:00.279 --> 00:08:05.120 high level, there are I call them callback approaches, so paper trail Audited, 91 00:08:05.560 --> 00:08:11.160 Mangoid History and gems like that. What they do they basically add callbacks 92 00:08:11.319 --> 00:08:16.279 and when you use ORM they would write history of these changes in the separate 93 00:08:16.279 --> 00:08:22.079 table. There are some tigger based approaches, so in a database you can 94 00:08:22.160 --> 00:08:26.759 write code, which is dangerous, could be dangerous, especially if you're trying 95 00:08:26.759 --> 00:08:31.199 to implement complex business logic, but for simple triggers it works. So you 96 00:08:31.240 --> 00:08:35.039 can try to implement triggers when, for example, a record is being updated, 97 00:08:35.120 --> 00:08:39.200 inserted, or deleted, please do something. And there is one gem 98 00:08:39.279 --> 00:08:43.240 called like ITZA. It's very hard to pronounce when you see it how it's 99 00:08:43.240 --> 00:08:50.159 spelled, but this gem implements this approach with triggers, and I think that 100 00:08:50.279 --> 00:08:54.519 for majority of the companies, especially small ones, these approaches are great. 101 00:08:54.840 --> 00:08:58.840 There is like low overhead. You just add them, maybe at them even 102 00:09:00.399 --> 00:09:05.279 just in case maybe you would do something dangerous and delete something by accident, 103 00:09:05.600 --> 00:09:11.600 or your logic was not tested well and it performed something crazy and you would 104 00:09:11.600 --> 00:09:15.639 want to roll back. That's kind of like good insurance. I would say. 105 00:09:16.399 --> 00:09:20.679 At DEMI we take slightly different approach. I can go more into details, 106 00:09:20.679 --> 00:09:26.559 but at a high level we use replication locks as a solution. There 107 00:09:26.600 --> 00:09:31.840 is a pattern called change data capture. There weird like birding, but it's 108 00:09:31.879 --> 00:09:37.399 a data pattern used by data engineers very often. What it does is basically, 109 00:09:39.399 --> 00:09:43.080 yeah, capturing data changes from the database and moving these data somewhere. 110 00:09:43.720 --> 00:09:50.200 The most popular use case is probably ETL or solutions like that, when you 111 00:09:50.240 --> 00:09:54.159 would use some kind of like solution that connects to your database for example my 112 00:09:54.279 --> 00:09:58.960 squol, Postbose or whatever, and it tracks all changes and moves these changes 113 00:10:00.080 --> 00:10:05.039 to warehouse or something like that. So we use this similar approach and change 114 00:10:05.120 --> 00:10:09.080 data capture CDC in short can be implemented in many different ways. There's like 115 00:10:09.159 --> 00:10:13.879 query based when you basically query by timestamp for example, give me all records 116 00:10:13.879 --> 00:10:20.600 that were updated from the last like hour when we checked. It could be 117 00:10:20.600 --> 00:10:26.120 implemented by using triggers again and beme, we use log based solution. So 118 00:10:26.639 --> 00:10:31.000 posgress for example, has vowve ride ahead log. My sequel has been log 119 00:10:31.360 --> 00:10:37.000 mangodib for example, it has uplog like a skew light has vowalve as well 120 00:10:37.279 --> 00:10:43.360 ride ahead log. So all these locks they again briefly describe like why they 121 00:10:43.399 --> 00:10:48.559 exist in the world. So these right ahead locks, they're designed to be 122 00:10:48.639 --> 00:10:58.679 able to quickly record changes in the database in append only log before writing all 123 00:10:58.720 --> 00:11:03.679 the chain just to the disc with data indexes and things like that. That 124 00:11:03.759 --> 00:11:09.200 are heavier operations and these locks they are crucial for data recovery. For example, 125 00:11:09.200 --> 00:11:13.679 if your instant crashes, it would try to recover from the last state, 126 00:11:13.919 --> 00:11:18.360 replace certain let's say changes from the wall and get to the later state. 127 00:11:20.360 --> 00:11:24.960 These replication logs can also be used for like cluster replication for example, 128 00:11:24.000 --> 00:11:30.919 if you have like primary and secondary, Secondary would connect to the ride ahead 129 00:11:30.960 --> 00:11:35.960 lock or replication log in a primary and try to read it and get and 130 00:11:35.000 --> 00:11:39.519 seek the state. We use basically the same solution at Theme. We are 131 00:11:39.600 --> 00:11:43.519 sort of connecting like a read replica, but we also decoded. It's called 132 00:11:43.639 --> 00:11:50.000 logical decoding. We decode all these changes so it's you can understand like it's 133 00:11:50.000 --> 00:11:56.919 not on a binary like file blog format. It comes from like decoded on 134 00:11:56.000 --> 00:12:01.759 a roll level, and you resee with all the changes and we simply stream 135 00:12:01.799 --> 00:12:07.159 and store them as a kind of like yeah in the separate table and people 136 00:12:07.159 --> 00:12:13.360 can use it and read it and people sometimes yeah, as I described, 137 00:12:13.399 --> 00:12:16.000 there are many different approaches like or use cases why you would use it, 138 00:12:16.399 --> 00:12:20.320 but at a high level, what we do we simply write a log of 139 00:12:20.399 --> 00:12:24.399 all changes that people want to track. They can specify like what table to 140 00:12:24.440 --> 00:12:28.000 subscribe to, and there are certain technologies on the postulous level that were recently 141 00:12:28.080 --> 00:12:33.720 released that may help with performance as well. Plus we have some nice UIs 142 00:12:33.720 --> 00:12:37.000 that people can use to see like what happened to these records and things like 143 00:12:37.000 --> 00:12:45.480 that. This is really interesting. I love the article that you have that 144 00:12:45.559 --> 00:12:50.039 kind of breaks down all of this on. I've never heard of change data 145 00:12:50.039 --> 00:12:54.360 capture before. It's kind of wild. There's a lot of pieces to the 146 00:12:54.919 --> 00:12:58.759 process here, which I think just comes honestly with a lot of the Avenger 147 00:13:00.039 --> 00:13:03.799 and kind of architectures. So I just want to take a step back, 148 00:13:03.919 --> 00:13:11.360 Like, let's let's walk through kind of like you know, why would somebody 149 00:13:11.000 --> 00:13:16.799 you know, aside from like, let's say, like what what are the 150 00:13:16.840 --> 00:13:22.399 primary use cases I guess for having your for having a need for that level 151 00:13:22.639 --> 00:13:30.279 of like you know, orchestration and like granular rollbacks. Right, Like what 152 00:13:30.320 --> 00:13:33.399 you're talking about is having like I guess, a massive amount of data that 153 00:13:33.480 --> 00:13:37.840 you want to granularize what you're auditing and be able to recover from those in 154 00:13:37.879 --> 00:13:41.399 case anything should happen. Is that your primary focus, like focusing on that 155 00:13:41.519 --> 00:13:45.879 recovery when you're trying to do the audit trail or like where do you see 156 00:13:46.000 --> 00:13:50.000 a lot of the you know, the most common use cases of having the 157 00:13:50.000 --> 00:13:56.159 audit trail and where like just having paper trail like starts to fail. Yeah, 158 00:13:56.279 --> 00:14:01.519 great question. Uh, we are very like we are noticing that it's 159 00:14:01.519 --> 00:14:05.639 a very horizontal solution and people use it for different purposes. That is why 160 00:14:05.879 --> 00:14:09.840 it's great when you give a solution for developers and they build something on top 161 00:14:09.879 --> 00:14:16.000 of it. We have maybe three main use cases. One use case is 162 00:14:16.080 --> 00:14:20.840 just for troubleshooting purposes. We have, for example, there is one logistics 163 00:14:20.879 --> 00:14:30.039 company that has thirty or forty customer support reps and they're in the logistics business, 164 00:14:30.200 --> 00:14:35.000 so there's lots of real time data from analog world, lots of unstructured 165 00:14:35.080 --> 00:14:39.279 data, and they leverage AI and things like that to help with it. 166 00:14:39.600 --> 00:14:46.080 But they need to see what's happening, maybe help customers, their customers and 167 00:14:46.360 --> 00:14:52.720 cargo companies and things like that, help them understand what happened to this shipment, 168 00:14:52.759 --> 00:14:56.200 for example, and they have lots of different inputs. It could be 169 00:14:56.279 --> 00:15:01.120 like constructured just an email. It could be a dog sign or something that 170 00:15:01.320 --> 00:15:05.279 changed the status of the shipment and they would need to troubleshoot and understand what's 171 00:15:05.320 --> 00:15:13.240 happening. So yeah, I previously like as developers, one solution is just 172 00:15:13.279 --> 00:15:16.080 going to a rails console or something like that. And if you use any 173 00:15:16.080 --> 00:15:20.799 of these solutions, you would just try to see like Querier record and get 174 00:15:20.960 --> 00:15:24.679 all recent changes that happen to you. When you need to understand like why 175 00:15:24.759 --> 00:15:31.000 is it in this weird state, why it's not in like different state, 176 00:15:31.080 --> 00:15:35.399 what happened there, like what API in point for example changed it, and 177 00:15:35.440 --> 00:15:41.000 things like that, some additional metadata. The second use case is related to 178 00:15:41.720 --> 00:15:46.879 just yeah rollbacks. For example, we have there is like one company related 179 00:15:46.919 --> 00:15:52.120 to carbon removal tech for example, and they use POSTS, which is like 180 00:15:52.200 --> 00:15:58.039 an extension for geolocation data on top of prosgress. So they do lots of 181 00:15:58.120 --> 00:16:04.360 changes in budgets and in certain cases when there is an error or something, 182 00:16:04.440 --> 00:16:08.360 they want to roll back to the previous state. And it's possible to just 183 00:16:08.440 --> 00:16:15.559 track certain selected tables and work with any extensions and things like that. And 184 00:16:15.879 --> 00:16:19.679 the third one is for many it's like table stakes, whether it's because of 185 00:16:21.039 --> 00:16:25.720 regulation or for example, there are in healthcare it's very popular. There are 186 00:16:25.759 --> 00:16:30.240 like clinical trial management systems for example, they need to track what happened to 187 00:16:30.360 --> 00:16:37.039 patients data. And many of these solutions. Like one fun story like at 188 00:16:37.039 --> 00:16:41.000 one company, we use paper trail and we had so many locks accumulated in 189 00:16:41.039 --> 00:16:48.879 the versions table that we ran out of integers and we had to Like the 190 00:16:48.919 --> 00:16:56.480 solution was simple. We basically okay, said start recording all changes in versions 191 00:16:56.759 --> 00:17:00.759 V two table, so we would keep the old one it will like cold 192 00:17:00.840 --> 00:17:04.240 archival, but all new changes please record in a separate table because we don't 193 00:17:04.240 --> 00:17:07.680 have any integers there and player IDs, So that is why it's good to 194 00:17:07.759 --> 00:17:15.359 use UIDs when possible. But one issue with for example pay Petrail and solutions 195 00:17:15.440 --> 00:17:22.079 like that is reliability. Maybe like people, maybe you heard that it's possible 196 00:17:22.119 --> 00:17:27.240 to even with rails, to execute certain queries that bypass the callbacks. For 197 00:17:27.319 --> 00:17:32.759 example, if you execute like delete all or update all or update column or 198 00:17:32.759 --> 00:17:38.319 something, it won't trigger the callback, does it won't record the history, 199 00:17:38.440 --> 00:17:45.119 so you will not have the complete history. So that is where like log 200 00:17:45.160 --> 00:17:51.000 based solutions are critical and could be very useful because even if someone goes directly 201 00:17:51.039 --> 00:17:55.880 to a database executes an SCO query, it will be recorded because it happens 202 00:17:55.880 --> 00:18:00.559 on their database level. And that's one of the kind of like main reasons. 203 00:18:00.599 --> 00:18:04.119 In terms of plus, there are some additional benefits like no performance penalty, 204 00:18:04.279 --> 00:18:07.640 so you're in the run time, you don't insert any additional records. 205 00:18:08.000 --> 00:18:15.319 It happens asynchroneously, but that's at a high level. Yeah, you made 206 00:18:15.319 --> 00:18:18.559 me think about because a lot of a lot of what we've been talking about 207 00:18:18.559 --> 00:18:23.920 has been surrounding like just auditing very specific processes around the data. Right, 208 00:18:23.920 --> 00:18:30.279 Like we get a you know, going back to the logistics approach, right, 209 00:18:30.359 --> 00:18:34.359 like somebody's submitted an order and they're tracking the process of that and the 210 00:18:34.480 --> 00:18:40.000 order you know, I imagine has a table of orders and various aspects of 211 00:18:40.039 --> 00:18:42.720 the order that are being tracked with it, and that gets updated in the 212 00:18:42.799 --> 00:18:45.720 database kind of the same way, and it gets tracked, you know, 213 00:18:45.799 --> 00:18:52.519 as it progresses through whatever business logic that we have. What I feel like 214 00:18:52.759 --> 00:18:59.319 is kind of missing from what we're talking about here is like that that overarching 215 00:18:59.519 --> 00:19:03.720 activity like almost it makes me think of open telemetry, right where we have 216 00:19:03.799 --> 00:19:10.720 like distributed systems right where they don't have to be necessarily you know, technologically 217 00:19:10.880 --> 00:19:15.160 right, but they are distributed events that happen throughout the business that fire off 218 00:19:15.200 --> 00:19:21.240 and some activity happens and that has effect on the database. What do we 219 00:19:21.279 --> 00:19:25.720 do for like cases like that, Like are there like tools available to audit 220 00:19:25.799 --> 00:19:30.400 that process and how that's affecting the individual audit trails that are happening, and 221 00:19:30.480 --> 00:19:36.240 like where how do you approach that aspect of it. Yeah, you're definitely 222 00:19:36.319 --> 00:19:40.839 right. There are some like tools for example data Dog and things like that 223 00:19:40.839 --> 00:19:47.319 that help with some visibility, observability, and tracing. Many of these tools 224 00:19:47.480 --> 00:19:52.880 they even implement a like just logging, so they would log what happens and 225 00:19:52.960 --> 00:19:56.759 for example, if you use like postcris, there is an extension popular one 226 00:19:56.799 --> 00:20:00.680 called pg audit. What it does it basically all queries that were executed. 227 00:20:02.119 --> 00:20:06.440 So if for security reasons, someone for example, gained access, you'll be 228 00:20:06.440 --> 00:20:11.279 able to, like during like cyber forensics operations or something, we will be 229 00:20:11.319 --> 00:20:15.839 able to detect what happened and who executed what. So there are some solutions 230 00:20:15.839 --> 00:20:22.079 that do just log in off activity that happens, and you can do it 231 00:20:22.160 --> 00:20:26.359 manually as well. There are some gems like public Activity or ahoy gems that 232 00:20:27.200 --> 00:20:32.839 maybe used like Google Analytics on the front end or something like that, or 233 00:20:32.880 --> 00:20:36.240 in the browser it's similar but on the back end, when you can manually 234 00:20:36.759 --> 00:20:41.519 trigger certain events that you want to track and they will be stored in structured 235 00:20:41.599 --> 00:20:45.039 format, not in locks, but in a database for example. There are 236 00:20:45.119 --> 00:20:53.039 also solutions that help with just performance, observability, open tracing and things like 237 00:20:53.079 --> 00:20:59.880 that. But there are not a lot of solutions that do something on the 238 00:21:00.160 --> 00:21:07.599 data level, and it's a little bit weird and funny because yeah, most 239 00:21:07.640 --> 00:21:11.799 of the products they use databases and that's the source of all data, and 240 00:21:11.880 --> 00:21:17.160 yet we don't track it like developers use geed for tracking and version in their 241 00:21:17.200 --> 00:21:23.240 code, but there is nothing for like databases. And to be fair, 242 00:21:23.279 --> 00:21:29.880 there are some solutions that people sometimes use. For example, there are maybe 243 00:21:30.400 --> 00:21:36.160 two technologies that may help you solve this problem. One is temporal tables and 244 00:21:36.240 --> 00:21:44.039 another one is event sourcing pattern that some companies and people use, so temporal 245 00:21:44.079 --> 00:21:51.079 tables. I'll try to summarize it in the simple words. Basically, it's 246 00:21:51.319 --> 00:21:57.640 within the SKAL standard since two thy eleven. What this feature does basically what 247 00:21:57.720 --> 00:22:03.240 I described it kind of like has a version of all records, So you 248 00:22:03.279 --> 00:22:07.519 have your record, plus you have a range and a timestamp basically a range 249 00:22:07.559 --> 00:22:14.400 of time stamps during which period this record was there. So for example, 250 00:22:14.440 --> 00:22:18.839 if you're updating a record two times, there will be two records in a 251 00:22:18.160 --> 00:22:26.440 temporal table, and the first record will have like an end time range when 252 00:22:26.839 --> 00:22:32.920 the second update happened, and second update will have like infinite time range starting 253 00:22:32.960 --> 00:22:37.119 with the time when this update happened. So with the functionality, and that's 254 00:22:37.200 --> 00:22:44.000 very powerful. It's usually helpful when you do for example, reporting I don't 255 00:22:44.000 --> 00:22:47.920 know, you need to generate some files or what happened last year, tax 256 00:22:48.039 --> 00:22:52.440 or something like that. That's very like powerful feature. Although many databases including 257 00:22:52.480 --> 00:22:59.880 postgars unfortunately don't have it implemented, so you would need to install some extension 258 00:23:00.279 --> 00:23:03.880 or build it yourself. So that's one option. And another one for just 259 00:23:04.200 --> 00:23:10.960 all dating changes is rethinking how you make data changes is event sourcing. Event 260 00:23:11.039 --> 00:23:17.359 sourcing in simple words, it's kind of like maybe the best analogy is geit. 261 00:23:18.079 --> 00:23:22.240 So when we use GIT, we basically use an event source system. 262 00:23:22.799 --> 00:23:29.519 So GIT records all events, commits and all the changes that happened and then 263 00:23:30.519 --> 00:23:33.839 and that's the source of truth. So committs become the main kind of like 264 00:23:36.759 --> 00:23:41.039 unit of data. And from these commitits, the powerful feature is that you 265 00:23:41.079 --> 00:23:45.240 can travel in time. So for example, if you want to roll back 266 00:23:45.319 --> 00:23:48.759 or see what was the state of like three commits back, you can do 267 00:23:48.839 --> 00:23:56.680 that by traveling and basically undoing the changes of each commit. I would say 268 00:23:56.039 --> 00:24:00.880 from my perspective, like in software engineering, event sourcing is like a unicorn 269 00:24:00.920 --> 00:24:08.519 idea. Everyone is talking how cool that is, but very rarely people implemented 270 00:24:08.960 --> 00:24:15.680 or do it successfully because it's a very challenging problem because you deal with mostly 271 00:24:15.720 --> 00:24:21.920 with immutability of these events. So every change basically needs to be mutable. 272 00:24:22.000 --> 00:24:25.519 But what if you need to change the structure, then you need to implement 273 00:24:25.559 --> 00:24:29.960 some kind of like versioning or replay the events and append some additional data or 274 00:24:30.400 --> 00:24:36.480 like change the things. Plus then you need to somehow get to the actual 275 00:24:36.519 --> 00:24:40.359 state. So you'd record all events. For example, shopping card, you'd 276 00:24:40.359 --> 00:24:44.920 record an event that somebody added an item to a shopping cart, somebody clicked 277 00:24:44.960 --> 00:24:48.759 on purchase, somebody purchased and it was paid or something like that. All 278 00:24:48.759 --> 00:24:52.960 these events would be recorded, but at the end of the day you need 279 00:24:53.000 --> 00:24:57.519 to understand what was purchased, for example, when showing the whole order history. 280 00:24:59.000 --> 00:25:03.799 To do that, people, you use projections sqres like for like separating 281 00:25:04.319 --> 00:25:11.039 reason rights, and lots of lots of other patterns from the book called the 282 00:25:11.039 --> 00:25:15.440 Main Driven Design DDITY, and that becomes very complex. I would say that 283 00:25:15.599 --> 00:25:23.680 event sourcing makes sense in certain regulated industries, and there are lots of great 284 00:25:25.279 --> 00:25:29.519 patterns or like how people do it for example double entry book keeping system in 285 00:25:29.680 --> 00:25:33.960 banks or like yeah, accounting and things like that. This is standard where 286 00:25:34.000 --> 00:25:37.559 you would probably try to use this approach from the get go. Yeah, 287 00:25:37.599 --> 00:25:45.480 you make a good point. I mean all of my even rooven design architecture 288 00:25:45.519 --> 00:25:48.440 related stuff as always, you know, the complexity raises as you start to 289 00:25:48.480 --> 00:25:52.359 spread things out and you try and like capture what you know, collect all 290 00:25:52.400 --> 00:25:56.640 of the things and merge you know, the state of things. You know, 291 00:25:56.640 --> 00:26:00.000 which pieces do you pick? Right? Like anybody that's had to merge 292 00:26:00.039 --> 00:26:04.000 conflict and get you know, can understand the issue here, right, like, 293 00:26:04.400 --> 00:26:08.359 well, which which changes do you use? Choose right? And that 294 00:26:08.440 --> 00:26:12.920 definitely becomes more of a problem as you start to spread things out with you 295 00:26:12.920 --> 00:26:17.880 know, your event driven architecture. Yeah, exactly. Yeah, it's much 296 00:26:17.920 --> 00:26:22.119 simpler to use, for example, like crowd operations for example, just update 297 00:26:22.160 --> 00:26:29.400 the record and that's transactional, right and do it or not. But it 298 00:26:29.440 --> 00:26:36.559 does make me kind of think like okay, well you know, as you 299 00:26:36.640 --> 00:26:41.839 know, it makes me wonder, like I guess if you have a need 300 00:26:41.920 --> 00:26:48.519 for right, like the things happening at different times, but really they're all 301 00:26:48.640 --> 00:26:53.319 the result of like one thing happening and having a simple audit trail, like 302 00:26:55.200 --> 00:26:56.960 it makes sense to just throw a gem in and then just have it handle 303 00:26:56.960 --> 00:27:04.000 all of the things. But when it's like when it's something like you know, 304 00:27:07.160 --> 00:27:10.880 a time series database or something like, are we just solving the wrong 305 00:27:10.960 --> 00:27:17.759 problem right? Like why not just have something like in a time series database 306 00:27:17.799 --> 00:27:22.920 that keeps track of the data as it happens versus spreading it out like what's 307 00:27:22.920 --> 00:27:26.720 the where do you see the benefit of having it be distributed like that over 308 00:27:27.160 --> 00:27:32.680 having it be more transactional and having things like Okay, well it happened in 309 00:27:32.720 --> 00:27:36.319 this series, Like you know, is it worth it what I'm saying, 310 00:27:36.680 --> 00:27:41.640 you know, to have that distributed nature versus just like taking the time and 311 00:27:41.720 --> 00:27:45.799 doing things in series. Yeah, that's a good question, and I think 312 00:27:45.000 --> 00:27:53.960 the best answer is favorite developers answer is it all depends. But I think 313 00:27:56.400 --> 00:28:00.720 if you need strong guarantees and you have scale, then it totally makes sense 314 00:28:00.759 --> 00:28:04.319 to use this system. The question is then whether you buy or do it 315 00:28:04.319 --> 00:28:08.200 yourself. If you do it yourself, then you would need to start thinking 316 00:28:08.240 --> 00:28:15.039 about all these like technologies school technologies like Kafka and maybe storing it in like 317 00:28:15.440 --> 00:28:22.440 yeah, time series databases and columnar databases and things like that or warehouses and 318 00:28:22.839 --> 00:28:26.319 use it somehow. If you need to use it from the app, then 319 00:28:26.359 --> 00:28:30.440 you need to somehow connect to I don't know snowflake to read it if you 320 00:28:30.480 --> 00:28:38.279 need it for like transactional or like operations. Basically I think like back to 321 00:28:38.319 --> 00:28:48.920 this like anecdote with large number of paper trail records, like anecdotically, like 322 00:28:48.000 --> 00:28:55.160 from talking to other companies, I saw that for many of them, just 323 00:28:55.839 --> 00:29:00.000 managing the database becomes much more expensive when you record it everything in the same 324 00:29:00.079 --> 00:29:06.759 database. You would notice that, like this version stable becomes your the most 325 00:29:06.799 --> 00:29:12.039 biggest table in your entire database, and that affects everything. It affects like 326 00:29:12.680 --> 00:29:18.680 backups, the storage, how fast your database would run, how difficult it 327 00:29:18.799 --> 00:29:23.440 is to upgrade it at scale, how fast it will start, how much 328 00:29:23.440 --> 00:29:30.920 traffic is there plus activity and things like that, and at some like that's 329 00:29:30.960 --> 00:29:37.759 probably like totally valid approach at a certain scale, but at some point you 330 00:29:37.759 --> 00:29:41.960 would probably notice, like why am I storing it in my database? That 331 00:29:41.079 --> 00:29:48.039 is being killed by this performance penalty because you're essentially doubling the throughput on every 332 00:29:48.200 --> 00:29:52.480 For example, insert you would insert an additional record, or it could be 333 00:29:52.519 --> 00:29:57.920 even worse. If you're inserting one hundred records in one query, it will 334 00:29:57.960 --> 00:30:03.119 be one to one one hundred plus extra one hundred queries. For each record 335 00:30:03.119 --> 00:30:10.359 that was inserted, it will be one hundred additional skill statements. And at 336 00:30:10.359 --> 00:30:15.799 some point it makes sense to somehow extract these data and it could be still 337 00:30:15.839 --> 00:30:18.759 in I don't know the same type of database that you use, and there 338 00:30:18.799 --> 00:30:23.000 are some benefits to it. For example, if your main transactional database is 339 00:30:23.000 --> 00:30:27.519 postgress, for example, what we do we store these changes and we call 340 00:30:27.559 --> 00:30:32.880 it destination database where the changes are stored from the source database. We store 341 00:30:32.880 --> 00:30:37.640 it still in postgress. The reason why is because one is easier to consume. 342 00:30:37.839 --> 00:30:40.839 You just use the same adapter, same or m For example, in 343 00:30:40.960 --> 00:30:45.079 rails, you can specify a couple of connections for example, for these models 344 00:30:45.160 --> 00:30:48.079 go to these database, but for this model go to the other database. 345 00:30:48.480 --> 00:30:52.240 So you're on the application layer. If you want to consume those, we 346 00:30:52.279 --> 00:30:57.920 don't need to change anything. And two, especially in postgress, there are 347 00:30:57.960 --> 00:31:03.279 lots of cool tech technologies that help you with scalability. One is maybe yeah 348 00:31:03.319 --> 00:31:08.119 time series. We mentioned, there's like time scale extension or super extension because 349 00:31:08.200 --> 00:31:15.000 it dramatically changes the way positives works and it allows you to scale, compress 350 00:31:15.119 --> 00:31:21.400 data, and record events in chronological order that it is designed for that with 351 00:31:21.599 --> 00:31:26.480 all these charding, partitioning and things like that. Yeah, that's really interesting. 352 00:31:26.000 --> 00:31:30.359 You know, that's definitely something I've seen, you know, scaling as 353 00:31:30.680 --> 00:31:33.359 things go. You you you know, you want to set it and forget 354 00:31:33.400 --> 00:31:36.960 it, but then all of a sudden you're like, well, why is 355 00:31:37.000 --> 00:31:41.799 the database crashing when when we want to like you know, audit the audit 356 00:31:41.839 --> 00:31:48.119 trailing. Yeah, in affects not just the database, even like your run 357 00:31:48.160 --> 00:31:52.519 time. So your application is slower as a result because it waits for database, 358 00:31:52.680 --> 00:31:56.839 but also tests. For example, if you don't disable for example versioning 359 00:31:56.000 --> 00:32:01.039 for your models in tests, Uh, imagine like eating a large like database 360 00:32:01.119 --> 00:32:06.160 or lots of things and testing things, it will just yeah, kill your 361 00:32:06.160 --> 00:32:12.160 test performance as well. That's a great point. So I mean, what 362 00:32:12.160 --> 00:32:16.680 what can we do to like avoid these pitfalls? Right like aside from like, 363 00:32:16.680 --> 00:32:21.160 like you said, you like using some of these other extensions, Like, 364 00:32:21.400 --> 00:32:24.200 are there like best practices just like out of the gate that you can 365 00:32:24.240 --> 00:32:30.400 do to like maybe avoid some of uh, you know the performance issues upfront 366 00:32:30.519 --> 00:32:35.279 without too much complexity. Yeah, I think there is no like silver bullet. 367 00:32:35.359 --> 00:32:39.480 Unfortunately, at smaller scale. For example, if you're a startup or 368 00:32:39.480 --> 00:32:45.119 something and you don't have lots of people or like customers or users, then 369 00:32:45.119 --> 00:32:49.119 it's probably not a problem and you can just buy yourself time in terms of 370 00:32:49.240 --> 00:32:57.319 velocity by building features and not thinking about optimizing things. But yeah, as 371 00:32:57.319 --> 00:33:02.400 you scale, unfortunately, companies hit this kind of like moment when they realize 372 00:33:02.440 --> 00:33:07.640 they need something and they already build it. Usually like cod basic becomes giant, 373 00:33:08.440 --> 00:33:13.519 there are already lots of like team members and engineers on your team, 374 00:33:14.039 --> 00:33:17.880 and now you have to migrate everything, and that becomes very difficult to do 375 00:33:19.000 --> 00:33:24.400 if you want to like change things dramatically and these migrations. Yeah, I 376 00:33:24.440 --> 00:33:29.480 talk to some like people who build similar systems at I don't know, Airbnb, 377 00:33:29.720 --> 00:33:36.000 Gusto and other companies. It's all like very difficult when you need reliability, 378 00:33:36.079 --> 00:33:40.880 performance and things like that. Yeah, I would say, yeah, 379 00:33:42.440 --> 00:33:45.880 I think it depends on your business. If you're starting and you think you 380 00:33:45.920 --> 00:33:50.400 would need all these kind of like tracking functionality and it's business critical. It 381 00:33:50.440 --> 00:33:53.880 makes you a business better. You can help, it can help you win 382 00:33:54.000 --> 00:33:59.000 customers and things like that. Then I would think from day one, like 383 00:33:59.240 --> 00:34:04.680 how to potentially designed this system either use like event sourcing, or use these 384 00:34:05.200 --> 00:34:09.360 CDC change data culture or like some products that out there something like that. 385 00:34:09.960 --> 00:34:15.400 But if it's just like secondary, then you can either buy at later stages 386 00:34:15.480 --> 00:34:22.480 if you're successful some solutions or just it's not like the main problem and it's 387 00:34:22.119 --> 00:34:28.639 probably will be deprioritized for many many years. You know. Part of me 388 00:34:28.679 --> 00:34:35.239 always wish that there was like a log rotate version for database tables or things 389 00:34:35.320 --> 00:34:38.719 like that, right where okay, after so many so much time has passed, 390 00:34:39.000 --> 00:34:43.559 you know, just take whatever's there and like stash it somewhere else. 391 00:34:44.679 --> 00:34:50.280 Yeah, almost like an auto database charter. Yeah right, yeah, yeah, 392 00:34:50.400 --> 00:34:53.440 Unfortunately it doesn't exist, but yeah, you need to build it yourself. 393 00:34:53.519 --> 00:34:58.079 Yeah, and we we plan to do that. We call it like 394 00:34:58.199 --> 00:35:01.760 retention period, so how long people need to retain their data and changes and 395 00:35:01.800 --> 00:35:07.480 audit Because storing everything in the hot database that can be quered at any time 396 00:35:07.599 --> 00:35:12.800 is expensive. That is why people sometimes move it to like some other cold 397 00:35:12.800 --> 00:35:15.920 storages and archival or something like that. Yeah, it's three buckets and that's 398 00:35:15.960 --> 00:35:20.960 awesome. I mean it's definitely needed. Uh. I wish so many times 399 00:35:21.000 --> 00:35:24.559 I had just like an option to buy that, right, Yeah, rather 400 00:35:24.599 --> 00:35:30.159 than have to build it and manage it and worry about performance of everything. 401 00:35:30.320 --> 00:35:37.599 Yeah, for sure. So where are you taking this from here? Like, 402 00:35:37.639 --> 00:35:40.199 what what are the next problems for you to solve? I mean we've 403 00:35:40.199 --> 00:35:45.239 talked about quite a lot of different so I mean we haven't dove in quite 404 00:35:45.840 --> 00:35:52.519 to maybe some of the specifics, like you know, doing various layers of 405 00:35:52.599 --> 00:35:58.239 logging, right, Like we've you've talked about like HTTP logging versus data based 406 00:35:58.239 --> 00:36:02.400 specific auditing, right and console log in which we haven't talked about, but 407 00:36:02.719 --> 00:36:08.079 really interesting stuff from like thirty seven signals right with Console eighty four and audits 408 00:36:08.519 --> 00:36:14.760 for doing that in the reals console. Yeah. There are many layers, 409 00:36:14.840 --> 00:36:19.239 Yeah, where so many layers? Yeah, many layers are covered and some 410 00:36:19.400 --> 00:36:23.079 are just there are huge gaps. So for example, you mentioned, yeah, 411 00:36:23.159 --> 00:36:28.239 HTTP logging, it's all done. For example, you can application execution, 412 00:36:28.480 --> 00:36:31.800 it's done. There are services data maybe it's solvable, and now we're 413 00:36:31.840 --> 00:36:37.719 trying to solve the dot on the database level console access. It's also I 414 00:36:37.719 --> 00:36:40.800 would say a little bit specific to the Ruby community. For example, in 415 00:36:40.840 --> 00:36:47.320 the JavaScript community, there is there are not usually good like repal consoles or 416 00:36:47.519 --> 00:36:53.239 that come with frameworks and things like that. And it's good and bad. 417 00:36:53.360 --> 00:36:58.320 I'm noticing that people. It's good that people don't go to this console to 418 00:36:58.360 --> 00:37:05.599 do crazy things there, but sometimes they go to scale console directly or connect 419 00:37:05.599 --> 00:37:09.119 to and scale and perform changes on the database level. I don't know which 420 00:37:09.159 --> 00:37:15.360 one is better. Yeah, with a review, Yeah, it's luckily we 421 00:37:15.440 --> 00:37:19.840 have great console experience. We have like gems like Pride that allow you to 422 00:37:20.039 --> 00:37:23.960 i don't know, go and jump into like classes and expect objects and things 423 00:37:24.000 --> 00:37:28.800 like that. And yeah, there are a couple of gems. One is 424 00:37:28.840 --> 00:37:32.440 Console nineteen eighty four and another one Audience nineteen eighty four. These are interesting 425 00:37:32.480 --> 00:37:38.280 gems. I personally haven't used them, but they seem to solve like very 426 00:37:39.519 --> 00:37:44.880 popular problem, especially among B to B companies. Probably when you have customer 427 00:37:44.960 --> 00:37:50.559 data or customer requests. You need to do something manually, but you want 428 00:37:50.599 --> 00:37:57.119 to audit what changes an engineer did. For example, what these gems do. 429 00:37:57.159 --> 00:38:00.400 They simply you install them in the rails project for example, when your 430 00:38:00.559 --> 00:38:06.519 law to your rails console. They'll prompt you and ask like why are you 431 00:38:06.760 --> 00:38:10.800 here, like what do you want to do? And engineers can specify like 432 00:38:12.119 --> 00:38:15.719 a ticket number and specify some context, I'm solving this customer problem blah blah 433 00:38:15.760 --> 00:38:20.599 blah, And you can type all the commands, and the commands may be 434 00:38:20.719 --> 00:38:25.559 locked in a separate database table. And the audience Jam nineteen eighty four allows 435 00:38:25.599 --> 00:38:31.199 you to inspect all these things. You can implement some additional approvals or like 436 00:38:31.360 --> 00:38:37.440 retroactive approvals. But these solutions are like and I don't know whether there is 437 00:38:37.480 --> 00:38:43.320 any better solution, but these solutions are not bulletproof in terms of reliability accuracy. 438 00:38:43.440 --> 00:38:46.320 Because it's Ruby. You can do whatever you want in a console. 439 00:38:46.400 --> 00:38:52.800 I can overwrite everything and then after that do crazy things that no one would 440 00:38:52.840 --> 00:38:58.719 see and notice. Of course, you're hoping that your colleagues or you won't 441 00:38:58.719 --> 00:39:04.920 do that breaking all the rules. But it requires discipline, and yeah, 442 00:39:04.960 --> 00:39:08.920 it's funny. I tried to implement something similar once at another company when we 443 00:39:09.039 --> 00:39:14.320 had a little bit more complex set up with SSOs to a WS accounts. 444 00:39:15.119 --> 00:39:20.679 These accounts would run on kumbernities certain process with the Rails console with some pre 445 00:39:20.800 --> 00:39:28.199 defined en viireables and these kind of like sets certain context for for example pay 446 00:39:28.199 --> 00:39:32.039 per trail or tools like that, so you can see who is making the 447 00:39:32.119 --> 00:39:37.119 changes. And yeah, I haven't shipped this feature or it was not like 448 00:39:37.239 --> 00:39:42.559 released, but it's still I would think that it's a problem for many So 449 00:39:42.639 --> 00:39:46.119 if you're listening to this podcast, yeah, that's one opportunity to improve how 450 00:39:46.199 --> 00:39:53.159 to improve log in and auditing of their console sessions because many companies just leave 451 00:39:53.239 --> 00:39:58.519 there like developers leave there troubleshooting purposes, yeah, and things like that. 452 00:39:59.679 --> 00:40:04.440 Yeah. Yeah, I remember first saying console nineteen eighty four and the audits 453 00:40:04.679 --> 00:40:09.000 nineteen eight for a kind of funny naming, but yeah, really cool. 454 00:40:09.039 --> 00:40:15.880 I've used them in a couple of projects. And it does have like you 455 00:40:15.000 --> 00:40:22.960 have to use like what is it the active record encryption stuff dataly so if 456 00:40:22.000 --> 00:40:27.159 you use maybe lock Box or something else, you're kind of like sol. 457 00:40:28.159 --> 00:40:32.639 Yeah, but it's really interesting, Like the interface is definitely pretty smooth, 458 00:40:34.159 --> 00:40:37.000 and I do like it. But you're right, like it doesn't really prevent 459 00:40:37.079 --> 00:40:39.320 anything. It's more just like all right, well, as long as you 460 00:40:39.320 --> 00:40:42.920 you know, and you know the goal is to hire people that you trust 461 00:40:42.960 --> 00:40:45.920 anyway, So as long as you have like at least a process in place 462 00:40:46.039 --> 00:40:51.800 to audit it, it definitely makes things much easier to track that. And 463 00:40:51.880 --> 00:40:54.159 from your experience, what did you do in these consoles? Usually, Like 464 00:40:54.920 --> 00:41:00.840 from my experience, it's usually either checking the data what's there because you don't 465 00:41:00.840 --> 00:41:06.440 have like fool UI for everything or something like that, or sometimes editing some 466 00:41:06.519 --> 00:41:12.679 stuff there as well. Some curious whether it's a common Yeah, I hate, 467 00:41:12.800 --> 00:41:16.639 I hate going in production console. So for me, it's mostly querying 468 00:41:16.679 --> 00:41:22.320 if anything, right, Like, you have a bunch of related objects and 469 00:41:22.360 --> 00:41:28.000 it's like almost not impossible, but like much worse to like go into the 470 00:41:28.079 --> 00:41:31.719 you know, a sequel dump or something like that and sift through the queries 471 00:41:31.760 --> 00:41:38.800 for it. But sometimes like you know, you need to, you know, 472 00:41:38.960 --> 00:41:45.480 just update some timestamp somewhere, and it's just like, you know, 473 00:41:45.599 --> 00:41:49.840 rather than having to generate a rag task that goes and runs, right, 474 00:41:50.559 --> 00:41:53.239 if you just have like one record that you're trying to get to that's problematic. 475 00:41:53.880 --> 00:42:00.199 That's like you know, failing a ton of repetitive workers or something like 476 00:42:00.239 --> 00:42:01.960 that, and you just want to stop it from doing that, you know, 477 00:42:02.760 --> 00:42:06.960 but you want to track that. It's that that that you were the 478 00:42:06.960 --> 00:42:10.239 one that like updated it, right, It's like it's very rare that you 479 00:42:10.320 --> 00:42:15.239 need to do that. But there are those cases. Yeah, in these 480 00:42:15.280 --> 00:42:21.039 cases when people need to update data. Yeah, I think there are many 481 00:42:21.119 --> 00:42:23.760 also approaches how you do that, like one one off, Yeah, you 482 00:42:23.760 --> 00:42:30.119 can write a rake task. Some people run it in migrations, some people 483 00:42:30.360 --> 00:42:34.840 execute stuff in rails, console or things like that. They think there is 484 00:42:34.880 --> 00:42:38.760 also like opportunity to improve things because writing, like it's good when it goes 485 00:42:38.840 --> 00:42:44.440 through the code, but the cycle can be in feedback cycle can be still 486 00:42:44.480 --> 00:42:46.719 like long and you don't know like whether you wrote everything perfectly or not. 487 00:42:47.880 --> 00:42:52.280 You need to test it and things like that. And yeah, I mean 488 00:42:52.440 --> 00:42:59.239 I would say in general like probably don't update things, yeah in a production 489 00:42:59.360 --> 00:43:02.440 box and general like no writing. I feel like that's a good rule of 490 00:43:02.480 --> 00:43:07.199 thumb. Many just ben yeah, access like only for example one person, 491 00:43:07.280 --> 00:43:13.480 or like certain like deblop steam can access or something for troubistioning or like certain 492 00:43:13.559 --> 00:43:17.639 emergencies. I know, I know some companies will do like you know only 493 00:43:17.960 --> 00:43:22.760 uh you know the console access only gets read replicas, so then it just 494 00:43:22.800 --> 00:43:25.159 prevents it in general, right, So then you just you can only read 495 00:43:25.159 --> 00:43:30.599 from the database in general anyway with your connected to console, which is really 496 00:43:30.639 --> 00:43:37.079 interesting and definitely like prevents it. Yeah yeah, but definitely not as flexible. 497 00:43:37.960 --> 00:43:40.880 Yeah, Ideally you'd need to tightly isolate everything, like there are certain 498 00:43:40.960 --> 00:43:46.880 other distributed systems like readis cash man cash people can also use and drop things 499 00:43:46.880 --> 00:43:52.719 by accident. Ideally would also want to isolate those, maybe certain HTTP requests, 500 00:43:52.559 --> 00:43:59.679 And we also like logging each TTP requests that are going from your application 501 00:44:00.239 --> 00:44:04.239 because I can use all environment viiables and send I don't know, stripe request 502 00:44:04.320 --> 00:44:08.360 to send myself like one million or something. Ideally it should be also traced 503 00:44:08.440 --> 00:44:15.639 like who made these HTTP request and or maybe ideally prevented from doing it in 504 00:44:15.639 --> 00:44:17.519 the console. So, I mean you bring up a good point, like 505 00:44:17.679 --> 00:44:22.840 I feel like we haven't touched much on like the recovery aspect of all of 506 00:44:22.880 --> 00:44:25.440 this, right, Like the whole point about and logging is in case you 507 00:44:25.440 --> 00:44:30.599 need to recover something, right like, or you need like next to reporting, 508 00:44:30.800 --> 00:44:34.360 right like, so reporting you can gather Okay, well we're doing a 509 00:44:34.639 --> 00:44:38.679 you know, audit specifically of everything that changed this particular thing. But like 510 00:44:39.320 --> 00:44:43.559 most of the times I feel like the reason for that is there's a recovery 511 00:44:43.599 --> 00:44:47.559 aspect following that up, right, Like, so where is that in this 512 00:44:47.599 --> 00:44:52.599 whole process? I know, like some of these jumps we've talked about have 513 00:44:52.800 --> 00:44:57.880 some of this, but I feel like it's still kind of missing, Like 514 00:44:57.880 --> 00:45:01.280 what is what is your like approach there, and like how are you working 515 00:45:01.400 --> 00:45:07.280 to like resolve this kind of like stuff. Yeah, recovery is very tricky 516 00:45:07.679 --> 00:45:14.000 problem in general. One issue, for example, is what if you somehow 517 00:45:14.159 --> 00:45:19.000 backed up some data or certain rows. You have the record historical version, 518 00:45:19.760 --> 00:45:23.480 but now your schema has changed. If you're using a scale like databases, 519 00:45:23.960 --> 00:45:28.639 Like, how do you resolve this conflict? What if you added the column 520 00:45:28.679 --> 00:45:31.000 you prepopulated it, but in your history you don't have it. Now you 521 00:45:31.079 --> 00:45:37.559 have to prepopulate it because the column may not like contain like shouldn't contain null 522 00:45:37.679 --> 00:45:42.360 values or something like that. So if that becomes a problem. In general, 523 00:45:42.440 --> 00:45:45.760 you have gems like paper Trail. They allow you to query the version 524 00:45:45.880 --> 00:45:51.039 find the right one. There is like a helper method called Rafi. You 525 00:45:51.079 --> 00:45:59.039 call it and it basically instantiates an active record record with the version, and 526 00:45:59.079 --> 00:46:05.920 then you can call say again and the tool overwrite and store the historical changes. 527 00:46:06.639 --> 00:46:10.039 As I mentioned, Yeah, there are some schema changes that they kind 528 00:46:10.079 --> 00:46:15.480 of like up to people to decide how to resolve these conflicts. There are 529 00:46:15.519 --> 00:46:22.519 some other different approaches that on the database layer people solve. So the one 530 00:46:22.639 --> 00:46:28.440 is very straightforward to snapshot or take back ups daily nightly. Ideally you maybe 531 00:46:28.519 --> 00:46:34.440 test your disaster recovery procedures and test whether you indeed can recover because very often 532 00:46:34.519 --> 00:46:40.239 no one tests those and just things that they're safe. And there are also 533 00:46:40.440 --> 00:46:49.519 new popular techniques maybe heard about point in time recovery, So that's popular feature 534 00:46:49.599 --> 00:46:54.920 that many database hosting providers advertise what they do at a like in terms of 535 00:46:55.000 --> 00:46:59.719 implementation, how it works, Okay, how would you use it? You 536 00:46:59.719 --> 00:47:05.559 would use it by simply selecting any time you want to recover data, and 537 00:47:05.639 --> 00:47:09.320 it will either create a new database instance with the state as of this time 538 00:47:09.360 --> 00:47:15.119 stamp or it will replace your database state. The way it works it uses 539 00:47:15.199 --> 00:47:22.119 these ride ahead locks that I mentioned previously. Because ride aheadlocks are like append 540 00:47:22.199 --> 00:47:29.840 only logs, these solutions they continuous to archive them. So some companies just 541 00:47:29.960 --> 00:47:36.079 take these valves and world records and they move them to S three and then 542 00:47:36.360 --> 00:47:39.440 if you need to roll back, they give you like either the timestamp when, 543 00:47:39.679 --> 00:47:45.559 for example, a certain blob was archived, or they may also do 544 00:47:45.679 --> 00:47:52.079 some binary search within a certain file to find the exact timestamp you want to 545 00:47:52.480 --> 00:47:57.840 try to recover to this data point. And there are some other like cool 546 00:47:57.920 --> 00:48:00.719 tag built on top of this idea. If you have like fowl and it's 547 00:48:01.320 --> 00:48:07.400 kind of like stored somewhere reliably, then you can do branching. So there 548 00:48:07.440 --> 00:48:12.320 are some serverless lots of server less hosting providers. What they allow you to 549 00:48:12.360 --> 00:48:20.039 do is to use basically copy on write copies for for example, your branch. 550 00:48:20.239 --> 00:48:22.679 When you want to test something on staging, what they allow you to 551 00:48:22.719 --> 00:48:29.079 do is basically create a database that will simply point to the state from a 552 00:48:29.159 --> 00:48:32.679 shared voeve kind of like log So instead of copying all data again and paying 553 00:48:32.719 --> 00:48:37.960 for all x ray resources, it will copy from the main let's say history, 554 00:48:37.400 --> 00:48:42.559 but any additional changes that you'll make, it will branch off and it 555 00:48:42.599 --> 00:48:47.639 will just incrementally record some additional stuff. So some super cool technologies that out 556 00:48:47.639 --> 00:48:54.159 there that help with branching as well as data recovery. Yeah, I've definitely 557 00:48:54.159 --> 00:48:58.800 seen the branching. It's pretty interesting, like is that a you know, 558 00:48:58.920 --> 00:49:02.599 it can just either wipe it off or like integrate it as as it merges 559 00:49:02.679 --> 00:49:07.880 down if you want it to, which is pretty interesting. Like, but 560 00:49:07.000 --> 00:49:10.559 in reality though, like what what have you seen? Kind of like things 561 00:49:10.559 --> 00:49:15.800 evolved too, like you know, what are people actually using? Are they 562 00:49:15.880 --> 00:49:19.480 using copy or on? Right? Like it seems like a very like complicated 563 00:49:19.519 --> 00:49:22.960 process to integrate with, like what what are the where the trend's going? 564 00:49:23.719 --> 00:49:31.840 Yeah? I think in general people prefer cloud managed databases usually for these reasons 565 00:49:31.880 --> 00:49:37.679 because they can manage backups reliably. So if you run something on RDS or 566 00:49:37.760 --> 00:49:40.960 like I don't know, render SAPA, base, HEROC or whatever, you 567 00:49:42.000 --> 00:49:47.280 can trust that the backups they take they're probably fine and you'll be able to 568 00:49:47.519 --> 00:49:52.880 roll back if necessary. That's one first line of defense. It's just relying 569 00:49:52.920 --> 00:49:59.920 on your hosting provider and any additional ones. Yeah, related to using additional 570 00:50:00.039 --> 00:50:04.639 tools, like the way I think about what we do. For example, 571 00:50:04.719 --> 00:50:09.719 at BEMI, we provide selective point in time recovery. What it means instead 572 00:50:09.719 --> 00:50:15.199 of like it's very hard and dangerous to roll back the whole database state, 573 00:50:15.559 --> 00:50:23.440 rather than like rolling back certain data records that were affected, and with tools 574 00:50:23.519 --> 00:50:29.599 like that, when you can filter out changes and select the ones that you 575 00:50:29.800 --> 00:50:36.159 want and just apply those, that's very useful because usually, yeah, I 576 00:50:36.280 --> 00:50:42.480 rarely saw when people would recover from like full database back up, because it 577 00:50:42.559 --> 00:50:46.039 means you would lose your history what happened after that, if certain other clients 578 00:50:46.119 --> 00:50:51.199 used it, and things like that, So it becomes very problematic and not 579 00:50:51.519 --> 00:50:55.199 very practical. So I have to ask this mostly because it's my interest, 580 00:50:55.280 --> 00:51:00.280 But like, do you see like AI fitting anywhere in this process us? 581 00:51:00.400 --> 00:51:04.199 Or is it kind of just like you know, there's this it's two sensitive 582 00:51:04.280 --> 00:51:07.639 of a topic to integrate with. Yeah, I think there are some, 583 00:51:09.199 --> 00:51:15.159 and I would share one idea that the problem we try to solve with AI, 584 00:51:15.239 --> 00:51:19.360 maybe we'll solve it, But it's okay if you have a desire to 585 00:51:19.400 --> 00:51:22.519 do it, to go for it, because it will be great if you 586 00:51:22.559 --> 00:51:29.199 can solve it. But one maybe obvious solution is like AI for SKL. 587 00:51:30.480 --> 00:51:37.599 There are many non technical people who want to use like query data in a 588 00:51:37.840 --> 00:51:42.360 like business intelligent tool, metabase or something else, but they don't have like 589 00:51:42.840 --> 00:51:47.360 the knowledge how to stitch things together for complex scenarios that's one idea. Another 590 00:51:47.440 --> 00:51:53.159 idea is we mentioned the audit trail and log and for example we have some 591 00:51:53.199 --> 00:52:00.559 customers, even non technical people use our UI for troubleshooting. But what could 592 00:52:00.679 --> 00:52:08.360 additionally help is data mapping. So usually database structure, it doesn't very well 593 00:52:08.400 --> 00:52:15.559 translate to real world language. You'd have some different ideas, maybe especially in 594 00:52:15.639 --> 00:52:19.719 Mongo. I remember like all days in Mango, it's like document based like 595 00:52:19.840 --> 00:52:24.159 database, and it doesn't have a structure, so the column names you may 596 00:52:24.199 --> 00:52:30.639 have different columns sort of columns. It was recommended for performance reasons to shorten 597 00:52:30.679 --> 00:52:34.719 your column names to like one or two letters or something like that because it 598 00:52:34.719 --> 00:52:38.920 would take less space. And imagine seeing like an audit trail with weird like 599 00:52:39.159 --> 00:52:45.519 data changes and with m is equal these timestan b is equal weird string. 600 00:52:45.159 --> 00:52:49.920 People would not be able to understand it. So where AI could feed it 601 00:52:50.199 --> 00:52:55.039 here is basically taking the structure and humanize it. Understand the data structure, 602 00:52:55.119 --> 00:53:00.800 understand maybe some business logic from the code base and muppets or anyone even non 603 00:53:00.840 --> 00:53:07.679 technical like customer support ors like the CEO could understand what's happening if they don't 604 00:53:07.760 --> 00:53:13.639 have like knowledge about how data is structured, and that's in general related to 605 00:53:13.679 --> 00:53:19.760 these ubiquitous language DVD concepts. Ideally everything should be the same, but that 606 00:53:19.880 --> 00:53:28.239 releively happens. That's one idea. Another one is like some companies they they 607 00:53:29.239 --> 00:53:37.280 maybe you heard about techniques like RAG for providing some additional context with custom data 608 00:53:38.119 --> 00:53:44.239 to your like AI system, so it can use internal data. Maybe if 609 00:53:44.320 --> 00:53:50.360 you've use perplexity. Perplexity uses other resources that maybe even like updated in real 610 00:53:50.400 --> 00:53:54.719 time just an hour ago. Plus they have all these citation links pointing to 611 00:53:54.760 --> 00:54:00.119 the source, so red techniques can help you to do that. But companies 612 00:54:00.119 --> 00:54:05.679 in general, when they deal with internal data, audit can also help. 613 00:54:06.920 --> 00:54:12.719 So you want to like have clean data, especially if you operate in the 614 00:54:12.760 --> 00:54:17.039 space when there are lots of documents and structured things maybe files, maybe emails. 615 00:54:19.760 --> 00:54:22.480 So I'm not sure, like it's for me, I don't know like 616 00:54:22.559 --> 00:54:28.000 how to productize it or what's the exact like solution. But the problem is 617 00:54:28.039 --> 00:54:30.440 that, yeah, people want to take this data, but they also want 618 00:54:30.480 --> 00:54:36.440 to understand like what happened, for example, why it was like that, 619 00:54:36.760 --> 00:54:40.000 why these AI systems aid it like that, maybe we can find unit and 620 00:54:40.039 --> 00:54:47.360 you'll see that with like, yeah, certain conflict changes your response rate and 621 00:54:47.400 --> 00:54:52.679 accuracy has improved and things like that, so it's important to audit the results 622 00:54:52.800 --> 00:55:00.159 of the AI queries as well. Yeah, super interesting takeaways there. Personally, 623 00:55:00.159 --> 00:55:04.320 I'm looking forward to, like, you know, just just getting some 624 00:55:04.360 --> 00:55:08.320 observation aspects of it out, so you know, like kind of just automate, 625 00:55:08.519 --> 00:55:13.920 automatically getting like the you know, bullet gem or something like that that 626 00:55:14.159 --> 00:55:16.840 just tells you, ay, like these things are not performing well, but 627 00:55:16.920 --> 00:55:23.239 without having to actually capture them. You just look at it sideways. Yeah, 628 00:55:23.480 --> 00:55:28.800 I hope. Yeah, things will improve and things will be implemented. 629 00:55:29.079 --> 00:55:32.800 Yeah, I use the topical pilot and yeah it's some people hate it, 630 00:55:32.880 --> 00:55:37.280 some people love it. I like it. When it changes your mind. 631 00:55:37.360 --> 00:55:42.599 You have to know and understand how to use it as well to better leverage 632 00:55:42.639 --> 00:55:45.360 it. Sometimes I would write a comment explaining what I'm trying to do and 633 00:55:45.599 --> 00:55:50.039 do it then so just me some code that more or less much is but 634 00:55:50.079 --> 00:55:52.559 then I still need to check sometimes and things like that. But yeah, 635 00:55:52.599 --> 00:55:59.400 there are lots of opportunities for AI. Very cool. Well, we've talked 636 00:55:59.440 --> 00:56:05.559 about so much here. Is there any other aspects you wanted to cover before 637 00:56:05.559 --> 00:56:08.920 we move into picks here. No, I think we covered a lot of 638 00:56:09.000 --> 00:56:15.159 ground. I think in general, Yeah, that's like tracking changes. Auditing 639 00:56:15.440 --> 00:56:20.159 is a big problem. There are tons of tons of ways how to solve 640 00:56:20.159 --> 00:56:25.199 this problem. Yeah, and I would recommend people try to evaliate them all 641 00:56:25.400 --> 00:56:31.239 instead of trying to just going with one by default because it's the hold this 642 00:56:31.360 --> 00:56:35.639 one or something like that. See what works best for you. Yeah, 643 00:56:35.719 --> 00:56:38.280 well, for sure people should check out bemy and your blog has a lot 644 00:56:38.320 --> 00:56:46.960 of great content around audit logging and tracking various activities and the database things you 645 00:56:47.000 --> 00:56:52.599 can add. I would definitely recommend people check that out for sure, because 646 00:56:52.039 --> 00:56:55.679 I'm already going to dive into quite a few articles on there. You've got 647 00:56:55.679 --> 00:57:01.519 a lot of great stuff there. All right, Well, let's dive in 648 00:57:01.519 --> 00:57:05.960 the picks. Do you have anything you want to share or I can go 649 00:57:06.000 --> 00:57:08.400 first if you want some time to think about it. I have two picks. 650 00:57:08.800 --> 00:57:14.119 One is non technical and one is technical. So the non technical one 651 00:57:14.280 --> 00:57:19.440 is maybe if you're old enough, you remember there was a technology called RSS 652 00:57:20.760 --> 00:57:24.360 and now I'm trying to use it on a day to day basis the way 653 00:57:24.400 --> 00:57:31.320 I'm using it, So there are for example, lots of great technical channels 654 00:57:31.400 --> 00:57:38.519 or content out there, and instead of these systems being pushed to me or 655 00:57:38.639 --> 00:57:43.480 me checking I don't know subreddits or like hacker news and things like that, 656 00:57:43.800 --> 00:57:50.239 I try to aggregate all my content as a kind of like funnel into an 657 00:57:50.360 --> 00:57:53.880 RSS feed. Some of them I use feedther, but there are many others 658 00:57:53.880 --> 00:58:01.079 and there's similar some of them they have help you get email address, so 659 00:58:01.119 --> 00:58:07.039 you can also subscribe to substack without polluting your in box. So I'm just 660 00:58:07.280 --> 00:58:10.079 organizing it and different subfolders I don't know tech related, I don't know, 661 00:58:10.159 --> 00:58:15.920 some engineering related and other topics, and I aggregate everything like stopstacks, top 662 00:58:16.159 --> 00:58:21.079 subreddits within a day or something. With some ural arguments, you can get 663 00:58:21.119 --> 00:58:25.320 to these feeds as well, tech blocks from different companies, top hockering news 664 00:58:25.360 --> 00:58:30.880 within a day or something, digest. So lots lots of resources that can 665 00:58:30.880 --> 00:58:34.719 be accumulated there, and I have just a queue that I'm going through and 666 00:58:34.760 --> 00:58:37.960 reading when I have time on mobile or on a desktop. And my second 667 00:58:38.079 --> 00:58:45.239 technical peak is Nix. I'm not sure if people heard about this system, 668 00:58:45.280 --> 00:58:51.239 but it's basically package manager and system configuration tool. It's sort of like our 669 00:58:51.360 --> 00:58:54.920 band r VM, SDF and all these things. But it's more powerful on 670 00:58:55.000 --> 00:59:05.119 the almost like operating system level. So it's not like just limited to programming 671 00:59:05.159 --> 00:59:09.400 languages for example, but the way we use it, for example, we 672 00:59:09.480 --> 00:59:15.480 have lots of projects in different languages. There is a full disclaimer. Nicks 673 00:59:15.599 --> 00:59:20.119 has, from my experience, has horrible devicts. It's very hard to use, 674 00:59:20.360 --> 00:59:22.880 very hard to understand what it does. So there are some wrappers around 675 00:59:22.920 --> 00:59:27.920 it, for example death box, but there are many others we use death 676 00:59:28.000 --> 00:59:31.679 box. So essentially, what it allows you to do is to bring any 677 00:59:31.760 --> 00:59:38.639 tax stack without Docker to your local machine, and it's isolated to your project, 678 00:59:39.119 --> 00:59:44.119 meaning it will create a separate folder for your postgress for your readis whatever, 679 00:59:44.440 --> 00:59:49.800 for your like jilascript engine or not Jazz or Rugby and things like that. 680 00:59:49.960 --> 00:59:52.800 You can specify all these dependencies so when you go to your project, 681 00:59:53.159 --> 00:59:58.880 you can keep using your I don't know favorite at a Sage orgh or whatever 682 00:59:59.000 --> 01:00:05.880 shell uh, and it just automatically enables all these additional dependencies within this project. 683 01:00:07.719 --> 01:00:14.239 Yeah, that these are my peeks for today. That's super cool. 684 01:00:14.679 --> 01:00:19.599 I'm gonna have to play around with that. Yeah, Nix is definitely a 685 01:00:19.679 --> 01:00:30.039 terrible, So thank you for sharing death box because that's pretty cool. So 686 01:00:30.280 --> 01:00:37.000 I only have one pick today. Uh. I my awesome co worker Anna, 687 01:00:37.199 --> 01:00:47.480 She wrote this incredible article on the Toroximity blog recently on evaluating GPD outputs 688 01:00:49.639 --> 01:00:53.719 and how we do it for medical professionals, uh and using golden data sets 689 01:00:53.920 --> 01:01:00.000 and programmatic emails and some human emails as well. Uh, to just make 690 01:01:00.039 --> 01:01:04.719 sure that uh, you know, that output you get from large language models 691 01:01:04.760 --> 01:01:07.480 is accurate, uh, at least following you know, a certain set of 692 01:01:07.480 --> 01:01:13.199 standards. So I recommend checking that out. It's really awesome and just like 693 01:01:13.320 --> 01:01:15.280 goes through a ton of stuff that leave them working out, which is really 694 01:01:15.280 --> 01:01:22.039 fun. That's great. Yeah, I hope that all healthcare providers not just 695 01:01:22.079 --> 01:01:31.280 blindly used AI without really too. Uh. Yeah, it's been great talking 696 01:01:31.320 --> 01:01:36.719 to you. I love to see all the work that you're doing, uh 697 01:01:36.480 --> 01:01:39.679 to help make auditing easier for a lot of businesses. And uh, you 698 01:01:39.719 --> 01:01:44.320 know, thank you for all the documentation and coming on the show today was 699 01:01:44.960 --> 01:01:49.559 really great. H you know, chatting with you today. If people want 700 01:01:49.559 --> 01:01:52.679 to get in contact with you, how can they reach out to you or 701 01:01:52.719 --> 01:01:59.280 find you on the web? Yes, I mostly and rarely use Twitter or 702 01:01:59.599 --> 01:02:06.320 x it's cold nowadays. I have like medium githop everywhere. My handle is 703 01:02:06.440 --> 01:02:10.760 exaspark. You can find me there. And yeah, thanks for having having 704 01:02:10.800 --> 01:02:15.880 me here Valentina today. It was nice chatting with you. Yeah, totally. 705 01:02:16.199 --> 01:02:20.400 Yeah. I mean until next time, everybody, I hope you learn 706 01:02:20.440 --> 01:02:23.320 a lot and can make use of a lot of this stuff because I know 707 01:02:23.360 --> 01:02:27.880 I will. All right, bye now, bye bye