WEBVTT 1 00:00:02.040 --> 00:00:05.440 So welcome everyone to another episode of Adventures in DevOps. 2 00:00:05.480 --> 00:00:08.960 Happy new Year, Warren, Happy new year. Thanks for joining 3 00:00:09.000 --> 00:00:09.400 me today. 4 00:00:10.279 --> 00:00:14.000 Thank you for having me back. Always a pleasure. Jillian, 5 00:00:14.240 --> 00:00:16.399 welcome back. How are you good? 6 00:00:16.480 --> 00:00:18.320 Good? How are you? And Happy New Year? 7 00:00:18.920 --> 00:00:19.679 I'm everyone. 8 00:00:20.559 --> 00:00:27.760 I'm pretty excited about today's episode because we have your background, AJ. 9 00:00:28.320 --> 00:00:30.160 So our guest today is a J Funk and his 10 00:00:30.280 --> 00:00:37.320 background is just so cool. So you're a software engineer 11 00:00:37.439 --> 00:00:43.320 for Rainforest, you are a you live in California, you're 12 00:00:43.359 --> 00:00:46.840 a snowboarder, and you are a previous d one and 13 00:00:47.039 --> 00:00:48.960 professional baseball player. 14 00:00:49.000 --> 00:00:49.799 Is that all correct? 15 00:00:50.320 --> 00:00:55.560 That is correct. I've taken an interesting career path, so yeah, 16 00:00:55.960 --> 00:00:58.439 definitely lots of stuff to do you here, and I'm 17 00:00:58.479 --> 00:01:01.359 in the Lake Tahoe area in California around the Porta 18 00:01:01.399 --> 00:01:05.480 in California and Nevada, So yeah, it's all true. 19 00:01:06.159 --> 00:01:08.120 What position did you play in baseball? 20 00:01:08.640 --> 00:01:09.400 I was a pitcher? 21 00:01:09.799 --> 00:01:11.519 A pitcher, Okay, cool. 22 00:01:11.319 --> 00:01:16.799 Oh that's what I know what it is. I was 23 00:01:16.799 --> 00:01:18.519 about to be like, I've told you what we're talking 24 00:01:18.560 --> 00:01:20.400 about here, but I know what the picture is. 25 00:01:21.760 --> 00:01:23.599 It's all about the little connections, Jillian. 26 00:01:24.120 --> 00:01:25.000 It really is. 27 00:01:26.400 --> 00:01:26.920 So cool. 28 00:01:27.760 --> 00:01:30.319 A j tell us a little bit about your role 29 00:01:30.439 --> 00:01:34.799 at Rainforest and what Rainforest does, and I think I'm 30 00:01:34.799 --> 00:01:41.439 interested to hear what led you to go from pitching 31 00:01:41.599 --> 00:01:44.120 Major League Baseball to pitching code? 32 00:01:45.760 --> 00:01:50.480 Yeah for sure. Yeah, so I started writing code when 33 00:01:50.640 --> 00:01:53.680 when I was a kid, really always as a hobby. 34 00:01:54.319 --> 00:01:57.719 My plan life was always to play baseball. You know, 35 00:01:57.799 --> 00:02:00.200 once you become an adult and you realize that's not 36 00:02:00.239 --> 00:02:04.280 always such a viable career path. I started realizing that 37 00:02:04.359 --> 00:02:06.159 this thing I do as a hobby I could do 38 00:02:06.239 --> 00:02:09.680 as a career. So just started kind of dabbling and 39 00:02:09.719 --> 00:02:12.439 realizing that I actually really enjoyed doing this, and it 40 00:02:12.560 --> 00:02:16.319 was a big pivot from you know, just athletics all 41 00:02:16.360 --> 00:02:21.240 the time. So yeah, now I ended up at Rainforest QA. 42 00:02:21.360 --> 00:02:24.319 I've been here for seven and a half years now. 43 00:02:24.879 --> 00:02:28.159 Oh wow, Yeah, it's been a long time, and it's 44 00:02:28.400 --> 00:02:31.240 it's been really fun. I work with really really great people, 45 00:02:31.840 --> 00:02:37.199 really intelligent experienced engineers, really great culture. We're distributed all 46 00:02:37.240 --> 00:02:39.080 over the world, so it's fun. We get to go 47 00:02:39.479 --> 00:02:41.719 meet up with each other every once in a while 48 00:02:42.000 --> 00:02:45.039 in some random place in the world, and so it's 49 00:02:45.120 --> 00:02:48.800 it's a really enjoyable place to work. I specialize in 50 00:02:48.800 --> 00:02:51.120 front and development, So I spent a lot of time 51 00:02:51.159 --> 00:02:54.680 with the product and design team shaping how our product works. 52 00:02:55.080 --> 00:02:58.680 And you know, as a quality assurance company, it's really 53 00:02:58.719 --> 00:03:01.639 important for us to have a really high bar of 54 00:03:01.759 --> 00:03:05.960 quality and reliability for a product. Obviously, if our app breaks, 55 00:03:06.240 --> 00:03:08.280 why are you going to trust us to make sure 56 00:03:08.280 --> 00:03:14.080 that your app doesn't break? It's reasonable, yeah, right, and 57 00:03:14.639 --> 00:03:17.479 you know, to meet that high bar, we use rainforest 58 00:03:17.560 --> 00:03:20.439 to test Rainforest. So being able to eat your own 59 00:03:20.479 --> 00:03:23.159 dog food on a daily basis is a really really 60 00:03:23.199 --> 00:03:26.680 good position to be in. It allows you to identify 61 00:03:26.680 --> 00:03:29.680 pain points in the user experience before your users do. 62 00:03:30.120 --> 00:03:32.879 Hopefully that's not always true, but we do our best, 63 00:03:33.560 --> 00:03:35.439 and it also means we spend a lot of time 64 00:03:35.479 --> 00:03:38.199 thinking about the best way to do QA, both from 65 00:03:38.240 --> 00:03:42.800 a kind of philosophical standpoint and from a practical implementation 66 00:03:43.080 --> 00:03:49.479 standpoint reality in other words, so this experience obviously helps 67 00:03:49.599 --> 00:03:52.639 guide our product roadmap, but it's also led me to 68 00:03:52.639 --> 00:03:55.560 develop a lot of strong opinions on how product and 69 00:03:55.599 --> 00:03:59.280 engineering team should shape their testing strategies, what kind of 70 00:03:59.280 --> 00:04:02.560 tools they should you'd be using, and how they can 71 00:04:02.560 --> 00:04:06.759 ship code quickly and continuously without having a sacrifice quality 72 00:04:08.240 --> 00:04:08.639 for sure. 73 00:04:08.680 --> 00:04:11.919 For me, that seems like one of those rabbit holes 74 00:04:11.919 --> 00:04:15.080 that it's really hard to find the balance on of 75 00:04:15.120 --> 00:04:19.399 like what's the minimum level of testing that you should 76 00:04:19.480 --> 00:04:24.079 be doing, and then like what's a what's an adequate 77 00:04:24.199 --> 00:04:29.399 level to get into where you're actually still getting good 78 00:04:29.480 --> 00:04:31.279 use for your time, you know, because obviously you can 79 00:04:32.399 --> 00:04:36.480 throw something at it that tests every possible combination or 80 00:04:37.439 --> 00:04:39.959 or path through your application that could ever be taken, 81 00:04:40.000 --> 00:04:42.879 and you're going to reach a point of diminishing returns there. 82 00:04:42.879 --> 00:04:44.759 So how do you figure out what that right spot is? 83 00:04:45.519 --> 00:04:50.160 Yeah? Absolutely, the myth of one hundred percent test coverage, right, Yeah, Yeah, 84 00:04:50.360 --> 00:04:53.600 that we we all strive for. I think they teach 85 00:04:53.639 --> 00:04:57.319 you very early on that, like one hundred test coverage 86 00:04:57.360 --> 00:05:00.439 is what we should be doing with every every time 87 00:05:00.439 --> 00:05:03.079 we push code there, we should make sure everything is covered. 88 00:05:03.519 --> 00:05:05.639 The reality is that's just not possible, right, What does 89 00:05:05.639 --> 00:05:07.439 that even mean? Does that mean every line of code 90 00:05:07.519 --> 00:05:11.800 is covered? Does that mean every possible edge case is covered? Like, 91 00:05:11.879 --> 00:05:14.160 you're never going to get all of those things. So 92 00:05:14.199 --> 00:05:18.639 the trick is finding that balance, right, We want to 93 00:05:18.639 --> 00:05:22.480 make sure that we have confidence in the thing that 94 00:05:22.519 --> 00:05:25.720 we're shipping that both the thing we're shipping works and 95 00:05:25.759 --> 00:05:28.360 that we're not breaking anything else. But we also want 96 00:05:28.399 --> 00:05:30.279 to be able to move quickly. We don't want this 97 00:05:30.399 --> 00:05:34.879 to get into our way, and so the main ways 98 00:05:34.879 --> 00:05:38.439 we go about that is really thinking about what your 99 00:05:38.439 --> 00:05:42.560 testing strategy is, right, What are you testing? What is 100 00:05:42.600 --> 00:05:46.040 your bar for quality? Because in reality sometimes we're okay 101 00:05:46.079 --> 00:05:49.240 if some things break, especially when we're in the early 102 00:05:49.279 --> 00:05:56.160 stages of prototyping something it's a beta feature, etc. And 103 00:05:57.920 --> 00:06:02.120 what are the layer of your testing, right, so we 104 00:06:02.199 --> 00:06:05.240 can we can think about our testing strategy as layers, 105 00:06:05.759 --> 00:06:11.680 with pictured as three layers, but more as a period. Right. 106 00:06:11.759 --> 00:06:17.480 So your foundation of your testing strategy is your unit tests, 107 00:06:17.480 --> 00:06:20.439 something we're all familiar with. We're using code to test 108 00:06:20.480 --> 00:06:21.800 code and small chunks. 109 00:06:21.879 --> 00:06:22.040 Right. 110 00:06:22.079 --> 00:06:25.000 What a unit is is totally up to you. We 111 00:06:25.040 --> 00:06:28.720 can define it as a single function or some class 112 00:06:28.720 --> 00:06:32.759 component whatever. The top of our pyramid is our end 113 00:06:32.800 --> 00:06:37.759 to end or our UI tests. That's testing our application 114 00:06:38.000 --> 00:06:41.279 in a kind of real world scenario. As we go 115 00:06:41.399 --> 00:06:46.639 up the pyramid, these tests are more comprehensive, we can 116 00:06:46.680 --> 00:06:50.920 rely on them more, but they're slower, they're more expensive. Right, 117 00:06:51.279 --> 00:06:53.199 So at the base of our pyramid, we have all 118 00:06:53.240 --> 00:06:55.959 of these unit tests that we could run constantly. So 119 00:06:56.040 --> 00:06:58.959 at Rainforest, we run these every single time you push 120 00:06:59.000 --> 00:07:01.399 code it. Just run them because it's cheap. We don't 121 00:07:01.399 --> 00:07:04.160 really care if you're lazy like me. I like to 122 00:07:04.240 --> 00:07:06.680 just push my code up and oh, something broke, I 123 00:07:06.759 --> 00:07:09.480 didn't notice, because I don't want to run the whole 124 00:07:09.480 --> 00:07:12.839 test suite all the time on my local machine. But 125 00:07:13.480 --> 00:07:17.639 as you move up that pyramid from unit tests, the 126 00:07:17.680 --> 00:07:20.120 middle of that pyramid would be our integration test, so 127 00:07:20.199 --> 00:07:23.680 basically testing multiple chunks of these units and how they 128 00:07:23.720 --> 00:07:26.759 interact with each other. That could be maybe one of 129 00:07:26.759 --> 00:07:30.079 your micro services talking to another micro service or something 130 00:07:30.120 --> 00:07:32.040 like that. As we get to the top of the 131 00:07:32.040 --> 00:07:34.240 pyramid and we start running these end to end tests, 132 00:07:34.680 --> 00:07:38.639 that's where I think strategy becomes much more important because, 133 00:07:39.120 --> 00:07:41.759 like I mentioned, they're slower, so we actually care about 134 00:07:41.759 --> 00:07:43.959 when we run these. We need to be more strategic 135 00:07:44.079 --> 00:07:48.759 about how often we run them. And they are more expensive, 136 00:07:49.040 --> 00:07:50.720 so we don't want to just blow a bunch of 137 00:07:51.120 --> 00:07:54.240 a whole bunch of money on them, right, So finding 138 00:07:54.279 --> 00:07:56.680 the balance between those things is the real trick. 139 00:07:57.199 --> 00:07:59.519 I like that you pulled out test pyramid and not 140 00:07:59.639 --> 00:08:04.600 one of the other newer hipster trends of the test 141 00:08:06.079 --> 00:08:10.519 diamond or a test climb model gone with the tried 142 00:08:10.560 --> 00:08:12.920 and true test pyramid. I mean, at least that's how 143 00:08:12.959 --> 00:08:15.480 I've always seen it. And you know, it's also really 144 00:08:15.480 --> 00:08:18.959 interesting that you bring up the one hundred percent test 145 00:08:18.959 --> 00:08:22.279 coverage is not possible at least from like anecdotal experience 146 00:08:22.319 --> 00:08:25.639 for me, I found it's almost like a preto distribution 147 00:08:26.360 --> 00:08:29.639 and follows the eighty twenty rule where if you wanted 148 00:08:29.639 --> 00:08:31.839 to have one hundred percent test coverage, it would actually 149 00:08:31.920 --> 00:08:33.879 require an infinite amount of time. 150 00:08:34.600 --> 00:08:41.120 Absolutely, yeah, we certainly strive to have all that test coverage, 151 00:08:41.120 --> 00:08:44.039 but I think the reality of one hundred percent test 152 00:08:44.080 --> 00:08:48.360 coverage is more along the lines of how you define it. 153 00:08:48.440 --> 00:08:52.399 Like your user workflows, Right, so the typical examples like 154 00:08:52.440 --> 00:08:57.080 your log and flow. What are the the main outcomes 155 00:08:57.120 --> 00:09:00.759 of the log in it's successful, log in, failed, log in? 156 00:09:00.960 --> 00:09:04.360 Maybe I forgot your password? And do you have test 157 00:09:04.399 --> 00:09:09.080 coverage and to end test coverage on that flow? If so, 158 00:09:09.279 --> 00:09:12.799 we could usually consider that cover. Do I have a 159 00:09:12.960 --> 00:09:15.840 unit test that covers every combination of things that I 160 00:09:15.879 --> 00:09:20.159 could type into that box. Absolutely not, but having some 161 00:09:20.279 --> 00:09:22.039 sort of test covered on it to make sure that 162 00:09:22.039 --> 00:09:25.519 it actually loads in some kind of real world scenario 163 00:09:25.759 --> 00:09:27.519 gives me much more confidence. 164 00:09:30.399 --> 00:09:31.960 With a lot of things like logins. 165 00:09:32.080 --> 00:09:35.559 And there's so many areas these days in developing software. 166 00:09:35.600 --> 00:09:42.200 You're using SaaS products as a as the mechanism for 167 00:09:42.279 --> 00:09:43.159 implementing that. 168 00:09:43.759 --> 00:09:45.279 What's your approach for. 169 00:09:47.039 --> 00:09:52.159 Dealing with that external dependency? Because you can you can 170 00:09:52.240 --> 00:09:54.960 mock it, or you can try to stimulate it, or 171 00:09:55.000 --> 00:09:56.120 you can actually call it. 172 00:09:56.519 --> 00:09:57.919 What do you what do you think about those? 173 00:09:58.240 --> 00:10:00.320 I feel like that the job at a me. Well, 174 00:10:00.440 --> 00:10:03.200 I see you, I see you're coming there. 175 00:10:04.399 --> 00:10:08.480 Now now I'm trying to Yeah, I got nothing came 176 00:10:08.559 --> 00:10:09.120 back to you. 177 00:10:12.919 --> 00:10:18.639 Yeah. So I think what you're asking is when we 178 00:10:18.840 --> 00:10:22.480 run any kind of tests, take We'll stick with unit 179 00:10:22.519 --> 00:10:26.360 tests for for the time being. There's a context that 180 00:10:26.399 --> 00:10:32.240 they run inside of, right, and if we wanted to 181 00:10:32.279 --> 00:10:35.919 test our app as close as possible to reality, we 182 00:10:35.960 --> 00:10:39.759 would test it in production on an actual machine with 183 00:10:39.840 --> 00:10:43.159 an actual human, which some people do. Right. There's obviously 184 00:10:43.200 --> 00:10:46.240 downsides to that testing and production probably don't have to 185 00:10:46.240 --> 00:10:49.399 get into why that's not a great idea. 186 00:10:49.720 --> 00:10:52.720 And the entire video game industry is wrong. 187 00:10:58.039 --> 00:11:01.240 And yes they are wrong. I mean, I'm not even 188 00:11:01.279 --> 00:11:02.200 sure that's true anymore. 189 00:11:02.240 --> 00:11:04.639 Like they don't even release games, right, It's just like 190 00:11:04.759 --> 00:11:07.039 content that you click download on and you pay for 191 00:11:07.120 --> 00:11:09.799 and then the game comes later or something like. I 192 00:11:09.919 --> 00:11:11.799 think that's what the game industry has gone towards. 193 00:11:11.960 --> 00:11:13.559 That does tend to be how it goes. But I 194 00:11:14.320 --> 00:11:16.240 still feel like they have the users doing an awful 195 00:11:16.279 --> 00:11:19.840 lot of acceptance testing in the video game industry and 196 00:11:19.919 --> 00:11:22.960 I'm like, too cheap for this nonsense. But anyways, I'm 197 00:11:22.960 --> 00:11:25.799 trying not to be rail the entire conversation today, so 198 00:11:25.919 --> 00:11:27.759 we can we can skip right on over that. 199 00:11:28.320 --> 00:11:31.519 No, I totally agree. It's like you do. Even when 200 00:11:31.559 --> 00:11:34.519 you do get a game, it's an incomplete game that's buggy, 201 00:11:34.600 --> 00:11:36.080 and you play it for an hour and you know, 202 00:11:36.120 --> 00:11:39.960 I'm never playing this again. So I'm a late adopter 203 00:11:40.080 --> 00:11:42.559 when it comes to these things. I wait till the 204 00:11:42.600 --> 00:11:49.720 Internet stops screaming about it and I started downloading. But yeah, 205 00:11:49.879 --> 00:11:53.399 when obviously we don't want to test our applications in 206 00:11:53.480 --> 00:11:55.679 production because we're smarter than that, and we have the 207 00:11:55.679 --> 00:12:00.559 ability to test these things in different environments. When we 208 00:12:00.919 --> 00:12:04.080 are running things like unit tests, we're kind of stuck 209 00:12:04.279 --> 00:12:09.840 inside of this artificial context. Right. If you're just running 210 00:12:09.879 --> 00:12:15.120 code to test code, it's inside of that specific code environment. 211 00:12:15.200 --> 00:12:18.720 It's given us inputs and outputs. Even as we go 212 00:12:18.879 --> 00:12:21.399 up the chain to some things that like to call 213 00:12:21.480 --> 00:12:24.240 themselves and the end tests which I kind of disagree with, 214 00:12:24.279 --> 00:12:28.720 which would be things like dombased testing, you're still stuck 215 00:12:28.759 --> 00:12:30.279 inside some kind of context. 216 00:12:30.440 --> 00:12:30.639 Right. 217 00:12:30.720 --> 00:12:33.480 So if the DOM, if you're not familiar, it's the 218 00:12:33.559 --> 00:12:39.200 Document Object Model, and it's essentially the application in interface 219 00:12:39.240 --> 00:12:41.960 that we have with the browser. So it's how our 220 00:12:42.039 --> 00:12:45.879 JavaScript code talks to the browser, how we manipulate things, 221 00:12:45.919 --> 00:12:49.120 how we read things from the browser. And so the 222 00:12:49.159 --> 00:12:53.399 important nuance here is that our code interacts with the DOM. 223 00:12:53.799 --> 00:12:57.200 A human being doesn't interact with the DOM. Right when 224 00:12:57.200 --> 00:12:59.440 you go click a button, you don't go talk to 225 00:12:59.480 --> 00:13:03.879 the DOM, interact with the user phase user interface. So 226 00:13:04.120 --> 00:13:07.600 we want to get our tests as close to actual 227 00:13:07.679 --> 00:13:10.000 end to end tests as possible, right, A human looking 228 00:13:10.039 --> 00:13:13.879 at the screen, a human interacting with the screen. And 229 00:13:15.039 --> 00:13:18.720 if we're not in that production environment, every step we 230 00:13:18.799 --> 00:13:21.399 take away from that gets us further from reality, right, 231 00:13:21.440 --> 00:13:27.639 it gives us this false sense of security. Sometimes there's 232 00:13:28.200 --> 00:13:32.000 a really common example with with DOM based tools, might 233 00:13:32.080 --> 00:13:36.360 be click this button, did it work right? Well, just 234 00:13:36.399 --> 00:13:39.000 because you can't interact with that button through the DOM 235 00:13:39.480 --> 00:13:42.559 doesn't mean your user can actually interact with that button right. 236 00:13:43.039 --> 00:13:45.879 There might be you know, some kind of overlay over 237 00:13:45.960 --> 00:13:48.080 my button. The button might be off of the screen. 238 00:13:48.720 --> 00:13:50.360 But when I asked the DOM, can I click the button, 239 00:13:50.399 --> 00:13:53.200 it says, yeah, we clicked it. It worked. Ship the 240 00:13:53.240 --> 00:13:55.200 code and now no one can log into your AP 241 00:13:55.240 --> 00:13:58.120 because no one can click the button right, And so 242 00:13:58.399 --> 00:14:02.440 doing ours as much as we can to get to 243 00:14:02.519 --> 00:14:06.080 that real world scenario, creating testing and staging environments that 244 00:14:06.159 --> 00:14:10.039 mirror production as much as possible, and loading these things 245 00:14:10.120 --> 00:14:15.639 into virtual machines with operating systems instead of a headless browser, 246 00:14:15.679 --> 00:14:18.039 which is, you know, basically a browser with no UI, 247 00:14:18.480 --> 00:14:21.600 and interacting with it in a way that human doesn't 248 00:14:21.639 --> 00:14:24.840 interact with it just gets us further away from reality. 249 00:14:25.679 --> 00:14:28.320 I mean, it's interesting you bring that up, and I'm 250 00:14:28.360 --> 00:14:30.720 sort of now I'm intrigued if you maybe want to 251 00:14:30.799 --> 00:14:34.320 roast all what we've been telling our customers. So obviously, 252 00:14:34.399 --> 00:14:37.399 where where we provide a third party product for our 253 00:14:37.440 --> 00:14:40.679 customers for a log in and access control, so they have, 254 00:14:41.240 --> 00:14:44.519 you know, providing them the off needs there, and I 255 00:14:44.559 --> 00:14:46.360 think the biggest advice that we end up giving them 256 00:14:46.440 --> 00:14:50.399 is like we are already testing that thing, Like, you know, 257 00:14:50.559 --> 00:14:54.440 don't focus on this. You're wasting your time duplicating our testing. 258 00:14:54.720 --> 00:14:57.360 If you felt the need to do that, it's almost 259 00:14:57.360 --> 00:15:00.320 like you don't trust us with our product, and then 260 00:15:00.399 --> 00:15:03.279 you probably should question why you're using that solution in 261 00:15:03.320 --> 00:15:06.120 the first place. If you get to that point, you know, 262 00:15:06.200 --> 00:15:10.320 that's actually a conversation more than it's a technical solution. However, 263 00:15:10.320 --> 00:15:12.600 we do find some customers still have a need to 264 00:15:12.639 --> 00:15:14.759 go a little bit further and the thing that we've done, 265 00:15:14.879 --> 00:15:17.240 I don't know this is the right answer, but we 266 00:15:17.320 --> 00:15:20.200 provide a given it is a SAS product, we provide 267 00:15:20.200 --> 00:15:23.000 a clone of our service as a container that can 268 00:15:23.080 --> 00:15:27.559 run that it is trimmed down, only has minor features, 269 00:15:27.600 --> 00:15:30.600 but allows the flows that you're going to test or 270 00:15:30.600 --> 00:15:35.000 you want to actually verify available without having to go 271 00:15:35.039 --> 00:15:37.360 through all the complexity that the service actually provides. 272 00:15:38.440 --> 00:15:40.240 Yeah, I mean, I think that's a good compromise. Right. 273 00:15:40.279 --> 00:15:43.519 So in this whole strategy, of fending balance between our 274 00:15:44.120 --> 00:15:49.720 confidence and our velocity and shipping. The reality is our 275 00:15:49.759 --> 00:15:52.320 testing environs are not going to match our production environments 276 00:15:52.320 --> 00:15:55.200 all the time, right, and a lot of times we're 277 00:15:55.240 --> 00:16:01.159 constrained by resources. So I think in a situation like that, 278 00:16:01.159 --> 00:16:03.600 that totally makes sense. You know, some kind of pared 279 00:16:03.639 --> 00:16:07.519 down version of your production application. I think the important 280 00:16:07.519 --> 00:16:11.879 thing is how you're testing it, right. If we are 281 00:16:11.919 --> 00:16:14.080 able to just kind of like strip down our product 282 00:16:14.240 --> 00:16:18.399 and test the bare bones version of it, as long 283 00:16:18.440 --> 00:16:20.919 as we are in an environment, right, they're clicking on 284 00:16:20.960 --> 00:16:23.120 it through I don't know, a web browser or whatever 285 00:16:23.200 --> 00:16:25.879 it might be, versus just like you know, running some 286 00:16:25.960 --> 00:16:28.039 script in the background. I think that's a really good 287 00:16:28.080 --> 00:16:30.799 balance between those two things. The key here is that 288 00:16:30.799 --> 00:16:34.720 we're still doing and testing right. I imagine it's you 289 00:16:34.759 --> 00:16:37.840 know that someone's typing in the box, a button's being clicked, 290 00:16:38.039 --> 00:16:40.559 there's an HTTP request or whatever it might be to 291 00:16:40.639 --> 00:16:42.919 an API that reads from a database, and we're checking 292 00:16:42.919 --> 00:16:46.559 out all of these things actually work together. So yeah, 293 00:16:46.720 --> 00:16:48.279 I think that's that's a good compromise. 294 00:16:49.720 --> 00:16:51.240 I think one of the things that actually comes up 295 00:16:51.279 --> 00:16:54.600 a lot is maybe just on a slight tangent is 296 00:16:54.639 --> 00:16:58.200 people are so focused on and testing they never stop 297 00:16:58.320 --> 00:17:02.399 to question should we like for that particular flow, is 298 00:17:02.840 --> 00:17:05.440 that where the value is for our company? Is it 299 00:17:05.799 --> 00:17:07.880 really where we should put a lot of resources? And 300 00:17:07.920 --> 00:17:11.559 do find that those that you're working with or your 301 00:17:11.559 --> 00:17:15.519 customers may or may not know where the highest value 302 00:17:15.680 --> 00:17:19.240 testing should be done, And then that's a conversation or 303 00:17:19.279 --> 00:17:19.839 maybe it's. 304 00:17:19.680 --> 00:17:24.240 Something that your tool provides. Yeah. Absolutely, And I think 305 00:17:24.319 --> 00:17:27.279 the trick between the trick for that is doing it 306 00:17:27.279 --> 00:17:32.599 early right. So, if you have a large application in 307 00:17:32.640 --> 00:17:34.480 a code base and you have not written any end 308 00:17:34.599 --> 00:17:38.039 n tests, it's hard to determine where to start, right 309 00:17:38.440 --> 00:17:42.079 versus when if you start early on, it kind of 310 00:17:42.079 --> 00:17:44.599 writes itself. Right. The first day is your login, you 311 00:17:44.640 --> 00:17:50.119 have some login coverage. Determining where the highest value is 312 00:17:50.119 --> 00:17:53.720 is certainly up to each usually the product team, Right, 313 00:17:53.759 --> 00:17:57.160 what do we care most about not breaking? And can 314 00:17:57.200 --> 00:18:00.400 we create some kind of smoke test that spans all 315 00:18:00.440 --> 00:18:03.359 of these right, So each one of these tests has 316 00:18:03.400 --> 00:18:08.039 a certain level of granularity to it. A good smoke 317 00:18:08.079 --> 00:18:10.799 test might be can I log into my application? Can 318 00:18:10.839 --> 00:18:13.759 I create a thing? Can I delete a thing? And 319 00:18:13.839 --> 00:18:18.359 things just like generally work. Those initially are your highest 320 00:18:18.440 --> 00:18:22.119 value tests because I know that my app actually loads 321 00:18:22.200 --> 00:18:24.960 in reality, right, regardless of what my unit tests say. 322 00:18:25.880 --> 00:18:30.559 After that, it's usually defined by what those user flows are. Right. So, 323 00:18:30.720 --> 00:18:33.480 as you're scoping something out with your product team, here's 324 00:18:33.519 --> 00:18:37.039 this new feature that we're building, it's really important in 325 00:18:37.119 --> 00:18:42.000 your planning process to include that right right tests for it. 326 00:18:42.640 --> 00:18:47.400 These are things that we usually as developers, kind of 327 00:18:47.440 --> 00:18:49.720 bake into our estimates, Right, I have the right unit 328 00:18:49.759 --> 00:18:53.839 tests for this. At Rainforest, we've shifted more towards baking 329 00:18:53.880 --> 00:18:56.880 and Rainforest tests for these things. We obviously have unit tests, 330 00:18:57.839 --> 00:19:01.480 But getting the coverage at the time of implementation or 331 00:19:01.519 --> 00:19:04.160 the time of release or whatever that might be, is 332 00:19:04.279 --> 00:19:07.359 usually your best bet to get that. If I have 333 00:19:07.480 --> 00:19:11.680 a large, large application and I don't have that coverage yet, 334 00:19:12.119 --> 00:19:15.400 it is certainly a balancing act figuring out what should 335 00:19:15.440 --> 00:19:18.079 be test first, Right, So I would certainly start with 336 00:19:18.119 --> 00:19:22.079 those kind of smoke tests, and then your highest used 337 00:19:22.319 --> 00:19:26.519 features is usually a really good place to start. The 338 00:19:26.559 --> 00:19:29.559 pitfall that you go into is putting too much nuance 339 00:19:29.599 --> 00:19:31.559 in a lot of these tests, right, What if they 340 00:19:31.559 --> 00:19:33.599 click into this and click out of that and then 341 00:19:33.680 --> 00:19:38.000 open this menu and whatever. Keeping them very coherent and 342 00:19:38.279 --> 00:19:40.920 legible and kind of focused on the thing that they're 343 00:19:40.960 --> 00:19:45.440 testing is the important piece of having efficient tests that 344 00:19:45.480 --> 00:19:46.599 you can maintain over time. 345 00:19:46.920 --> 00:19:49.440 I saw Will smarting there, and I know he's just 346 00:19:49.599 --> 00:19:52.880 entered into a new, glorious position at his organization, so 347 00:19:52.880 --> 00:19:56.559 maybe he has some unique insight that he's interested in 348 00:19:57.039 --> 00:19:58.759 blessing else with No. 349 00:19:59.000 --> 00:20:02.160 I was curious because this is an opportunity for me 350 00:20:02.200 --> 00:20:05.920 to throw in a buzzword that's trending, and so once 351 00:20:06.000 --> 00:20:09.599 the episode is transcribed, we'll just go viral on that. 352 00:20:09.720 --> 00:20:12.079 So does AI play a role in. 353 00:20:12.079 --> 00:20:15.960 Helping figure out that type of user flow? In the 354 00:20:16.000 --> 00:20:19.039 different like odd places you can end up. 355 00:20:20.400 --> 00:20:25.039 Good question? That's my knowledge. Yet, you know, as really 356 00:20:25.039 --> 00:20:27.400 good at some things and really bad at some things, 357 00:20:27.440 --> 00:20:31.359 and we haven't quite figured out how to make it 358 00:20:31.720 --> 00:20:34.640 give it enough context to understand how it should go 359 00:20:34.640 --> 00:20:38.559 about testing your app. Right, we do have some really 360 00:20:38.599 --> 00:20:44.160 cool AI tools that rain for us. They don't determine 361 00:20:44.200 --> 00:20:46.960 what your test coverage should be. Rather that's kind of 362 00:20:47.039 --> 00:20:49.640 left up to you and then it helps you write 363 00:20:49.680 --> 00:20:53.359 the test. So what we have is entering a prompt, 364 00:20:53.440 --> 00:20:56.400 you know, say, it could be something pretty generic. Log 365 00:20:56.480 --> 00:20:58.960 in and Adam add an item to the cart and 366 00:20:59.039 --> 00:21:01.680 check out something like that, and it will generate your 367 00:21:01.759 --> 00:21:06.759 reinforce steps for you. So during execution, AI is left 368 00:21:06.759 --> 00:21:08.720 out of it. Right, It does the initial generation and 369 00:21:08.720 --> 00:21:12.680 then we just execute things normally and then we have 370 00:21:12.759 --> 00:21:16.640 some self healing functionality. So it fails on something that 371 00:21:16.680 --> 00:21:19.480 we generated, we're going to try and regenerate those steps. 372 00:21:20.119 --> 00:21:24.640 And what's really nice about that is since Rainforest is 373 00:21:24.720 --> 00:21:30.319 a visual tool, we identify things on the screen based 374 00:21:30.359 --> 00:21:34.920 on screenshots. Right, it's possible for you to make slight 375 00:21:35.039 --> 00:21:38.200 visual changes, and now that image doesn't quite match up, 376 00:21:38.279 --> 00:21:40.359 your test might fail. You don't want to have to 377 00:21:40.359 --> 00:21:43.599 go back in and retake all of their screenshots. But 378 00:21:43.880 --> 00:21:46.799 since it's generated by AI, it could go back follow 379 00:21:46.799 --> 00:21:49.119 the same steps and realize this is what the button 380 00:21:49.200 --> 00:21:52.240 is here. It would be really cool if it could 381 00:21:52.960 --> 00:21:54.839 kind of add that test coverage for you, or like 382 00:21:54.920 --> 00:21:58.279 tell you what you should be testing. We've poked at 383 00:21:58.319 --> 00:22:01.240 that a few times and it's obviously just really dumb 384 00:22:01.279 --> 00:22:03.920 in that aspect and doesn't really give you anything useful. 385 00:22:04.559 --> 00:22:06.720 It's like, yeah, go go test all the things and 386 00:22:06.759 --> 00:22:09.119 make sure things work and it's like cool, yeah, I 387 00:22:09.279 --> 00:22:17.440 knew that, thank you, Yes they get smart. I feel like. 388 00:22:17.519 --> 00:22:20.279 Part of the answer is also the domain you're in. 389 00:22:20.640 --> 00:22:22.640 I know, something we haven't talked about is like really, 390 00:22:22.640 --> 00:22:25.319 at the top of the test pyramid is exploratory testing, 391 00:22:25.640 --> 00:22:30.480 whereas like add your creative human instincts to where bugs 392 00:22:30.480 --> 00:22:33.119 could potentially pop up while you're looking at an interface 393 00:22:33.279 --> 00:22:37.440 or API. And I don't think we're doing anything wrong 394 00:22:37.599 --> 00:22:40.359 in the creation of AI and LM models. It's removing 395 00:22:40.359 --> 00:22:44.160 the creativity from them, and I think that that harms 396 00:22:44.240 --> 00:22:47.319 us here. But there has been one area, especially within 397 00:22:47.680 --> 00:22:53.759 things like protocol creation or SDKs interfaces for the services, 398 00:22:54.359 --> 00:22:57.799 and that's I think the keyword is fuzzing. So trying 399 00:22:57.960 --> 00:23:01.640 an LM, any sort of AI can spam with almost 400 00:23:01.640 --> 00:23:04.920 a more intelligent root for strategy about what sorts of 401 00:23:04.920 --> 00:23:08.920