Speaker 1: So welcome everyone to another episode of Adventures in DevOps.
Speaker 1: Happy new Year, Warren, Happy new year. Thanks for joining
Speaker 1: me today.
Speaker 2: Thank you for having me back. Always a pleasure. Jillian,
Speaker 2: welcome back. How are you good?
Speaker 3: Good? How are you? And Happy New Year?
Speaker 4: I'm everyone.
Speaker 1: I'm pretty excited about today's episode because we have your background, AJ.
Speaker 1: So our guest today is a J Funk and his
Speaker 1: background is just so cool. So you're a software engineer
Speaker 1: for Rainforest, you are a you live in California, you're
Speaker 1: a snowboarder, and you are a previous d one and
Speaker 1: professional baseball player.
Speaker 2: Is that all correct?
Speaker 5: That is correct. I've taken an interesting career path, so yeah,
Speaker 5: definitely lots of stuff to do you here, and I'm
Speaker 5: in the Lake Tahoe area in California around the Porta
Speaker 5: in California and Nevada, So yeah, it's all true.
Speaker 2: What position did you play in baseball?
Speaker 5: I was a pitcher?
Speaker 2: A pitcher, Okay, cool.
Speaker 3: Oh that's what I know what it is. I was
Speaker 3: about to be like, I've told you what we're talking
Speaker 3: about here, but I know what the picture is.
Speaker 2: It's all about the little connections, Jillian.
Speaker 3: It really is.
Speaker 2: So cool.
Speaker 1: A j tell us a little bit about your role
Speaker 1: at Rainforest and what Rainforest does, and I think I'm
Speaker 1: interested to hear what led you to go from pitching
Speaker 1: Major League Baseball to pitching code?
Speaker 5: Yeah for sure. Yeah, so I started writing code when
Speaker 5: when I was a kid, really always as a hobby.
Speaker 5: My plan life was always to play baseball. You know,
Speaker 5: once you become an adult and you realize that's not
Speaker 5: always such a viable career path. I started realizing that
Speaker 5: this thing I do as a hobby I could do
Speaker 5: as a career. So just started kind of dabbling and
Speaker 5: realizing that I actually really enjoyed doing this, and it
Speaker 5: was a big pivot from you know, just athletics all
Speaker 5: the time. So yeah, now I ended up at Rainforest QA.
Speaker 5: I've been here for seven and a half years now.
Speaker 5: Oh wow, Yeah, it's been a long time, and it's
Speaker 5: it's been really fun. I work with really really great people,
Speaker 5: really intelligent experienced engineers, really great culture. We're distributed all
Speaker 5: over the world, so it's fun. We get to go
Speaker 5: meet up with each other every once in a while
Speaker 5: in some random place in the world, and so it's
Speaker 5: it's a really enjoyable place to work. I specialize in
Speaker 5: front and development, So I spent a lot of time
Speaker 5: with the product and design team shaping how our product works.
Speaker 5: And you know, as a quality assurance company, it's really
Speaker 5: important for us to have a really high bar of
Speaker 5: quality and reliability for a product. Obviously, if our app breaks,
Speaker 5: why are you going to trust us to make sure
Speaker 5: that your app doesn't break? It's reasonable, yeah, right, and
Speaker 5: you know, to meet that high bar, we use rainforest
Speaker 5: to test Rainforest. So being able to eat your own
Speaker 5: dog food on a daily basis is a really really
Speaker 5: good position to be in. It allows you to identify
Speaker 5: pain points in the user experience before your users do.
Speaker 5: Hopefully that's not always true, but we do our best,
Speaker 5: and it also means we spend a lot of time
Speaker 5: thinking about the best way to do QA, both from
Speaker 5: a kind of philosophical standpoint and from a practical implementation
Speaker 5: standpoint reality in other words, so this experience obviously helps
Speaker 5: guide our product roadmap, but it's also led me to
Speaker 5: develop a lot of strong opinions on how product and
Speaker 5: engineering team should shape their testing strategies, what kind of
Speaker 5: tools they should you'd be using, and how they can
Speaker 5: ship code quickly and continuously without having a sacrifice quality
Speaker 5: for sure.
Speaker 1: For me, that seems like one of those rabbit holes
Speaker 1: that it's really hard to find the balance on of
Speaker 1: like what's the minimum level of testing that you should
Speaker 1: be doing, and then like what's a what's an adequate
Speaker 1: level to get into where you're actually still getting good
Speaker 1: use for your time, you know, because obviously you can
Speaker 1: throw something at it that tests every possible combination or
Speaker 1: or path through your application that could ever be taken,
Speaker 1: and you're going to reach a point of diminishing returns there.
Speaker 2: So how do you figure out what that right spot is?
Speaker 5: Yeah? Absolutely, the myth of one hundred percent test coverage, right, Yeah, Yeah,
Speaker 5: that we we all strive for. I think they teach
Speaker 5: you very early on that, like one hundred test coverage
Speaker 5: is what we should be doing with every every time
Speaker 5: we push code there, we should make sure everything is covered.
Speaker 5: The reality is that's just not possible, right, What does
Speaker 5: that even mean? Does that mean every line of code
Speaker 5: is covered? Does that mean every possible edge case is covered? Like,
Speaker 5: you're never going to get all of those things. So
Speaker 5: the trick is finding that balance, right, We want to
Speaker 5: make sure that we have confidence in the thing that
Speaker 5: we're shipping that both the thing we're shipping works and
Speaker 5: that we're not breaking anything else. But we also want
Speaker 5: to be able to move quickly. We don't want this
Speaker 5: to get into our way, and so the main ways
Speaker 5: we go about that is really thinking about what your
Speaker 5: testing strategy is, right, What are you testing? What is
Speaker 5: your bar for quality? Because in reality sometimes we're okay
Speaker 5: if some things break, especially when we're in the early
Speaker 5: stages of prototyping something it's a beta feature, etc. And
Speaker 5: what are the layer of your testing, right, so we
Speaker 5: can we can think about our testing strategy as layers,
Speaker 5: with pictured as three layers, but more as a period. Right.
Speaker 5: So your foundation of your testing strategy is your unit tests,
Speaker 5: something we're all familiar with. We're using code to test
Speaker 5: code and small chunks.
Speaker 2: Right.
Speaker 5: What a unit is is totally up to you. We
Speaker 5: can define it as a single function or some class
Speaker 5: component whatever. The top of our pyramid is our end
Speaker 5: to end or our UI tests. That's testing our application
Speaker 5: in a kind of real world scenario. As we go
Speaker 5: up the pyramid, these tests are more comprehensive, we can
Speaker 5: rely on them more, but they're slower, they're more expensive. Right,
Speaker 5: So at the base of our pyramid, we have all
Speaker 5: of these unit tests that we could run constantly. So
Speaker 5: at Rainforest, we run these every single time you push
Speaker 5: code it. Just run them because it's cheap. We don't
Speaker 5: really care if you're lazy like me. I like to
Speaker 5: just push my code up and oh, something broke, I
Speaker 5: didn't notice, because I don't want to run the whole
Speaker 5: test suite all the time on my local machine. But
Speaker 5: as you move up that pyramid from unit tests, the
Speaker 5: middle of that pyramid would be our integration test, so
Speaker 5: basically testing multiple chunks of these units and how they
Speaker 5: interact with each other. That could be maybe one of
Speaker 5: your micro services talking to another micro service or something
Speaker 5: like that. As we get to the top of the
Speaker 5: pyramid and we start running these end to end tests,
Speaker 5: that's where I think strategy becomes much more important because,
Speaker 5: like I mentioned, they're slower, so we actually care about
Speaker 5: when we run these. We need to be more strategic
Speaker 5: about how often we run them. And they are more expensive,
Speaker 5: so we don't want to just blow a bunch of
Speaker 5: a whole bunch of money on them, right, So finding
Speaker 5: the balance between those things is the real trick.
Speaker 4: I like that you pulled out test pyramid and not
Speaker 4: one of the other newer hipster trends of the test
Speaker 4: diamond or a test climb model gone with the tried
Speaker 4: and true test pyramid. I mean, at least that's how
Speaker 4: I've always seen it. And you know, it's also really
Speaker 4: interesting that you bring up the one hundred percent test
Speaker 4: coverage is not possible at least from like anecdotal experience
Speaker 4: for me, I found it's almost like a preto distribution
Speaker 4: and follows the eighty twenty rule where if you wanted
Speaker 4: to have one hundred percent test coverage, it would actually
Speaker 4: require an infinite amount of time.
Speaker 5: Absolutely, yeah, we certainly strive to have all that test coverage,
Speaker 5: but I think the reality of one hundred percent test
Speaker 5: coverage is more along the lines of how you define it.
Speaker 5: Like your user workflows, Right, so the typical examples like
Speaker 5: your log and flow. What are the the main outcomes
Speaker 5: of the log in it's successful, log in, failed, log in?
Speaker 5: Maybe I forgot your password? And do you have test
Speaker 5: coverage and to end test coverage on that flow? If so,
Speaker 5: we could usually consider that cover. Do I have a
Speaker 5: unit test that covers every combination of things that I
Speaker 5: could type into that box. Absolutely not, but having some
Speaker 5: sort of test covered on it to make sure that
Speaker 5: it actually loads in some kind of real world scenario
Speaker 5: gives me much more confidence.
Speaker 2: With a lot of things like logins.
Speaker 1: And there's so many areas these days in developing software.
Speaker 1: You're using SaaS products as a as the mechanism for
Speaker 1: implementing that.
Speaker 2: What's your approach for.
Speaker 1: Dealing with that external dependency? Because you can you can
Speaker 1: mock it, or you can try to stimulate it, or
Speaker 1: you can actually call it.
Speaker 2: What do you what do you think about those?
Speaker 4: I feel like that the job at a me. Well,
Speaker 4: I see you, I see you're coming there.
Speaker 2: Now now I'm trying to Yeah, I got nothing came
Speaker 2: back to you.
Speaker 5: Yeah. So I think what you're asking is when we
Speaker 5: run any kind of tests, take We'll stick with unit
Speaker 5: tests for for the time being. There's a context that
Speaker 5: they run inside of, right, and if we wanted to
Speaker 5: test our app as close as possible to reality, we
Speaker 5: would test it in production on an actual machine with
Speaker 5: an actual human, which some people do. Right. There's obviously
Speaker 5: downsides to that testing and production probably don't have to
Speaker 5: get into why that's not a great idea.
Speaker 3: And the entire video game industry is wrong.
Speaker 5: And yes they are wrong. I mean, I'm not even
Speaker 5: sure that's true anymore.
Speaker 4: Like they don't even release games, right, It's just like
Speaker 4: content that you click download on and you pay for
Speaker 4: and then the game comes later or something like. I
Speaker 4: think that's what the game industry has gone towards.
Speaker 3: That does tend to be how it goes. But I
Speaker 3: still feel like they have the users doing an awful
Speaker 3: lot of acceptance testing in the video game industry and
Speaker 3: I'm like, too cheap for this nonsense. But anyways, I'm
Speaker 3: trying not to be rail the entire conversation today, so
Speaker 3: we can we can skip right on over that.
Speaker 5: No, I totally agree. It's like you do. Even when
Speaker 5: you do get a game, it's an incomplete game that's buggy,
Speaker 5: and you play it for an hour and you know,
Speaker 5: I'm never playing this again. So I'm a late adopter
Speaker 5: when it comes to these things. I wait till the
Speaker 5: Internet stops screaming about it and I started downloading. But yeah,
Speaker 5: when obviously we don't want to test our applications in
Speaker 5: production because we're smarter than that, and we have the
Speaker 5: ability to test these things in different environments. When we
Speaker 5: are running things like unit tests, we're kind of stuck
Speaker 5: inside of this artificial context. Right. If you're just running
Speaker 5: code to test code, it's inside of that specific code environment.
Speaker 5: It's given us inputs and outputs. Even as we go
Speaker 5: up the chain to some things that like to call
Speaker 5: themselves and the end tests which I kind of disagree with,
Speaker 5: which would be things like dombased testing, you're still stuck
Speaker 5: inside some kind of context.
Speaker 1: Right.
Speaker 5: So if the DOM, if you're not familiar, it's the
Speaker 5: Document Object Model, and it's essentially the application in interface
Speaker 5: that we have with the browser. So it's how our
Speaker 5: JavaScript code talks to the browser, how we manipulate things,
Speaker 5: how we read things from the browser. And so the
Speaker 5: important nuance here is that our code interacts with the DOM.
Speaker 5: A human being doesn't interact with the DOM. Right when
Speaker 5: you go click a button, you don't go talk to
Speaker 5: the DOM, interact with the user phase user interface. So
Speaker 5: we want to get our tests as close to actual
Speaker 5: end to end tests as possible, right, A human looking
Speaker 5: at the screen, a human interacting with the screen. And
Speaker 5: if we're not in that production environment, every step we
Speaker 5: take away from that gets us further from reality, right,
Speaker 5: it gives us this false sense of security. Sometimes there's
Speaker 5: a really common example with with DOM based tools, might
Speaker 5: be click this button, did it work right? Well, just
Speaker 5: because you can't interact with that button through the DOM
Speaker 5: doesn't mean your user can actually interact with that button right.
Speaker 5: There might be you know, some kind of overlay over
Speaker 5: my button. The button might be off of the screen.
Speaker 5: But when I asked the DOM, can I click the button,
Speaker 5: it says, yeah, we clicked it. It worked. Ship the
Speaker 5: code and now no one can log into your AP
Speaker 5: because no one can click the button right, And so
Speaker 5: doing ours as much as we can to get to
Speaker 5: that real world scenario, creating testing and staging environments that
Speaker 5: mirror production as much as possible, and loading these things
Speaker 5: into virtual machines with operating systems instead of a headless browser,
Speaker 5: which is, you know, basically a browser with no UI,
Speaker 5: and interacting with it in a way that human doesn't
Speaker 5: interact with it just gets us further away from reality.
Speaker 4: I mean, it's interesting you bring that up, and I'm
Speaker 4: sort of now I'm intrigued if you maybe want to
Speaker 4: roast all what we've been telling our customers. So obviously,
Speaker 4: where where we provide a third party product for our
Speaker 4: customers for a log in and access control, so they have,
Speaker 4: you know, providing them the off needs there, and I
Speaker 4: think the biggest advice that we end up giving them
Speaker 4: is like we are already testing that thing, Like, you know,
Speaker 4: don't focus on this. You're wasting your time duplicating our testing.
Speaker 4: If you felt the need to do that, it's almost
Speaker 4: like you don't trust us with our product, and then
Speaker 4: you probably should question why you're using that solution in
Speaker 4: the first place. If you get to that point, you know,
Speaker 4: that's actually a conversation more than it's a technical solution. However,
Speaker 4: we do find some customers still have a need to
Speaker 4: go a little bit further and the thing that we've done,
Speaker 4: I don't know this is the right answer, but we
Speaker 4: provide a given it is a SAS product, we provide
Speaker 4: a clone of our service as a container that can
Speaker 4: run that it is trimmed down, only has minor features,
Speaker 4: but allows the flows that you're going to test or
Speaker 4: you want to actually verify available without having to go
Speaker 4: through all the complexity that the service actually provides.
Speaker 5: Yeah, I mean, I think that's a good compromise. Right.
Speaker 5: So in this whole strategy, of fending balance between our
Speaker 5: confidence and our velocity and shipping. The reality is our
Speaker 5: testing environs are not going to match our production environments
Speaker 5: all the time, right, and a lot of times we're
Speaker 5: constrained by resources. So I think in a situation like that,
Speaker 5: that totally makes sense. You know, some kind of pared
Speaker 5: down version of your production application. I think the important
Speaker 5: thing is how you're testing it, right. If we are
Speaker 5: able to just kind of like strip down our product
Speaker 5: and test the bare bones version of it, as long
Speaker 5: as we are in an environment, right, they're clicking on
Speaker 5: it through I don't know, a web browser or whatever
Speaker 5: it might be, versus just like you know, running some
Speaker 5: script in the background. I think that's a really good
Speaker 5: balance between those two things. The key here is that
Speaker 5: we're still doing and testing right. I imagine it's you
Speaker 5: know that someone's typing in the box, a button's being clicked,
Speaker 5: there's an HTTP request or whatever it might be to
Speaker 5: an API that reads from a database, and we're checking
Speaker 5: out all of these things actually work together. So yeah,
Speaker 5: I think that's that's a good compromise.
Speaker 4: I think one of the things that actually comes up
Speaker 4: a lot is maybe just on a slight tangent is
Speaker 4: people are so focused on and testing they never stop
Speaker 4: to question should we like for that particular flow, is
Speaker 4: that where the value is for our company? Is it
Speaker 4: really where we should put a lot of resources? And
Speaker 4: do find that those that you're working with or your
Speaker 4: customers may or may not know where the highest value
Speaker 4: testing should be done, And then that's a conversation or
Speaker 4: maybe it's.
Speaker 5: Something that your tool provides. Yeah. Absolutely, And I think
Speaker 5: the trick between the trick for that is doing it
Speaker 5: early right. So, if you have a large application in
Speaker 5: a code base and you have not written any end
Speaker 5: n tests, it's hard to determine where to start, right
Speaker 5: versus when if you start early on, it kind of
Speaker 5: writes itself. Right. The first day is your login, you
Speaker 5: have some login coverage. Determining where the highest value is
Speaker 5: is certainly up to each usually the product team, Right,
Speaker 5: what do we care most about not breaking? And can
Speaker 5: we create some kind of smoke test that spans all
Speaker 5: of these right, So each one of these tests has
Speaker 5: a certain level of granularity to it. A good smoke
Speaker 5: test might be can I log into my application? Can
Speaker 5: I create a thing? Can I delete a thing? And
Speaker 5: things just like generally work. Those initially are your highest
Speaker 5: value tests because I know that my app actually loads
Speaker 5: in reality, right, regardless of what my unit tests say.
Speaker 5: After that, it's usually defined by what those user flows are. Right. So,
Speaker 5: as you're scoping something out with your product team, here's
Speaker 5: this new feature that we're building, it's really important in
Speaker 5: your planning process to include that right right tests for it.
Speaker 5: These are things that we usually as developers, kind of
Speaker 5: bake into our estimates, Right, I have the right unit
Speaker 5: tests for this. At Rainforest, we've shifted more towards baking
Speaker 5: and Rainforest tests for these things. We obviously have unit tests,
Speaker 5: But getting the coverage at the time of implementation or
Speaker 5: the time of release or whatever that might be, is
Speaker 5: usually your best bet to get that. If I have
Speaker 5: a large, large application and I don't have that coverage yet,
Speaker 5: it is certainly a balancing act figuring out what should
Speaker 5: be test first, Right, So I would certainly start with
Speaker 5: those kind of smoke tests, and then your highest used
Speaker 5: features is usually a really good place to start. The
Speaker 5: pitfall that you go into is putting too much nuance
Speaker 5: in a lot of these tests, right, What if they
Speaker 5: click into this and click out of that and then
Speaker 5: open this menu and whatever. Keeping them very coherent and
Speaker 5: legible and kind of focused on the thing that they're
Speaker 5: testing is the important piece of having efficient tests that
Speaker 5: you can maintain over time.
Speaker 4: I saw Will smarting there, and I know he's just
Speaker 4: entered into a new, glorious position at his organization, so
Speaker 4: maybe he has some unique insight that he's interested in
Speaker 4: blessing else with No.
Speaker 1: I was curious because this is an opportunity for me
Speaker 1: to throw in a buzzword that's trending, and so once
Speaker 1: the episode is transcribed, we'll just go viral on that.
Speaker 2: So does AI play a role in.
Speaker 1: Helping figure out that type of user flow? In the
Speaker 1: different like odd places you can end up.
Speaker 5: Good question? That's my knowledge. Yet, you know, as really
Speaker 5: good at some things and really bad at some things,
Speaker 5: and we haven't quite figured out how to make it
Speaker 5: give it enough context to understand how it should go
Speaker 5: about testing your app. Right, we do have some really
Speaker 5: cool AI tools that rain for us. They don't determine
Speaker 5: what your test coverage should be. Rather that's kind of
Speaker 5: left up to you and then it helps you write
Speaker 5: the test. So what we have is entering a prompt,
Speaker 5: you know, say, it could be something pretty generic. Log
Speaker 5: in and Adam add an item to the cart and
Speaker 5: check out something like that, and it will generate your
Speaker 5: reinforce steps for you. So during execution, AI is left
Speaker 5: out of it. Right, It does the initial generation and
Speaker 5: then we just execute things normally and then we have
Speaker 5: some self healing functionality. So it fails on something that
Speaker 5: we generated, we're going to try and regenerate those steps.
Speaker 5: And what's really nice about that is since Rainforest is
Speaker 5: a visual tool, we identify things on the screen based
Speaker 5: on screenshots. Right, it's possible for you to make slight
Speaker 5: visual changes, and now that image doesn't quite match up,
Speaker 5: your test might fail. You don't want to have to
Speaker 5: go back in and retake all of their screenshots. But
Speaker 5: since it's generated by AI, it could go back follow
Speaker 5: the same steps and realize this is what the button
Speaker 5: is here. It would be really cool if it could
Speaker 5: kind of add that test coverage for you, or like
Speaker 5: tell you what you should be testing. We've poked at
Speaker 5: that a few times and it's obviously just really dumb
Speaker 5: in that aspect and doesn't really give you anything useful.
Speaker 5: It's like, yeah, go go test all the things and
Speaker 5: make sure things work and it's like cool, yeah, I
Speaker 5: knew that, thank you, Yes they get smart. I feel like.
Speaker 4: Part of the answer is also the domain you're in.
Speaker 4: I know, something we haven't talked about is like really,
Speaker 4: at the top of the test pyramid is exploratory testing,
Speaker 4: whereas like add your creative human instincts to where bugs
Speaker 4: could potentially pop up while you're looking at an interface
Speaker 4: or API. And I don't think we're doing anything wrong
Speaker 4: in the creation of AI and LM models. It's removing
Speaker 4: the creativity from them, and I think that that harms
Speaker 4: us here. But there has been one area, especially within
Speaker 4: things like protocol creation or SDKs interfaces for the services,
Speaker 4: and that's I think the keyword is fuzzing. So trying
Speaker 4: an LM, any sort of AI can spam with almost
Speaker 4: a more intelligent root for strategy about what sorts of
Speaker 4: inputs tend to break your interface or your service or
Speaker 4: your product and then use that as a potential test
Speaker 4: that you can commit longer term. And again it's not
Speaker 4: for everything, like I don't think it really works so
Speaker 4: much in a UI world, but definitely depending on what
Speaker 4: your service or interface is doing. Stuff in the crypto
Speaker 4: space cryptography, not blockchain.
Speaker 1: Just to be clear, I feel like that was a
Speaker 1: dig there, Warren, What are you trying to say?
Speaker 4: You know, it's not the sort of thing I want
Speaker 4: to bring up on an episode.
Speaker 3: Well, you don't want a record.
Speaker 4: Huh yeah, definitely not on a record. We do cryptography
Speaker 4: because we're really into security and deep there, and we
Speaker 4: do not building our own crypto, but hi, we're very
Speaker 4: high users of it. Everything JAWT creation or jot creation,
Speaker 4: every single different kind of algorithm strategy we end up
Speaker 4: utilizing these, and so finding where we're not using libraries
Speaker 4: effectively is certainly an area that we've potentially looked into. Actually,
Speaker 4: according to our company by laws, we're not allowed to
Speaker 4: do anything regarding cryptocurrency, Like it's actually not allowed by
Speaker 4: the country of Switzerland for us to get involved in
Speaker 4: any way. We can't accept payments, we can't pay people
Speaker 4: in crypto. We can't even think about consulting for companies
Speaker 4: that I want to do something crypto related.
Speaker 2: That's discrimination.
Speaker 3: I work at HPC and I'm pretty sure like some
Speaker 3: of the admins will just kind of use like a
Speaker 3: little bit of the compute power from different clusters that
Speaker 3: they have to be running different crypto schemes, but I haven't.
Speaker 3: I haven't like one hundred percent caught anybody. But I'm
Speaker 3: just I'm waiting for the day. I'm waiting for it.
Speaker 3: Not to tell on them, I just want to know
Speaker 3: because I'm super nosy and like I just like knowing
Speaker 3: things like this.
Speaker 2: No, I'm going to tell on as long as they
Speaker 2: cut you in. Yeah.
Speaker 3: It's like when my you know, when my website not
Speaker 3: hacked by that Chinese jewelry store and I was like, guys, like,
Speaker 3: if you if you would just give me a cut,
Speaker 3: this would be fine. It was nice.
Speaker 4: I mean I think that's I think that really is
Speaker 4: expert advice from our resident mL expert here, because.
Speaker 3: Like just making.
Speaker 4: Yeah no, because AS just came out and said that
Speaker 4: the strategy of sharing reservations across customer a WOS accounts,
Speaker 4: Like if you're a consultant that does bundling, for instance,
Speaker 4: reservations or compute reservations, you no longer can pass along
Speaker 4: that savings to the customer. I mean, what are they
Speaker 4: going to do with all this excess capacity now other
Speaker 4: than some good old fashioned bitcoin mining.
Speaker 3: I don't know, I don't know. I mean we could
Speaker 3: be making drugs for autoimmune diseases and cancer, or or
Speaker 3: you could be making some call part cash. I don't know.
Speaker 1: It all.
Speaker 3: Such both it's not in either. Or there's plenty of
Speaker 3: compute power these guys have. You know, they're spinning up
Speaker 3: plenty of a W Westerns. They're not going to notice
Speaker 3: if that last ten is using crypto.
Speaker 1: So when this episode launches and we all get blocked
Speaker 1: from a respective AWS accounts, we can just reflect on
Speaker 1: this moment fondly.
Speaker 4: So, I mean, for the record, AWS is going to
Speaker 4: block you because the ROI on utilizing cloud resources to
Speaker 4: mind crypto is so low that you're pretty much just
Speaker 4: paying AWS. But it is a good indication that there
Speaker 4: is malicious activity happening on your account. So it is
Speaker 4: something that they will for sure investigate, and that I
Speaker 4: think is as much as a tangent on this that
Speaker 4: I want to go down for today.
Speaker 3: Right, Yeah, I think we should talk about the low
Speaker 3: code with Rainforest. I love low code stuff. How did
Speaker 3: how did this come about? And like how does it work?
Speaker 3: I just I want to know all about it.
Speaker 5: Yeah, for sure. So when I first joined Rainforest over
Speaker 5: seven years ago, our model is a bit different. We
Speaker 5: had a bunch of human testers. It was kind of
Speaker 5: the gig uber model of I have something I want
Speaker 5: to test. Here's my test cases. They're all written in
Speaker 5: plain English, and we'll provide a bunch of humans for
Speaker 5: you to go test your application, right, including some exploratory
Speaker 5: stuff like you mentioned go click all over this page
Speaker 5: and try and find problems with it. And that worked
Speaker 5: really well. It was true and the end testing, we
Speaker 5: load your app in a in a virtual machine inside
Speaker 5: of a web browser. They're actually clicking the buttons and
Speaker 5: confirming there's things on the screen. But what we found
Speaker 5: is that humans are inefficient and expensive, as we all know.
Speaker 5: That's why we have automation, right, and so we kind
Speaker 5: of shifted over to automation, but we wanted to do
Speaker 5: something a bit different from what everyone else was doing,
Speaker 5: which is these code based tools uh dom based interactions,
Speaker 5: and instead we built it all on the visual layer.
Speaker 5: So the way it works is you go in, you
Speaker 5: load your app and essentially, just like tix stree shots
Speaker 5: of things, right click on this type into this field.
Speaker 5: I can give it an ai an AI prompt and say,
Speaker 5: you know, log in and check check out in the
Speaker 5: cart and then when you execute things, it loads into
Speaker 5: the same environment. Right, you have your your staging environment.
Speaker 5: Hopefully you have some seed data with login information. You
Speaker 5: can load that all in the rainforest. It goes in
Speaker 5: and runs this whole workflow for you. The output of
Speaker 5: it is a video of the thing being tested. Results
Speaker 5: on each step, things like HTTP logs, jobscript, console logs,
Speaker 5: all the information that you need to actually debug things
Speaker 5: when something instead of it just saying you know, like
Speaker 5: in a unit test and it's like failure like one
Speaker 5: does not equal to And so by doing things at
Speaker 5: that visual level, it offers a lot of flexibility. The
Speaker 5: first thing is that we're out stuck inside of the browser. Right,
Speaker 5: we do primarily focus on web based testing, but that
Speaker 5: does not mean you're stuck inside of the browser. It
Speaker 5: means you can do things like install a Chrome extension. Right,
Speaker 5: open another tab in your browser, install a Chrome extension,
Speaker 5: interact with that extension. Because while you're still inside the browser,
Speaker 5: you're outside of the scope of that web page where
Speaker 5: you usually are interacting just through the domin you can
Speaker 5: you know, install some type of desktop application and test
Speaker 5: it through there. Because since we're working at the visual layer,
Speaker 5: it doesn't care what you're testing. It doesn't care what
Speaker 5: your tech stack is. It just cares that it loads
Speaker 5: in the machine. And it also offers a lot of
Speaker 5: like more flexible and robust in avoiding flakiness and brittleness
Speaker 5: to small changes. We have fallback methods. As much as
Speaker 5: I've been kind of hammering that testing with the DOM
Speaker 5: is not a great idea, we do offer DOM fallbacks
Speaker 5: because sometimes it makes sense. Sometimes I don't care about
Speaker 5: the visual appearance of the button, and all I care
Speaker 5: about is that there's a button there. Right. In reality,
Speaker 5: there are variables that we can't control. Right. A very
Speaker 5: common scenario is my marketing team is running experiments. Every
Speaker 5: time I load this page, the button says something different,
Speaker 5: it looks different, and so we don't want to tie
Speaker 5: the visual appearance to the past fail result of this test.
Speaker 5: So I'll use something else. I'll use a DOM selector.
Speaker 5: We also have like AI search. You could say something
Speaker 5: like the button, the log and button at the bottom
Speaker 5: of the page. And so the important point here is
Speaker 5: you don't write any code whatsoever. We have an intuitive
Speaker 5: UI that you do all of this through, which means
Speaker 5: you don't need skilled engineers to do it right. A
Speaker 5: lot of teams have QA engineers that their job is
Speaker 5: to just write tests all the time. Other teams, you
Speaker 5: are read the engineer is responsible for writing these tests,
Speaker 5: but they need very specific domain knowledge, right. I need
Speaker 5: to know about the thing that I'm testing, like from
Speaker 5: a product standpoint, what does this thing do? I need
Speaker 5: to understand the code. I need to know how to
Speaker 5: write these tests with a no code solution. Anybody can
Speaker 5: do this, right. It's up to your team who owns quality.
Speaker 5: Who owns these tests. For us, it is usually the
Speaker 5: engineer that is shipping the code we write the reinforest
Speaker 5: tests with it, or our product and design team owns
Speaker 5: it because, like I was saying, they're very tied to
Speaker 5: our user workflows, right, They're like, this is how we
Speaker 5: design this thing. Engineers are going to build it, and
Speaker 5: then any of us that have knowledge about how this
Speaker 5: userflow is supposed to work can own this test. So
Speaker 5: it makes it much easier to both write and maintain
Speaker 5: your tests over time. And then you know, if that
Speaker 5: person leaves your company and has all of that domain
Speaker 5: knowledge and how it works, you just need someone who
Speaker 5: knows how the app works and they can update your
Speaker 5: test suite.
Speaker 4: Now, I feel like one of the biggest mistakes that
Speaker 4: I keep seeing over the course of my career is
Speaker 4: as companies grow, they tend to have more, allegedly more software,
Speaker 4: more code, which may or may not end up in
Speaker 4: a giant ball of mud mass or even an extensive
Speaker 4: number of quote unquote micro services that communicate and really
Speaker 4: depend on each other. And there was always this challenge
Speaker 4: by someone who wanted to have a test that required
Speaker 4: somehow interacting with all of these components, and they never
Speaker 4: really could understand that one of the whole point so
Speaker 4: micro services was to isolate testing. But I think we
Speaker 4: live in the reality, which is there are some companies
Speaker 4: that do have a giant ball of mud that have
Speaker 4: thousands of binaries that have to be installed on running servers.
Speaker 4: Is there a strategy that I don't I don't mean to,
Speaker 4: you know, pick on your company.
Speaker 5: I don't know.
Speaker 4: If I don't think there is a strategy. I think
Speaker 4: the strategy is right micro services. But I can imagine that.
Speaker 4: You know, as a SaaS company, the last thing we
Speaker 4: want to tell our customers is, yeah, have you tried
Speaker 4: not having that problem?
Speaker 3: Is there a package manager that tends to do this
Speaker 3: solution that I see that what is everybody's favorite.
Speaker 4: Yeah, distributed monolith always works. Publish all your binaries that
Speaker 4: remotely depend on each other to a third party solution
Speaker 4: and then pull those out at run time always works.
Speaker 4: Best solution ever. Maybe Aja, you have some insight here
Speaker 4: on either something that works or something that works with
Speaker 4: Rainforest QA to deal with those situations, or maybe you
Speaker 4: just you know, it's not something that is handled today.
Speaker 5: Yeah. Sure, we do have a bunch of different micro
Speaker 5: services running, and I think I'm going to refer back
Speaker 5: to the testing pyramid. Right, is we test each one
Speaker 5: of those micro services in isolation. Absolutely, Maybe we test
Speaker 5: as we go up to the next layer of our
Speaker 5: integration test, we test some interactions between them, right, the
Speaker 5: kind of core you know, handshake interactions, whatever is the
Speaker 5: main functionality of these two microservices talking to each other.
Speaker 5: Maybe we have some tests there, But at the end
Speaker 5: of the day, these end to end tests are comprehensive. Right,
Speaker 5: If anything in that micro service architecture is failing, presumably
Speaker 5: my test is going to fail. And at the end
Speaker 5: of the day, all we really care about in theory
Speaker 5: is what the user gets when they're interacting with it.
Speaker 5: So if I'm just clicking a button, maybe there's a
Speaker 5: thousand micro services that are involved in this, and maybe
Speaker 5: I'm not directly testing each one of those, but by
Speaker 5: implementing it as an end to end test, I am
Speaker 5: very confident that they're all working because my test passed.
Speaker 5: And so being smart about how you implement each one
Speaker 5: of those layers in an efficient way. Right, lots of
Speaker 5: unit tests on each micro service and then this overarching
Speaker 5: test that just make sure everything is working together is
Speaker 5: usually a way to go about this.
Speaker 4: I mean, maybe it's a technical implementation question, like where
Speaker 4: is the environment running that the rainforest tests are actually
Speaker 4: executing us? Is this some sort of binary or CLI
Speaker 4: that's run on the client side, or are they sharing
Speaker 4: with you a set of micro services with deployment instruction
Speaker 4: so that you can run them within your own infrastructure.
Speaker 5: Sure, Usually our requirement is that you need to be
Speaker 5: able to access it via a web URL. So the
Speaker 5: kind of standard way that a Rainforests is run is
Speaker 5: we provide you a VM, and that VM has a
Speaker 5: browser on it, so I can specify Chrome on Windows
Speaker 5: eleven and the first step of your test is going
Speaker 5: to be a navigation navigates this URL. This is where
Speaker 5: my web app lives. There are some different use cases
Speaker 5: where you can absolutely go download a Binari and install
Speaker 5: it and do whatever you want with it or out
Speaker 5: of the box functionality is to primarily test those web apps,
Speaker 5: so that's where we focus. But you certainly have the
Speaker 5: flexibility to do whatever you want with those vms.
Speaker 4: No, I mean, I think I think that approaches genius. Basically,
Speaker 4: it's out of scope for setting up the environment unless
Speaker 4: you want it to be in scope of which you know.
Speaker 4: Then it's a it's a virtual machine. Go to town
Speaker 4: on how they want to deal with it there.
Speaker 5: Right, and by kind of forcing people to give it
Speaker 5: a public URL, we're nudging them towards good practices. Right
Speaker 5: for sure is set up a staging environment a QA
Speaker 5: environment and make it mirror production as much as possible,
Speaker 5: which includes being able to navigate to it in a URL.
Speaker 5: And these are small things that we see with some
Speaker 5: of our new clients are like, well, I don't have
Speaker 5: a staging environment, and sure, I guess you could load
Speaker 5: your production environment in there, but let's show you how
Speaker 5: to test this properly and not shoot yourself on the foot.
Speaker 4: I'm like, I love that you're saying that. The number
Speaker 4: one feedback I've always seen here is we can't expose
Speaker 4: our non production environment publicly, Like people can't know what
Speaker 4: we're currently working on. They will use that information maliciously
Speaker 4: against our company in some way.
Speaker 5: Yeah, like what are they going to do with it?
Speaker 3: Though?
Speaker 5: You know, like if I have seen some interesting mistakes,
Speaker 5: like you know, maybe we're cloning our production database and
Speaker 5: not sanitizing sensitive information from it or something like that. Yes,
Speaker 5: absolutely you're doing some bad things, but there are certainly
Speaker 5: ways to do this. And I'm of the opinion who
Speaker 5: cares if people are in your testing environment, Like worst
Speaker 5: case they blow up your testing environment and whatever.
Speaker 1: And in that case you figured out how they could
Speaker 1: blow up your production environment.
Speaker 5: Without losing pride exactly.
Speaker 2: Yeah, I think there's.
Speaker 1: That's, you know, like part of the undocumented learning curve
Speaker 1: of working in this industry, you know, because people who
Speaker 1: are early in their careers think things like, oh, I
Speaker 1: shouldn't expose staging until you know, they learned that that's
Speaker 1: actually probably a good thing.
Speaker 2: But like, nowhere in any.
Speaker 1: Computer science or course or or boot camp or anything
Speaker 1: do they cover these kinds of things, and so I
Speaker 1: think that's actually like a really valuable add on service
Speaker 1: that you know, you get from rain forest, or that
Speaker 1: you get from working with people who are more experienced,
Speaker 1: is just like learning that tribal knowledge that's going to
Speaker 1: help you out later in your career, so you don't
Speaker 1: have to reinvent the wheel and solve problems that we
Speaker 1: actually solved thirty years ago.
Speaker 2: I mean, that's will.
Speaker 4: I mean we've unfortunately had to append our documentation with
Speaker 4: like here explicitly are the sensitive pieces of data that
Speaker 4: are relevant to our third party application. This is sensitive.
Speaker 4: This is sensitive, Like this is not sensitive. This is
Speaker 4: like the application I need not sensitive. Like, do not
Speaker 4: try to encrypt this, Do not try to secure it,
Speaker 4: because people will try like how do I do this?
Speaker 4: I'm like, you can't stop it, Like this has to
Speaker 4: be public on your website, in your application. People have
Speaker 4: to be able to see it. You're not going to
Speaker 4: be able to get around that. And I feel like
Speaker 4: it's more than just experience. I feel like there's a
Speaker 4: whole level of pragmatism there, like weighing the cost versus
Speaker 4: the reward of actually trying to sanitize that piece of information.
Speaker 4: And having a third party testing service, as you mentioned,
Speaker 4: just reinforces that in a way, like you are going
Speaker 4: to have to expose that to be tested, must show
Speaker 4: that it's not actually sensitive information.
Speaker 5: Yeah, definitely. To me, it just reminds me of like
Speaker 5: the myth of the one hundred percent test coverage, right,
Speaker 5: It's like, can we one hundred percent encrypt everything? Absolutely not?
Speaker 5: Like people, your end users need to see this information.
Speaker 5: I see. I've seen some interesting attempts to kind of
Speaker 5: obfuscate those things, Like I've seen some libraries that prevent
Speaker 5: you from opening the davascript console, for example, and it's like,
Speaker 5: what are you hiding in there? Maybe maybe you should
Speaker 5: just not put sensitive things.
Speaker 2: Here's here's a wild thought. I just don't do that.
Speaker 3: Credentials like encoded in the HTML on your page.
Speaker 5: Maybe.
Speaker 4: I mean, I can't believe you two are joking about this, honestly,
Speaker 4: like one of the most common attacks against So I
Speaker 4: don't do.
Speaker 3: You why so I can joke about all this because
Speaker 3: none of them It's okay. Tone You'll have.
Speaker 4: Jillian, You'll have plenty of opportunity to get your models
Speaker 4: encoded with AWS access keys in secrets, and then you
Speaker 4: just ask the model, Hey, can I have an access
Speaker 4: key and secret that are valid that work for any
Speaker 4: aw's account.
Speaker 3: I did actually accidentally push my AWS credentials to get
Speaker 3: huged once, and like the amounted emails that I got
Speaker 3: from AWS was just like it was unreal. It was
Speaker 3: very It was a very bad day for me. It
Speaker 3: was a very very bad day. So I've done other
Speaker 3: stupid things, but I don't do the same stupid things,
Speaker 3: so I can sit here and be very smug about this.
Speaker 5: Like.
Speaker 1: That is like that is I've always I've often wondered
Speaker 1: about that, Like the speed that AWS and other malicious
Speaker 1: people can identify that you committed an AWS access key
Speaker 1: to a GitHub repo.
Speaker 3: It was instant. It was right then, because as soon
Speaker 3: as I did it, I was like, oh no, and
Speaker 3: tried to you know, and like try to like make
Speaker 3: the gi hub repil private, and no, it was instant.
Speaker 3: They knew, they knew it was out there.
Speaker 4: Yeah, I mean it's bad. I mean I think I
Speaker 4: saw a bunch of statistics on this that for AWS
Speaker 4: keys and on GitHub, it's about thirty seconds to two
Speaker 4: minutes having been exposed in the repository anywhere in any format,
Speaker 4: So like commit at the beginning of the repository where
Speaker 4: it was there but then got removed, so it's not
Speaker 4: in plain text anymore. You have to go back through
Speaker 4: the get history still about two minutes. Then there's exposure
Speaker 4: on like stack overflow and places like I don't know
Speaker 4: who uses Facebook in connection with their work, but that
Speaker 4: was another place, and then Instagram and read it somewhere
Speaker 4: between two and four or five days, and then there's
Speaker 4: a couple other ones where it's six and more.
Speaker 5: Some of those you have to.
Speaker 4: Thank, like GitHub for like they'll actually discover secrets there.
Speaker 4: So if you provide a third party application that has credentials,
Speaker 4: like at authors, we have our secret keys registered there,
Speaker 4: so if one of our customers exposes keys for our
Speaker 4: service on GitHub, we'll get notified, automatically revoke those keys
Speaker 4: and send them an email telling them that they did
Speaker 4: something that they probably did not want to do, multiple
Speaker 4: times if that's if necessary, because that's happened as well.
Speaker 1: I want to switch topics here real quick, AJ, because
Speaker 1: you've been with Rainforest QA for over seven years now,
Speaker 1: which is unusual in the tech industry, So I'm curious
Speaker 1: about what are the what are the things that you
Speaker 1: look for in a job, that have been fulfilled at Rainforest,
Speaker 1: that keep you there that long.
Speaker 5: Yeah, for sure. First and foremost is the people. Actually,
Speaker 5: when I was interviewing with Rainforest, the last person I
Speaker 5: talked to told me something like I say, at Rainforest
Speaker 5: because of the people, And I was like, okay, that's
Speaker 5: that's a way whateveryone said, And then I quickly drank
Speaker 5: the kool aid, I think, and I found myself saying
Speaker 5: that on interviews and I'm like, I know, this sounds
Speaker 5: like a load of crap, and I think, you know,
Speaker 5: the hiring process is super super important, right, Yeah, both
Speaker 5: finding people that are qualified for the job obviously, but
Speaker 5: are good culture fits. We have a pretty small team,
Speaker 5: so there's nowhere to hide. If you are not doing
Speaker 5: your job where you're not up to par, you're going
Speaker 5: to be exposed pretty quickly, which leads us to have
Speaker 5: a very reliable team. You know, we are distributed globally,
Speaker 5: so there's a lot of handoff. You know, I'm going
Speaker 5: to sleep, you're waking up. Here's what I did, and
Speaker 5: I trust when I wake up that you're just going
Speaker 5: to have this thing done. And if you're not one
Speaker 5: of those people, you're probably not going to fit at Rainforest.
Speaker 5: So really really qualified, experienced, smart, reliable people makes life
Speaker 5: so much easier. And then the other piece of it is,
Speaker 5: you know the mission that we're on, the technology that
Speaker 5: we're building. I think when I was first exposed to it,
Speaker 5: the first time I ship code with Rainforest, it was
Speaker 5: kind of like, Wow, how have I been shipping code
Speaker 5: before this? And the answer was I was probably breaking
Speaker 5: things all of the time and you don't notice until
Speaker 5: user catches it in production two days later or whatever.
Speaker 5: And it's something that I'm really passionate about. I think
Speaker 5: as a front and engineer, we get really caught up
Speaker 5: on the details. Right, There's all these visual layers, there's
Speaker 5: these very specific human interactions. I like building things that
Speaker 5: humans are actually interacting with, and that kind of naturally
Speaker 5: leads you to a quality assurance mindset, Right, I want
Speaker 5: everything to be perfect all of the time. How do
Speaker 5: I ensure this? And so the combination of really great
Speaker 5: people and working on something that I'm actually really passionate
Speaker 5: about and I want to see the rest of the
Speaker 5: world adopt these correct ways of testing things, in my
Speaker 5: opinion of course, just makes it makes it easy to
Speaker 5: work area.
Speaker 2: Right on. That's cool, that's cool.
Speaker 1: Is there you guys obviously do a lot of front
Speaker 1: end type testing. Is there a particular industry or vertical
Speaker 1: that you have got a lot of experience it's in,
Speaker 1: or something that has worked really well that makes a
Speaker 1: really cool story.
Speaker 5: Something I've been involved in that makes a cool story. Oh,
Speaker 5: I don't know if I have a good answer for you. Honestly,
Speaker 5: I've been, like I said, I've been at Rainforest for
Speaker 5: so long. That's all I didn't think about. I guess, right.
Speaker 1: Do you do you attract like a certain like customers
Speaker 1: with like financial apps or with like web based gaming apps,
Speaker 1: or is there a particular vertical that tends to gravitate
Speaker 1: towards your service?
Speaker 5: I think not really, And I think that's one of
Speaker 5: the things that makes it cool is it's a very
Speaker 5: generic testing tool. Right There are some some limitations, but
Speaker 5: in general, if you could load your app on a machine,
Speaker 5: you could probably test it with Rainforest, not caring about
Speaker 5: what the tech stack is those kinds of things. So
Speaker 5: there's a very wide range of users that we have from. Yeah,
Speaker 5: there's some financial some financial companies do some What I
Speaker 5: always finds interesting kind of like testing visually things like
Speaker 5: spreadsheet style apps like their tables and things like that.
Speaker 5: And then we have some really cool like visual tools
Speaker 5: like like drag and drop interfaces where you're building things
Speaker 5: like you know, lego style building, where there's probably, to
Speaker 5: my knowledge, not any other great way to test something
Speaker 5: like that, Like what you say, are all my legos
Speaker 5: on the page? Yeah they are? Are they kind of
Speaker 5: oriented this way? Like? Yeah they are? But how does
Speaker 5: it look right? What does the user see? So the
Speaker 5: real sweet spot is really visual based applications because I
Speaker 5: don't think there's other great solutions for them out there.
Speaker 5: But in general, being a kind of generic visual testing application,
Speaker 5: it really applies to anything right on.
Speaker 1: For a lot of web based front ends, it's all
Speaker 1: no JS based. Do you have a favorite no JS
Speaker 1: type tool? Are you like a React fan or next
Speaker 1: JS or view give.
Speaker 2: A personal preference?
Speaker 5: Yes, I am a Ract fanboy for sure. I started.
Speaker 5: Uh you know, we rewind all the way back to
Speaker 5: like the j Query days and stuff, right, I see that,
Speaker 5: and I have nightmares still we have some we actually
Speaker 5: have some of that floating around and our like like
Speaker 5: our admin applications and stuff where it's like a rails
Speaker 5: back in and they're like, yeah, we got jQuery in there.
Speaker 5: And then my first thought was always like, well, like
Speaker 5: how do you test that jQuery? And the answer is
Speaker 5: we don't. Uh throw a couple of wait for us
Speaker 5: tests that and call it good. And I started. I
Speaker 5: started with Angular back in the day, all right, back
Speaker 5: Angular one anyways was kind of the reverse of React.
Speaker 5: We're like, we're gonna put your JavaScript in your HTML.
Speaker 5: React took the approach ever to put your HTML and
Speaker 5: your JavaScript, you know, just smush it all together. And
Speaker 5: it's come a very very long way, I must say. So. Yeah,
Speaker 5: I find working with React very easy and intuitive, and
Speaker 5: it's very nice that the general JavaScript community has supported
Speaker 5: that and has pushed that forward because especially, you know,
Speaker 5: with with all software and technology, but especially in front
Speaker 5: of development, it's really easy to pick the wrong tool
Speaker 5: long term, Right, I picked this thing, it's great, and
Speaker 5: then we find a better way to do it, and
Speaker 5: they just abandon the project. Right. This is true with
Speaker 5: I mean anything open source, and we've run into this
Speaker 5: a lot of times, right, even with open source testing tools.
Speaker 5: And actually we had a very large Enzyme test suite
Speaker 5: on our React application, and we ran into something like this.
Speaker 5: There was a new way of testing React apps, which
Speaker 5: was the React Testing library, and Enzyme kind of said, yep,
Speaker 5: that's a better way to do it. We're going to
Speaker 5: stop supporting at after I forgot version like REX sixteen
Speaker 5: ORRAC seventeen, like, well we want to upgrade to React seventeen.
Speaker 5: It's like, well none of your enzyme tests work, so yeah, exactly,
Speaker 5: exactly too bad for us. And so now you start
Speaker 5: weighing the options of well, how do we upgrade? Right?
Speaker 5: Do we just say let's not upgrade, which is going
Speaker 5: to bite you really quickly, right, especially at the pace
Speaker 5: all these JavaScript libraries remain updated. I want that new
Speaker 5: shiny thing, I want support for that thing, and I
Speaker 5: don't want to be stuck in the past. The more
Speaker 5: you get stuck in the past, the harder it is
Speaker 5: to catch up with everything else. Right, And so our
Speaker 5: options were basically rewrite all these however many thousand enzyme tests,
Speaker 5: or we could just nuke them all, which is it
Speaker 5: reminds me of like these these memes I see about
Speaker 5: like junior engineers and the intern where they're like they're
Speaker 5: commit messages. I knuked all the tests because they were failing,
Speaker 5: and I kind of make them pass turn true and
Speaker 5: all the tests because you know, that's one of the.
Speaker 3: Past, the only reasonable way to do things.
Speaker 5: Yeah, and it sounds kind of like an overreaction, but
Speaker 5: as we started to kind of think about these testing philosophies,
Speaker 5: we're like, we have end and test coverage on all
Speaker 5: of these things, right, And a lot of the front
Speaker 5: end tests, even though they're unit tests, they they load
Speaker 5: things in a headless browser and we're kind of recreating
Speaker 5: what an end end test does. So we chose to
Speaker 5: keep all of our actual unit tests, all the kind
Speaker 5: of business logic that didn't use enzyme new call the
Speaker 5: enzyme tests, and just lean into our rainforest test because
Speaker 5: we know if the rainforest tests are passing, we don't
Speaker 5: need all of these redundant tests anymore. And instant productivity
Speaker 5: boosts Like I don't have to maintain all of these
Speaker 5: things anymore. I don't have to upgrade them. I could
Speaker 5: just get them out of my way and I can
Speaker 5: upgrade all my dependencies. And because we have really good
Speaker 5: end and testsage, we could do that confidently and know
Speaker 5: that we're not breaking things. So, yeah, choosing dependencies can
Speaker 5: be quite tricky sometimes, especially in the job script world.
Speaker 4: Did you find some places that you still wanted to
Speaker 4: reintroduce some of the React testing library for I don't know,
Speaker 4: component level testing of the UI or have you kept
Speaker 4: with like one hundred percent of the decision to not
Speaker 4: have that layer of testing anymore regarding the UI components
Speaker 4: because you focus on the full picture and the end
Speaker 4: testing for the user flow and also whatever you have
Speaker 4: with the interaction with the back end.
Speaker 5: Yeah, we still have some of it, and we drew
Speaker 5: the line at user interactions. Right. So React has this
Speaker 5: idea of hooks, which are basically just chunks of logic.
Speaker 5: It's just a function that I can use inside of
Speaker 5: a component. We stopped having any React testing library tests
Speaker 5: that were actual user interactions, no clicking on things, and
Speaker 5: instead we used it to test the functionality the logic
Speaker 5: of those hooks. So it's essentially a unit test, but
Speaker 5: it's testing a specific React thing, and it requires the
Speaker 5: testing library to do that. Everything else kind of gets
Speaker 5: hoisted up to the end to end testing level, and
Speaker 5: it's nice to just say, hey, designer, hay, product manager, like,
Speaker 5: go at this test coverage while I'm busy hacking on things,
Speaker 5: and I don't have to worry about this anymore.
Speaker 2: So that's actually a great Jillian.
Speaker 3: Oh, I was just gonna say, I'm so impressed with
Speaker 3: people who can keep up with like the UI and
Speaker 3: JavaScript plan because I've tried. I've tried like a few
Speaker 3: times and it just then everything changed. I was like,
Speaker 3: all right, I'm not doing this anymore, and we go
Speaker 3: I'm gonna go do high performance computing that hasn't changed
Speaker 3: in like thirty years.
Speaker 5: It's gonna be Yeah, you definitely start feeling like Sissyphis.
Speaker 5: You're just pushing that rock up the hill and every
Speaker 5: time you get to the top, someone tells you that
Speaker 5: you're actually on the wrong hill.
Speaker 1: I was gonna say, that seems like a really interesting
Speaker 1: approach that I hadn't thought of when we initially started talking.
Speaker 1: But like you, you can replace having to write a
Speaker 1: lot of your tests in your React app by using
Speaker 1: Rainforest right. By just focusing on what the end user
Speaker 1: experience is and testing for that, you can save yourself
Speaker 1: from having to write a lot of tests in the
Speaker 1: React standard library.
Speaker 4: So that's where the trade off is though, right, because
Speaker 4: these tests then are testing more functionality at once and
Speaker 4: so if there is a problem, you don't necessarily know
Speaker 4: like which line of code is causing the issue or
Speaker 4: which interaction there is, so you know there really is
Speaker 4: Like how valuable is that flow? I think? And that's
Speaker 4: something that, as you pointed out Agent, like you sort
Speaker 4: of have to determine upfront, like where is the value
Speaker 4: of your testing and how do you get the most
Speaker 4: value out of which pieces.
Speaker 5: You're adding and where you're validating an et cetera. And so, yeah,
Speaker 5: I mean.
Speaker 4: In your case, and tis weren't actually providing the right
Speaker 4: value in the first place, so definitely switch them over.
Speaker 5: Yeah, absolutely. And it's kind of a question of redundancy too, right,
Speaker 5: Like is redundancy good? Sometimes like I can be really
Speaker 5: really sure and I can have some extra confidence that
Speaker 5: the things isn't going to break, But most of the
Speaker 5: time it just slows us down. Right. I find that
Speaker 5: the often the best time to add more unit test
Speaker 5: coverage is when something breaks, right, because if my end
Speaker 5: to end tests are all passing, but something's broken, very
Speaker 5: often it's some kind of edge case, right, It's some
Speaker 5: either some weird user behavior, some weird input, some weird
Speaker 5: sequence of events, and those things are usually better captured
Speaker 5: in a unit test because it's it's easier to kind
Speaker 5: of implement that specific scenario, that specific line of code
Speaker 5: that is the offender here versus creating you know, a
Speaker 5: whole new end the end test and just cover some
Speaker 5: edge case. Those tests are going to just get longer
Speaker 5: and longer and just be kind of confusing. Honestly, it's like, well,
Speaker 5: why am I just like clicking in all the these
Speaker 5: random spots doing these things trying to cover these edge cases?
Speaker 5: Like just reread a unit test for it and call good.
Speaker 3: I really like the emphasis on, you know, testing for
Speaker 3: business logic and just in general having not everything controlled
Speaker 3: by the engineers, because I find for myself, you know,
Speaker 3: like I'll write something and then I'll hand it off
Speaker 3: to a user and then they immediately start using it
Speaker 3: in some way that I didn't even think of, and then,
Speaker 3: you know, and then we do like a couple of
Speaker 3: rounds of this. So being able to cut back on
Speaker 3: that person who writes thing, who does not actually use
Speaker 3: the thing, and then just immediately being able to push
Speaker 3: it off to an end user.
Speaker 5: Yeah, absolutely, Yeah, And things like testing and staging environments
Speaker 5: are great for this. We push code of those environments
Speaker 5: all the time, give them to a PM and say
Speaker 5: go run and try and try and break this thing, right.
Speaker 5: Don't always want to do that in production, right, like
Speaker 5: if the thing's not fully baked, I don't want to
Speaker 5: corrupt something in the database or whatever. And so having
Speaker 5: places to push that and have people early in the
Speaker 5: process on this, finding the major bugs, the minor bugs,
Speaker 5: the stylistic bugs is super super valuable then having one
Speaker 5: of your users find it later.
Speaker 2: Right on. So you live in Tahoe. Do you get
Speaker 2: outdoors a lot?
Speaker 5: I do. I live here with my wife and my dog.
Speaker 5: He's a lab husky mix, so he kind of thrives
Speaker 5: in the summer, thrives in the winter. There's lots of
Speaker 5: snow out right now, and so we're outside pretty much
Speaker 5: every day, you know, snowboarding, hiking, kayaking and all that
Speaker 5: kind of stuff.
Speaker 2: All Right on, how long have you lived in the
Speaker 2: Tahoe area.
Speaker 5: I've been here for about five years now. I grew
Speaker 5: up in the San Francisco Bay area and I was
Speaker 5: part of the great COVID migration out here. We always
Speaker 5: wanted to get here eventually, and I was lucky. I
Speaker 5: was still working at rainforest at the time and already remote.
Speaker 5: So the transfer up here to from remote near an
Speaker 5: office to remote it actually doesn't matter how far you
Speaker 5: are from the office, was very easy. And we're also
Speaker 5: real fortunate that we were not the only ones doing
Speaker 5: this migration. So we've made lots of friends that were like, yeah,
Speaker 5: we lived down the street from you in the city
Speaker 5: and we all live here now. So it's a very
Speaker 5: different life, but we love it and I don't think
Speaker 5: we're leaving.
Speaker 2: Ah, that's cool. Yeah, right on, Tahoe is a beautiful area.
Speaker 5: Yeah, it really is.
Speaker 2: Cool. All right. Should we move on to some picks
Speaker 2: before we do any final thoughts on.
Speaker 1: QA rainforest tips guidance that you want to leave us
Speaker 1: with aj.
Speaker 5: I think just recapping is finding the balance between confidence
Speaker 5: and velocity, right, everybody needs to set their own bar
Speaker 5: for quality, like how what is my ratio between confidence
Speaker 5: and velocity? Determining that for yourself is the most important
Speaker 5: thing here and keeping in mind that not only is
Speaker 5: it a lot, but a lot of times it's the
Speaker 5: sanity of your engineers, right, Like, we don't want to
Speaker 5: spend all of our time writing tests, So finding that
Speaker 5: balance and doing things in an efficient way is the
Speaker 5: key to success.
Speaker 1: Right on. And I think that's very use case specific too,
Speaker 1: you know, because the right answer for a financial app
Speaker 1: is going to be very different than the right answer
Speaker 1: for like.
Speaker 2: A social media app.
Speaker 1: Absolutely cool, All right, Jillian calling you out first? What'd
Speaker 1: you bring for a pick today?
Speaker 3: I am going to pick Drive by Dave Kellett. It
Speaker 3: is a sci fi graphic novel and I think it's
Speaker 3: on it's I think it's like releasing the fourth one
Speaker 3: this summer. But it's so good and it's so nice
Speaker 3: and wholesome, which is very nice because like, I really
Speaker 3: like sci fi, but I don't really like violence or
Speaker 3: gore or you know, ikey fluids or I don't like
Speaker 3: any of that. Okay, I don't like any of it.
Speaker 3: And this is just so wholesome and adorable and the
Speaker 3: main character is very cute. So that's it. That's the pick.
Speaker 3: I've got a whole bunch of copies for Christmas and
Speaker 3: I'm like making people read them and I'm gonna have
Speaker 3: like a little little intigraphic novel cult going on soon enough.
Speaker 2: Great, all right, Warren, would you bring? Yeah?
Speaker 4: So I just got back from a long hiatus being
Speaker 4: away from the show I was on vacation, and so
Speaker 4: I think this pick is a really accurate, very short book.
Speaker 4: I highly recommend Dowdishing by Laozza, which is the founder
Speaker 4: of Taoism spelt Taoism in case in case. You've seen
Speaker 4: it written but never pronounced before. And there's just so
Speaker 4: much good stuff that is in the book that can
Speaker 4: be applied to everyday life, working environment, etc. It's incredibly short.
Speaker 4: There's only like one hundred and eight principles or so,
Speaker 4: and it starts off great with the dow that can
Speaker 4: be told. Is not the eternal tao Like, you can't
Speaker 4: write down the whole truth. There is something that's never said.
Speaker 4: It's impossible to convey everything. And I know it sounds
Speaker 4: so philosophical, you know, to go down this path, but
Speaker 4: I feel like going through these really helps to put
Speaker 4: into perspective thinking outside the box with solving certain problems
Speaker 4: or interactions or the communication we have every day.
Speaker 2: Highlight right on, cool heja, what you got for us?
Speaker 5: Yeah. My reading and listening choices are kind of all
Speaker 5: over the map. But I did have an interesting one recently.
Speaker 5: It was called The Light Eaters. It's about plants and
Speaker 5: specifically this idea of plant intelligence. So obviously intelligence is
Speaker 5: a loaded word. They're not intelligent like you and I
Speaker 5: either not debating QA strategies and things like that, but
Speaker 5: they do have a lot of intelligent like behavior. You know,
Speaker 5: they communicate, they recognize their kin, they hear sounds, they
Speaker 5: transform themselves based on the visual appearance of the environment
Speaker 5: around them, And so I found it really interesting and
Speaker 5: gave me a lot to think about, especially you know,
Speaker 5: when I'm out in nature with the with the wife
Speaker 5: and dog, just kind of staring at trees and stuff.
Speaker 4: Check out that are intelligent for sure, one hundred percent
Speaker 4: totally with you. There's a there's a good one. If
Speaker 4: you are out and there's plants or grass being cut
Speaker 4: and you notice the smell of you know, freshly cut grass.
Speaker 5: What is that?
Speaker 4: It's a fear intently, a fear pheromone that's been sent
Speaker 4: off to warn other grass that that there is danger around.
Speaker 4: Like that is the sign of intelligent life.
Speaker 5: Yeah, for sure. There's lots of super interesting examples in
Speaker 5: this book, and just like plants acting like animals essentially,
Speaker 5: and it's it's kind of mind blowing experience.
Speaker 1: I read a book recently and I can't remember which
Speaker 1: one it was, but I've been studying mushrooms a lot lately,
Speaker 1: and this book showed where mushrooms actually act as a
Speaker 1: communication agent for trees in the forest, and so like
Speaker 1: a specific you know, set of insects can start attacking
Speaker 1: trees on one end of the forest, and then the mushroom,
Speaker 1: because it's the mycelium that grows underneath the entire forest floor,
Speaker 1: will relay that information to the other trees in the forest.
Speaker 1: And so by the time the insects work their way
Speaker 1: down to those trees, that those trees are producing a
Speaker 1: sind or a pheromone that actually repels the insects by
Speaker 1: the time they get there.
Speaker 2: And I thought that was super cool.
Speaker 3: That is cool. I'm in on a like a mushroom
Speaker 3: and foraging Facebook group and everybody just like takes pictures
Speaker 3: of fun mushrooms that they find when when they're out
Speaker 3: and about, and it's it's just such a nice little
Speaker 3: group because it's so chill. That's it. There's like, there's
Speaker 3: no drama, there's no nothing, it's just look at this mushroom.
Speaker 1: I found there's an app called Naturalist that I use
Speaker 1: for that you can take a picture of not just mushrooms,
Speaker 1: but anything you find that you can't identify, and then
Speaker 1: upload it to I Naturalists and it will it will
Speaker 1: try to auto detect what it is for you, but
Speaker 1: then other people will come in and confirm or tell
Speaker 1: you what that actually is.
Speaker 2: That was pretty cool.
Speaker 3: I used to do that a lot as a kid.
Speaker 3: I'd have like the field guides and go out with
Speaker 3: my field guide and try to like identify all the plants.
Speaker 3: But now we have an app for that.
Speaker 1: There's an app for that always will what's your what's
Speaker 1: your pick? My pick is There's a series on Netflix
Speaker 1: called Kunk on Earth and I thought my sense of
Speaker 1: humor was like really really dry, but this lady, she's amazing,
Speaker 1: takes it.
Speaker 2: She takes it to a whole new level.
Speaker 1: This series is just hilarious because she sits down it's
Speaker 1: you know, like a the History of Earth basically, but
Speaker 1: she'll sit down with legitimate world renow experts in their
Speaker 1: field and ask them the most off the wall questions.
Speaker 1: And that, to me, thet was the highlight of the series.
Speaker 1: Is just the looks on their faces when she would
Speaker 1: ask them these questions that had absolutely nothing to do
Speaker 1: with what they were an expert in, but super entertaining series.
Speaker 1: Definitely ten out of ten stars. Kunk on Earth on Netflix,
Speaker 1: and you know, there's actually two other things. There's Kunk on.
Speaker 4: Britain I think, and then there's like one on Christmas
Speaker 4: and Shakespeare, so you too.
Speaker 5: Oh sweet.
Speaker 2: I will have to check those out because I love
Speaker 2: her sense of humor.
Speaker 3: I've never heard that's fun.
Speaker 1: Yeah, it's very very accurate.
Speaker 2: All right.
Speaker 1: That brings us to the end of the episode. Thank
Speaker 1: you everyone for listening. Jillian Warren, thank you for joining
Speaker 1: me hosting the show. And aj thanks for coming on
Speaker 1: the show man. It's been a pleasure talking to you.
Speaker 5: Thanks so much for having me.
Speaker 1: It was a lot of fun right on. Glad to
Speaker 1: hear that, and I will see everyone next week.
Speaker 5: M