Speaker 1: So welcome everyone to another episode of Adventures in DevOps.
Happy new Year, Warren, Happy new year. Thanks for joining
me today.
Speaker 2: Thank you for having me back. Always a pleasure. Jillian,
welcome back. How are you good?
Speaker 3: Good? How are you? And Happy New Year?
Speaker 4: I'm everyone.
Speaker 1: I'm pretty excited about today's episode because we have your background, AJ.
So our guest today is a J Funk and his
background is just so cool. So you're a software engineer
for Rainforest, you are a you live in California, you're
a snowboarder, and you are a previous d one and
professional baseball player.
Speaker 2: Is that all correct?
Speaker 5: That is correct. I've taken an interesting career path, so yeah,
definitely lots of stuff to do you here, and I'm
in the Lake Tahoe area in California around the Porta
in California and Nevada, So yeah, it's all true.
Speaker 2: What position did you play in baseball?
Speaker 5: I was a pitcher?
Speaker 2: A pitcher, Okay, cool.
Speaker 3: Oh that's what I know what it is. I was
about to be like, I've told you what we're talking
about here, but I know what the picture is.
Speaker 2: It's all about the little connections, Jillian.
Speaker 3: It really is.
Speaker 2: So cool.
Speaker 1: A j tell us a little bit about your role
at Rainforest and what Rainforest does, and I think I'm
interested to hear what led you to go from pitching
Major League Baseball to pitching code?
Speaker 5: Yeah for sure. Yeah, so I started writing code when
when I was a kid, really always as a hobby.
My plan life was always to play baseball. You know,
once you become an adult and you realize that's not
always such a viable career path. I started realizing that
this thing I do as a hobby I could do
as a career. So just started kind of dabbling and
realizing that I actually really enjoyed doing this, and it
was a big pivot from you know, just athletics all
the time. So yeah, now I ended up at Rainforest QA.
I've been here for seven and a half years now.
Oh wow, Yeah, it's been a long time, and it's
it's been really fun. I work with really really great people,
really intelligent experienced engineers, really great culture. We're distributed all
over the world, so it's fun. We get to go
meet up with each other every once in a while
in some random place in the world, and so it's
it's a really enjoyable place to work. I specialize in
front and development, So I spent a lot of time
with the product and design team shaping how our product works.
And you know, as a quality assurance company, it's really
important for us to have a really high bar of
quality and reliability for a product. Obviously, if our app breaks,
why are you going to trust us to make sure
that your app doesn't break? It's reasonable, yeah, right, and
you know, to meet that high bar, we use rainforest
to test Rainforest. So being able to eat your own
dog food on a daily basis is a really really
good position to be in. It allows you to identify
pain points in the user experience before your users do.
Hopefully that's not always true, but we do our best,
and it also means we spend a lot of time
thinking about the best way to do QA, both from
a kind of philosophical standpoint and from a practical implementation
standpoint reality in other words, so this experience obviously helps
guide our product roadmap, but it's also led me to
develop a lot of strong opinions on how product and
engineering team should shape their testing strategies, what kind of
tools they should you'd be using, and how they can
ship code quickly and continuously without having a sacrifice quality
for sure.
Speaker 1: For me, that seems like one of those rabbit holes
that it's really hard to find the balance on of
like what's the minimum level of testing that you should
be doing, and then like what's a what's an adequate
level to get into where you're actually still getting good
use for your time, you know, because obviously you can
throw something at it that tests every possible combination or
or path through your application that could ever be taken,
and you're going to reach a point of diminishing returns there.
Speaker 2: So how do you figure out what that right spot is?
Speaker 5: Yeah? Absolutely, the myth of one hundred percent test coverage, right, Yeah, Yeah,
that we we all strive for. I think they teach
you very early on that, like one hundred test coverage
is what we should be doing with every every time
we push code there, we should make sure everything is covered.
The reality is that's just not possible, right, What does
that even mean? Does that mean every line of code
is covered? Does that mean every possible edge case is covered? Like,
you're never going to get all of those things. So
the trick is finding that balance, right, We want to
make sure that we have confidence in the thing that
we're shipping that both the thing we're shipping works and
that we're not breaking anything else. But we also want
to be able to move quickly. We don't want this
to get into our way, and so the main ways
we go about that is really thinking about what your
testing strategy is, right, What are you testing? What is
your bar for quality? Because in reality sometimes we're okay
if some things break, especially when we're in the early
stages of prototyping something it's a beta feature, etc. And
what are the layer of your testing, right, so we
can we can think about our testing strategy as layers,
with pictured as three layers, but more as a period. Right.
So your foundation of your testing strategy is your unit tests,
something we're all familiar with. We're using code to test
code and small chunks.
Speaker 2: Right.
Speaker 5: What a unit is is totally up to you. We
can define it as a single function or some class
component whatever. The top of our pyramid is our end
to end or our UI tests. That's testing our application
in a kind of real world scenario. As we go
up the pyramid, these tests are more comprehensive, we can
rely on them more, but they're slower, they're more expensive. Right,
So at the base of our pyramid, we have all
of these unit tests that we could run constantly. So
at Rainforest, we run these every single time you push
code it. Just run them because it's cheap. We don't
really care if you're lazy like me. I like to
just push my code up and oh, something broke, I
didn't notice, because I don't want to run the whole
test suite all the time on my local machine. But
as you move up that pyramid from unit tests, the
middle of that pyramid would be our integration test, so
basically testing multiple chunks of these units and how they
interact with each other. That could be maybe one of
your micro services talking to another micro service or something
like that. As we get to the top of the
pyramid and we start running these end to end tests,
that's where I think strategy becomes much more important because,
like I mentioned, they're slower, so we actually care about
when we run these. We need to be more strategic
about how often we run them. And they are more expensive,
so we don't want to just blow a bunch of
a whole bunch of money on them, right, So finding
the balance between those things is the real trick.
Speaker 4: I like that you pulled out test pyramid and not
one of the other newer hipster trends of the test
diamond or a test climb model gone with the tried
and true test pyramid. I mean, at least that's how
I've always seen it. And you know, it's also really
interesting that you bring up the one hundred percent test
coverage is not possible at least from like anecdotal experience
for me, I found it's almost like a preto distribution
and follows the eighty twenty rule where if you wanted
to have one hundred percent test coverage, it would actually
require an infinite amount of time.
Speaker 5: Absolutely, yeah, we certainly strive to have all that test coverage,
but I think the reality of one hundred percent test
coverage is more along the lines of how you define it.
Like your user workflows, Right, so the typical examples like
your log and flow. What are the the main outcomes
of the log in it's successful, log in, failed, log in?
Maybe I forgot your password? And do you have test
coverage and to end test coverage on that flow? If so,
we could usually consider that cover. Do I have a
unit test that covers every combination of things that I
could type into that box. Absolutely not, but having some
sort of test covered on it to make sure that
it actually loads in some kind of real world scenario
gives me much more confidence.
Speaker 2: With a lot of things like logins.
Speaker 1: And there's so many areas these days in developing software.
You're using SaaS products as a as the mechanism for
implementing that.
Speaker 2: What's your approach for.
Speaker 1: Dealing with that external dependency? Because you can you can
mock it, or you can try to stimulate it, or
you can actually call it.
Speaker 2: What do you what do you think about those?
Speaker 4: I feel like that the job at a me. Well,
I see you, I see you're coming there.
Speaker 2: Now now I'm trying to Yeah, I got nothing came
back to you.
Speaker 5: Yeah. So I think what you're asking is when we
run any kind of tests, take We'll stick with unit
tests for for the time being. There's a context that
they run inside of, right, and if we wanted to
test our app as close as possible to reality, we
would test it in production on an actual machine with
an actual human, which some people do. Right. There's obviously
downsides to that testing and production probably don't have to
get into why that's not a great idea.
Speaker 3: And the entire video game industry is wrong.
Speaker 5: And yes they are wrong. I mean, I'm not even
sure that's true anymore.
Speaker 4: Like they don't even release games, right, It's just like
content that you click download on and you pay for
and then the game comes later or something like. I
think that's what the game industry has gone towards.
Speaker 3: That does tend to be how it goes. But I
still feel like they have the users doing an awful
lot of acceptance testing in the video game industry and
I'm like, too cheap for this nonsense. But anyways, I'm
trying not to be rail the entire conversation today, so
we can we can skip right on over that.
Speaker 5: No, I totally agree. It's like you do. Even when
you do get a game, it's an incomplete game that's buggy,
and you play it for an hour and you know,
I'm never playing this again. So I'm a late adopter
when it comes to these things. I wait till the
Internet stops screaming about it and I started downloading. But yeah,
when obviously we don't want to test our applications in
production because we're smarter than that, and we have the
ability to test these things in different environments. When we
are running things like unit tests, we're kind of stuck
inside of this artificial context. Right. If you're just running
code to test code, it's inside of that specific code environment.
It's given us inputs and outputs. Even as we go
up the chain to some things that like to call
themselves and the end tests which I kind of disagree with,
which would be things like dombased testing, you're still stuck
inside some kind of context.
Speaker 1: Right.
Speaker 5: So if the DOM, if you're not familiar, it's the
Document Object Model, and it's essentially the application in interface
that we have with the browser. So it's how our
JavaScript code talks to the browser, how we manipulate things,
how we read things from the browser. And so the
important nuance here is that our code interacts with the DOM.
A human being doesn't interact with the DOM. Right when
you go click a button, you don't go talk to
the DOM, interact with the user phase user interface. So
we want to get our tests as close to actual
end to end tests as possible, right, A human looking
at the screen, a human interacting with the screen. And
if we're not in that production environment, every step we
take away from that gets us further from reality, right,
it gives us this false sense of security. Sometimes there's
a really common example with with DOM based tools, might
be click this button, did it work right? Well, just
because you can't interact with that button through the DOM
doesn't mean your user can actually interact with that button right.
There might be you know, some kind of overlay over
my button. The button might be off of the screen.
But when I asked the DOM, can I click the button,
it says, yeah, we clicked it. It worked. Ship the
code and now no one can log into your AP
because no one can click the button right, And so
doing ours as much as we can to get to
that real world scenario, creating testing and staging environments that
mirror production as much as possible, and loading these things
into virtual machines with operating systems instead of a headless browser,
which is, you know, basically a browser with no UI,
and interacting with it in a way that human doesn't
interact with it just gets us further away from reality.
Speaker 4: I mean, it's interesting you bring that up, and I'm
sort of now I'm intrigued if you maybe want to
roast all what we've been telling our customers. So obviously,
where where we provide a third party product for our
customers for a log in and access control, so they have,
you know, providing them the off needs there, and I
think the biggest advice that we end up giving them
is like we are already testing that thing, Like, you know,
don't focus on this. You're wasting your time duplicating our testing.
If you felt the need to do that, it's almost
like you don't trust us with our product, and then
you probably should question why you're using that solution in
the first place. If you get to that point, you know,
that's actually a conversation more than it's a technical solution. However,
we do find some customers still have a need to
go a little bit further and the thing that we've done,
I don't know this is the right answer, but we
provide a given it is a SAS product, we provide
a clone of our service as a container that can
run that it is trimmed down, only has minor features,
but allows the flows that you're going to test or
you want to actually verify available without having to go
through all the complexity that the service actually provides.
Speaker 5: Yeah, I mean, I think that's a good compromise. Right.
So in this whole strategy, of fending balance between our
confidence and our velocity and shipping. The reality is our
testing environs are not going to match our production environments
all the time, right, and a lot of times we're
constrained by resources. So I think in a situation like that,
that totally makes sense. You know, some kind of pared
down version of your production application. I think the important
thing is how you're testing it, right. If we are
able to just kind of like strip down our product
and test the bare bones version of it, as long
as we are in an environment, right, they're clicking on
it through I don't know, a web browser or whatever
it might be, versus just like you know, running some
script in the background. I think that's a really good
balance between those two things. The key here is that
we're still doing and testing right. I imagine it's you
know that someone's typing in the box, a button's being clicked,
there's an HTTP request or whatever it might be to
an API that reads from a database, and we're checking
out all of these things actually work together. So yeah,
I think that's that's a good compromise.
Speaker 4: I think one of the things that actually comes up
a lot is maybe just on a slight tangent is
people are so focused on and testing they never stop
to question should we like for that particular flow, is
that where the value is for our company? Is it
really where we should put a lot of resources? And
do find that those that you're working with or your
customers may or may not know where the highest value
testing should be done, And then that's a conversation or
maybe it's.
Speaker 5: Something that your tool provides. Yeah. Absolutely, And I think
the trick between the trick for that is doing it
early right. So, if you have a large application in
a code base and you have not written any end
n tests, it's hard to determine where to start, right
versus when if you start early on, it kind of
writes itself. Right. The first day is your login, you
have some login coverage. Determining where the highest value is
is certainly up to each usually the product team, Right,
what do we care most about not breaking? And can
we create some kind of smoke test that spans all
of these right, So each one of these tests has
a certain level of granularity to it. A good smoke
test might be can I log into my application? Can
I create a thing? Can I delete a thing? And
things just like generally work. Those initially are your highest
value tests because I know that my app actually loads
in reality, right, regardless of what my unit tests say.
After that, it's usually defined by what those user flows are. Right. So,
as you're scoping something out with your product team, here's
this new feature that we're building, it's really important in
your planning process to include that right right tests for it.
These are things that we usually as developers, kind of
bake into our estimates, Right, I have the right unit
tests for this. At Rainforest, we've shifted more towards baking
and Rainforest tests for these things. We obviously have unit tests,
But getting the coverage at the time of implementation or
the time of release or whatever that might be, is
usually your best bet to get that. If I have
a large, large application and I don't have that coverage yet,
it is certainly a balancing act figuring out what should
be test first, Right, So I would certainly start with
those kind of smoke tests, and then your highest used
features is usually a really good place to start. The
pitfall that you go into is putting too much nuance
in a lot of these tests, right, What if they
click into this and click out of that and then
open this menu and whatever. Keeping them very coherent and
legible and kind of focused on the thing that they're
testing is the important piece of having efficient tests that
you can maintain over time.
Speaker 4: I saw Will smarting there, and I know he's just
entered into a new, glorious position at his organization, so
maybe he has some unique insight that he's interested in
blessing else with No.
Speaker 1: I was curious because this is an opportunity for me
to throw in a buzzword that's trending, and so once
the episode is transcribed, we'll just go viral on that.
Speaker 2: So does AI play a role in.
Speaker 1: Helping figure out that type of user flow? In the
different like odd places you can end up.
Speaker 5: Good question? That's my knowledge. Yet, you know, as really
good at some things and really bad at some things,
and we haven't quite figured out how to make it
give it enough context to understand how it should go
about testing your app. Right, we do have some really
cool AI tools that rain for us. They don't determine
what your test coverage should be. Rather that's kind of
left up to you and then it helps you write
the test. So what we have is entering a prompt,
you know, say, it could be something pretty generic. Log
in and Adam add an item to the cart and
check out something like that, and it will generate your
reinforce steps for you. So during execution, AI is left
out of it. Right, It does the initial generation and
then we just execute things normally and then we have
some self healing functionality. So it fails on something that
we generated, we're going to try and regenerate those steps.
And what's really nice about that is since Rainforest is
a visual tool, we identify things on the screen based
on screenshots. Right, it's possible for you to make slight
visual changes, and now that image doesn't quite match up,
your test might fail. You don't want to have to
go back in and retake all of their screenshots. But
since it's generated by AI, it could go back follow
the same steps and realize this is what the button
is here. It would be really cool if it could
kind of add that test coverage for you, or like
tell you what you should be testing. We've poked at
that a few times and it's obviously just really dumb
in that aspect and doesn't really give you anything useful.
It's like, yeah, go go test all the things and
make sure things work and it's like cool, yeah, I
knew that, thank you, Yes they get smart. I feel like.
Speaker 4: Part of the answer is also the domain you're in.
I know, something we haven't talked about is like really,
at the top of the test pyramid is exploratory testing,
whereas like add your creative human instincts to where bugs
could potentially pop up while you're looking at an interface
or API. And I don't think we're doing anything wrong
in the creation of AI and LM models. It's removing
the creativity from them, and I think that that harms
us here. But there has been one area, especially within
things like protocol creation or SDKs interfaces for the services,
and that's I think the keyword is fuzzing. So trying
an LM, any sort of AI can spam with almost
a more intelligent root for strategy about what sorts of
inputs tend to break your interface or your service or
your product and then use that as a potential test
that you can commit longer term. And again it's not
for everything, like I don't think it really works so
much in a UI world, but definitely depending on what
your service or interface is doing. Stuff in the crypto
space cryptography, not blockchain.
Speaker 1: Just to be clear, I feel like that was a
dig there, Warren, What are you trying to say?
Speaker 4: You know, it's not the sort of thing I want
to bring up on an episode.
Speaker 3: Well, you don't want a record.
Speaker 4: Huh yeah, definitely not on a record. We do cryptography
because we're really into security and deep there, and we
do not building our own crypto, but hi, we're very
high users of it. Everything JAWT creation or jot creation,
every single different kind of algorithm strategy we end up
utilizing these, and so finding where we're not using libraries
effectively is certainly an area that we've potentially looked into. Actually,
according to our company by laws, we're not allowed to
do anything regarding cryptocurrency, Like it's actually not allowed by
the country of Switzerland for us to get involved in
any way. We can't accept payments, we can't pay people
in crypto. We can't even think about consulting for companies
that I want to do something crypto related.
Speaker 2: That's discrimination.
Speaker 3: I work at HPC and I'm pretty sure like some
of the admins will just kind of use like a
little bit of the compute power from different clusters that
they have to be running different crypto schemes, but I haven't.
I haven't like one hundred percent caught anybody. But I'm
just I'm waiting for the day. I'm waiting for it.
Not to tell on them, I just want to know
because I'm super nosy and like I just like knowing
things like this.
Speaker 2: No, I'm going to tell on as long as they
cut you in. Yeah.
Speaker 3: It's like when my you know, when my website not
hacked by that Chinese jewelry store and I was like, guys, like,
if you if you would just give me a cut,
this would be fine. It was nice.
Speaker 4: I mean I think that's I think that really is
expert advice from our resident mL expert here, because.
Speaker 3: Like just making.
Speaker 4: Yeah no, because AS just came out and said that
the strategy of sharing reservations across customer a WOS accounts,
Like if you're a consultant that does bundling, for instance,
reservations or compute reservations, you no longer can pass along
that savings to the customer. I mean, what are they
going to do with all this excess capacity now other
than some good old fashioned bitcoin mining.
Speaker 3: I don't know, I don't know. I mean we could
be making drugs for autoimmune diseases and cancer, or or
you could be making some call part cash. I don't know.
Speaker 1: It all.
Speaker 3: Such both it's not in either. Or there's plenty of
compute power these guys have. You know, they're spinning up
plenty of a W Westerns. They're not going to notice
if that last ten is using crypto.
Speaker 1: So when this episode launches and we all get blocked
from a respective AWS accounts, we can just reflect on
this moment fondly.
Speaker 4: So, I mean, for the record, AWS is going to
block you because the ROI on utilizing cloud resources to
mind crypto is so low that you're pretty much just
paying AWS. But it is a good indication that there
is malicious activity happening on your account. So it is
something that they will for sure investigate, and that I
think is as much as a tangent on this that
I want to go down for today.
Speaker 3: Right, Yeah, I think we should talk about the low
code with Rainforest. I love low code stuff. How did
how did this come about? And like how does it work?
I just I want to know all about it.
Speaker 5: Yeah, for sure. So when I first joined Rainforest over
seven years ago, our model is a bit different. We
had a bunch of human testers. It was kind of
the gig uber model of I have something I want
to test. Here's my test cases. They're all written in
plain English, and we'll provide a bunch of humans for
you to go test your application, right, including some exploratory
stuff like you mentioned go click all over this page
and try and find problems with it. And that worked
really well. It was true and the end testing, we
load your app in a in a virtual machine inside
of a web browser. They're actually clicking the buttons and
confirming there's things on the screen. But what we found
is that humans are inefficient and expensive, as we all know.
That's why we have automation, right, and so we kind
of shifted over to automation, but we wanted to do
something a bit different from what everyone else was doing,
which is these code based tools uh dom based interactions,
and instead we built it all on the visual layer.
So the way it works is you go in, you
load your app and essentially, just like tix stree shots
of things, right click on this type into this field.
I can give it an ai an AI prompt and say,
you know, log in and check check out in the
cart and then when you execute things, it loads into
the same environment. Right, you have your your staging environment.
Hopefully you have some seed data with login information. You
can load that all in the rainforest. It goes in
and runs this whole workflow for you. The output of
it is a video of the thing being tested. Results
on each step, things like HTTP logs, jobscript, console logs,
all the information that you need to actually debug things
when something instead of it just saying you know, like
in a unit test and it's like failure like one
does not equal to And so by doing things at
that visual level, it offers a lot of flexibility. The
first thing is that we're out stuck inside of the browser. Right,
we do primarily focus on web based testing, but that
does not mean you're stuck inside of the browser. It
means you can do things like install a Chrome extension. Right,
open another tab in your browser, install a Chrome extension,
interact with that extension. Because while you're still inside the browser,
you're outside of the scope of that web page where
you usually are interacting just through the domin you can
you know, install some type of desktop application and test
it through there. Because since we're working at the visual layer,
it doesn't care what you're testing. It doesn't care what
your tech stack is. It just cares that it loads
in the machine. And it also offers a lot of
like more flexible and robust in avoiding flakiness and brittleness
to small changes. We have fallback methods. As much as
I've been kind of hammering that testing with the DOM
is not a great idea, we do offer DOM fallbacks
because sometimes it makes sense. Sometimes I don't care about
the visual appearance of the button, and all I care
about is that there's a button there. Right. In reality,
there are variables that we can't control. Right. A very
common scenario is my marketing team is running experiments. Every
time I load this page, the button says something different,
it looks different, and so we don't want to tie
the visual appearance to the past fail result of this test.
So I'll use something else. I'll use a DOM selector.
We also have like AI search. You could say something
like the button, the log and button at the bottom
of the page. And so the important point here is
you don't write any code whatsoever. We have an intuitive
UI that you do all of this through, which means
you don't need skilled engineers to do it right. A
lot of teams have QA engineers that their job is
to just write tests all the time. Other teams, you
are read the engineer is responsible for writing these tests,
but they need very specific domain knowledge, right. I need
to know about the thing that I'm testing, like from
a product standpoint, what does this thing do? I need
to understand the code. I need to know how to
write these tests with a no code solution. Anybody can
do this, right. It's up to your team who owns quality.
Who owns these tests. For us, it is usually the
engineer that is shipping the code we write the reinforest
tests with it, or our product and design team owns
it because, like I was saying, they're very tied to
our user workflows, right, They're like, this is how we
design this thing. Engineers are going to build it, and
then any of us that have knowledge about how this
userflow is supposed to work can own this test. So
it makes it much easier to both write and maintain
your tests over time. And then you know, if that
person leaves your company and has all of that domain
knowledge and how it works, you just need someone who
knows how the app works and they can update your
test suite.
Speaker 4: Now, I feel like one of the biggest mistakes that
I keep seeing over the course of my career is
as companies grow, they tend to have more, allegedly more software,
more code, which may or may not end up in
a giant ball of mud mass or even an extensive
number of quote unquote micro services that communicate and really
depend on each other. And there was always this challenge
by someone who wanted to have a test that required
somehow interacting with all of these components, and they never
really could understand that one of the whole point so
micro services was to isolate testing. But I think we
live in the reality, which is there are some companies
that do have a giant ball of mud that have
thousands of binaries that have to be installed on running servers.
Is there a strategy that I don't I don't mean to,
you know, pick on your company.
Speaker 5: I don't know.
Speaker 4: If I don't think there is a strategy. I think
the strategy is right micro services. But I can imagine that.
You know, as a SaaS company, the last thing we
want to tell our customers is, yeah, have you tried
not having that problem?
Speaker 3: Is there a package manager that tends to do this
solution that I see that what is everybody's favorite.
Speaker 4: Yeah, distributed monolith always works. Publish all your binaries that
remotely depend on each other to a third party solution
and then pull those out at run time always works.
Best solution ever. Maybe Aja, you have some insight here
on either something that works or something that works with
Rainforest QA to deal with those situations, or maybe you
just you know, it's not something that is handled today.
Speaker 5: Yeah. Sure, we do have a bunch of different micro
services running, and I think I'm going to refer back
to the testing pyramid. Right, is we test each one
of those micro services in isolation. Absolutely, Maybe we test
as we go up to the next layer of our
integration test, we test some interactions between them, right, the
kind of core you know, handshake interactions, whatever is the
main functionality of these two microservices talking to each other.
Maybe we have some tests there, But at the end
of the day, these end to end tests are comprehensive. Right,
If anything in that micro service architecture is failing, presumably
my test is going to fail. And at the end
of the day, all we really care about in theory
is what the user gets when they're interacting with it.
So if I'm just clicking a button, maybe there's a
thousand micro services that are involved in this, and maybe
I'm not directly testing each one of those, but by
implementing it as an end to end test, I am
very confident that they're all working because my test passed.
And so being smart about how you implement each one
of those layers in an efficient way. Right, lots of
unit tests on each micro service and then this overarching
test that just make sure everything is working together is
usually a way to go about this.
Speaker 4: I mean, maybe it's a technical implementation question, like where
is the environment running that the rainforest tests are actually
executing us? Is this some sort of binary or CLI
that's run on the client side, or are they sharing
with you a set of micro services with deployment instruction
so that you can run them within your own infrastructure.
Speaker 5: Sure, Usually our requirement is that you need to be
able to access it via a web URL. So the
kind of standard way that a Rainforests is run is
we provide you a VM, and that VM has a
browser on it, so I can specify Chrome on Windows
eleven and the first step of your test is going
to be a navigation navigates this URL. This is where
my web app lives. There are some different use cases
where you can absolutely go download a Binari and install
it and do whatever you want with it or out
of the box functionality is to primarily test those web apps,
so that's where we focus. But you certainly have the
flexibility to do whatever you want with those vms.
Speaker 4: No, I mean, I think I think that approaches genius. Basically,
it's out of scope for setting up the environment unless
you want it to be in scope of which you know.
Then it's a it's a virtual machine. Go to town
on how they want to deal with it there.
Speaker 5: Right, and by kind of forcing people to give it
a public URL, we're nudging them towards good practices. Right
for sure is set up a staging environment a QA
environment and make it mirror production as much as possible,
which includes being able to navigate to it in a URL.
And these are small things that we see with some
of our new clients are like, well, I don't have
a staging environment, and sure, I guess you could load
your production environment in there, but let's show you how
to test this properly and not shoot yourself on the foot.
Speaker 4: I'm like, I love that you're saying that. The number
one feedback I've always seen here is we can't expose
our non production environment publicly, Like people can't know what
we're currently working on. They will use that information maliciously
against our company in some way.
Speaker 5: Yeah, like what are they going to do with it?
Speaker 3: Though?
Speaker 5: You know, like if I have seen some interesting mistakes,
like you know, maybe we're cloning our production database and
not sanitizing sensitive information from it or something like that. Yes,
absolutely you're doing some bad things, but there are certainly
ways to do this. And I'm of the opinion who
cares if people are in your testing environment, Like worst
case they blow up your testing environment and whatever.
Speaker 1: And in that case you figured out how they could
blow up your production environment.
Speaker 5: Without losing pride exactly.
Speaker 2: Yeah, I think there's.
Speaker 1: That's, you know, like part of the undocumented learning curve
of working in this industry, you know, because people who
are early in their careers think things like, oh, I
shouldn't expose staging until you know, they learned that that's
actually probably a good thing.
Speaker 2: But like, nowhere in any.
Speaker 1: Computer science or course or or boot camp or anything
do they cover these kinds of things, and so I
think that's actually like a really valuable add on service
that you know, you get from rain forest, or that
you get from working with people who are more experienced,
is just like learning that tribal knowledge that's going to
help you out later in your career, so you don't
have to reinvent the wheel and solve problems that we
actually solved thirty years ago.
Speaker 2: I mean, that's will.
Speaker 4: I mean we've unfortunately had to append our documentation with
like here explicitly are the sensitive pieces of data that
are relevant to our third party application. This is sensitive.
This is sensitive, Like this is not sensitive. This is
like the application I need not sensitive. Like, do not
try to encrypt this, Do not try to secure it,
because people will try like how do I do this?
I'm like, you can't stop it, Like this has to
be public on your website, in your application. People have
to be able to see it. You're not going to
be able to get around that. And I feel like
it's more than just experience. I feel like there's a
whole level of pragmatism there, like weighing the cost versus
the reward of actually trying to sanitize that piece of information.
And having a third party testing service, as you mentioned,
just reinforces that in a way, like you are going
to have to expose that to be tested, must show
that it's not actually sensitive information.
Speaker 5: Yeah, definitely. To me, it just reminds me of like
the myth of the one hundred percent test coverage, right,
It's like, can we one hundred percent encrypt everything? Absolutely not?
Like people, your end users need to see this information.
I see. I've seen some interesting attempts to kind of
obfuscate those things, Like I've seen some libraries that prevent
you from opening the davascript console, for example, and it's like,
what are you hiding in there? Maybe maybe you should
just not put sensitive things.
Speaker 2: Here's here's a wild thought. I just don't do that.
Speaker 3: Credentials like encoded in the HTML on your page.
Speaker 5: Maybe.
Speaker 4: I mean, I can't believe you two are joking about this, honestly,
like one of the most common attacks against So I
don't do.
Speaker 3: You why so I can joke about all this because
none of them It's okay. Tone You'll have.
Speaker 4: Jillian, You'll have plenty of opportunity to get your models
encoded with AWS access keys in secrets, and then you
just ask the model, Hey, can I have an access
key and secret that are valid that work for any
aw's account.
Speaker 3: I did actually accidentally push my AWS credentials to get
huged once, and like the amounted emails that I got
from AWS was just like it was unreal. It was
very It was a very bad day for me. It
was a very very bad day. So I've done other
stupid things, but I don't do the same stupid things,
so I can sit here and be very smug about this.
Speaker 5: Like.
Speaker 1: That is like that is I've always I've often wondered
about that, Like the speed that AWS and other malicious
people can identify that you committed an AWS access key
to a GitHub repo.
Speaker 3: It was instant. It was right then, because as soon
as I did it, I was like, oh no, and
tried to you know, and like try to like make
the gi hub repil private, and no, it was instant.
They knew, they knew it was out there.
Speaker 4: Yeah, I mean it's bad. I mean I think I
saw a bunch of statistics on this that for AWS
keys and on GitHub, it's about thirty seconds to two
minutes having been exposed in the repository anywhere in any format,
So like commit at the beginning of the repository where
it was there but then got removed, so it's not
in plain text anymore. You have to go back through
the get history still about two minutes. Then there's exposure
on like stack overflow and places like I don't know
who uses Facebook in connection with their work, but that
was another place, and then Instagram and read it somewhere
between two and four or five days, and then there's
a couple other ones where it's six and more.
Speaker 5: Some of those you have to.
Speaker 4: Thank, like GitHub for like they'll actually discover secrets there.
So if you provide a third party application that has credentials,
like at authors, we have our secret keys registered there,
so if one of our customers exposes keys for our
service on GitHub, we'll get notified, automatically revoke those keys
and send them an email telling them that they did
something that they probably did not want to do, multiple
times if that's if necessary, because that's happened as well.
Speaker 1: I want to switch topics here real quick, AJ, because
you've been with Rainforest QA for over seven years now,
which is unusual in the tech industry, So I'm curious
about what are the what are the things that you
look for in a job, that have been fulfilled at Rainforest,
that keep you there that long.
Speaker 5: Yeah, for sure. First and foremost is the people. Actually,
when I was interviewing with Rainforest, the last person I
talked to told me something like I say, at Rainforest
because of the people, And I was like, okay, that's
that's a way whateveryone said, And then I quickly drank
the kool aid, I think, and I found myself saying
that on interviews and I'm like, I know, this sounds
like a load of crap, and I think, you know,
the hiring process is super super important, right, Yeah, both
finding people that are qualified for the job obviously, but
are good culture fits. We have a pretty small team,
so there's nowhere to hide. If you are not doing
your job where you're not up to par, you're going
to be exposed pretty quickly, which leads us to have
a very reliable team. You know, we are distributed globally,
so there's a lot of handoff. You know, I'm going
to sleep, you're waking up. Here's what I did, and
I trust when I wake up that you're just going
to have this thing done. And if you're not one
of those people, you're probably not going to fit at Rainforest.
So really really qualified, experienced, smart, reliable people makes life
so much easier. And then the other piece of it is,
you know the mission that we're on, the technology that
we're building. I think when I was first exposed to it,
the first time I ship code with Rainforest, it was
kind of like, Wow, how have I been shipping code
before this? And the answer was I was probably breaking
things all of the time and you don't notice until
user catches it in production two days later or whatever.
And it's something that I'm really passionate about. I think
as a front and engineer, we get really caught up
on the details. Right, There's all these visual layers, there's
these very specific human interactions. I like building things that
humans are actually interacting with, and that kind of naturally
leads you to a quality assurance mindset, Right, I want
everything to be perfect all of the time. How do
I ensure this? And so the combination of really great
people and working on something that I'm actually really passionate
about and I want to see the rest of the
world adopt these correct ways of testing things, in my
opinion of course, just makes it makes it easy to
work area.
Speaker 2: Right on. That's cool, that's cool.
Speaker 1: Is there you guys obviously do a lot of front
end type testing. Is there a particular industry or vertical
that you have got a lot of experience it's in,
or something that has worked really well that makes a
really cool story.
Speaker 5: Something I've been involved in that makes a cool story. Oh,
I don't know if I have a good answer for you. Honestly,
I've been, like I said, I've been at Rainforest for
so long. That's all I didn't think about. I guess, right.
Speaker 1: Do you do you attract like a certain like customers
with like financial apps or with like web based gaming apps,
or is there a particular vertical that tends to gravitate
towards your service?
Speaker 5: I think not really, And I think that's one of
the things that makes it cool is it's a very
generic testing tool. Right There are some some limitations, but
in general, if you could load your app on a machine,
you could probably test it with Rainforest, not caring about
what the tech stack is those kinds of things. So
there's a very wide range of users that we have from. Yeah,
there's some financial some financial companies do some What I
always finds interesting kind of like testing visually things like
spreadsheet style apps like their tables and things like that.
And then we have some really cool like visual tools
like like drag and drop interfaces where you're building things
like you know, lego style building, where there's probably, to
my knowledge, not any other great way to test something
like that, Like what you say, are all my legos
on the page? Yeah they are? Are they kind of
oriented this way? Like? Yeah they are? But how does
it look right? What does the user see? So the
real sweet spot is really visual based applications because I
don't think there's other great solutions for them out there.
But in general, being a kind of generic visual testing application,
it really applies to anything right on.
Speaker 1: For a lot of web based front ends, it's all
no JS based. Do you have a favorite no JS
type tool? Are you like a React fan or next
JS or view give.
Speaker 2: A personal preference?
Speaker 5: Yes, I am a Ract fanboy for sure. I started.
Uh you know, we rewind all the way back to
like the j Query days and stuff, right, I see that,
and I have nightmares still we have some we actually
have some of that floating around and our like like
our admin applications and stuff where it's like a rails
back in and they're like, yeah, we got jQuery in there.
And then my first thought was always like, well, like
how do you test that jQuery? And the answer is
we don't. Uh throw a couple of wait for us
tests that and call it good. And I started. I
started with Angular back in the day, all right, back
Angular one anyways was kind of the reverse of React.
We're like, we're gonna put your JavaScript in your HTML.
React took the approach ever to put your HTML and
your JavaScript, you know, just smush it all together. And
it's come a very very long way, I must say. So. Yeah,
I find working with React very easy and intuitive, and
it's very nice that the general JavaScript community has supported
that and has pushed that forward because especially, you know,
with with all software and technology, but especially in front
of development, it's really easy to pick the wrong tool
long term, Right, I picked this thing, it's great, and
then we find a better way to do it, and
they just abandon the project. Right. This is true with
I mean anything open source, and we've run into this
a lot of times, right, even with open source testing tools.
And actually we had a very large Enzyme test suite
on our React application, and we ran into something like this.
There was a new way of testing React apps, which
was the React Testing library, and Enzyme kind of said, yep,
that's a better way to do it. We're going to
stop supporting at after I forgot version like REX sixteen
ORRAC seventeen, like, well we want to upgrade to React seventeen.
It's like, well none of your enzyme tests work, so yeah, exactly,
exactly too bad for us. And so now you start
weighing the options of well, how do we upgrade? Right?
Do we just say let's not upgrade, which is going
to bite you really quickly, right, especially at the pace
all these JavaScript libraries remain updated. I want that new
shiny thing, I want support for that thing, and I
don't want to be stuck in the past. The more
you get stuck in the past, the harder it is
to catch up with everything else. Right, And so our
options were basically rewrite all these however many thousand enzyme tests,
or we could just nuke them all, which is it
reminds me of like these these memes I see about
like junior engineers and the intern where they're like they're
commit messages. I knuked all the tests because they were failing,
and I kind of make them pass turn true and
all the tests because you know, that's one of the.
Speaker 3: Past, the only reasonable way to do things.
Speaker 5: Yeah, and it sounds kind of like an overreaction, but
as we started to kind of think about these testing philosophies,
we're like, we have end and test coverage on all
of these things, right, And a lot of the front
end tests, even though they're unit tests, they they load
things in a headless browser and we're kind of recreating
what an end end test does. So we chose to
keep all of our actual unit tests, all the kind
of business logic that didn't use enzyme new call the
enzyme tests, and just lean into our rainforest test because
we know if the rainforest tests are passing, we don't
need all of these redundant tests anymore. And instant productivity
boosts Like I don't have to maintain all of these
things anymore. I don't have to upgrade them. I could
just get them out of my way and I can
upgrade all my dependencies. And because we have really good
end and testsage, we could do that confidently and know
that we're not breaking things. So, yeah, choosing dependencies can
be quite tricky sometimes, especially in the job script world.
Speaker 4: Did you find some places that you still wanted to
reintroduce some of the React testing library for I don't know,
component level testing of the UI or have you kept
with like one hundred percent of the decision to not
have that layer of testing anymore regarding the UI components
because you focus on the full picture and the end
testing for the user flow and also whatever you have
with the interaction with the back end.
Speaker 5: Yeah, we still have some of it, and we drew
the line at user interactions. Right. So React has this
idea of hooks, which are basically just chunks of logic.
It's just a function that I can use inside of
a component. We stopped having any React testing library tests
that were actual user interactions, no clicking on things, and
instead we used it to test the functionality the logic
of those hooks. So it's essentially a unit test, but
it's testing a specific React thing, and it requires the
testing library to do that. Everything else kind of gets
hoisted up to the end to end testing level, and
it's nice to just say, hey, designer, hay, product manager, like,
go at this test coverage while I'm busy hacking on things,
and I don't have to worry about this anymore.
Speaker 2: So that's actually a great Jillian.
Speaker 3: Oh, I was just gonna say, I'm so impressed with
people who can keep up with like the UI and
JavaScript plan because I've tried. I've tried like a few
times and it just then everything changed. I was like,
all right, I'm not doing this anymore, and we go
I'm gonna go do high performance computing that hasn't changed
in like thirty years.
Speaker 5: It's gonna be Yeah, you definitely start feeling like Sissyphis.
You're just pushing that rock up the hill and every
time you get to the top, someone tells you that
you're actually on the wrong hill.
Speaker 1: I was gonna say, that seems like a really interesting
approach that I hadn't thought of when we initially started talking.
But like you, you can replace having to write a
lot of your tests in your React app by using
Rainforest right. By just focusing on what the end user
experience is and testing for that, you can save yourself
from having to write a lot of tests in the
React standard library.
Speaker 4: So that's where the trade off is though, right, because
these tests then are testing more functionality at once and
so if there is a problem, you don't necessarily know
like which line of code is causing the issue or
which interaction there is, so you know there really is
Like how valuable is that flow? I think? And that's
something that, as you pointed out Agent, like you sort
of have to determine upfront, like where is the value
of your testing and how do you get the most
value out of which pieces.
Speaker 5: You're adding and where you're validating an et cetera. And so, yeah,
I mean.
Speaker 4: In your case, and tis weren't actually providing the right
value in the first place, so definitely switch them over.
Speaker 5: Yeah, absolutely. And it's kind of a question of redundancy too, right,
Like is redundancy good? Sometimes like I can be really
really sure and I can have some extra confidence that
the things isn't going to break, But most of the
time it just slows us down. Right. I find that
the often the best time to add more unit test
coverage is when something breaks, right, because if my end
to end tests are all passing, but something's broken, very
often it's some kind of edge case, right, It's some
either some weird user behavior, some weird input, some weird
sequence of events, and those things are usually better captured
in a unit test because it's it's easier to kind
of implement that specific scenario, that specific line of code
that is the offender here versus creating you know, a
whole new end the end test and just cover some
edge case. Those tests are going to just get longer
and longer and just be kind of confusing. Honestly, it's like, well,
why am I just like clicking in all the these
random spots doing these things trying to cover these edge cases?
Like just reread a unit test for it and call good.
Speaker 3: I really like the emphasis on, you know, testing for
business logic and just in general having not everything controlled
by the engineers, because I find for myself, you know,
like I'll write something and then I'll hand it off
to a user and then they immediately start using it
in some way that I didn't even think of, and then,
you know, and then we do like a couple of
rounds of this. So being able to cut back on
that person who writes thing, who does not actually use
the thing, and then just immediately being able to push
it off to an end user.
Speaker 5: Yeah, absolutely, Yeah, And things like testing and staging environments
are great for this. We push code of those environments
all the time, give them to a PM and say
go run and try and try and break this thing, right.
Don't always want to do that in production, right, like
if the thing's not fully baked, I don't want to
corrupt something in the database or whatever. And so having
places to push that and have people early in the
process on this, finding the major bugs, the minor bugs,
the stylistic bugs is super super valuable then having one
of your users find it later.
Speaker 2: Right on. So you live in Tahoe. Do you get
outdoors a lot?
Speaker 5: I do. I live here with my wife and my dog.
He's a lab husky mix, so he kind of thrives
in the summer, thrives in the winter. There's lots of
snow out right now, and so we're outside pretty much
every day, you know, snowboarding, hiking, kayaking and all that
kind of stuff.
Speaker 2: All Right on, how long have you lived in the
Tahoe area.
Speaker 5: I've been here for about five years now. I grew
up in the San Francisco Bay area and I was
part of the great COVID migration out here. We always
wanted to get here eventually, and I was lucky. I
was still working at rainforest at the time and already remote.
So the transfer up here to from remote near an
office to remote it actually doesn't matter how far you
are from the office, was very easy. And we're also
real fortunate that we were not the only ones doing
this migration. So we've made lots of friends that were like, yeah,
we lived down the street from you in the city
and we all live here now. So it's a very
different life, but we love it and I don't think
we're leaving.
Speaker 2: Ah, that's cool. Yeah, right on, Tahoe is a beautiful area.
Speaker 5: Yeah, it really is.
Speaker 2: Cool. All right. Should we move on to some picks
before we do any final thoughts on.
Speaker 1: QA rainforest tips guidance that you want to leave us
with aj.
Speaker 5: I think just recapping is finding the balance between confidence
and velocity, right, everybody needs to set their own bar
for quality, like how what is my ratio between confidence
and velocity? Determining that for yourself is the most important
thing here and keeping in mind that not only is
it a lot, but a lot of times it's the
sanity of your engineers, right, Like, we don't want to
spend all of our time writing tests, So finding that
balance and doing things in an efficient way is the
key to success.
Speaker 1: Right on. And I think that's very use case specific too,
you know, because the right answer for a financial app
is going to be very different than the right answer
for like.
Speaker 2: A social media app.
Speaker 1: Absolutely cool, All right, Jillian calling you out first? What'd
you bring for a pick today?
Speaker 3: I am going to pick Drive by Dave Kellett. It
is a sci fi graphic novel and I think it's
on it's I think it's like releasing the fourth one
this summer. But it's so good and it's so nice
and wholesome, which is very nice because like, I really
like sci fi, but I don't really like violence or
gore or you know, ikey fluids or I don't like
any of that. Okay, I don't like any of it.
And this is just so wholesome and adorable and the
main character is very cute. So that's it. That's the pick.
I've got a whole bunch of copies for Christmas and
I'm like making people read them and I'm gonna have
like a little little intigraphic novel cult going on soon enough.
Speaker 2: Great, all right, Warren, would you bring? Yeah?
Speaker 4: So I just got back from a long hiatus being
away from the show I was on vacation, and so
I think this pick is a really accurate, very short book.
I highly recommend Dowdishing by Laozza, which is the founder
of Taoism spelt Taoism in case in case. You've seen
it written but never pronounced before. And there's just so
much good stuff that is in the book that can
be applied to everyday life, working environment, etc. It's incredibly short.
There's only like one hundred and eight principles or so,
and it starts off great with the dow that can
be told. Is not the eternal tao Like, you can't
write down the whole truth. There is something that's never said.
It's impossible to convey everything. And I know it sounds
so philosophical, you know, to go down this path, but
I feel like going through these really helps to put
into perspective thinking outside the box with solving certain problems
or interactions or the communication we have every day.
Speaker 2: Highlight right on, cool heja, what you got for us?
Speaker 5: Yeah. My reading and listening choices are kind of all
over the map. But I did have an interesting one recently.
It was called The Light Eaters. It's about plants and
specifically this idea of plant intelligence. So obviously intelligence is
a loaded word. They're not intelligent like you and I
either not debating QA strategies and things like that, but
they do have a lot of intelligent like behavior. You know,
they communicate, they recognize their kin, they hear sounds, they
transform themselves based on the visual appearance of the environment
around them, And so I found it really interesting and
gave me a lot to think about, especially you know,
when I'm out in nature with the with the wife
and dog, just kind of staring at trees and stuff.
Speaker 4: Check out that are intelligent for sure, one hundred percent
totally with you. There's a there's a good one. If
you are out and there's plants or grass being cut
and you notice the smell of you know, freshly cut grass.
Speaker 5: What is that?
Speaker 4: It's a fear intently, a fear pheromone that's been sent
off to warn other grass that that there is danger around.
Like that is the sign of intelligent life.
Speaker 5: Yeah, for sure. There's lots of super interesting examples in
this book, and just like plants acting like animals essentially,
and it's it's kind of mind blowing experience.
Speaker 1: I read a book recently and I can't remember which
one it was, but I've been studying mushrooms a lot lately,
and this book showed where mushrooms actually act as a
communication agent for trees in the forest, and so like
a specific you know, set of insects can start attacking
trees on one end of the forest, and then the mushroom,
because it's the mycelium that grows underneath the entire forest floor,
will relay that information to the other trees in the forest.
And so by the time the insects work their way
down to those trees, that those trees are producing a
sind or a pheromone that actually repels the insects by
the time they get there.
Speaker 2: And I thought that was super cool.
Speaker 3: That is cool. I'm in on a like a mushroom
and foraging Facebook group and everybody just like takes pictures
of fun mushrooms that they find when when they're out
and about, and it's it's just such a nice little
group because it's so chill. That's it. There's like, there's
no drama, there's no nothing, it's just look at this mushroom.
Speaker 1: I found there's an app called Naturalist that I use
for that you can take a picture of not just mushrooms,
but anything you find that you can't identify, and then
upload it to I Naturalists and it will it will
try to auto detect what it is for you, but
then other people will come in and confirm or tell
you what that actually is.
Speaker 2: That was pretty cool.
Speaker 3: I used to do that a lot as a kid.
I'd have like the field guides and go out with
my field guide and try to like identify all the plants.
But now we have an app for that.
Speaker 1: There's an app for that always will what's your what's
your pick? My pick is There's a series on Netflix
called Kunk on Earth and I thought my sense of
humor was like really really dry, but this lady, she's amazing,
takes it.
Speaker 2: She takes it to a whole new level.
Speaker 1: This series is just hilarious because she sits down it's
you know, like a the History of Earth basically, but
she'll sit down with legitimate world renow experts in their
field and ask them the most off the wall questions.
And that, to me, thet was the highlight of the series.
Is just the looks on their faces when she would
ask them these questions that had absolutely nothing to do
with what they were an expert in, but super entertaining series.
Definitely ten out of ten stars. Kunk on Earth on Netflix,
and you know, there's actually two other things. There's Kunk on.
Speaker 4: Britain I think, and then there's like one on Christmas
and Shakespeare, so you too.
Speaker 5: Oh sweet.
Speaker 2: I will have to check those out because I love
her sense of humor.
Speaker 3: I've never heard that's fun.
Speaker 1: Yeah, it's very very accurate.
Speaker 2: All right.
Speaker 1: That brings us to the end of the episode. Thank
you everyone for listening. Jillian Warren, thank you for joining
me hosting the show. And aj thanks for coming on
the show man. It's been a pleasure talking to you.
Speaker 5: Thanks so much for having me.
Speaker 1: It was a lot of fun right on. Glad to
hear that, and I will see everyone next week.
Speaker 5: M