Speaker 1: And we're live for anyone who doesn't know. We also
live stream this on LinkedIn, YouTube and twitch. At what
time is it? It's uh what ten thirty Eastern time
on Tuesday morning? It's two thirty pm UTC. If you're
not in one of those time zones, you're gonna have
to do the math yourself. But I love you for
listening and watching the live stream. Meanwhile, what's up born?
Speaker 2: Hey, thanks for turning that up. And I always get
the time zones, you know, totally wrong living in Switzerland
where at plus two UTC at Yeah, I mean some
of those special ones in North America are quite challenging.
Speaker 1: Yeah yeah. On my team currently, I'm in GMT minus six.
I have a person in GMT mine three, one in
GMT plus one, and one in GMT plus two.
Speaker 2: So you know, I only is thickering here because we
built a product a stand up bought called the stand
Up and Prosper and it was part of an old
product that we had built a long time ago, and
very quickly we realized, you know, we may actually be
one of those companies that now has to understand really
lots of intricacies about how time zones work. Because like
we have to send we'd send messages to people when
it's eight pm or eight am in their time zone,
and we try to rely on like the Slack time
zone for that and it's not always valuable. And then
we also have to display it an RUIS and also
get it right on Slack display and trying to schedule
in a different time zone is always such a challenge there.
Speaker 1: Yeah, I worked at a startup.
Speaker 3: One time.
Speaker 1: We did medical imaging for trauma hospitals on their trauma
patients and they decided to rewrite the core application in Java.
We're not going to go down that path, but that's
what they did. Right the very next time that daylight
savings time went into effect for the US, we had
about two thousand trauma patients disappear from our system because
they didn't account for daylight savings time. So that was
a good time.
Speaker 2: Yeah, I mean, there are definitely some things where we
allow events in the future and then we try to
hide them being displayed, just because prepopulation of data is
so important for production scale applications because you don't want
to wait until you know something actually happens before having
those events in your database. So I can totally imagine
that they're omitted in the future, so don't display them
in any of the UIs to users because that would
be confusing.
Speaker 1: Right. But speaking time, it's that time of year that
we're coming up on the end of year, the holiday season,
which can have a lot of it can mean a
lot of things depending on what industry you're in, how
your company handles closing the books at your end, and
so that's what this episode is all about, is just
kind of I think it's kind of just us talking
out loud as reminder to ourselves as well as a
reminder to the listeners of our show that those things
are coming and to start thinking about them, because we're
going to be executing those tasks here before you know it.
Speaker 2: You say that, and the first thing that comes to
mind is when it hits to be cyber Monday, I guess,
or you know, three weeks ahead of that, because realistically,
that's when every year it just gets soon and sooner.
And I remember working at so many companies where someone
was telling me, you know, something's going to be different now,
But as an engineer, I'm just like, I don't. I
don't know what that means. I don't I don't know
how to what do want to do that information right?
Speaker 1: For me, the hardest part has always been getting that
information out of marketing because they spend a lot of
time and effort doing campaigns, marketing campaigns where this's ads,
email blasts or whatever to build up these big sales events,
And in doing this for three decades, I have never
been successful in convincing someone from marketing that, hey, if
you're gonna funnel an extra two million people to the site,
did you ever think about giving someone an engineering like
a heads up?
Speaker 2: Yeah? I mean that's always the most ridiculous thing. I
think it leads the conversation sometimes in an disappointing direction, though,
because it goes from like, Okay, well we're going to
sign out two million emails, but how many? How many
of those are going to be opened? And then you
realize by many marketing departments, the answer is we have
no idea, and it's like, is that two million users
coming to our site? Or is it two hundred thousand,
is it twenty thousand? And so a lot of times
that stuff happens and there's zero impact to the business,
And the most effective thing you can do is, you know,
just stop performing those experiments in that way unless you
actually know what the click through rate is, what the
conversions are going to be. And that's interesting you say
that because when we build authors a while back, one
of the most common concerns we heard from our customers
that we're using our competitors is how they charge by
monthly active users. And that means if you sign in,
you get immediately dinged for a user count that then
is amateurized through the whole month, which means you're basically
paying for thirty days worth of your advertisement, which is
a huge cost, especially when these marketing campaigns go out.
But yeah, I mean yeah, engineering very rarely knows what
to do there, and so we went to a pure
charge by usage, so you know, if they only log
in once, there's not a problem. But I still think
most of those marketing organizations they have no idea how
many users are going to actually come to your site. Yeah.
Speaker 1: I think the big takeaway from that is that I
chose the wrong career profession. I chose one where my
performance can be measured. That's my fault.
Speaker 2: Oh you say that, and now I really wonder how
are you? How are you being measured? Because performance measurement
in selfware engineering, I think, like anything is, it's much
more difficult than widget production, which is, you know, highly
repetitive in which you can actually measure the throughput and
you know, quality metrics of the individual machine that's that's
manufacturing those widgets, whether it's human or automated.
Speaker 1: Yeah, from my perspective, since I'm mostly infrastructure based, a
good bit of my performance evaluation is based on uptime
and ease of deployment and error rate and things like that.
But I would I don't have any data to back
this up, but I would say that a bigger part
of my performance evaluation comes from perception of the teams
that I support. Do they feel like they're being Do
they feel like they're getting the level of support that
they need? And that just comes down to customer service.
Recently had a situation on that exact same topic where
someone felt like they were not getting the level of
support that they needed, And the only thing I changed
was at the end of every meeting asking them, Hey,
is there anything that you are blocked on because you're
waiting on us to do something, and just doing that,
just calling it out to attention, you know, was enough
to change the perception of the performance.
Speaker 2: Yeah, I mean, I think that was one of the
biggest learnings I had from when I started professionally, is
I thought that my deliverables as I coded them, as
I architected them, was worth something. But I knew that
there was this thing of No, you're judged by politics
is the word I would use to describe it, which
I really think is a misnomer honestly, because you aren't
fairly or equality judged on what your outputs. You know,
it is really about the perception. And talking with whoever
is in charge of promoting you is really the first
important step that everyone really needs to take on day
one when they get a job. How am I actually
going to be evaluated and over time whether the things
you're working on are actually relevant to being promoted or
even just staying in your job.
Speaker 1: Yeah, for sure, that's the whole like social side of
your career. Like your technical skills are only part of it.
Speaker 2: I mean, if you can get by working like not
writing any code every single day and just you know,
saying like spitting a good game to your managers, you know,
you can get really really far, so much so you laugh,
but so much so that there's actually a name for this,
and it's called effective managing up And you actually see
a lot of not helpful, probably not good managers doing
this and that's how they got to those positions that
they're really effective. It commits other people that they're doing things.
Speaker 3: Yeah.
Speaker 1: Yeah, it's which goes back to your earlier comment that
is very political in the US, that's the basis of
our entire government. Without turning this into a political comment,
let's not go there.
Speaker 3: We shouldn't do. Yeah.
Speaker 1: But so back on topic here, you're in what does
what does your end look like for you or office?
Speaker 2: Honestly, we don't really pay too much attention to it
because we don't go through the marketing cycles. Uh, there
is usually some amount of uptick that actually starts at
the beginning of the year, especially for for two reasons.
One for our products that deal more around the ASIC nature.
So I mentioned we have a like number one or
two slack bot for handling stand ups. And at the
beginning towards the end of the year. Beginning of the year,
there's a lot of vacations. You know, people aren't all
in office. It's more remote work, and so figuring out
how to manage your stand ups in an acinc way
is usually valuable for authors, which you know, obviously is
our main product. Budget comes in at the beginning of
the year, and so that's when a lot of teams
start thinking about more what's going on there, and towards
the end of the year they're dealing with all of
the tech debt, all of the mistakes they've made in
the previous six to nine to eleven months before they
get to that point. And I think one of the
biggest ones is the marketing spam, you know, having actual
users come on board. And I actually remember that earlier
on in my career, I was working at a company
that was had billions of dollars in revenue coming in
every year at the cyber Monday, Black Friday, you know,
huge deal days, right for getting in lots of users
to the site, like not even having to ship out orders.
I mean, the biggest problem is actually the manufacturing world
having to then turn around and manufacture and then ship
that stuff out when it's not just sitting on your shelves.
So much so that I remember that frequently management quote unquote,
I don't know who these people were, you know, you
hear it through the grapevine, like it's coming. You know,
there's going to be a massive wave. You need to prepare,
prepare for that, so only really important software should be
done at the end of the year. I feel like
it's the even more ridiculous version of don't merge code
on Friday afternoons. This is don't don't even think about
code between now and after Christmas. And also there aren't
any people there in the office, so maybe also you know,
push that out to like, you know, January eighth, so
from now until January eighth, everyone's on vacation. And that
just always struck me as so utterly ridiculous that a
software engineers could anyone really, but I'm gonna blame software
engineers could determine the difference between like a critical thing
that needs to get the production and like feature and
something that's not because you're often driven by that perception
of this needs to get done, and you're often not
in the situation of being able to even make that decision.
Speaker 1: Yeah, you know, up until recently, I had never participated
in the year end code freeze strategy. But at Polygon
we actually have certain platforms where we do a year
in code freeze, and it's frustrating to me because I'm like, no,
just just just know. This is the part of the
show where I could wish, where I wish.
Speaker 3: It could cuss.
Speaker 1: But but the truth is is where we have some
very very early stage products and some we've built some
things that, for all practical purposes, have never been done before.
So when it comes to monitoring and alerting, we have
room to improve on that because like, if you've never
done this thing, how do you know when the thing
is doing the thing that it's supposed to do and
how are you going to alert on that? And there's
there's certain stages where we don't We just kind of
watch it until we get a like get the pulse
of it, and then we can put monitoring on that.
And so that's the I think that's a big reason
why we're reluctant to do.
Speaker 3: Your end releases.
Speaker 1: And then also, you know, combined with that, like you mentioned,
a lot of people are out taking vacation and then
we're completely globally distributed to so we you know, async
communications are very very async.
Speaker 2: Yeah, I mean there's a platform thing there, right, Like
if you write games as your company's product, there's a
validation process, same with I mean similar to if you're
in healthcare some other apps, if you're deploying a mobile
app for iOS or Android, there's a turnaround time there.
And so even if it's even if your company is
absolutely perfect, we do live in a more just in
time world, a more agile world where you are depending
on third party partners. So even if you do the work,
you know realistically it's not going to be actually able
to get to production. So rather than having it stack
up where you're creating a lot of waste there that
someone still like, you don't want tons of merger requests
sitting open waiting for someone to come and test and
review your code. You want to get those production as
soon as possible. There's no reason to start that work
because you know it's going to be blocked somewhere. So
I do totally get that, especially if you're depending on clients, right,
you actually don't want to do any work unto your
client already says why is it not delivered already? Right,
Because if they're not already waiting on you for a chance,
you're going to do the work and then have to
wait on your client and then you're like, Wow, those
three weeks to three months are longer. Unfortunately in some
concern cases that I'm sure you've got well, that you
could have been doing something more valuable in that time.
Speaker 3: Yeah, for sure.
Speaker 1: That makes me think of doing mobile development and getting
your app approved to go into the Apple App Store?
Is that three days? Is that three months? Who knows?
Speaker 3: Is it consistent?
Speaker 2: Nope?
Speaker 1: It could be one right after the other every single time,
and that makes it really challenging to roll out new features.
And then compounding that, just because you release a new
version in the App Store doesn't mean people are going
to update. I remember when I was active, we had
a very very small group of people, I mean less
than ten that hadn't updated their app in two years.
They were running on a two year old version of
the app and perfectly happy, but it was causing us
some painful technical debt. And we finally had our marketing team.
They did a marketing and customer service. They jumped through
all kinds of hoops to actually figure out who these
people were, and then I think they were able to
contact them and ask them and send them a gift
card or whatever to upgrade their app.
Speaker 2: I got two great stories on that point. Actually, the
first one is having built a very technical API. People
talk about deprecating old stuff, but realistically, if you write it,
you should just pretend it's never going to get removed.
It's so much like the amount of churn challenge, complexity
of trying to convince your customers, as you described, to
make a change there where there's zero, like, there's negative
value in making the change. Right, it's all work, it's
working today. Why would they why would they realistically change?
It's just not worth thinking about in most cases. And
there are so many companies I've worked with who had
this mindset of, oh, we'll get them to change, and
so they then have like five or six versions, like
one every couple of years. Every time there was a
new engineering lead team you know, owner of that product
roll out something new. And so we started having version
numbers in our endpoints and things like that, and so
you see all of them available in production. Still some
of those companies I worked with, you know, they still
have those oldest versions still there because they don't have
that amount of pull within their organization with their customers.
So actually, one of the things we did when we
wrote Authoris is that we just completely discarded that notion,
like there are no versions, it's will be like this
forever likely. It's it's so totally unrealistic. Now, we do
make some changes where we then like update our documentation,
we may remove reference to fields that we still support,
but we don't remove them because the amount of overhead
it goes in it is really high. The other thing
I'll say is I am totally that person that has
old apps on their phone, like from multiple and I
think there's sort of like a two sided extreme here
where both of them are not great. It's there are
tons of apps out there that every single time there's
a new version, it's like you must update to use
the app, and I hate it. And I'm gonna call
out Signal in Google because they both do this. They
tell me all the time, Hey, there's a new version Google.
At least you can skip it. But the pop up
always happens at the wrong time where I'm like, can
you interact with me at the end of my workflow
because then I'll accept the upgrade, and Signal just says no,
I'm sorry, you can't look at your messages until you upgrade.
And that's just an incredible pain and I totally get
the other side where people are using the app multiple
years later. I'm just so afraid of it breaking something
or draining my battery life because I can't trust these
apps and there's no way to stop them from interacting
with the Google Play integrations to get battery hell or
send messages, notifications, et cetera.
Speaker 1: Sure that's the I'm the same way, Like as a developer,
I'm like, why are these people not updating? But as
a user, I'm like, I'm not updating that it's working
just fine. There's no way I'm going to risk this
thing breaking.
Speaker 2: I'll tell you a secret. And maybe this is letting
the cat out of the bag here. I upgrade apps
when something looks like it doesn't work, and I won't
know why it doesn't work. So you could, for sure
if you had some way of feature flagging, like a
couple buttons just breaking in a weird way, right, Like
you know, I use Plex and if you hit the
like you know, if I hit the play button and
it like pauses it or you know, starts playing and
it like does it weird jump in the video and
it keeps like it just starts happening, I'll be like,
oh man, I got an upgrade. So that's the secret
for convincing your users to upgrade is really, you know,
have something where they don't think is really your fault,
that you would have done intentionally to get the benefit.
Speaker 1: And now if you productize that as a service, you've
got you next huge startup idea.
Speaker 2: Yeah, graceful like non graceful degradation. And yeah, we like
our core metric for our ICP would be like the
number of broken apps we have in production at that moment,
right our customers are currently destroyed the user experience of
one million users that could.
Speaker 3: Be that's funny, write that down right.
Speaker 1: But meanwhile, back to your end, Yeah, because that was
our chosen topic till we wandered off, I.
Speaker 2: Think it's really challenging.
Speaker 1: Yeah, So in my mind, there's a couple of different
categories of things that you have that come up at
your end. One of which you brought up that I
hadn't remembered until you brought it up is budgeting for
the next year. So you got to start campaigning for
whatever it is you want to do and building a
use case so that it makes it into the budget.
And then like we already mentioned potentially depending on your industry,
increase in traffic for your end sales and special and
you know, for a lot of companies that's just largely
a scaling issue, Like, yeah, I've got auto scaling, it's fine,
but I always like to go ahead and just kind
of cross check everything before we get to that point,
because we've probably made a lot of changes throughout the
years since we last tested this really hard, and so
I just want to make sure that we didn't inadvertently
turn something off that we were counting on or something
like that. And then their category for me is like
ye're in closing of the books, you know, which in
my mind that's that's a financial thing, but I also
think there's an infrastructure side of it too.
Speaker 3: It's a good chance to.
Speaker 1: Check all your backups and your data retention policy and
you know, restore from backups into a dev network and
validate those and just make sure that all of that
stuff is working as well.
Speaker 2: Yeah, I mean, you don't want to be that Australian
pension fund company where they lost all of all of
their one hundred billion dollars worth of financial bookings for
all of the citizens of the country because they were using.
Speaker 3: One cloud platform that.
Speaker 2: Decided to be extra helpful in setting up their environment
one day. So I don't know if you know the
story here, but apparently yeah, so apparently there was an
internal So the first the first knit is that they
were using VMware and on prem which I totally get,
I've been there. But then they wanted to do a
lift and shift, so they did it and were hoping
that the cloud of choice we're going to offer them
VMware as a service, and they did so they shifted
to GCP and the problem was the configuration that GCP
provided for VMware was not sufficient to meet their needs
memory ram zpu, I don't know what it was, and
so they filed the support ticket to ask for them
to basically create a sphere cluster for them manually. And
if you've ever built a technical product that has a
high requirement on reliability or quality, you'll know that the
thing that you really want to do is make sure
you go through the public interfaces. And if the public
interfaces don't support that, then you know that someone's going
to have a bad day sometime future. And realistically, it's
a it's a good reminder where requests from one team
to another one like the support team to the ops
team or product engineering team should be quick to resolve
either know we're never going to do that, or support
it or yeah, of course we'll get that in in
this case updating whatever public API. They had to be
able to utilize it to actually generate the cluster in
a needed way. But instead of going through that in
the good old fashion and we can do it ourselves strategy. Uh,
this is the team responsible for solving this task for
the pension fund. UH ran some scripts to dynamically generate
a cluster using a tool that was built for non
production environments to automatically self implode after a period of time.
Uh that you the default was one year and uh,
you know, probably to stop probably to stop development environments
from consuming a lot of resources after they lost their value.
And it was used for this, and so a year
later after creation of their environment, it disappeared, just as
decided by the engineers of this internal tool.
Speaker 3: Nice.
Speaker 4: Yeah, so you know, if you may need to look
all the way back to what happened last year at
this time and use it as a metric for what
could possibly be going wrong.
Speaker 2: I remember one year at one of the companies I
was working with. We had a production outage soon after holiday,
and that's who we called it. And it was related
to the fact we had a table called calendar table,
and I kid you not, the calendar table ran out
of days and so it crashed production for all of
our one thousand windows services that we had running. And
the calendar table, you know, may sound like it's a
pretty clever thing, but I what it actually was is
a day count number as the index, followed by the
day of the week that that day was, and that
was the whole table. And so if it didn't know
what today was, it wouldn't know what the day a
week it was, so it wouldn't be able to do
planning to figure out how much actual planning, both on
the manufacturing side but also planning how many people that
needed to be in the office manufacturing plan in order
to run the plan effectively. And so the fix, of
course was to insert more dacent the table. And of
course the fix was someone runs a sequel script that
just inserts, you know, one hundred thousand more rows into
the table and no one thinks about this problem. Ever.
Speaker 1: Again, this sounds like mayan calendar as a service.
Speaker 2: I guess I guess. You know, maybe the lesson learned
here is that when it comes to the end of
the year, things are going to break it away.
Speaker 1: Yeah, because there's always always things like that where someone
you know hard coded a year or didn't you know
they're doing like math between months and then so whenever
the month number decreases, it screws something up. And that's
a really good point. Those are things too. I don't
know that you can effectively look and identify those ahead
of time without just a complete audit of every piece
of code that you have, but it's something to keep
in mind after the at the end of the year,
when you go into the new year, when something starts
acting really weird, don't rule those kinds of things out.
Speaker 2: I wonder if there's a framework that could actually be
used and executed here, like like the thing to do
on January second is control shift F through your entire
code base or grock or whatever you're using and search
for like WTF or slash slash to do and see
what actually comes up.
Speaker 3: That's a great idea.
Speaker 2: Just schedule a reminder, like every team could potentially do this.
I mean, I hate these comments. They don't there's so
completely worthless in a lot of ways because it's like
there's something wrong with the code that I'm looking at,
but I don't know, I'm too lazy to do something
about it, or there's just a time bomb here waiting
to go off. At least explain what the problem is, right,
like what am I looking at? Why is this weird?
Why is it confusing? And maybe twods are okay, but
realistically I feel like either it's something you want to do,
in which case you know, file a ticket, or be
like this is intended behavior. And actually one of the
things we do is we add an explicit log statement,
so we don't actually implement those cases, but we do
log whenever there's a weird thing that gets hit, so
it doesn't instead of WTF, it's like WTF, like log
track this message? Why is this even happening? And someone
should investigate so that if we do hit that use case,
we know to do something, and if we never hit it,
no one.
Speaker 3: Has to care. Yeah, for sure, I tend to.
Speaker 1: You know, I can't say I've never created those, but
I have gotten better over time at in the comments
putting heywill or if you're reading this, so that it
kind of stands out and then try to set the
stage so that future me knows why past me screwed
me over like.
Speaker 2: This, Yeah, yeah, I totally hear you. I'm definitely Like
there's lots of paragraphs related of content and comments in
front of things that just aren't super obvious if you
look at them, like why are we comparing this to
the number seven, for instance, and like you don't want
the comments like we compare this to the day of
the week, Like that's not helpful, Like it's helpful when
you explain why from a business standpoint or a product standpoint,
this whole code section is split this way. Like if
you're doing a like a merged sort or any sort
of sort algorithm, or you're trying to find a maximum
value or a minimum value or some arbitrary P value,
Like don't say, oh, we try to find the P
value of ninety nine, Say we want to find the
P value of ninet ninety because and then fill that
out so that you're actually explaining why the code is
the way it is, because often that's not going to
be included in the ticket. Maybe it is that generated,
but usually not. And obviously you don't want it to
jump to that or look at the code comment that's
that's related there.
Speaker 1: Yeah. I think it was Don Felker that had to
tweet a really great tweet that said that comments should
explain why you're doing something, not.
Speaker 3: What you did.
Speaker 1: The code itself tells you what you did, but the
comment should tell you why you chose to do that.
Speaker 2: Yeah, for sure, one hundred percent agree. But I don't
think anyone's going to explain to my previous senior engineer
who is working on a different team in a previous
company at the end of the year, who at my
request to not run production software on their own machine,
decided to run production software for on their own machine
for a critical piece of software for the entire company,
And during a long vacation break in the company, the
cleaning staff unplugged the machine and crushed the service. I
kid you not. And the really ridiculous part about this
story is it was actually right around a holiday. Like
it was not not a good story. The ridiculous part
is we were migrating to AWS at that time, and
this particular engineer, who I actually think was a like
a equivalent of a staff or senior staff right now,
said I don't understand why we're going to aws. We
have working software on prem in all of our data centers.
That's sufficient. We have you know, x or it's running
these data centers. And well, I think that statement sort
of stands for itself.
Speaker 1: Yeah, I'm sure you've seen the meme of like the
laptop where the lids partially closed with the sticky note
on it that.
Speaker 3: Says, this is a production server. Please don't close lid.
Speaker 2: It's not a joke. Like I have seen that in
my career like four or five times now, Like somewhere
I am. That is a thing, And it's always in
places that have a really experienced engineering division, and they
have engineers in not experienced engineering divisions or staff that
aren't it focused who were forced to install something and
run that application themselves. So if you're on an OPS
team and you tell one of your customers, hey, we're
not going to do that for you because that's not
our job, this is what they go and do. They
go and install that production requirement that the marketing software
that ends out all the emails to your customers on
a single laptop somewhere and put a sticker on it
and then actually find out about a problem. Like I
would be sitting somewhere there'll be a laptop there and
someone will go over and unplug it and I'll be like,
you just turn that off? I have no idea why
it was like that. It's weird, Like you should know
that it was weird. Why did you do that? And
then like two hours later, someone will randomly show and
be like, hey, this laptop. Did you see what happened here?
Did someone like turn it off? Like what happened? This
has to stay on? And we're like, I don't know.
Someone just came over and unplugged it and then laughed, right.
Speaker 1: And my favorite part is when that happens is someone
gets really mad that you had the audacity to turn
off that laptop, like how dare you turn off a laptop?
It's like, wait, no, why are you running production software
on a laptop? That's the core issue here. It's not
the fact that I turned it off, it's the fact
that you're running production software on your laptop.
Speaker 2: Yeah, you know. I try to think of it from
a multiple perspectives. If there's a way to some how
put myself in their shoes of they just don't have
an understanding of what the criticalness is or the interim,
the lack of flexibility, the likelihood of this causing a
problem of the actions they're taking. I think human beings
by nature are so myopically focused on short term and
not deal it thinking about the long term. But also,
I mean, we're the ones in the position of thinking
about reliability and the operations of the software that's running
and how to keep it reliable, and you've got to
imagine that other people are not. Yeah.
Speaker 1: Absolutely, it's so hard to look at things from other perspectives.
Oh for sure, because like, whenever I'm doing infrastructure work,
you know, I'm pretty tightly focused on you know, high availability,
fault tolerance, redundancy, all of that kind of stuff.
Speaker 3: But then.
Speaker 1: Like the next day, I can be writing code and
you can look at the code that I'm writing and
be like, has this guy ever heard of high availability?
Because you just get so focused on solving that problem
and fail to step back and think, wait, what other
what other pieces do I need to make this?
Speaker 2: That sounds like there was a story there.
Speaker 3: Oh, there's probably a lot of stories there.
Speaker 2: And it's those that like at the time, you're just like,
I'm all read in the face I'm like, I can't
believe this is happening right now. I have to totally
chill out. And they're the ones that make for the
best laughs later. I mean, maybe it's a little cliche,
but those really terrible experiences that you're you're gonna have
between now and the end of the year, Uh, they're
great stories. Like I got another one. We were the face.
I was running a team that was the face for
our customers U I and a bunch of back end services,
and we didn't get any public praise. Is all about
the perception like our stuff just ran no problem, And
when it came time for the end of the year
and increased load, we would often say, we don't need
to do anything. We've looked at last year and the
year before, and also what's happened in the last couple of months,
and it looks like we have tons of capacity. What
we're going to do, anyway, is just allocate twice as
much capacity for no reason whatsoever and just run like that,
because who cares. It's our infrastructure. We've done a lot
of cost auto optimization, so we don't have to worry
about if a couple of months or even a week
or so we just bite the bullet there and it
was always a good idea, but totally unnecessary in almost
every situation. That's one team now we dependent on. The
reason we were so we were so critical of the
organization is because we depended on a lot of services
from other teams who did not write such reliable software,
who were not as concerned about soft were services that
would never be interacted with directed by the customers. So
we had a lot of things in our code like
automatic retries, acing processing, pre caching, loading from cash like.
We would call lots of services and they would fail,
and then we would have really complex workfloths to figure out,
like predict what the value was going to be in
order to utilize it in most cases where we could.
I mean, obviously you can't do it in every case,
otherwise you wouldn't need the service. However, there was this
other team who had problems. Every single week, there was something.
It was always firefighting one hundred percent, and because it
seemed like they were constantly under pressure, that means there
were also constantly solving problems, which gave them a positive perception,
and when it came to a huge load, their services
would always get pounded into. So the moral of that
story is actually two more teams had to be spun
up just to support the load of when something went wrong.
So a support to teams in a support organization to
talk to customers and deal with integrations, because those software
issues of having a service that's not reliable have real
reaching impacts to customers, and they got rewarded for that,
which has just always been not a thing I've looked
kindly back in my career over but they would say
the most ridiculous things ever where, Like on the surface
may sound really intelligent, but if you dig down into it,
it's like, so, here's one. We know that there should
be high load all the time that we often can't handle,
so when there's no load, in order to make up
for it, we know that load is going to come later.
So whenever we experience a drop in volume, we scale
up all of our services as much as we can
to prepare for that waterfall that's coming next. And on
the surface, it's like, well, that's actually kind of clever.
You know, if you expect one hundred requests per second
and you don't get any or ten seconds, well where
did those thousand requests go? Someone was maybe hoarding them
for that period of time. I mean, that's at least
the thought. However, you also realize, well, like that's not
really a good sustainable scaling strategy. You might as well
just figure out how to deal with the burst load
when it comes in, you know, figure out use asynchronous
software to use services that will actually allow you to
support pulling or pushing that load off and so you
can actually normalize the curve over time. Don't rely on
superstition or tradition to determine what your scaling policy should be.
Speaker 3: Seems reasonable, you know.
Speaker 1: That type of scenario makes me think, and I think
this is specific to the topic of today, of your
end preparation, where you have a lot of different pieces
coming together. There can be certain scenarios where it's useful
to create a temporary team and pull in people from
different disciplines, different teams that are just focused on that
temporary problem. You know, like say, okay, we know that
we're going to have a massive increase in traffic because
of the holiday sales this year, and that's going to
impact that's going to be driven by marketing, but also
it's going to affect the mobile app team and the
back end API team and the infrastructure team bears the
brunt of that. So rather than try to manage all
of those separately, it's often a good good time to
just pull key people from those teams into a temporary
team and set their scope. Your scope is to manage
this incoming load and make sure that we're successful at
handling it from our customer's perspective. And then after that
event's passed, the team disbands and goes back to their
normal job.
Speaker 2: Yeah. And I think a product a project team in
that regard, in that situation is like the one time
where it actually makes sense. And as long as they're
not they're not building anything. They're actually trying to prevent
problems by using only what's already available. And because there's
no long term output really from that process, you don't
have to worry about disbanding. And at the end, like
that's a really good idea.
Speaker 1: Yeah, And I think one of the reasons it's successful
is because you clearly define the scope. And when I
talk about defining the scope, I'm talking about not only saying, hey,
this is what you're going to be focused on, but
you have full authority to say no to everything. That
is not that thing, which is really important. I've seen
that a lot throughout my career. We say yes to
things almost with like this hidden assumption that there's always
more bandwidth available, So we'll do everything that we were
doing plus this new thing. And at some point you
reach critical mass where you in order to say one
more yes, you have to say one more no to
something else that you're not going to be doing.
Speaker 2: I actually put it stronger. Everyone should be at that
point already right now, whether you think you are or not.
Like if you like, even if you think you have
three pre capacity, you probably don't. I know it may
feel like that, but realistically, I know as individuals we're
very bad at evaluating our own free capacity and be
able to do things. And if you actually had free
capacity to do extra things, do the thing you're doing
more like faster, like why is it just not already
getting done? There's no reason to put something else in
that spot. So I think realistically, everything you do something
next like I would always use this, especially for inexperienced
engineers at the beginning of their career. You know, we
should solve this, we should fix this problem, we should
refactor this code, we should switch to. Now it's Rust,
which I can finally get behind. But it used to
be you know, have assembly, or we should switch to
whatever fancy new version of React that's out there. I'm
so happy we don't use React office for so many reasons.
There are so many better alternatives. And I'd always say
the same thing, like, you know, great, we can absolutely
do that. Which thing that you're working on right now?
Do you want to stop working on so that you
can work on that instead? And really having that priority
first conversation, because well, most of the time of the
year may actually be practiced, it doesn't matter if they
pick up this extra thing. It really helps to think
about the priority order. So when huge important events come up,
like the end of the year closing or whenever you're
end of the year closing in, or if you have
a big marketing event, whenever that happens, whether you can
know or not, it really is the production version of
can I prioritize effectively?
Speaker 1: Yeah?
Speaker 2: For sure.
Speaker 3: So that's a.
Speaker 1: That's a it's sort of along the same topic here.
You talked about, you know, Rust, and a lot of
times as we talk about end of year and we're
looking at projects for the coming year, and there a
lot of times there is a topic, you know, like
oh we should we should rewrite this in rust, or
we should rewrite this in you know whatever.
Speaker 3: You know, you can pick whatever.
Speaker 1: How do you go about making the case for that, Like,
if you're campaigning for time and budget for the coming year,
how are you going to structure that campaign?
Speaker 2: I mean I get to make those decisions, So I
guess I should. I'll I'll put this maybe in the
flip side of like what I'm looking for when someone
comes and excellent point to create this whatever that change is.
I mean, I don't feel like I go out and
hire engineers, Like that's sort of like a skill capability
you have. Maybe you're systems thinker, like you're able to
understand how doing one thing impacts doing another thing. Yeah,
for sure, you can write in some a couple of
different languages, But realistically, the teams exist to solve some
business problem. That's why they're there. And so looking at
the business problem short, medium, and long term is what
I care about. So you know, tell me a story
that solves short, medium and long term problems. So maybe
there's a problem today and it needs to get resolved,
and so you know, you want to deliver quick little
improvements today, you know, what does that look like to
solve that? And then medium you want to do you know,
a bigger project to make sure that we never have
to worry about that again. And long term, like why
are we even discussing this problem? Like this problem is
too small scale for us. It should get automatically solved
by what we have, by our infrastructure, by a single
new feature ticket. And so think about conveying those things
when discussing potential ideas of what to work on. So
the change to rust, talk to me about error budget,
Talk to me about speed of delivery for new features.
Talk to me about maybe they're all ability of what
we have, the quality of what we have, the ability
to deliver new things based on the frameworks that are available, etc. Etc. Right,
the actual impact of doing that and nothing comes for free? Right?
You know, I want to know the cost of if
we make this switch, how long are we going to
pay paying the cost? Short term, medium term, long term?
And I got to see some benefit there. Now. I
don't expect a perfect financially planned strategy for every single
part of the process, but some thought has to have
gone into it. Right, how many engineers know about this
programming languages? How many of them want to know hiring
in the market is.
Speaker 3: A challenge there?
Speaker 1: Right?
Speaker 2: Maybe maybe not? Do we have big projects coming up
where we already started with something we have and now
we need to potentially put that on hold in order
to do the switch, or we'd have to write it twice. Right,
So they's throwaway work, and an understanding of that tells
me that you've thought enough about this problem that I
could let you go and run with it, can delegate
it to you effectively. Otherwise you're basically saying I want
someone else to solve this problem. Right when you come
up with a problem and you share with someone else, Hey,
we should do this thing, try to figure out what
you're doing. Right. Are you complaining about it I don't
like X, or are you saying I want you to
fix why? Or are you saying I want to do this?
This is what it will look like. This is the
impact to the team and the organization, and then I'll
say great, sounds good, or if it's a bigger impact
of the business and some way, it'll probably get into
our initiatives for the next quarter and our quarterly planning
and they will just show up there and someone who
is most relevant to be working on that at that
moment may or may not be you. Right, you know,
just because you came up with a good idea doesn't
mean that it makes sense for your team to work
on it. We'll take over and start running with it.
Speaker 1: Yeah, for me, the starting point of all of those
conversations has to be one of two things. How does
this increase revenue for the company or how does it
decrease operating costs for the company?
Speaker 2: I think you're maybe being a little bit too unfair there.
I mean, I totally agree that should be the actual
impact at the end of the day, Like, it has
to be one of those two things realistically. I mean
there's also social capital as well, right, Like, maybe you're
scaling up longer term and hiring additional people is important.
So you know, maybe you've got PHP or Ruby you're
using and you want to sell it's difficult for us
to get engineers. It's difficult for us, you know, two
or three years down the line, we're going to have
to make this switch at some point. It has nothing
to do with the operating cost today or making future
making revenue right now, but longer term there's a lot
of trade offs. I it needs to be thought about,
but like you don't have to come to the table
with like this is the number of dollars that we're
going to say now maybe, I mean, will your boss apparently,
I mean it's much better, right, you know, if you
could actually do some sort of analysis, you know, feel
free to take that. But it's sort of the thing
where like a pull request on architecture, you don't need
to come with the code perfect all written for the
whole feature. You know, I have some idea. Should I
even start thinking about what the next step is here?
Or should I forget about it? Like, you know, give
me some feedback because this is a good idea to
switching to RUSS maybe a good idea. And I will say,
could be maybe you know, tell me about it. And
then you say, I actually don't know, let me go
do some investigation. I'll be like, okay, sounds good.
Speaker 3: Yeah. Now I'm still stuck on those two I think.
Speaker 2: You know, for a long time I was as well.
The thing that broke me was companies that focus on
some sort of social good which is neither necessarily cost
reduction or revenue generation. So if there's a tragedy of
a common situation where you want to get out from
under it, you know, take the climate crisis, right, we
have a thing that's neither revenue nor cost. I mean,
of course, you can model the the criticalness of our
direness of our human society at this moment, it's a
huge problem. You could measure the rate of temperature asmospheric
increase or oceanic temperature increase as your metric they're going after. So,
I mean, whatever it is for your business, right, you know,
if you're making money, then for sure, I'll say profit,
not necessarily revenue. Both of them are bad metrics for
different reasons, depending on how you're abusing them, right, And
you're right startups care about revenue, not profit, and real
companies care about profit and not revenue. But you could
just be making a lot of money a return on
investment that really isn't worth it. So you know, what
is the important thing for your company? I totally agree with.
Speaker 1: You, will, yeah, for sure. And that's probably something that
happens in my mind that I don't express enough is
whenever I talk about currency, it's not always dollars. It's
the product itself. Like if you're an open source company,
you don't sell your software, but you still have a currency.
The currency is the number of people who download and
use your software and build on top of it. So
when you talk about increasing when I talk about increasing revenue,
it may not be US dollars or euros, it's whatever
currency measures the success of your product.
Speaker 2: So Solona or Ethereum.
Speaker 1: Yeah, or Polygon, you have your you have your own
toke gun we do yeah, Matic okay, yeah, yeah.
Speaker 4: So yeah, that one, that's that's that's the most.
Speaker 2: Important currency to to actually own, so to measure everything
in madic and then you'll know whether your idea is
a good one.
Speaker 1: But that's that's a really good point because as we're
an open source company. Yeah, and so for us, the
currency is getting people to build on the Polygon network.
Speaker 3: Matic.
Speaker 1: The token itself is actually a byproduct of that. That's
just a an exchange ticket that all of these people
who build on top of our network use used for transactions.
The success of Polygon itself is building a network that
enables people to do that. So for us, revenue is
the number of people adopting the Polygon network as their
blockchain framework or blockchain.
Speaker 2: I mean, I mean it is money in a way.
Each transaction that happens is you're getting the revenue out
of that that the number of transactions can be your revenue, right.
Speaker 1: Right, Yeah, And the actual ematic monetary mattic associated with
that goes to the validators who Polygon is some of
those right now, but mostly is other people. So we're
we're even only partially affected by that, with the longer
term goal of not having anything to do with that.
Speaker 2: I mean, arguably it's not even necessarily relevant for you.
I mean, that's happening outside. You're a contributor in some way,
and so whatever that is is more important than what's
necessarily happening in the network. I mean, well, the network
is super critical and important for success, it may not
be something you can directly affect, right.
Speaker 1: Cool, So all right, closing thoughts on your end.
Speaker 2: You know, I never met an engineering team where if
I said, like, you know, today, you've got to do
everything differently because it's a particular set of numbers on
the calendar, and then actually went and made some fundamental
changes in their process. I don't think it really works
that way. Unfortunately, like you it's not like you change
how you're doing code reviews. You're not all of a
sudden more secure. You don't automatically change what you're doing,
or maybe you add extra reviewers. You don't change the
values of your team, the identity, or your processes just
like that. I think preparing for this is a cultural chef.
It's what happened throughout the whole year that got you,
that makes you prepared for situations like this. As well
said testing your backups, other out of band processes that
you want to validate, or potentially building specific teams that
have an identity that matches the execution you want. But yeah,
telling software developers to you know, don't commit as much
code it doesn't doesn't whole lot of water.
Speaker 3: Yeah, I would agree with that.
Speaker 1: I think for me, the big takeaway is it's a
good time of year to pause, think about it, figure
out how it applies to your situation, and then take
action away from that may have an impact, it may
not have an impact, but I think it's worth taking
a moment to just stop and think about it because
we get, you know, so focused in the day to day,
like today, I'm focused on these tickets and here's the thing,
And so I think the big takeawy for me is just.
Speaker 3: The call out.
Speaker 1: Today's ticket is to take a step back and look
at things from an entire organizational perspective.
Speaker 2: I mean, I think that's a really good point that
is lost. I was reading a paper about improving based
on training and the tickets. The standard work is sort
of your production mode. You're doing the work, you're delivering it.
In order to become better or even sustain yourself, you
need a practice mode. You need to go where there's
it's safer to learn new things even and when do
you actually deliberately go and do that. If you don't,
then you're going to just you're burning your ability to
deliver effectively. And could be a good reminder that at
the end of the year it's time to start doing
some things deliberately that's different than your standard strategy. Actually
thinking about how you do that work or what it is,
and even a simple thing like validating some of your
processes huge benefit.
Speaker 1: Yeah, for sure, even if you're Yeah, maybe especially if
you're junior in your career. It's a good training exercise
because as you advance in your career, these things are
going to be more relevant to your role. So it's
a good chance to think about that and then have
that conversation with your manager or your boss and start
gaining some perspective that way to help yourself.
Speaker 3: Later on in your career.
Speaker 2: Yeah, for sure.
Speaker 3: Cool. Should we do some picks?
Speaker 2: Yeah?
Speaker 3: All right? Which good?
Speaker 2: So I've been playing a lot of one particular game
since the summer Steams sale. I bought a couple, and
I mean, I am I just like keeping old apps
on my phone. I'm playing games from years ago because
they're cheaper, and I don't mind playing out of date content.
It's like thirty more years and I'll probably play Breath
to the Wild. Okay, So my my pick this time
is actually a game called frost Punk. It's the best
way I can describe it is it's a rts against
the cold weather climate, and you, instead of trying to
defeat an alien race or some other civilizations that are
out to attack you, you need to deal with the
temperature rapidly decreasing in a ridiculous scenario, and so as
a resource collection and et cetera. Except but you know,
civilization building, it's not not that you like you link
get to like six hundred people or something so good
number And it's not technically an RTS, but I think
RTS against the weather as how I'll categorize it right
on cool, it's good, it's really stressful. I'll say, like
I I will go at like actually during the day
because I played it into the night, you know, And
I'm not signed kind of person that gets stressed over things.
But if you ever, like worked, and you said something
that you felt like was stupid to your manager, and
or you didn't you didn't think it was stupid, but
then your manager said something it made you feel like
what you said was stupid, and then you think about
it all the way into the night and while you're
sleeping on it. Like, that's what happens to me with
this game.
Speaker 3: Gotcha, I am.
Speaker 1: I don't hardly play games anymore, and I do miss
it because I used to love playing games. But the
few times I've gone back to play games, like, my
skills are so rusty. I just suck so bad that
it's not even fun. The exception to that, I'll have
to actually make this my pick. Did you ever play
the half Life series?
Speaker 2: I did not.
Speaker 3: I loved the half Life series.
Speaker 1: That was just such a great game for me, and
I've got the Oculus VR headset, and so there's a
new relatively new this game is several years old.
Speaker 3: Now. There's a new addition to.
Speaker 1: The series called Alex that's in VR, and the game
mechanics of it are just so well done. When you
put on the headset, whenever you pick something up, you know,
you get the little vibration slap in there, and it's
very intuitive, you know, you put the headset on and
within like thirty seconds top, you're completely immersed in this world,
having tuned out the real world that it's just that
well done. So, wow, that was a cool game.
Speaker 2: So no Half Life three, but there's a subset for VR.
I mean, that's gotta be the biggest joke of them all.
Speaker 1: Yeah, yeah, no half Life three, but there's four add
on expansions for Half Life two and the side quest
half Life Alex.
Speaker 2: Wow, that's that's a very ridiculous.
Speaker 1: You know.
Speaker 2: I used to play all sorts of games, many different kinds,
and I never really figured out what time of gamer
I was, And over time I realized there are certain
things I just don't want to feel like work, and
so like a lot of games I realized just became
work like like RPGs that are the JRPG style just
felt like like boss grinding, just so much work. Lots
of games just ended become like resource management games, just
like a lot of extra Like I just spent all
day thinking about architectures and software and then I go
home and I'm playing basically a game where I have
to do something. It's just a huge nightmare. And so
like trying to really narrow down the types of games
I play. I did play fps IS for quite a
long period of my life. Though.
Speaker 3: Yeah, I hear what you're saying on that.
Speaker 1: I'm the same way, Like I don't want to be
challenged in the game. I'm not here to level up.
I'm looking for more of like an interactive movie. No interesting,
that's my gaming style. I want to be entertained.
Speaker 2: If there's no plot to the game, that's like immediately
I don't know why I'm playing this as well be
playing with a fidget toy. So that's for sure number
one requirement. The second one is it's gotta like I
almost don't like combat, honestly, like it just it feels
so as like an extra chore. Puzzles, I like, I
like puzzles in a game, so like those two things
super important for me. But most of the other things,
like if I got to do the same thing over
and over again, I'm just it's not for me. So
it sort of eliminates a lot of things, like I
don't know how people do the rogue like games like
The Diabolos, which you just go into the dungeon and
murder a bunch of things over and over again.
Speaker 3: For hours time Slaughterfest.
Speaker 2: I mean, I've done it. You know, it's sort of
interesting the first couple of times, but after that, it's
like I've seen the story, I played the game, is there?
I mean, maybe this is a second ending somewhere. You know,
it's got to be something at the end that you get.
I used to be achievements driven, but after I realized, yeah,
it's pretty much just extra work.
Speaker 3: I don't get anything for it.
Speaker 2: Then. I mean, if a game pulled me, if I
get through all the achieved, they would give me like
like a tenth of my money back. I would totally
do that. I would play the game longer. I would
definitely play all the game through for sure.
Speaker 3: Yeah, would you do it?
Speaker 1: For like social validation achievements like you get this, you
get this special icon on LinkedIn or whatever.
Speaker 2: I did the Xbox achievements way back, a long time ago.
I did. I tried to do all of them. I
used to be a completionist, and the game that broke
me was Assassin's Creed. I absolutely love the game. I
know everyone's going to telp me. If you've ever played, oh,
two and three or three and four, whatever, you know,
the ones with scot artillery are so much better than
then what was his name, Desmond playing al Tayer, But
I'll say that much better. But there were a thousand
of these flags in the game. We had to go
around and find all of them, and there's no indication
of where they are or which ones you've gone. So
even if you know where all the flags are, you
have to go to every single location in this giant
open world in order to actually find them. And I
just like, after the third time of going through literally
the whole game, in every single place where every flag is,
and I still couldn't find some, I'm just like, that's it.
I'm done. And I realized, like, who is looking at
these achievements like they're a sort of for me, like,
I don't have like tons of friends who are like,
oh wow, Warren has all the achievements for all of
these games like social validation. I mean, what is that so.
Speaker 1: Supermodel sliding into your DMS? Hey, I saw your achievement there.
Speaker 2: I mean, if that happened, I think I'd be a
different person today. But so I guess it's safe to
say that never happened. I mean I don't know, you know,
I guess I wish I had lived your life will
where you know, that was something to look forward too.
Speaker 3: That keeps the voices in my head entertained.
Speaker 2: So that's what's important, right, Well, there are all there
are all all the voices.
Speaker 1: In your head. Sure, yeah, go with that, all right, cool, Warren,
thank you Free Time. It's great with you today, and
thank you for listening to this episode. And we'll be
back next week. I don't actually know what next.
Speaker 3: Week's topic is. I have no idea, but uh, you
will all be surprised.
Speaker 1: Yeah, all right, thanks everyone, We'll see y'all.