How'd you like to listen to dot net rocks with no ads? Easy?
Become a patron for just five dollars a month. You get access to a
private RSS feed where all the shows have no ads. Twenty dollars a month,
we'll get you that and a special dot net Rocks patron mug. Sign
up now at Patreon dot dot net rocks dot com. Hey Carlin Richard here.
As you may have heard, NDC is back offering their incredible in person
conferences around the world, and we'd like to tell you about them. NDC
Copenhagen is happening August twenty seventh through the thirty first. Go to NDC Copenhagen
dot com for more information. NDC Porto is happening October sixteenth through the twentieth.
The early bird discount for ADC Porto ends July twenty first. Go to
Dcporto dot com to register and check out the full lineup of conferences at NDC
conferences dot com. Hey there, this is Jeff Fritz, the purple blazer
guy from like or Soft, letting you in on a little secret about my
friend Carl Franklin. You know, the guy who started dot net rocks.
The first podcast about dot net in two thousand and two. The guy who's
been teaching Blazer on YouTube since twenty twenty, Yeah, that Carl Franklin.
Well, Carl's joined up with the folks from Code in a Castle to teach
a week long hands on Blazer class at Are you Ready Get This? At
a castle slash villa in Tuscany. It's sort of a luxury vacation with Blazer
learning built in. Carl's calling it the Blazer master Class. You'll learn Blazer
from the ground up, finishing the week with the ability to build and deploy
Blazer applications. Since the training happens for only four hours in the morning over
six days, you can bring your significant other, your partner with you and
you should right This part of Italy is absolutely beautiful. There's so much to
see and do, and in Larion Marco from Code into Castle are organizing daily
activities both at the castle and in the area. The castle is in the
Marema, a less touristed region of Tuscany, offering both classic Tuscan hill country
as well as easy access to the Etruscan Riviera, with sublime local food,
wine and olive oil around every corner. Breakfast is included. Every day there
will be two communal dinners at the castle book ending the experience, and most
other meals and all activities are included. And did I mention you'll learn Blazer
in person from Carl Franklin. Listen, space is limited and for very good
reason. This is quality training in a beautiful setting. Go to code in
Acastle dot com slash Blazer twenty twenty three. That's b L A z R
two zero two three to take advantage of this amazing opportunity to join Carl in
Tuscany for an unforgettable week of La dolce vita while advancing your programming skills in
this important new technology. Welcome back to dot and rocks. This is Carl
Franklin and this is Richard cavill Man. You know I got a story for
you. Oh my new studio is coming along. Have you seeing any of
the videos I have seen? You know what I saw? I saw red?
Yes, you do? Like the red walls. I do. And
it's the same exact fabric that I had at Plato point zero. Interesting,
Okay, the same exact stuff and it's that was really two point zero right,
No, No, that was one point well, okay two point oh,
yeah, I guess. I mean that the old State Street place you
had it was the training space first, then it became the studio space,
and then you have built this studios. Yeah, yeah, you're right,
like you definitely went through some iterations. We might as well say studio one
was my basement in Waterford where night arguably community. And now you're staring at
five. Yeah, I just call it. I call it PLoP three point
zero. Okay, I'll buy that. Yeah, we'll buy that. The
third version is the best version anyway, everything you know. Yeah, anyway,
putting in the floor in a couple of weeks. The split HRAC system
is going in on Tuesday. Nice and the band is going to start rehearsing
there in just a few weeks. So I'm really excul that's really great.
Anyway, let's start with better no framework, roll the clar music at awesome.
All right, what do you got? This is um something I know
you're going to be interested. You probably already know this because you know everything,
but it's a It's an article in Electrek Dot and CEO. Sweden is
building the world's first permanent electric road that charges moving evs that's cool, isn't
that cool? Yeah, And there's a picture of a bus and it's driving
over this blue strip of charging some things in it. Magnetic charging, I
don't know, you probably know. Yeah, it's probably cute style charging or
just impedance or field style charging, which is final. They'll only charge so
fast, but you and it's pretty smart because you know, it doesn't have
to charge really powerfully because you're constantly going over it. Right. Yeah,
if every foot you get a little bit of charge, enough to go at
least a foot, then you're positive. Right, Yeah, I don't think
I don't think you're gonna be that net benefit, but hey, anything that
adds a little range, yeah, you know, it stretches a bit further,
so much the better. I mean, clearly, this project is an
experiment. They're talking twenty kilometers, right, so it's one piece of one
thing, and who knows how many more they're going to do. But you
know, obviously after their experiments they said, hey, this piece makes sense
that we're going to keep doing this and we'll see what happens. It's great
to see, you know, infrastructure come into play. Not to derail the
conversation, but you also saw Ford signing on a now GM signing on with
Tesla to do the North American charging standard like this. Here's an interesting point.
The model team released in nineteen o one, first gas station opens nineteen
thirteen. Wow, what do they do for twelve years? Yeah, I
don't know. You bought your gas in containers from the chemist. That's what
you did, chemists. Yes, well you know what we now call a
pharmacy, but basically exactly. But I mean to that point, you know,
watch how these things emerge, the idea of the standard eyes gas filler
that you know, one gas station had different tanks for different kinds of fuel.
Right, you can get diesel, you can get we have to do
a new geek out on electric cars and electric transportation perhaps, but you know
they we're at this point now where it's like, hey, we want standards
around this, we want these things to work correctly. So again, the
experiments continue. This This electro tech article is a great example of that of
more experiments. Well, you know, it's a nice experiment, but will
it scale? That's see what I did there. I like that. I
know it's very clever. Are you're the one trying to stay on subject?
That's weirded out, all right, who's talking to us? Richard grabbed a
comment off a show seventeen seventy eight. That's the one we did back in
twenty twenty two about pro micro services and dot at six. That was Sean
white Cell, Rod Richardson and Matthew Groves. And this comment comes from Mark
Essex, who said, Wow, what a timely podcast. We're in the
early stages of hoarding our monolithic web forms app, which has been in production
for twenty plus years in some form to micro services. I immediately went out
and purchased the book and I will force myself to make time to read all
four hundred and eight pages. Well, I'm sure Sean and Robin Matthew will
be to listen to this show first. Yeah, I thought discussion was so
relevant, really hit on some key points that we've been discussing internally about areas
of concerning issues it might arise. I'm scared to death. I'm about how
to debug this thing, or to test it, or to ensure the same
quality that we've enjoyed with our current application. I mean, if you've running
an app for twenty plus years, like people are pretty used to it,
right, and as somebody else mentioned in the comments. I really hope we
don't have to decide to go back to our monolithic app as that would have
a whole other set of challenges. So thanks so much for a great podcast.
Yeah, with any luck, Mark, you'll be that. You know
that was a year ago admittedly, so you're probably pretty far down the path.
Maybe the show isn't timely enough for you. But here's to hoping.
We're certainly having a conversation about you know, there is no one perfect solution
for anything, and there's a bunch of ways is all of this problem,
and that's certainly what this show will be about. Yes, sir so Mark,
thank you so much for your comment. And a copy of music Go
Buy is on its way to you. And if you'd like a copy of
music Go Buy, write a comment on the website at dot net rocks dot
com or on the facebooks. We publish every show there and if you comment
there and everything in the show, we'll let you a copy of music Go
Buy. And you should definitely follow us on something. You can follow us
on Twitter if you want to, we were there, or Facebook. As
Richard said, but the cool kids are over on Mastodon, right now I'm
at Carl Franklin at tech hub dot social, and I'm Rich Campbell at Mastodon
dot Social send Institute, Rudy Tooty, Fresh and fruity. Okay, let's
bring on Derek. We actually started the conversation before we started recording, so
we'll probably pick up there. But Derek Comartin is a software developer. Over
two decades of professional experience, He's written software for various business domains such as
consumer goods, distribution, transportation, manufacturing, and accounting. Derek has a
very active blog on YouTube channel called code Opinion, which focuses on software architecture
and design. Welcome Derek, thanks for having me. I wanted to say
I'm a first time color longtime listener, but I was not sure if anybody
would realize what that is. Yeah, well we get it. School.
Yeah, longtime listener, first time Yeah, I love it well. And
I'm a longtime reader of your blog, sir. I often tweet out my
favorite of your blog. Yes, yeah, I appreciate that. Yeah for
sure. So scaling in general is seems like an insurmountable task when you know
the boss comes and says we have to talk about scaling. Uh, okay,
because we can't. We can't imagine to what degree we need to scale.
And most companies think, if they're public facing websites, for example,
they think that they're going to be the next Netflix or the next Amazon or
something, and so it may be completely over architected and completely overextended. And
then you have the companies that are small now but think that they will grow
to be these big companies. So we want to be ready to scale,
and that may involve a lot of extra architecture developers don't want. I mean,
there's just so many gotchas in scaling, and I guess that's why we
brought you on to help clear up some of this nonsense. It is like
it's an interesting topic because, like you said, there's this kind of this
notion where right from the get go, we need to decide now in stone
exactly what it's going to be so we can be the next big thing or
that we think we might be. And I think there's just this right now,
even though we've been swinging back, I feel like a last couple of
years worth of this. Okay, we've been doing the kind of getting more
focused on micro services, like we're talking about the comment, and I think
it's just it's viewed the wrong way. I think it's you can go pretty
far for if you have them onolith or if you're starting that way, if
you think about how you're building your app, not necessarily in a physical sense
right away, but more in a logical sense, how you design it that
way, you can then decide how you want to scale. And there's a
variety of different ways to think about scaling. I mean, I think primarily
if you were to stay cheese, I don't know, fifteen twenty years ago,
before a cloud, you would it was a little bit more difficult in
the sense that you were thinking, okay, we got to buy new hardware,
and that was a little bit more difficult to scale up or scale out
potentially. But now it's just a little bit easier. But the kind of
the same thought process is still it's still the same for me. It's thinking
about just so the traditional things of are we scaling up, are we scaling
out? But they all kind of have implications about what I really call moving
the bottleneck. Usually when you're changing one thing down the road, you're likely
changing something else. So it's it's kind of an evolving thing. That's why
I think it's a little bit of a fallacy to think up front, oh,
we're going to create microservices and that's gonna just solve all our scaling problems.
When scaling of water actually we're talking about there, because in most times
that's more of an organizational thing than it was necessarily you said the word microservices.
I could tell that a million sphincters just tightened. I thought I saw
your eye twitch. Actually, carl As, we've got video because of the
sphincter tightening. I think you know, it occurred to me. It's like
we should switch to microservices is like the new version of we should have written
this in Ruby on rails right Like it's it's the hey, we can't fix
this problem, so let's come up with a solution so complicated that you'll leave
us alone for six months. It's interesting because we were kind of alluding to
this before we started, which I was bringing up months ago. I can't
remember how long ago it was. There was Amazon Prime Video had a video
or had a blog post about kind of their transition from using lambda's aws lambda
and a combination of S three and some other things, and they migrated that
to just be containers in ecs, and there was seemingly kind of an uproar
of various opinions, including my own, about what that really was, which
people were saying, well, they went back to a monolith, which even
in their own posts they did, but I kind of said, it really
wasn't And there's a lot of people claiming, oh, well, how does
Amazon doesn't even know AWS doesn't even know really what lambdas are good for,
And it was just a lot of hot takes I guess about Okay, they
had this service specifically for I believe it was for kind of trying to determine
the quality of a stream and letting know the streamer know, like when quality
was degrading, etc. And it initially said in the post that it was
just it wasn't designed for the scale that it ended up becoming, which is
okay, great, I guess, and they evolved it. It's like,
what's so terrible about deciding, Okay, well this worked at one scale,
but it's not currently how it is now in the volume that we need.
Yeah, it's funny how people want the one right way right, like they
well, if it doesn't work for this, and it wouldn't work for anything,
right, I mean I read the original post and what my immediate thought
was, why were you using servi lists here? To me, the reason
you servillists is you you have an elastic load that has to abruptly scale up
a lot and abruptly drop back down. The audio video consumption on Prime Video
is probably pretty high all the time. It probably goes from high to really
freaking high, Like what do they need that, I'm out of elasticity for
well, as LAMB does though they they can shut down, right, Yeah,
that's the whole thing, right is it very elastic? But it also
takes time for them to spin back up again, so that could you know
very few people, you know, the first person that has to wait for
that has to wait. Yeah. And another piece of that was from what
I got from it was just the cost in the issues of moving data back
and forth from those landas two S three like back and forth, like pull
them that data from there. Yeah, so going to ECS and being a
container, you're pulling it once they're running their detectors. It's all at that
point in memory, once they fetched it once, rather than the however many
times they needed to do it. And in theory serverlists should have if you're
used to service properly, it saves you money because you only use instances when
you need them. But it's got to be the shape of the workload,
right, And I guess as they understood the shape of their workload, they
could implement a lower cost solution for that workload exactly. It's just it's I
always say context this king like at the starting of it, maybe I don't
know what their situation was other than what they wrote in the post. It
just it didn't seem like the initially the solution work because it it'sn't at the
scale that they were headed towards, and they had no intention to it.
So it evolved. It's like, yeah, okay, great, you evolved.
It like makes sense to me. This is like saying what you're a
teenager, childhood's a waste of time. Yeah, it's a great point.
And it is really like like you said, it's a hot take and then
and a hot takes in general are incorrect. I think anybody's actually built and
cared for software, Like I spent a lot of time helping people scale,
and you always start the conversation with corridulations. You have a good problem and
enough people are using your stuff that scale matters. If nobody was using it,
it'd worked great, exactly when you kind of in for I would say,
I'm making the assumption here that it's not an abrupt oh oh my gosh,
I need to scale like from zero to one hundred. It's usually kind
of you know, I mean over time as you're gaining traction, if some
type of sas product, whatever the case may be. And usually when that's
coming up, that's when you're trying to determine what needs the scales. So,
like I was mentioning, if it's it could just be scaling up kind
of that app service that you're running. So let's say you're just running some
web app, some ht B API, whatever the case may be. Maybe
that first step is okay, just scaling up, giving it more resources,
giving it more memory, whatever it's kind of starving on. But like I
said, when doing that, usually the next step it might not be instantly,
but it might be okay, well, now we can handle more inbound
requests. So what is that going to affect downstream? Right nowadays, it's
not just usually some app running, it's app running with obviously a database,
some other infrastructure, there's a lot of moving parts, and usually when you
scale, when you fix that one bottleneck, let's say it's at the very
beginning of that app compute, you're likely gonna start seeing a bottleneck somewhere else
and that yeah, in some other infrastructure. So then you're kind of tackling
that piece of the puzzle if you need more performance, right, like,
yes, absolutely, yeah, And there's a lot of ways of handling that.
It's I think it's just it's not immediately. I think when people get
into it, and it's kind of a slow process of having to fix each
individual place that may be experiencing performance issues if you will or you can't handle
that type of load. I think intuitively people start figuring it out of where
what to fix where, or whether it be like it said, scaling up,
scaling out at adding more instances. But I think that a bigger challenge
oftentimes is more related to infrastructure that you don't necessarily like it's not your app,
so that whether that be your database and depending on what type of database
you're using. Right, So it's easy to say, okay, well I'm
just gonna scale up my relational database. It's not so easy to all of
a sudden say Okay, well we're going to introduce a read replica and that's
just gonna be no problem, right, And that's a major re architect Like
that's it can be your data quite differently, you want to talk through a
read replica or folks who haven't dealt with it exactly, Like, it's not
all of a sudden, it's well, I always say this about cashing.
It's like I always get the sense that there's this people talk about it.
I just I mean, add some cashing problem solved, everything'll be fine.
It's like I always like people say that cash invalidation is a hard problem.
I don't think it's like it's it's a hard problem, don't get me wrong.
But in the sense of an application like these types of systems, it's
I find it more of an issue of just dealing with a cash in general.
There's so much that can go wrong with it, sure, right,
so well, and it has direct impact on business, right, Like every
airline consolidation site you've ever used doesn't check with the airlines every time they show
you a flight, they've cashed all of that, And every so often you're
going to click on a flight for a price, and then it's gonna go,
oh, sorry, I don't have that flight. Then maybe it's one
percent of the time they're going to annoy you that way, and they've decided
that's sufficient. I dealt with an inventory system that was too slow and they
wanted to add cashing to it, and I said, that means occasional you're
going to sell stuff you don't have. What do you want to do about
that, right, And the answer was, we put in a back order
system, Like we accept the fact that we don't have it, and so
we now have. We never had back orders before, we'll add back ordering.
So if you've ordered something that wasn't in the we're still in the cash,
but it wasn't on the shelf, it's like, hey, really sorry,
we put you on back order. If they choose to cancel whatever,
but at least we're doing investor can and we literally change the business flow to
compensate for performance exactly trade offs, right, And a lot of the stuff
is usually built into in most cases, like if you're talking about whatever industry.
I love that you brought up that example because I always said that if
you oversell something like in distribution and warehousing, it wasn't a sales problem.
It's a purchasing problem, right, so they usually have stuff in place for
this to begin with. But whether it would be a read replica that you're
introducing a cash when we're talking about something that's hivan stale data and how you're
going to deal with that. But even just like to touch on the cash
again, it's even from a technical point of view, is it always available?
Is it responding to you like you're using a cash typically because you don't
want to reach out, you want to offload some of those requests from your
database. What happens when the cash isn't available or it's not as responses of
you do want it to be right, right, guess what you're doing.
You're usually going back to the database. So right, so if all of
a sudden everything's sailing great one day and then there's hiccups, there's performance issues
with your cash. Now all of a sudden, these requests that we're hitting
the cash or cash misses and they're going back to your database. You just
flood your database with a pile of cash miss is that it normally isn't expecting
Yeah, let's talk through the read replicub model, because I think it's really
interesting for folks, and I don't know, I don't know there's a product
off the shelf that can let you do read replica, Like you have to
code for this, this idea that you're writing to one database that then is
synchronizing to one or more reading databases. Yeah, I mean there's options to
have some type of proxies there so and then rules built around that so you
can be offloading them. But it's just still a matter of if that re
replica is a synchronous and it's eventually consistent, then that's more of the issue
than necessarily like you need to know it. The whole world has been taught
that by Facebook because Facebook clearly uses read replicas, and when you didn't immediately
see your post, you would post it again and then you see them both.
Like everybody knows about that problem now just because they learned Facebook. Yeah,
I think there's definitely something to it in terms of note like people are
a little bit more familiar with with it, whether they're technical or not,
that can sometimes probably guess what's happening. Yeah, but still it's not the
greatest obviously, and I think it's similar whether it's a re replica or whether
it's a cash they're both are potentially stale at any given time. It's like
it's contact specific. Most users, I would think, in your example,
if they performed some right, which they may not know they're doing, but
yeah, if they're performing some right and you're immediately going to provide them with
some read of that right, they're expecting it. Yeah, and you And
there's lots of ways to cheat on that too, absolutely and say I already
you won't You just want to see your data, so I'll keep a copy
of your data around to show you right away while it spends X many milliseconds
getting synchronized on the back end. I mean, we're maybe we're getting a
little nitty gritty here, but I found the bigger thing for performance in the
rereplica model was that the right database A was insert only and B had minimal
indexes on it because you didn't you didn't need them. You were never going
to read from them, and index has caused the law of a blocking and
locking that they then you would later synchronize it later being milliseconds later, would
synchronize to a read replica that was orgoned. It only had one place of
right path coming from the right database, and then did have all that indexing
to optimize for read that as well as which is another option as we're talking
about options for scaling, is creating what I'll call like there's different ways of
determining that calling this is like a materialized view, right, creating some type
of projection where that's specifically for that's something that's query optimized. Right. It's
that kind of separating these concerns of Okay, I have these rights, like
you're saying you want them to be fast or kind of off floating indexes.
There that contention and doing the same thing on that read side, which is,
let's optimize for reads, whether that be indexes, whether that be more
denormalized in regardless of what type of data store, it is just more structured
to the way that you want to fetch that data. Well, it really
salient point. It doesn't all have to be relation databases, like one could
argue, and I have made the argument before and on this show that when
you've received an order, which is basically a glob of objects, now,
making the customer wait while you decompose that in the rows and columns is kind
of daft, like just store the object. It is the source of truth,
send the customer on their way, and do your decomposition into the read
system separately. I'll take it a step further and say where we're going with
this is offloading work. So that offloading of that decomposition to whatever immune secreture
process exactly. But that cannot be just like that view and composing that view.
It's actually composing the work, like actually performing the work. So like
I said, like if you're placing an order, it's I don't necessarily need
to persist the actual order and perform the payment in everything, It's okay,
here's my message dropped a CE. Do this asynchronously where that I can perform
that work. And in terms of scaling, that's like I would say,
kind of the I think it's a way that people immediately can think of in
your system. If you're not using any type of messaging, not using queues
or anything, and you want to start somewhere down the road, that way
is just look at something simple that you know can be asynchronous, like an
email. Right when you click that button that says do something, and it
performs some type of tasks it's say persisting data and then needs to send out
an email, It's like nobody expects that email to be instantly there. Right,
So that's kind of the immediate place you can look at your system.
Where can I offload some of this work? Where can I run this asynchronously?
Yeah? Yeah, And I love the expectation part. Nobody expects it
like they're used to. This is asynchronous. That's and it's going to have
some latency and that's fine, right, And I think it's how you're how
you're providing it to the user. Kind of then I hate to use user
experience, but it is. It's it's how what their expectation is. Like
I said, was reading your right is do they expect to see immediately the
receipt with their transaction number of their credit card? Like not necessarily so that
it's just a lot of this work can happen. It's how you're presenting it
to them. Thank you, We've I mean, we've accepted your orgale.
Will let you know when it's processed. I like e commerce systems like that,
like okay, I've received your order, all right, we've yeah,
we definitely have. All another emails is I've we've picked your product. We
know we have it all. Now I've build your card, here's the receipt
like and that I a lot of ways. Like I like that better because
you see it's clearly a workflow. They're not making anything up, they're validating
as they go, and more importantly, we'll step off if there's a problem
and they're using known methods to communicate email, text, whatever. What I
don't like is when you may want to track your package install this app.
Yeah, no, go away, thanks, no spying, no spying,
thanks very much. Yeah. The queuing model is you talk about a programming
intensive modification to a piece of software, is to go from a synchronous database
connection to a que approach like a takes some time to get your head around
that, but it takes time for the users to get used to the fact
that all of that is now different too. Yes, for sure, an
existing app depending on like what we said, is depending on how you're implementing
it. But like I said, I think there's a lot of places that
you could probably look in your if you think if people are listening or now
thinking of where can I do that, it's it's just trying to find those
natural places that are asynchronous. And I think that's just an easier place to
start, especially, like you said, wrapping your head around at how it's
working in your system. But it's there's more places probably than not that you
can that are really good fits. Yeah, there are some building gotchas with
that kind of system too, such as let's just take a very simple example.
You have a model and you want to add this data to the database
through this model, and the model doesn't have a primary key because there's no
primary key created in the database. Yet. You might have a you might
have an identifier for that model, but it's not a database identifier. But
then you can't really use it until you get that and you get that from
the database, right, the database is going to assign you and tell you
what that unique identifier is. So in the meantime, you can't do any
edits, you can't do any updates, and you can't even do a delete.
I mean you really kind of have to for that record that you're you
just created. Your user is in limbo. Yeah, that is one.
There's a bunch of gotcha's here, for sure. That's probably the one that
I think people run into most. But oftentimes a solution there is depending on
where you're making the requests from, is just providing that identifier from the get
go. It may not be the primary key, but it's some other way
that you can look up separately with what you've generated. Yeah, right,
So if it's some type of guid, I've moved to GUIDs for primary keys
instead of letting the database do it for me. I mean, anybody can
generate guid at any time and it's going to be unique. Yeah, exactly,
It's just you're at the client, and I don't mean the client as
the end user. I mean is wherever that needs to initially make that request
is essentially providing some identifier that wherever else it needs to be doing those lookups
like you said, so it doesn't necessarily need to know when the request was
processed. Right. The example I love to use in terms of visually two
is if you're in AS, you're you're in AWS wherever, and let's say
you go to stop, restart or do something with some type of resource.
You know that's not happening immediately. You're not just like making that request.
Then know this VM is stopping like it just stopped like right, Like it's
an asynchronous process, you know for sure, that that thing behind the scenes
is queuing that up for that request to be process. Now, in terms
of feedback, like we're talking about, if that is a thing where you're
behind the scenes providing some identifier, it may be using i mean some type
of web sockets connection to get pushed back down of that update happening, which
most of them do, so that you kind of get that instant feedback,
but it's not really full proof if you've ever used it before. So you
are refreshing to kind of see the latest, and there's usually your little refresh
button there for that purpose, but you know it's in the indicated to you.
It's like it's stopping, it's restarting and shutting down. It's you know,
it's not instant. Yeah, absolutely, And Derek coming in and run
for one moment for this very important message. There is always something new from
our sponsor, text Control. As a developer, do you need to integrate
PDF generation, document editing, or electronic signatures into your asp net corp or
angular applications, or you want to learn more about the differences between electronic and
digital signatures. Text Control is offering a free consulting service to educate you about
digital document processing and how text control products can help you add these features to
your applications. Go to text control dot com, slash contact and request your
free personal consultation and we're back. It's done at Rocks. I'm Richard Campbell.
That's Carl Franklin, Hey, and talking to our new friend Derek O.
Martin, who writes his great blog posts on Code Opinion and as an
excellent YouTube channel, talking about scaling the monolith. That being said, have
you seen scenarios where micro services were the right choice? I think so.
I think, Oh well, let me start by saying I can't deal with
the micro part of the word there. Yeah, it's just silly. Yeah
it is, but yeah, I know, um, but I think it's
I think it's applicable. I think I wish the industry would get more on
thinking in terms of monolith and micro services, but rather thinking about logical boundaries
and physical boundaries. I wish we could just term it that way. So
I mean by that right is thinking about if you're defining, like if you
have some type of large system, define logical boundaries, which I say are
incredibly important to do, yet they're really difficult to quote unquote get right.
Because of what right kind of changes as things evolved. Sure, but it's
thinking about what does your system do, like what what pieces of functionality?
Where's the value defining those piece of functionality, grouping that altogether logically, and
then deciding, okay, once we have all this, how do we deploy
this? What's the physical aspect of this? Right? Can this all to
be deployed together? Can this be all composed into a single unit? And
maybe this is just deployed as a container, maybe always built separate correcting because
a unit yep, deployed as a unit. Maybe we are using messaging and
queuing and it's still underneath the hood using the same like all that same code,
but the entry points are different rather than let's say we had our first
piece of the puzzle here was an htb API or some web AP, but
there's still a queueing component of that, which could be just a separate container
that's listening to whatever queuing message bus or you know what I mean, broker
that we're using, still using the same underlying code. So there's just this
aspect of we could compose this however physically that we want to. One logical
boundary could be deployed independently if it needs to scale and it has different resources.
That's a certainly situation I've run into where we realized we're scaling this large
set of classes, most of which are not being called in each of the
scale instances. But you know, there's just one problem a child that gets
called a lot, and if we peel it out of that larger ball of
goo, it'll scaree. It's easier to ramp that up, it's lighter weight
to do so, and the rest you don't need is a bunch of the
other stuff. So it's just like pull the too part together in exchange to
the complexity of now we have another security boundary, we have another API boundary,
like all of those things had to be addressed in exchange. Absolutely,
Like it's trade offs, right if you all of a sudden want to decide,
Okay, we're gonna peel out this part, and a lot of this
comes down to how you're coupling between these logical boundaries, right, So that's
kind of a big factor in whether you're going to be able to extract something,
but how you're doing that. But if you decide that is something you
need to do, like you said, for scale, purposes. Then you
have a lot of options about just in terms of how you're communicating with the
other logical boundaries. Whether that's something that you hopefully from the start maybe went
down a more loosely coupled route, but I realized that's not always the case.
But you realize that, Okay, if you weren't, and you were
doing in process calls, say between logical we'll call service A and service B,
and now you're going to do that over HTTP, gRPC, whatever the
case with some request response that's latency. There's a whole lot of that's not
like, oh, we're loosely coupled now because we have two services that communicate
over the network. No, you just created a huge performance bottle, exactly
done, exactly so, thinking that's why I said I don't love the monolith
versus micro services. I'd rather the conversation was around, Okay, let's define
logical boundaries and then separately, let's talk about how these things are deployed.
Right, So, what are the logical boundaries? I mean, scale is
one of them, but I'm sure there's others in well, So the way
I often think of logical boundaries, the way I describe it because I lived
in warehousing and distribution for a long time. And my explanation of this of
how to think about it is, Okay, yes, you have a warehouse
system. It's enormous, right, it does everything, It does a whole
lackload of things. It runs the entire company essentially. So what does that
mean though, Well, it's if you think about it, if you were
just to think about the business itself, there's a lot of departments, like
they're shipping and receiving, there's sales, there's purchasing, accounting, the warehouse
itself, people working in the warehouse, and if you want to think of
it that way, at a sense, that is a logical boundary, and
that would be a logical boundary in your system. And kind of the way
that I like to use to describe this is if you were working in that
system and you're talking to anybody that works in that business, and you're to
go to them and you say, like, what's a product, they would
all give you different answers because each one of them has a different concern about
what a product is. Right, So if you're talking to somebody in sales,
they care about the price. If you're talking to somebody and purchasing they're
talking about like the vendor cost manufacturer. If you're talking to something in the
warehouse, they care none of that. They just care about where's that product
physically in the warehouse or warehouses. They just have different concerns. So I
think that's where the kind of the I think people get in trouble is thinking
of quote unquote entities as there's like one singular like be all end all of
a product, right, But there's not. It's it's logically it's the same
product, but it just exists in different ways, in different logical boundaries.
Yeah, and that's normal. It's always been true. It just becomes more
apparent as you press against these kinds of problems. And that's the thing is
that when you think about when you think about them that way, and you
think, okay, okay, I have this subset of functionality that we're performing
on, like in sales, for example, about how we sell products,
their prices, their discounts, all those types of things, and you have
some other like let's say purchasing. As was using the example and thinking about
vendors. It's kind of liberating in a sense because they're different ways of thinking
about it and not conflating two ideas together, which then in terms of your
codebase, you're starting to think about, okay, well it's not just this
what I call entity services where it has all this mixed behavior about all these
things. It's no. Then you become a little bit more focused on what
the how I say is, like, what the capabilities are of that logical
boundary. The biggest part is usually is inevitably these things need to be composed
in some way. Yeah, that represent a product. Like if some users
on a screen, they might want to see, okay, well what's the
sale price and what's the actual purchase price? Right, So how you're composing
those things if you're in process together, well, maybe you are just making
API calls, I mean in process from as it was. Let's use the
example of you are making these things different projects within a solution. Maybe it's
just some interface that you're calling, some type of class that you're calling to
get that data. But there's still that boundary there physically that you're doing.
It's just how you wanted to then do that. If like you're saying,
if you're kind of ripping it apart, you need to be aware about when
you're going ahead of this, like ahead of time. Okay, we're gonna
rip this out and then we have all these in process calls. That's going
to be a problem. Yeah, where where is it going to? Where
is it going to impact things? You can't just pull it out and hope
for the best. Do You need to have some sense of organization and where
where the code dependencies actually exist and decide if it's actually worthwhile too, Right,
there can be cases where it's like the performance gains just not enough for
the complexity being added. Yeah, that's one of the thing with complexity is
it's it's you're moving. You're getting pretty high when you don't need to if
you don't need to move stuff out of process. Right, It's just I
guess there's this sense that moving stuff out of process, are starting that way
immediately for the sake of scale. Is I'm genuinely curious sometimes that people really
know what they're getting into in terms of all the like just deployment in I
realize things like open telemetry are awesome, but not that simple. It's not
that simple. Well, this is this is the developer credo. We we
did these things not because they're easy, but because we thought they were easy.
Yeah, I want to hear you say that with a John F.
Kennedy accent. Not because they are easy easy, but you know, we
have this terrible disease in software development where we keep calling stuff easy that clearly
has never been easy. Like virtually none of this is easy. You may
know how to do it, but that doesn't make it easy. All everything
is effort, and everything comes with a cost. And and you know,
and you're and you're shifting problems around. There's no free lunch here, not
a bit. I think it's a good way of thinking about as shifting problems
are like just creating I was creating problems you're yeah, I mean you're trying
to come up with solutions. But everything we've mentioned right in terms of cashing
or read replicas and queuing, they all come with a lot of trade offs,
a lot of complexity that you're adding that you need to be like that
it's got to come with value, like you actually need need it. And
if you're unfamiliar with it, some of these things that you're going to get
a root awakening, right, Well, that's an overhead too. If how
long is it going to take to learn this to do it decently. But
also I don't think there's any way to predict the outcome, Like you can't
do a calculus and go, hey, if we do this, we'll get
sufficient performance for effort. In some ways, there's almost no way around putting
out the effort to get to a point to even know this has a return.
And that's certainly the challenge I had was scaling anything was we're going to
try this, well, you do it as quickly as we can, but
we're not really going to know the results to we do it. I mean,
hopefully we measured before we started so that we can measure after we've done
it to say there was a return. The number of times I've seen stuff
people just do things and go does it feel faster to you? LIKEE know,
faster. That's the thing brings up a good point in terms of measuring,
because a lot of this now for me is it's it's purely based on
that of knowing, just having metrics, having some type of visibility, and
how your systems performing. Besides the well, this seems slow, It's like
yeah, is it? It only seems slow because the CTO keeps telling me
it's slow. Yeah, everything seems slow. Nothing's quite fast enough generally.
Yeah, well and is that return right? Like, how much faster can
I make this? Yeah, I'm used to. I'm used to using an
I nine computer that I built myself. That's just the fastest thing I've ever
used in my life. And the other day I set up a webcam using
one of those little Intel Mini stick computers for gigs of RAM or something like
that. And if you know, watching that thing just crawl, you know,
just sit there at a prompt thinking and thinking and thinking, and I'm
thinking, wait for computers like this and we don't even realize it because they
were they were so slow, they weren't realize there's a slow until we've got
something faster. Well, I think when we move back kind of in the
conversation and we were talking about, I mean being elastic and having those options,
it's it really is incredible now to be able to do it in terms
of land as in anything, even if we're talking about just the ability to
scale up quickly with minimal minimal downtime kind of gives you that option when even
if it's just for a moment of time where you don't really know what the
heck's going on and you're having issues, you can do it and at least
kind of weather the storm a little bit so you can investigate what what you're
going to do. But I generally find most of these things they're they're kind
of if you have metrics, if you know what's happening in your system,
you can kind of tell slowly over time where things are going to start going
south, whether like what infrastructure it is, what resources it is it is
it your app compute, is it your day to base, is it your
cash whatever services you're using. Ultimately you can kind of start telling. But
it's I find one of those things that's usually you know the metrics, usually
when it's somewhat too later, you don't have the foresight to know what you
need to measure now. I mean when you really go back and read that
post by Prime Video, you see exactly that. Over time as we measured
it, we saw performance problems, we saw them with problems, and we
saw costs increases. Like they didn't know when they went into this. I
mean one would argue they built it server list because it was quick, it
was an easy way to go, and it's only after it ran for a
while and it got more popular. It's like, huh, now, let's
and they literally say, let's revisit architecture because you have that option and got
great results from the revisit, Like this is actually a good news story.
I think it was, And especially at the end of it, I believe
at the end of that post they were saying how most of the code was
reused. It was it's basically pulling it out of landas that were howard they
were being invoked, and it's putting it into that process that's being executed in
as a container in ECS, like they did a platform shift, like this
is actually wicked cool that a lot of this code was just fine, but
when you shifted the platform around, and if you shifted the platform around,
you got rid of a bunch of contention, it reduced costs, Like there
was all these benefits as different architecture. Would it work for you? Done?
O, You got to go measure what you're doing right now to try
and compare against other architectures. I think that's the most difficult thing. I
think you alluded to you earlier. It's just like we want this cookie cutter
solution that this is just going to work all the time, and it's just
not the case. It's just like the one right way. Please. Yeah,
it's just not the case. It doesn't exist. It's very it's very
funny just that. Yeah. I wish it was that simple, but there's
too many different kinds of workloads, and there's too many different ways at things
evolved. You know, I wonder if this was a great solution when they
started. I think it was. I how I remember it correctly is like
I think their exact kind of message was we never intended it to be at
the scale that it was. Yeah, so when and I dealt with lots
of e commerce sites like that, we thought it was only going to have
this many customers, and then it got all this much bigger. It's like,
wow, great problem, Like part of this is his whole You ain't
gonna need it. Like how do you go into a complex architecture having no
idea if the product's really going to take off or not? To build it
as simply as possible, because delivering it is an important value too, you
know. One way to make sure it never provides value is never finished the
darn thing, because you made an architecture is so complicated it takes forever to
get it built. Yeah, it is. And when we go back to
the kind of the one way that this is going to work is it is
it's it's starting simply based on what you're aware of kind of that space.
But like I said, if you if you focus to me, if you
focus on defining logical boundaries, trying its best you can to understand the coupling
between them, You've got a lot of options about why you can choose.
Then you can choose how you want to do this stuff. Like you said,
if there's one particular part of your system that is heavily overloaded, yeah,
maybe that's where you choose to remove it and peel it off and scale
it on its own and see if that's sufficient. You know. The other
thing about that good architecture, that separation and concerns is the experiments are easier
too. Yeah, well, I to based on experiments. I would also
say that not everything needs to be created the same way, right right,
So that means that if each if each boundary has defines how it wants to
do a storage and that could be still the same underlying physical database. Instance,
Let's say it like, hey, we're using this relational database for the
entire thing, the exact same data we got it. Yeah, we're not
gonna like split these things up more costs also just to use the exact same
instance. It could be in the same physical database. But hey, these
particular tables are owned by this particular boundary. Don't go start muddling in my
stuff, right, Like, defind some type of API we want to exchange
this if we need to. Yeah, but just define those boundaries and then
from there maybe you decide later on, you know what, like this particular
boundary that relational just wasn't a good choice here isn't needed. So yeah,
so like when you're staying like experiment, it's like you get that option,
right, It's and it's not like you're doing this because you want that option.
It's just you're getting it kind of. It kind of is a little
free lunch. And that's it comes at the complexity of you have to define
these boundaries. It's not just one massive thinks about it. Yeah, but
also now that you have that boundary, it's like, hey, would this
be better served in an object store instead of a relational database. Well,
let's let's see what it takes to do that. You know, maybe a
little spike on it to do some experimentation, and it's like, hey,
this works pretty good. I'll finish implementing it, and then let's do some
benchmarks. You know, I've added that they you know, then we'll maybe
deal with some of the complexity around it. Like it is. These are
fun explorers too, right, Like I found I had pretty good patterns for
scaling websites because I did a ton of them, and I was often right.
But you'd get into more esoteric systems, more interior type architectures, you're
like, yeah, I see that this is likely the bottleneck. I see
three ways for us to try and scale us. I can't tell you which
one's gonna win, Like, we've got to try exactly. And I think
kind of my point when I wrote this post even some of the videos is
just there's just you have different ways of dealing with it, right, So
whether that be scaling up, scaling out, knowing where you're going to kind
of move that ballneck move that problem, whether you're off floating work with queues,
messaging, whether it be materialized views for queries, and the other point
I want to make is, at least in my experience, is generally there's
hot paths. Right, Like, especially in big systems, I always kind
of use eighty twenty rule, is that you're going to have the twenty percent
that's causing all your havoc basically right, and just being able to optimize those
So whether that be never mind from an infrastructure point of view, but just
let's look at where those routes are, Like, let's look at those code
pass What can we be doing there? Specifically? Are we making too many
round trips in the database? Maybe we're just doing something absurd where we're just
hitting our database and fetching able to a whole lot of nonsense that we don't
need. You'd be surprised how much that happens. So it's even just and
it can be cost effective, as crazy as sounds. Sometimes it's like,
let's just spend some time doing this, Like, let's just look at what
we're actually doing. The question is do you know what twenty percent or do
you have to get out of the field and run it for a while ago,
Oh, this is the twenty percent that's the problem shelves. I think
metrics, but I think intuitively in a large system, least in my experience,
you know what that core part of functionality is generally I think even from
if you were to ask, say, people on the business side, like
hey, and the app, where do you think the most action is at
the because that's what they're in most of the time, right, Like it's
I would say, there's kind of the core part of your system as well
as that fringe supporting more generic stuff. So like when I was talking about
distribution, say let's say like CRM or something like that, maybe you're like,
you know what, that's not a big part of our business. We
have some rudimentary thing for dealing with customers. It's really not that big a
deal. Like our bread and butter is warehouse management, and that's where all
the complexity is, right, so you kind of know to a degree,
I think even on the business side, they could probably give you some hints.
Well, in the web world, nobody ever went wrong tuning a landing
page, you know, because it just had so much traffic, like it
was pretty much a guaranteed win. It spends some time in there or usually
it was just you got to a point where it's like, hey, the
incremental improvement left is so small you can't justifind more time on it. There
are easier winds elsewhere, but they may not be as important. Ah,
it's a fun battle, you know. It's not like there's any simple solution
to this. You've got to go out measure and feel an ound and hypothesize
and build some tests and push a little further. But I wonder if we're
speaking heresy here, because last time I look, there's a cloud and I
just have a knob I have to turn up right, Like, isn't there?
They go faster knob of everything related to the cloud. You just spend
more. I mean too, I like we're joking, but like somewhat yea
in this sense, right, it's at least being able to scale up in
a lot of circumstances, like I can always shift to a larger VM,
and they are perfectly willing to sell you a bigger and bigger VM these days.
And seeing two is that you can do that with at least with app
services on Azure you can. You can. It's really trivial to add and
scale it up exactly like if you if you already are in the situation where
all we we scale out, say maybe our web app and we're running three
instances, but all of a sudden things are going south, and we'll just
add more instances. But like I said, and that may work. You
had one instance and then all of a sudden, you start noticing, like
I said, downstream start suffering, and then it just bottleneck and you get
a back end bottleneck. Like well, yeah, okay, you also have
to be your code has to be able to handle multiple spanning, multiple machines.
That's an old talk I used to do from one server to two.
Yeah, you know, it's a pretty hard leap just to get to that
point, you know what I mean. With Blazer server anyway, which is
sort of like in my wheelhouse now that if you're using their um the signal
ar service, it's pretty trivial. You there's affinity is turned on by default,
so whatever, the first server that you logged into, that's the one
that you're going to use for the rest of your session. And that's that's
nice to not have to worry about that. Yeah, I think I think
the scale ups scale out just with cloud just making it so much easier is
great, and I leverage it quite often when things start going south. But
I guess in experience, I kind of okay, well, like we got
some thresholds here where I know, okay, if this is scaling out,
this is when it's a little bit too far. I just can't infinitely scale
out and everything's going to be going to be great. And to that effect
too, A lot of it is things that aren't even in your control.
So if you're leveraging, say some external services, and you're making requests to
them, I don't know the examples, like say you're making a call to
being Maps or Google Maps or any other service, some type of payment gateway,
and you expect that to on the happy time, it only takes like
hunter milliseconds, it's fine, and all of a sudden that thing starts taking
seconds. Yeah, right, like all of a sudden. It's so,
what the heck's going on? Where what we were mentioning with metrics that comes
into play because you know, how matter how much you scale out, you
ain't going to help that. Yeah, yeah, the simple So what's next
for you, Derek? What's in your same old thing of just creating videos
and blogs at all? Kind of pertain to everything we just talked about pretty
much for the most part. Well, I'm going to urge our listeners to
go check out your blog and continue reading it. Thanks so much for spending
a time with us. It's been enlightening. Yeah, I appreciate this was
this was really cool, all right, and we'll see you next time on
dot net rocks. Dot net Rocks is brought to you by Franklin's Net and
produced by Pop Studios, a full service audio, video and post production facility
located physically in New London, Connecticut, and of course in the cloud online
at pwop dot com. Visit our website at dt nt R o c ks
dot com for RSS feeds, downloads, mobile apps, comments, and access
to the full archives going back to show number one, recorded in September two
thousand and two. And make sure you check out our sponsors. They keep
us in business. Now go write some code, See you next time.
You got a vand