What's going on everybody. I'm your host today Will Button four Adventures in dev
Ops And before we get started, I do want to remind everyone, or
tell you for the first time maybe that we are now doing these shows live
in addition to the podcast, so if you do want to catch it live,
we're recording on Tuesdays at nine to thirty Central Time. That's GMT minus
six, and actually it's like nine thirty ish because we start at nine thirty,
but then usually have a little bit of a pre chat to get our
guests up to speed on how the process works, and then a click to
go live button, So shortly after nine thirty Central time, you can catch
us live on Facebook, LinkedIn, and YouTube. And speaking of our guests,
today, I have Segee Brody, chief Technology officer of Opti nine,
consultant former software developer, according to his own words, still has code and
production that probably shouldn't be and I can definitely relate to that. Today we're
going to be talking about resilience and disaster recovery in the cloud, how it's
relevant, why you still need it, and then dig into that. See
welcome to the show, Thank you great to be here with you. Will
I'm excited to speak to a technical audience, which is not always the taste.
So you know, see if I get called out on anything. But
it's great to be able to yet as deep as I want to be with
not having to pull myself back, right. That's always my biggest fear in
doing talks and live shows like this. It's the part of the show that
I call stump the chump where you say something wrong and somebody just calls you
out on it. That pressure, that pressure can be very useful with harness
the right way. I used to force myself to volunteer to speak on highly
technical topics at conventions that I knew nothing about, and I had like maybe
two months and so these topics that I was putting on the bottom of my
list that I knew I needed to learn, but I was just dragging my
feet on. You know. Now, now I have a time and day
where I'm the expert and I'm going to potentially get stumped, and so,
you know, fear of embarrassment is a very great motivator. Oh for sure,
I like the approach. That's bold, but that's definitely going to be
effective. I like it. I may steal that from you. Yeah,
cool, So tell tell our viewers a little bit about your background and how
you got to be the CTEO of Optanine. Sure. Yeah, So when
I was you know, like like probably many who were listening, you know,
it got into sort of this industry as a teenager, just you know,
screwing around with computers and the Internet and having fun and that you know,
that sort of curiosity somehow turned into a job, which is great.
So it was late nineties, my co founder of a company called webear.
We were kind of in the right place at the right time, and we
just started hosting websites for our friends, so you can kind of think of
it as a hosting company. So we were working with technologies like mi FreeBSD
and Apache and you know, the typical sort of web stack. And this
was great because it was before before Google and before things like PHP, and
before things like customer service or support, and it was kind of like sinker
swim, and so we just built everything ourselves and just sort of scaled up
with our customers, grew that business, sort of pivoted towards enterprise about maybe
ten years after that, and started to focus on management of private cloud deployments.
Management of public clouds, orchestration and sort of owning the glue in between
these hybrid cloun environments. So a lot of networking, which is which is
always fun, and then got into b C d R so disaster recovery as
a service backups. Networking was always our secret sauce, which is fun,
you know, saying things like it's great that you're you're copying your data somewhere.
How are you going to consume it? You know, what does consumption
look like? You know, these are networking problems. So I've always been
a big a big network I eventually we sold that business to private equity.
I stayed on as the CTO and we rebranded to opt and nine after we
bought or merged with two other companies. And I'm still here mostly in a
sort of a more of a you know, chief Technology Product officer, which
is starting to become a thing now, you know, which which you know,
CTO is so vague, there's different personas. So really my role is
sort of product focused company, you know, sort of customer focus. What
are we building for customers, how are we helping them? How are we
working within the bounds of the third part of technologies that we use. From
an integration perspective, how do we push down bloat and the multi doing some
consulting on the side and just trying to stay busy right on. I think
that's cool perspective or a cool journey, because for a lot of us,
we end up spending a few years at a company and then jump to another
company, and so we end up going from company to company. I've done
it myself just to well, let's be honest, I've done it for salary
increases, but also because of the opportunity to work on different technologies that I
wanted to go deeper into. But I think that's a cool path and an
unusual one in the fact that you've been with the same company even though there
have been mergers and acquisitions along the way. So you've really built your skill
set in driving the company as it matures versus driving your skill your skill set
as you mature. Yeah, you know, I would say we we were
a service provider, and a service provider is interesting. It's very different from
an enterprise environment, and a lot of people don't realize the nuances and differences.
It's funny whenever a vendor used to call us and try to sell us
something to be like, all right, cool, do you have multi tenant
capabilities? No? I'm like, okay, do you know what a service
provider is? Like? You know, realize like and then we'd say,
well, listen, if you can, if you can make a service better
happy, you can probably make anybody happy. So you know, we'll tell
you what you need to do. But but you're absolutely right. What was
great for us was that our customers were, you know, sort of in
multiple industries and verticals, trying to run different applications and solve different problems.
I'd say the market is more sort of segmented now and matured now. But
what's great about being a service provider is your customer customers are coming to you
with the with the problems that need to be solved, with the use cases,
as the as the industry changes and grows and as there's new shiny objects,
they're coming back to you and pushing you and saying, hey, we
heard about this really cool thing. We want to use it. Can we
use it? And it's like, uh, you know it's it's like yes,
of course, well and then we'll figure out on the back end.
Or no, it's like you want to lose a customer and listen if there's
value in what they're saying, and you've heard it more than more than once
in the last four weeks, and then you listen to it. And if
you think someone else can benefit from it, and you listen to it.
So it's our customers have been pushing us always and that's sort of driven innovation,
and so we never had to. Okay, maybe you're not like going
way outside of your out of your target zone, but you're constantly pivoting.
You're constantly trying to keep the leg up on your competitors, because if your
customers are asking you for that, then competitors are hearing the same thing.
And so I think where we've done well is we've owned our own you know,
we've never owned our own IP, but we've owned our own glue and
empowered us to be able to mix and match best and breed and just and
just innovate and and be at the forefront. So I agree with you.
I think service providers are a great place to be and even for you know,
listen, I was a founder, so maybe it's a little different,
but within our environment, over the years, you know, a new problem
to be solved would come in and maybe one employee would sort of just jump
on it and just be like, hey, you know that's cool, I
can do that. And as a smaller company we'd be like all right,
you know, like uld say. The silly example is when you know,
like uh, you know, uh no, SQL platforms got big and people
wanted us to manage Mango and we'd have we had one gentleman who is like,
yeah, I'd love to do that. It's like all right, you
know, great, next week. You know that guy is the Mango expert.
Everything have, any is goes goes to him. And so there's just
a lot of opportunity for self growth there if you can recognize and take it.
Yeah, for sure. And I think that's the key to longevity in
this space is a desire to continually grow and learn new skills. Yeah,
but there's also some old skills that we can't let go. One of those
being disaster recovery and backups. And you mentioned it before we started recording.
It's one of those that seems to have been pushed on the back burner over
the last ten years or so. But doing so, it has some definite,
some definite impacts to your business, So talk to us a little bit
about resilience in DR in the cloud. Yeah, I'd love to. So,
you know, we have we have traditionally provided a disaster recovery as a
service offering for you know, I don't want to say legacy, you know,
sort of non non sort of cloud native applications, so things that are
not necessarily running on AWS or Azure, and so over the years we would
look at out maybe a deployment running on VMS, running on VMware or hyper
v or KBM. We would basically provide an entire ecosystem needed so that your
applications would continue to operate despite some sort of outers aut the production site or
cybersecurity event or somebody fat fingering database and so you know what that looks like
is obviously replicating the data, but more important than that is sort of understanding
what is consumption. What does consumption look like? How are your users going
to consume the application from the DR side as they did in production. And
that's a big sort of networking sort of task or challenge to deal with,
and then also dealing with dependencies, like what about all these shared services that
they're relying upon, authentication or networking, c p IPM, stuff like that.
And so we take ownership of authoring the WRUNG books not only for fail
over fail back, but what if it's just one application that you want to
faill over and what do you do with shared resources? You know, if
you have a legacy database server, which is a weird thing to say,
and it's running, you know that is hosting databases, tend different applications and
you want to fail over one, do you bring the database server with you?
And so always interesting situations and challenges. And then you know, when
when public clouds started getting popular, you know, I had a pretty pessimistic
look on disaster recovery in general, and I think I think the entire sort
of industry was excited about the fact that, like, we won't have to
deal with that anymore. We have ability now to just build applications that are
inherently resilient, you know, from the bottom up, and you know,
we'll deploy them on the cloud. They'll be self healing, and then we
won't have to deal with this, you know. And I think you know
what's happened is that people have tried that, and people have tried to build
these applications that will run let's say, in multiple eight of US regions,
and they realize the complexity involved in building the applications from the start with that
thought in mind is just it is just far beyond the bounds of what they
want to deal with. And we see that even when you invest time and
resources into that, it doesn't necessarily mean that it's going to work. You
know, every time like AWS East has an outage or goes down, you
know how many very large popular sites, you know, household name sites go
down that are technology companies that we know are deploying the multiple regions. So
why are they down? Because it's almost like it's like this impossible thing to
build, and it's not always their fault, right, Like the interdependence between
their applications and even third party you know SaaS or their party pass mean that
can they actually test this thing, can actually test their resilience plan without you
know, without actually affecting production. So what I've seen is sort of like
the industry going towards the middle ground where where some people don't even realize you
can do this, but you can basically employ an application in a in a
single region not have to sort of build this whole resilience concept into your application
from day one, and then employ traditional disaster recovery strategies towards you know,
sort of gaining you know, resilience of your app. So the point on
one region and maybe now we can use replication tools that are more cloud native
focused, and then we could still take all of those things that we learned
over the years from from traditional disaster recovery, things like dependency mapping, building
run books to deal with different situations, building sort of network strategies so that
I can test at the DR site without poisoning my production data. You know,
if your production app is connected with the sales Salesforce API and you bring
up your app and DR and you start playing with records like oops, we're
modifying production data. So you know, all of these sort of you know,
sort of core disaster recovery strategies. Give them a modern data mover that
knows how to replicate or rewrite, rewrite of resources. Let's they use the
terraform or cloud formations. Give me something modern data mover and then apply everything
from traditional DR and you can actually achieve resilience without having to go crazy from
the development. Yeah, one thing that you mentioned there a couple of times
that I think is is really key is testing that, And it reminds me
every time I think about that. It reminds me way back early in my
career decades ago, my boss asking our team, Hey, are you guys
ready for a disaster and ology And we're like, oh yeah, we're all
set and he's like, okay, great, everybody show up on Saturday.
And so we showed up on this Saturday and we went out to he ran
into a conference room in a hotel, had some servers sitting there, and
he had our backup tapes. He's got great restore everything, you know.
And we didn't even make it five minutes into the process before we realized,
oh wait, we don't have the floppy disk to update our bios or we
don't have the boot disc to reinstall the operating system. And it was a
really, really long and painful day. But the lessons have stuck for a
couple decades now. Yeah, you know, that's a It's great when people
are sort of overly focused on some sort of data application when it comes to
disaster recovery or even where some people just think that their backup strategy is also
their sort of resilience or disaster recovery strategy, and won't get too much into
that, but you know, you have two separate goals with you know,
sort of two separate strategies to be employed. So yeah, you really need
to sort of pre author the run book. And I think today what's interesting
now too that we're seeing is that if you look at an event like a
like a ransomware attack or a cybersecurity event, you know it's the incident response
plan or the sort of the and order the disaster recovery run book or something
like that, it's not it's it's not something that a single team would be
dealing with. Right, Like a DevOps team is responsible for sort of the
the the uptime and resilience of an application, and presumably they own sort of
all this orchestration for production to dr multi regions and fail over. That's great.
But now if you bring in this sort of you know, this sort
of security aspect that this has this need to fail over was in relation to
a security event, now you have a completely new team. Maybe it's an
internal soccer security team or an external MSSP. And now you have these two
teams that unfortunately many organizations don't speak that much, and now they need to
be lockstep as part of the incident. And you know, if you think
about a CTO or CIO at a higher level, you know, they kind
of become the quarterback between these teams during an incident. And it's not something
that I think they even realized that they were ever going to have to deal
with. And so the incident response plans, the disaster recovery rum books need
to be inclusive of who is who owns what during a you know, sort
of a security incident. You know, can you even bring up the application?
You know at the r site do you want to? So how does
a team that maybe they recognize that their their dr the resilience plan isn't where
it should be. What what what are the first steps like because to get
this done you need to devote time and resources and it has to be prioritized
and sometimes that you have to prioritize it above like day to day operations.
And I think specifically it comes down to what are you going to say no
to and so that you can so that you do have the bandwidth to say
yes to this. So what are some good early steps for people once they
recognize that they're they're not where they should be. Yeah, so I think
the good question. I think the first thing that they need to do,
and I think this is I think that the market has matured a bit here
and this is austraily obvious now, but you know, the teams need to
kind of sit down and figure out, you know, what they have an
appetite to take ownership and responsibility for in this realm. And so if you
look at a traditional you know, DevOps sort of how this goes and general
for like a DevOps conversation is, you know, are we are we application
developers? Are we sres? You know? Who is responsible for ongoing ongoing
management? You know sort of metrics collection efficiency? And obviously that that's a
there's no right or wrong with any of these things. And a lot of
it has to do with sort of the the DNA of the company and what
they kind of want to be when they grow up and do they want their
you know, certain IT teams adding value to the business or managing infrastructure,
and so you know, we'll see I'll see smaller organizations that are like,
you know, we're a small team. We own everything, so we're going
to just internalize it, and also see very large organizations that have, you
know, an abundance of resources, and they basically make the they basically make
the decision that we don't want to be in the business of managing disaster recovery.
We don't want to be responsible for it. We'd rather outsource it.
And an interesting thing to think about here is, you know, the the
complexity of all of all of our applications and our deployments are. They're not
getting simpler, They're getting more complex. In fact, I think you can
argue that part of the goal of of DevOps these days, part of one
of the things they should be striving for, and maybe even a key metric
to focus on is to what extent am I making my you know, the
deployment that I'm managing, to what extent I make? Am I making it
simpler and less complex? And obviously the more complex, the harder it is
to manage, to monitor, to scale, to secure, and to make
and to make resilience. So I think people need to acknowledge that. And
when you have that conversation, you know, one of the answers that comes
out of that conversation could be, Hey, we want to make it simpler
how do we make it simple? Well, how do why don't we outsource
certain layers and certain responsibilities and disaster recovery and resilience is an easy one to
outsource. It's low hanging fruit, you know, typically it does not affect
your production too much. If you can use sort of that middle ground strategy
that I mentioned at the beginning, you don't have to modify your application,
you know, much at all in order to be able to achieve resilience.
So that that would be my answer. The first thing I'll do is sit
down and figure out, you know, what is your appetite to manage and
own that internally? Yeah, for sure. Yeah, And I think that's
a huge selling point. If you have a strategy where you don't have to
find your existing infrastructure application a whole lot, that's always going to be a
big selling point. Let's let's do this. Take a step back and help
me understand why moving to the cloud or using the cloud cloud providers like ABS
is not a dr strategy in itself. Yeah. Well, you know,
they give you the right deserverbody it knows right it's going on homes on a
home depot, and they're giving you the right tools and you got to you
know, makeup if you want. So you have to look at it on
a per sort of platform, you know, uh, per platform sort of
environment. So if you look at something like S three, which obviously is
being you know, is being stored in multiple local zones within within a region,
or even has the ability to sort of have its own inherent built in
sort of cross region replication, you're probably good there from a you know,
if you wanted to, if you wanted to build a disaster recovery strategy between
let's say East and West, when as three perspective, it is, it
is fairly straightforward. You can kind of put a check next to that layer.
As far as your data being available at the at the dr site,
you know, recovering from a cyber attack or sort of a manipulation of the
data, that's another story. But if if you're if you know, if
the entire interviews goes down and you want your application back up and running within
within a set rt O, you know, you can kind of put a
check there for other for other sort of you know, other platforms. It's
not always the case that that that that is done. Typically it's not,
you know, and so there are snapshot capabilities that exist, but then there's
this entire orchestration task that sits on top of all that. So you have
all of your all of your configurations and resources, maybe have another site,
but now your applications are not necessarily written to be able to reference those at
those reference ideas at the at the the R site. And so now we're
so so it's really a replication orchestration strategy, right. And so what we'll
do is we'll look at your various applications and then we'll look at the A
w U S and we're doing this mostly for a w S today in addition
to the legacy environments which I mentioned before, but for a public cloud.
We'll look at the various platforms that your application is using, and we will
employ underlying AWS technologies to ensure that data is up to date at the DR
site. And so maybe that's maybe that is cross regent snapshots, or maybe
that is a w S d r S which works very well for certain platforms
but can be expensive. So now we get into the application criticality question of
you know, how critical is each application to be up and running and sort
of match the right replication technologies to the cost and to the application criticality.
Beyond that, you know, we're using orchestration tools and one of them is
called r PO that we'll use that will orchestrate some of this back and forth.
And our PO might be something that's great for a team that wants to
internalize all this and just say we got a tool, let's use it.
Where OPT and I comes in is it's not just about the tool, it's
you know, who is you know, do you want to take owner ship
of the sailover process and the sailback process? Do you want the ownership of
the testing, building the network integration strategy, building the automations into let's say,
you know, d n S, maybe sd WAN policies, so on
and so forth. So we kind of sit on top and own the entire
process, you know, suit to notts, so that DevOps teams and IT
teams can just wash their hands of it and focus on building applications right on.
Yeah, I'm actually an RBO customer and it's a it's a great tool.
It's just it's it's one of the few tools I've seen that just does
what it says it's going to do at an exceptional level. But just like
you mentioned, you know, that's only part of it that handles the infrastructure.
There's still the whole human aspect of it of verifying what you've replicated and
doing a failover to it and testing it and making sure it works. And
that's another full time job in itself, it is. And what the funny
thing is for us is again have being a company that has been doing and
providing disaster recovery as a service for you know, uh VMware platforms, physical
servers, IBM I series, you know zen KBM based applications. The funny
thing is, you know we are we're not. You can say we're a
technology company, but it's really that glue that we're owning. But we have
we have broad invest in breed data movers and sort of replication tools to to
you know, to focus on specific platforms and our and and so when brought
in our PO it's like, hey, here's the best and breed tool for
cloud native adobs apps. But everything else that we're doing, all the value
we're providing, and all the wrappers around around the replication tool like they're all
the same as we were doing five ten years ago, which is actually pretty
cool. It's like if you can stay up with the tech, and you
can build a platform that can support multiple rations in a modular way like you
can, you can stay relevant through all of these crazy croudchets, for sure.
I should. We had Doug from RBO on the podcast a few weeks
ago. I should do another episode with both you and him and just go
into a deep dive on this week, so him and I and I've known
him for a while and I really am super polish on their platform. I
think it's amazing. Him and I are doing a webinar tomorrow actually about all
this in detail. Oh right, I will get that from you and make
sure that that's in our show notes when this episode goes live. That will
cool talk. When it comes to DR in the cloud, you mentioned that
providers like AWS have a lot of the tools built in. You just have
to look at them on a case by case basis, see what those tools
are and it make sure that they're enabled and that they're working properly for you.
How often do you see the need or do you recommend cross provider DR
strategies like backing up our AWS or replicating our AWS environment in as your or
GCP, because that brings with it a whole, like an exponential increase in
overhead as well as costs. Yeah, that's a great question. You know.
I think that you kind of have to look at at three buckets here
in general. You know, you have your high available you know, the
ability to achieve high availability, right, which which maybe is sort of you
know, I think in order to build high availability for your application cross region
or cross cloud, you're really not going to be able to get away from
sort of building your application with that intent from day one and having to apply
so much more complexity to your application, to your c c D process,
and really the the level of expertise that you need from your developers. Just
I think it's it's on another level, right. And so if if you're
just starting the process of building an application now and that is your goal,
you can't you can't go back. You can't go back later and just be
like, oh, we'll just do that later. No, it has to
be. It has to be in the DNA of your application. This is
also an interesting point when you start to think about integrations with third parties.
You start to think about all of the third party providers that you're going to
utilize. From an EPI perspective or from a data perspective. You know,
if you if you have this mandate to have resilience and high availability as part
of your application or security, and you build a framework or a requirement around
that you need to have, you need to have those conversations with those third
parties, you know, before you start using them and not after, because
if they're the if they're the weakest link in the chain from that perspective,
if they don't have great resiliency to provide you with the options you need,
then then you're stuck. I think too many companies go and they'll you know,
they have the SaaS sprawl or they just start using them and then you
know, you might spend I don't know years ago being an AA application that
that works a cross cloud, but one of your vendors you know, is
not locked up, and boom, you know you achieve nothing. But so
understand the difference is a j backups and sort of traditional dr applied here and
really sort of figure out, I would say, figure out where do you
want to where do you want your sort of vendor lock in to be right
if it's if it's data, If you're okay with with vendor lock in with
one cloud, that's fine. I don't think there's anything wrong with that,
especially again if you're if you're building your application with forethought into that, and
maybe you know, we see people, I'm sure you've seen it many times,
people that are like, I'm going to use a WUS multiple regions,
but I'm purposely not going to use any any platform services. You're going to
run my own SQL instances and kind of go backwards in that way. Fine,
you know, if you're using our r PO, as far as I'm
aware, it isn't today, it does not have any cross cloud replication capabilities,
but let's say it did. Great now your vendor lockin is on that
level. So I would say a lot of this is sort of risk,
you know, risk aversion, risk mitigation. I think the likelihood of all
of a w US going down and having a need for sort of cross cloud
is, you know, hopefully very little to none. But I think a
single region outage as we've seen is you know, fairly, it's definitely in
realm possibility and happens. But I do think what you're saying makes sense from
a backup perspective, right, Maybe we don't need, you know, an
rto of being able to fail over from a of US to Azure within four
hours or twenty four hours. But if we're copying our data, if we're
having a copy of our data there and we and we understand what the path
to sort of bringing it back up looks like, I think that you're in
better shape than most are today. Yeah, I agree with that one hundred
percent. I've had as a consultant. I've had multiple companies come to me
over the years and say, hey, we need to implement dr so we
want to we can't trust AIDA, or we don't want to trust AWS,
so we want to use multiple cloud providers. And my approach with them has
always been, you know, I don't think AWS is and not picking on
AWS here, but I don't think that's the weak point. And then we
go through and look at their stack, and it always comes down to the
fact that you know that hasn't been the weak point. You know, they've
chosen to use a managed database provider and so all of their data is not
even in their AWS environment, or they have all of these external dependencies like
Salesforce or different things like that, and it's like, okay, if you
can replicate all your infrastructure over to another provider, but this third party tendancy
is still a single point of failure and much more painful that goes down.
Which makes me think along those lines, since you work with a lot of
companies in this how willing are third party vendors to talk about what their own
internal d are and high availability strategy is? They all have they all have
their boiler plate off the cuff answer that they have to provide, you know,
and it's always going to be pretty vague, and you're probably going to
have to go back two or three more times. And sometimes I'll just refer
you to the s l A and obviously their s l A credit mechanisms,
like most are going to be just a joke, right, And so it
is it is a risk. I mean, I will say on the on
the compute side, I do think that you know, Kubernetes has has democratized
sort of the the compute layer and has made it very easy to sort of
deploy you know, your your code where you want when you want to.
But but you're right, it is it is the database layer, uh,
and sort of the rest of the shared services layers, and that's kind of
as it is kind of a hard pill to swallow because you know, again,
what if if you kind of want to manage and run and operate your
own databases, that's fine. It'll be less expensive that way, you'll save
you'll save money, and you'll have more control and you will be able to
to sort of make good on this sort of cross cross cloud resilience if you
want to. But now the operational overhead has has increased, and so you
know, part of what we've done and sort of what I've sort of been
dabbling in, you know, with some consulting is just doing that dependency mapping,
application mapping and figuring out what we what we want to do. And
by the way, just because you're using paths in production, doesn't mean that
you can't have sort of a single database deployment in dr with some sort of
you know, sort of sort of you know, snapshot or replication mechanism in
place as a as a backup. And look, it takes you two days
to get that to get all the tweets work out, you know, post
event, you know, most people will say that's not the end of the
world, and they will accept that as a solution because, to be honest,
a lot of folks are looking unfortunately, they're looking to check the box
on a DR strategy, are having one in place for compliance and having the
DR strategy does not necessarily mean that you have you have a run book or
you have super low R t O s R pos. It just means that
you have sat down and written what you would do during an event, even
if it hasn't been fully tested. And so if that's what you're after,
that's what your goal is. Because maybe you are not a you know,
a fully technology based platform, as a as a as a business, as
a revenue generation oftentimes that isn't a Yeah. I think the having the conversation
about rt O the recovery time objective is really important to have because all my
entire career, you know, I've never worked for companies like Google. Well,
there's been one exception where I had one of my employers. We were
doing health care for trauma patients, so we had to had to move quickly
there. But for most businesses, having having that RTO conversation is very helpful
because while ideally you would like to say, oh, yeah we can,
we can fail over in two hours. That's cool, but it comes with
a set of costs and acknowledging the fact that you know it would be embarrassing
to tell your customers will be up in two days. Maybe that is the
right strategy based on your your business. Yeah, you got to start somewhere,
right when when when I've worked with companies to build a disaster recovery strategy
and actually roll it out, you know, the first thing we'll ask is
what what are the what are the business goals you're trying to achieve? And
and some of the questions might be, you know, do you do you
need and only looking to protect against a sort of a full failure at the
production site where all the applications need to be filled over concurrently, or are
you're looking to protect against situations where you might need to fail over individual applications.
And then there might be other questions like do you want to faill over
if there's like one server that is sort of ransomwared And you know, of
course everybody says yes to everything, yes, we want all that. The
problem is the sort of the more situation, sort of the increased complexity,
and you know, ironically enough, the full failover event everything needs to come
over at the same time is actually much easier to build for and to achieve
than all the others because typically you have the sort of interdependence between applications are
sitting maybe behind the same firewall and the same VPC, the same network,
and so if you can keep them on all the same IP addresses and keep
references intact, then it is much easier. And so typically we'll employ a
phase approach. Will let's let's be able to achieve that, improve that,
show that it works, and then we'll sort of peel back the rest of
the layers of the onion and strive for more. Yeah. It reminds me
of an analogy from drag car drag car racing, speed, cost money,
How fast can you afford to go? Yeah? Yeah, And I really
think it's interesting when you think about these things and you think about the burdens
if you're looking for complete aha, multi region or even multi cloud, the
burden, the extra burden that you're putting on your you know, DevOps or
app dev teams, you know, and what is what does that translate into
just sort of the business impact? You know, how much longer are your
development cycles because of that? And what are you not being what? What
features are you not able to work on because the oldest extra time put into
the forethought of this high availability. That's why I like. I like the
middle ground approach where let's have our developers focus on developing a application that runs
on a single let's say a WUS region and you know, hands head down,
hands to keyboard, focus on building applications, which they probably have,
probably have a lot of experience doing that. You know, this whole multi
region thing is typically fairly new to someone and they're going to go off on
a tangent. So APT devs you focus on building, you know, an
application that is resilient within a region. AWS makes it fairly straightforward to do
that. And then maybe a separate team or SR team or a company like
Optinine kind of comes in over the top and says we are going to employ
a disaster recovery as a service to that single region deployment and achieve resilience using
tools like RPO and using proven strategies, and that way the APT devs can
just highly focused. I think that's such a win win, and honestly,
I don't even know that there's a ton of developers out there that can even
achieve the HA with the high degree success. Yeah. I think one of
the other benefits of that approach is discovering tribal knowledge, because in a lot
of the scenarios I've been involved with, we do things and we take certain
steps or actions because of this tribal knowledge that we happen to know. And
in many cases we don't even know that we're making decisions based on tribal knowledge.
But when you bring in a third party like opt to nine, then
you're you're coming at it from a fresh perspective without the tribal knowledge, and
it works really well to expose that. It's like, oh, okay,
now we have this piece of information that has to be documented and formalized.
Absolutely, And like I said at the beginning, you know, when I
hear tribal knowledge, you know, I hear complexity, and I gin that.
I think there's this whole idea of managing complexity, managing complexity sprawl,
you know, fighting to reduce complexity. It's not it is not being pushed
enough, you know, from an industry perspective. In fact, I think
we have the opposite problem. I think we have a lot of folks out
there and I'll even you know, different times in my career. It definitely
have been guilty of this. You know, we have we have shiny objects
syndrome, and we want to be able to be exposed to all the latest
and greatest tools. You know, I think we're all curious people in the
indrodustry and we like playing with new things. I think I think part of
it also is just maybe a little bit of fear and ensuring that we have
the latest and greatest acronyms on our resumes. Sure, but shiny objects syndrome
is you know, is I think the complete opposite of I want to keep
my environment simple so that it's manageable, so that I can reduce the need
for tribal knowledge. And this kind of goes into you know, like other
soft skills, right like, you know, if I want the person at
four a m to be able to fix what I built. You know,
to what extent am I a good technical documenter? And to what extent do
I take pride in that as a standalone skill that I'm good at, you
know, as as a developer or an SR or you know or DevOps person.
Yeah, agreed, And I just speaking from personal experience, I'm not
good at documenting. I'll write something that just seems to be as clear as
it can be, and then usually me six months later looks at it and
was like, who's the moron it wrote it? Oh wait, never mind?
Yeah, I think we've all been there, right. I mean it's
when I manage a lot of technical teams, and it's always that last ten
percent when you see the documentation. How are we monitoring it? How do
we know if it goes down? How are you backing it up? I
mean, we want to build cool things, right, and then we just
want to pass it off. But I do think that that us as sort
of you know, DevOps engineers, we need to start taking pride in in
sort of skills that are outside of the hands to keyboard technical documentation. Taking
pride and being able to walk away, go on vacation and people knowing what's
going on by reading my documentation without calling me. You know, I think
also being good troubleshooter and this kind this kind of kind of goes back into
the complexity and sort of disaster of every conversation. But to what extent,
you know, is my troubleshooting skills set high? And I think unfortunately a
lot of the soft skills don't have great KPIO metrics that you can kind of
throw on a resume that can show how well you do with those things.
But but I love I love honing the troubleshooting skill and being brought into a
problem that I know nothing about and you know, figuring it out, you
know, quickly compared to maybe folks that wrote it or have been dealing with
it. It's you know, that's fun. It's a great little challenge.
Yeah, that's one thing I've advocated for for years now is my role as
a DevOps engineer is to work myself out of a job, you know,
to set everything up so that it runs and when it doesn't, it's clearly
documented and what stuff's to do, and someone new can come on board and
get their app to production without having to rely on and do so in a
way that makes sure that they honor the constraints of the business. And if
I can do that, then there's no reason for me to be at that
company anymore. And I think that's my own personal metric for job success.
Yeah. I think, actually, you know, not not to not to
pull mortion any objects into the conversation. I think so jen Ai, I
think has a huge potential to help in the screen. In fact, I'm
talking to some startups that are already starting to do this where you will plug
them into all of your internal documentation and they will basically just give you a
chat bot where you can just ask questions and so you know, having service
provider experience. This is this is really interesting because you know, if we're
managing multiple customer deployments. You know, part of what optenine does is pro
we're doing managed cloud ops for managing AWS deployments on the pass of our customers.
But you know, not to say we don't want every kind of similar
to be the owned science project, but there is always going to be this
balance of standardization and customization. And so we have very detailed documentation on each
customer's deployment and diagrams and all that. But it's very hard to scale that,
especially for the person at four am that gets the phone call that something
is down and having to sit through and read all that documentation and catch up.
You know, it's like it's an impossible task to do when you need
to spend hours catching up before you can even begin into troubleshoot. And this,
I think is which is really cool. It's just where Jenny I can
help where if you have this you know LLM that's constantly looking at this data,
and you can have a bot where you say, hey, where's this
customer stuff deployed? When was the last time something was deployed? When was
there a change? And you can just quickly get those answers to me as
someone who's managed twenty four seven teams, I mean, that's just super exciting
and that really helps us scale, you know, the Knock and the stock
organization. Yeah, for sure, because context switching is huge, and that's
where it seems to really raise its visibility of how painful and expensive context switching
is. And I think you probably are very familiar with it from your experience
at OPTU nine when you switch from not only project a project, but customer
to customer and so you are working on one customer's environment that's built this way,
and then you know, the pager goes off and you have to switch
to a completely different environment. And so how do you minimize that amount of
time where you're just sitting there with a blank stare trying to figure out where
to begin in this environment that could have infinite number of combinations yeah, and
i'd say, like now based on you know, you know, saying like
the complexity, it's almost impossible, to be honest, it really is.
And having you know, the tribal knowledge and the experience working on a specific
customer's environment, you know, helps greatly. So what we do is we
obviously try to have as as many standardized tools as we can standardize and monitoring.
I like looking at different monitoring strategies where we have we build monitoring again
into the the CICD work focus far as what we're going to monitor. But
what I'd like to do is to really have sort of macro level alerts go
off at the same time as sort of micro level alerts go off. So
if my application is down, if we're monitoring a specific query and we want
to see that it's returning you know, greater than twenty five results from the
customer perspective, if that goes down, I would like to see you know,
four or five different monitors you know, they're monitoring specific layers of the
back end or specific API and points also going down at the same time.
So the poor technician at four am, we're kind of spoon feeding them,
Hey something, you know, there's a serious problem. But at the same
time, hey, we also noticed these four things that are out of black,
and so instead instead of having a start from scratch that can kind of
work backwards from the lowest hangings. Yeah, just giving them a series of
bread crumbs to follow. Yeah, and again, I mean I think that's
that's a strategy. I mean to me, that's is that a technical Is
that a is that a sort of technical skill or is that sort of a
quasi non technical strategy that you need to employ, you know, with this
resilience or sr HA you know for sure sort of DevOps right there in the
middle of DevOps, I think. Yeah. One of the things I like
to do is in all of my alerts, I like to include like,
hey, here's the alert obviously, here's why it went off, and then
here is a link to the application dashboard and the run book for that,
just you know, to leave those breadcrumbs and help minimize that context switching time.
Absolutely. Absolutely. Documentation. You've mentioned that multiple times, and it's
a pet peeve of mine because I don't like Confluence, I don't like notion,
I don't like read me pretty much. I don't like any of the
documentation tools but you mentioned standardizing on tools. Do you have a preferred documentation
tool. I don't. I've used, I've used sort of all of the
above. You know, I would I would say the answer, I don't
think that there's one tool that's better than the other. Right, And this
is this is a cliche, but right, it's more about the use.
It's like talking about it's like talking about the best diet. Right, it's
the best. It's the one that you can do consistently over time. Yeah,
I say one of the and I think so as long as you as
long as it's simple and you can build them into your work, quote fairly
easily, that that is the best tool. I'll tell you one one win
related to that that I that I experienced years ago. It happened to be
with Confluence, But the same example I know is the same sort of capability,
and it was available and almost all documentation tools now years ago we used
to use. We used to use Zio to create like you know, diagrams,
and then we'd upload them into the documentation tool. And that whole process
of sort of you know, bringing the bringing the work or the output from
one tool and the other. That process, like people don't want to do
that. They'll end up just sort of keeping the diagram, let's say in
their own let's say they're using you know, Lucid Charts or Gliffy or something
like that, they'll end up just keeping it in that account. So a
big win for me was was when Confluence started adding in these plugins where you
can actually create the diagram without having to go out of the documentation system and
have the diagram embedded right into the documents right there, instead of having a
building in a separate tool and an important copy and all that, and so,
you know, I think that was great because now I mean it,
I'm authoring a document, I want to show a visual representation. I'm a
I'm a big visual person, and I can just create the diagram right there
without having to leave the page. It's saved, and now the actual the
actual ip of that diagram is embedded into the document. It can never be
pulled apart. Nobody can ever tell me, oh, yeah, I never
upload I never uploaded the latest version of the diagram into the document. So
there's that whole concept of of you know, working in the updating of documentations
and diagrams into your workflow. I think it's a really good example of how
you can do that. Obviously with them, I think with Youurra and get
help, you can do that. But I don't think that that capability exists
enough for more of an infrastructure operations a sor re perspective, right, Han,
when it comes to like making sure things are up to date, whether
that's documentation or run books or your failover strategy, what's the minimum frequency you
would recommend someone reviewing that, Well, I'd say twice a year is probably
the minimum. But then you also you need to add you need to add
hooks into your change control. Right, anytime you maybe deploy a new service,
you know that should be a hook to whoever is responsible for resilience,
maybe an outside vendor like but it's not if it's an outside vendor like Optie.
You know. So if I'm sitting in the customer seat, I'm going
to add as much as many hooks as possible, and I'm going to say
to my vendor like, hey, we just change this, we just change
that, make sure our make sure our million still works now. On the
flip side, if I'm an opti and ized seat, I might say,
yeah, no problem, We've updated it, which which we'll do in earnest
and we have to but hey, you know, we did what we had
to do, but we got to retest now, right, So you got
you you do have to find that balance. Then it doesn't mean you can't,
you know, update these things in an ongoing basis and then kind of
have a list of what you want to ensure functions during the next test.
I will say, though, one of the important things with testing is you
don't want to just have the I T teams doing the application testing. You
really need to have users testing. Maybe it's QA, maybe you have internal
staff that are using the system. You need people that can smell out a
problem with the application, can smell out the fact that it's maybe a little
bit more sluggish, or that certain functionality doesn't work as good. And this
is a big miss. A lot of our teams try to internalize it because
they want to just move past it. For sure. Yeah, just as
an it background, my my overall objective is to avoid as many conversations with
other humans as possible. But this is one of those areas where you just
kind of can't do that. And I'm guilty of doing it too, of
performing a failover looking, Yeah, all the health checks pass, no alarms,
that must be good and then moving on. Yeah, and I like
the idea of almost making product managers responsible for some of this. You know,
if if resilience and high availability is you know, is a feature,
a component you know of sort of the outward product, then I do think
that they can be the liaison between the developers, third parties or whoever is
whoever's owning the resilience. You do need a quarterback there. And if and
if there is a product product management function, I think this is a great
aspect for them to ensure continuity of long term Yeah, agreed, Like a
seasoned product manager is just worth their weight and goal because they understand all of
these different layers of complexity and interactions between the teams and and just by job
definition, they're really good at at orchestrating and pulling in the right resources at
the time that they're needed. Yeah, and with third parties, right,
if they get win salesforce, you know, what they're going to want to
do is you know, potentially pull in the you know, whatever positive capabilities
are being pulled pulled through, maybe they're pulling it into a product feature step
they also need to better understand that, you know, what it means for
the outward messaging on the resilience or if they can still make good on that
promise. All right, well we are coming up on an hour here,
is there anything else that you feel like we should be covering when it comes
to resilience, dr and managing complexity? So I'd say, like the most
important thing, and this might be a little cliche these days, but you
know it's just make no assumptions, Make no assumptions on any of the platforms
that you're using in regards to what built in resilience or redundancy exists. And
also keep in mind that high availability and resilience does not always equate your ability
to recover from specific types of events. You know, if you're hit with
a cyber attack and your data is corrupted in production systems, you know,
having a replica or having high availability even with multiple regions does not mean you
can recover from that. There are other sort of strategies that you need to
employ. Obviously, you know how far back is your is your you know,
is your snapshot history, your journal, and you know you'll need to
have separate run books for that type of situation then sort of the high availability
type of situations. So just understand there's sort of you know, those are
completely separate and again make no assumptions. Yeah, it's almost like this would
make a really good board game. M that would make a good board game.
Yeah, we should do like a jump to conclusion. Uh yeah,
well done on the office phase reference. Yeah, nice to see that.
Well played. Cool. So if folks, if our listeners want to talk
more about this or reach out to you directly with additional questions, what's the
best way for them to do it? Uh, find me on LinkedIn.
That's probably a time that I'm most active on, you know, or or
right to me on there or at optinin tech dot com or you'll find me.
My name is kind of unique, so I have no doubts that that
anyone who's listening to the show will will not will be able to not be
able to find me. So your name is unique? Is that short for
something? Sagi is a is a Hebrew name. Like other Hebrew names,
they can kind of get butchered, you know, in these parts. But
there are much worse Hebrew names that I, you know, so I don't
have it that bad, but it is. I mean, it is nice
because when I get cold calls, I immediately know that this person never spoke
to me before the land the built in screening feature. Yeah yeah, that's
good. Awesome. Well, thank you so much for joining me today.
This has been a cool conversation and I think it's one that we we need
to spend more time I'm talking about because it often gets overlooked or assumed,
Like you said, make no assumptions. Absolutely cool. Well, and thank
you, thank you. I mean, this has been fun. You're a
great presenter, and it's nice to talk to someone who's kind of also lived
it and been through it as well. Right as old guys have to group
together and tell war story once in a while. All that right, All
right, thanks again the Gie, thank you for listening, and we will
see y'all next week.