How'd you like to listen to dot net rocks with no ads? Easy?
Become a patron for just five dollars a month. You get access to a
private RSS feed where all the shows have no ads. Twenty dollars a month.
We'll get you that and a special dot net Rocks patron mug. Sign
up now at Patreon dot dot net rocks dot com. Hey Carlin Richard here.
As you may have heard, NDC is back offering their incredible in person
conferences around the world. DC Porto is happening October sixteenth through the twentieth.
Go to dc Porto dot com to register and check out the full lineup of
conferences at NDC conferences dot com. Hey, guess what it's time for dot
net rocks Danish style. Yeah, in an airstream, I mean of the
most American of American trailers. You can imagine an are in an airstream.
So this I guess it was n dc Copenhagen at one point, but they
rebranded it dev Festival. Yeah. Yeah, and it's very festivally tense.
It is. They have the Boston Dynamics robot here, one of them.
Yeah, it looks like a dog and it walks around, looks around.
It's got the arm on it. Yeah. It doesn't do anything threatening though,
like you know the one. It doesn't even backflip. Like I said,
you can to do a backflip. He's like, no, no,
that's Stlas. But it's very Uh, it's very cool. We talked about
not making it non threatening too, because apparently they've now made Atlas smaller,
like instead of being human heights, it's quite short, and I think it's
all part about making people comfortable with these machines. Yeah, because they are
creepy and they're hard and heavy. Yeah. I picked up the battery for
that little spot robot. It it's it's a good ten pounds. Yeah,
yeah, it's crazy. Anyway, great show. Really excited to be here.
Yeah, very very excited to be here. Grishmo Jenna is here and
we're gonna be talking with her in a little bit, but first it's better
no framework roll it all right? Man? What do you got? Was
something that Joel helan Uh told me about. It's a it's a it's a
language called wing and you can get it at wing lang dot io, a
programming language for the cloud. Kind of reminds me of Pollumi a little bit.
Oh, I'm not really sure how it differs, but it combines infrastructure
and runtime code in one language, enabling and I'm reading here enabling developers to
stay in their creative flow and to deliver better software, faster and more securely.
And just looking on it, you know, here, here's a little
sample clip. Let bucket equal new cloud dot bucket Grand brand. That seems
pretty powerful, right, Yeah, bucket dot put so very cool, And
there's a local simulator to stay in your creative flow with minimal context switching and
immediate feedback. Like I said, it looks good. I haven't run it,
but you know, Joel seemed to like it and it looks pretty cool.
Yeah, that's awesome. And there is a video there, so if
you want to just watch the YouTube video, you'll probably get a lot more
information about it. Wing lang dot io go get it. Who's talking to
us? Richard grabbed a comment off a show eighteen twenty two, the one
we did with Billy Hollis back in December talking about high level design, and
of course Billy's one of our great thinkers in the UX space, and there's
lots of good comments on the show. This particular one is from par nine
one one. I don't know what that means. Is that an x or
Twitter account? Possibly? Yeah, it didn't map to one, but he
says I always love a conversation to discusses workflow and design. As a developer
who's heavily into windforms development, ensuring design compliments workflows and vice versa is important
and the hardest part of developing a solution. One of the best ways I've
worked on on how to improve this is to actually do the role of the
solution is designed to assist. He's shouldn't find repetitive tasks that you can automate,
So I think what he's really talking about here is do the work,
like what is the workflow? Where can way through it? And the UX
can kind of be dictated from that. So spending your time with the app,
not just thinking about it abstractly. Yeah. Yeah, no, substitute
for that fewest number of clicks, use numbers of change of hand position like
to the mouse and to back again. All those kinds of things make a
better UI. Yeah, it's always the same. Right. You talk to
the customer and you say, if you could write this program, like,
how would you want to operate it start Where would you want to start?
Where would you want to go? What are the generals? What are the
details? And how do you want to get through that? And often people
don't believe you. They like they think that they're stuck in a box of
a form or this or that. But you say, you know, if
this was a car, how would you drive it? Yeah, Chrishma is
shaking her head, nodding her head. Yes, she's been there, no
question. And you know that part of that is watching somebody work, like,
look, how do you normally do this? And all? And you
realize those little jumps outs even after you've built the app, going back and
watching people use it, you know, a few months later going wow,
that's not what I always think. Yeah, right, hey, par thank
you so much. Here a comment at a copy of music Coby. It
is on its way to you. And if you'd like a copy of music
Coby, I read a comment on the website at dot net rocks dot com
or on the facebooks. We publish every show there and if you comment Darren,
we're reading the show. We'll say your copy music cobuy. And you
can definitely follow us on Twitter or x as it's called now, but that's
fine, but the real fund is happening over on Mastodon. I met Carl
Franklin at tech Hub dot social and I'm Rich Campbell at mastodon dot social.
Send us too, and you know you could send us a comment by two
if you like, you can and we will respond. We will respond,
We read them all. Okay, let me introduce Chris Chrishma Jenna is a
data scientist with the UX Insights Team at IBM in San Francisco. I should
just stop there, because that's amazing. That's awesome, the full amazing full
stop right. I mean, you're the only data scientist in the org.
She supports eighty plus user researchers and designers and uses data to understand user struggles
and opportunities to enhance product experiences. She's delivered fifty plus talks and workshops at
multiple conferences around the globe, including piicon US. Grishma is extremely passionate about
encouraging, mentoring, and empowering people, especially women and students, in the
world of technology. She's been recognized as a Fellow by the Python Software Foundation
and also serves on advisory and leadership boards for nonprofits and other organizations. In
other words, a whole lot of awesome. Thanks for going with us,
Thank you so much for inviting me. Yeah, what did you think about
the comment that Richard read there? Very interesting? I think right about the
workflows and just having the usocentric mindset, so that something we focus a lot
at IBEM as well. You must have heard that IBM's one of the frontiers
and on the frontier of the design. So they came up with this Enterprise
Design thinking framework, which is all about making the process user centric, coming
up with something called an empathy map, which tries to put yourself in the
use of the users and be like, Okay, what is this user thinking,
what is the feeling? What is this user doing, and what is
the current you know, state the ass scenario and then hopefully going to the
TV which is the future scenario where there are hopefully no use of frustrations or
struggles and it's a very seamless end to end EXPERI yeah, because of course
my instinct as soon need to say what is user feeling? It's like I'm
expecting frustrated lunch. Absolutely where does the data science fall into this? Because
that's what grabbed me, was this idea. Yeah, you know, so
much of UX design seems intuitive based rather than is there a data approach to
us? A science there is? Yeah, I think this is definitely something
that's a little novel And a lot of people ask me, Okay, you're
a data scientist, but on the UX side, isn't that what's happening,
what's what's going on? Like, shouldn't you be on this quantitative side with
the logical, practical thinking. Yeah, and this is this is definitely a
very unusual role. I will say it's not a stereotypical role. But where
data false is that data is the language that customers used to talk to about.
So I think there's a huge, huge potential there to actually mind all
of that data, all of that feedback that customers are giving us, whether
that's quantitatively through the instrumentation of the products of qualitatively, well you know,
they're on a customers about called saying yes, office sucks, I'm going to
go to the next competitor. You know, I could have done this with
Excel. Yeah, my brother could do this in access. You're you're reminding
me of Mark Miller's Science of Great UI talk where you know, the first
time we talked to him about what you mean science of UI, Like you
said, it seems like into would be taking over, but he talked about
you know, there are certain things about the way your eye sees things and
the way your brain reacts, and and that can be other way. Contrast
is perceived as importance. The things that are higher contrasts tend to be more
important than lower contrasts, And so that can translate directly into rules that you
can use for a better user interface, a better UX experience. And so
that's exactly what I thought of when you said, you know your data science
around UX. There's a lot to it. There's a lot to it.
But I also see we're good. We're getting good at instrumenting stuff now,
the open telemetriies and things, the lists, like we have the tremendous amount
of data we can pull from people using the app, not even talking about
the information of putting into it, but how they interact with the interface,
And I wonder how well we can use that data. Like I've worked on
a bunch of that, and we end up with doing heat maps where it's
like, oh, their mouse is always over here, and it's it's rarely
over there that always do you go to eye track. I haven't dealt with
eye tracking data myself, but yes, I know that some of the teams
use the eye tracking data. But again, like you mentioned, right Richard,
with a lot of instrumentation data, where are the clicks happening, where's
the mouse activity happening? How much are they strolling through the page? Are
they skipping over some demos because they're like, yeah, yeah, I already
know this stuff, like you know, just just get me to the good
stuff. I know all of this basic stuff. But I would say it
also really comes down to two things. Number one is what data are you
exactly capturing? How are those metrics defined? Right? Because what might be
a conversion for my product or for my perspective might not be a conversion according
to you. So I think just having that level of standardization of Okay,
this is how we're going to track a log in, this is where you
know, this particular workflow starts, this is how this particular workflow ends.
What happens if there's some drop off in the middle, or you know what
conversion means someone who gets to the end of a work yes, exactly right,
So whether that's a purchase order going through or somebody putting you know,
starting a workflow again? Now does that mean do you start a workflow?
Doesn't mean ending the workflow? Where do you exactly when a motion points?
I mean we usually think about conversions from an e commerce perspective, Yeah,
but I love the idea of applying that to an internal business workflow. That'd
be a great number to have. I've never thought about having. Is how
many times have people started down a form? Yeah, and then a band
abandoned it? Yeah? And if you could see a pattern that data is
that they consistently abandoned here, yes, which might be they need additional piece
of information that interrupts them enough they have to leave the app to do that.
Like now you've clearly got a feature waiting to be built. Absolutely,
And I think the other thing is also the amount of data are capturing right
that that plays a huge role because unfortunately I've seen too many of these instances,
especially now given that AI and Jenny I chat GPTY is pretty much do
you know all people are talking about executives coming and say, Okay, we're
going to build this genitive AI model with deep learning in machine learning and all
of that. But when you go to look at the data. There isn't
a lot of data. There is not even good quality data available to do
those modelings. So I think having some realistic expectations really help speed up the
process. It also strikes me, especially when talking about the large language models
and even the general mL stuff, that's a pretty big hammer for what is
not that big a problem if you're thinking about it, well, like I
just wonder. I mean, it's cool, it's cool hammer, but it's
like, really do you want to That's like it's almost like, hey,
we just rewrote this and Ruby on rails, our problems go away. But
the more important thing is like you're going to leave me alone for six months
while I do that, Like if you pull out the mL hammer, you're
really just getting a three month pass. Yeah, when actually studying the data
meaningfully could get you to a solutions faster and simpler yea, or at a
minimum, give you better inputs for that machine learning model in the end.
Anyway, Yes, completely, I agree. I heard this. You know.
An ology that someone used is that machine learning or deep lunning models are
often the ferraris when you might even need like a bike to I'm talking a
pickup truck, a two wheeler will do without a motor. Yeah, so
I mean that I've found these days with telemetry specifically, it's either way too
little and you never you can't get anything useful lot of it, or it's
so much it's just unmanageable. Yeah, how do you try and thin the
data out? There? Is there a tooling approach? I mean, if
it takes too little basimbly, nothing we can do. So we kind of
got to go with it too much. So now what Yeah, I think
the kind of approach I leaned towards is start with things you are absolutely confident
about. We become up with two metrics that you say, okay, the
conversion rate and you know, maybe the number number of logins per month or
you know something like that. Just come with two metrics that you're absolutely confident
and that you can manually check that, Okay, what's being instrumented, what's
being captured, is what we are expecting, and then you feel confident.
Maybe keep building onto that sect thing through a bunch of obvious and errors for
a log in, yeah right, the speil log and failed log at an
incomplete loggin. I don't think it's more than that. And now can we
see them all can we now see a number behind that? Or would you
start with the happy path? Depends depends. So the way I like to
do it is that let's pick up two metrics in the instrumentation phase that we
are confident to it that we can check. Once we have that in plates
lists, start thinking about what future questions we would want to answer. So
I might not have the data today, but three months down the line,
I want to see how many users are abandoning on a particular workflow, right,
are converting to a particular workflow, all right, and then start those
that reverse engineering of okay, to be able to answer that question. The
kind of data I would need is the abandons, maybe the Paige speed load.
Maybe you know how many demos they are watching, how much time they
are spending on this particular product, And then go back and see what other
metrics that you could come up with actually start tracking all of that data.
Now, when you're talking about data science and in this I know you mentioned
AI a little bit passively, but do you use predictive analytics on this data
to try to figure things out or any other kind of you know AI if
you will, Yeah, yeah, okay, So a bit of a disclaimer,
I'm not really sure where the lines between artificial intelligence and data science,
you know, blur, but I always feel yeah, I mean data science
is really at the very very lowest level of AI. It is all statistics,
it's data sensors, machine learning. So just for the purposes of our
conversation, I'm going to interchangeable use or just maybe stick. Do you know
data sciensing machine learning? Yes, Predictive analytics is definitely something very popular.
I think one of the most most abundantly available use cases is predicting if a
customer is about to leave your product, So it's basically predicting customer attrition.
So you can maybe start looking at you know, has their overall usage of
the product decreased? Have they opened a lot of support tickets recently? Maybe
those support tickets have not given resolutions to them right, or maybe the average
time taken to handle its support ticket is increasing. You also have a lot
of feedback surveys that they give, right like how happy are you with this
experience? Would you recommend our product to a friend or a colleague, and
then just looking at the trend of those goals. So that is definitely one
case where you know the predictive models can help and alert the product team that
here this customers at a high risk of actually, you know, terminating the
contract. Yeah, so so better go ahead and maybe have like a touch
point of okay before the customers actually talking about canceling exactly. The idea and
the idea that you would contact them and say, hey, you've been having
a time can we help you? Like, what can I do? Extra?
Yeah? Why did you post this on Twitter's angry Sweet? Yeah?
Just call support? Yeah, you know I have. I think I talk
about this the show that Bloody Dishwasher, the Melee. Yeah. I kept
breaking Yeah, and I couldn't get regular support. Regular sport wouldn't respond to
me four weeks. But if I tweeted and tweeted yea that day. Yeah.
And at one point I'm like, listen, I'm not that mean a
guy, right, I just didn't get anywhere I would Is there a better
way to do this? And the and the person on the other end's like,
literally, nope, this works, just use it. I'm I'm in
that limbo period right now with the HVAC system in my studio. Oh man,
yeah, they's a new one. Well they I don't want to take
up too much time with this story. But they sold me a twenty four
thousand BTU unit for a six hundred and eighty square foot room that's insulated to
our twenty R fifty R fifty okay, really freaking insulated. Yeah. Yeah,
you don't have to pump very much cold air in there. Yeah.
So it cools off in about five seconds with this thing, and then it
stops, and so the humidity rises to sixty five seventy percent, Yeah,
and it gets unpleasant. And so I had to beg with them to you
know, get somebody out here to fix this, and they finally decided to
get a smaller unit in there, but then they wanted to charge me more.
That's like, wait a second, you screwed up, and you want
to charge me more. Yeah, And so now we're going we didn't get
anywhere with their subcontractor, and now we're going to the actual contractor, who
company who we bought it from, right and there, and they are unresponsive
right now. And if they come back with I'm sorry, you're gonna have
to pay it, then then I'm going to unleash the kracking. Yeah,
do the thing, yep, yep, you just need to send a tweet
to them, right, if there's to start, but begin the public humiliation,
right, let it begin. Well, we'll keep you posted on that.
Yeah, no kidding, but I mean you still your language to me
strikes as external customers reacting to product, right, I mean, and that's
fine, Like those are all good things, and it is interesting to synthesize
data from your tech support side as well as perhaps a sales channel. Maybe
you're doing some monitoring at the social media level, to say, do we
have some sentiment analysis going on? Coordinate this information, but then to add
the utilization of the software. Yeah, Like this is not just a person
being randomly grumpy, right, it's this is where they're falling off on the
workflow and they do that consistently. Like, this is not a whining person.
They're trying to get it done and they're abandoning for whatever reason. Like
that that is such actionable data. I am you are watching this customer try,
Yeah, Like shouldn't you do something about that? Yeah? Yeah?
And I think one case where data science comes in really helpful is that maybe
as a front front facing you know, you're facing the client and you're having
these conversations, maybe it's a product manage. Maybe it's a develops having a
conversation with a very frustrated customer, right saying, Okay, you know this
isn't working the way it's expected. I'm not going to do a bunch of
issues. And that's a data point that you have. But when you look
at the data perspective or quant perspective it you get that scale, right,
So now you start to notice this is not just one isolated customer, it's
actually ten customers, hundred customers, right, thousands of customers that are doing
that. And I think that helps to also remove that bias, right,
which it could work in both ways, not just that customers the pain exactly.
I guess the next question you asked when you talk about like a failed
workflow like gus, well, how many succeed? Yeah? How often do
we succeed? How many other abandons do we have? I really like that
you take the personality element out of it to some degree, and it's like,
here's a set of data and you have found with someone willing to communicate
with you about the problem, don't make them the enemy. Here's all these
people that aren't communicating about this problem but are having the same problems. Yeah,
exactly, yeah, and it works two ways, right, Like the
user researcher could maybe be in a conversation and then notice that, okay,
the customer is struggling with the product, and then they go to the data
science team or the analytics team and say, okay, we saw this one
person. Are there other people who are struggling with this? And then you
notice that you know, this effect is that scale. And another thing it
also does is like often product managers or you know, a client facing teams
here from really important like it's called the hippo the highest paid person's opinion,
and it's usually the highest paid customer, like we hate this feature. When
are you in unveiling? You know, XOY is that feature and they're just
going to be you know, maximum attention to them because that's where the money
is exactly, all right. But then when you combine it with the data,
you can actually realize that is this person's problem? You know, actually
yes, is it unique to the customer or is it a little more pervasive
where other customers are also facing this? And then it's probably a future worth
including on the road. Oh man, now you've now you make me feel
like it's a superpower, like if you applying data science to all of these
kinds of problems so that you're always talking from more than the immediate issue in
front of you. Exactly. My automatic reaction to a person's having a problem
with this feature is to be able to apply some data analytics, say how
many other people? How many six? What are the other scenarios? Like,
I mean, even think aboufter our past scenario where you've got a customer
talk and now you've found out this other customers failing, Like lots of people
exclude themselves from serving, but they don't want to do that. But to
call that customer say hey, you know, I'm from such a such a
company and we'll see you're struggling with this thing and we're trying to address it.
Would you help us to understand what you're trying to do, Like,
I think you'd get good response from that. Yeah, I mean, you're
going to hum the fine line of being creepy. You know, we've been
watching you and we see you're not having a good time. Better to say
that you're a hired psychic. You've discovered that they're the spirits. Have told
me about your unsatisfaction. But I know. I do think it's an interesting
part of this is to just have that reflex of let's can we quickly gather
more data, like do you see an opportunity to build up tooling? Like
I'd really want to make that mechanism simple. I don't want to have to
call you Krishma as the expert to say, can you give me analysis on
this? Yeah, I wanted to be a button the moment it pops up.
What is the IBM system that you talked about earlier, You're sort sort
of hinted at it. What is that play? Yeah? I think we
have a bunch of different tooling and framework and to be honest, one of
the issues we're facing a standardization because just the software unit that I work with,
we have around eighty two hundred different products and all of them have absolutely
varying degrees of data maturity, right from like we haven't really put this product
into production, or we only have like two clients, all the way to
we have thousands plus, you know, users every month, and this is
just a huge amount of data. So I think it's it's hard to generalize
what that looks like, which is why where I need to have a personalized
approach for the team I'm dealing with, the product, I'm dealing with,
what is the exact objective or the problem statement that we're trying to find automation.
That's that's very interesting because that is definitely something that I'm trying to do
as well as unit I've seen a lot of other company needs do as well.
Is this concept of anomally detection, right, right, So throwing all
of the data into a model that will help you understand if a particular user
activity or a particular user is you know, I'm showing up as an anomaly.
They're doing something different, they're doing something unexpected, maybe they're running into
a bunch of issues or errors. And then that also goes into with incident
management. So a lot of security teams do that to see it, right,
maybe there's a DDoS attack happening, that's anomally for you, so that's
an incident, right, So using kind of the same principles, So that
is definitely one way of automating. But again that only comes into play when
you have a highly mature team in terms of the instrumentation, in terms of
the data, in terms of understanding how many users you have, yeah,
and what those scenarios look like like it does feel like something I'd want to
build into a tech support client. So not only am I pulling up the
anomaly, but then provides analytics run the anomally say this is the four hundred
times this has happened this week. Yeah, right, and with this many
different customers, just to give a sense of urgency and scope to that.
Like I said, I love the idea of being able to resist the hippo
because they're asking for a fitture that's distinct to them. The majority of customers
are fine. Yeah, And at the same time, when you have a
problem, show up to see, okay, well this is unique to that
customer. I can approach it differently. And I'm not saying like ignore it
but so much, but recognize it's an unusual work Yeah, as of course
to it's a usual workflow that often fails, and maybe you can get a
shape for it. But also you know, maybe it fails ten percent of
the time and ninety percent time it succeeds. Like now you've got a different
shape of how you're going to approach it. Looking at the problem, it
feels like it's time to take a break, So we'll be back after these
very important messages. Hey, we're back. It's Carl Franklin. This is
Richard Campbell, and we're here with Grish Magenna and we're talking data science and
UX and I'm very fascinated by this UX sort of a science framework that IBM
has. Well that is there any one of those things that you would use
at the onset of a project to help design the US or are they sort
of things that you work with after the fact, Like this last example you
gave, seems like you have a mature team and a mature project. Yeah.
Yeah, I think again going back to like at the start, it's
more of let's start thinking about what are the questions you want answer and what
are the metrics you need to track, So it's probably at the earliest stage
of data collection and just deciding what those metrics should look like. I think
one thing I am trying to do as well is when I'm working with these
mature teams and royalizing there are sort of inconsistencies with the data. I'm actually
trying to create this guide of Hey, I worked with this team that had
great data maturity, but we still had inconstituencies, we still had issues figuring
out, you know, what was going on. So as a heads up
to you before you even start instrumenting, learn from our mistakes, take take
our failures into account, and here are the recommendations that would make you know.
So that is definitely some sort of you know, activity that I'm trying
to do earlier in the pipeline. But yeah, I think, I mean,
that's that's all you can do, right because you can speculate. But
at the end of the day, when the data starts full flowing, and
that's when you can actually do some expiation and and that it's an understand is
this good enough? Is this good quality data? Does this make sense?
Or do you maybe need to you know, modify a few things. And
do you have a sort of general knowledge database of things that you've gleaned from
projects working? You know, when somebody says, you know, customer says
that we're thinking of doing it like this, and you say, well,
you know, based on our experience, that approach hasn't really worked well and
years why, like, do you have this sort of general knowledge that you're
gathering that you used too? Yeah, I do have it unformally, but
formally I think this reminds me of what is my primary responsibility is to create
this insights repository of sorts, which is where all the user researchers are coming
in and they have findings, and more often than not, all of these
findings are stowed away and like you know, drives and folded. And let's
say somebody leaves the company two years later, product manages like, oh that
thing was done two months, two years ago, where is the token?
I find? You need to talk to ten people? So we're definitely about
knowledge management. Yeah, it would be really cool to upload that to a
chat chypt like things that you could ask it questions directly. So what do
you think about this? Well based on past experience, Yeah, exactly.
But then also am I other risk of automating my job away? I don't
know. I mean I've always had the experience that there's we're never getting to
the bottom or to do list anyway. So knocking automating stuff, you're just
going to do more faster, We're just going to do more things. But
you have worked with some interesting customers I saw. I haven't seen the top,
but I've seen the abstract for the top where you talked about folks like
Airbnb, and Spotify and so forth, like can you dig into any of
those scenarios. I don't want to get any any trouble but here, but
it's like that just sounds cool. Yeah, nom, So those are those
are case studies. I wish I could claim that those are the projects that
I haven't done. Those are case studies, though, but very interesting ones.
So I think Airbnb and Spotify does a really good job of having their
ux research teams, design teams and their data science teams work in very tight
conjunction with each other. Yeah. So, like I mentioned, right,
you could have a user researchers that says, Okay, this customer, this
users having something odd, let's go to data scientists. But conversely, you
could have data scientists are looking at that aggregated level of data and saying,
hmm, this is kind of unexpected. We didn't really expect such low conversions.
To check the metrics. The web pages are working fine, the instrumentation
is okay, what's happening? Can you use a researcher go and talk to
the customer and try to find out what's exactly going on. That's where the
quantitative and the qualitative aspects of it. This is this is the title from
your child, right, the US data analysts they should be friends, Yes,
they should be best friends friends. Okay, well, because you get
this problem with ux of it only being intuitive and a dodo. Yeah to
the data science folks are going to come in, are going to help you
beef up your case for work absolutely and vice versa. There the data folks
are going to see these anomalies in the arc, like the larger data sets,
and be able to come back with have you looked at and maybe turn
up some other issues exactly exactly, And I think especially as more companies and
executives go into this data driven decision model, right, it really helps to
have that evidence. So you can say, my instincts or my intuition based
on all the research I've done, say that we should, you know,
modify the workflow, we should introduce this feature. But here's the data to
back it up. We spoke to our users or we have looked at instrumentation
data. So here's the evidence that I can you know, help prove across
a point with Because anytime you change a working data flow, you're impairing its
productivity for some period of time. Like people know how to use this,
so the improvement has to be big enough that thing initial impact of the change
is offset by the net long term net benefit. I guess an interesting ROI
question. It's like, hey, we're not going to change this because it
affects so many users and the inkroll improvement doesn't appear to be substantial enough.
That'd be an interesting place, like I don't know if any organizations I've ever
dealt with where they were that mature. Thank you. See, this is
a design improvement, but it's not a big enough design improvement. So I
think we're going to do it and until we can find something large enough that
it's worth disrupting the workflow flow yep, yep. And even then, I
think communication is really important. You need to tell the users here, this
is our latest updated version, and this is where now this menu has actually
gone over to the right side, and these are the new features that have
been on wheals. So I think that communication and almost change management, is
it really important so that the expectations of the users are you know, kept
kept in check. Yeah sure, and yeah, I certainly. I talked
to folks who have client fatigue that stuff, especially in the SaaS products and
cloud stuff. It changes too often, right, And I spend enough time
on the sytems. It's like I get tickets all the time, and I
can see the wave of oh they updated, decline again this week, and
I'm getting all these tickets like somebody move my cheese? Right, and you're
back to what the heck happened? How is it different? Here's how we
deal with it. Talk to you next week. Yeah, and that This
is another case where data science comes in really handy, because then you can
start doing AB testing right and actually look at the data. Rights this change
actually useful? Does this make sense to users? Or are the users are
frustrated and they just keep going back to like, you know, take me
to the old layout, take me to the old format? Can I turn
it off? The first question? Every time? And I love you know,
AB testing isn't always obvious, right, Like how do you know which
to pick A or B? Like? Why was it better? Yeah?
You know, Okay, you serve them fifty fifty percent of the time.
Like what's the metric that shows it was a superior outcome? Higher completion rate?
I guess would be the one? Yes, yes, definitely. I
think you also need to look at the demographics of users, right, did
you do the branching based on where the users are based? And if so,
were there any cultural differences demographical differences that contributed to that? So I
think just it's definitely a very complex topic. How do you have those confounding
variables? How do you make sure they're bearable to attracking, are independent of
each other, or at least as independent as they could an experimental sets.
Now you also have to look at things like is there a geographic event happening
right now? An earthquake, a flood, a heat wave, not that
that would ever happen, Nah, a forest fire? Looking at you,
Richard. Yeah, this comes down to really, how do you how do
you separate A and B? Like? Why? Because my instinct as a
developer is just gonna make it random. But nothing's actually a random So what
do you mean when you say random? So, I mean, at a
minimum, if you're thinking webish here, I've got a cookie, I'm going
to stick a given cookie to A, and the next new cookie I create,
I'm going to stick them to B. Yes? Is that sufficiently?
It's not random at all? Actually, it's a cyclical measure. You haven't
I mean, you're ignoring all the other factors that good enough. I don't
know me the data can tell yeah, yeah, how much do we know
about that cookie to actually say well this is who got it? What was
a separation on them? Yeah, I don't know that I've ever looked at
the data that way, Like Ashley dug In and said, did we make
a good A B test from a client selection perspective before we say hey A
out A outcompleted B therefore A is better? Not even making me think too
much, christ Man, It's all about the biases, right, selection bias,
confirmation bias, recency bias. Everything I've thought about it is a lie.
Yeah, well even in fact that I'm just selecting new cookies so my
existing users don't see it at all, Right, like that that could be
in effect as well. So many opportunities to lie, yeah, or to
just or to select a truth yes in advance ways to filter? Do you
ever have to tackle those kinds of problems like how are we going to separate
to request these request streams to make sure or we get a good AB mix?
Like what does that even look like? I don't think I've actually done
a lot of AB testing on this current role, not not so much.
But yeah, again, I think I'm challenging more with that volume. And
you know, data quality is shoes right now. So hopefully in the future
I'm hoping to get to those good, nicer problems to have. But right
now it's a lot about education and awareness about how the instrumentation needs to happen
and what other things that's possible, because oftentimes it's also just the black box
perspective of AI, right, like, oh, it can do anything and
everything, but we know it's not true to start from. I'm also thinking
from the friends perspective of often it's just a developer decided to do that.
We've decided to do an ab testing group of developers there, it's like you
should be calling your data science people at that moment to talk about how we're
going to discriminate these Yeah, let those experts get into the play and even
know that they're doing that, because that can give us a make the get
a better result if we work at it a bit. Exactly. Yeah,
And I think even having these multicultural, diverse teams all hell because there's a
lot of communication messaging involved. For example, I recently came across this website
that says that each color has very different connotations culturally, like read in some
cases might mean danger or something negative, but in other cases it's actually a
sign of prosperity, right, well, and then the Chinese us right,
yes, exactly exa and the Chinese dragon is a very powerful positive thing,
and European culture it's a negative, right yeah. So yeah, where you're
using that stuff matters as well. And certainly words, right, I mean
it's some words that translate to other languages to mean embarrassing things. Yes,
we see that over and over again. It's the famous Nova story, right,
yeah, Nova, it doesn't go ye no, Chevy Nova, don't
call a card that in a Spanish speaking control mistake, or the slogan Coke
ads life when translated to Japanese. I think it was our Chinese one of
the languages Coke will bring your ancestors back from the dead. Yeah. I
think the most recent one was the video came out with their latest emmal model
or one of those framebooks, and basically translated to Spanish, it meant bottoms.
That's why you see like all these nonsense words as names of products totally
like that, because they're just they don't exist otherwise boid. It helps,
like, for example, work just Fine, which is the sound beforehead slap,
also is the name of my production company. So you know, we've
been digging in purely on the telemetry side, all of this very quantitative data,
but there's a whole qualitative side, right Doing hot surveying well, I've
come to appreciate incredibly hard to do. But as soon as you you know,
you want people to type something or write something, now you got to
analyze it all, like what do you do? Oh? Use researchers are
actually pretty good at this, but unfortunately it's in manual efforts, so scaling
it up can be very time consuming for sure. That's another place where I've
kind of loved this use case is to help analyze surveys or reviews or you
know, Twitter complaints or whatever social media mentions that you have. So what
I like to do is, you know, run all of these through Python
scripts that I've developed and then try to understand what are those top teams,
keywords and sentiments coming up. And while I will be the first to admit
it's never going to be a hundred percent perfect, it's usually a really good
starting point that then I can handle over to the user researchers and they'll get
a lay of the land almost like Okay, you know, there are a
lot of mentioning about us, or there's a lot of negative sentiment attached to
this new version that we had. Let me go in and manually dig in
a bit and see, you know, what are those more insightful things that
I can come up with. So that's another you know, symbaetic relationship we
have. It seems like a multiple choice survey kind of thing is the way
a lot of people do that kind of stuff. But I mean there's there's
problems with that too, right, Like what if you don't provide the right
option, you know, and then you have other and now you give them
an Irish text. Now you're back to text again. So what are some
of the other problems with that approach, the multiple choice survey kind of thing.
I think there's always the order that the options are presented in, right,
especially if you're not really paying attention to the survey and you're just like
it's like the first option and just you know, I want to quickly complete
the survey, So that that is definitely a bias that can creep in as
well, and if people just don't want to do it, they will just
pick random things exactly exactly. Yeah, I think language really the way you
write that option it is left to it's subject to interpretation. Right. What
I might interpret as option it might not be what others interpret as OPTIONE.
So having that, I always like having those open ended text responses because then
the users can add in a lot more information. They can also give you
a context that hey, you know, it makes sense if I'm doing X,
y Z. But actually I would like to choose option B on a
regular basis because I do another workflow way more often than I do option A.
Brian McKay works for a company Roster, that does AI analysis of text
of comments that people leave and determines whether they're relevant or not and actually turns
them into data. And they're using GPT. Yes, yes, that is
That is definitely a very interesting use cases. How do you identify if one
this topic or this comment is relevant and two is this actionable? Right?
Even if it was relevant, you could just have the comments saying this product
is awesome, I love it, or this product sucks I hate it,
but there's nothing really actionable from there. And oftentimes some of the best.
Some of the best ideas for the products come from the users themselves because they
are at the center of this. They use this day in and day out,
and they are probably the ones that are going to say, hey,
have you thought about this feature? Or your competitor already has this feature?
You know, I would love to see this coming in here. Sort these
comments by actionability. Yes, that's a cool concept, and there you like,
that's I really wanted to get into this. Where would I use machine
learning? What is the right thing to do? That's something? Sort these
by action ability? How do you get the tool to assess that? But
I guess this is what LM's in theory or could app Yeah, yeah,
yeah, given given the right domain knowledge, because I think a lot of
this, you know, enterprise products especially have so much technical juggy. Yeah.
Domin knowledge is really really important, right, because maybe what means workflow
to us, like in general language, might be a very very specific activity
that's happened. Defining addiction what do we call it? A glossary of vocabulary
right at the front of a project makes so much sense. I got bit
by this once. I had a customer that we developed an app for.
I'm not going to say who it was. But at the end of it,
they basically said, well, where's the the feedback page. I think
it was feedback or maybe it was a comment screen, right, and I'm
thinking we did a fill out form, right, But what they meant by
a comment screen was a live chat. And they were like, oh no,
this we said this in the steck, we want to comment screen.
I'm like, yeah, but that definition was never defined anywa. To them,
it meant a live chat, which hack of a lot more software,
a lot more than writing a comment and having a store exactly. Yeah,
yeah, surprise, yeah, yeah. So when you say comment screen,
what do you mean exactly? And it's it's one of those things that we
all take for granted because you know, we thought comment everybody knows what that
means. You know, you leave a comment or an it's a contact page,
a contact page, contact page. So when I say contact page to
you, it's fill in this form. We'll get back to your later,
that's right, not a live chat. Yeah. So, hey, it
pays to go over every fine detail when we're doing that. Well, I'm
picking back to our comment at the top of the show with par about you
know, watching people in the workflow, are actually going through the workflow.
Yeah, Like if you've done that and you've gotten to the contact, yeah,
it would have been and now we chat back and forth. You mean
chat screen. Yeah, Oh well it's a different thing entirely. I certainly
on my radar to do more shows around building a large language model sort of
domain specific, because that seems to be we don't need existential conversation with software.
The generalized ones are amusing. We can debate the value of search on
that, but the idea that it understands a particular domain and can help organize
large amounts of data. Effective summarizer, a prioritizer like those seem like very
practical implementations of this. Being able to sort comments into ones that are actionable.
That's a huge amount of time that you could really nail down. It's
just an interesting metric of is it relevant, you know, does it talk
about a product in a way, and then you know those are all elements
that come down to this concept of actionable ability. Yeah, but it does
mean we're going to be training our own I'm hoping the tooling gets better.
Like it. This still seems hard to approach it, does it does?
Yeah? And again it really goes back to the amount of data. How
do you define actionability? Do you have a ground source of shoot that says,
these are the comments that are actionable because you know, these keywords were
used, or maybe they were really long comments, which again there's a correlation
right to having longer comments has a lot of more information in there, versus
these are the ones that seem actionable or maybe aren't even relevant. I think
this one time I was doing some reviews analysis and there were those automated you
know spambots, you know, putting in really random advertisms. I'm like,
no, that is not relevant at all, and I don't even know why
it's in the data set. So I think that's definitely going to be an
interesting thing to deal with. Yeah, yeah, interesting times and yeah,
definitely new tools emerging. Still, we're going to get better at this.
All the more reason to buy pizza for the data team. Of course,
I'm a big believer in you know, folk city together, and so anytime
you can do this cross team, it's like, how do we help each
other? Like you really got me thinking about the US team wanting to have
access to more data before they act on those problems. I'd do the same
thing, but with barbecue that total yea and vice versa. That you'll hope
the data analytics team that's looking at telemetry and so forth are I think routinely
just saying hey, here's what we're seeing today, because often they may not
even it may not occur to them that what they're seeing is unusual. Yeah,
so they just show it to the US to periodically say the US folks,
here's how we see how your apps being used. Hope that fits with
what your expectations were, because it might just kick off a conversation that finds
something for you lose customers over it. Absolutely, and that toll triangulation of
that qualitative plus quantitative or UX resocier or plus data scientists coming together and giving
you those different plus spectives is so so powerful, right because it almost feels
like as a data scientist or as UX resocial you're looking at a very small
part of the picture. But then when you start talking to different teams and
you know more, get that vital perspective, that's where you can see the
entire picture of oh, okay, this is what's actually happening with our users
in the product. Awesome. Yeah, so what's next for you? Have
you done your talk yet? No? I am busy all day on Friday.
That's what we wanted to do this on Wednesday. So because you've got
a busy day on Friday, I looked at your schedules like, I don't
know, did you lose a bet, Like you're working the whole day.
Yes, I have no idea why that's been scheduled that way. But you
know, you enjoy the conference for a while. I guess, I guess,
yeah. I would rather get out of the way any other conference and
then have more time. But well, it's Christmas. Has been great talking
to you. Thank you some excellent ideas. I really appreciate it. Thank
you so much. All right, and we'll talk to you next time.
I'm dot net What is it? Oh? Yeah? Dot net rock Nice.
Dot Net Rocks is brought to you by Franklin's Net and produced by Pop
Studios, a full service audio, video and post production facility located physically in
New London, Connecticut, and of course in the cloud online at pwop dot
com. Visit our website at dt n et r o c ks dot com
for RSS feeds, downloads, mobile apps, comments, and access to the
full archives, going back to show number one, recorded in September two thousand
and two, and make sure you check out our sponsors. They keep us
in business. Now go write some code, see you next time. It
is harder than my Texas