WEBVTT 1 00:00:01.000 --> 00:00:04.759 How'd you like to listen to dot net rocks with no ads? Easy? 2 00:00:05.320 --> 00:00:09.400 Become a patron for just five dollars a month. You get access to a 3 00:00:09.480 --> 00:00:14.240 private RSS feed where all the shows have no ads. Twenty dollars a month. 4 00:00:14.240 --> 00:00:18.399 We'll get you that and a special dot net Rocks patron mug. Sign 5 00:00:18.440 --> 00:00:23.679 up now at Patreon dot dot net rocks dot com. Hey Carlin Richard here. 6 00:00:24.039 --> 00:00:28.480 As you may have heard, NDC is back offering their incredible in person 7 00:00:28.559 --> 00:00:33.960 conferences around the world. DC Porto is happening October sixteenth through the twentieth. 8 00:00:34.240 --> 00:00:38.280 Go to dc Porto dot com to register and check out the full lineup of 9 00:00:38.320 --> 00:00:55.280 conferences at NDC conferences dot com. Hey, guess what it's time for dot 10 00:00:55.280 --> 00:00:59.719 net rocks Danish style. Yeah, in an airstream, I mean of the 11 00:00:59.759 --> 00:01:03.920 most American of American trailers. You can imagine an are in an airstream. 12 00:01:04.159 --> 00:01:07.879 So this I guess it was n dc Copenhagen at one point, but they 13 00:01:07.879 --> 00:01:15.200 rebranded it dev Festival. Yeah. Yeah, and it's very festivally tense. 14 00:01:15.359 --> 00:01:19.920 It is. They have the Boston Dynamics robot here, one of them. 15 00:01:19.000 --> 00:01:22.959 Yeah, it looks like a dog and it walks around, looks around. 16 00:01:23.000 --> 00:01:26.439 It's got the arm on it. Yeah. It doesn't do anything threatening though, 17 00:01:26.599 --> 00:01:29.760 like you know the one. It doesn't even backflip. Like I said, 18 00:01:29.760 --> 00:01:30.359 you can to do a backflip. He's like, no, no, 19 00:01:30.640 --> 00:01:34.079 that's Stlas. But it's very Uh, it's very cool. We talked about 20 00:01:34.159 --> 00:01:38.799 not making it non threatening too, because apparently they've now made Atlas smaller, 21 00:01:38.920 --> 00:01:42.599 like instead of being human heights, it's quite short, and I think it's 22 00:01:42.599 --> 00:01:45.760 all part about making people comfortable with these machines. Yeah, because they are 23 00:01:45.920 --> 00:01:49.319 creepy and they're hard and heavy. Yeah. I picked up the battery for 24 00:01:49.319 --> 00:01:53.760 that little spot robot. It it's it's a good ten pounds. Yeah, 25 00:01:53.840 --> 00:01:56.560 yeah, it's crazy. Anyway, great show. Really excited to be here. 26 00:01:56.640 --> 00:01:59.799 Yeah, very very excited to be here. Grishmo Jenna is here and 27 00:01:59.799 --> 00:02:01.799 we're gonna be talking with her in a little bit, but first it's better 28 00:02:01.840 --> 00:02:13.159 no framework roll it all right? Man? What do you got? Was 29 00:02:13.199 --> 00:02:17.960 something that Joel helan Uh told me about. It's a it's a it's a 30 00:02:19.039 --> 00:02:23.639 language called wing and you can get it at wing lang dot io, a 31 00:02:23.800 --> 00:02:27.080 programming language for the cloud. Kind of reminds me of Pollumi a little bit. 32 00:02:27.159 --> 00:02:30.240 Oh, I'm not really sure how it differs, but it combines infrastructure 33 00:02:30.280 --> 00:02:36.039 and runtime code in one language, enabling and I'm reading here enabling developers to 34 00:02:36.080 --> 00:02:39.919 stay in their creative flow and to deliver better software, faster and more securely. 35 00:02:40.840 --> 00:02:46.199 And just looking on it, you know, here, here's a little 36 00:02:46.240 --> 00:02:53.120 sample clip. Let bucket equal new cloud dot bucket Grand brand. That seems 37 00:02:53.120 --> 00:03:00.400 pretty powerful, right, Yeah, bucket dot put so very cool, And 38 00:03:00.919 --> 00:03:07.400 there's a local simulator to stay in your creative flow with minimal context switching and 39 00:03:07.479 --> 00:03:10.719 immediate feedback. Like I said, it looks good. I haven't run it, 40 00:03:10.840 --> 00:03:15.759 but you know, Joel seemed to like it and it looks pretty cool. 41 00:03:15.879 --> 00:03:17.360 Yeah, that's awesome. And there is a video there, so if 42 00:03:17.400 --> 00:03:21.719 you want to just watch the YouTube video, you'll probably get a lot more 43 00:03:22.840 --> 00:03:28.080 information about it. Wing lang dot io go get it. Who's talking to 44 00:03:28.159 --> 00:03:30.919 us? Richard grabbed a comment off a show eighteen twenty two, the one 45 00:03:30.919 --> 00:03:34.680 we did with Billy Hollis back in December talking about high level design, and 46 00:03:34.719 --> 00:03:37.560 of course Billy's one of our great thinkers in the UX space, and there's 47 00:03:37.599 --> 00:03:40.120 lots of good comments on the show. This particular one is from par nine 48 00:03:40.240 --> 00:03:45.240 one one. I don't know what that means. Is that an x or 49 00:03:45.360 --> 00:03:49.840 Twitter account? Possibly? Yeah, it didn't map to one, but he 50 00:03:49.879 --> 00:03:53.159 says I always love a conversation to discusses workflow and design. As a developer 51 00:03:53.240 --> 00:04:00.639 who's heavily into windforms development, ensuring design compliments workflows and vice versa is important 52 00:04:00.639 --> 00:04:02.520 and the hardest part of developing a solution. One of the best ways I've 53 00:04:02.520 --> 00:04:06.319 worked on on how to improve this is to actually do the role of the 54 00:04:06.360 --> 00:04:12.360 solution is designed to assist. He's shouldn't find repetitive tasks that you can automate, 55 00:04:12.400 --> 00:04:15.840 So I think what he's really talking about here is do the work, 56 00:04:15.240 --> 00:04:18.199 like what is the workflow? Where can way through it? And the UX 57 00:04:18.240 --> 00:04:23.160 can kind of be dictated from that. So spending your time with the app, 58 00:04:23.240 --> 00:04:26.319 not just thinking about it abstractly. Yeah. Yeah, no, substitute 59 00:04:26.360 --> 00:04:29.920 for that fewest number of clicks, use numbers of change of hand position like 60 00:04:30.120 --> 00:04:31.839 to the mouse and to back again. All those kinds of things make a 61 00:04:31.879 --> 00:04:35.639 better UI. Yeah, it's always the same. Right. You talk to 62 00:04:35.680 --> 00:04:41.079 the customer and you say, if you could write this program, like, 63 00:04:41.199 --> 00:04:44.319 how would you want to operate it start Where would you want to start? 64 00:04:44.319 --> 00:04:46.759 Where would you want to go? What are the generals? What are the 65 00:04:46.800 --> 00:04:50.160 details? And how do you want to get through that? And often people 66 00:04:50.240 --> 00:04:55.079 don't believe you. They like they think that they're stuck in a box of 67 00:04:55.120 --> 00:04:58.000 a form or this or that. But you say, you know, if 68 00:04:58.000 --> 00:05:00.399 this was a car, how would you drive it? Yeah, Chrishma is 69 00:05:00.399 --> 00:05:03.639 shaking her head, nodding her head. Yes, she's been there, no 70 00:05:03.800 --> 00:05:08.199 question. And you know that part of that is watching somebody work, like, 71 00:05:08.240 --> 00:05:10.399 look, how do you normally do this? And all? And you 72 00:05:10.480 --> 00:05:13.759 realize those little jumps outs even after you've built the app, going back and 73 00:05:13.800 --> 00:05:16.000 watching people use it, you know, a few months later going wow, 74 00:05:16.079 --> 00:05:20.199 that's not what I always think. Yeah, right, hey, par thank 75 00:05:20.240 --> 00:05:23.160 you so much. Here a comment at a copy of music Coby. It 76 00:05:23.199 --> 00:05:25.120 is on its way to you. And if you'd like a copy of music 77 00:05:25.160 --> 00:05:28.279 Coby, I read a comment on the website at dot net rocks dot com 78 00:05:28.399 --> 00:05:30.800 or on the facebooks. We publish every show there and if you comment Darren, 79 00:05:30.800 --> 00:05:32.319 we're reading the show. We'll say your copy music cobuy. And you 80 00:05:32.399 --> 00:05:35.800 can definitely follow us on Twitter or x as it's called now, but that's 81 00:05:35.839 --> 00:05:40.759 fine, but the real fund is happening over on Mastodon. I met Carl 82 00:05:41.000 --> 00:05:45.600 Franklin at tech Hub dot social and I'm Rich Campbell at mastodon dot social. 83 00:05:45.800 --> 00:05:49.160 Send us too, and you know you could send us a comment by two 84 00:05:49.560 --> 00:05:53.439 if you like, you can and we will respond. We will respond, 85 00:05:53.519 --> 00:05:57.800 We read them all. Okay, let me introduce Chris Chrishma Jenna is a 86 00:05:57.920 --> 00:06:02.680 data scientist with the UX Insights Team at IBM in San Francisco. I should 87 00:06:02.720 --> 00:06:08.480 just stop there, because that's amazing. That's awesome, the full amazing full 88 00:06:08.560 --> 00:06:11.839 stop right. I mean, you're the only data scientist in the org. 89 00:06:13.079 --> 00:06:17.879 She supports eighty plus user researchers and designers and uses data to understand user struggles 90 00:06:18.399 --> 00:06:25.480 and opportunities to enhance product experiences. She's delivered fifty plus talks and workshops at 91 00:06:25.560 --> 00:06:30.920 multiple conferences around the globe, including piicon US. Grishma is extremely passionate about 92 00:06:31.040 --> 00:06:35.600 encouraging, mentoring, and empowering people, especially women and students, in the 93 00:06:35.680 --> 00:06:41.319 world of technology. She's been recognized as a Fellow by the Python Software Foundation 94 00:06:41.439 --> 00:06:46.519 and also serves on advisory and leadership boards for nonprofits and other organizations. In 95 00:06:46.600 --> 00:06:48.199 other words, a whole lot of awesome. Thanks for going with us, 96 00:06:48.680 --> 00:06:51.720 Thank you so much for inviting me. Yeah, what did you think about 97 00:06:51.759 --> 00:06:56.639 the comment that Richard read there? Very interesting? I think right about the 98 00:06:56.720 --> 00:07:00.480 workflows and just having the usocentric mindset, so that something we focus a lot 99 00:07:00.560 --> 00:07:04.839 at IBEM as well. You must have heard that IBM's one of the frontiers 100 00:07:05.000 --> 00:07:09.240 and on the frontier of the design. So they came up with this Enterprise 101 00:07:09.319 --> 00:07:13.839 Design thinking framework, which is all about making the process user centric, coming 102 00:07:13.920 --> 00:07:17.199 up with something called an empathy map, which tries to put yourself in the 103 00:07:17.319 --> 00:07:19.759 use of the users and be like, Okay, what is this user thinking, 104 00:07:19.800 --> 00:07:23.759 what is the feeling? What is this user doing, and what is 105 00:07:23.800 --> 00:07:27.600 the current you know, state the ass scenario and then hopefully going to the 106 00:07:27.639 --> 00:07:31.079 TV which is the future scenario where there are hopefully no use of frustrations or 107 00:07:31.079 --> 00:07:34.079 struggles and it's a very seamless end to end EXPERI yeah, because of course 108 00:07:34.120 --> 00:07:36.319 my instinct as soon need to say what is user feeling? It's like I'm 109 00:07:36.360 --> 00:07:45.759 expecting frustrated lunch. Absolutely where does the data science fall into this? Because 110 00:07:45.959 --> 00:07:47.680 that's what grabbed me, was this idea. Yeah, you know, so 111 00:07:47.839 --> 00:07:55.759 much of UX design seems intuitive based rather than is there a data approach to 112 00:07:55.879 --> 00:07:59.879 us? A science there is? Yeah, I think this is definitely something 113 00:08:00.079 --> 00:08:03.360 that's a little novel And a lot of people ask me, Okay, you're 114 00:08:03.399 --> 00:08:09.759 a data scientist, but on the UX side, isn't that what's happening, 115 00:08:09.879 --> 00:08:13.600 what's what's going on? Like, shouldn't you be on this quantitative side with 116 00:08:13.680 --> 00:08:16.759 the logical, practical thinking. Yeah, and this is this is definitely a 117 00:08:16.879 --> 00:08:20.759 very unusual role. I will say it's not a stereotypical role. But where 118 00:08:20.879 --> 00:08:24.639 data false is that data is the language that customers used to talk to about. 119 00:08:26.720 --> 00:08:31.120 So I think there's a huge, huge potential there to actually mind all 120 00:08:31.160 --> 00:08:33.200 of that data, all of that feedback that customers are giving us, whether 121 00:08:33.320 --> 00:08:37.600 that's quantitatively through the instrumentation of the products of qualitatively, well you know, 122 00:08:37.799 --> 00:08:41.879 they're on a customers about called saying yes, office sucks, I'm going to 123 00:08:41.919 --> 00:08:43.039 go to the next competitor. You know, I could have done this with 124 00:08:43.120 --> 00:08:48.399 Excel. Yeah, my brother could do this in access. You're you're reminding 125 00:08:48.480 --> 00:08:54.320 me of Mark Miller's Science of Great UI talk where you know, the first 126 00:08:54.360 --> 00:08:58.200 time we talked to him about what you mean science of UI, Like you 127 00:08:58.320 --> 00:09:01.639 said, it seems like into would be taking over, but he talked about 128 00:09:01.759 --> 00:09:05.240 you know, there are certain things about the way your eye sees things and 129 00:09:05.279 --> 00:09:11.120 the way your brain reacts, and and that can be other way. Contrast 130 00:09:11.320 --> 00:09:15.639 is perceived as importance. The things that are higher contrasts tend to be more 131 00:09:15.679 --> 00:09:20.159 important than lower contrasts, And so that can translate directly into rules that you 132 00:09:20.279 --> 00:09:24.519 can use for a better user interface, a better UX experience. And so 133 00:09:26.080 --> 00:09:30.279 that's exactly what I thought of when you said, you know your data science 134 00:09:30.360 --> 00:09:33.080 around UX. There's a lot to it. There's a lot to it. 135 00:09:33.279 --> 00:09:37.679 But I also see we're good. We're getting good at instrumenting stuff now, 136 00:09:37.799 --> 00:09:39.840 the open telemetriies and things, the lists, like we have the tremendous amount 137 00:09:39.840 --> 00:09:43.799 of data we can pull from people using the app, not even talking about 138 00:09:43.799 --> 00:09:46.399 the information of putting into it, but how they interact with the interface, 139 00:09:48.000 --> 00:09:50.279 And I wonder how well we can use that data. Like I've worked on 140 00:09:50.320 --> 00:09:54.200 a bunch of that, and we end up with doing heat maps where it's 141 00:09:54.240 --> 00:09:56.440 like, oh, their mouse is always over here, and it's it's rarely 142 00:09:56.519 --> 00:10:01.399 over there that always do you go to eye track. I haven't dealt with 143 00:10:01.480 --> 00:10:05.879 eye tracking data myself, but yes, I know that some of the teams 144 00:10:05.000 --> 00:10:09.399 use the eye tracking data. But again, like you mentioned, right Richard, 145 00:10:09.519 --> 00:10:11.639 with a lot of instrumentation data, where are the clicks happening, where's 146 00:10:11.639 --> 00:10:15.879 the mouse activity happening? How much are they strolling through the page? Are 147 00:10:15.919 --> 00:10:18.639 they skipping over some demos because they're like, yeah, yeah, I already 148 00:10:18.679 --> 00:10:20.679 know this stuff, like you know, just just get me to the good 149 00:10:20.720 --> 00:10:22.759 stuff. I know all of this basic stuff. But I would say it 150 00:10:22.840 --> 00:10:28.519 also really comes down to two things. Number one is what data are you 151 00:10:28.720 --> 00:10:31.799 exactly capturing? How are those metrics defined? Right? Because what might be 152 00:10:31.919 --> 00:10:37.759 a conversion for my product or for my perspective might not be a conversion according 153 00:10:37.799 --> 00:10:41.039 to you. So I think just having that level of standardization of Okay, 154 00:10:41.080 --> 00:10:43.879 this is how we're going to track a log in, this is where you 155 00:10:43.960 --> 00:10:48.600 know, this particular workflow starts, this is how this particular workflow ends. 156 00:10:48.960 --> 00:10:50.799 What happens if there's some drop off in the middle, or you know what 157 00:10:52.600 --> 00:10:54.679 conversion means someone who gets to the end of a work yes, exactly right, 158 00:10:54.720 --> 00:10:58.399 So whether that's a purchase order going through or somebody putting you know, 159 00:10:58.600 --> 00:11:03.480 starting a workflow again? Now does that mean do you start a workflow? 160 00:11:03.519 --> 00:11:07.039 Doesn't mean ending the workflow? Where do you exactly when a motion points? 161 00:11:07.120 --> 00:11:11.360 I mean we usually think about conversions from an e commerce perspective, Yeah, 162 00:11:11.399 --> 00:11:16.720 but I love the idea of applying that to an internal business workflow. That'd 163 00:11:16.759 --> 00:11:18.080 be a great number to have. I've never thought about having. Is how 164 00:11:18.080 --> 00:11:22.759 many times have people started down a form? Yeah, and then a band 165 00:11:22.840 --> 00:11:26.919 abandoned it? Yeah? And if you could see a pattern that data is 166 00:11:26.000 --> 00:11:31.039 that they consistently abandoned here, yes, which might be they need additional piece 167 00:11:31.120 --> 00:11:33.600 of information that interrupts them enough they have to leave the app to do that. 168 00:11:35.399 --> 00:11:39.960 Like now you've clearly got a feature waiting to be built. Absolutely, 169 00:11:39.559 --> 00:11:45.360 And I think the other thing is also the amount of data are capturing right 170 00:11:45.919 --> 00:11:48.399 that that plays a huge role because unfortunately I've seen too many of these instances, 171 00:11:48.519 --> 00:11:54.120 especially now given that AI and Jenny I chat GPTY is pretty much do 172 00:11:54.159 --> 00:11:56.120 you know all people are talking about executives coming and say, Okay, we're 173 00:11:56.159 --> 00:12:01.039 going to build this genitive AI model with deep learning in machine learning and all 174 00:12:01.080 --> 00:12:03.320 of that. But when you go to look at the data. There isn't 175 00:12:03.360 --> 00:12:07.519 a lot of data. There is not even good quality data available to do 176 00:12:07.679 --> 00:12:11.600 those modelings. So I think having some realistic expectations really help speed up the 177 00:12:11.639 --> 00:12:16.320 process. It also strikes me, especially when talking about the large language models 178 00:12:16.360 --> 00:12:20.039 and even the general mL stuff, that's a pretty big hammer for what is 179 00:12:20.279 --> 00:12:26.159 not that big a problem if you're thinking about it, well, like I 180 00:12:26.279 --> 00:12:28.879 just wonder. I mean, it's cool, it's cool hammer, but it's 181 00:12:28.879 --> 00:12:31.159 like, really do you want to That's like it's almost like, hey, 182 00:12:31.240 --> 00:12:33.919 we just rewrote this and Ruby on rails, our problems go away. But 183 00:12:35.039 --> 00:12:37.399 the more important thing is like you're going to leave me alone for six months 184 00:12:37.440 --> 00:12:39.519 while I do that, Like if you pull out the mL hammer, you're 185 00:12:39.559 --> 00:12:45.559 really just getting a three month pass. Yeah, when actually studying the data 186 00:12:45.639 --> 00:12:50.559 meaningfully could get you to a solutions faster and simpler yea, or at a 187 00:12:50.639 --> 00:12:54.360 minimum, give you better inputs for that machine learning model in the end. 188 00:12:54.399 --> 00:12:58.279 Anyway, Yes, completely, I agree. I heard this. You know. 189 00:12:58.440 --> 00:13:03.480 An ology that someone used is that machine learning or deep lunning models are 190 00:13:03.519 --> 00:13:07.279 often the ferraris when you might even need like a bike to I'm talking a 191 00:13:07.440 --> 00:13:16.240 pickup truck, a two wheeler will do without a motor. Yeah, so 192 00:13:16.440 --> 00:13:22.960 I mean that I've found these days with telemetry specifically, it's either way too 193 00:13:24.039 --> 00:13:26.919 little and you never you can't get anything useful lot of it, or it's 194 00:13:26.039 --> 00:13:31.360 so much it's just unmanageable. Yeah, how do you try and thin the 195 00:13:31.440 --> 00:13:35.440 data out? There? Is there a tooling approach? I mean, if 196 00:13:35.440 --> 00:13:37.000 it takes too little basimbly, nothing we can do. So we kind of 197 00:13:37.039 --> 00:13:41.600 got to go with it too much. So now what Yeah, I think 198 00:13:41.639 --> 00:13:46.559 the kind of approach I leaned towards is start with things you are absolutely confident 199 00:13:46.639 --> 00:13:50.440 about. We become up with two metrics that you say, okay, the 200 00:13:50.480 --> 00:13:54.879 conversion rate and you know, maybe the number number of logins per month or 201 00:13:54.960 --> 00:13:58.519 you know something like that. Just come with two metrics that you're absolutely confident 202 00:13:58.600 --> 00:14:01.519 and that you can manually check that, Okay, what's being instrumented, what's 203 00:14:01.519 --> 00:14:05.039 being captured, is what we are expecting, and then you feel confident. 204 00:14:05.120 --> 00:14:09.519 Maybe keep building onto that sect thing through a bunch of obvious and errors for 205 00:14:09.519 --> 00:14:13.039 a log in, yeah right, the speil log and failed log at an 206 00:14:13.039 --> 00:14:16.600 incomplete loggin. I don't think it's more than that. And now can we 207 00:14:16.720 --> 00:14:18.639 see them all can we now see a number behind that? Or would you 208 00:14:18.720 --> 00:14:24.720 start with the happy path? Depends depends. So the way I like to 209 00:14:24.799 --> 00:14:28.480 do it is that let's pick up two metrics in the instrumentation phase that we 210 00:14:28.559 --> 00:14:31.720 are confident to it that we can check. Once we have that in plates 211 00:14:31.799 --> 00:14:35.519 lists, start thinking about what future questions we would want to answer. So 212 00:14:35.559 --> 00:14:37.960 I might not have the data today, but three months down the line, 213 00:14:37.960 --> 00:14:41.279 I want to see how many users are abandoning on a particular workflow, right, 214 00:14:41.519 --> 00:14:43.519 are converting to a particular workflow, all right, and then start those 215 00:14:43.679 --> 00:14:46.679 that reverse engineering of okay, to be able to answer that question. The 216 00:14:46.759 --> 00:14:50.279 kind of data I would need is the abandons, maybe the Paige speed load. 217 00:14:50.440 --> 00:14:54.320 Maybe you know how many demos they are watching, how much time they 218 00:14:54.320 --> 00:14:58.080 are spending on this particular product, And then go back and see what other 219 00:14:58.159 --> 00:15:01.320 metrics that you could come up with actually start tracking all of that data. 220 00:15:01.720 --> 00:15:07.759 Now, when you're talking about data science and in this I know you mentioned 221 00:15:07.840 --> 00:15:13.120 AI a little bit passively, but do you use predictive analytics on this data 222 00:15:13.240 --> 00:15:18.720 to try to figure things out or any other kind of you know AI if 223 00:15:18.759 --> 00:15:22.840 you will, Yeah, yeah, okay, So a bit of a disclaimer, 224 00:15:22.919 --> 00:15:26.240 I'm not really sure where the lines between artificial intelligence and data science, 225 00:15:26.360 --> 00:15:31.039 you know, blur, but I always feel yeah, I mean data science 226 00:15:31.120 --> 00:15:35.080 is really at the very very lowest level of AI. It is all statistics, 227 00:15:35.120 --> 00:15:39.279 it's data sensors, machine learning. So just for the purposes of our 228 00:15:39.320 --> 00:15:41.440 conversation, I'm going to interchangeable use or just maybe stick. Do you know 229 00:15:41.559 --> 00:15:48.080 data sciensing machine learning? Yes, Predictive analytics is definitely something very popular. 230 00:15:48.679 --> 00:15:52.679 I think one of the most most abundantly available use cases is predicting if a 231 00:15:52.799 --> 00:15:58.320 customer is about to leave your product, So it's basically predicting customer attrition. 232 00:15:58.960 --> 00:16:03.080 So you can maybe start looking at you know, has their overall usage of 233 00:16:03.120 --> 00:16:07.720 the product decreased? Have they opened a lot of support tickets recently? Maybe 234 00:16:07.759 --> 00:16:11.159 those support tickets have not given resolutions to them right, or maybe the average 235 00:16:11.200 --> 00:16:15.480 time taken to handle its support ticket is increasing. You also have a lot 236 00:16:15.519 --> 00:16:18.240 of feedback surveys that they give, right like how happy are you with this 237 00:16:18.360 --> 00:16:21.879 experience? Would you recommend our product to a friend or a colleague, and 238 00:16:21.919 --> 00:16:23.960 then just looking at the trend of those goals. So that is definitely one 239 00:16:25.039 --> 00:16:29.759 case where you know the predictive models can help and alert the product team that 240 00:16:29.879 --> 00:16:33.240 here this customers at a high risk of actually, you know, terminating the 241 00:16:33.360 --> 00:16:37.080 contract. Yeah, so so better go ahead and maybe have like a touch 242 00:16:37.159 --> 00:16:41.840 point of okay before the customers actually talking about canceling exactly. The idea and 243 00:16:41.720 --> 00:16:45.799 the idea that you would contact them and say, hey, you've been having 244 00:16:45.840 --> 00:16:48.480 a time can we help you? Like, what can I do? Extra? 245 00:16:48.840 --> 00:16:53.240 Yeah? Why did you post this on Twitter's angry Sweet? Yeah? 246 00:16:53.399 --> 00:17:00.440 Just call support? Yeah, you know I have. I think I talk 247 00:17:00.480 --> 00:17:03.000 about this the show that Bloody Dishwasher, the Melee. Yeah. I kept 248 00:17:03.039 --> 00:17:07.720 breaking Yeah, and I couldn't get regular support. Regular sport wouldn't respond to 249 00:17:07.759 --> 00:17:12.240 me four weeks. But if I tweeted and tweeted yea that day. Yeah. 250 00:17:12.440 --> 00:17:15.559 And at one point I'm like, listen, I'm not that mean a 251 00:17:15.680 --> 00:17:18.839 guy, right, I just didn't get anywhere I would Is there a better 252 00:17:18.880 --> 00:17:22.039 way to do this? And the and the person on the other end's like, 253 00:17:22.119 --> 00:17:26.240 literally, nope, this works, just use it. I'm I'm in 254 00:17:26.440 --> 00:17:30.920 that limbo period right now with the HVAC system in my studio. Oh man, 255 00:17:30.400 --> 00:17:33.240 yeah, they's a new one. Well they I don't want to take 256 00:17:33.319 --> 00:17:37.559 up too much time with this story. But they sold me a twenty four 257 00:17:37.640 --> 00:17:41.960 thousand BTU unit for a six hundred and eighty square foot room that's insulated to 258 00:17:41.039 --> 00:17:47.759 our twenty R fifty R fifty okay, really freaking insulated. Yeah. Yeah, 259 00:17:47.880 --> 00:17:48.960 you don't have to pump very much cold air in there. Yeah. 260 00:17:49.039 --> 00:17:53.160 So it cools off in about five seconds with this thing, and then it 261 00:17:53.240 --> 00:17:57.680 stops, and so the humidity rises to sixty five seventy percent, Yeah, 262 00:17:57.720 --> 00:18:02.079 and it gets unpleasant. And so I had to beg with them to you 263 00:18:02.160 --> 00:18:04.039 know, get somebody out here to fix this, and they finally decided to 264 00:18:04.119 --> 00:18:07.000 get a smaller unit in there, but then they wanted to charge me more. 265 00:18:08.400 --> 00:18:11.920 That's like, wait a second, you screwed up, and you want 266 00:18:11.960 --> 00:18:15.279 to charge me more. Yeah, And so now we're going we didn't get 267 00:18:15.319 --> 00:18:18.119 anywhere with their subcontractor, and now we're going to the actual contractor, who 268 00:18:18.359 --> 00:18:22.680 company who we bought it from, right and there, and they are unresponsive 269 00:18:22.799 --> 00:18:26.480 right now. And if they come back with I'm sorry, you're gonna have 270 00:18:26.519 --> 00:18:29.240 to pay it, then then I'm going to unleash the kracking. Yeah, 271 00:18:29.440 --> 00:18:32.400 do the thing, yep, yep, you just need to send a tweet 272 00:18:32.440 --> 00:18:37.680 to them, right, if there's to start, but begin the public humiliation, 273 00:18:37.920 --> 00:18:41.519 right, let it begin. Well, we'll keep you posted on that. 274 00:18:41.720 --> 00:18:45.839 Yeah, no kidding, but I mean you still your language to me 275 00:18:45.920 --> 00:18:52.599 strikes as external customers reacting to product, right, I mean, and that's 276 00:18:52.680 --> 00:18:56.720 fine, Like those are all good things, and it is interesting to synthesize 277 00:18:56.799 --> 00:19:02.440 data from your tech support side as well as perhaps a sales channel. Maybe 278 00:19:02.519 --> 00:19:04.640 you're doing some monitoring at the social media level, to say, do we 279 00:19:04.720 --> 00:19:11.000 have some sentiment analysis going on? Coordinate this information, but then to add 280 00:19:11.039 --> 00:19:14.920 the utilization of the software. Yeah, Like this is not just a person 281 00:19:15.079 --> 00:19:18.279 being randomly grumpy, right, it's this is where they're falling off on the 282 00:19:18.319 --> 00:19:22.119 workflow and they do that consistently. Like, this is not a whining person. 283 00:19:22.200 --> 00:19:27.039 They're trying to get it done and they're abandoning for whatever reason. Like 284 00:19:27.160 --> 00:19:32.720 that that is such actionable data. I am you are watching this customer try, 285 00:19:32.880 --> 00:19:34.759 Yeah, Like shouldn't you do something about that? Yeah? Yeah? 286 00:19:34.799 --> 00:19:38.519 And I think one case where data science comes in really helpful is that maybe 287 00:19:38.680 --> 00:19:42.640 as a front front facing you know, you're facing the client and you're having 288 00:19:42.680 --> 00:19:47.160 these conversations, maybe it's a product manage. Maybe it's a develops having a 289 00:19:47.240 --> 00:19:49.319 conversation with a very frustrated customer, right saying, Okay, you know this 290 00:19:49.440 --> 00:19:52.720 isn't working the way it's expected. I'm not going to do a bunch of 291 00:19:52.799 --> 00:19:56.920 issues. And that's a data point that you have. But when you look 292 00:19:56.039 --> 00:20:00.920 at the data perspective or quant perspective it you get that scale, right, 293 00:20:00.000 --> 00:20:04.000 So now you start to notice this is not just one isolated customer, it's 294 00:20:04.039 --> 00:20:08.119 actually ten customers, hundred customers, right, thousands of customers that are doing 295 00:20:08.200 --> 00:20:11.240 that. And I think that helps to also remove that bias, right, 296 00:20:11.319 --> 00:20:15.000 which it could work in both ways, not just that customers the pain exactly. 297 00:20:15.079 --> 00:20:18.400 I guess the next question you asked when you talk about like a failed 298 00:20:18.440 --> 00:20:21.480 workflow like gus, well, how many succeed? Yeah? How often do 299 00:20:21.559 --> 00:20:25.160 we succeed? How many other abandons do we have? I really like that 300 00:20:25.359 --> 00:20:29.359 you take the personality element out of it to some degree, and it's like, 301 00:20:29.519 --> 00:20:33.839 here's a set of data and you have found with someone willing to communicate 302 00:20:33.920 --> 00:20:37.880 with you about the problem, don't make them the enemy. Here's all these 303 00:20:37.920 --> 00:20:41.319 people that aren't communicating about this problem but are having the same problems. Yeah, 304 00:20:41.480 --> 00:20:45.079 exactly, yeah, and it works two ways, right, Like the 305 00:20:45.240 --> 00:20:48.759 user researcher could maybe be in a conversation and then notice that, okay, 306 00:20:48.839 --> 00:20:51.920 the customer is struggling with the product, and then they go to the data 307 00:20:51.960 --> 00:20:53.599 science team or the analytics team and say, okay, we saw this one 308 00:20:53.680 --> 00:20:56.839 person. Are there other people who are struggling with this? And then you 309 00:20:57.119 --> 00:21:00.000 notice that you know, this effect is that scale. And another thing it 310 00:21:00.079 --> 00:21:03.279 also does is like often product managers or you know, a client facing teams 311 00:21:03.680 --> 00:21:08.839 here from really important like it's called the hippo the highest paid person's opinion, 312 00:21:10.319 --> 00:21:14.880 and it's usually the highest paid customer, like we hate this feature. When 313 00:21:14.880 --> 00:21:17.880 are you in unveiling? You know, XOY is that feature and they're just 314 00:21:17.960 --> 00:21:21.119 going to be you know, maximum attention to them because that's where the money 315 00:21:21.240 --> 00:21:26.799 is exactly, all right. But then when you combine it with the data, 316 00:21:26.960 --> 00:21:32.559 you can actually realize that is this person's problem? You know, actually 317 00:21:32.880 --> 00:21:34.920 yes, is it unique to the customer or is it a little more pervasive 318 00:21:34.960 --> 00:21:38.599 where other customers are also facing this? And then it's probably a future worth 319 00:21:38.640 --> 00:21:41.680 including on the road. Oh man, now you've now you make me feel 320 00:21:41.680 --> 00:21:45.920 like it's a superpower, like if you applying data science to all of these 321 00:21:47.039 --> 00:21:51.119 kinds of problems so that you're always talking from more than the immediate issue in 322 00:21:51.160 --> 00:21:53.720 front of you. Exactly. My automatic reaction to a person's having a problem 323 00:21:53.720 --> 00:21:57.240 with this feature is to be able to apply some data analytics, say how 324 00:21:57.279 --> 00:22:00.839 many other people? How many six? What are the other scenarios? Like, 325 00:22:02.119 --> 00:22:04.039 I mean, even think aboufter our past scenario where you've got a customer 326 00:22:04.079 --> 00:22:08.359 talk and now you've found out this other customers failing, Like lots of people 327 00:22:08.640 --> 00:22:14.440 exclude themselves from serving, but they don't want to do that. But to 328 00:22:14.720 --> 00:22:17.839 call that customer say hey, you know, I'm from such a such a 329 00:22:17.880 --> 00:22:21.920 company and we'll see you're struggling with this thing and we're trying to address it. 330 00:22:21.960 --> 00:22:23.559 Would you help us to understand what you're trying to do, Like, 331 00:22:23.680 --> 00:22:26.279 I think you'd get good response from that. Yeah, I mean, you're 332 00:22:26.319 --> 00:22:30.559 going to hum the fine line of being creepy. You know, we've been 333 00:22:30.640 --> 00:22:37.160 watching you and we see you're not having a good time. Better to say 334 00:22:37.279 --> 00:22:44.119 that you're a hired psychic. You've discovered that they're the spirits. Have told 335 00:22:44.200 --> 00:22:48.440 me about your unsatisfaction. But I know. I do think it's an interesting 336 00:22:48.599 --> 00:22:52.400 part of this is to just have that reflex of let's can we quickly gather 337 00:22:52.559 --> 00:22:56.480 more data, like do you see an opportunity to build up tooling? Like 338 00:22:56.559 --> 00:22:59.519 I'd really want to make that mechanism simple. I don't want to have to 339 00:22:59.599 --> 00:23:03.799 call you Krishma as the expert to say, can you give me analysis on 340 00:23:03.880 --> 00:23:06.680 this? Yeah, I wanted to be a button the moment it pops up. 341 00:23:07.000 --> 00:23:11.839 What is the IBM system that you talked about earlier, You're sort sort 342 00:23:11.880 --> 00:23:15.400 of hinted at it. What is that play? Yeah? I think we 343 00:23:15.519 --> 00:23:18.359 have a bunch of different tooling and framework and to be honest, one of 344 00:23:18.359 --> 00:23:22.519 the issues we're facing a standardization because just the software unit that I work with, 345 00:23:22.720 --> 00:23:26.519 we have around eighty two hundred different products and all of them have absolutely 346 00:23:26.640 --> 00:23:30.759 varying degrees of data maturity, right from like we haven't really put this product 347 00:23:30.799 --> 00:23:34.000 into production, or we only have like two clients, all the way to 348 00:23:34.119 --> 00:23:37.680 we have thousands plus, you know, users every month, and this is 349 00:23:37.839 --> 00:23:41.319 just a huge amount of data. So I think it's it's hard to generalize 350 00:23:41.359 --> 00:23:45.440 what that looks like, which is why where I need to have a personalized 351 00:23:45.440 --> 00:23:48.279 approach for the team I'm dealing with, the product, I'm dealing with, 352 00:23:48.480 --> 00:23:53.079 what is the exact objective or the problem statement that we're trying to find automation. 353 00:23:53.200 --> 00:23:57.119 That's that's very interesting because that is definitely something that I'm trying to do 354 00:23:57.200 --> 00:24:00.759 as well as unit I've seen a lot of other company needs do as well. 355 00:24:00.240 --> 00:24:03.359 Is this concept of anomally detection, right, right, So throwing all 356 00:24:03.359 --> 00:24:07.240 of the data into a model that will help you understand if a particular user 357 00:24:07.279 --> 00:24:11.720 activity or a particular user is you know, I'm showing up as an anomaly. 358 00:24:11.799 --> 00:24:15.000 They're doing something different, they're doing something unexpected, maybe they're running into 359 00:24:15.039 --> 00:24:18.920 a bunch of issues or errors. And then that also goes into with incident 360 00:24:19.000 --> 00:24:22.000 management. So a lot of security teams do that to see it, right, 361 00:24:22.039 --> 00:24:26.400 maybe there's a DDoS attack happening, that's anomally for you, so that's 362 00:24:26.440 --> 00:24:29.319 an incident, right, So using kind of the same principles, So that 363 00:24:29.480 --> 00:24:32.720 is definitely one way of automating. But again that only comes into play when 364 00:24:32.799 --> 00:24:36.960 you have a highly mature team in terms of the instrumentation, in terms of 365 00:24:37.000 --> 00:24:40.640 the data, in terms of understanding how many users you have, yeah, 366 00:24:40.799 --> 00:24:44.599 and what those scenarios look like like it does feel like something I'd want to 367 00:24:44.720 --> 00:24:49.079 build into a tech support client. So not only am I pulling up the 368 00:24:49.160 --> 00:24:55.799 anomaly, but then provides analytics run the anomally say this is the four hundred 369 00:24:55.880 --> 00:24:57.920 times this has happened this week. Yeah, right, and with this many 370 00:24:57.960 --> 00:25:03.039 different customers, just to give a sense of urgency and scope to that. 371 00:25:03.799 --> 00:25:06.440 Like I said, I love the idea of being able to resist the hippo 372 00:25:06.480 --> 00:25:10.119 because they're asking for a fitture that's distinct to them. The majority of customers 373 00:25:10.160 --> 00:25:12.960 are fine. Yeah, And at the same time, when you have a 374 00:25:14.039 --> 00:25:15.720 problem, show up to see, okay, well this is unique to that 375 00:25:15.880 --> 00:25:18.880 customer. I can approach it differently. And I'm not saying like ignore it 376 00:25:18.920 --> 00:25:22.559 but so much, but recognize it's an unusual work Yeah, as of course 377 00:25:22.640 --> 00:25:27.720 to it's a usual workflow that often fails, and maybe you can get a 378 00:25:27.759 --> 00:25:30.160 shape for it. But also you know, maybe it fails ten percent of 379 00:25:30.240 --> 00:25:34.240 the time and ninety percent time it succeeds. Like now you've got a different 380 00:25:34.279 --> 00:25:37.519 shape of how you're going to approach it. Looking at the problem, it 381 00:25:37.640 --> 00:25:41.000 feels like it's time to take a break, So we'll be back after these 382 00:25:41.119 --> 00:25:49.079 very important messages. Hey, we're back. It's Carl Franklin. This is 383 00:25:49.160 --> 00:25:52.599 Richard Campbell, and we're here with Grish Magenna and we're talking data science and 384 00:25:52.720 --> 00:26:02.880 UX and I'm very fascinated by this UX sort of a science framework that IBM 385 00:26:02.920 --> 00:26:07.440 has. Well that is there any one of those things that you would use 386 00:26:07.480 --> 00:26:12.039 at the onset of a project to help design the US or are they sort 387 00:26:12.079 --> 00:26:17.319 of things that you work with after the fact, Like this last example you 388 00:26:17.400 --> 00:26:19.480 gave, seems like you have a mature team and a mature project. Yeah. 389 00:26:19.680 --> 00:26:22.440 Yeah, I think again going back to like at the start, it's 390 00:26:22.519 --> 00:26:26.000 more of let's start thinking about what are the questions you want answer and what 391 00:26:26.119 --> 00:26:30.759 are the metrics you need to track, So it's probably at the earliest stage 392 00:26:30.759 --> 00:26:34.480 of data collection and just deciding what those metrics should look like. I think 393 00:26:34.559 --> 00:26:37.279 one thing I am trying to do as well is when I'm working with these 394 00:26:37.359 --> 00:26:41.359 mature teams and royalizing there are sort of inconsistencies with the data. I'm actually 395 00:26:41.400 --> 00:26:45.440 trying to create this guide of Hey, I worked with this team that had 396 00:26:45.559 --> 00:26:48.440 great data maturity, but we still had inconstituencies, we still had issues figuring 397 00:26:48.480 --> 00:26:52.359 out, you know, what was going on. So as a heads up 398 00:26:52.440 --> 00:26:56.880 to you before you even start instrumenting, learn from our mistakes, take take 399 00:26:56.920 --> 00:27:00.599 our failures into account, and here are the recommendations that would make you know. 400 00:27:00.799 --> 00:27:03.240 So that is definitely some sort of you know, activity that I'm trying 401 00:27:03.279 --> 00:27:07.559 to do earlier in the pipeline. But yeah, I think, I mean, 402 00:27:07.640 --> 00:27:10.119 that's that's all you can do, right because you can speculate. But 403 00:27:10.279 --> 00:27:12.000 at the end of the day, when the data starts full flowing, and 404 00:27:12.119 --> 00:27:15.400 that's when you can actually do some expiation and and that it's an understand is 405 00:27:15.440 --> 00:27:18.200 this good enough? Is this good quality data? Does this make sense? 406 00:27:18.319 --> 00:27:21.519 Or do you maybe need to you know, modify a few things. And 407 00:27:21.519 --> 00:27:25.960 do you have a sort of general knowledge database of things that you've gleaned from 408 00:27:26.119 --> 00:27:30.000 projects working? You know, when somebody says, you know, customer says 409 00:27:30.039 --> 00:27:32.960 that we're thinking of doing it like this, and you say, well, 410 00:27:33.079 --> 00:27:36.640 you know, based on our experience, that approach hasn't really worked well and 411 00:27:36.759 --> 00:27:40.440 years why, like, do you have this sort of general knowledge that you're 412 00:27:40.680 --> 00:27:42.759 gathering that you used too? Yeah, I do have it unformally, but 413 00:27:44.359 --> 00:27:48.160 formally I think this reminds me of what is my primary responsibility is to create 414 00:27:48.240 --> 00:27:53.079 this insights repository of sorts, which is where all the user researchers are coming 415 00:27:53.160 --> 00:27:56.200 in and they have findings, and more often than not, all of these 416 00:27:56.279 --> 00:28:00.920 findings are stowed away and like you know, drives and folded. And let's 417 00:28:00.920 --> 00:28:03.920 say somebody leaves the company two years later, product manages like, oh that 418 00:28:03.119 --> 00:28:07.599 thing was done two months, two years ago, where is the token? 419 00:28:07.680 --> 00:28:10.759 I find? You need to talk to ten people? So we're definitely about 420 00:28:10.920 --> 00:28:14.279 knowledge management. Yeah, it would be really cool to upload that to a 421 00:28:14.559 --> 00:28:18.839 chat chypt like things that you could ask it questions directly. So what do 422 00:28:18.880 --> 00:28:22.519 you think about this? Well based on past experience, Yeah, exactly. 423 00:28:22.000 --> 00:28:26.839 But then also am I other risk of automating my job away? I don't 424 00:28:26.880 --> 00:28:29.960 know. I mean I've always had the experience that there's we're never getting to 425 00:28:29.960 --> 00:28:33.880 the bottom or to do list anyway. So knocking automating stuff, you're just 426 00:28:33.000 --> 00:28:37.920 going to do more faster, We're just going to do more things. But 427 00:28:37.119 --> 00:28:41.759 you have worked with some interesting customers I saw. I haven't seen the top, 428 00:28:41.839 --> 00:28:45.480 but I've seen the abstract for the top where you talked about folks like 429 00:28:45.559 --> 00:28:48.079 Airbnb, and Spotify and so forth, like can you dig into any of 430 00:28:48.119 --> 00:28:51.319 those scenarios. I don't want to get any any trouble but here, but 431 00:28:51.440 --> 00:28:55.119 it's like that just sounds cool. Yeah, nom, So those are those 432 00:28:55.160 --> 00:28:59.000 are case studies. I wish I could claim that those are the projects that 433 00:28:59.079 --> 00:29:02.680 I haven't done. Those are case studies, though, but very interesting ones. 434 00:29:02.759 --> 00:29:06.519 So I think Airbnb and Spotify does a really good job of having their 435 00:29:06.720 --> 00:29:11.079 ux research teams, design teams and their data science teams work in very tight 436 00:29:11.680 --> 00:29:15.039 conjunction with each other. Yeah. So, like I mentioned, right, 437 00:29:15.079 --> 00:29:18.240 you could have a user researchers that says, Okay, this customer, this 438 00:29:18.359 --> 00:29:22.720 users having something odd, let's go to data scientists. But conversely, you 439 00:29:22.799 --> 00:29:26.160 could have data scientists are looking at that aggregated level of data and saying, 440 00:29:26.480 --> 00:29:30.680 hmm, this is kind of unexpected. We didn't really expect such low conversions. 441 00:29:30.039 --> 00:29:33.359 To check the metrics. The web pages are working fine, the instrumentation 442 00:29:33.480 --> 00:29:37.880 is okay, what's happening? Can you use a researcher go and talk to 443 00:29:37.960 --> 00:29:41.559 the customer and try to find out what's exactly going on. That's where the 444 00:29:41.640 --> 00:29:45.240 quantitative and the qualitative aspects of it. This is this is the title from 445 00:29:45.240 --> 00:29:48.279 your child, right, the US data analysts they should be friends, Yes, 446 00:29:48.400 --> 00:29:52.319 they should be best friends friends. Okay, well, because you get 447 00:29:52.400 --> 00:30:00.240 this problem with ux of it only being intuitive and a dodo. Yeah to 448 00:30:00.480 --> 00:30:03.680 the data science folks are going to come in, are going to help you 449 00:30:03.799 --> 00:30:07.640 beef up your case for work absolutely and vice versa. There the data folks 450 00:30:07.680 --> 00:30:11.160 are going to see these anomalies in the arc, like the larger data sets, 451 00:30:11.200 --> 00:30:15.319 and be able to come back with have you looked at and maybe turn 452 00:30:15.440 --> 00:30:21.000 up some other issues exactly exactly, And I think especially as more companies and 453 00:30:21.079 --> 00:30:23.960 executives go into this data driven decision model, right, it really helps to 454 00:30:25.039 --> 00:30:27.400 have that evidence. So you can say, my instincts or my intuition based 455 00:30:27.440 --> 00:30:30.000 on all the research I've done, say that we should, you know, 456 00:30:30.160 --> 00:30:34.200 modify the workflow, we should introduce this feature. But here's the data to 457 00:30:34.279 --> 00:30:37.640 back it up. We spoke to our users or we have looked at instrumentation 458 00:30:37.759 --> 00:30:41.680 data. So here's the evidence that I can you know, help prove across 459 00:30:41.759 --> 00:30:45.960 a point with Because anytime you change a working data flow, you're impairing its 460 00:30:47.039 --> 00:30:49.880 productivity for some period of time. Like people know how to use this, 461 00:30:51.799 --> 00:30:56.680 so the improvement has to be big enough that thing initial impact of the change 462 00:30:56.880 --> 00:31:00.759 is offset by the net long term net benefit. I guess an interesting ROI 463 00:31:00.880 --> 00:31:02.559 question. It's like, hey, we're not going to change this because it 464 00:31:02.640 --> 00:31:07.799 affects so many users and the inkroll improvement doesn't appear to be substantial enough. 465 00:31:07.559 --> 00:31:11.240 That'd be an interesting place, like I don't know if any organizations I've ever 466 00:31:11.319 --> 00:31:14.240 dealt with where they were that mature. Thank you. See, this is 467 00:31:14.279 --> 00:31:17.359 a design improvement, but it's not a big enough design improvement. So I 468 00:31:17.519 --> 00:31:19.720 think we're going to do it and until we can find something large enough that 469 00:31:19.759 --> 00:31:23.559 it's worth disrupting the workflow flow yep, yep. And even then, I 470 00:31:23.599 --> 00:31:27.039 think communication is really important. You need to tell the users here, this 471 00:31:27.160 --> 00:31:32.599 is our latest updated version, and this is where now this menu has actually 472 00:31:32.640 --> 00:31:34.359 gone over to the right side, and these are the new features that have 473 00:31:34.480 --> 00:31:37.640 been on wheals. So I think that communication and almost change management, is 474 00:31:37.640 --> 00:31:41.240 it really important so that the expectations of the users are you know, kept 475 00:31:41.319 --> 00:31:45.319 kept in check. Yeah sure, and yeah, I certainly. I talked 476 00:31:45.319 --> 00:31:51.319 to folks who have client fatigue that stuff, especially in the SaaS products and 477 00:31:51.680 --> 00:31:55.319 cloud stuff. It changes too often, right, And I spend enough time 478 00:31:55.319 --> 00:31:57.000 on the sytems. It's like I get tickets all the time, and I 479 00:31:57.039 --> 00:32:01.720 can see the wave of oh they updated, decline again this week, and 480 00:32:01.759 --> 00:32:05.720 I'm getting all these tickets like somebody move my cheese? Right, and you're 481 00:32:05.759 --> 00:32:07.880 back to what the heck happened? How is it different? Here's how we 482 00:32:07.960 --> 00:32:10.839 deal with it. Talk to you next week. Yeah, and that This 483 00:32:12.000 --> 00:32:14.880 is another case where data science comes in really handy, because then you can 484 00:32:14.920 --> 00:32:17.440 start doing AB testing right and actually look at the data. Rights this change 485 00:32:17.519 --> 00:32:21.960 actually useful? Does this make sense to users? Or are the users are 486 00:32:22.000 --> 00:32:23.680 frustrated and they just keep going back to like, you know, take me 487 00:32:23.759 --> 00:32:27.000 to the old layout, take me to the old format? Can I turn 488 00:32:27.039 --> 00:32:30.160 it off? The first question? Every time? And I love you know, 489 00:32:30.759 --> 00:32:34.839 AB testing isn't always obvious, right, Like how do you know which 490 00:32:34.920 --> 00:32:37.680 to pick A or B? Like? Why was it better? Yeah? 491 00:32:37.920 --> 00:32:39.759 You know, Okay, you serve them fifty fifty percent of the time. 492 00:32:39.839 --> 00:32:45.359 Like what's the metric that shows it was a superior outcome? Higher completion rate? 493 00:32:45.400 --> 00:32:46.920 I guess would be the one? Yes, yes, definitely. I 494 00:32:47.000 --> 00:32:51.640 think you also need to look at the demographics of users, right, did 495 00:32:51.720 --> 00:32:54.000 you do the branching based on where the users are based? And if so, 496 00:32:54.200 --> 00:33:00.240 were there any cultural differences demographical differences that contributed to that? So I 497 00:33:00.359 --> 00:33:04.319 think just it's definitely a very complex topic. How do you have those confounding 498 00:33:04.400 --> 00:33:07.480 variables? How do you make sure they're bearable to attracking, are independent of 499 00:33:07.559 --> 00:33:10.200 each other, or at least as independent as they could an experimental sets. 500 00:33:10.359 --> 00:33:15.200 Now you also have to look at things like is there a geographic event happening 501 00:33:15.279 --> 00:33:17.680 right now? An earthquake, a flood, a heat wave, not that 502 00:33:17.799 --> 00:33:22.720 that would ever happen, Nah, a forest fire? Looking at you, 503 00:33:22.000 --> 00:33:27.400 Richard. Yeah, this comes down to really, how do you how do 504 00:33:27.519 --> 00:33:30.640 you separate A and B? Like? Why? Because my instinct as a 505 00:33:30.720 --> 00:33:37.200 developer is just gonna make it random. But nothing's actually a random So what 506 00:33:37.279 --> 00:33:40.559 do you mean when you say random? So, I mean, at a 507 00:33:40.559 --> 00:33:45.359 minimum, if you're thinking webish here, I've got a cookie, I'm going 508 00:33:45.400 --> 00:33:49.119 to stick a given cookie to A, and the next new cookie I create, 509 00:33:49.200 --> 00:33:52.119 I'm going to stick them to B. Yes? Is that sufficiently? 510 00:33:52.200 --> 00:33:55.119 It's not random at all? Actually, it's a cyclical measure. You haven't 511 00:33:55.559 --> 00:34:00.880 I mean, you're ignoring all the other factors that good enough. I don't 512 00:34:00.880 --> 00:34:05.279 know me the data can tell yeah, yeah, how much do we know 513 00:34:05.359 --> 00:34:08.239 about that cookie to actually say well this is who got it? What was 514 00:34:08.320 --> 00:34:12.320 a separation on them? Yeah, I don't know that I've ever looked at 515 00:34:12.360 --> 00:34:15.519 the data that way, Like Ashley dug In and said, did we make 516 00:34:15.599 --> 00:34:22.239 a good A B test from a client selection perspective before we say hey A 517 00:34:22.440 --> 00:34:27.960 out A outcompleted B therefore A is better? Not even making me think too 518 00:34:28.000 --> 00:34:31.280 much, christ Man, It's all about the biases, right, selection bias, 519 00:34:31.400 --> 00:34:36.400 confirmation bias, recency bias. Everything I've thought about it is a lie. 520 00:34:36.840 --> 00:34:39.079 Yeah, well even in fact that I'm just selecting new cookies so my 521 00:34:39.159 --> 00:34:44.039 existing users don't see it at all, Right, like that that could be 522 00:34:44.119 --> 00:34:46.880 in effect as well. So many opportunities to lie, yeah, or to 523 00:34:47.000 --> 00:34:53.320 just or to select a truth yes in advance ways to filter? Do you 524 00:34:53.400 --> 00:34:57.639 ever have to tackle those kinds of problems like how are we going to separate 525 00:34:58.159 --> 00:35:00.039 to request these request streams to make sure or we get a good AB mix? 526 00:35:00.199 --> 00:35:04.760 Like what does that even look like? I don't think I've actually done 527 00:35:04.760 --> 00:35:07.079 a lot of AB testing on this current role, not not so much. 528 00:35:07.280 --> 00:35:10.719 But yeah, again, I think I'm challenging more with that volume. And 529 00:35:12.159 --> 00:35:14.880 you know, data quality is shoes right now. So hopefully in the future 530 00:35:14.920 --> 00:35:17.960 I'm hoping to get to those good, nicer problems to have. But right 531 00:35:19.039 --> 00:35:22.239 now it's a lot about education and awareness about how the instrumentation needs to happen 532 00:35:22.400 --> 00:35:27.960 and what other things that's possible, because oftentimes it's also just the black box 533 00:35:28.639 --> 00:35:30.280 perspective of AI, right, like, oh, it can do anything and 534 00:35:30.400 --> 00:35:36.079 everything, but we know it's not true to start from. I'm also thinking 535 00:35:36.119 --> 00:35:38.400 from the friends perspective of often it's just a developer decided to do that. 536 00:35:38.480 --> 00:35:40.719 We've decided to do an ab testing group of developers there, it's like you 537 00:35:40.760 --> 00:35:45.280 should be calling your data science people at that moment to talk about how we're 538 00:35:45.280 --> 00:35:49.719 going to discriminate these Yeah, let those experts get into the play and even 539 00:35:49.840 --> 00:35:52.440 know that they're doing that, because that can give us a make the get 540 00:35:52.599 --> 00:35:55.679 a better result if we work at it a bit. Exactly. Yeah, 541 00:35:55.719 --> 00:36:00.719 And I think even having these multicultural, diverse teams all hell because there's a 542 00:36:00.760 --> 00:36:05.400 lot of communication messaging involved. For example, I recently came across this website 543 00:36:05.440 --> 00:36:09.559 that says that each color has very different connotations culturally, like read in some 544 00:36:09.719 --> 00:36:14.760 cases might mean danger or something negative, but in other cases it's actually a 545 00:36:14.800 --> 00:36:17.440 sign of prosperity, right, well, and then the Chinese us right, 546 00:36:17.840 --> 00:36:24.159 yes, exactly exa and the Chinese dragon is a very powerful positive thing, 547 00:36:24.280 --> 00:36:29.880 and European culture it's a negative, right yeah. So yeah, where you're 548 00:36:29.920 --> 00:36:31.920 using that stuff matters as well. And certainly words, right, I mean 549 00:36:32.000 --> 00:36:37.119 it's some words that translate to other languages to mean embarrassing things. Yes, 550 00:36:37.519 --> 00:36:40.920 we see that over and over again. It's the famous Nova story, right, 551 00:36:42.039 --> 00:36:46.000 yeah, Nova, it doesn't go ye no, Chevy Nova, don't 552 00:36:46.159 --> 00:36:52.920 call a card that in a Spanish speaking control mistake, or the slogan Coke 553 00:36:52.000 --> 00:36:57.400 ads life when translated to Japanese. I think it was our Chinese one of 554 00:36:57.440 --> 00:37:04.960 the languages Coke will bring your ancestors back from the dead. Yeah. I 555 00:37:05.039 --> 00:37:09.400 think the most recent one was the video came out with their latest emmal model 556 00:37:09.519 --> 00:37:16.400 or one of those framebooks, and basically translated to Spanish, it meant bottoms. 557 00:37:21.239 --> 00:37:24.280 That's why you see like all these nonsense words as names of products totally 558 00:37:24.480 --> 00:37:28.639 like that, because they're just they don't exist otherwise boid. It helps, 559 00:37:28.679 --> 00:37:34.280 like, for example, work just Fine, which is the sound beforehead slap, 560 00:37:34.400 --> 00:37:38.039 also is the name of my production company. So you know, we've 561 00:37:38.079 --> 00:37:44.920 been digging in purely on the telemetry side, all of this very quantitative data, 562 00:37:45.119 --> 00:37:49.599 but there's a whole qualitative side, right Doing hot surveying well, I've 563 00:37:49.639 --> 00:37:52.159 come to appreciate incredibly hard to do. But as soon as you you know, 564 00:37:52.280 --> 00:37:55.960 you want people to type something or write something, now you got to 565 00:37:55.960 --> 00:38:00.639 analyze it all, like what do you do? Oh? Use researchers are 566 00:38:00.679 --> 00:38:04.440 actually pretty good at this, but unfortunately it's in manual efforts, so scaling 567 00:38:04.480 --> 00:38:08.880 it up can be very time consuming for sure. That's another place where I've 568 00:38:09.239 --> 00:38:14.920 kind of loved this use case is to help analyze surveys or reviews or you 569 00:38:15.000 --> 00:38:19.320 know, Twitter complaints or whatever social media mentions that you have. So what 570 00:38:19.400 --> 00:38:22.199 I like to do is, you know, run all of these through Python 571 00:38:22.280 --> 00:38:24.679 scripts that I've developed and then try to understand what are those top teams, 572 00:38:24.840 --> 00:38:29.880 keywords and sentiments coming up. And while I will be the first to admit 573 00:38:29.960 --> 00:38:31.880 it's never going to be a hundred percent perfect, it's usually a really good 574 00:38:31.920 --> 00:38:36.760 starting point that then I can handle over to the user researchers and they'll get 575 00:38:36.800 --> 00:38:38.360 a lay of the land almost like Okay, you know, there are a 576 00:38:38.400 --> 00:38:42.039 lot of mentioning about us, or there's a lot of negative sentiment attached to 577 00:38:42.119 --> 00:38:45.440 this new version that we had. Let me go in and manually dig in 578 00:38:45.519 --> 00:38:49.199 a bit and see, you know, what are those more insightful things that 579 00:38:49.239 --> 00:38:52.800 I can come up with. So that's another you know, symbaetic relationship we 580 00:38:52.920 --> 00:38:57.239 have. It seems like a multiple choice survey kind of thing is the way 581 00:38:57.639 --> 00:39:00.880 a lot of people do that kind of stuff. But I mean there's there's 582 00:39:00.960 --> 00:39:04.800 problems with that too, right, Like what if you don't provide the right 583 00:39:04.840 --> 00:39:07.800 option, you know, and then you have other and now you give them 584 00:39:07.840 --> 00:39:12.159 an Irish text. Now you're back to text again. So what are some 585 00:39:12.239 --> 00:39:15.920 of the other problems with that approach, the multiple choice survey kind of thing. 586 00:39:15.800 --> 00:39:21.239 I think there's always the order that the options are presented in, right, 587 00:39:21.360 --> 00:39:22.920 especially if you're not really paying attention to the survey and you're just like 588 00:39:23.039 --> 00:39:25.679 it's like the first option and just you know, I want to quickly complete 589 00:39:25.679 --> 00:39:30.519 the survey, So that that is definitely a bias that can creep in as 590 00:39:30.559 --> 00:39:32.519 well, and if people just don't want to do it, they will just 591 00:39:32.639 --> 00:39:37.320 pick random things exactly exactly. Yeah, I think language really the way you 592 00:39:37.559 --> 00:39:42.960 write that option it is left to it's subject to interpretation. Right. What 593 00:39:43.119 --> 00:39:45.559 I might interpret as option it might not be what others interpret as OPTIONE. 594 00:39:46.000 --> 00:39:50.719 So having that, I always like having those open ended text responses because then 595 00:39:50.719 --> 00:39:52.440 the users can add in a lot more information. They can also give you 596 00:39:52.440 --> 00:39:55.480 a context that hey, you know, it makes sense if I'm doing X, 597 00:39:55.639 --> 00:39:58.920 y Z. But actually I would like to choose option B on a 598 00:39:59.000 --> 00:40:02.360 regular basis because I do another workflow way more often than I do option A. 599 00:40:04.920 --> 00:40:10.119 Brian McKay works for a company Roster, that does AI analysis of text 600 00:40:10.760 --> 00:40:15.440 of comments that people leave and determines whether they're relevant or not and actually turns 601 00:40:15.480 --> 00:40:20.119 them into data. And they're using GPT. Yes, yes, that is 602 00:40:20.239 --> 00:40:23.599 That is definitely a very interesting use cases. How do you identify if one 603 00:40:23.760 --> 00:40:29.400 this topic or this comment is relevant and two is this actionable? Right? 604 00:40:29.559 --> 00:40:31.320 Even if it was relevant, you could just have the comments saying this product 605 00:40:31.440 --> 00:40:34.719 is awesome, I love it, or this product sucks I hate it, 606 00:40:35.159 --> 00:40:37.920 but there's nothing really actionable from there. And oftentimes some of the best. 607 00:40:38.599 --> 00:40:43.360 Some of the best ideas for the products come from the users themselves because they 608 00:40:43.400 --> 00:40:45.840 are at the center of this. They use this day in and day out, 609 00:40:45.199 --> 00:40:47.000 and they are probably the ones that are going to say, hey, 610 00:40:47.199 --> 00:40:51.800 have you thought about this feature? Or your competitor already has this feature? 611 00:40:51.920 --> 00:40:53.679 You know, I would love to see this coming in here. Sort these 612 00:40:53.800 --> 00:41:00.119 comments by actionability. Yes, that's a cool concept, and there you like, 613 00:41:00.679 --> 00:41:02.440 that's I really wanted to get into this. Where would I use machine 614 00:41:02.519 --> 00:41:07.000 learning? What is the right thing to do? That's something? Sort these 615 00:41:07.079 --> 00:41:10.639 by action ability? How do you get the tool to assess that? But 616 00:41:10.719 --> 00:41:15.440 I guess this is what LM's in theory or could app Yeah, yeah, 617 00:41:15.599 --> 00:41:17.760 yeah, given given the right domain knowledge, because I think a lot of 618 00:41:17.880 --> 00:41:23.000 this, you know, enterprise products especially have so much technical juggy. Yeah. 619 00:41:23.519 --> 00:41:28.199 Domin knowledge is really really important, right, because maybe what means workflow 620 00:41:28.239 --> 00:41:30.639 to us, like in general language, might be a very very specific activity 621 00:41:30.719 --> 00:41:37.719 that's happened. Defining addiction what do we call it? A glossary of vocabulary 622 00:41:37.880 --> 00:41:39.599 right at the front of a project makes so much sense. I got bit 623 00:41:39.679 --> 00:41:44.079 by this once. I had a customer that we developed an app for. 624 00:41:44.199 --> 00:41:46.760 I'm not going to say who it was. But at the end of it, 625 00:41:46.960 --> 00:41:52.559 they basically said, well, where's the the feedback page. I think 626 00:41:52.599 --> 00:41:57.880 it was feedback or maybe it was a comment screen, right, and I'm 627 00:41:57.920 --> 00:42:00.360 thinking we did a fill out form, right, But what they meant by 628 00:42:00.400 --> 00:42:07.400 a comment screen was a live chat. And they were like, oh no, 629 00:42:07.840 --> 00:42:09.719 this we said this in the steck, we want to comment screen. 630 00:42:09.800 --> 00:42:14.920 I'm like, yeah, but that definition was never defined anywa. To them, 631 00:42:15.039 --> 00:42:17.039 it meant a live chat, which hack of a lot more software, 632 00:42:17.280 --> 00:42:21.760 a lot more than writing a comment and having a store exactly. Yeah, 633 00:42:22.000 --> 00:42:27.199 yeah, surprise, yeah, yeah. So when you say comment screen, 634 00:42:27.280 --> 00:42:30.760 what do you mean exactly? And it's it's one of those things that we 635 00:42:30.840 --> 00:42:34.920 all take for granted because you know, we thought comment everybody knows what that 636 00:42:35.079 --> 00:42:37.639 means. You know, you leave a comment or an it's a contact page, 637 00:42:37.679 --> 00:42:43.000 a contact page, contact page. So when I say contact page to 638 00:42:43.119 --> 00:42:45.599 you, it's fill in this form. We'll get back to your later, 639 00:42:45.719 --> 00:42:49.480 that's right, not a live chat. Yeah. So, hey, it 640 00:42:49.599 --> 00:42:52.360 pays to go over every fine detail when we're doing that. Well, I'm 641 00:42:52.360 --> 00:42:55.199 picking back to our comment at the top of the show with par about you 642 00:42:55.280 --> 00:42:59.559 know, watching people in the workflow, are actually going through the workflow. 643 00:42:59.719 --> 00:43:02.280 Yeah, Like if you've done that and you've gotten to the contact, yeah, 644 00:43:02.840 --> 00:43:05.920 it would have been and now we chat back and forth. You mean 645 00:43:06.320 --> 00:43:12.760 chat screen. Yeah, Oh well it's a different thing entirely. I certainly 646 00:43:12.840 --> 00:43:16.639 on my radar to do more shows around building a large language model sort of 647 00:43:16.679 --> 00:43:22.559 domain specific, because that seems to be we don't need existential conversation with software. 648 00:43:22.519 --> 00:43:28.079 The generalized ones are amusing. We can debate the value of search on 649 00:43:28.239 --> 00:43:32.559 that, but the idea that it understands a particular domain and can help organize 650 00:43:32.639 --> 00:43:37.599 large amounts of data. Effective summarizer, a prioritizer like those seem like very 651 00:43:37.840 --> 00:43:43.320 practical implementations of this. Being able to sort comments into ones that are actionable. 652 00:43:43.800 --> 00:43:47.239 That's a huge amount of time that you could really nail down. It's 653 00:43:47.480 --> 00:43:52.360 just an interesting metric of is it relevant, you know, does it talk 654 00:43:52.400 --> 00:43:55.159 about a product in a way, and then you know those are all elements 655 00:43:55.199 --> 00:44:00.239 that come down to this concept of actionable ability. Yeah, but it does 656 00:44:00.320 --> 00:44:02.199 mean we're going to be training our own I'm hoping the tooling gets better. 657 00:44:02.320 --> 00:44:06.360 Like it. This still seems hard to approach it, does it does? 658 00:44:06.519 --> 00:44:08.719 Yeah? And again it really goes back to the amount of data. How 659 00:44:08.800 --> 00:44:13.360 do you define actionability? Do you have a ground source of shoot that says, 660 00:44:13.679 --> 00:44:15.960 these are the comments that are actionable because you know, these keywords were 661 00:44:16.039 --> 00:44:20.400 used, or maybe they were really long comments, which again there's a correlation 662 00:44:20.559 --> 00:44:23.920 right to having longer comments has a lot of more information in there, versus 663 00:44:24.239 --> 00:44:29.679 these are the ones that seem actionable or maybe aren't even relevant. I think 664 00:44:29.679 --> 00:44:34.079 this one time I was doing some reviews analysis and there were those automated you 665 00:44:34.159 --> 00:44:37.039 know spambots, you know, putting in really random advertisms. I'm like, 666 00:44:37.280 --> 00:44:40.000 no, that is not relevant at all, and I don't even know why 667 00:44:40.039 --> 00:44:44.480 it's in the data set. So I think that's definitely going to be an 668 00:44:44.840 --> 00:44:47.840 interesting thing to deal with. Yeah, yeah, interesting times and yeah, 669 00:44:49.159 --> 00:44:51.760 definitely new tools emerging. Still, we're going to get better at this. 670 00:44:52.239 --> 00:44:58.440 All the more reason to buy pizza for the data team. Of course, 671 00:44:58.719 --> 00:45:02.079 I'm a big believer in you know, folk city together, and so anytime 672 00:45:02.159 --> 00:45:05.400 you can do this cross team, it's like, how do we help each 673 00:45:05.440 --> 00:45:08.679 other? Like you really got me thinking about the US team wanting to have 674 00:45:08.760 --> 00:45:13.159 access to more data before they act on those problems. I'd do the same 675 00:45:13.199 --> 00:45:17.000 thing, but with barbecue that total yea and vice versa. That you'll hope 676 00:45:17.000 --> 00:45:22.119 the data analytics team that's looking at telemetry and so forth are I think routinely 677 00:45:22.360 --> 00:45:28.320 just saying hey, here's what we're seeing today, because often they may not 678 00:45:28.480 --> 00:45:30.639 even it may not occur to them that what they're seeing is unusual. Yeah, 679 00:45:30.719 --> 00:45:34.639 so they just show it to the US to periodically say the US folks, 680 00:45:34.679 --> 00:45:37.400 here's how we see how your apps being used. Hope that fits with 681 00:45:37.519 --> 00:45:42.320 what your expectations were, because it might just kick off a conversation that finds 682 00:45:42.360 --> 00:45:45.639 something for you lose customers over it. Absolutely, and that toll triangulation of 683 00:45:45.760 --> 00:45:52.519 that qualitative plus quantitative or UX resocier or plus data scientists coming together and giving 684 00:45:52.559 --> 00:45:55.519 you those different plus spectives is so so powerful, right because it almost feels 685 00:45:55.599 --> 00:45:59.800 like as a data scientist or as UX resocial you're looking at a very small 686 00:46:00.039 --> 00:46:02.000 part of the picture. But then when you start talking to different teams and 687 00:46:02.119 --> 00:46:06.360 you know more, get that vital perspective, that's where you can see the 688 00:46:06.519 --> 00:46:09.159 entire picture of oh, okay, this is what's actually happening with our users 689 00:46:09.199 --> 00:46:13.519 in the product. Awesome. Yeah, so what's next for you? Have 690 00:46:13.639 --> 00:46:15.920 you done your talk yet? No? I am busy all day on Friday. 691 00:46:16.840 --> 00:46:20.599 That's what we wanted to do this on Wednesday. So because you've got 692 00:46:20.679 --> 00:46:22.880 a busy day on Friday, I looked at your schedules like, I don't 693 00:46:22.880 --> 00:46:25.599 know, did you lose a bet, Like you're working the whole day. 694 00:46:27.519 --> 00:46:30.840 Yes, I have no idea why that's been scheduled that way. But you 695 00:46:30.920 --> 00:46:34.400 know, you enjoy the conference for a while. I guess, I guess, 696 00:46:34.519 --> 00:46:38.119 yeah. I would rather get out of the way any other conference and 697 00:46:38.239 --> 00:46:43.239 then have more time. But well, it's Christmas. Has been great talking 698 00:46:43.280 --> 00:46:45.360 to you. Thank you some excellent ideas. I really appreciate it. Thank 699 00:46:45.360 --> 00:46:49.360 you so much. All right, and we'll talk to you next time. 700 00:46:49.840 --> 00:46:52.519 I'm dot net What is it? Oh? Yeah? Dot net rock Nice. 701 00:47:15.039 --> 00:47:17.960 Dot Net Rocks is brought to you by Franklin's Net and produced by Pop 702 00:47:19.119 --> 00:47:23.760 Studios, a full service audio, video and post production facility located physically in 703 00:47:23.840 --> 00:47:30.079 New London, Connecticut, and of course in the cloud online at pwop dot 704 00:47:30.199 --> 00:47:34.920 com. Visit our website at dt n et r o c ks dot com 705 00:47:35.440 --> 00:47:39.199 for RSS feeds, downloads, mobile apps, comments, and access to the 706 00:47:39.280 --> 00:47:44.880 full archives, going back to show number one, recorded in September two thousand 707 00:47:44.880 --> 00:47:47.280 and two, and make sure you check out our sponsors. They keep us 708 00:47:47.320 --> 00:48:00.079 in business. Now go write some code, see you next time. It 709 00:48:00.320 --> 00:48:01.239 is harder than my Texas