1 00:00:01,000 --> 00:00:04,759 How'd you like to listen to dot net rocks with no ads? Easy? 2 00:00:05,320 --> 00:00:09,400 Become a patron for just five dollars a month. You get access to a 3 00:00:09,480 --> 00:00:14,240 private RSS feed where all the shows have no ads. Twenty dollars a month. 4 00:00:14,240 --> 00:00:18,399 We'll get you that and a special dot net Rocks patron mug. Sign 5 00:00:18,440 --> 00:00:23,679 up now at Patreon dot dot net rocks dot com. Hey Carlin Richard here. 6 00:00:24,039 --> 00:00:28,480 As you may have heard, NDC is back offering their incredible in person 7 00:00:28,559 --> 00:00:33,960 conferences around the world. DC Porto is happening October sixteenth through the twentieth. 8 00:00:34,240 --> 00:00:38,280 Go to dc Porto dot com to register and check out the full lineup of 9 00:00:38,320 --> 00:00:55,280 conferences at NDC conferences dot com. Hey, guess what it's time for dot 10 00:00:55,280 --> 00:00:59,719 net rocks Danish style. Yeah, in an airstream, I mean of the 11 00:00:59,759 --> 00:01:03,920 most American of American trailers. You can imagine an are in an airstream. 12 00:01:04,159 --> 00:01:07,879 So this I guess it was n dc Copenhagen at one point, but they 13 00:01:07,879 --> 00:01:15,200 rebranded it dev Festival. Yeah. Yeah, and it's very festivally tense. 14 00:01:15,359 --> 00:01:19,920 It is. They have the Boston Dynamics robot here, one of them. 15 00:01:19,000 --> 00:01:22,959 Yeah, it looks like a dog and it walks around, looks around. 16 00:01:23,000 --> 00:01:26,439 It's got the arm on it. Yeah. It doesn't do anything threatening though, 17 00:01:26,599 --> 00:01:29,760 like you know the one. It doesn't even backflip. Like I said, 18 00:01:29,760 --> 00:01:30,359 you can to do a backflip. He's like, no, no, 19 00:01:30,640 --> 00:01:34,079 that's Stlas. But it's very Uh, it's very cool. We talked about 20 00:01:34,159 --> 00:01:38,799 not making it non threatening too, because apparently they've now made Atlas smaller, 21 00:01:38,920 --> 00:01:42,599 like instead of being human heights, it's quite short, and I think it's 22 00:01:42,599 --> 00:01:45,760 all part about making people comfortable with these machines. Yeah, because they are 23 00:01:45,920 --> 00:01:49,319 creepy and they're hard and heavy. Yeah. I picked up the battery for 24 00:01:49,319 --> 00:01:53,760 that little spot robot. It it's it's a good ten pounds. Yeah, 25 00:01:53,840 --> 00:01:56,560 yeah, it's crazy. Anyway, great show. Really excited to be here. 26 00:01:56,640 --> 00:01:59,799 Yeah, very very excited to be here. Grishmo Jenna is here and 27 00:01:59,799 --> 00:02:01,799 we're gonna be talking with her in a little bit, but first it's better 28 00:02:01,840 --> 00:02:13,159 no framework roll it all right? Man? What do you got? Was 29 00:02:13,199 --> 00:02:17,960 something that Joel helan Uh told me about. It's a it's a it's a 30 00:02:19,039 --> 00:02:23,639 language called wing and you can get it at wing lang dot io, a 31 00:02:23,800 --> 00:02:27,080 programming language for the cloud. Kind of reminds me of Pollumi a little bit. 32 00:02:27,159 --> 00:02:30,240 Oh, I'm not really sure how it differs, but it combines infrastructure 33 00:02:30,280 --> 00:02:36,039 and runtime code in one language, enabling and I'm reading here enabling developers to 34 00:02:36,080 --> 00:02:39,919 stay in their creative flow and to deliver better software, faster and more securely. 35 00:02:40,840 --> 00:02:46,199 And just looking on it, you know, here, here's a little 36 00:02:46,240 --> 00:02:53,120 sample clip. Let bucket equal new cloud dot bucket Grand brand. That seems 37 00:02:53,120 --> 00:03:00,400 pretty powerful, right, Yeah, bucket dot put so very cool, And 38 00:03:00,919 --> 00:03:07,400 there's a local simulator to stay in your creative flow with minimal context switching and 39 00:03:07,479 --> 00:03:10,719 immediate feedback. Like I said, it looks good. I haven't run it, 40 00:03:10,840 --> 00:03:15,759 but you know, Joel seemed to like it and it looks pretty cool. 41 00:03:15,879 --> 00:03:17,360 Yeah, that's awesome. And there is a video there, so if 42 00:03:17,400 --> 00:03:21,719 you want to just watch the YouTube video, you'll probably get a lot more 43 00:03:22,840 --> 00:03:28,080 information about it. Wing lang dot io go get it. Who's talking to 44 00:03:28,159 --> 00:03:30,919 us? Richard grabbed a comment off a show eighteen twenty two, the one 45 00:03:30,919 --> 00:03:34,680 we did with Billy Hollis back in December talking about high level design, and 46 00:03:34,719 --> 00:03:37,560 of course Billy's one of our great thinkers in the UX space, and there's 47 00:03:37,599 --> 00:03:40,120 lots of good comments on the show. This particular one is from par nine 48 00:03:40,240 --> 00:03:45,240 one one. I don't know what that means. Is that an x or 49 00:03:45,360 --> 00:03:49,840 Twitter account? Possibly? Yeah, it didn't map to one, but he 50 00:03:49,879 --> 00:03:53,159 says I always love a conversation to discusses workflow and design. As a developer 51 00:03:53,240 --> 00:04:00,639 who's heavily into windforms development, ensuring design compliments workflows and vice versa is important 52 00:04:00,639 --> 00:04:02,520 and the hardest part of developing a solution. One of the best ways I've 53 00:04:02,520 --> 00:04:06,319 worked on on how to improve this is to actually do the role of the 54 00:04:06,360 --> 00:04:12,360 solution is designed to assist. He's shouldn't find repetitive tasks that you can automate, 55 00:04:12,400 --> 00:04:15,840 So I think what he's really talking about here is do the work, 56 00:04:15,240 --> 00:04:18,199 like what is the workflow? Where can way through it? And the UX 57 00:04:18,240 --> 00:04:23,160 can kind of be dictated from that. So spending your time with the app, 58 00:04:23,240 --> 00:04:26,319 not just thinking about it abstractly. Yeah. Yeah, no, substitute 59 00:04:26,360 --> 00:04:29,920 for that fewest number of clicks, use numbers of change of hand position like 60 00:04:30,120 --> 00:04:31,839 to the mouse and to back again. All those kinds of things make a 61 00:04:31,879 --> 00:04:35,639 better UI. Yeah, it's always the same. Right. You talk to 62 00:04:35,680 --> 00:04:41,079 the customer and you say, if you could write this program, like, 63 00:04:41,199 --> 00:04:44,319 how would you want to operate it start Where would you want to start? 64 00:04:44,319 --> 00:04:46,759 Where would you want to go? What are the generals? What are the 65 00:04:46,800 --> 00:04:50,160 details? And how do you want to get through that? And often people 66 00:04:50,240 --> 00:04:55,079 don't believe you. They like they think that they're stuck in a box of 67 00:04:55,120 --> 00:04:58,000 a form or this or that. But you say, you know, if 68 00:04:58,000 --> 00:05:00,399 this was a car, how would you drive it? Yeah, Chrishma is 69 00:05:00,399 --> 00:05:03,639 shaking her head, nodding her head. Yes, she's been there, no 70 00:05:03,800 --> 00:05:08,199 question. And you know that part of that is watching somebody work, like, 71 00:05:08,240 --> 00:05:10,399 look, how do you normally do this? And all? And you 72 00:05:10,480 --> 00:05:13,759 realize those little jumps outs even after you've built the app, going back and 73 00:05:13,800 --> 00:05:16,000 watching people use it, you know, a few months later going wow, 74 00:05:16,079 --> 00:05:20,199 that's not what I always think. Yeah, right, hey, par thank 75 00:05:20,240 --> 00:05:23,160 you so much. Here a comment at a copy of music Coby. It 76 00:05:23,199 --> 00:05:25,120 is on its way to you. And if you'd like a copy of music 77 00:05:25,160 --> 00:05:28,279 Coby, I read a comment on the website at dot net rocks dot com 78 00:05:28,399 --> 00:05:30,800 or on the facebooks. We publish every show there and if you comment Darren, 79 00:05:30,800 --> 00:05:32,319 we're reading the show. We'll say your copy music cobuy. And you 80 00:05:32,399 --> 00:05:35,800 can definitely follow us on Twitter or x as it's called now, but that's 81 00:05:35,839 --> 00:05:40,759 fine, but the real fund is happening over on Mastodon. I met Carl 82 00:05:41,000 --> 00:05:45,600 Franklin at tech Hub dot social and I'm Rich Campbell at mastodon dot social. 83 00:05:45,800 --> 00:05:49,160 Send us too, and you know you could send us a comment by two 84 00:05:49,560 --> 00:05:53,439 if you like, you can and we will respond. We will respond, 85 00:05:53,519 --> 00:05:57,800 We read them all. Okay, let me introduce Chris Chrishma Jenna is a 86 00:05:57,920 --> 00:06:02,680 data scientist with the UX Insights Team at IBM in San Francisco. I should 87 00:06:02,720 --> 00:06:08,480 just stop there, because that's amazing. That's awesome, the full amazing full 88 00:06:08,560 --> 00:06:11,839 stop right. I mean, you're the only data scientist in the org. 89 00:06:13,079 --> 00:06:17,879 She supports eighty plus user researchers and designers and uses data to understand user struggles 90 00:06:18,399 --> 00:06:25,480 and opportunities to enhance product experiences. She's delivered fifty plus talks and workshops at 91 00:06:25,560 --> 00:06:30,920 multiple conferences around the globe, including piicon US. Grishma is extremely passionate about 92 00:06:31,040 --> 00:06:35,600 encouraging, mentoring, and empowering people, especially women and students, in the 93 00:06:35,680 --> 00:06:41,319 world of technology. She's been recognized as a Fellow by the Python Software Foundation 94 00:06:41,439 --> 00:06:46,519 and also serves on advisory and leadership boards for nonprofits and other organizations. In 95 00:06:46,600 --> 00:06:48,199 other words, a whole lot of awesome. Thanks for going with us, 96 00:06:48,680 --> 00:06:51,720 Thank you so much for inviting me. Yeah, what did you think about 97 00:06:51,759 --> 00:06:56,639 the comment that Richard read there? Very interesting? I think right about the 98 00:06:56,720 --> 00:07:00,480 workflows and just having the usocentric mindset, so that something we focus a lot 99 00:07:00,560 --> 00:07:04,839 at IBEM as well. You must have heard that IBM's one of the frontiers 100 00:07:05,000 --> 00:07:09,240 and on the frontier of the design. So they came up with this Enterprise 101 00:07:09,319 --> 00:07:13,839 Design thinking framework, which is all about making the process user centric, coming 102 00:07:13,920 --> 00:07:17,199 up with something called an empathy map, which tries to put yourself in the 103 00:07:17,319 --> 00:07:19,759 use of the users and be like, Okay, what is this user thinking, 104 00:07:19,800 --> 00:07:23,759 what is the feeling? What is this user doing, and what is 105 00:07:23,800 --> 00:07:27,600 the current you know, state the ass scenario and then hopefully going to the 106 00:07:27,639 --> 00:07:31,079 TV which is the future scenario where there are hopefully no use of frustrations or 107 00:07:31,079 --> 00:07:34,079 struggles and it's a very seamless end to end EXPERI yeah, because of course 108 00:07:34,120 --> 00:07:36,319 my instinct as soon need to say what is user feeling? It's like I'm 109 00:07:36,360 --> 00:07:45,759 expecting frustrated lunch. Absolutely where does the data science fall into this? Because 110 00:07:45,959 --> 00:07:47,680 that's what grabbed me, was this idea. Yeah, you know, so 111 00:07:47,839 --> 00:07:55,759 much of UX design seems intuitive based rather than is there a data approach to 112 00:07:55,879 --> 00:07:59,879 us? A science there is? Yeah, I think this is definitely something 113 00:08:00,079 --> 00:08:03,360 that's a little novel And a lot of people ask me, Okay, you're 114 00:08:03,399 --> 00:08:09,759 a data scientist, but on the UX side, isn't that what's happening, 115 00:08:09,879 --> 00:08:13,600 what's what's going on? Like, shouldn't you be on this quantitative side with 116 00:08:13,680 --> 00:08:16,759 the logical, practical thinking. Yeah, and this is this is definitely a 117 00:08:16,879 --> 00:08:20,759 very unusual role. I will say it's not a stereotypical role. But where 118 00:08:20,879 --> 00:08:24,639 data false is that data is the language that customers used to talk to about. 119 00:08:26,720 --> 00:08:31,120 So I think there's a huge, huge potential there to actually mind all 120 00:08:31,160 --> 00:08:33,200 of that data, all of that feedback that customers are giving us, whether 121 00:08:33,320 --> 00:08:37,600 that's quantitatively through the instrumentation of the products of qualitatively, well you know, 122 00:08:37,799 --> 00:08:41,879 they're on a customers about called saying yes, office sucks, I'm going to 123 00:08:41,919 --> 00:08:43,039 go to the next competitor. You know, I could have done this with 124 00:08:43,120 --> 00:08:48,399 Excel. Yeah, my brother could do this in access. You're you're reminding 125 00:08:48,480 --> 00:08:54,320 me of Mark Miller's Science of Great UI talk where you know, the first 126 00:08:54,360 --> 00:08:58,200 time we talked to him about what you mean science of UI, Like you 127 00:08:58,320 --> 00:09:01,639 said, it seems like into would be taking over, but he talked about 128 00:09:01,759 --> 00:09:05,240 you know, there are certain things about the way your eye sees things and 129 00:09:05,279 --> 00:09:11,120 the way your brain reacts, and and that can be other way. Contrast 130 00:09:11,320 --> 00:09:15,639 is perceived as importance. The things that are higher contrasts tend to be more 131 00:09:15,679 --> 00:09:20,159 important than lower contrasts, And so that can translate directly into rules that you 132 00:09:20,279 --> 00:09:24,519 can use for a better user interface, a better UX experience. And so 133 00:09:26,080 --> 00:09:30,279 that's exactly what I thought of when you said, you know your data science 134 00:09:30,360 --> 00:09:33,080 around UX. There's a lot to it. There's a lot to it. 135 00:09:33,279 --> 00:09:37,679 But I also see we're good. We're getting good at instrumenting stuff now, 136 00:09:37,799 --> 00:09:39,840 the open telemetriies and things, the lists, like we have the tremendous amount 137 00:09:39,840 --> 00:09:43,799 of data we can pull from people using the app, not even talking about 138 00:09:43,799 --> 00:09:46,399 the information of putting into it, but how they interact with the interface, 139 00:09:48,000 --> 00:09:50,279 And I wonder how well we can use that data. Like I've worked on 140 00:09:50,320 --> 00:09:54,200 a bunch of that, and we end up with doing heat maps where it's 141 00:09:54,240 --> 00:09:56,440 like, oh, their mouse is always over here, and it's it's rarely 142 00:09:56,519 --> 00:10:01,399 over there that always do you go to eye track. I haven't dealt with 143 00:10:01,480 --> 00:10:05,879 eye tracking data myself, but yes, I know that some of the teams 144 00:10:05,000 --> 00:10:09,399 use the eye tracking data. But again, like you mentioned, right Richard, 145 00:10:09,519 --> 00:10:11,639 with a lot of instrumentation data, where are the clicks happening, where's 146 00:10:11,639 --> 00:10:15,879 the mouse activity happening? How much are they strolling through the page? Are 147 00:10:15,919 --> 00:10:18,639 they skipping over some demos because they're like, yeah, yeah, I already 148 00:10:18,679 --> 00:10:20,679 know this stuff, like you know, just just get me to the good 149 00:10:20,720 --> 00:10:22,759 stuff. I know all of this basic stuff. But I would say it 150 00:10:22,840 --> 00:10:28,519 also really comes down to two things. Number one is what data are you 151 00:10:28,720 --> 00:10:31,799 exactly capturing? How are those metrics defined? Right? Because what might be 152 00:10:31,919 --> 00:10:37,759 a conversion for my product or for my perspective might not be a conversion according 153 00:10:37,799 --> 00:10:41,039 to you. So I think just having that level of standardization of Okay, 154 00:10:41,080 --> 00:10:43,879 this is how we're going to track a log in, this is where you 155 00:10:43,960 --> 00:10:48,600 know, this particular workflow starts, this is how this particular workflow ends. 156 00:10:48,960 --> 00:10:50,799 What happens if there's some drop off in the middle, or you know what 157 00:10:52,600 --> 00:10:54,679 conversion means someone who gets to the end of a work yes, exactly right, 158 00:10:54,720 --> 00:10:58,399 So whether that's a purchase order going through or somebody putting you know, 159 00:10:58,600 --> 00:11:03,480 starting a workflow again? Now does that mean do you start a workflow? 160 00:11:03,519 --> 00:11:07,039 Doesn't mean ending the workflow? Where do you exactly when a motion points? 161 00:11:07,120 --> 00:11:11,360 I mean we usually think about conversions from an e commerce perspective, Yeah, 162 00:11:11,399 --> 00:11:16,720 but I love the idea of applying that to an internal business workflow. That'd 163 00:11:16,759 --> 00:11:18,080 be a great number to have. I've never thought about having. Is how 164 00:11:18,080 --> 00:11:22,759 many times have people started down a form? Yeah, and then a band 165 00:11:22,840 --> 00:11:26,919 abandoned it? Yeah? And if you could see a pattern that data is 166 00:11:26,000 --> 00:11:31,039 that they consistently abandoned here, yes, which might be they need additional piece 167 00:11:31,120 --> 00:11:33,600 of information that interrupts them enough they have to leave the app to do that. 168 00:11:35,399 --> 00:11:39,960 Like now you've clearly got a feature waiting to be built. Absolutely, 169 00:11:39,559 --> 00:11:45,360 And I think the other thing is also the amount of data are capturing right 170 00:11:45,919 --> 00:11:48,399 that that plays a huge role because unfortunately I've seen too many of these instances, 171 00:11:48,519 --> 00:11:54,120 especially now given that AI and Jenny I chat GPTY is pretty much do 172 00:11:54,159 --> 00:11:56,120 you know all people are talking about executives coming and say, Okay, we're 173 00:11:56,159 --> 00:12:01,039 going to build this genitive AI model with deep learning in machine learning and all 174 00:12:01,080 --> 00:12:03,320 of that. But when you go to look at the data. There isn't 175 00:12:03,360 --> 00:12:07,519 a lot of data. There is not even good quality data available to do 176 00:12:07,679 --> 00:12:11,600 those modelings. So I think having some realistic expectations really help speed up the 177 00:12:11,639 --> 00:12:16,320 process. It also strikes me, especially when talking about the large language models 178 00:12:16,360 --> 00:12:20,039 and even the general mL stuff, that's a pretty big hammer for what is 179 00:12:20,279 --> 00:12:26,159 not that big a problem if you're thinking about it, well, like I 180 00:12:26,279 --> 00:12:28,879 just wonder. I mean, it's cool, it's cool hammer, but it's 181 00:12:28,879 --> 00:12:31,159 like, really do you want to That's like it's almost like, hey, 182 00:12:31,240 --> 00:12:33,919 we just rewrote this and Ruby on rails, our problems go away. But 183 00:12:35,039 --> 00:12:37,399 the more important thing is like you're going to leave me alone for six months 184 00:12:37,440 --> 00:12:39,519 while I do that, Like if you pull out the mL hammer, you're 185 00:12:39,559 --> 00:12:45,559 really just getting a three month pass. Yeah, when actually studying the data 186 00:12:45,639 --> 00:12:50,559 meaningfully could get you to a solutions faster and simpler yea, or at a 187 00:12:50,639 --> 00:12:54,360 minimum, give you better inputs for that machine learning model in the end. 188 00:12:54,399 --> 00:12:58,279 Anyway, Yes, completely, I agree. I heard this. You know. 189 00:12:58,440 --> 00:13:03,480 An ology that someone used is that machine learning or deep lunning models are 190 00:13:03,519 --> 00:13:07,279 often the ferraris when you might even need like a bike to I'm talking a 191 00:13:07,440 --> 00:13:16,240 pickup truck, a two wheeler will do without a motor. Yeah, so 192 00:13:16,440 --> 00:13:22,960 I mean that I've found these days with telemetry specifically, it's either way too 193 00:13:24,039 --> 00:13:26,919 little and you never you can't get anything useful lot of it, or it's 194 00:13:26,039 --> 00:13:31,360 so much it's just unmanageable. Yeah, how do you try and thin the 195 00:13:31,440 --> 00:13:35,440 data out? There? Is there a tooling approach? I mean, if 196 00:13:35,440 --> 00:13:37,000 it takes too little basimbly, nothing we can do. So we kind of 197 00:13:37,039 --> 00:13:41,600 got to go with it too much. So now what Yeah, I think 198 00:13:41,639 --> 00:13:46,559 the kind of approach I leaned towards is start with things you are absolutely confident 199 00:13:46,639 --> 00:13:50,440 about. We become up with two metrics that you say, okay, the 200 00:13:50,480 --> 00:13:54,879 conversion rate and you know, maybe the number number of logins per month or 201 00:13:54,960 --> 00:13:58,519 you know something like that. Just come with two metrics that you're absolutely confident 202 00:13:58,600 --> 00:14:01,519 and that you can manually check that, Okay, what's being instrumented, what's 203 00:14:01,519 --> 00:14:05,039 being captured, is what we are expecting, and then you feel confident. 204 00:14:05,120 --> 00:14:09,519 Maybe keep building onto that sect thing through a bunch of obvious and errors for 205 00:14:09,519 --> 00:14:13,039 a log in, yeah right, the speil log and failed log at an 206 00:14:13,039 --> 00:14:16,600 incomplete loggin. I don't think it's more than that. And now can we 207 00:14:16,720 --> 00:14:18,639 see them all can we now see a number behind that? Or would you 208 00:14:18,720 --> 00:14:24,720 start with the happy path? Depends depends. So the way I like to 209 00:14:24,799 --> 00:14:28,480 do it is that let's pick up two metrics in the instrumentation phase that we 210 00:14:28,559 --> 00:14:31,720 are confident to it that we can check. Once we have that in plates 211 00:14:31,799 --> 00:14:35,519 lists, start thinking about what future questions we would want to answer. So 212 00:14:35,559 --> 00:14:37,960 I might not have the data today, but three months down the line, 213 00:14:37,960 --> 00:14:41,279 I want to see how many users are abandoning on a particular workflow, right, 214 00:14:41,519 --> 00:14:43,519 are converting to a particular workflow, all right, and then start those 215 00:14:43,679 --> 00:14:46,679 that reverse engineering of okay, to be able to answer that question. The 216 00:14:46,759 --> 00:14:50,279 kind of data I would need is the abandons, maybe the Paige speed load. 217 00:14:50,440 --> 00:14:54,320 Maybe you know how many demos they are watching, how much time they 218 00:14:54,320 --> 00:14:58,080 are spending on this particular product, And then go back and see what other 219 00:14:58,159 --> 00:15:01,320 metrics that you could come up with actually start tracking all of that data. 220 00:15:01,720 --> 00:15:07,759 Now, when you're talking about data science and in this I know you mentioned 221 00:15:07,840 --> 00:15:13,120 AI a little bit passively, but do you use predictive analytics on this data 222 00:15:13,240 --> 00:15:18,720 to try to figure things out or any other kind of you know AI if 223 00:15:18,759 --> 00:15:22,840 you will, Yeah, yeah, okay, So a bit of a disclaimer, 224 00:15:22,919 --> 00:15:26,240 I'm not really sure where the lines between artificial intelligence and data science, 225 00:15:26,360 --> 00:15:31,039 you know, blur, but I always feel yeah, I mean data science 226 00:15:31,120 --> 00:15:35,080 is really at the very very lowest level of AI. It is all statistics, 227 00:15:35,120 --> 00:15:39,279 it's data sensors, machine learning. So just for the purposes of our 228 00:15:39,320 --> 00:15:41,440 conversation, I'm going to interchangeable use or just maybe stick. Do you know 229 00:15:41,559 --> 00:15:48,080 data sciensing machine learning? Yes, Predictive analytics is definitely something very popular. 230 00:15:48,679 --> 00:15:52,679 I think one of the most most abundantly available use cases is predicting if a 231 00:15:52,799 --> 00:15:58,320 customer is about to leave your product, So it's basically predicting customer attrition. 232 00:15:58,960 --> 00:16:03,080 So you can maybe start looking at you know, has their overall usage of 233 00:16:03,120 --> 00:16:07,720 the product decreased? Have they opened a lot of support tickets recently? Maybe 234 00:16:07,759 --> 00:16:11,159 those support tickets have not given resolutions to them right, or maybe the average 235 00:16:11,200 --> 00:16:15,480 time taken to handle its support ticket is increasing. You also have a lot 236 00:16:15,519 --> 00:16:18,240 of feedback surveys that they give, right like how happy are you with this 237 00:16:18,360 --> 00:16:21,879 experience? Would you recommend our product to a friend or a colleague, and 238 00:16:21,919 --> 00:16:23,960 then just looking at the trend of those goals. So that is definitely one 239 00:16:25,039 --> 00:16:29,759 case where you know the predictive models can help and alert the product team that 240 00:16:29,879 --> 00:16:33,240 here this customers at a high risk of actually, you know, terminating the 241 00:16:33,360 --> 00:16:37,080 contract. Yeah, so so better go ahead and maybe have like a touch 242 00:16:37,159 --> 00:16:41,840 point of okay before the customers actually talking about canceling exactly. The idea and 243 00:16:41,720 --> 00:16:45,799 the idea that you would contact them and say, hey, you've been having 244 00:16:45,840 --> 00:16:48,480 a time can we help you? Like, what can I do? Extra? 245 00:16:48,840 --> 00:16:53,240 Yeah? Why did you post this on Twitter's angry Sweet? Yeah? 246 00:16:53,399 --> 00:17:00,440 Just call support? Yeah, you know I have. I think I talk 247 00:17:00,480 --> 00:17:03,000 about this the show that Bloody Dishwasher, the Melee. Yeah. I kept 248 00:17:03,039 --> 00:17:07,720 breaking Yeah, and I couldn't get regular support. Regular sport wouldn't respond to 249 00:17:07,759 --> 00:17:12,240 me four weeks. But if I tweeted and tweeted yea that day. Yeah. 250 00:17:12,440 --> 00:17:15,559 And at one point I'm like, listen, I'm not that mean a 251 00:17:15,680 --> 00:17:18,839 guy, right, I just didn't get anywhere I would Is there a better 252 00:17:18,880 --> 00:17:22,039 way to do this? And the and the person on the other end's like, 253 00:17:22,119 --> 00:17:26,240 literally, nope, this works, just use it. I'm I'm in 254 00:17:26,440 --> 00:17:30,920 that limbo period right now with the HVAC system in my studio. Oh man, 255 00:17:30,400 --> 00:17:33,240 yeah, they's a new one. Well they I don't want to take 256 00:17:33,319 --> 00:17:37,559 up too much time with this story. But they sold me a twenty four 257 00:17:37,640 --> 00:17:41,960 thousand BTU unit for a six hundred and eighty square foot room that's insulated to 258 00:17:41,039 --> 00:17:47,759 our twenty R fifty R fifty okay, really freaking insulated. Yeah. Yeah, 259 00:17:47,880 --> 00:17:48,960 you don't have to pump very much cold air in there. Yeah. 260 00:17:49,039 --> 00:17:53,160 So it cools off in about five seconds with this thing, and then it 261 00:17:53,240 --> 00:17:57,680 stops, and so the humidity rises to sixty five seventy percent, Yeah, 262 00:17:57,720 --> 00:18:02,079 and it gets unpleasant. And so I had to beg with them to you 263 00:18:02,160 --> 00:18:04,039 know, get somebody out here to fix this, and they finally decided to 264 00:18:04,119 --> 00:18:07,000 get a smaller unit in there, but then they wanted to charge me more. 265 00:18:08,400 --> 00:18:11,920 That's like, wait a second, you screwed up, and you want 266 00:18:11,960 --> 00:18:15,279 to charge me more. Yeah, And so now we're going we didn't get 267 00:18:15,319 --> 00:18:18,119 anywhere with their subcontractor, and now we're going to the actual contractor, who 268 00:18:18,359 --> 00:18:22,680 company who we bought it from, right and there, and they are unresponsive 269 00:18:22,799 --> 00:18:26,480 right now. And if they come back with I'm sorry, you're gonna have 270 00:18:26,519 --> 00:18:29,240 to pay it, then then I'm going to unleash the kracking. Yeah, 271 00:18:29,440 --> 00:18:32,400 do the thing, yep, yep, you just need to send a tweet 272 00:18:32,440 --> 00:18:37,680 to them, right, if there's to start, but begin the public humiliation, 273 00:18:37,920 --> 00:18:41,519 right, let it begin. Well, we'll keep you posted on that. 274 00:18:41,720 --> 00:18:45,839 Yeah, no kidding, but I mean you still your language to me 275 00:18:45,920 --> 00:18:52,599 strikes as external customers reacting to product, right, I mean, and that's 276 00:18:52,680 --> 00:18:56,720 fine, Like those are all good things, and it is interesting to synthesize 277 00:18:56,799 --> 00:19:02,440 data from your tech support side as well as perhaps a sales channel. Maybe 278 00:19:02,519 --> 00:19:04,640 you're doing some monitoring at the social media level, to say, do we 279 00:19:04,720 --> 00:19:11,000 have some sentiment analysis going on? Coordinate this information, but then to add 280 00:19:11,039 --> 00:19:14,920 the utilization of the software. Yeah, Like this is not just a person 281 00:19:15,079 --> 00:19:18,279 being randomly grumpy, right, it's this is where they're falling off on the 282 00:19:18,319 --> 00:19:22,119 workflow and they do that consistently. Like, this is not a whining person. 283 00:19:22,200 --> 00:19:27,039 They're trying to get it done and they're abandoning for whatever reason. Like 284 00:19:27,160 --> 00:19:32,720 that that is such actionable data. I am you are watching this customer try, 285 00:19:32,880 --> 00:19:34,759 Yeah, Like shouldn't you do something about that? Yeah? Yeah? 286 00:19:34,799 --> 00:19:38,519 And I think one case where data science comes in really helpful is that maybe 287 00:19:38,680 --> 00:19:42,640 as a front front facing you know, you're facing the client and you're having 288 00:19:42,680 --> 00:19:47,160 these conversations, maybe it's a product manage. Maybe it's a develops having a 289 00:19:47,240 --> 00:19:49,319 conversation with a very frustrated customer, right saying, Okay, you know this 290 00:19:49,440 --> 00:19:52,720 isn't working the way it's expected. I'm not going to do a bunch of 291 00:19:52,799 --> 00:19:56,920 issues. And that's a data point that you have. But when you look 292 00:19:56,039 --> 00:20:00,920 at the data perspective or quant perspective it you get that scale, right, 293 00:20:00,000 --> 00:20:04,000 So now you start to notice this is not just one isolated customer, it's 294 00:20:04,039 --> 00:20:08,119 actually ten customers, hundred customers, right, thousands of customers that are doing 295 00:20:08,200 --> 00:20:11,240 that. And I think that helps to also remove that bias, right, 296 00:20:11,319 --> 00:20:15,000 which it could work in both ways, not just that customers the pain exactly. 297 00:20:15,079 --> 00:20:18,400 I guess the next question you asked when you talk about like a failed 298 00:20:18,440 --> 00:20:21,480 workflow like gus, well, how many succeed? Yeah? How often do 299 00:20:21,559 --> 00:20:25,160 we succeed? How many other abandons do we have? I really like that 300 00:20:25,359 --> 00:20:29,359 you take the personality element out of it to some degree, and it's like, 301 00:20:29,519 --> 00:20:33,839 here's a set of data and you have found with someone willing to communicate 302 00:20:33,920 --> 00:20:37,880 with you about the problem, don't make them the enemy. Here's all these 303 00:20:37,920 --> 00:20:41,319 people that aren't communicating about this problem but are having the same problems. Yeah, 304 00:20:41,480 --> 00:20:45,079 exactly, yeah, and it works two ways, right, Like the 305 00:20:45,240 --> 00:20:48,759 user researcher could maybe be in a conversation and then notice that, okay, 306 00:20:48,839 --> 00:20:51,920 the customer is struggling with the product, and then they go to the data 307 00:20:51,960 --> 00:20:53,599 science team or the analytics team and say, okay, we saw this one 308 00:20:53,680 --> 00:20:56,839 person. Are there other people who are struggling with this? And then you 309 00:20:57,119 --> 00:21:00,000 notice that you know, this effect is that scale. And another thing it 310 00:21:00,079 --> 00:21:03,279 also does is like often product managers or you know, a client facing teams 311 00:21:03,680 --> 00:21:08,839 here from really important like it's called the hippo the highest paid person's opinion, 312 00:21:10,319 --> 00:21:14,880 and it's usually the highest paid customer, like we hate this feature. When 313 00:21:14,880 --> 00:21:17,880 are you in unveiling? You know, XOY is that feature and they're just 314 00:21:17,960 --> 00:21:21,119 going to be you know, maximum attention to them because that's where the money 315 00:21:21,240 --> 00:21:26,799 is exactly, all right. But then when you combine it with the data, 316 00:21:26,960 --> 00:21:32,559 you can actually realize that is this person's problem? You know, actually 317 00:21:32,880 --> 00:21:34,920 yes, is it unique to the customer or is it a little more pervasive 318 00:21:34,960 --> 00:21:38,599 where other customers are also facing this? And then it's probably a future worth 319 00:21:38,640 --> 00:21:41,680 including on the road. Oh man, now you've now you make me feel 320 00:21:41,680 --> 00:21:45,920 like it's a superpower, like if you applying data science to all of these 321 00:21:47,039 --> 00:21:51,119 kinds of problems so that you're always talking from more than the immediate issue in 322 00:21:51,160 --> 00:21:53,720 front of you. Exactly. My automatic reaction to a person's having a problem 323 00:21:53,720 --> 00:21:57,240 with this feature is to be able to apply some data analytics, say how 324 00:21:57,279 --> 00:22:00,839 many other people? How many six? What are the other scenarios? Like, 325 00:22:02,119 --> 00:22:04,039 I mean, even think aboufter our past scenario where you've got a customer 326 00:22:04,079 --> 00:22:08,359 talk and now you've found out this other customers failing, Like lots of people 327 00:22:08,640 --> 00:22:14,440 exclude themselves from serving, but they don't want to do that. But to 328 00:22:14,720 --> 00:22:17,839 call that customer say hey, you know, I'm from such a such a 329 00:22:17,880 --> 00:22:21,920 company and we'll see you're struggling with this thing and we're trying to address it. 330 00:22:21,960 --> 00:22:23,559 Would you help us to understand what you're trying to do, Like, 331 00:22:23,680 --> 00:22:26,279 I think you'd get good response from that. Yeah, I mean, you're 332 00:22:26,319 --> 00:22:30,559 going to hum the fine line of being creepy. You know, we've been 333 00:22:30,640 --> 00:22:37,160 watching you and we see you're not having a good time. Better to say 334 00:22:37,279 --> 00:22:44,119 that you're a hired psychic. You've discovered that they're the spirits. Have told 335 00:22:44,200 --> 00:22:48,440 me about your unsatisfaction. But I know. I do think it's an interesting 336 00:22:48,599 --> 00:22:52,400 part of this is to just have that reflex of let's can we quickly gather 337 00:22:52,559 --> 00:22:56,480 more data, like do you see an opportunity to build up tooling? Like 338 00:22:56,559 --> 00:22:59,519 I'd really want to make that mechanism simple. I don't want to have to 339 00:22:59,599 --> 00:23:03,799 call you Krishma as the expert to say, can you give me analysis on 340 00:23:03,880 --> 00:23:06,680 this? Yeah, I wanted to be a button the moment it pops up. 341 00:23:07,000 --> 00:23:11,839 What is the IBM system that you talked about earlier, You're sort sort 342 00:23:11,880 --> 00:23:15,400 of hinted at it. What is that play? Yeah? I think we 343 00:23:15,519 --> 00:23:18,359 have a bunch of different tooling and framework and to be honest, one of 344 00:23:18,359 --> 00:23:22,519 the issues we're facing a standardization because just the software unit that I work with, 345 00:23:22,720 --> 00:23:26,519 we have around eighty two hundred different products and all of them have absolutely 346 00:23:26,640 --> 00:23:30,759 varying degrees of data maturity, right from like we haven't really put this product 347 00:23:30,799 --> 00:23:34,000 into production, or we only have like two clients, all the way to 348 00:23:34,119 --> 00:23:37,680 we have thousands plus, you know, users every month, and this is 349 00:23:37,839 --> 00:23:41,319 just a huge amount of data. So I think it's it's hard to generalize 350 00:23:41,359 --> 00:23:45,440 what that looks like, which is why where I need to have a personalized 351 00:23:45,440 --> 00:23:48,279 approach for the team I'm dealing with, the product, I'm dealing with, 352 00:23:48,480 --> 00:23:53,079 what is the exact objective or the problem statement that we're trying to find automation. 353 00:23:53,200 --> 00:23:57,119 That's that's very interesting because that is definitely something that I'm trying to do 354 00:23:57,200 --> 00:24:00,759 as well as unit I've seen a lot of other company needs do as well. 355 00:24:00,240 --> 00:24:03,359 Is this concept of anomally detection, right, right, So throwing all 356 00:24:03,359 --> 00:24:07,240 of the data into a model that will help you understand if a particular user 357 00:24:07,279 --> 00:24:11,720 activity or a particular user is you know, I'm showing up as an anomaly. 358 00:24:11,799 --> 00:24:15,000 They're doing something different, they're doing something unexpected, maybe they're running into 359 00:24:15,039 --> 00:24:18,920 a bunch of issues or errors. And then that also goes into with incident 360 00:24:19,000 --> 00:24:22,000 management. So a lot of security teams do that to see it, right, 361 00:24:22,039 --> 00:24:26,400 maybe there's a DDoS attack happening, that's anomally for you, so that's 362 00:24:26,440 --> 00:24:29,319 an incident, right, So using kind of the same principles, So that 363 00:24:29,480 --> 00:24:32,720 is definitely one way of automating. But again that only comes into play when 364 00:24:32,799 --> 00:24:36,960 you have a highly mature team in terms of the instrumentation, in terms of 365 00:24:37,000 --> 00:24:40,640 the data, in terms of understanding how many users you have, yeah, 366 00:24:40,799 --> 00:24:44,599 and what those scenarios look like like it does feel like something I'd want to 367 00:24:44,720 --> 00:24:49,079 build into a tech support client. So not only am I pulling up the 368 00:24:49,160 --> 00:24:55,799 anomaly, but then provides analytics run the anomally say this is the four hundred 369 00:24:55,880 --> 00:24:57,920 times this has happened this week. Yeah, right, and with this many 370 00:24:57,960 --> 00:25:03,039 different customers, just to give a sense of urgency and scope to that. 371 00:25:03,799 --> 00:25:06,440 Like I said, I love the idea of being able to resist the hippo 372 00:25:06,480 --> 00:25:10,119 because they're asking for a fitture that's distinct to them. The majority of customers 373 00:25:10,160 --> 00:25:12,960 are fine. Yeah, And at the same time, when you have a 374 00:25:14,039 --> 00:25:15,720 problem, show up to see, okay, well this is unique to that 375 00:25:15,880 --> 00:25:18,880 customer. I can approach it differently. And I'm not saying like ignore it 376 00:25:18,920 --> 00:25:22,559 but so much, but recognize it's an unusual work Yeah, as of course 377 00:25:22,640 --> 00:25:27,720 to it's a usual workflow that often fails, and maybe you can get a 378 00:25:27,759 --> 00:25:30,160 shape for it. But also you know, maybe it fails ten percent of 379 00:25:30,240 --> 00:25:34,240 the time and ninety percent time it succeeds. Like now you've got a different 380 00:25:34,279 --> 00:25:37,519 shape of how you're going to approach it. Looking at the problem, it 381 00:25:37,640 --> 00:25:41,000 feels like it's time to take a break, So we'll be back after these 382 00:25:41,119 --> 00:25:49,079 very important messages. Hey, we're back. It's Carl Franklin. This is 383 00:25:49,160 --> 00:25:52,599 Richard Campbell, and we're here with Grish Magenna and we're talking data science and 384 00:25:52,720 --> 00:26:02,880 UX and I'm very fascinated by this UX sort of a science framework that IBM 385 00:26:02,920 --> 00:26:07,440 has. Well that is there any one of those things that you would use 386 00:26:07,480 --> 00:26:12,039 at the onset of a project to help design the US or are they sort 387 00:26:12,079 --> 00:26:17,319 of things that you work with after the fact, Like this last example you 388 00:26:17,400 --> 00:26:19,480 gave, seems like you have a mature team and a mature project. Yeah. 389 00:26:19,680 --> 00:26:22,440 Yeah, I think again going back to like at the start, it's 390 00:26:22,519 --> 00:26:26,000 more of let's start thinking about what are the questions you want answer and what 391 00:26:26,119 --> 00:26:30,759 are the metrics you need to track, So it's probably at the earliest stage 392 00:26:30,759 --> 00:26:34,480 of data collection and just deciding what those metrics should look like. I think 393 00:26:34,559 --> 00:26:37,279 one thing I am trying to do as well is when I'm working with these 394 00:26:37,359 --> 00:26:41,359 mature teams and royalizing there are sort of inconsistencies with the data. I'm actually 395 00:26:41,400 --> 00:26:45,440 trying to create this guide of Hey, I worked with this team that had 396 00:26:45,559 --> 00:26:48,440 great data maturity, but we still had inconstituencies, we still had issues figuring 397 00:26:48,480 --> 00:26:52,359 out, you know, what was going on. So as a heads up 398 00:26:52,440 --> 00:26:56,880 to you before you even start instrumenting, learn from our mistakes, take take 399 00:26:56,920 --> 00:27:00,599 our failures into account, and here are the recommendations that would make you know. 400 00:27:00,799 --> 00:27:03,240 So that is definitely some sort of you know, activity that I'm trying 401 00:27:03,279 --> 00:27:07,559 to do earlier in the pipeline. But yeah, I think, I mean, 402 00:27:07,640 --> 00:27:10,119 that's that's all you can do, right because you can speculate. But 403 00:27:10,279 --> 00:27:12,000 at the end of the day, when the data starts full flowing, and 404 00:27:12,119 --> 00:27:15,400 that's when you can actually do some expiation and and that it's an understand is 405 00:27:15,440 --> 00:27:18,200 this good enough? Is this good quality data? Does this make sense? 406 00:27:18,319 --> 00:27:21,519 Or do you maybe need to you know, modify a few things. And 407 00:27:21,519 --> 00:27:25,960 do you have a sort of general knowledge database of things that you've gleaned from 408 00:27:26,119 --> 00:27:30,000 projects working? You know, when somebody says, you know, customer says 409 00:27:30,039 --> 00:27:32,960 that we're thinking of doing it like this, and you say, well, 410 00:27:33,079 --> 00:27:36,640 you know, based on our experience, that approach hasn't really worked well and 411 00:27:36,759 --> 00:27:40,440 years why, like, do you have this sort of general knowledge that you're 412 00:27:40,680 --> 00:27:42,759 gathering that you used too? Yeah, I do have it unformally, but 413 00:27:44,359 --> 00:27:48,160 formally I think this reminds me of what is my primary responsibility is to create 414 00:27:48,240 --> 00:27:53,079 this insights repository of sorts, which is where all the user researchers are coming 415 00:27:53,160 --> 00:27:56,200 in and they have findings, and more often than not, all of these 416 00:27:56,279 --> 00:28:00,920 findings are stowed away and like you know, drives and folded. And let's 417 00:28:00,920 --> 00:28:03,920 say somebody leaves the company two years later, product manages like, oh that 418 00:28:03,119 --> 00:28:07,599 thing was done two months, two years ago, where is the token? 419 00:28:07,680 --> 00:28:10,759 I find? You need to talk to ten people? So we're definitely about 420 00:28:10,920 --> 00:28:14,279 knowledge management. Yeah, it would be really cool to upload that to a 421 00:28:14,559 --> 00:28:18,839 chat chypt like things that you could ask it questions directly. So what do 422 00:28:18,880 --> 00:28:22,519 you think about this? Well based on past experience, Yeah, exactly. 423 00:28:22,000 --> 00:28:26,839 But then also am I other risk of automating my job away? I don't 424 00:28:26,880 --> 00:28:29,960 know. I mean I've always had the experience that there's we're never getting to 425 00:28:29,960 --> 00:28:33,880 the bottom or to do list anyway. So knocking automating stuff, you're just 426 00:28:33,000 --> 00:28:37,920 going to do more faster, We're just going to do more things. But 427 00:28:37,119 --> 00:28:41,759 you have worked with some interesting customers I saw. I haven't seen the top, 428 00:28:41,839 --> 00:28:45,480 but I've seen the abstract for the top where you talked about folks like 429 00:28:45,559 --> 00:28:48,079 Airbnb, and Spotify and so forth, like can you dig into any of 430 00:28:48,119 --> 00:28:51,319 those scenarios. I don't want to get any any trouble but here, but 431 00:28:51,440 --> 00:28:55,119 it's like that just sounds cool. Yeah, nom, So those are those 432 00:28:55,160 --> 00:28:59,000 are case studies. I wish I could claim that those are the projects that 433 00:28:59,079 --> 00:29:02,680 I haven't done. Those are case studies, though, but very interesting ones. 434 00:29:02,759 --> 00:29:06,519 So I think Airbnb and Spotify does a really good job of having their 435 00:29:06,720 --> 00:29:11,079 ux research teams, design teams and their data science teams work in very tight 436 00:29:11,680 --> 00:29:15,039 conjunction with each other. Yeah. So, like I mentioned, right, 437 00:29:15,079 --> 00:29:18,240 you could have a user researchers that says, Okay, this customer, this 438 00:29:18,359 --> 00:29:22,720 users having something odd, let's go to data scientists. But conversely, you 439 00:29:22,799 --> 00:29:26,160 could have data scientists are looking at that aggregated level of data and saying, 440 00:29:26,480 --> 00:29:30,680 hmm, this is kind of unexpected. We didn't really expect such low conversions. 441 00:29:30,039 --> 00:29:33,359 To check the metrics. The web pages are working fine, the instrumentation 442 00:29:33,480 --> 00:29:37,880 is okay, what's happening? Can you use a researcher go and talk to 443 00:29:37,960 --> 00:29:41,559 the customer and try to find out what's exactly going on. That's where the 444 00:29:41,640 --> 00:29:45,240 quantitative and the qualitative aspects of it. This is this is the title from 445 00:29:45,240 --> 00:29:48,279 your child, right, the US data analysts they should be friends, Yes, 446 00:29:48,400 --> 00:29:52,319 they should be best friends friends. Okay, well, because you get 447 00:29:52,400 --> 00:30:00,240 this problem with ux of it only being intuitive and a dodo. Yeah to 448 00:30:00,480 --> 00:30:03,680 the data science folks are going to come in, are going to help you 449 00:30:03,799 --> 00:30:07,640 beef up your case for work absolutely and vice versa. There the data folks 450 00:30:07,680 --> 00:30:11,160 are going to see these anomalies in the arc, like the larger data sets, 451 00:30:11,200 --> 00:30:15,319 and be able to come back with have you looked at and maybe turn 452 00:30:15,440 --> 00:30:21,000 up some other issues exactly exactly, And I think especially as more companies and 453 00:30:21,079 --> 00:30:23,960 executives go into this data driven decision model, right, it really helps to 454 00:30:25,039 --> 00:30:27,400 have that evidence. So you can say, my instincts or my intuition based 455 00:30:27,440 --> 00:30:30,000 on all the research I've done, say that we should, you know, 456 00:30:30,160 --> 00:30:34,200 modify the workflow, we should introduce this feature. But here's the data to 457 00:30:34,279 --> 00:30:37,640 back it up. We spoke to our users or we have looked at instrumentation 458 00:30:37,759 --> 00:30:41,680 data. So here's the evidence that I can you know, help prove across 459 00:30:41,759 --> 00:30:45,960 a point with Because anytime you change a working data flow, you're impairing its 460 00:30:47,039 --> 00:30:49,880 productivity for some period of time. Like people know how to use this, 461 00:30:51,799 --> 00:30:56,680 so the improvement has to be big enough that thing initial impact of the change 462 00:30:56,880 --> 00:31:00,759 is offset by the net long term net benefit. I guess an interesting ROI 463 00:31:00,880 --> 00:31:02,559 question. It's like, hey, we're not going to change this because it 464 00:31:02,640 --> 00:31:07,799 affects so many users and the inkroll improvement doesn't appear to be substantial enough. 465 00:31:07,559 --> 00:31:11,240 That'd be an interesting place, like I don't know if any organizations I've ever 466 00:31:11,319 --> 00:31:14,240 dealt with where they were that mature. Thank you. See, this is 467 00:31:14,279 --> 00:31:17,359 a design improvement, but it's not a big enough design improvement. So I 468 00:31:17,519 --> 00:31:19,720 think we're going to do it and until we can find something large enough that 469 00:31:19,759 --> 00:31:23,559 it's worth disrupting the workflow flow yep, yep. And even then, I 470 00:31:23,599 --> 00:31:27,039 think communication is really important. You need to tell the users here, this 471 00:31:27,160 --> 00:31:32,599 is our latest updated version, and this is where now this menu has actually 472 00:31:32,640 --> 00:31:34,359 gone over to the right side, and these are the new features that have 473 00:31:34,480 --> 00:31:37,640 been on wheals. So I think that communication and almost change management, is 474 00:31:37,640 --> 00:31:41,240 it really important so that the expectations of the users are you know, kept 475 00:31:41,319 --> 00:31:45,319 kept in check. Yeah sure, and yeah, I certainly. I talked 476 00:31:45,319 --> 00:31:51,319 to folks who have client fatigue that stuff, especially in the SaaS products and 477 00:31:51,680 --> 00:31:55,319 cloud stuff. It changes too often, right, And I spend enough time 478 00:31:55,319 --> 00:31:57,000 on the sytems. It's like I get tickets all the time, and I 479 00:31:57,039 --> 00:32:01,720 can see the wave of oh they updated, decline again this week, and 480 00:32:01,759 --> 00:32:05,720 I'm getting all these tickets like somebody move my cheese? Right, and you're 481 00:32:05,759 --> 00:32:07,880 back to what the heck happened? How is it different? Here's how we 482 00:32:07,960 --> 00:32:10,839 deal with it. Talk to you next week. Yeah, and that This 483 00:32:12,000 --> 00:32:14,880 is another case where data science comes in really handy, because then you can 484 00:32:14,920 --> 00:32:17,440 start doing AB testing right and actually look at the data. Rights this change 485 00:32:17,519 --> 00:32:21,960 actually useful? Does this make sense to users? Or are the users are 486 00:32:22,000 --> 00:32:23,680 frustrated and they just keep going back to like, you know, take me 487 00:32:23,759 --> 00:32:27,000 to the old layout, take me to the old format? Can I turn 488 00:32:27,039 --> 00:32:30,160 it off? The first question? Every time? And I love you know, 489 00:32:30,759 --> 00:32:34,839 AB testing isn't always obvious, right, Like how do you know which 490 00:32:34,920 --> 00:32:37,680 to pick A or B? Like? Why was it better? Yeah? 491 00:32:37,920 --> 00:32:39,759 You know, Okay, you serve them fifty fifty percent of the time. 492 00:32:39,839 --> 00:32:45,359 Like what's the metric that shows it was a superior outcome? Higher completion rate? 493 00:32:45,400 --> 00:32:46,920 I guess would be the one? Yes, yes, definitely. I 494 00:32:47,000 --> 00:32:51,640 think you also need to look at the demographics of users, right, did 495 00:32:51,720 --> 00:32:54,000 you do the branching based on where the users are based? And if so, 496 00:32:54,200 --> 00:33:00,240 were there any cultural differences demographical differences that contributed to that? So I 497 00:33:00,359 --> 00:33:04,319 think just it's definitely a very complex topic. How do you have those confounding 498 00:33:04,400 --> 00:33:07,480 variables? How do you make sure they're bearable to attracking, are independent of 499 00:33:07,559 --> 00:33:10,200 each other, or at least as independent as they could an experimental sets. 500 00:33:10,359 --> 00:33:15,200 Now you also have to look at things like is there a geographic event happening 501 00:33:15,279 --> 00:33:17,680 right now? An earthquake, a flood, a heat wave, not that 502 00:33:17,799 --> 00:33:22,720 that would ever happen, Nah, a forest fire? Looking at you, 503 00:33:22,000 --> 00:33:27,400 Richard. Yeah, this comes down to really, how do you how do 504 00:33:27,519 --> 00:33:30,640 you separate A and B? Like? Why? Because my instinct as a 505 00:33:30,720 --> 00:33:37,200 developer is just gonna make it random. But nothing's actually a random So what 506 00:33:37,279 --> 00:33:40,559 do you mean when you say random? So, I mean, at a 507 00:33:40,559 --> 00:33:45,359 minimum, if you're thinking webish here, I've got a cookie, I'm going 508 00:33:45,400 --> 00:33:49,119 to stick a given cookie to A, and the next new cookie I create, 509 00:33:49,200 --> 00:33:52,119 I'm going to stick them to B. Yes? Is that sufficiently? 510 00:33:52,200 --> 00:33:55,119 It's not random at all? Actually, it's a cyclical measure. You haven't 511 00:33:55,559 --> 00:34:00,880 I mean, you're ignoring all the other factors that good enough. I don't 512 00:34:00,880 --> 00:34:05,279 know me the data can tell yeah, yeah, how much do we know 513 00:34:05,359 --> 00:34:08,239 about that cookie to actually say well this is who got it? What was 514 00:34:08,320 --> 00:34:12,320 a separation on them? Yeah, I don't know that I've ever looked at 515 00:34:12,360 --> 00:34:15,519 the data that way, Like Ashley dug In and said, did we make 516 00:34:15,599 --> 00:34:22,239 a good A B test from a client selection perspective before we say hey A 517 00:34:22,440 --> 00:34:27,960 out A outcompleted B therefore A is better? Not even making me think too 518 00:34:28,000 --> 00:34:31,280 much, christ Man, It's all about the biases, right, selection bias, 519 00:34:31,400 --> 00:34:36,400 confirmation bias, recency bias. Everything I've thought about it is a lie. 520 00:34:36,840 --> 00:34:39,079 Yeah, well even in fact that I'm just selecting new cookies so my 521 00:34:39,159 --> 00:34:44,039 existing users don't see it at all, Right, like that that could be 522 00:34:44,119 --> 00:34:46,880 in effect as well. So many opportunities to lie, yeah, or to 523 00:34:47,000 --> 00:34:53,320 just or to select a truth yes in advance ways to filter? Do you 524 00:34:53,400 --> 00:34:57,639 ever have to tackle those kinds of problems like how are we going to separate 525 00:34:58,159 --> 00:35:00,039 to request these request streams to make sure or we get a good AB mix? 526 00:35:00,199 --> 00:35:04,760 Like what does that even look like? I don't think I've actually done 527 00:35:04,760 --> 00:35:07,079 a lot of AB testing on this current role, not not so much. 528 00:35:07,280 --> 00:35:10,719 But yeah, again, I think I'm challenging more with that volume. And 529 00:35:12,159 --> 00:35:14,880 you know, data quality is shoes right now. So hopefully in the future 530 00:35:14,920 --> 00:35:17,960 I'm hoping to get to those good, nicer problems to have. But right 531 00:35:19,039 --> 00:35:22,239 now it's a lot about education and awareness about how the instrumentation needs to happen 532 00:35:22,400 --> 00:35:27,960 and what other things that's possible, because oftentimes it's also just the black box 533 00:35:28,639 --> 00:35:30,280 perspective of AI, right, like, oh, it can do anything and 534 00:35:30,400 --> 00:35:36,079 everything, but we know it's not true to start from. I'm also thinking 535 00:35:36,119 --> 00:35:38,400 from the friends perspective of often it's just a developer decided to do that. 536 00:35:38,480 --> 00:35:40,719 We've decided to do an ab testing group of developers there, it's like you 537 00:35:40,760 --> 00:35:45,280 should be calling your data science people at that moment to talk about how we're 538 00:35:45,280 --> 00:35:49,719 going to discriminate these Yeah, let those experts get into the play and even 539 00:35:49,840 --> 00:35:52,440 know that they're doing that, because that can give us a make the get 540 00:35:52,599 --> 00:35:55,679 a better result if we work at it a bit. Exactly. Yeah, 541 00:35:55,719 --> 00:36:00,719 And I think even having these multicultural, diverse teams all hell because there's a 542 00:36:00,760 --> 00:36:05,400 lot of communication messaging involved. For example, I recently came across this website 543 00:36:05,440 --> 00:36:09,559 that says that each color has very different connotations culturally, like read in some 544 00:36:09,719 --> 00:36:14,760 cases might mean danger or something negative, but in other cases it's actually a 545 00:36:14,800 --> 00:36:17,440 sign of prosperity, right, well, and then the Chinese us right, 546 00:36:17,840 --> 00:36:24,159 yes, exactly exa and the Chinese dragon is a very powerful positive thing, 547 00:36:24,280 --> 00:36:29,880 and European culture it's a negative, right yeah. So yeah, where you're 548 00:36:29,920 --> 00:36:31,920 using that stuff matters as well. And certainly words, right, I mean 549 00:36:32,000 --> 00:36:37,119 it's some words that translate to other languages to mean embarrassing things. Yes, 550 00:36:37,519 --> 00:36:40,920 we see that over and over again. It's the famous Nova story, right, 551 00:36:42,039 --> 00:36:46,000 yeah, Nova, it doesn't go ye no, Chevy Nova, don't 552 00:36:46,159 --> 00:36:52,920 call a card that in a Spanish speaking control mistake, or the slogan Coke 553 00:36:52,000 --> 00:36:57,400 ads life when translated to Japanese. I think it was our Chinese one of 554 00:36:57,440 --> 00:37:04,960 the languages Coke will bring your ancestors back from the dead. Yeah. I 555 00:37:05,039 --> 00:37:09,400 think the most recent one was the video came out with their latest emmal model 556 00:37:09,519 --> 00:37:16,400 or one of those framebooks, and basically translated to Spanish, it meant bottoms. 557 00:37:21,239 --> 00:37:24,280 That's why you see like all these nonsense words as names of products totally 558 00:37:24,480 --> 00:37:28,639 like that, because they're just they don't exist otherwise boid. It helps, 559 00:37:28,679 --> 00:37:34,280 like, for example, work just Fine, which is the sound beforehead slap, 560 00:37:34,400 --> 00:37:38,039 also is the name of my production company. So you know, we've 561 00:37:38,079 --> 00:37:44,920 been digging in purely on the telemetry side, all of this very quantitative data, 562 00:37:45,119 --> 00:37:49,599 but there's a whole qualitative side, right Doing hot surveying well, I've 563 00:37:49,639 --> 00:37:52,159 come to appreciate incredibly hard to do. But as soon as you you know, 564 00:37:52,280 --> 00:37:55,960 you want people to type something or write something, now you got to 565 00:37:55,960 --> 00:38:00,639 analyze it all, like what do you do? Oh? Use researchers are 566 00:38:00,679 --> 00:38:04,440 actually pretty good at this, but unfortunately it's in manual efforts, so scaling 567 00:38:04,480 --> 00:38:08,880 it up can be very time consuming for sure. That's another place where I've 568 00:38:09,239 --> 00:38:14,920 kind of loved this use case is to help analyze surveys or reviews or you 569 00:38:15,000 --> 00:38:19,320 know, Twitter complaints or whatever social media mentions that you have. So what 570 00:38:19,400 --> 00:38:22,199 I like to do is, you know, run all of these through Python 571 00:38:22,280 --> 00:38:24,679 scripts that I've developed and then try to understand what are those top teams, 572 00:38:24,840 --> 00:38:29,880 keywords and sentiments coming up. And while I will be the first to admit 573 00:38:29,960 --> 00:38:31,880 it's never going to be a hundred percent perfect, it's usually a really good 574 00:38:31,920 --> 00:38:36,760 starting point that then I can handle over to the user researchers and they'll get 575 00:38:36,800 --> 00:38:38,360 a lay of the land almost like Okay, you know, there are a 576 00:38:38,400 --> 00:38:42,039 lot of mentioning about us, or there's a lot of negative sentiment attached to 577 00:38:42,119 --> 00:38:45,440 this new version that we had. Let me go in and manually dig in 578 00:38:45,519 --> 00:38:49,199 a bit and see, you know, what are those more insightful things that 579 00:38:49,239 --> 00:38:52,800 I can come up with. So that's another you know, symbaetic relationship we 580 00:38:52,920 --> 00:38:57,239 have. It seems like a multiple choice survey kind of thing is the way 581 00:38:57,639 --> 00:39:00,880 a lot of people do that kind of stuff. But I mean there's there's 582 00:39:00,960 --> 00:39:04,800 problems with that too, right, Like what if you don't provide the right 583 00:39:04,840 --> 00:39:07,800 option, you know, and then you have other and now you give them 584 00:39:07,840 --> 00:39:12,159 an Irish text. Now you're back to text again. So what are some 585 00:39:12,239 --> 00:39:15,920 of the other problems with that approach, the multiple choice survey kind of thing. 586 00:39:15,800 --> 00:39:21,239 I think there's always the order that the options are presented in, right, 587 00:39:21,360 --> 00:39:22,920 especially if you're not really paying attention to the survey and you're just like 588 00:39:23,039 --> 00:39:25,679 it's like the first option and just you know, I want to quickly complete 589 00:39:25,679 --> 00:39:30,519 the survey, So that that is definitely a bias that can creep in as 590 00:39:30,559 --> 00:39:32,519 well, and if people just don't want to do it, they will just 591 00:39:32,639 --> 00:39:37,320 pick random things exactly exactly. Yeah, I think language really the way you 592 00:39:37,559 --> 00:39:42,960 write that option it is left to it's subject to interpretation. Right. What 593 00:39:43,119 --> 00:39:45,559 I might interpret as option it might not be what others interpret as OPTIONE. 594 00:39:46,000 --> 00:39:50,719 So having that, I always like having those open ended text responses because then 595 00:39:50,719 --> 00:39:52,440 the users can add in a lot more information. They can also give you 596 00:39:52,440 --> 00:39:55,480 a context that hey, you know, it makes sense if I'm doing X, 597 00:39:55,639 --> 00:39:58,920 y Z. But actually I would like to choose option B on a 598 00:39:59,000 --> 00:40:02,360 regular basis because I do another workflow way more often than I do option A. 599 00:40:04,920 --> 00:40:10,119 Brian McKay works for a company Roster, that does AI analysis of text 600 00:40:10,760 --> 00:40:15,440 of comments that people leave and determines whether they're relevant or not and actually turns 601 00:40:15,480 --> 00:40:20,119 them into data. And they're using GPT. Yes, yes, that is 602 00:40:20,239 --> 00:40:23,599 That is definitely a very interesting use cases. How do you identify if one 603 00:40:23,760 --> 00:40:29,400 this topic or this comment is relevant and two is this actionable? Right? 604 00:40:29,559 --> 00:40:31,320 Even if it was relevant, you could just have the comments saying this product 605 00:40:31,440 --> 00:40:34,719 is awesome, I love it, or this product sucks I hate it, 606 00:40:35,159 --> 00:40:37,920 but there's nothing really actionable from there. And oftentimes some of the best. 607 00:40:38,599 --> 00:40:43,360 Some of the best ideas for the products come from the users themselves because they 608 00:40:43,400 --> 00:40:45,840 are at the center of this. They use this day in and day out, 609 00:40:45,199 --> 00:40:47,000 and they are probably the ones that are going to say, hey, 610 00:40:47,199 --> 00:40:51,800 have you thought about this feature? Or your competitor already has this feature? 611 00:40:51,920 --> 00:40:53,679 You know, I would love to see this coming in here. Sort these 612 00:40:53,800 --> 00:41:00,119 comments by actionability. Yes, that's a cool concept, and there you like, 613 00:41:00,679 --> 00:41:02,440 that's I really wanted to get into this. Where would I use machine 614 00:41:02,519 --> 00:41:07,000 learning? What is the right thing to do? That's something? Sort these 615 00:41:07,079 --> 00:41:10,639 by action ability? How do you get the tool to assess that? But 616 00:41:10,719 --> 00:41:15,440 I guess this is what LM's in theory or could app Yeah, yeah, 617 00:41:15,599 --> 00:41:17,760 yeah, given given the right domain knowledge, because I think a lot of 618 00:41:17,880 --> 00:41:23,000 this, you know, enterprise products especially have so much technical juggy. Yeah. 619 00:41:23,519 --> 00:41:28,199 Domin knowledge is really really important, right, because maybe what means workflow 620 00:41:28,239 --> 00:41:30,639 to us, like in general language, might be a very very specific activity 621 00:41:30,719 --> 00:41:37,719 that's happened. Defining addiction what do we call it? A glossary of vocabulary 622 00:41:37,880 --> 00:41:39,599 right at the front of a project makes so much sense. I got bit 623 00:41:39,679 --> 00:41:44,079 by this once. I had a customer that we developed an app for. 624 00:41:44,199 --> 00:41:46,760 I'm not going to say who it was. But at the end of it, 625 00:41:46,960 --> 00:41:52,559 they basically said, well, where's the the feedback page. I think 626 00:41:52,599 --> 00:41:57,880 it was feedback or maybe it was a comment screen, right, and I'm 627 00:41:57,920 --> 00:42:00,360 thinking we did a fill out form, right, But what they meant by 628 00:42:00,400 --> 00:42:07,400 a comment screen was a live chat. And they were like, oh no, 629 00:42:07,840 --> 00:42:09,719 this we said this in the steck, we want to comment screen. 630 00:42:09,800 --> 00:42:14,920 I'm like, yeah, but that definition was never defined anywa. To them, 631 00:42:15,039 --> 00:42:17,039 it meant a live chat, which hack of a lot more software, 632 00:42:17,280 --> 00:42:21,760 a lot more than writing a comment and having a store exactly. Yeah, 633 00:42:22,000 --> 00:42:27,199 yeah, surprise, yeah, yeah. So when you say comment screen, 634 00:42:27,280 --> 00:42:30,760 what do you mean exactly? And it's it's one of those things that we 635 00:42:30,840 --> 00:42:34,920 all take for granted because you know, we thought comment everybody knows what that 636 00:42:35,079 --> 00:42:37,639 means. You know, you leave a comment or an it's a contact page, 637 00:42:37,679 --> 00:42:43,000 a contact page, contact page. So when I say contact page to 638 00:42:43,119 --> 00:42:45,599 you, it's fill in this form. We'll get back to your later, 639 00:42:45,719 --> 00:42:49,480 that's right, not a live chat. Yeah. So, hey, it 640 00:42:49,599 --> 00:42:52,360 pays to go over every fine detail when we're doing that. Well, I'm 641 00:42:52,360 --> 00:42:55,199 picking back to our comment at the top of the show with par about you 642 00:42:55,280 --> 00:42:59,559 know, watching people in the workflow, are actually going through the workflow. 643 00:42:59,719 --> 00:43:02,280 Yeah, Like if you've done that and you've gotten to the contact, yeah, 644 00:43:02,840 --> 00:43:05,920 it would have been and now we chat back and forth. You mean 645 00:43:06,320 --> 00:43:12,760 chat screen. Yeah, Oh well it's a different thing entirely. I certainly 646 00:43:12,840 --> 00:43:16,639 on my radar to do more shows around building a large language model sort of 647 00:43:16,679 --> 00:43:22,559 domain specific, because that seems to be we don't need existential conversation with software. 648 00:43:22,519 --> 00:43:28,079 The generalized ones are amusing. We can debate the value of search on 649 00:43:28,239 --> 00:43:32,559 that, but the idea that it understands a particular domain and can help organize 650 00:43:32,639 --> 00:43:37,599 large amounts of data. Effective summarizer, a prioritizer like those seem like very 651 00:43:37,840 --> 00:43:43,320 practical implementations of this. Being able to sort comments into ones that are actionable. 652 00:43:43,800 --> 00:43:47,239 That's a huge amount of time that you could really nail down. It's 653 00:43:47,480 --> 00:43:52,360 just an interesting metric of is it relevant, you know, does it talk 654 00:43:52,400 --> 00:43:55,159 about a product in a way, and then you know those are all elements 655 00:43:55,199 --> 00:44:00,239 that come down to this concept of actionable ability. Yeah, but it does 656 00:44:00,320 --> 00:44:02,199 mean we're going to be training our own I'm hoping the tooling gets better. 657 00:44:02,320 --> 00:44:06,360 Like it. This still seems hard to approach it, does it does? 658 00:44:06,519 --> 00:44:08,719 Yeah? And again it really goes back to the amount of data. How 659 00:44:08,800 --> 00:44:13,360 do you define actionability? Do you have a ground source of shoot that says, 660 00:44:13,679 --> 00:44:15,960 these are the comments that are actionable because you know, these keywords were 661 00:44:16,039 --> 00:44:20,400 used, or maybe they were really long comments, which again there's a correlation 662 00:44:20,559 --> 00:44:23,920 right to having longer comments has a lot of more information in there, versus 663 00:44:24,239 --> 00:44:29,679 these are the ones that seem actionable or maybe aren't even relevant. I think 664 00:44:29,679 --> 00:44:34,079 this one time I was doing some reviews analysis and there were those automated you 665 00:44:34,159 --> 00:44:37,039 know spambots, you know, putting in really random advertisms. I'm like, 666 00:44:37,280 --> 00:44:40,000 no, that is not relevant at all, and I don't even know why 667 00:44:40,039 --> 00:44:44,480 it's in the data set. So I think that's definitely going to be an 668 00:44:44,840 --> 00:44:47,840 interesting thing to deal with. Yeah, yeah, interesting times and yeah, 669 00:44:49,159 --> 00:44:51,760 definitely new tools emerging. Still, we're going to get better at this. 670 00:44:52,239 --> 00:44:58,440 All the more reason to buy pizza for the data team. Of course, 671 00:44:58,719 --> 00:45:02,079 I'm a big believer in you know, folk city together, and so anytime 672 00:45:02,159 --> 00:45:05,400 you can do this cross team, it's like, how do we help each 673 00:45:05,440 --> 00:45:08,679 other? Like you really got me thinking about the US team wanting to have 674 00:45:08,760 --> 00:45:13,159 access to more data before they act on those problems. I'd do the same 675 00:45:13,199 --> 00:45:17,000 thing, but with barbecue that total yea and vice versa. That you'll hope 676 00:45:17,000 --> 00:45:22,119 the data analytics team that's looking at telemetry and so forth are I think routinely 677 00:45:22,360 --> 00:45:28,320 just saying hey, here's what we're seeing today, because often they may not 678 00:45:28,480 --> 00:45:30,639 even it may not occur to them that what they're seeing is unusual. Yeah, 679 00:45:30,719 --> 00:45:34,639 so they just show it to the US to periodically say the US folks, 680 00:45:34,679 --> 00:45:37,400 here's how we see how your apps being used. Hope that fits with 681 00:45:37,519 --> 00:45:42,320 what your expectations were, because it might just kick off a conversation that finds 682 00:45:42,360 --> 00:45:45,639 something for you lose customers over it. Absolutely, and that toll triangulation of 683 00:45:45,760 --> 00:45:52,519 that qualitative plus quantitative or UX resocier or plus data scientists coming together and giving 684 00:45:52,559 --> 00:45:55,519 you those different plus spectives is so so powerful, right because it almost feels 685 00:45:55,599 --> 00:45:59,800 like as a data scientist or as UX resocial you're looking at a very small 686 00:46:00,039 --> 00:46:02,000 part of the picture. But then when you start talking to different teams and 687 00:46:02,119 --> 00:46:06,360 you know more, get that vital perspective, that's where you can see the 688 00:46:06,519 --> 00:46:09,159 entire picture of oh, okay, this is what's actually happening with our users 689 00:46:09,199 --> 00:46:13,519 in the product. Awesome. Yeah, so what's next for you? Have 690 00:46:13,639 --> 00:46:15,920 you done your talk yet? No? I am busy all day on Friday. 691 00:46:16,840 --> 00:46:20,599 That's what we wanted to do this on Wednesday. So because you've got 692 00:46:20,679 --> 00:46:22,880 a busy day on Friday, I looked at your schedules like, I don't 693 00:46:22,880 --> 00:46:25,599 know, did you lose a bet, Like you're working the whole day. 694 00:46:27,519 --> 00:46:30,840 Yes, I have no idea why that's been scheduled that way. But you 695 00:46:30,920 --> 00:46:34,400 know, you enjoy the conference for a while. I guess, I guess, 696 00:46:34,519 --> 00:46:38,119 yeah. I would rather get out of the way any other conference and 697 00:46:38,239 --> 00:46:43,239 then have more time. But well, it's Christmas. Has been great talking 698 00:46:43,280 --> 00:46:45,360 to you. Thank you some excellent ideas. I really appreciate it. Thank 699 00:46:45,360 --> 00:46:49,360 you so much. All right, and we'll talk to you next time. 700 00:46:49,840 --> 00:46:52,519 I'm dot net What is it? Oh? Yeah? Dot net rock Nice. 701 00:47:15,039 --> 00:47:17,960 Dot Net Rocks is brought to you by Franklin's Net and produced by Pop 702 00:47:19,119 --> 00:47:23,760 Studios, a full service audio, video and post production facility located physically in 703 00:47:23,840 --> 00:47:30,079 New London, Connecticut, and of course in the cloud online at pwop dot 704 00:47:30,199 --> 00:47:34,920 com. Visit our website at dt n et r o c ks dot com 705 00:47:35,440 --> 00:47:39,199 for RSS feeds, downloads, mobile apps, comments, and access to the 706 00:47:39,280 --> 00:47:44,880 full archives, going back to show number one, recorded in September two thousand 707 00:47:44,880 --> 00:47:47,280 and two, and make sure you check out our sponsors. They keep us 708 00:47:47,320 --> 00:48:00,079 in business. Now go write some code, see you next time. It 709 00:48:00,320 --> 00:48:01,239 is harder than my Texas