WEBVTT 1 00:00:00.320 --> 00:00:04.679 Hey, Carl and Richard here with your twenty twenty four NDC schedule. We'll 2 00:00:04.719 --> 00:00:09.160 be at as many NDC conferences as possible this year, and you should consider 3 00:00:09.199 --> 00:00:14.279 attending no matter what. The Copenhagen Developers Festival happens August twenty sixth through the 4 00:00:14.359 --> 00:00:21.519 thirtieth. Tickets at Cphdevfest dot com. NDC Porto is happening October fourteenth through 5 00:00:21.519 --> 00:00:27.399 the eighteenth. The early discount ends June fourteenth. Tickets at Ndcporto dot com. 6 00:00:27.440 --> 00:00:43.320 We'll see you there, we hope. Hey, gis what it's dot 7 00:00:43.399 --> 00:00:47.960 net rocks. I'm Carl Franklin and amateur campbell fishwes Lele is here with us 8 00:00:48.000 --> 00:00:51.000 and we're excited to talk to him. How are you doing, my friend, 9 00:00:51.359 --> 00:00:54.159 seen any hotters this year? Oh? Yeah, well, the otters 10 00:00:54.159 --> 00:00:57.560 are everywhere right. It's been summer up on the coast, so it's hard 11 00:00:57.560 --> 00:01:00.920 to be on happy up here. But we've got the kayak out, cleaned 12 00:01:00.960 --> 00:01:03.039 them up, did some repairs, all the things you need to do, 13 00:01:03.079 --> 00:01:04.239 and then you get to go, you know, meet the otters where they 14 00:01:04.239 --> 00:01:07.159 are. How's that? Do they taste good? Well? They think no, 15 00:01:07.319 --> 00:01:11.400 they don't. They smell and they smell terrible. But now, don't 16 00:01:11.400 --> 00:01:14.879 eat otter. That's that's not a good eating carnivores in general, not a 17 00:01:14.879 --> 00:01:18.120 good idea. That's a good idea. But no, they've got to stink 18 00:01:18.159 --> 00:01:19.840 all their own. Man Like, it's not hard to find an otter den 19 00:01:19.920 --> 00:01:25.760 you'll smell it first. Wow. Yeah, okay, I have no such 20 00:01:25.840 --> 00:01:32.000 things to report. But summer into your place means barbecue, right like the 21 00:01:32.040 --> 00:01:34.760 girls are running. I just bought two new twelve hundred watt subway first for 22 00:01:34.840 --> 00:01:38.680 the band. Well, he's that kind of guy. You guys have been 23 00:01:38.680 --> 00:01:41.359 playing up a tornado these days. I mean I only see it from your 24 00:01:41.400 --> 00:01:44.920 calendar. I could seep seeing every every Friday, every Saturday, blocked out 25 00:01:44.959 --> 00:01:49.040 through the summer. One of the guys that saw us at Ocean Beach just 26 00:01:49.120 --> 00:01:55.000 sent me like fifteen videos that he took of us playing wow out there. 27 00:01:55.560 --> 00:01:57.920 I don't know what to do with them. They're little clips. You know, 28 00:01:57.120 --> 00:02:00.599 you put them on TikTok, dude, that's what the hip kids do. 29 00:02:00.879 --> 00:02:06.359 I guess. Yeah, while it still exists, yeah, before while 30 00:02:06.359 --> 00:02:07.840 it's look, it's not going away. It's just a question of who's going 31 00:02:07.919 --> 00:02:12.080 to own it. Yeah, maybe it's the reality. All right, Well, 32 00:02:12.159 --> 00:02:15.759 I have something really funny and very cool from our friend Simon Crop for 33 00:02:15.919 --> 00:02:27.080 better no framework, roll the music. Alright, man, what do you 34 00:02:27.120 --> 00:02:30.439 got? I love Simon Simon's and Assi and he comes up with really great 35 00:02:30.479 --> 00:02:36.520 stuff. Every once in a while he crosses the line into comedy, which 36 00:02:36.919 --> 00:02:43.719 is also always helpful. But he's got this tool called waffle generator for the 37 00:02:43.719 --> 00:02:49.360 big waffles. Well, in a way produces text which on first glance looks 38 00:02:49.439 --> 00:02:55.000 like real ponderous pros replete with cliches. So this is like you know your 39 00:02:55.120 --> 00:03:00.000 lorem ipsem if you just want text to fill up the screen instead of using 40 00:03:00.360 --> 00:03:05.919 Latin random Latin? Is this lo ifs in the large language model era? 41 00:03:06.120 --> 00:03:08.319 Is that what this is? Oh? No, yeah, so this is 42 00:03:08.400 --> 00:03:13.120 great. Look at the example content on the Githovery boat. This stuff, 43 00:03:13.159 --> 00:03:20.560 I might say, the aesthetic of economico social disposition quote in this regard the 44 00:03:20.639 --> 00:03:24.439 underlying surrealism of the take home message should not divert attention from the esthetic of 45 00:03:24.680 --> 00:03:31.840 economic economic social disposition Humphrey Yokomoto in the Journal of the Total Entative Item. 46 00:03:32.360 --> 00:03:38.719 And then there's some on any rational basis A particular factor such as the functional 47 00:03:38.719 --> 00:03:44.159 baseline, the analogy of object, the strategic requirements, or the principle overriding 48 00:03:44.199 --> 00:03:50.840 programming, provides an interesting insight into the complementary functional derivation. This trend man 49 00:03:50.919 --> 00:03:58.319 dissipate due to the measurable proficiency complete and utter nonsense, But it sounds good. 50 00:04:00.280 --> 00:04:03.759 This is LLM degeneration, you know, toombs for a purpose. It's 51 00:04:03.800 --> 00:04:09.840 so awesome. So it's hilarious. Uh yeah, so he basically get this. 52 00:04:09.960 --> 00:04:13.199 He has a blazer wrapper around it, of course, that's what it's 53 00:04:13.240 --> 00:04:21.000 called blazing waffles. Blazing waffles. Oh my god, it's, oh, 54 00:04:21.120 --> 00:04:26.399 simon, so awesome, brilliantly awesome. If you need to make text. 55 00:04:26.720 --> 00:04:30.920 Yeah, he's also got an extension for bogus to use waffle generator, which 56 00:04:30.000 --> 00:04:34.079 is great. Okay, so good. He must be stopped though. Sometimes 57 00:04:34.439 --> 00:04:41.240 I got to ask him about this. What I'm usually the one he says 58 00:04:41.240 --> 00:04:46.560 stuff like that, he must be stopped. What do you think of that? 59 00:04:46.680 --> 00:04:54.800 Vishwaz Yeah, it's great, it's good. You know, whenever every 60 00:04:54.839 --> 00:04:58.680 time we get something that's both a functional tool that we're going to use and 61 00:04:58.839 --> 00:05:01.319 makes us laugh, it's like the perfect better no framework. It's a good 62 00:05:01.399 --> 00:05:04.240 day. It's a good day, every day, no question. All right, 63 00:05:04.279 --> 00:05:08.680 So who's talking to us? Richard grabbed a coma Hoff Shows nineteen oh 64 00:05:08.720 --> 00:05:11.720 four, the when we shot back at build with Mark Brown. We were 65 00:05:11.720 --> 00:05:15.120 talking about COSMOSDB and its role in large language models like chat, GPT and 66 00:05:15.240 --> 00:05:18.240 others, you know the role of AI and Cosmos DUB in general, and 67 00:05:18.279 --> 00:05:23.120 our friend Brent vander Mead had this comment. He said, oh yeah, 68 00:05:23.160 --> 00:05:29.079 thanks for the post traumatic event. At minute thirty two, Carl talks about 69 00:05:29.120 --> 00:05:31.560 trying to put together training on artificial intelligence and be patient for the tools coming 70 00:05:31.560 --> 00:05:33.639 out. Oh yeah, And Carl says, you know, this stuff is 71 00:05:33.680 --> 00:05:36.759 changing so fast, this is not the time. We just got to wait, 72 00:05:36.800 --> 00:05:40.319 wait it out and see what the major tools shake out like. And 73 00:05:40.399 --> 00:05:44.240 I said, it only gets easier. Carl said, you could imagine going 74 00:05:44.279 --> 00:05:46.120 into training class like right before Windows came out, and someone teaching you to 75 00:05:46.120 --> 00:05:50.079 write your own windowing system. YEP, exactly what I was thinking. And 76 00:05:50.120 --> 00:05:54.920 Brent, that was my point. That's why I said we shouldn't be doing 77 00:05:55.000 --> 00:05:58.160 this, just hang on. Yeah, and Brent comes back with I immediately 78 00:05:58.199 --> 00:06:01.920 was thrown back into setting up a three t your team foundations on premises server 79 00:06:02.079 --> 00:06:06.920 cluster. If only the rapid changes were occurring back then, and how people 80 00:06:06.959 --> 00:06:13.519 were willing to suffer setting up Jenkins pipelines for automation into AWS. Oh man, 81 00:06:13.759 --> 00:06:18.519 am I happy with aks Helm terraform as your DevOps and get hub actions. 82 00:06:18.879 --> 00:06:24.160 Thanks for the great show. I mean it's a great example. Think 83 00:06:24.199 --> 00:06:29.240 about how many shows we did talking about continuous integration back in the day, 84 00:06:29.319 --> 00:06:34.560 sort of building your own tool set every Now this bespoke pipeline and today it's 85 00:06:34.639 --> 00:06:39.759 set of services right, you turn them on right and you've got it set 86 00:06:39.800 --> 00:06:43.839 up even as I AD as infrastructure as code, so you're able to just 87 00:06:44.120 --> 00:06:46.079 really run a script. Is like players my pipeline, let's go. We 88 00:06:46.120 --> 00:06:50.480 set up CI for many people in many companies back in the day. Yeah, 89 00:06:50.560 --> 00:06:55.639 it was a whole business. You know, you're right the folks that 90 00:06:55.680 --> 00:06:58.879 would specialize in just getting that right for you so you didn't have to figure 91 00:06:58.879 --> 00:07:00.759 it out. I pretty sure you got a copy of music coe by, 92 00:07:00.879 --> 00:07:03.600 but we'll hook you up with something cool. Give me a ring at Richard 93 00:07:03.639 --> 00:07:06.240 Aguapp dot com. Thanks you so much for your comment and a copy of 94 00:07:06.319 --> 00:07:09.399 music Cobe is on its way to you, and if you'd like a copy 95 00:07:09.399 --> 00:07:12.000 of music, co buy I read a comment on the website at donnat Rocks 96 00:07:12.040 --> 00:07:14.560 dot com or on the facebooks. We publish every show there, and if 97 00:07:14.560 --> 00:07:15.680 you comment there and I read on the show, we'll send your copy of 98 00:07:15.720 --> 00:07:19.000 music go by very good. And you know you can always follow us on 99 00:07:19.160 --> 00:07:23.720 ex Twitter as I call it. But we've been there for many years. 100 00:07:23.759 --> 00:07:27.920 But the cool kids are hanging out. I'm masked on I'm at Carl Franklin 101 00:07:27.959 --> 00:07:30.639 at tech Hub dot social, and I'm Ridge Campbell at mass it. I'm 102 00:07:30.680 --> 00:07:34.079 dot social. Send us a two Rudy two two Rudy two dy fresh and 103 00:07:34.120 --> 00:07:42.360 fruity and uh weird. The last time vishwas Lele was on. We had 104 00:07:42.519 --> 00:07:47.000 such a great conversation and I was under the impression at no. I said 105 00:07:47.079 --> 00:07:49.360 to him, you were on just like a month or two ago. It 106 00:07:49.439 --> 00:07:54.319 was nine months ago. Yeah, yeah, I don't feel nine months older. 107 00:07:54.879 --> 00:07:58.600 But obviously I am been pretty crazy. And first show is all the 108 00:07:58.639 --> 00:08:01.079 way back in two thousand and seven. I like he's been Oh yeah, 109 00:08:01.079 --> 00:08:05.000 he's an og pretty sure we owe my sub sandwich at this point or something. 110 00:08:05.079 --> 00:08:09.879 Yeah, so his title has changed. He is now co founder in 111 00:08:11.040 --> 00:08:16.040 CEO at pwin dot Ai. He's also a noted industry speaker and author, 112 00:08:16.240 --> 00:08:22.160 and the regional director for Washington DC. Welcome back, vishwas, thank you 113 00:08:22.240 --> 00:08:26.360 both. Glad to be back a new job. Dude, what are you 114 00:08:26.519 --> 00:08:31.279 doing. You've been AIS CTO for ten fifteen years? Yeah, almost sixteen 115 00:08:31.360 --> 00:08:39.799 years. I've worked at AIS for almost thirty years, So quite a big 116 00:08:39.919 --> 00:08:43.960 change here since the dast But it does seem to me like you're not alone 117 00:08:43.159 --> 00:08:48.240 either. Some really great tech people I know are jumping into AI startups like 118 00:08:48.360 --> 00:08:52.759 it's the hip thing to do right now. It's such a massive opportunity. 119 00:08:52.000 --> 00:08:56.159 Yeah, so, Richard, I was not going to be started on this 120 00:08:56.279 --> 00:09:00.960 idea. I was not envisioning. If you asked me like nine months ago, 121 00:09:01.120 --> 00:09:07.840 what was your plan. The plan was whenever a new technology like Azure, 122 00:09:07.879 --> 00:09:11.840 open ai or open ai comes along, We've always had this culture of 123 00:09:11.200 --> 00:09:15.879 let's can we find a few people jump into it, try to understand it, 124 00:09:15.879 --> 00:09:20.039 try to build something so that we can understand it better and take it 125 00:09:20.080 --> 00:09:22.240 to our customers. I mean, that's what we're planning to do. And 126 00:09:22.960 --> 00:09:26.879 with Azure open Ai, we went through a bunch of use cases and then 127 00:09:26.960 --> 00:09:33.000 said, hey, can we work on a use case which is can help 128 00:09:33.159 --> 00:09:37.159 our internal folks. We have a pretty sizable proposal team, so it can 129 00:09:37.200 --> 00:09:41.120 we build something that can help you. So we build something, and then 130 00:09:41.200 --> 00:09:43.399 we took it out to other people and one thing led to another and here 131 00:09:43.440 --> 00:09:48.480 we are so really good because you were making things that people wanted. It's 132 00:09:48.519 --> 00:09:52.279 like this is actually a product and we should do more of it. Yeah. 133 00:09:52.320 --> 00:09:54.639 So what we did, Richard, was first we build an m VP 134 00:09:54.799 --> 00:10:01.240 for our internal teams and got some feedback from them, some good, some 135 00:10:01.440 --> 00:10:03.799 bad, all of it. And then we said, okay, let's do 136 00:10:03.919 --> 00:10:11.480 this. Can we before we build any more functionality into this, can we 137 00:10:11.480 --> 00:10:13.759 go to eight or ten companies? Ended up going to a dozen or so 138 00:10:13.879 --> 00:10:18.840 companies and said, would you allow us to generate some content for you and 139 00:10:18.080 --> 00:10:22.120 then we'll talk about this later. But this is a co pilot for proposal 140 00:10:22.159 --> 00:10:26.279 writers. So we went to a bunch of companies and said, can we 141 00:10:26.320 --> 00:10:30.440 generate the first draft of a proposal for you? Maybe eight or ten companies. 142 00:10:30.919 --> 00:10:35.559 It's going to be called the Aesthetic of Economico Social Disposition. You're going 143 00:10:35.600 --> 00:10:45.120 to love it, so sorry, that's great, dash waffle waffle false. 144 00:10:46.879 --> 00:10:52.039 So we we said, okay, we took it to eight or ten companies 145 00:10:52.080 --> 00:10:56.279 and they got some feedback. And what I learned from it, Richard, 146 00:10:56.360 --> 00:10:58.000 to your point, A lot of people are jumping in, and as they 147 00:10:58.039 --> 00:11:03.039 should. This is definitely a disruptive technology. Can I can feel it from 148 00:11:03.399 --> 00:11:07.159 early on in the share Point days you remember two thousand and seven show Getting 149 00:11:07.200 --> 00:11:11.240 into share Point, or remember the cloud show early days. This seems like 150 00:11:11.399 --> 00:11:16.039 much much bigger than all of those things combined. And there's this idea that 151 00:11:16.320 --> 00:11:22.240 you hear technology that can reason on your behalf. Forget the content generation pieces 152 00:11:22.320 --> 00:11:24.559 right, forget that can summarize text for you, but the fact that it 153 00:11:24.559 --> 00:11:28.639 can reason. Given some things, it can reason and give you an output 154 00:11:28.679 --> 00:11:33.679 back. Think about it as applicability all across what we do in it. 155 00:11:33.279 --> 00:11:39.360 But we took one use case and maybe in hindsight, if I think about 156 00:11:39.399 --> 00:11:43.799 it, you can do a lot bit generative AI. But the generative AI 157 00:11:43.919 --> 00:11:48.879 models are hard to sort of control their good But how do you get them 158 00:11:48.919 --> 00:11:56.879 to focus in and generate one twenty thirty forty page document and that has an 159 00:11:56.960 --> 00:12:01.360 ROI associated with that because you know come, sure are time consuming to build? 160 00:12:01.639 --> 00:12:03.519 Yes, they are time consisting to build. So I think, in 161 00:12:03.639 --> 00:12:11.440 hindsight, picking one use case which aligned well with the superpower of generative AI, 162 00:12:11.519 --> 00:12:16.639 which is to generate content, but then to solve a problem where people 163 00:12:16.639 --> 00:12:22.799 were spending struggling writing this content and then generating the first draft. And it 164 00:12:22.879 --> 00:12:26.120 is important to note that it is a co pilot very much using the term 165 00:12:26.200 --> 00:12:30.600 copilot. Yeah, so you're still the pilot. It's your fault. It's 166 00:12:30.600 --> 00:12:35.600 still your fault. Yes, we're just getting you to this first draft. 167 00:12:35.679 --> 00:12:39.960 And later on in the show, I can talk about as a CTO, 168 00:12:39.000 --> 00:12:43.639 as an architect, I have supported proposal writing for at least two decades, 169 00:12:45.000 --> 00:12:48.720 right, and I can talk about all of the trouble and trial and tabulation 170 00:12:48.919 --> 00:12:54.919 supporting that, and then how that product helps with all of those challenges that 171 00:12:54.960 --> 00:12:58.600 the organizations face. So let us go off. I mean, I'm all 172 00:12:58.639 --> 00:13:01.840 about the summarization only existing content, like I get that from a large language 173 00:13:01.840 --> 00:13:07.720 model. But you talked about reasoning. Yes, where is the reasoning needed 174 00:13:07.879 --> 00:13:13.639 and how does the tool do? Yeah, so the reasoning is needed because 175 00:13:16.600 --> 00:13:20.399 let's just literature Defied's okay, I'll take a step back and sort of describe 176 00:13:20.440 --> 00:13:24.360 the problem and then we can talk about how we solved it. Right. 177 00:13:24.879 --> 00:13:30.240 So, I remember nine months ago as you were ending that show. At 178 00:13:30.240 --> 00:13:33.000 that time M three sixty five Copilot had not come out or it was a 179 00:13:33.080 --> 00:13:37.039 yeah, it was being talked about. I think it available in a limited 180 00:13:37.120 --> 00:13:41.759 review or something like that. And Richard, I remember vividly that you asked 181 00:13:41.799 --> 00:13:48.440 a question that if you're generating the content based on your past responses, right, 182 00:13:50.080 --> 00:13:54.919 and let's say you've submitted twenty thirty forty RFP submissions in the past, 183 00:13:56.279 --> 00:14:00.159 you stored them into a SharePoint site. Perhaps now you're M three sixty if 184 00:14:00.159 --> 00:14:07.799 I have co pilot has now scanned those documents, and you're now able to 185 00:14:07.879 --> 00:14:09.600 go and say, hey, generate me an answer to this question. And 186 00:14:11.159 --> 00:14:15.759 you at that point said, why would I not use M three sixty five 187 00:14:16.080 --> 00:14:20.840 co pilot for that? Right? Why do you need this special purpose co 188 00:14:20.960 --> 00:14:24.559 pilot to do what N three sixty five copilot seems to be doing out of 189 00:14:24.559 --> 00:14:28.759 the box That that was a question proposed to do. Yeah, nobody really 190 00:14:28.799 --> 00:14:33.360 knew at that point, right, And the answer to that question is that 191 00:14:33.559 --> 00:14:39.440 M three sixty five Copilot is a very good general purpose chat You give it 192 00:14:39.519 --> 00:14:43.559 tons and tons of content, could be any content in your share point one 193 00:14:43.720 --> 00:14:46.600 drive teams, what have you. And now you can use the graph connector 194 00:14:46.600 --> 00:14:50.440 to pull in content from anywhere else you want to putting data from Salesforce, 195 00:14:50.440 --> 00:14:54.279 you can do that. But the fact is that it is a general purpose 196 00:14:56.679 --> 00:15:01.039 chat bot where the content is broken up. And let's say there's some sort 197 00:15:01.039 --> 00:15:05.120 of a chunking strategy. You chunk it by the page numbers, or you 198 00:15:05.200 --> 00:15:09.120 chunked it by the discussion, or you chunk it by sections. You chunk 199 00:15:09.159 --> 00:15:13.039 it some way. You store it in a vector database, and there are 200 00:15:13.080 --> 00:15:16.519 lots and lots of them available. But then you are asking a question that 201 00:15:16.679 --> 00:15:22.840 Hey, based on the information you have, can you answer this question right? 202 00:15:22.240 --> 00:15:26.440 And it goes about answering that question. I keep in mind that chat 203 00:15:26.480 --> 00:15:31.799 bought including chat gpt our System one chat butt And what do I mean by 204 00:15:31.840 --> 00:15:37.399 system one. I'm really talking about the Daniel famous book from Daniel Canman System 205 00:15:37.399 --> 00:15:43.320 one system I'm exactly I'm talking about, right, and it is can you 206 00:15:43.320 --> 00:15:48.240 elaborate on that visual lot of it? Yeah, So there's a famous book 207 00:15:48.600 --> 00:15:56.080 by by Nobel Laureate Daniel Canman called Thinking Fast and Slow. Highly recommend folks 208 00:15:56.120 --> 00:16:00.279 read it very good. Book really talked about two things. And this book 209 00:16:00.360 --> 00:16:06.080 came out many many years ago, well before CHATGBT was a thing. But 210 00:16:06.720 --> 00:16:10.320 in this book they talk about system one and system to system one. Think 211 00:16:10.320 --> 00:16:17.000 of that as the way your brain works that allows you to avoid a ditch. 212 00:16:17.440 --> 00:16:19.080 Right, You don't have to think too much, You instinctively take a 213 00:16:19.320 --> 00:16:26.519 reaction. System two is when you're thinking about reptile brain exactly. So chat 214 00:16:26.559 --> 00:16:32.320 GPT and other chatbots today are system one. Right, you can go up 215 00:16:32.360 --> 00:16:34.759 to chat GPT and say, I'm going to ask you a question, but 216 00:16:34.879 --> 00:16:37.960 this is a really detailed question. I want you to think about it for 217 00:16:38.000 --> 00:16:41.759 thirty minutes and then only respond. There's no way to do that. It 218 00:16:41.799 --> 00:16:48.279 starts responding in three seconds or whatever the latency is. And so when we 219 00:16:48.360 --> 00:16:55.440 talk about a special purpose chatbot in this case, in the case of proposals, 220 00:16:55.919 --> 00:16:59.840 you really have to think about the proposal domain. Think about the important 221 00:17:00.120 --> 00:17:04.279 entities in the proposal domain. As you look through past documents, you need 222 00:17:04.359 --> 00:17:10.839 to collate them in a certain entities which are very domain specific. And then 223 00:17:11.279 --> 00:17:15.839 generating a response to a question. There are some best practices that are available 224 00:17:15.880 --> 00:17:22.759 out there in terms of how do you write persuasive content right and how do 225 00:17:22.880 --> 00:17:29.160 you reason about So these questions can be long form questions, open ended questions 226 00:17:29.160 --> 00:17:34.960 in RFPs, like we have this problem about modernization and this system needs to 227 00:17:36.000 --> 00:17:38.799 be modernized, and these are the challenges we are running into. It could 228 00:17:38.799 --> 00:17:41.759 be anything, right, I'm just giving you an IT example, could be 229 00:17:41.720 --> 00:17:45.960 a healthcare example, could be something else. So these are questions that you 230 00:17:47.000 --> 00:17:49.640 have to respond to. Their very broad based questions. But then there are 231 00:17:49.720 --> 00:17:56.000 best practices in terms of how you write persuasive text in responding to that question. 232 00:17:56.119 --> 00:18:06.279 And that's where the reasoning comes in. Where our our domain specific copilot 233 00:18:07.200 --> 00:18:12.960 understands some of these best practices. When it ingest the content, it automatically 234 00:18:14.119 --> 00:18:19.920 organizes them based on the entities things like that I can I can make an 235 00:18:19.920 --> 00:18:26.799 analogy here if you go to chat GPT to ask it for a problem and 236 00:18:26.880 --> 00:18:32.880 how to configure this middleware or whatever. Right, you're you're taking a chance, 237 00:18:32.960 --> 00:18:37.079 whether it's going to assume that you're using startup CS or program cs right 238 00:18:37.160 --> 00:18:41.480 a dot net six and before version of asp net core, or you know, 239 00:18:41.960 --> 00:18:45.400 or the new, new or modern stuff. And if it understood this, 240 00:18:45.559 --> 00:18:48.920 there would be another layer before it started spitting this stuff out that said, 241 00:18:49.000 --> 00:18:53.839 oh, we have this contact that says it should know we were using 242 00:18:55.119 --> 00:18:59.079 dot net eight and we're using the latest version of see sharp, and we 243 00:18:59.160 --> 00:19:03.359 want to you know, frame the answer in that context. Right. So 244 00:19:03.440 --> 00:19:08.039 that's setting up a system prompt and making sure that chat, GPT or any 245 00:19:08.079 --> 00:19:15.079 language model has the right context so that it can give you an answer that 246 00:19:15.200 --> 00:19:19.680 is very much aligned with your background and your needs. I'm also talking about 247 00:19:21.119 --> 00:19:26.319 KYLA. I'm also talking about a scenario where in some cases you're trying to 248 00:19:26.359 --> 00:19:30.920 answer a question. So let's just take that t T example, and I 249 00:19:30.920 --> 00:19:33.559 think that'll appeal to the listeners here as well. So there's a question about 250 00:19:34.000 --> 00:19:41.200 somebody, some government agency or some enterprise issued in RFP with a question about 251 00:19:41.400 --> 00:19:45.519 modernization of some data platform, right, and they talked about, hey, 252 00:19:47.720 --> 00:19:52.279 this platform needs to be modernized. How do you go about have you done 253 00:19:52.279 --> 00:19:55.599 this kind of work before? How do you go about solving this problem? 254 00:19:55.759 --> 00:20:00.160 Again, I'm over simplifying this, but let's say that's starf right. So 255 00:20:00.200 --> 00:20:03.799 in this case, we would have to go find out if you have done 256 00:20:03.880 --> 00:20:07.599 this kind of work as an organization, right, and then we will have 257 00:20:07.640 --> 00:20:15.599 to search through your past RFP responses, your core competencies. Maybe you've written 258 00:20:15.599 --> 00:20:21.720 some white papers, you've written blogs, you have transcriptions of podcasts and things 259 00:20:21.799 --> 00:20:25.920 like that. We go look through that, and that's a common RAG pattern 260 00:20:25.960 --> 00:20:27.839 that I'm talking about that people are familiar with. But we have to go 261 00:20:27.880 --> 00:20:32.160 find a couple of examples, or maybe three examples of where we have done 262 00:20:32.279 --> 00:20:38.799 this kind of data modernization. So you get that text which says, well, 263 00:20:38.799 --> 00:20:44.440 these three examples match somewhat the requirement of the RFP. But then the 264 00:20:44.480 --> 00:20:48.400 reasoning comes in, how good a match is this? Right? Should we 265 00:20:48.519 --> 00:20:52.680 use this as an example? If it is not a good match, should 266 00:20:52.680 --> 00:20:56.799 we be changing our prompt and looking for a different kind of an answer. 267 00:20:56.079 --> 00:21:02.960 So all of that is happening dynamically in run time, and we are not 268 00:21:03.119 --> 00:21:07.079 asking our users to write any of the proms, right. We are essentially 269 00:21:08.559 --> 00:21:14.880 generating all of the prompts for them based on the best practices that I talked 270 00:21:14.920 --> 00:21:18.400 about. Right, And there's an interesting inverted ratio that we have learned about. 271 00:21:18.519 --> 00:21:22.200 So if we're generating a twenty page document, in many cases we end 272 00:21:22.279 --> 00:21:27.359 up generating two hundred pages. That's the inverted ratio of proms, which includes 273 00:21:27.400 --> 00:21:30.400 all of the prompts and the context that you're feeding into the language model. 274 00:21:30.519 --> 00:21:34.680 I mean, that's not that weird when it comes to a good search string. 275 00:21:34.799 --> 00:21:37.759 Even like when you focus in on a specific thing, you need to 276 00:21:37.799 --> 00:21:44.000 know you have to be very descriptive to get the answer. That's right. 277 00:21:44.079 --> 00:21:47.240 That's a good analogy. Yeah, I mean, I think it's useful for 278 00:21:47.240 --> 00:21:49.319 folks t als to understand. You're in Washington, DC, like you guys 279 00:21:49.319 --> 00:21:52.880 are in the business of writing RFPs for government, which tend to be lengthy 280 00:21:53.559 --> 00:21:59.680 and have very specific rules, like it's not a trivial thing, it is 281 00:21:59.680 --> 00:22:04.400 not a real thing. There is a multi hundred page document called the Federal 282 00:22:04.480 --> 00:22:11.519 Acquisition Regulation which talks about all of the details of an RFP. What would 283 00:22:11.599 --> 00:22:15.519 go in section M, what would go in section L, Section C, 284 00:22:15.240 --> 00:22:18.839 things like that, so the rules around that. And I mean, I 285 00:22:18.839 --> 00:22:23.000 think a language mode would be good at given this rule set and this goal 286 00:22:23.279 --> 00:22:29.680 and these capabilities, write me the paragraphs right, and what we are finding 287 00:22:29.759 --> 00:22:33.799 is as good as these language models are. Just like human beings, if 288 00:22:33.839 --> 00:22:36.799 you tell them twelve things to do. Yeah, they will do the first 289 00:22:36.880 --> 00:22:40.279 second, and then we will do the eleventh and twelfth and forget everything everything 290 00:22:40.319 --> 00:22:44.440 in between. That's so, is that interesting? It's very interesting, But 291 00:22:44.480 --> 00:22:48.680 it does sound like the in building this RFP generator, you literally want to 292 00:22:48.720 --> 00:22:55.000 go section to section and write a separate prompt for each one. That's effectively. 293 00:22:55.680 --> 00:22:57.799 Again, because the rule set's so consistent, you should be able to 294 00:22:57.880 --> 00:23:02.880 generate almost all of this, right, right, So let me actually take 295 00:23:02.920 --> 00:23:07.720 you back into the dane in life of a technical architect or an engineer like 296 00:23:07.799 --> 00:23:11.359 me who is supporting RFPs, and then we'll come back to this point. 297 00:23:11.400 --> 00:23:15.559 So I'll try to be very quick here. So let's say an RFP arrives. 298 00:23:15.599 --> 00:23:18.839 It's a fifty sixty or one hundred page PDF which is montrivial to read. 299 00:23:18.920 --> 00:23:22.039 Somebody has to read it. This is the request for you to propose 300 00:23:22.079 --> 00:23:27.119 to them. This is the request. Your proposal will be larger, although 301 00:23:27.160 --> 00:23:30.599 in some cases they want you to write it. They have very strict page 302 00:23:30.640 --> 00:23:33.440 limits into how much you can write, right, don't know. They can 303 00:23:33.480 --> 00:23:37.759 send you as many pages as they want, but you can only send them. 304 00:23:37.960 --> 00:23:41.480 You can only and then you know, all the tricks about you know, 305 00:23:41.640 --> 00:23:45.799 creating really dense diagrams to sort of save on words comes into play. 306 00:23:45.839 --> 00:23:51.200 But let's just say you've got a fifty page complicated RFP. Now some human 307 00:23:51.240 --> 00:23:56.119 being has to read that and understand what is happening in this RFP, and 308 00:23:56.160 --> 00:23:59.240 then let's say just spend a day or two to figure out. And then 309 00:23:59.279 --> 00:24:03.640 the organize is a meeting where they invite different people. There's a proposal manager, 310 00:24:04.119 --> 00:24:11.079 there's a technical sme there is account team. Right, there is no 311 00:24:11.240 --> 00:24:18.279 point in responding to an RFP if you don't know anything about that customer. 312 00:24:18.440 --> 00:24:21.400 That is a general rule. That's not just in government. That's a general 313 00:24:21.480 --> 00:24:23.839 rule. Right. Why waste you know, there's no price for the second 314 00:24:23.839 --> 00:24:30.359 and third and fourth place based on your response, So you have to read 315 00:24:30.400 --> 00:24:37.279 that. You organize a meeting and they present what is requested in the RFP, 316 00:24:37.400 --> 00:24:41.480 and then somebody like me, an engineer, would join that meeting and 317 00:24:41.519 --> 00:24:45.279 say, yes, we can do this. This is SharePoint related, this 318 00:24:45.400 --> 00:24:48.279 is records management. Oh, we will use this tool and we will architect 319 00:24:48.359 --> 00:24:53.559 these pieces. And somebody will say fine, you go off, and the 320 00:24:53.599 --> 00:24:59.359 other architects go off and start writing the first draft. Now I don't want 321 00:24:59.440 --> 00:25:04.240 to write anything because I'm already busy with my day job. And they give 322 00:25:04.240 --> 00:25:07.880 me two weeks. I go off and start writing, and turns out that 323 00:25:07.920 --> 00:25:11.519 I don't know about other my colleagues. I'm not a good writer. So 324 00:25:11.559 --> 00:25:17.240 two weeks go by, and then we're expected to show back, come back 325 00:25:17.279 --> 00:25:21.279 to this meeting with what we have written. And that's when the yelling begins, 326 00:25:21.319 --> 00:25:25.880 because you know they don't like what we have written. Everybody has written 327 00:25:25.920 --> 00:25:30.000 it differently, and you know, of course I am over emphasizing on all 328 00:25:30.039 --> 00:25:33.960 of the projects that I've worked on, and the same is true with my 329 00:25:34.039 --> 00:25:40.000 colleagues, and there's a rich set of experienced library, but I'm only focused 330 00:25:40.039 --> 00:25:41.480 on the ones I've worked on, right, because that's what I know best. 331 00:25:42.039 --> 00:25:48.319 So now two weeks have gone by, and really a draft is there, 332 00:25:48.359 --> 00:25:52.599 which is like poor quality draft, which reads like ten people have written 333 00:25:52.640 --> 00:25:56.359 it. And now you've only two more weeks left to the final submission. 334 00:25:56.799 --> 00:26:03.319 Right. And it is often said that proposals are one either at the beginning 335 00:26:03.400 --> 00:26:06.480 or then. What they mean is either at the beginning, you understand that 336 00:26:06.519 --> 00:26:10.039 you're going to win, you put the right team in place, or at 337 00:26:10.119 --> 00:26:14.559 the end, if you have enough time left, you want to take a 338 00:26:14.559 --> 00:26:19.000 step back and think about what should be a creative pricing strategy. Who else 339 00:26:19.039 --> 00:26:23.039 would be bidding this right? How can I ghost my competition? If you 340 00:26:23.079 --> 00:26:26.799 do those things, your chances of winning goes up. But the sad reality 341 00:26:26.839 --> 00:26:32.440 of proposal writing is you spend most of the time in the middle, chasing 342 00:26:32.480 --> 00:26:37.079 the first draft, chasing the second draft. So the value proposition for the 343 00:26:37.119 --> 00:26:41.319 tool, and especially designed for people like me, you have taken all of 344 00:26:41.359 --> 00:26:45.559 your past performances, stored them in a library, and then when the RFP 345 00:26:45.720 --> 00:26:49.960 arrives, our engine, peven engine, will take a look at the RFP, 346 00:26:52.400 --> 00:26:56.799 parse it down into what is exactly being asked, and then rather than 347 00:26:56.839 --> 00:27:03.359 having to write ten pages of pros and return it, you're really dealing at 348 00:27:03.400 --> 00:27:07.519 a higher level. You're providing hints to our model. We call it flight 349 00:27:07.559 --> 00:27:12.720 plan and very creative. Right if you're calling ourselves copilot, a flight planner 350 00:27:12.799 --> 00:27:19.079 seems to be logical. So there, so you set up a flight plan, 351 00:27:19.119 --> 00:27:23.359 which is really and this is Richard and cal This is where I think 352 00:27:23.400 --> 00:27:26.759 this is a very key concept in my mind. Again, looking back, 353 00:27:26.799 --> 00:27:32.319 it seems trivial, but this is one of the concepts that we landed on 354 00:27:32.400 --> 00:27:36.079 earlier on that has served as well. So if you think of calling yourself 355 00:27:36.079 --> 00:27:38.880 a co pilot, which means we are not here to replace the proposal writers. 356 00:27:40.000 --> 00:27:44.319 You know, these people have gained experience and skill or many many years. 357 00:27:45.039 --> 00:27:47.960 We are here to augment them. And how do they work with the 358 00:27:48.039 --> 00:27:52.920 model without becoming a prompt experts themselves or without becoming experts in a RAG pattern. 359 00:27:53.880 --> 00:27:56.400 What we ask them to do is we ask them to fill out a 360 00:27:56.400 --> 00:28:03.240 flight plan, essentially tell us in great how should we be responding. So 361 00:28:03.440 --> 00:28:07.960 these are the solutions we want to incorporate. Here are the win themes. 362 00:28:07.480 --> 00:28:11.480 We think we are differentiated because we are the best Microsoft partner in the world. 363 00:28:12.279 --> 00:28:15.559 What are the pain points? If you really know the customer, you 364 00:28:15.599 --> 00:28:22.559 should know their motivation for issuing this RFP, So you provide that vision in 365 00:28:22.599 --> 00:28:26.279 the form of a flight plan, and then our engine will take that flight 366 00:28:26.400 --> 00:28:33.160 plan and then generate these prompts dynamically as we talked about, and then generate 367 00:28:33.000 --> 00:28:38.200 a twenty thirty forty page response. And why twenty thirty forty pages, because 368 00:28:38.200 --> 00:28:42.119 that's what's been asked, And that was the other innovation. If you go 369 00:28:42.200 --> 00:28:48.200 to any other co pilot, you'll end up getting one or two pages. 370 00:28:48.200 --> 00:28:49.720 Even chat GPT, if we ask a complex question, it will say, 371 00:28:49.759 --> 00:28:52.519 hey, this is a complex question. Let me just tell you how you 372 00:28:52.559 --> 00:28:56.759 can get to this answer. In our case, we can't tell our users 373 00:28:56.920 --> 00:29:00.559 how you can get to an answer. They want to see twenty pages. 374 00:29:00.039 --> 00:29:04.680 So we have two. Like I was saying earlier, these language models, 375 00:29:04.680 --> 00:29:08.319 as good they are, they tend to forget things in the middle. So 376 00:29:08.359 --> 00:29:12.279 it's our job to take two requirements, give it to the language model, 377 00:29:15.079 --> 00:29:19.160 get them to generate something, and evaluate it at the end of that and 378 00:29:19.200 --> 00:29:22.440 see if they really answered the question. If not, send it back again, 379 00:29:22.519 --> 00:29:26.359 send it back again. Now you have section one, you do that 380 00:29:26.400 --> 00:29:30.319 for section two, section three instead of you do that repeatedly. And that's 381 00:29:30.359 --> 00:29:33.799 why people ask me, why is it taking you guys four or five hours 382 00:29:33.839 --> 00:29:38.519 to generate a twenty page document? And when I ask a question of chat 383 00:29:38.559 --> 00:29:41.279 GPT, it gives me an answer in seconds. Why is it taking you 384 00:29:41.319 --> 00:29:47.000 four or five hours? Well, the answer is that we're having to iterate 385 00:29:47.160 --> 00:29:51.880 over the content that's being developed, and at every step of the way we 386 00:29:51.920 --> 00:29:55.519 are evaluating it for the sake of completeness. And by the way, we 387 00:29:55.599 --> 00:30:00.960 also do hallucination checks as the content is being generated, right, you have 388 00:30:00.039 --> 00:30:07.680 to without that human to check things out. Customers can't just expect you to 389 00:30:07.720 --> 00:30:12.000 plug stuff in and copy and paste out of a chat bot, you know, 390 00:30:12.279 --> 00:30:17.480 right, right, So I agree hallucination checks up. That's why very 391 00:30:17.480 --> 00:30:21.279 important. And you know, going back to my example, Carl, the 392 00:30:21.400 --> 00:30:26.319 RFP is asking for companies who have done a five thousand or fifty fifty thousand 393 00:30:26.400 --> 00:30:33.359 notes sorry, migration to the cloud. And that's what the and now the 394 00:30:33.400 --> 00:30:37.640 text that was generated says you have done five thousand note migration. Yeah, 395 00:30:37.799 --> 00:30:41.000 we can't find a reference to a five thousand note migration in your entire past 396 00:30:41.079 --> 00:30:45.319 history. Say, hey, you may want to go double check this. 397 00:30:45.160 --> 00:30:52.119 Well, how effective has it been in generating proposals? So well, you 398 00:30:52.119 --> 00:30:57.200 should ask our customers that. But one, well sounds like they're happy but 399 00:30:57.359 --> 00:31:00.759 happy with it. Yeah, I think I'll lot more work needs to be 400 00:31:00.799 --> 00:31:06.839 done, obviously. Language models building, Richard, Building a production grade application 401 00:31:06.920 --> 00:31:11.960 with language models is hard because they have their own personality. When Microsoft goes 402 00:31:11.000 --> 00:31:17.359 from four to four turbo to four omni. There's some personality aspect of these 403 00:31:17.440 --> 00:31:19.519 language models can change, and we have to account for that, especially if 404 00:31:19.519 --> 00:31:25.000 you're generating a multi layer prompt, you have to account for that. So 405 00:31:25.160 --> 00:31:30.759 take some additional time. But so far we have had customers who said that 406 00:31:30.799 --> 00:31:34.359 we've accelerated their timelines, given them more time. We said, you reached 407 00:31:34.400 --> 00:31:38.200 out to a dozen customers, were they all pretty happy with what they got? 408 00:31:38.559 --> 00:31:44.039 So I didn't complete the story. We built an MVP. We went 409 00:31:44.079 --> 00:31:47.559 to a dozen customers and said, we just want you to try our product. 410 00:31:47.680 --> 00:31:52.400 We didn't even have a UI at that point, right, And because 411 00:31:52.480 --> 00:31:56.000 what's the point of the UI if the content is not of the quality where 412 00:31:56.039 --> 00:32:00.519 it is helping you with saving time and improving the outcomes. Right, So 413 00:32:00.559 --> 00:32:05.839 we went to them, got some really good feedback, added more capabilities, 414 00:32:06.160 --> 00:32:08.599 got the UI, and of course our UI is SharePoint because it turns out 415 00:32:08.599 --> 00:32:14.039 that eighty to ninety percent of proposal writers live in share points. So why 416 00:32:14.079 --> 00:32:21.000 should why should we build something outside an environment that they're already Why blow against 417 00:32:21.039 --> 00:32:24.319 the wind? Yeah? So then, Richard, what we did was back 418 00:32:24.359 --> 00:32:29.440 in February or March. This has been a very fast moving thing lest in 419 00:32:29.559 --> 00:32:34.359 nine months, we said, okay, we're going to offer this in a 420 00:32:34.400 --> 00:32:38.119 commercial manner as a paid service, and we went to many of those original 421 00:32:38.160 --> 00:32:43.160 data customers. Many of them signed up, so we were at this point 422 00:32:43.279 --> 00:32:50.200 running several production customers who are paying customers of ours. And I'm really proud 423 00:32:50.240 --> 00:32:54.880 to say that Microsoft's own proposal Center of Excellence. Microsoft has this group called 424 00:32:54.880 --> 00:33:01.240 the Global Proposal Center of Excellence. We went through an evaluation with them and 425 00:33:01.279 --> 00:33:07.920 they just onboarded as wow the tool. Wow, congratulations, congrats. Yeah, 426 00:33:07.960 --> 00:33:12.160 that's great, and gentlemen, we should pause for one moment for these 427 00:33:12.240 --> 00:33:16.000 very important messages. Now we're back. It's not that Rocks. I'm Richard 428 00:33:16.039 --> 00:33:22.720 Campbell. That's Carl Franklin. Hey here with our friend vishchwas Lela now the 429 00:33:22.799 --> 00:33:27.519 lead on this new Pewin company. And obviously you know your MVP has gone 430 00:33:27.519 --> 00:33:30.559 well, you've done some initial testing. You know. I'll throw my sissedmin 431 00:33:30.680 --> 00:33:35.279 hat on because I've talked to a lot of CTO CIO types who are looking 432 00:33:35.359 --> 00:33:38.480 at these technologies and they keep being told by Microsoft, you should have your 433 00:33:38.559 --> 00:33:45.079 quote data estate in order, like what I'm afraid of with the generalized tools 434 00:33:45.119 --> 00:33:49.480 like this, is that your data state is not in order, and so 435 00:33:49.920 --> 00:33:53.400 data is going to be pulled into the model and then subsequently revealed in prompts 436 00:33:53.680 --> 00:33:58.839 that everyone's going to be very unhappy about. And there is no it's not 437 00:33:58.880 --> 00:34:01.359 like a sign pop out of a server somewhere when your data state is in 438 00:34:01.440 --> 00:34:06.480 order. You don't know, it's like your system should be secure. It's 439 00:34:06.640 --> 00:34:09.719 just as impossible. Like, one of the things that appeals to me about 440 00:34:10.199 --> 00:34:16.920 your approach to this is you're encapsulating the data set and arguably you're taking that 441 00:34:17.000 --> 00:34:22.199 responsibility saying where the folks who will make sure that the data that appears in 442 00:34:22.239 --> 00:34:25.760 the model is appropriate. I mean, I'm wondering. I'm just wondering. 443 00:34:25.760 --> 00:34:30.880 How much of work is that for you? Yeah, that's a huge amount 444 00:34:30.920 --> 00:34:36.800 of work. If I had to look back, we've spent thousands of hours 445 00:34:37.119 --> 00:34:42.519 data science hours in building the multi layer prompt engine. I would say an 446 00:34:42.559 --> 00:34:46.440 equal amount of time in worrying about ingestion of the data. And now, 447 00:34:46.639 --> 00:34:54.239 fortunately for us, this is why a domain specific copilot made sense, because 448 00:34:54.280 --> 00:34:58.320 you know you're limiting the data set, right. Some of the things that 449 00:34:58.360 --> 00:35:05.079 you're talking about is you might have hundreds of thousands of documents in SharePoint and 450 00:35:05.119 --> 00:35:08.480 stored somewhere on one drive, and that one document which nobody is found until 451 00:35:08.519 --> 00:35:13.000 now. You can be sure that M three sixty five copilet would find it. 452 00:35:13.039 --> 00:35:16.840 And somebody asks the question, that question will get answered inappropriately because that 453 00:35:16.920 --> 00:35:22.719 document was available, but nobody knew about that document. In our case, 454 00:35:22.840 --> 00:35:29.880 it helped that we limit the kind of data that goes into the engine. 455 00:35:30.199 --> 00:35:36.039 Right And as I mentioned, your past RFP submissions, your white papers, 456 00:35:36.639 --> 00:35:43.400 and things like SPARS, which is a government tissued document which says you've worked 457 00:35:43.440 --> 00:35:49.599 on these technologies well structured. So we limit ourselves to those documents. We 458 00:35:49.679 --> 00:35:54.800 do things like many of these proposals include resumes of people, and we exclude 459 00:35:54.840 --> 00:36:01.519 those resumes because now you're experienced from a previous employer might pollute your current employer's 460 00:36:01.599 --> 00:36:07.880 data, so we exclude that. So as part of the ingestion Richard, 461 00:36:07.920 --> 00:36:13.360 we break the documents down into logical sections. We use the language models for 462 00:36:13.440 --> 00:36:17.119 some classification. Even so we don't have human taggers to say this is a 463 00:36:17.159 --> 00:36:21.559 management plan, this is an executive somebody. We're using language models to do 464 00:36:21.639 --> 00:36:27.519 some tagging of the documents themselves, and I'm seeing this now inside of perview 465 00:36:27.639 --> 00:36:32.159 Microsoft's and again this IT conversation where it's getting really good at calling out PII. 466 00:36:34.039 --> 00:36:37.519 Even before you got around a tagging. It knows the shape of identifiable 467 00:36:37.559 --> 00:36:44.000 information. So starting with a limited data set helped us. And you know 468 00:36:44.199 --> 00:36:46.920 that may not be applicable to other people here, and frankly, that's the 469 00:36:47.039 --> 00:36:52.119 challenge that Microsoft had with m TH sixty five copilot, because you have everything, 470 00:36:52.599 --> 00:36:57.119 and of course they've added capabilities like hey, don't include these documents in 471 00:36:57.159 --> 00:37:01.920 my index or leave this library out right. So obviously those governance controls have 472 00:37:02.079 --> 00:37:06.000 come into play. In our case, we started a small data set, 473 00:37:06.039 --> 00:37:08.360 but even then it was really important to parse it. I'll give you an 474 00:37:08.400 --> 00:37:15.840 example. You might partner with a company when you're applying for a certain piece 475 00:37:15.880 --> 00:37:22.400 of work. That company may end up teaming with a competitor reviewers in a 476 00:37:22.440 --> 00:37:28.119 subsequent RFP. Now if you've taken their content included into your knowledge base, 477 00:37:28.960 --> 00:37:30.920 chances are that that content will show up, so you have to be careful 478 00:37:30.960 --> 00:37:38.440 about that in terms of how you're using maybe partner's data. So all of 479 00:37:38.480 --> 00:37:45.400 those things, Richard apply and we're breaking that content down again. I talked 480 00:37:45.400 --> 00:37:49.880 about entities and then we use those for our content generation. I got a 481 00:37:49.960 --> 00:37:52.920 question, did you, and you may have said this early on, I 482 00:37:52.960 --> 00:37:59.280 didn't hear it, but we're using Microsoft copilot Studio to create your're We're not 483 00:37:59.320 --> 00:38:04.039 losing max of copallet studio. Have you used it, I've used it, 484 00:38:04.559 --> 00:38:13.639 But this we started our development well before copiallet Studio. And secondly, our 485 00:38:13.800 --> 00:38:19.119 response engine at this point is so complicated that we needed to go directly build 486 00:38:19.199 --> 00:38:23.039 things in Python and asure machine learning notebooks and things like that. Gotcha. 487 00:38:23.159 --> 00:38:30.119 I mean what I like about this is you clearly brought your own meta understanding 488 00:38:30.320 --> 00:38:37.320 of RFP writing to the tool set that's right to then make into a product 489 00:38:37.480 --> 00:38:42.199 that works well for individual companies against their data. But you know, there's 490 00:38:42.239 --> 00:38:46.039 three pieces here. There is the language model as it stands. There is 491 00:38:46.239 --> 00:38:52.960 the understanding of the rules of writing RFPs effectively and applying those. And then 492 00:38:52.000 --> 00:38:58.239 there is the data set that is the companies for what their capabilities are and 493 00:38:58.519 --> 00:39:01.719 how they want to respond to RFPs like those. You need all of those 494 00:39:01.760 --> 00:39:07.840 things. It's not there's no magic wand here. Smart people had to work 495 00:39:07.920 --> 00:39:12.239 really hard to make this work. But when it works, the fact that 496 00:39:12.280 --> 00:39:16.039 you're going to be able to respond to more RFPs and to write better responses 497 00:39:16.199 --> 00:39:22.039 like this is money in the bank if done why if done right? And 498 00:39:22.360 --> 00:39:27.440 I should add one other thing which I should have mentioned earlier. I believe 499 00:39:27.679 --> 00:39:34.480 that in order to develop any effective AI solution, you need to have two 500 00:39:34.480 --> 00:39:37.239 good mL engineering. Of course that's a given, but then you have to 501 00:39:37.239 --> 00:39:43.119 bring domain expertise. And you alluded to that that we brought some domain expertise, 502 00:39:43.320 --> 00:39:46.119 yes, but we are not experts in proposal writing right. So what 503 00:39:46.199 --> 00:39:53.519 we did Richard early on is we went to a company called Shipley Shipley and 504 00:39:53.960 --> 00:39:58.719 you may not be familiar with them if you're not in the proposal space. 505 00:39:58.760 --> 00:40:05.679 But Steven Shipley fifty years ago wrote the book on proposal writing right, and 506 00:40:06.360 --> 00:40:09.760 really an influential book in terms of how should you be thinking about your RFP 507 00:40:09.880 --> 00:40:15.639 responses? And that translated into a company. People can go to Shipley wins 508 00:40:15.679 --> 00:40:21.119 dot com fifty year old company and their job it is to teach classes about 509 00:40:21.119 --> 00:40:25.840 better proposal writing, teach nuances of writing compliant proposals and when you need a 510 00:40:25.920 --> 00:40:30.039 surge capacity, they make their consultants available. Wow, okay, yeah, 511 00:40:30.079 --> 00:40:32.960 so they literally the business around it. That's a business around it. So 512 00:40:32.960 --> 00:40:36.559 we went to them and we said, you know, we are a tech 513 00:40:36.599 --> 00:40:39.199 company. We of course write our own proposals, but we are nowhere near 514 00:40:39.239 --> 00:40:44.360 the subject matter expertise and the name that you have, and we would like 515 00:40:44.400 --> 00:40:49.199 to partner you to be co creators of this tool for us. Right and 516 00:40:50.239 --> 00:40:52.960 Shipley had never partnered with a company, and I was clear that I'm not 517 00:40:53.039 --> 00:40:57.760 just looking for a referral partnership that hey recommend us. I want you to 518 00:40:57.840 --> 00:41:02.280 be joining the team so that you're constantly helping us improve the prompts. You're 519 00:41:02.320 --> 00:41:08.000 bringing that domain expertise. And they went through their own evaluation, Richard, 520 00:41:08.079 --> 00:41:13.559 for a period of time, they looked at other tools, and very proud 521 00:41:13.639 --> 00:41:15.639 to say that they decided to join in as an exclusive partner, as a 522 00:41:15.639 --> 00:41:23.199 conciliation partner. Right and having them on our engineering scrums and saying, generate 523 00:41:23.320 --> 00:41:29.199 this content in this manner, because what is very interesting about proposal writing is 524 00:41:30.880 --> 00:41:35.800 if you don't write it in a certain manner right there, you are restating 525 00:41:35.840 --> 00:41:39.519 the problem. You're saying you've done these things in the past, You're reinforcing 526 00:41:39.559 --> 00:41:45.360 the point. There's a certain format, and proposal writers tend to be very 527 00:41:45.400 --> 00:41:51.159 personniketty about that, right, if you don't follow that format wrong or all 528 00:41:51.199 --> 00:41:54.079 that's are off right, Yeah, and to the three of us, if 529 00:41:54.079 --> 00:42:00.119 you read one page of highly curated proposal writing ready stuff and then some other 530 00:42:00.159 --> 00:42:04.400 stuff, you would say, hey, reasonable people can argue about it. 531 00:42:04.400 --> 00:42:07.800 Oh yeah, your sentence is here, your reference is here. But they 532 00:42:07.800 --> 00:42:10.920 don't like that, right, They want the format to be exactly, So 533 00:42:12.800 --> 00:42:16.719 how do you write content that is persuasive, that follows the structure. And 534 00:42:16.760 --> 00:42:22.360 having a domain expert like them, so so fortunate to have them. They've 535 00:42:22.400 --> 00:42:25.760 taught people the process, the methodology, the best practice is I'm betting they're 536 00:42:25.960 --> 00:42:30.000 testing like that. They're evaluating the output. Absolutely did a show on run 537 00:42:30.039 --> 00:42:32.760 as with Lyn Langett. Now that logo we call the hard part of machine 538 00:42:32.800 --> 00:42:37.199 learning. It's like test test, test, Like you've got to evaluate these 539 00:42:37.199 --> 00:42:42.400 outputs, right, and it takes experts to even know what's good and what's 540 00:42:42.480 --> 00:42:46.000 bad. It takes experts and you cannot use just chat GiB to say which 541 00:42:46.000 --> 00:42:52.239 section is better, right because it requires experience, skill, all of that. 542 00:42:52.760 --> 00:42:55.760 So they're helping us with testing, they're helping us write this. So 543 00:42:55.840 --> 00:43:00.239 that was an important partnership. And that's maybe one tip to the list us 544 00:43:00.239 --> 00:43:04.719 out there that in order for an YI solution to be effective, you can't 545 00:43:04.800 --> 00:43:07.719 just have five spart engineers go to any domain and say I have a solution 546 00:43:07.840 --> 00:43:12.920 for your problem because I know as are mL and Python better than you. 547 00:43:13.039 --> 00:43:15.360 Well, that does not matter. Now you'd know my problem set. Well, 548 00:43:15.400 --> 00:43:22.400 there's plenty of smart developers that have built quality software that was wrong because 549 00:43:22.440 --> 00:43:27.519 they didn't have the domain experts involved to make the right thing. This is 550 00:43:27.559 --> 00:43:30.719 the same roles. You sit a different place in some respects. You're not 551 00:43:30.800 --> 00:43:35.639 qualified to evaluate this output. You need an expert to evaluate this out. 552 00:43:35.679 --> 00:43:38.800 So having that, I was just finishing that other point that you succinctly made, 553 00:43:38.840 --> 00:43:42.960 which is it's really important to have a domain expertise, And I was 554 00:43:43.039 --> 00:43:47.760 just adding that our expertise was not enough alone and we needed to partner with 555 00:43:47.800 --> 00:43:51.840 the best in the business to do that, and we're fortunate to have them. 556 00:43:51.880 --> 00:43:53.840 I mean, I've got to think there's somebody who's made their career to 557 00:43:53.920 --> 00:44:01.360 writing great RFPs. This is still pretty threatening like that, at least but 558 00:44:01.440 --> 00:44:05.119 admittedly you presented the right way. It's like it's only going to give you 559 00:44:05.119 --> 00:44:08.760 a draft. We're going to need your help right every time, at least 560 00:44:08.840 --> 00:44:14.679 to still be read. Yeah, Richard's that's a good point. And at 561 00:44:14.679 --> 00:44:19.239 some level we are all concerned about the socizeful impact of generative AI and all 562 00:44:19.280 --> 00:44:23.320 of the things that we do. But the way I presented two important points. 563 00:44:24.199 --> 00:44:28.920 One is, remember the notion of flight plan that I talked about. 564 00:44:29.559 --> 00:44:32.880 An expert proposal writer can quickly fill out the flight plan because think about that, 565 00:44:32.920 --> 00:44:37.920 you're setting the vision, or you're setting the intellectual momentum for you to 566 00:44:37.039 --> 00:44:42.199 win. Without a vision, nobody is winning any RFPs, right, And 567 00:44:42.320 --> 00:44:45.079 this is where the proposal directors or proposal writers come in and they fill out 568 00:44:45.119 --> 00:44:50.039 the flight plan. And we have been talking to them about how their job 569 00:44:50.199 --> 00:44:54.480 changes with generative AI, where they are operating at the object layer. We 570 00:44:54.559 --> 00:44:59.119 talk about them, We talk to them about objects, and then we are 571 00:44:59.280 --> 00:45:01.800 dealing at the work words level. Right, Yeah, they're changing the objects 572 00:45:01.800 --> 00:45:08.079 and attributes and methods maybe, but then we are generating all the code for 573 00:45:08.159 --> 00:45:12.400 them. In other words, that we are generating all the words for them. 574 00:45:12.480 --> 00:45:16.280 So that's one, and it is a model driven architecture in the sense 575 00:45:16.360 --> 00:45:21.599 that you change the flight plan, you look at the content and you feel 576 00:45:21.639 --> 00:45:23.559 like, hey, this came out pretty flat. Let me go back and 577 00:45:23.639 --> 00:45:29.079 change the flight plan and regenerate and regenerate and regenerate. Right, So it's 578 00:45:29.079 --> 00:45:32.679 a model driven architecture and the better they get it with that setup, they 579 00:45:32.679 --> 00:45:37.039 can make these quick changes. That's one. The second important point is we 580 00:45:37.079 --> 00:45:44.320 are not claiming authorship. Right. I get this request from some people that 581 00:45:44.440 --> 00:45:50.400 hey, can we use your tool? Our knowledge repository is pretty sparse. 582 00:45:50.719 --> 00:45:53.320 We have not done this type of a work. Can you help us generate 583 00:45:53.400 --> 00:45:58.920 good content? And well, this is not cheating as a service. You 584 00:45:58.960 --> 00:46:02.119 can just push up button and you get a proposal response. Right. If 585 00:46:02.119 --> 00:46:07.000 you don't, if you don't have good past performances, if you don't have 586 00:46:07.159 --> 00:46:13.639 good background, if you don't have creative architects who can come up with good 587 00:46:13.679 --> 00:46:17.239 creative solutions to the problem, chances are that what we generate is going to 588 00:46:17.280 --> 00:46:22.400 look very flat to you. So we're not claiming authorship. Those things have 589 00:46:22.480 --> 00:46:25.639 to be in place for an organization to be able to use over tool. 590 00:46:27.119 --> 00:46:31.199 So do you do you ask your customers. When you give a proposal, 591 00:46:31.280 --> 00:46:37.000 do you mind if we use this as and feed it back into the machine, 592 00:46:37.800 --> 00:46:42.480 so as as training data. So cal one important thing to note is 593 00:46:43.000 --> 00:46:46.000 we are not training the models. And this is a question that I get 594 00:46:46.039 --> 00:46:52.639 asked often. We're not training the model at all, because it's not like 595 00:46:52.760 --> 00:46:55.400 your data is going to be trained and then and then use for someone else. 596 00:46:55.760 --> 00:47:00.880 But you do you do use those as? Yes, we use them 597 00:47:00.920 --> 00:47:05.440 that we use them in a rag manner. Where right? And so I 598 00:47:05.519 --> 00:47:07.599 was just about to say this is retrieve all mana generation. This model is 599 00:47:07.639 --> 00:47:14.559 the model, I would say, you still have to ask permission. Yeah, 600 00:47:14.639 --> 00:47:17.239 So, so just to be clear, if somebody acquires our tool, 601 00:47:19.400 --> 00:47:22.559 they go through an onboarding stage where we look at the documents and we tell 602 00:47:22.599 --> 00:47:28.559 them some best practices and then we ingest those documents. Of course, that's 603 00:47:29.360 --> 00:47:32.480 their asset and that data belongs to them. And whenever engine runs, it 604 00:47:32.559 --> 00:47:37.199 dynamically makes a call into this database to look for right pieces of information. 605 00:47:38.559 --> 00:47:44.079 So of course we are using their data, but then we dynamically call into 606 00:47:44.119 --> 00:47:49.159 this engine. If there is no results to be returned because they have not 607 00:47:49.280 --> 00:47:52.239 done this kind of work, then don't have any white papers and that technology, 608 00:47:52.760 --> 00:47:57.880 then it's going to return pretty flat results and we can't infuse that content 609 00:47:57.960 --> 00:48:02.400 back into language Models and c rate the content appropriate. Okay, soa vishwas 610 00:48:02.440 --> 00:48:07.000 what's on your to do list? What's next? Well, the next is 611 00:48:07.039 --> 00:48:15.679 to continue to improve the product. That's what I'm focused on for the foreseeable 612 00:48:15.760 --> 00:48:21.920 future. Take advantage of all of the amazing enhancements that are coming to the 613 00:48:22.000 --> 00:48:25.960 Language Model space. What I love about this is this allows me to tap 614 00:48:27.000 --> 00:48:30.440 into all of the Azure work that I've done in the last ten years. 615 00:48:30.639 --> 00:48:36.559 Because ultimately Language Model is just one service. There are thirty eight other Azure 616 00:48:36.639 --> 00:48:39.760 services that are powering this solution. So a lot of things that we've learned 617 00:48:40.360 --> 00:48:45.119 over the last ten years with the community are coming into play. So continue 618 00:48:45.480 --> 00:48:49.559 to see how far we can take this, how we can support our customers 619 00:48:49.599 --> 00:48:54.440 better. So that's what next for me. Fantastic. Hey, it's always 620 00:48:54.519 --> 00:48:58.440 great to talk to you. We always learned something, we always get a 621 00:48:58.480 --> 00:49:02.320 new perspective on AI and when we talk to you, and thanks, thank 622 00:49:02.360 --> 00:49:06.800 you, all right, and we'll talk to you the next time at dot 623 00:49:06.840 --> 00:49:30.559 NetBox. Dot net Rocks is brought to you by Franklin's Net and produced by 624 00:49:30.639 --> 00:49:37.559 Pop Studios, a full service audio, video and post production facility located physically 625 00:49:37.599 --> 00:49:42.559 in New London, Connecticut, and of course in the cloud online at pwop 626 00:49:42.800 --> 00:49:45.679 dot com. Visit our website at d O T N E T R O 627 00:49:45.760 --> 00:49:52.039 c k S dot com for RSS feeds, downloads, mobile apps, comments, 628 00:49:52.400 --> 00:49:55.480 and access to the full archives going back to show number one, recorded 629 00:49:55.519 --> 00:49:59.960 in September two thousand and two. And make sure you check out our spot 630 00:50:00.000 --> 00:50:02.400 sponsors. They keep us in business. Now, go write some code. 631 00:50:04.000 --> 00:50:13.719 See you next time. You got jad middle vans down there that means hard 632 00:50:14.199 --> 00:50:15.360 than my taxes are