WEBVTT 1 00:00:00.440 --> 00:00:03.799 Thank everyone. My name is say It Heather. I will 2 00:00:03.839 --> 00:00:09.880 be moderating session. Welcome to the panel. How forarma is 3 00:00:09.919 --> 00:00:16.320 adopting jen AI across medical development? I think previous speaker 4 00:00:16.359 --> 00:00:19.920 has set us well. Those are the things that hopefully 5 00:00:19.920 --> 00:00:23.519 we will be addressing answering for you, and some additional 6 00:00:23.600 --> 00:00:26.519 questions may come up as well. So with that, I 7 00:00:26.600 --> 00:00:29.679 just wanted to quickly introduce myself. I work at MERC 8 00:00:30.000 --> 00:00:35.880 Pharmaceutical and I lead AI in clinical trial operations, help 9 00:00:36.359 --> 00:00:41.719 support AIU spass strategy and so on, and in my 10 00:00:41.799 --> 00:00:44.960 leisure time, I'm also an adjunct faculty at Columbia University, 11 00:00:45.840 --> 00:00:50.320 where I am affiliated with Data Science Institute. So okay, 12 00:00:50.399 --> 00:00:56.759 with that, I can I can start with our panelist here. 13 00:00:57.320 --> 00:01:00.920 So on my right is Henry Wee is from Regional 14 00:01:01.439 --> 00:01:05.640 s JATA, and then we have Shamir Kadir from Sonfi 15 00:01:06.319 --> 00:01:11.519 Jatasha from Moderna and Dave Appel from JANJA. And I 16 00:01:11.599 --> 00:01:14.200 ask you to introduce yourselves when you answer the questions. 17 00:01:14.519 --> 00:01:14.920 Thank you. 18 00:01:16.840 --> 00:01:20.840 So, like I said, we're well said to address these 19 00:01:21.680 --> 00:01:25.040 topics that we will be I will be asking questions 20 00:01:25.040 --> 00:01:28.879 of our testing panelists and we will learn more about it. 21 00:01:31.400 --> 00:01:35.120 So The first question I asked is for Sujata, share 22 00:01:35.159 --> 00:01:44.239 your organization's experience regarding citizen JENNYI for each user or employee. 23 00:01:43.959 --> 00:01:48.680 Thank you sired everyone. Hi, my name is Sujatasha. I 24 00:01:48.760 --> 00:01:54.599 am from Maderna and I am kind of not in. 25 00:01:54.560 --> 00:01:56.959 The digital team. I come from medical writing. 26 00:01:58.519 --> 00:02:01.680 But we've been exploring this idea of AI for quite 27 00:02:01.680 --> 00:02:07.120 some time at Maderna. If you kind of take the 28 00:02:07.200 --> 00:02:11.039 journey of evolution with me for just a moment. Remember 29 00:02:11.080 --> 00:02:15.800 when we had these big supercomputers that just a few nerds, 30 00:02:15.840 --> 00:02:17.719 as David said, used. 31 00:02:19.039 --> 00:02:20.159 And then they become. 32 00:02:19.919 --> 00:02:21.599 Smaller and smaller and smaller. 33 00:02:21.319 --> 00:02:23.800 Until we now have supercomputers in our hands and we 34 00:02:23.879 --> 00:02:25.439 use them all the time for everything. 35 00:02:26.639 --> 00:02:28.039 That's kind of the evolution that. 36 00:02:28.080 --> 00:02:32.879 AI has been taking. So, as El mentioned earlier today, 37 00:02:33.639 --> 00:02:34.599 AI is not new. 38 00:02:34.759 --> 00:02:37.520 We've been using it, but it's only been. 39 00:02:37.439 --> 00:02:41.439 Used by a select few people that were experts digital people. 40 00:02:42.199 --> 00:02:44.960 But now it's getting to be like the smartphone in 41 00:02:45.000 --> 00:02:48.879 our hands. We all have access to it. And at Maderna, 42 00:02:49.759 --> 00:02:53.319 the leadership team realized the evolution that was taking place 43 00:02:53.400 --> 00:02:57.360 in the world of technology pretty early, and so a 44 00:02:57.400 --> 00:03:01.400 few years ago we launched what was called the AI Academy, 45 00:03:02.360 --> 00:03:06.280 and everyone in the company was required to attend AI Academy, 46 00:03:07.159 --> 00:03:09.919 where we learned what was happening in the world of. 47 00:03:09.879 --> 00:03:12.840 AI, how things were evolving. Pretty soon we were all 48 00:03:12.879 --> 00:03:14.199 going to have access to AI. 49 00:03:14.360 --> 00:03:19.560 So it started creating this buzz around AI at Maderna. 50 00:03:20.639 --> 00:03:24.240 From that, from the AI Academy, we launched a hackathon 51 00:03:24.840 --> 00:03:27.159 and that kind of helped us identify this. 52 00:03:27.159 --> 00:03:29.039 Lead group of people that were not. 53 00:03:28.919 --> 00:03:34.759 Necessarily digital, but we're really enjoying AI, and so then 54 00:03:35.000 --> 00:03:39.479 we made it really attractive. We created this really exclusive 55 00:03:39.599 --> 00:03:44.520 club called the AI Champions Team, and these were the 56 00:03:44.520 --> 00:03:47.879 people that had adopted AI. They had participated in the hackathon, 57 00:03:48.680 --> 00:03:51.439 and they were from all different parts of the company, 58 00:03:51.560 --> 00:03:54.520 so they were talking within their teams about how AI 59 00:03:54.639 --> 00:03:57.560 was going to help us do our jobs better. And 60 00:03:57.639 --> 00:04:00.280 because it was an exclusive club, as you know with 61 00:04:00.400 --> 00:04:04.000 any exclusive clubs, everyone wanted to join it because these 62 00:04:04.000 --> 00:04:06.319 were the people that were getting early access to the 63 00:04:06.719 --> 00:04:10.400 newer technologies as they were becoming available. And I know 64 00:04:10.599 --> 00:04:13.960 one of my colleagues was on that team, and every 65 00:04:14.039 --> 00:04:15.719 day I would ask him, how do I get on this? 66 00:04:15.960 --> 00:04:17.279 I want to be part of this, I want to 67 00:04:17.319 --> 00:04:20.839 get I want to get access to the technology, and 68 00:04:20.920 --> 00:04:24.920 so that just created that buzz of everyone wanting to 69 00:04:25.040 --> 00:04:29.319 use AI. And from there we started a partnership with 70 00:04:29.680 --> 00:04:33.199 open Ai, which allowed us to get enterprise level licenses 71 00:04:33.240 --> 00:04:39.680 for ched GPT for the entire company. And at this point, 72 00:04:39.759 --> 00:04:43.240 what we've done is we've opened up the creation of 73 00:04:43.360 --> 00:04:48.519 agents or GPTs to everyone, and it's very transparent to everyone. 74 00:04:48.680 --> 00:04:51.600 So I can see the GPT that David has built 75 00:04:52.120 --> 00:04:54.680 and say, oh, I can take parts of your GPT 76 00:04:54.959 --> 00:04:58.000 and adapt it to something I want to do in 77 00:04:58.000 --> 00:05:02.680 my job function point. We built about seven hundred and 78 00:05:02.800 --> 00:05:05.759 fifty was the count last week. It's probably eight hundred 79 00:05:05.759 --> 00:05:09.480 by now or so. Agents within Maderna that everyone is 80 00:05:09.519 --> 00:05:14.639 allowed to use. It ranges from everything like make this 81 00:05:14.759 --> 00:05:19.399 email a friendlier or what are the benefits that are 82 00:05:19.439 --> 00:05:22.920 available to me through Maderna, or all the way to 83 00:05:23.040 --> 00:05:27.399 data analysis and being able to identify the dose in 84 00:05:27.480 --> 00:05:32.839 a study. All of this of course is with engagement 85 00:05:32.879 --> 00:05:36.160 with human beings. So the human judgment is of course 86 00:05:36.199 --> 00:05:39.120 a part of all of this, and that that's where 87 00:05:39.120 --> 00:05:41.160 I'll stop, and that someone else. 88 00:05:41.600 --> 00:05:42.560 Thank you, go ahead, David. 89 00:05:42.560 --> 00:05:46.480 You share with day A bell I lead External Innovation, 90 00:05:46.759 --> 00:05:51.439 so partnerships for RNI data Science at Jane Jay Innovative Medicine. 91 00:05:52.199 --> 00:05:56.160 What you said totally totally resonated with me, and it 92 00:05:56.199 --> 00:05:57.959 was great to see Mary Jill and Bardie stalk right 93 00:05:58.040 --> 00:06:00.720 before this, which actually has data on a lot. 94 00:06:00.600 --> 00:06:01.319 Of these topics. 95 00:06:02.160 --> 00:06:04.680 Not to put us down button contrast with the opinions 96 00:06:04.720 --> 00:06:07.120 we will and maybe a little bit experience that will 97 00:06:07.160 --> 00:06:08.319 offer here today. 98 00:06:08.759 --> 00:06:09.839 So I mean we're. 99 00:06:09.680 --> 00:06:14.240 Now not quite two years into the chat GPT era, 100 00:06:14.959 --> 00:06:19.720 which is you know, that's really the moment where leaders 101 00:06:19.879 --> 00:06:23.360 at companies and pharma were said, oh aha, there's this 102 00:06:23.480 --> 00:06:26.439 huge opportunity here and even if we can't fight quite 103 00:06:26.480 --> 00:06:30.079 grasp it yet, we see the We see that data 104 00:06:30.120 --> 00:06:33.480 science and AIII is not going to be a niche technology. 105 00:06:33.560 --> 00:06:35.920 This has the opportunity to improve a lot of what 106 00:06:36.000 --> 00:06:39.079 we do. And so I mean we're working across all 107 00:06:39.120 --> 00:06:42.680 of the different vectors there, meaning deploying AI tools for 108 00:06:42.720 --> 00:06:46.800 everyone's you know, regular workflows like writing emails, which unfortunately 109 00:06:46.839 --> 00:06:49.959 takes a lot of time for probably everyone in this room, 110 00:06:50.319 --> 00:06:53.759 and then also specific use cases like medical writing, which 111 00:06:53.800 --> 00:06:56.000 a lot of companies, including ourselves are looking at as 112 00:06:56.079 --> 00:06:57.160 areas of opportunity. 113 00:06:57.279 --> 00:06:58.720 One last thing you said rested. 114 00:06:58.439 --> 00:07:00.759 To do with me, which is getting people excited about 115 00:07:00.759 --> 00:07:05.040 this right, and in particular you highlighted the uh, the 116 00:07:05.079 --> 00:07:09.000 power of the fear of missing out. FOMO is an 117 00:07:09.040 --> 00:07:13.879 excellent an excellent driver of human behavior and it is 118 00:07:14.680 --> 00:07:15.600 for better and worse. 119 00:07:15.800 --> 00:07:16.199 Uh. 120 00:07:16.399 --> 00:07:18.160 It creates a lot of a lot of a lot 121 00:07:18.160 --> 00:07:20.439 of buzz. And so in contrast with two years ago 122 00:07:20.879 --> 00:07:24.399 where you know AI and generative AI, we're seen as 123 00:07:24.399 --> 00:07:29.439 this niche, niche tool, you know, living in honestly in 124 00:07:29.480 --> 00:07:33.240 like discovery use cases for protein folding and structural biology. 125 00:07:34.120 --> 00:07:36.160 Now everyone wants to be a part of it that 126 00:07:36.199 --> 00:07:38.879 creates its own problems, but it's also really exciting for 127 00:07:39.000 --> 00:07:41.319 those of us who are excited about the opportunity and 128 00:07:41.959 --> 00:07:44.959 want to realize that opportunity for the benefit of the 129 00:07:45.000 --> 00:07:46.279 company and the benefit of patients. 130 00:07:46.279 --> 00:07:50.600 Most importantly, absolutely, thank you both, and I just can 131 00:07:51.680 --> 00:07:56.439 just re emphasize that you know AI education, AI engagement 132 00:07:56.600 --> 00:07:59.839 is very important for driving and building their trust confidence. 133 00:08:00.120 --> 00:08:03.720 We saw from Mary Joe's presentation building that you know, 134 00:08:03.879 --> 00:08:11.319 trust deficit. So my next question is for Dave, what 135 00:08:11.399 --> 00:08:14.639 is strategy by a form should adopt for you know, 136 00:08:14.920 --> 00:08:20.120 deploying these solutions at a scale, so as we. 137 00:08:20.040 --> 00:08:23.560 Saw earlier from Mary Joe and I think reflecting a 138 00:08:23.600 --> 00:08:26.120 lot of probably reflecting the experience of the people there. 139 00:08:26.160 --> 00:08:30.680 There are many many use cases in development, however, there 140 00:08:30.720 --> 00:08:34.080 are few that have been truly scaled and that that 141 00:08:34.159 --> 00:08:38.440 makes good sense, right like pharma is uh risk averse 142 00:08:38.480 --> 00:08:38.960 and and and. 143 00:08:39.440 --> 00:08:41.720 That's appropriate. 144 00:08:43.600 --> 00:08:46.320 Given the you know, the the importance and risk of 145 00:08:46.360 --> 00:08:47.879 what we're we're working on. 146 00:08:48.000 --> 00:08:51.600 The advice I would give is to take a. 147 00:08:51.080 --> 00:08:55.720 Testimonlear an approach to run pilots with with partners who 148 00:08:55.799 --> 00:08:59.480 run pilots internally in use cases. 149 00:08:59.159 --> 00:08:59.840 That are. 150 00:09:02.919 --> 00:09:07.279 Either uh you know, lower risk for for patients or 151 00:09:07.279 --> 00:09:13.519 the company, or in in areas where it's kind of 152 00:09:13.559 --> 00:09:16.080 contained and isolated. And in doing that, you can actually 153 00:09:16.120 --> 00:09:18.559 address a bunch of the other questions. You can get 154 00:09:18.559 --> 00:09:20.399 the buy and you can need, you can illustrate the 155 00:09:20.519 --> 00:09:22.360 r o I, and then after doing that you can 156 00:09:22.480 --> 00:09:24.960 you can scale up. It's the same as any new technology. 157 00:09:25.480 --> 00:09:27.679 It's something that I think a lot of companies are 158 00:09:27.679 --> 00:09:31.399 struggling with. We are actively building that muscle and kind 159 00:09:31.440 --> 00:09:34.840 of at the edge of moving things from you know, 160 00:09:34.919 --> 00:09:36.679 pilot stage to. 161 00:09:36.440 --> 00:09:39.159 Scaling up a bunch, So that would be my advice. 162 00:09:39.200 --> 00:09:40.360 You got to you gotta get started. 163 00:09:41.240 --> 00:09:41.639 Thank you. 164 00:09:42.360 --> 00:09:43.639 I could not agree more. 165 00:09:44.080 --> 00:09:49.480 And you know, thinking about value proposition, thinking about feasibility, 166 00:09:50.039 --> 00:09:52.440 and we are all at nissant stage. We saw that 167 00:09:52.559 --> 00:09:56.600 from Mary Joe's sort way that the industry overall is 168 00:09:56.639 --> 00:09:59.200 at a nis into stage and adopting AI, so we're 169 00:09:59.200 --> 00:10:04.200 all in it for other. My next question is for Henry, 170 00:10:04.960 --> 00:10:08.559 what approaches have you seen to separate the high chrome reality? 171 00:10:08.600 --> 00:10:10.240 For GENNYI use cases. 172 00:10:10.720 --> 00:10:11.840 Thanks. Hi, I'm Henry. 173 00:10:11.840 --> 00:10:14.279 I'm a doctor by background, and I had an innovation 174 00:10:14.480 --> 00:10:17.600 for global development at Regeneron and we try and make 175 00:10:17.919 --> 00:10:20.960 trials faster and cheaper and better, so to separate the 176 00:10:21.039 --> 00:10:21.720 hype and reality. 177 00:10:21.720 --> 00:10:23.039 I used to be a lot meaner about this. 178 00:10:23.399 --> 00:10:26.000 So if any of has heerd about like for Matt's 179 00:10:26.039 --> 00:10:29.519 last theorem, sorry apologies and salespeople in the room, But 180 00:10:29.559 --> 00:10:32.120 I would basically make a word version of that and 181 00:10:32.159 --> 00:10:35.159 say like, oh, can your solution basically solve for integers 182 00:10:35.159 --> 00:10:38.279 that you know were the exponent is? And they'd be 183 00:10:38.279 --> 00:10:41.799 like oh yeah, yeah, big time, And that's like mathematically 184 00:10:41.840 --> 00:10:46.000 difficult to prove like from a historical perspective. So that's 185 00:10:46.080 --> 00:10:47.679 one way of flushing it out to just kind of 186 00:10:47.679 --> 00:10:50.240 pose questions that we know to be actually hard problems 187 00:10:50.279 --> 00:10:53.399 for humanity. I just not say there impossible. The more 188 00:10:53.440 --> 00:10:56.559 practical actually follows what Maderna did, which is that we 189 00:10:56.600 --> 00:11:00.000 actually have an open innovation model that we recent implement. 190 00:11:00.240 --> 00:11:03.759 I'd invite folks to go to regeneron dot depthpost dot com. 191 00:11:04.399 --> 00:11:04.960 There's kind of a. 192 00:11:04.960 --> 00:11:08.279 Dumb video of me there, but there are about fifty 193 00:11:08.320 --> 00:11:12.799 some entries where we actually asked folks without actually implying 194 00:11:12.919 --> 00:11:16.919 use of AI or not, solve useful problems to solve, 195 00:11:17.879 --> 00:11:19.960 and many of them actually tried to implement AI, and we 196 00:11:20.000 --> 00:11:21.960 got to see whether or not it was feasible or not. 197 00:11:22.279 --> 00:11:24.279 We also got to see patterns of how folks were 198 00:11:24.279 --> 00:11:28.879 obfuscating and trying to cover up the lack of utility 199 00:11:28.919 --> 00:11:30.679 of their AI, because we were able to pop up 200 00:11:30.679 --> 00:11:32.840 in the code and see it was basically an empty shell. 201 00:11:33.279 --> 00:11:35.559 So that was very instructive for us of having a 202 00:11:35.600 --> 00:11:39.679 diverse external group. Hammer at that and other companies have 203 00:11:39.759 --> 00:11:43.279 done similar exercise. A non to the literature and our 204 00:11:43.320 --> 00:11:46.879 peers doing great work with CSR writing I've seen in 205 00:11:46.919 --> 00:11:49.399 the literature, so I'm a big fan of that kind 206 00:11:49.440 --> 00:11:54.279 of inclusive, all hands on deck type model to really 207 00:11:54.360 --> 00:11:55.720 figure out what works early on. 208 00:11:56.600 --> 00:11:59.840 Absolutely, thank you, and I think I saw one of 209 00:11:59.840 --> 00:12:03.799 them talk yesterday. We're on the gardener hive cycle. We 210 00:12:03.840 --> 00:12:06.240 are now on the other side of the curve, and 211 00:12:06.840 --> 00:12:11.279 now reality is hitting the door, so it cannot be 212 00:12:12.360 --> 00:12:13.039 more true. 213 00:12:13.720 --> 00:12:14.120 All right. 214 00:12:14.279 --> 00:12:18.519 My next question is for smir how do data types 215 00:12:18.559 --> 00:12:22.799 influence the performance and design of algorithms and why is 216 00:12:23.360 --> 00:12:26.879 it often such that the quality and the structure of 217 00:12:26.960 --> 00:12:29.519 data are more critical than algorithms themselves. 218 00:12:30.759 --> 00:12:32.600 Thank you saying thanks for having me. And this is 219 00:12:32.639 --> 00:12:36.039 actually my third d farm and last two times i've 220 00:12:36.080 --> 00:12:41.559 given talks around advanced algorithms and graph EMIL and approaches, 221 00:12:41.639 --> 00:12:46.000 and I never seen a crowd lighting like this. So 222 00:12:46.039 --> 00:12:48.480 first of all, let me acknowledge that you know, for 223 00:12:48.559 --> 00:12:52.399 sure GPT is the iPhone moment that we're all looking for, right, 224 00:12:53.360 --> 00:12:55.960 and then at the same time that that core piece 225 00:12:56.000 --> 00:13:01.240 of technology is only six years old. It's built on 226 00:13:01.320 --> 00:13:03.320 top of a lot of other technologies, you know, whether 227 00:13:03.320 --> 00:13:06.559 it's transformers and bert and others that's built, but it's 228 00:13:06.679 --> 00:13:09.600 really geept's a six year old technology. Just so just 229 00:13:09.679 --> 00:13:14.759 imagine defarming another six years and in a way that 230 00:13:14.840 --> 00:13:18.879 but from a clinical development perspective, we should think about it. 231 00:13:18.960 --> 00:13:21.720 Like you know, I'm sure many of you use algorithms. 232 00:13:22.120 --> 00:13:28.000 What drives an algorithm sixty percents your data? How good 233 00:13:28.080 --> 00:13:31.200 is that data? How well informed, well connected is that data? 234 00:13:31.240 --> 00:13:33.960 So this is something that you all have to think 235 00:13:34.000 --> 00:13:38.200 about for that matter, whenever you're designing and investing in 236 00:13:38.759 --> 00:13:41.840 an initiative like this, because right now you know, all 237 00:13:41.919 --> 00:13:45.039 the way from your tech team to the c suit, 238 00:13:45.440 --> 00:13:49.679 everybody is interested in going GENAA. But I know there's 239 00:13:49.720 --> 00:13:52.600 also session on GENAI. But I want to tell you 240 00:13:52.600 --> 00:13:55.720 that many times you might not need a GENAI strategy. There, 241 00:13:55.840 --> 00:13:58.080 you might need an AI strategy. You might need a 242 00:13:58.120 --> 00:14:02.120 digital transformation strategy. By the way of introduction, I come 243 00:14:02.159 --> 00:14:06.120 from the Christian Medicine and Computational Biology function and safe 244 00:14:06.240 --> 00:14:10.600 we support our end their immunoscience and other therapeutic areas. 245 00:14:10.759 --> 00:14:14.799 I lead the computational commutational team within Santa FE and 246 00:14:14.879 --> 00:14:17.840 in my personal opinion is that you know, the data 247 00:14:17.919 --> 00:14:21.679 matters a lot A lot of time we invest in 248 00:14:22.320 --> 00:14:25.840 the latest trend latest technology, but we often forget the 249 00:14:25.879 --> 00:14:28.840 importance of the data. So you know, down the lane. 250 00:14:29.039 --> 00:14:32.480 Over the next six years, I can tell you that 251 00:14:32.600 --> 00:14:35.679 all of our companies will have their own foundation models 252 00:14:36.039 --> 00:14:39.039 built using our own internal data. So if you don't 253 00:14:39.039 --> 00:14:41.679 have data centric efforts in the company, maybe this's a 254 00:14:41.679 --> 00:14:44.480 good time to start. I may just also add that 255 00:14:44.519 --> 00:14:47.679 from a from an AI perspective, Santa Fe is a 256 00:14:47.720 --> 00:14:50.600 company that's on a mission, on a transformation to be 257 00:14:52.080 --> 00:14:56.000 AI first in its approach and then also immuinoscience centric. 258 00:14:56.039 --> 00:14:59.519 And then we are investing heavily in both what we 259 00:14:59.600 --> 00:15:02.480 call as our CEO or the piece, and I think 260 00:15:02.600 --> 00:15:06.879 Folbes as a snackable AI that's open to almost every 261 00:15:06.879 --> 00:15:09.840 employee in the company and an expert AI where you 262 00:15:10.000 --> 00:15:13.360 need advanced AI you know all code, low code or 263 00:15:13.399 --> 00:15:16.159 you know, no code skills to do that interpretation. So 264 00:15:16.200 --> 00:15:17.399 we're seeing that already. 265 00:15:17.440 --> 00:15:17.639 You know. 266 00:15:17.679 --> 00:15:19.960 Then they are a large organization with you know, fifty 267 00:15:19.960 --> 00:15:23.559 one hundred thousand people are already embracing AI, and almost 268 00:15:23.639 --> 00:15:27.960 every SIGNI is now one a chat interface to the 269 00:15:28.000 --> 00:15:31.799 scientific databases or to have an interface with the patient 270 00:15:32.200 --> 00:15:35.799 instead of having a normal challenge in people. One generative 271 00:15:35.799 --> 00:15:39.480 AI challenge and so this is here. But remember you 272 00:15:39.519 --> 00:15:42.320 know your data matters a lot because a lot of 273 00:15:42.360 --> 00:15:45.720 the other pieces, like the whether it's a GPT or others, 274 00:15:45.799 --> 00:15:48.360 they are now commodity assets. Now you know, you know, 275 00:15:48.399 --> 00:15:50.840 you can, you can plug and play with different algorithms. 276 00:15:51.080 --> 00:15:53.639 They're all getting better over time. So let's make sure 277 00:15:53.720 --> 00:15:57.120 that you know the data. Data is the king in 278 00:15:57.159 --> 00:15:59.320 my opinion, Thank you, Shamir. 279 00:15:59.639 --> 00:16:02.679 And with data, I recall that people usually ask me 280 00:16:03.200 --> 00:16:05.720 when this model was trained and we have to tell 281 00:16:05.799 --> 00:16:08.639 like it knows the world as of August twenty twenty 282 00:16:08.639 --> 00:16:12.559 three or September twenty twenty one. You know, so these 283 00:16:12.559 --> 00:16:15.080 models are trained at a certain time, so they don't 284 00:16:15.120 --> 00:16:17.360 know the data that has occurred since then. So a 285 00:16:17.440 --> 00:16:22.399 great feedback. Thank you. My next question is again for Henry. 286 00:16:23.240 --> 00:16:27.720 How are you thinking of assessing the performance of GENI solutions. 287 00:16:28.240 --> 00:16:29.759 Yeah, I'm very. 288 00:16:29.600 --> 00:16:33.440 Excited about this, and I would nod to a colleague, 289 00:16:34.200 --> 00:16:38.080 John Curry, International Man of Mystery, one of our closest 290 00:16:38.080 --> 00:16:41.679 working colleagues, having a really solid data foundation, not in 291 00:16:41.799 --> 00:16:45.360 training data, not in kind of data for in France, 292 00:16:45.840 --> 00:16:50.200 but actually instrumenting your operations and having great discipline around 293 00:16:50.240 --> 00:16:54.200 measuring cycle times and performance and other attributes of human 294 00:16:54.240 --> 00:16:58.799 activity is extraordinarily helpful at helping you identify where the 295 00:16:58.799 --> 00:17:01.559 true pain points are and whether or not these solutions 296 00:17:01.600 --> 00:17:04.319 are actually helping things along. So a lot of this 297 00:17:04.359 --> 00:17:08.960 is predicated on just great operating discipline and strategy being 298 00:17:09.480 --> 00:17:13.119 put in place well in advance of folks actually even 299 00:17:13.279 --> 00:17:16.279 considering JENII, and it pays div ends right there and 300 00:17:16.279 --> 00:17:19.160 then so I can't say enough for just yeah, how 301 00:17:19.440 --> 00:17:22.079 are we thinking of assessing the performance of Jeni? It's 302 00:17:22.119 --> 00:17:25.119 the question is actually how are we thinking of assessing performance? 303 00:17:26.079 --> 00:17:29.000 Very nice, thank you. Yeah, this is a key for 304 00:17:29.160 --> 00:17:35.559 AI success implementation that we are able to compute the metrics, 305 00:17:35.640 --> 00:17:39.880 collect the metrics, and then demonstrate that actually helps leadership 306 00:17:40.359 --> 00:17:45.200 and the business to understand and adopt we are making value, 307 00:17:45.279 --> 00:17:48.359 offering value and where things still or may not be 308 00:17:48.359 --> 00:17:52.279 as feasible at the moment. So thank your gain and just. 309 00:17:52.279 --> 00:17:54.039 Quick yeah, very quickly. 310 00:17:54.119 --> 00:17:59.599 So from a medical writing perspective, we're using and testing 311 00:17:59.640 --> 00:18:03.480 AI to summarize tables or do the simple things that 312 00:18:03.519 --> 00:18:10.039 medical writers do. And here we really realize how important 313 00:18:10.799 --> 00:18:15.519 the quality check is because hallucinations are common with AI, 314 00:18:15.799 --> 00:18:18.200 and a lot of our medical writers will say, well, 315 00:18:18.240 --> 00:18:22.079 I'll use AI when the hallucinations are all gone. I so, well, 316 00:18:22.319 --> 00:18:25.119 if the hallucinations are all gone, then what are you doing? 317 00:18:26.519 --> 00:18:30.839 So the importance of that human judgment in every step 318 00:18:30.880 --> 00:18:33.000 of the ways is really important. 319 00:18:33.559 --> 00:18:37.799 This is one of the most critical and trickiest topics 320 00:18:39.599 --> 00:18:44.319 because all of these models are proliferating, all these use 321 00:18:44.359 --> 00:18:45.519 cases are proliferating. 322 00:18:46.119 --> 00:18:49.960 So many companies are approaching. 323 00:18:51.839 --> 00:18:57.680 BARMA customers with solutions and trying to figure out which 324 00:18:57.720 --> 00:19:01.200 of the good ones is. You know, a considerable effort, 325 00:19:01.240 --> 00:19:06.640 and it's sort of a like by definition, a multidisciplinary effort. 326 00:19:06.680 --> 00:19:09.640 You need the data science tist in the room, who 327 00:19:09.720 --> 00:19:14.000 as assessing the architecture, who's assessing benchmarks to the extent 328 00:19:14.119 --> 00:19:15.680 that those are available. 329 00:19:15.759 --> 00:19:16.839 But then you also need. 330 00:19:16.680 --> 00:19:19.759 The domain experts, right so in medical writing you need 331 00:19:19.799 --> 00:19:20.880 the medical writing. 332 00:19:20.920 --> 00:19:22.839 Expert to look at this and saint does this work 333 00:19:22.880 --> 00:19:25.160 with workflows? Are they solving the right problem? 334 00:19:25.839 --> 00:19:28.960 And this is challenging, But I think what's interesting about 335 00:19:29.000 --> 00:19:32.079 it is jen AI has forced a lot of people 336 00:19:32.160 --> 00:19:35.319 to work together and come together that never work together before. 337 00:19:35.400 --> 00:19:37.839 So Jane Jay, the data science team which a month. 338 00:19:38.000 --> 00:19:40.440 We're now working super closely with the medical writing team, 339 00:19:40.440 --> 00:19:42.160 but I think like two years ago that was not 340 00:19:42.240 --> 00:19:45.519 the case and there have been no collaboration. So you know, 341 00:19:45.559 --> 00:19:47.599 you need to bring both parties to the table and 342 00:19:47.720 --> 00:19:50.240 kind of work through the kings there. But you need 343 00:19:51.359 --> 00:19:54.039 a whole bunch of people to really do this, do 344 00:19:54.119 --> 00:19:57.799 this effectively. And so the data science side, I would 345 00:19:57.839 --> 00:20:00.000 just say, we, you know, it's critical that we value 346 00:20:00.079 --> 00:20:03.119 the expertise of our of our functional business partners who 347 00:20:03.160 --> 00:20:05.279 really own the problem and understand it at a level 348 00:20:05.319 --> 00:20:08.039 maybe the data scientist down, So take some humility. 349 00:20:07.599 --> 00:20:10.599 To absolutely and I think that, Oh, go ahead, tru. 350 00:20:11.000 --> 00:20:13.480 I just want to, you know, just with respect to 351 00:20:13.519 --> 00:20:16.440 the performance of Genia solutions, maybe I just want to 352 00:20:16.480 --> 00:20:19.319 ask a question, with your permission, how many of you 353 00:20:19.160 --> 00:20:22.640 you think you currently use an AI solution as part 354 00:20:22.680 --> 00:20:28.839 of your clinical development? Fifty thirty percentage maybe in the room? 355 00:20:29.240 --> 00:20:32.079 How many of you have taken a decision based on 356 00:20:32.400 --> 00:20:40.559 a recommendation from an AI solution? Maybe thirty percentage of that? 357 00:20:41.640 --> 00:20:43.519 So you see that, right, like you know, when you 358 00:20:43.559 --> 00:20:45.559 when you you know the performance, you can come up 359 00:20:45.599 --> 00:20:49.240 with any type of KPI and OKR and metrics like that, 360 00:20:49.599 --> 00:20:52.000 but it's really up to them, those who actually have 361 00:20:52.200 --> 00:20:56.519 taken an action based on that feedback. So that human element, 362 00:20:56.720 --> 00:21:01.920 it's very difficult to capture, but it's very they actually 363 00:21:02.000 --> 00:21:05.519 brought that AA solution into their decision making. It augmented 364 00:21:05.519 --> 00:21:08.000 the team. It make the team move faster. That's not 365 00:21:08.039 --> 00:21:10.640 the case with all the use cases, but at least 366 00:21:10.640 --> 00:21:12.400 in some of the use cases, we see that here. 367 00:21:12.400 --> 00:21:15.799 So think about that, and it's an important that you should, 368 00:21:15.799 --> 00:21:21.200 every one of us to teach our colleagues, our executives 369 00:21:21.240 --> 00:21:25.480 about this. AA is not a magic wand it needs training, 370 00:21:25.519 --> 00:21:28.920 it needs help, and it's just another tool and you know, 371 00:21:29.440 --> 00:21:32.599 and it's moment right now, but think about its limitations. 372 00:21:32.640 --> 00:21:36.160 The real magic comes when your subject same is use 373 00:21:36.279 --> 00:21:38.720 this to I don't know, ten X or you know 374 00:21:38.720 --> 00:21:43.359 whatever improvement in their productivity. That's what you always think about. 375 00:21:43.440 --> 00:21:46.240 Metrics are great, but don't forget the human factor. 376 00:21:47.279 --> 00:21:49.680 Yeah. In fact, you know this term has been around 377 00:21:49.759 --> 00:21:53.559 for some time, maybe more than ten years. Citizen data scientists. 378 00:21:53.960 --> 00:21:57.240 I think in all practical purposes, it is coming to 379 00:21:57.319 --> 00:22:01.519 life that GENEI empowers each one of us as a 380 00:22:01.599 --> 00:22:04.759 data scientist. Think from that perspective when you're crafting your 381 00:22:04.839 --> 00:22:08.359 prompt you know, when you're asking a question, it just 382 00:22:08.480 --> 00:22:12.559 empowers so as as. And then the multi is disciplinary. 383 00:22:12.599 --> 00:22:18.839 I was attending FDA workshop last month in DC, and 384 00:22:19.000 --> 00:22:22.759 multidisciplinary was one of the key them that you know, 385 00:22:22.880 --> 00:22:26.559 you need teams which are multi disciplinary. You need data 386 00:22:26.599 --> 00:22:30.839 scientists or medical writers or medical writers who want to 387 00:22:30.839 --> 00:22:32.000 be data scientists. 388 00:22:32.319 --> 00:22:33.599 The lines are blurring. 389 00:22:34.119 --> 00:22:40.359 AI has just democratized these skills, these opportunities that the 390 00:22:40.440 --> 00:22:42.880 lines are blurring. So thank you again. 391 00:22:43.920 --> 00:22:44.680 With that, I. 392 00:22:44.680 --> 00:22:49.720 Will we'll move to call for action. Dave should be 393 00:22:50.000 --> 00:22:51.640 start with you the call for action. 394 00:22:52.079 --> 00:22:55.799 Sure, I would encourage everyone in this room to embrace 395 00:22:55.839 --> 00:23:01.319 the ambiguity and get started in trying to use AI 396 00:23:01.440 --> 00:23:05.079 and jen Ai in your in your data workflows. It 397 00:23:05.200 --> 00:23:08.920 is coming for all of us, and we have a 398 00:23:08.960 --> 00:23:15.720 responsibility to patients to make our trials and get therapies 399 00:23:15.759 --> 00:23:19.519 to patients as quickly and as effectively as impossible. So 400 00:23:20.400 --> 00:23:24.519 get started in using this in your workflows, bring ideas forward, 401 00:23:25.200 --> 00:23:25.880