Speaker 1: Thank everyone. My name is say It Heather. I will
Speaker 1: be moderating session. Welcome to the panel. How forarma is
Speaker 1: adopting jen AI across medical development? I think previous speaker
Speaker 1: has set us well. Those are the things that hopefully
Speaker 1: we will be addressing answering for you, and some additional
Speaker 1: questions may come up as well. So with that, I
Speaker 1: just wanted to quickly introduce myself. I work at MERC
Speaker 1: Pharmaceutical and I lead AI in clinical trial operations, help
Speaker 1: support AIU spass strategy and so on, and in my
Speaker 1: leisure time, I'm also an adjunct faculty at Columbia University,
Speaker 1: where I am affiliated with Data Science Institute. So okay,
Speaker 1: with that, I can I can start with our panelist here.
Speaker 1: So on my right is Henry Wee is from Regional
Speaker 1: s JATA, and then we have Shamir Kadir from Sonfi
Speaker 1: Jatasha from Moderna and Dave Appel from JANJA. And I
Speaker 1: ask you to introduce yourselves when you answer the questions.
Speaker 2: Thank you.
Speaker 1: So, like I said, we're well said to address these
Speaker 1: topics that we will be I will be asking questions
Speaker 1: of our testing panelists and we will learn more about it.
Speaker 1: So The first question I asked is for Sujata, share
Speaker 1: your organization's experience regarding citizen JENNYI for each user or employee.
Speaker 3: Thank you sired everyone. Hi, my name is Sujatasha. I
Speaker 3: am from Maderna and I am kind of not in.
Speaker 2: The digital team. I come from medical writing.
Speaker 3: But we've been exploring this idea of AI for quite
Speaker 3: some time at Maderna. If you kind of take the
Speaker 3: journey of evolution with me for just a moment. Remember
Speaker 3: when we had these big supercomputers that just a few nerds,
Speaker 3: as David said, used.
Speaker 4: And then they become.
Speaker 2: Smaller and smaller and smaller.
Speaker 3: Until we now have supercomputers in our hands and we
Speaker 3: use them all the time for everything.
Speaker 2: That's kind of the evolution that.
Speaker 3: AI has been taking. So, as El mentioned earlier today,
Speaker 3: AI is not new.
Speaker 2: We've been using it, but it's only been.
Speaker 3: Used by a select few people that were experts digital people.
Speaker 3: But now it's getting to be like the smartphone in
Speaker 3: our hands. We all have access to it. And at Maderna,
Speaker 3: the leadership team realized the evolution that was taking place
Speaker 3: in the world of technology pretty early, and so a
Speaker 3: few years ago we launched what was called the AI Academy,
Speaker 3: and everyone in the company was required to attend AI Academy,
Speaker 3: where we learned what was happening in the world of.
Speaker 2: AI, how things were evolving. Pretty soon we were all
Speaker 2: going to have access to AI.
Speaker 3: So it started creating this buzz around AI at Maderna.
Speaker 3: From that, from the AI Academy, we launched a hackathon
Speaker 3: and that kind of helped us identify this.
Speaker 2: Lead group of people that were not.
Speaker 3: Necessarily digital, but we're really enjoying AI, and so then
Speaker 3: we made it really attractive. We created this really exclusive
Speaker 3: club called the AI Champions Team, and these were the
Speaker 3: people that had adopted AI. They had participated in the hackathon,
Speaker 3: and they were from all different parts of the company,
Speaker 3: so they were talking within their teams about how AI
Speaker 3: was going to help us do our jobs better. And
Speaker 3: because it was an exclusive club, as you know with
Speaker 3: any exclusive clubs, everyone wanted to join it because these
Speaker 3: were the people that were getting early access to the
Speaker 3: newer technologies as they were becoming available. And I know
Speaker 3: one of my colleagues was on that team, and every
Speaker 3: day I would ask him, how do I get on this?
Speaker 3: I want to be part of this, I want to
Speaker 3: get I want to get access to the technology, and
Speaker 3: so that just created that buzz of everyone wanting to
Speaker 3: use AI. And from there we started a partnership with
Speaker 3: open Ai, which allowed us to get enterprise level licenses
Speaker 3: for ched GPT for the entire company. And at this point,
Speaker 3: what we've done is we've opened up the creation of
Speaker 3: agents or GPTs to everyone, and it's very transparent to everyone.
Speaker 3: So I can see the GPT that David has built
Speaker 3: and say, oh, I can take parts of your GPT
Speaker 3: and adapt it to something I want to do in
Speaker 3: my job function point. We built about seven hundred and
Speaker 3: fifty was the count last week. It's probably eight hundred
Speaker 3: by now or so. Agents within Maderna that everyone is
Speaker 3: allowed to use. It ranges from everything like make this
Speaker 3: email a friendlier or what are the benefits that are
Speaker 3: available to me through Maderna, or all the way to
Speaker 3: data analysis and being able to identify the dose in
Speaker 3: a study. All of this of course is with engagement
Speaker 3: with human beings. So the human judgment is of course
Speaker 3: a part of all of this, and that that's where
Speaker 3: I'll stop, and that someone else.
Speaker 4: Thank you, go ahead, David.
Speaker 5: You share with day A bell I lead External Innovation,
Speaker 5: so partnerships for RNI data Science at Jane Jay Innovative Medicine.
Speaker 5: What you said totally totally resonated with me, and it
Speaker 5: was great to see Mary Jill and Bardie stalk right
Speaker 5: before this, which actually has data on a lot.
Speaker 4: Of these topics.
Speaker 5: Not to put us down button contrast with the opinions
Speaker 5: we will and maybe a little bit experience that will
Speaker 5: offer here today.
Speaker 4: So I mean we're.
Speaker 5: Now not quite two years into the chat GPT era,
Speaker 5: which is you know, that's really the moment where leaders
Speaker 5: at companies and pharma were said, oh aha, there's this
Speaker 5: huge opportunity here and even if we can't fight quite
Speaker 5: grasp it yet, we see the We see that data
Speaker 5: science and AIII is not going to be a niche technology.
Speaker 5: This has the opportunity to improve a lot of what
Speaker 5: we do. And so I mean we're working across all
Speaker 5: of the different vectors there, meaning deploying AI tools for
Speaker 5: everyone's you know, regular workflows like writing emails, which unfortunately
Speaker 5: takes a lot of time for probably everyone in this room,
Speaker 5: and then also specific use cases like medical writing, which
Speaker 5: a lot of companies, including ourselves are looking at as
Speaker 5: areas of opportunity.
Speaker 4: One last thing you said rested.
Speaker 5: To do with me, which is getting people excited about
Speaker 5: this right, and in particular you highlighted the uh, the
Speaker 5: power of the fear of missing out. FOMO is an
Speaker 5: excellent an excellent driver of human behavior and it is
Speaker 5: for better and worse.
Speaker 6: Uh.
Speaker 5: It creates a lot of a lot of a lot
Speaker 5: of buzz. And so in contrast with two years ago
Speaker 5: where you know AI and generative AI, we're seen as
Speaker 5: this niche, niche tool, you know, living in honestly in
Speaker 5: like discovery use cases for protein folding and structural biology.
Speaker 5: Now everyone wants to be a part of it that
Speaker 5: creates its own problems, but it's also really exciting for
Speaker 5: those of us who are excited about the opportunity and
Speaker 5: want to realize that opportunity for the benefit of the
Speaker 5: company and the benefit of patients.
Speaker 1: Most importantly, absolutely, thank you both, and I just can
Speaker 1: just re emphasize that you know AI education, AI engagement
Speaker 1: is very important for driving and building their trust confidence.
Speaker 1: We saw from Mary Joe's presentation building that you know,
Speaker 1: trust deficit. So my next question is for Dave, what
Speaker 1: is strategy by a form should adopt for you know,
Speaker 1: deploying these solutions at a scale, so as we.
Speaker 5: Saw earlier from Mary Joe and I think reflecting a
Speaker 5: lot of probably reflecting the experience of the people there.
Speaker 5: There are many many use cases in development, however, there
Speaker 5: are few that have been truly scaled and that that
Speaker 5: makes good sense, right like pharma is uh risk averse
Speaker 5: and and and.
Speaker 4: That's appropriate.
Speaker 5: Given the you know, the the importance and risk of
Speaker 5: what we're we're working on.
Speaker 4: The advice I would give is to take a.
Speaker 5: Testimonlear an approach to run pilots with with partners who
Speaker 5: run pilots internally in use cases.
Speaker 1: That are.
Speaker 5: Either uh you know, lower risk for for patients or
Speaker 5: the company, or in in areas where it's kind of
Speaker 5: contained and isolated. And in doing that, you can actually
Speaker 5: address a bunch of the other questions. You can get
Speaker 5: the buy and you can need, you can illustrate the
Speaker 5: r o I, and then after doing that you can
Speaker 5: you can scale up. It's the same as any new technology.
Speaker 5: It's something that I think a lot of companies are
Speaker 5: struggling with. We are actively building that muscle and kind
Speaker 5: of at the edge of moving things from you know,
Speaker 5: pilot stage to.
Speaker 4: Scaling up a bunch, So that would be my advice.
Speaker 4: You got to you gotta get started.
Speaker 1: Thank you.
Speaker 6: I could not agree more.
Speaker 1: And you know, thinking about value proposition, thinking about feasibility,
Speaker 1: and we are all at nissant stage. We saw that
Speaker 1: from Mary Joe's sort way that the industry overall is
Speaker 1: at a nis into stage and adopting AI, so we're
Speaker 1: all in it for other. My next question is for Henry,
Speaker 1: what approaches have you seen to separate the high chrome reality?
Speaker 1: For GENNYI use cases.
Speaker 4: Thanks. Hi, I'm Henry.
Speaker 7: I'm a doctor by background, and I had an innovation
Speaker 7: for global development at Regeneron and we try and make
Speaker 7: trials faster and cheaper and better, so to separate the
Speaker 7: hype and reality.
Speaker 4: I used to be a lot meaner about this.
Speaker 7: So if any of has heerd about like for Matt's
Speaker 7: last theorem, sorry apologies and salespeople in the room, But
Speaker 7: I would basically make a word version of that and
Speaker 7: say like, oh, can your solution basically solve for integers
Speaker 7: that you know were the exponent is? And they'd be
Speaker 7: like oh yeah, yeah, big time, And that's like mathematically
Speaker 7: difficult to prove like from a historical perspective. So that's
Speaker 7: one way of flushing it out to just kind of
Speaker 7: pose questions that we know to be actually hard problems
Speaker 7: for humanity. I just not say there impossible. The more
Speaker 7: practical actually follows what Maderna did, which is that we
Speaker 7: actually have an open innovation model that we recent implement.
Speaker 7: I'd invite folks to go to regeneron dot depthpost dot com.
Speaker 6: There's kind of a.
Speaker 7: Dumb video of me there, but there are about fifty
Speaker 7: some entries where we actually asked folks without actually implying
Speaker 7: use of AI or not, solve useful problems to solve,
Speaker 7: and many of them actually tried to implement AI, and we
Speaker 7: got to see whether or not it was feasible or not.
Speaker 7: We also got to see patterns of how folks were
Speaker 7: obfuscating and trying to cover up the lack of utility
Speaker 7: of their AI, because we were able to pop up
Speaker 7: in the code and see it was basically an empty shell.
Speaker 7: So that was very instructive for us of having a
Speaker 7: diverse external group. Hammer at that and other companies have
Speaker 7: done similar exercise. A non to the literature and our
Speaker 7: peers doing great work with CSR writing I've seen in
Speaker 7: the literature, so I'm a big fan of that kind
Speaker 7: of inclusive, all hands on deck type model to really
Speaker 7: figure out what works early on.
Speaker 1: Absolutely, thank you, and I think I saw one of
Speaker 1: them talk yesterday. We're on the gardener hive cycle. We
Speaker 1: are now on the other side of the curve, and
Speaker 1: now reality is hitting the door, so it cannot be
Speaker 1: more true.
Speaker 4: All right.
Speaker 1: My next question is for smir how do data types
Speaker 1: influence the performance and design of algorithms and why is
Speaker 1: it often such that the quality and the structure of
Speaker 1: data are more critical than algorithms themselves.
Speaker 6: Thank you saying thanks for having me. And this is
Speaker 6: actually my third d farm and last two times i've
Speaker 6: given talks around advanced algorithms and graph EMIL and approaches,
Speaker 6: and I never seen a crowd lighting like this. So
Speaker 6: first of all, let me acknowledge that you know, for
Speaker 6: sure GPT is the iPhone moment that we're all looking for, right,
Speaker 6: and then at the same time that that core piece
Speaker 6: of technology is only six years old. It's built on
Speaker 6: top of a lot of other technologies, you know, whether
Speaker 6: it's transformers and bert and others that's built, but it's
Speaker 6: really geept's a six year old technology. Just so just
Speaker 6: imagine defarming another six years and in a way that
Speaker 6: but from a clinical development perspective, we should think about it.
Speaker 6: Like you know, I'm sure many of you use algorithms.
Speaker 6: What drives an algorithm sixty percents your data? How good
Speaker 6: is that data? How well informed, well connected is that data?
Speaker 6: So this is something that you all have to think
Speaker 6: about for that matter, whenever you're designing and investing in
Speaker 6: an initiative like this, because right now you know, all
Speaker 6: the way from your tech team to the c suit,
Speaker 6: everybody is interested in going GENAA. But I know there's
Speaker 6: also session on GENAI. But I want to tell you
Speaker 6: that many times you might not need a GENAI strategy. There,
Speaker 6: you might need an AI strategy. You might need a
Speaker 6: digital transformation strategy. By the way of introduction, I come
Speaker 6: from the Christian Medicine and Computational Biology function and safe
Speaker 6: we support our end their immunoscience and other therapeutic areas.
Speaker 6: I lead the computational commutational team within Santa FE and
Speaker 6: in my personal opinion is that you know, the data
Speaker 6: matters a lot A lot of time we invest in
Speaker 6: the latest trend latest technology, but we often forget the
Speaker 6: importance of the data. So you know, down the lane.
Speaker 6: Over the next six years, I can tell you that
Speaker 6: all of our companies will have their own foundation models
Speaker 6: built using our own internal data. So if you don't
Speaker 6: have data centric efforts in the company, maybe this's a
Speaker 6: good time to start. I may just also add that
Speaker 6: from a from an AI perspective, Santa Fe is a
Speaker 6: company that's on a mission, on a transformation to be
Speaker 6: AI first in its approach and then also immuinoscience centric.
Speaker 6: And then we are investing heavily in both what we
Speaker 6: call as our CEO or the piece, and I think
Speaker 6: Folbes as a snackable AI that's open to almost every
Speaker 6: employee in the company and an expert AI where you
Speaker 6: need advanced AI you know all code, low code or
Speaker 6: you know, no code skills to do that interpretation. So
Speaker 6: we're seeing that already.
Speaker 1: You know.
Speaker 6: Then they are a large organization with you know, fifty
Speaker 6: one hundred thousand people are already embracing AI, and almost
Speaker 6: every SIGNI is now one a chat interface to the
Speaker 6: scientific databases or to have an interface with the patient
Speaker 6: instead of having a normal challenge in people. One generative
Speaker 6: AI challenge and so this is here. But remember you
Speaker 6: know your data matters a lot because a lot of
Speaker 6: the other pieces, like the whether it's a GPT or others,
Speaker 6: they are now commodity assets. Now you know, you know,
Speaker 6: you can, you can plug and play with different algorithms.
Speaker 6: They're all getting better over time. So let's make sure
Speaker 6: that you know the data. Data is the king in
Speaker 6: my opinion, Thank you, Shamir.
Speaker 1: And with data, I recall that people usually ask me
Speaker 1: when this model was trained and we have to tell
Speaker 1: like it knows the world as of August twenty twenty
Speaker 1: three or September twenty twenty one. You know, so these
Speaker 1: models are trained at a certain time, so they don't
Speaker 1: know the data that has occurred since then. So a
Speaker 1: great feedback. Thank you. My next question is again for Henry.
Speaker 1: How are you thinking of assessing the performance of GENI solutions.
Speaker 6: Yeah, I'm very.
Speaker 7: Excited about this, and I would nod to a colleague,
Speaker 7: John Curry, International Man of Mystery, one of our closest
Speaker 7: working colleagues, having a really solid data foundation, not in
Speaker 7: training data, not in kind of data for in France,
Speaker 7: but actually instrumenting your operations and having great discipline around
Speaker 7: measuring cycle times and performance and other attributes of human
Speaker 7: activity is extraordinarily helpful at helping you identify where the
Speaker 7: true pain points are and whether or not these solutions
Speaker 7: are actually helping things along. So a lot of this
Speaker 7: is predicated on just great operating discipline and strategy being
Speaker 7: put in place well in advance of folks actually even
Speaker 7: considering JENII, and it pays div ends right there and
Speaker 7: then so I can't say enough for just yeah, how
Speaker 7: are we thinking of assessing the performance of Jeni? It's
Speaker 7: the question is actually how are we thinking of assessing performance?
Speaker 1: Very nice, thank you. Yeah, this is a key for
Speaker 1: AI success implementation that we are able to compute the metrics,
Speaker 1: collect the metrics, and then demonstrate that actually helps leadership
Speaker 1: and the business to understand and adopt we are making value,
Speaker 1: offering value and where things still or may not be
Speaker 1: as feasible at the moment. So thank your gain and just.
Speaker 2: Quick yeah, very quickly.
Speaker 3: So from a medical writing perspective, we're using and testing
Speaker 3: AI to summarize tables or do the simple things that
Speaker 3: medical writers do. And here we really realize how important
Speaker 3: the quality check is because hallucinations are common with AI,
Speaker 3: and a lot of our medical writers will say, well,
Speaker 3: I'll use AI when the hallucinations are all gone. I so, well,
Speaker 3: if the hallucinations are all gone, then what are you doing?
Speaker 3: So the importance of that human judgment in every step
Speaker 3: of the ways is really important.
Speaker 5: This is one of the most critical and trickiest topics
Speaker 5: because all of these models are proliferating, all these use
Speaker 5: cases are proliferating.
Speaker 4: So many companies are approaching.
Speaker 5: BARMA customers with solutions and trying to figure out which
Speaker 5: of the good ones is. You know, a considerable effort,
Speaker 5: and it's sort of a like by definition, a multidisciplinary effort.
Speaker 5: You need the data science tist in the room, who
Speaker 5: as assessing the architecture, who's assessing benchmarks to the extent
Speaker 5: that those are available.
Speaker 4: But then you also need.
Speaker 5: The domain experts, right so in medical writing you need
Speaker 5: the medical writing.
Speaker 4: Expert to look at this and saint does this work
Speaker 4: with workflows? Are they solving the right problem?
Speaker 5: And this is challenging, But I think what's interesting about
Speaker 5: it is jen AI has forced a lot of people
Speaker 5: to work together and come together that never work together before.
Speaker 4: So Jane Jay, the data science team which a month.
Speaker 5: We're now working super closely with the medical writing team,
Speaker 5: but I think like two years ago that was not
Speaker 5: the case and there have been no collaboration. So you know,
Speaker 5: you need to bring both parties to the table and
Speaker 5: kind of work through the kings there. But you need
Speaker 5: a whole bunch of people to really do this, do
Speaker 5: this effectively. And so the data science side, I would
Speaker 5: just say, we, you know, it's critical that we value
Speaker 5: the expertise of our of our functional business partners who
Speaker 5: really own the problem and understand it at a level
Speaker 5: maybe the data scientist down, So take some humility.
Speaker 1: To absolutely and I think that, Oh, go ahead, tru.
Speaker 6: I just want to, you know, just with respect to
Speaker 6: the performance of Genia solutions, maybe I just want to
Speaker 6: ask a question, with your permission, how many of you
Speaker 6: you think you currently use an AI solution as part
Speaker 6: of your clinical development? Fifty thirty percentage maybe in the room?
Speaker 6: How many of you have taken a decision based on
Speaker 6: a recommendation from an AI solution? Maybe thirty percentage of that?
Speaker 6: So you see that, right, like you know, when you
Speaker 6: when you you know the performance, you can come up
Speaker 6: with any type of KPI and OKR and metrics like that,
Speaker 6: but it's really up to them, those who actually have
Speaker 6: taken an action based on that feedback. So that human element,
Speaker 6: it's very difficult to capture, but it's very they actually
Speaker 6: brought that AA solution into their decision making. It augmented
Speaker 6: the team. It make the team move faster. That's not
Speaker 6: the case with all the use cases, but at least
Speaker 6: in some of the use cases, we see that here.
Speaker 6: So think about that, and it's an important that you should,
Speaker 6: every one of us to teach our colleagues, our executives
Speaker 6: about this. AA is not a magic wand it needs training,
Speaker 6: it needs help, and it's just another tool and you know,
Speaker 6: and it's moment right now, but think about its limitations.
Speaker 6: The real magic comes when your subject same is use
Speaker 6: this to I don't know, ten X or you know
Speaker 6: whatever improvement in their productivity. That's what you always think about.
Speaker 6: Metrics are great, but don't forget the human factor.
Speaker 1: Yeah. In fact, you know this term has been around
Speaker 1: for some time, maybe more than ten years. Citizen data scientists.
Speaker 1: I think in all practical purposes, it is coming to
Speaker 1: life that GENEI empowers each one of us as a
Speaker 1: data scientist. Think from that perspective when you're crafting your
Speaker 1: prompt you know, when you're asking a question, it just
Speaker 1: empowers so as as. And then the multi is disciplinary.
Speaker 1: I was attending FDA workshop last month in DC, and
Speaker 1: multidisciplinary was one of the key them that you know,
Speaker 1: you need teams which are multi disciplinary. You need data
Speaker 1: scientists or medical writers or medical writers who want to
Speaker 1: be data scientists.
Speaker 6: The lines are blurring.
Speaker 1: AI has just democratized these skills, these opportunities that the
Speaker 1: lines are blurring. So thank you again.
Speaker 4: With that, I.
Speaker 1: Will we'll move to call for action. Dave should be
Speaker 1: start with you the call for action.
Speaker 5: Sure, I would encourage everyone in this room to embrace
Speaker 5: the ambiguity and get started in trying to use AI
Speaker 5: and jen Ai in your in your data workflows. It
Speaker 5: is coming for all of us, and we have a
Speaker 5: responsibility to patients to make our trials and get therapies
Speaker 5: to patients as quickly and as effectively as impossible. So
Speaker 5: get started in using this in your workflows, bring ideas forward,
Speaker 5: test it out.
Speaker 1: Thank you to Jota.
Speaker 3: Please, my call for action is really simple, just start
Speaker 3: using it. I hear all the time, Oh I need
Speaker 3: a reference manual I need this to start using DPT.
Speaker 3: Did you have a reference manual to use your iPhone? No,
Speaker 3: it's just like that.
Speaker 2: Just start using it.
Speaker 6: I would say, you know behind you know, we're all
Speaker 6: in this business of finding medicines, you know, life saving medicines,
Speaker 6: and behind every data point that we handle the is
Speaker 6: a patient. So you know, be responsible about that. A
Speaker 6: I can make mistakes, you know. That's why we need semes.
Speaker 6: That's why we need I like this term thoughtful AI.
Speaker 6: You know you we need to have that filter. These
Speaker 6: are machines trained using massive amount of data. They can
Speaker 6: make mistakes, but it's an amazing tool that if you
Speaker 6: can tackle it to solve appropriate problems, and it's it's
Speaker 6: probably the most exciting time to be in our field
Speaker 6: as well, because we're all building it together. So enjoy
Speaker 6: the ride. I would say, for for sure, I would.
Speaker 7: Say, make sure you're finding that path is a very
Speaker 7: deep empathy.
Speaker 6: This is scary.
Speaker 7: This is unfamiliar terrain for folks who don't have a
Speaker 7: quantitative background. And if you're like a sociopath and you're
Speaker 7: incapable of empathy, just fake it. But somehow, you know,
Speaker 7: really understand where other folks are coming from first.
Speaker 1: Thank you very insightful. I would I would suggest that
Speaker 1: you know, I recall a little bit of my tenure
Speaker 1: at Behavioral Science that there is this theory called theory
Speaker 1: of planned behavior, which is about motivation, opportunity, and skills
Speaker 1: when you want a desired behavior, so I think AI
Speaker 1: education and sharing success stories uplifts the motivation provides the
Speaker 1: skills and just unlocks that, you know, adoption that we
Speaker 1: are looking for. Thank you everyone for being here and
Speaker 1: providing your perspective.