Speaker 1: I would like to now invite up meritisosis from ut
Speaker 1: Health San Antonio. Thanks so much for having me. So,
Speaker 1: you know, on using clinical research as a care option,
Speaker 1: it really relies on taking data from routine care and
Speaker 1: using it in research, and or taking the data from
Speaker 1: research and getting it back into routine care where it's
Speaker 1: relevant to routine care. And so I'm going to talk
Speaker 1: a little bit about the goes In's part of that,
Speaker 1: which is using routine care data for research. So doing
Speaker 1: this using routine care.
Speaker 2: Data for research.
Speaker 1: Literally has been the holy grail. It's been a dream
Speaker 1: since the first uses of computers in medicine, and doing
Speaker 1: so would hugely facilitate our ability to integrate care and
Speaker 1: research because both care and research are incredibly information intensive endeavors,
Speaker 1: and so people can't make the decisions that they need
Speaker 1: in either of those aspects without having the correct information. Unfortunately,
Speaker 1: we're not there yet, and I would say we're probably
Speaker 1: not nearly as close as we would like to be,
Speaker 1: even though there's a ton of standards and a ton
Speaker 1: of really good technology out there.
Speaker 2: So why is using.
Speaker 1: EHR data such a big deal in care and research? Well, first,
Speaker 1: is the cost of redundant data collections high. I'll never
Speaker 1: forget the day that I sat in a meeting at
Speaker 1: my old institution and one of my colleagues walked in
Speaker 1: and he said, oh my god, I just had a
Speaker 1: clinical trial visit and you're not going to believe this.
Speaker 1: And so all of us are clinical trials sitting around
Speaker 1: the table. We're like, doc, what happened. He looked around
Speaker 1: the table and he said, I got stuck twice. We said,
Speaker 1: what do you mean, you got stuck twice? Well, the
Speaker 1: study needed a blood draw and my doctor needed my
Speaker 1: annual physical and they couldn't use one piece of information
Speaker 1: for the other. They took two blood draws from me
Speaker 1: thirty minutes apart. Thus, we really want to end that
Speaker 1: redundant data collection and the redundant participant burden, and the
Speaker 1: redundant burden on sites. Other things like it's hard to
Speaker 1: find investigators to conduct studies. We have many, many, many,
Speaker 1: many questions that are still unanswered in clinical care. Evidence
Speaker 1: based medicine is not exactly evidence based, and I've seen
Speaker 1: estimates as high as about half of the cases, and
Speaker 1: I put some interesting examples on the bottom of the slide.
Speaker 1: The other is that just from a healthcare facility operations standpoint.
Speaker 2: I mean we care.
Speaker 1: About research at healthcare facility all across the spectrum. Sure,
Speaker 1: we want to make new therapeutics and discover new things,
Speaker 1: but dag on it, we also want our operations to
Speaker 1: perform better and to provide safer and better care. So
Speaker 1: we care about research in all of those aspects and
Speaker 1: have the same needs as others that conduct research in
Speaker 1: all of those aspects. So this is a big deal
Speaker 1: to the FDA using EHR data and research. They have
Speaker 1: released a slew of guidance recently to try and help
Speaker 1: folks do this. It's a big deal to an organization
Speaker 1: known as the Patient Centered Outcomes Research Institute or for COORY.
Speaker 1: One of its fundamental goals is this pocornet network, which
Speaker 1: is a network of a network of networks. In each
Speaker 1: of those networks is multiple health care facilities that provide
Speaker 1: electronic health record data into a large national data set
Speaker 1: that can be used to support research. It's also a
Speaker 1: big deal to National Institutes of Health. Not only do
Speaker 1: they have probably the largest single collection of EHR data
Speaker 1: for the COVID clinical coport, but they're also they have
Speaker 1: finished recently a pilot for broadening that up to all
Speaker 1: of EHR data, and making those data available in a
Speaker 1: research enclave for secondary use for research.
Speaker 2: So it's a big deal to a lot of people.
Speaker 1: Yet using EHR data in research has still very much
Speaker 1: eluded us from an automated, helpful standpoint. Now for years,
Speaker 1: we've gone over to the medical record and whether it's
Speaker 1: paper or electronic, looked at values, scanned through page by page,
Speaker 1: read the notes, get the data that we need, and
Speaker 1: enter it into an electronic data capture system. So everybody
Speaker 1: uses EHR data in that way, but dag on it.
Speaker 1: Not only is that expensive, but it's also the most
Speaker 1: error prone process in collecting and managing clinical research data.
Speaker 1: Error rates are common up to ten or twenty percent
Speaker 1: of the data, and that's high enough to flip a
Speaker 1: P value in a study. So we've been eluded until now.
Speaker 1: And I'm going to talk about two different examples ways
Speaker 1: of using DHR data in research and some very recent data.
Speaker 1: One of the studies, the data literally came out of
Speaker 1: the field last week. The other one is an older study.
Speaker 1: In the other two or three year in between. But
Speaker 1: I want to mention that in looking from prospective to
Speaker 1: retrospective studies. The gentleman that came up had a similar
Speaker 1: concern about asking if the policy was going to cover
Speaker 1: both clinical trials and observational research.
Speaker 2: So there's a spectrum.
Speaker 1: There that we need to cover, and there's different ways
Speaker 1: of using EHR data and thinking about use of EHR
Speaker 1: data when we move from observational studies to some interventional
Speaker 1: studies and maybe all the way forward to randomized control
Speaker 1: clinical studies. So the first example that I'm going to
Speaker 1: give this is the oldest of the studies. It was
Speaker 1: a seven thousand patient study.
Speaker 2: About fifty nine.
Speaker 1: Hundred of them were eligible for this data quality study,
Speaker 1: meaning that they had EHR data, and they also had
Speaker 1: participants self report data for thirty four medical conditions, a procedures, hospitalizations,
Speaker 1: and smoking status. And so we compared them. That was
Speaker 1: the goal of the study was to compare them and
Speaker 1: to measure the quality of patient reported data versus electronic
Speaker 1: health record data.
Speaker 2: And so when we.
Speaker 1: Did that, first of all, we found out that ninety
Speaker 1: four point five percent of the participants had one or
Speaker 1: more discrepancy. Okay, that's kind of high. We were hoping
Speaker 1: for a lower number, but you know, it's reasonable. It's
Speaker 1: a lot of data, right, lots of opportunities for disagreement
Speaker 1: in the data. What we also found was that ten
Speaker 1: out of the forty five assessed parameters, those thirty four conditions,
Speaker 1: A procedures, hospitalizations, and smoking status had less than eighty
Speaker 1: percent overall agreement. And I will tell you that of
Speaker 1: the many and we can discuss later in the hall
Speaker 1: if you want a more detailed discussion, but overall agreement
Speaker 1: is the easiest bar to get over for data quality measurement.
Speaker 1: So we then took six hundred and eleven of the
Speaker 1: participants and we interviewed them about the discrepancies to figure out, well,
Speaker 1: your EHR says this, and you reported this, help us
Speaker 1: understand the difference. And then after the inner with the participant,
Speaker 1: which was often quite conclusive in many of the cases,
Speaker 1: they're like, oh, yeah, no, that records from my healthcare
Speaker 1: record in Cannapolis, North Carolina, and I had them down
Speaker 1: in Florida five years ago. Oh okay, we get it.
Speaker 1: You showed up with a cannapolists er because you fell
Speaker 1: and you broke your ankle. So we for those six
Speaker 1: hundred and eleven patients, we interviewed them and in for
Speaker 1: the Arkansas data, and this data was collected in regions
Speaker 1: across two states, one in North Carolina and the other
Speaker 1: in the state of Arkansas. The sensitivity of the EHR
Speaker 1: data was less than eighty percent for thirty items. So
Speaker 1: sensitivity basically means the ability of the EHR to detect
Speaker 1: that a diagnosis, that a patient has a diagnosis, given
Speaker 1: that they have a diagnosis. So sensitivity of eighty percent
Speaker 1: means that I missed twenty percent of the diagnoses in
Speaker 1: the EHR, or that EHR itself miss twenty percent of
Speaker 1: the diagnoses. And we've so when we look at the data,
Speaker 1: the bars are the ninety five percent confidence intervals.
Speaker 2: This is the accuracy data.
Speaker 1: So it's only in that six hundred and eleven patient
Speaker 1: sample we've got the top limit of the ninety five
Speaker 1: percent confident center well meeting the area where we're in
Speaker 1: ninety five percent sure that that region covers the air
Speaker 1: rate that we're seeing so many of them. The top
Speaker 1: region of that confidence interval is under eighty percent, which
Speaker 1: if you talk to a clinical trialist who is writing
Speaker 1: a clinical studies report submitting this data to FDA, they
Speaker 1: would fall out of their chair if you said the
Speaker 1: aer rate of the data might be twenty percent.
Speaker 2: I also want to just point out.
Speaker 1: For those of you that are using hospitalization as an
Speaker 1: outcome man, EHR data is terrible for that.
Speaker 2: You have to have.
Speaker 1: Another option for that. So can you find the culprit?
Speaker 1: Why so many errors in the discrepancies? What do you
Speaker 1: think it could have been? Or which parameter up there
Speaker 1: is the culprit? Since we're running out of time, I'll
Speaker 1: make it easy. It's the sensitivity of the EHR data.
Speaker 2: But it's the sensitivity of the EHR data.
Speaker 1: Note North Carolina and Arkansas look very different from an
Speaker 1: EHR sensitivity basis, and that.
Speaker 2: Really caused a lot of concern.
Speaker 1: That kind of caused us to scratch our heads a bit,
Speaker 1: and that was a little painful. So the first thing
Speaker 1: that we concluded from that, and we did sort of
Speaker 1: find the smoking gun with the sensitivity for the EHR data.
Speaker 1: The data in Arkansas came from an integrated health system
Speaker 1: with community sites around the four corners of the state,
Speaker 1: but all the data from those community sites were integrated
Speaker 1: centrally at the Academic Medical Center in Arkansas, so it
Speaker 1: was all integrated and warehoused in.
Speaker 2: In North Carolina.
Speaker 1: It was very different. The data came primarily from community
Speaker 1: sites to echo another theme of reaching people out in
Speaker 1: the communities, and their care tended to be much more fragmented.
Speaker 1: There were two large health systems in the area we
Speaker 1: had EHR data from one. The other refused to participate
Speaker 1: in the study, and the care was incredibly fragmented for
Speaker 1: those patients. And that really for us when we look
Speaker 1: detailed into it. That's what we penned, given the limitations
Speaker 1: of an observational study, sort of what we think is
Speaker 1: the smoking gun is the care fragmentation and subsequent data
Speaker 1: fragmentation in the region.
Speaker 2: In general.
Speaker 1: The sensitivity of the self report data was higher than
Speaker 1: that for the EHR data, meaning maybe we should just
Speaker 1: ask the patients, or at a minimum, maybe we should
Speaker 1: ask the patients whenever we use EHR data, because asking
Speaker 1: thes can help increase the quality of the data. Together,
Speaker 1: huge benefit from us. And you know what, nobody we called,
Speaker 1: all those six hundred and eleven people in Arkansas and
Speaker 1: North Carolina, nobody we called was offended that we, you know,
Speaker 1: said that there was a discrepancy in their data. They
Speaker 1: were engaged in helping us figure it out and nobody
Speaker 1: that we called even for consent to participate in the
Speaker 1: broad study of the five nine hundred people, Nobody said, wait,
Speaker 1: you have my EHR data. What do you mean you're
Speaker 1: going to use my EHR data. The North Carolina people
Speaker 1: were prospectively consented for use of EHR data, and the
Speaker 1: Arkansas people were consented for use of HR data at
Speaker 1: the time of their self report collection. And so even
Speaker 1: in North Carolina, when that initial consent could have been
Speaker 1: seven years before, nobody came back and said, what I
Speaker 1: didn't consent to that? I don't remember that for hospitalization,
Speaker 1: neither source alone demonstrated good sensitivity. So for a hospitalization outcome,
Speaker 1: we need another option and the last sort of conclusion,
Speaker 1: and I put the reference for the PECORI technical report
Speaker 1: for the seven thousand patient study. It's our It's out
Speaker 1: on the web, the full report. So there's a lot
Speaker 1: of detail out there. But the one of the recommendations
Speaker 1: from the report echoes that that which has been said elsewhere,
Speaker 1: that the sensitivity and specificity of data in a multi
Speaker 1: center study from EHRs really needs to be assessed in
Speaker 1: each center. We saw a huge difference in the two sites.
Speaker 1: You remember the red bar in the were's waldo huge difference,
Speaker 1: and when you pool data all together, combining that the
Speaker 1: data all together can just wash out any of that
Speaker 1: and give you a very wrong answer. So then, what
Speaker 1: what does it mean for the use of e HR
Speaker 1: data to support integration of healthcare and research? And what
Speaker 1: about all those unanswered questions that we hoped that kind
Speaker 1: of answer using the e HR data was going to
Speaker 1: make it a little easier for us. Well incomes example
Speaker 1: number two, completely different situation. The first was taking EHR
Speaker 1: data and that it's already been collected in routine care
Speaker 1: and looking back over it using it for some totally
Speaker 1: different purpose.
Speaker 2: This next is.
Speaker 1: Something that's been proposed and pursued in clinical research trials
Speaker 1: in particular, but broader than trials is taking the data
Speaker 1: in the context of.
Speaker 2: A structured protocol, where.
Speaker 1: A site happens to enter the data in the EHR
Speaker 1: as the source. So the original documentation of the data,
Speaker 1: given that the study is done in the context of
Speaker 1: a structured clinical protocol, maybe those data are higher quality
Speaker 1: because there is that structure there. So we did this
Speaker 1: in two oncology studies conducted in the United States. At
Speaker 1: one site, it happened to be our site in Texas
Speaker 1: thanks to a group software known as Incartes or in
Speaker 1: coup and also working with these two studies were SWAG
Speaker 1: Studies used formerly known as the Southwest Oncology Group. It's
Speaker 1: an NIH funded cancer cooperative group.
Speaker 2: The next the.
Speaker 1: Single study was a study conducted at Shiba Medical Center
Speaker 1: in Israel, so outside the US. Didn't even use the
Speaker 1: healthcare the Health Level seven fire interoperability standards, so a
Speaker 1: very different way to even access the data and the
Speaker 1: electronic health records. So what we found is in all
Speaker 1: cases in the three separate studies, we measured a zero
Speaker 1: percent error rate. Literally found no errors in the data.
Speaker 1: And it was mainly labs, meds, vital signs, this simple
Speaker 1: kind of common data that was mapped, not the full
Speaker 1: study CRF.
Speaker 2: And even in.
Speaker 1: Just the few patients that we used, really reasonable confidence
Speaker 1: intervals and competence intervals that don't overlap between the error
Speaker 1: rate real error rate as in all all calls errors,
Speaker 1: all errors counted in the data compared to an adjudicated
Speaker 1: gold standard, very different than the EDC error rates.
Speaker 2: So there's an indication there.
Speaker 1: Granted, these are observational data quality studies themselves, but there's
Speaker 1: an indication that actually using the data from the EHR
Speaker 1: that's collected in the context of a structured protocol can
Speaker 1: better the data quality that we get from from today's standards.
Speaker 1: So comparing the examples we just did that, I'll skip
Speaker 1: that and save more time for questions, but it's there
Speaker 1: in the slides if you need it.
Speaker 2: Some closing thoughts.
Speaker 1: First, the EHR to DC and what we've seen so
Speaker 1: far is associated with a better error rate. The program
Speaker 1: that we're doing this under is called ASRWD. It's Ancillary
Speaker 1: Studies to evaluate real world data quality, and we're working
Speaker 1: with sponsors and technology providers who are willing to have their.
Speaker 2: Data intopend evaluated.
Speaker 1: They send us all the discrepancies, we adjudicate them one
Speaker 1: by one, go through them, call a site all sorts
Speaker 1: of stuff, have the site look back in the EHR,
Speaker 1: so really an independent evaluation, and then we adjudicate a
Speaker 1: gold standard based on that full medical record review at
Speaker 1: the site by the site study coordinator.
Speaker 2: And so in doing.
Speaker 1: That, the ACE is just starting to generate this look
Speaker 1: at what we can achieve with integration of care and
Speaker 1: research at a site where we're reducing what's a huge
Speaker 1: burden for a site in terms of data collection. Hopefully
Speaker 1: we're reducing the I got stuck twice phenomenon from our
Speaker 1: patients in clinical studies. So making sort of the final
Speaker 1: point that when all sizes, it's not all equal, right
Speaker 1: when you compare use of EHR data using it retrospectively
Speaker 1: where it was collected in the course of routine care
Speaker 1: outside the context of a clinical studies, completely different from
Speaker 1: when the site has the context of that clinical study
Speaker 1: protocol and may even have aspects of that protocol implemented
Speaker 1: in their EHR. And as we as sites all get
Speaker 1: better and more fascial with our EHR systems, that aspect
Speaker 1: is going to get even better. So we have ways
Speaker 1: to go to catch up with Denise and what she's
Speaker 1: implemented at Duke. With respect to that last thing I'll
Speaker 1: mention is the ASRWD program is ongoing.
Speaker 2: We've got two years left in it.
Speaker 1: We're always happy to do these independent assessments or work
Speaker 1: with somebody that's got both to do an independent assessment
Speaker 1: of the data. We're doing this so that technology providers, sponsors, regulators,
Speaker 1: everybody has a shared pool of information that's cross site,
Speaker 1: cross therapeutic area cross study, so that they can have
Speaker 1: much better information about what the actual quality of the
Speaker 1: data is moving forward, and that we can get best
Speaker 1: practices out of how to implement this and keep the
Speaker 1: error slow and uh, Thursday or I guess it's actually
Speaker 1: Wednesday afternoon. Sorry, we'll talk more about how we actually
Speaker 1: measured it and measurement methods if folks are interested in
Speaker 1: doing that. Did I mention ACE is still recruiting. I'm
Speaker 1: just saying I do in the interest of fully disclosing there.
Speaker 1: This work is really hard to get funded, right because
Speaker 1: very few people care about data quality and at this
Speaker 1: little detail level and want to pay extra money to
Speaker 1: get the studies done. So huge shout out to Cory
Speaker 1: for the funding for the older study, n CI National
Speaker 1: Institutes of Health for the cancer study, or our Cancer
Speaker 1: Institute at Texas, the Borough's Welcome Fund which which funds
Speaker 1: the Coordinating Center for the ASR w D program, UH
Speaker 1: in Coop and Karts Platform for e h R to
Speaker 1: e d C and Yonahlink for their platform and in
Speaker 1: kind contribution uh the data. Huge thank you to these guys.
Speaker 1: I'm happy for questions if we have time.
Speaker 2: We do have time.
Speaker 1: For one or two questions.
Speaker 3: Go ahead. I thank you for that wonderful study and presentation.
Speaker 3: One thing that I would add, if I might, to
Speaker 3: the general topic of HR data for research is the
Speaker 3: insecurity of vhrs, the most hacked and breached data sources
Speaker 3: in the United States. And I understand that the Office
Speaker 3: of National Coordinator is promoting TEFKA, the Trusted Exchange Framework
Speaker 3: and Common Agreement Program as a way of making more
Speaker 3: EHR data using tokens presumably to link to put together
Speaker 3: the pieces of patient data, so you have supposedly single
Speaker 3: individual data for research purposes, for pandemic use, for major
Speaker 3: healthcare uses. And yet what they're really doing is cascading
Speaker 3: the error that can come in, not just the kind
Speaker 3: of errors that you were talking about, but from hackle
Speaker 3: actual hacking. And I think there's another The White House
Speaker 3: Office of National Security does have a major concern about
Speaker 3: that as a national security risk, because TEFKA is really
Speaker 3: just expanding the contact surface and the opportunity for bad
Speaker 3: actors to throw wrong data in. But what could be
Speaker 3: happening even now, as I understand it, is that bad
Speaker 3: actors are doing precisely that they can get in and
Speaker 3: without anyone knowing it, just alter data here and there,
Speaker 3: and so researchers have really no sense of what they're what.
Speaker 3: It might seem accurate, but anyway, that's just another concern
Speaker 3: the whole security question.
Speaker 1: So yet, yes and no, I mean definitely there are
Speaker 1: those of us who have been hacked who don't even
Speaker 1: know it yet, and there are hundreds of thousands of
Speaker 1: hacking tries, if the numbers that low across our country
Speaker 1: today in EHR systems. I will say though, that the
Speaker 1: hacker's primary interest is selling the data or getting getting
Speaker 1: funds for the data, and they don't get much for
Speaker 1: going in and playfully changing values in our EHRs. It's
Speaker 1: when they sell the identifiers on the record. So there's
Speaker 1: a lack of a financial motivation so much from.
Speaker 3: Concern of state actors deliberately trying to create problems in
Speaker 3: the United States.
Speaker 1: Yep. Well, there's those of us that haven't been hacked unfortunately,
Speaker 1: will uh, we'll deal with it in the future and
Speaker 1: and have you know, be hacked at some point most
Speaker 1: of us.
Speaker 4: So thank you.
Speaker 2: That was great.
Speaker 4: So my question relates to practice based research networks, and
Speaker 4: I just got a big thirty million dollar fund we're
Speaker 4: all going after.
Speaker 2: My question really.
Speaker 4: Relates to the quality of data in clinical practice sites
Speaker 4: family medicine, internal medicine, primary care, general pediatrics. What we've
Speaker 4: found with our PBRN is that smaller sites that aren't
Speaker 4: necessarily part of a larger health system actually need help
Speaker 4: understanding how to make data driven decisions. And so if
Speaker 4: data is not all that accurate going into their electronic
Speaker 4: health record to start with, and we're trying to help
Speaker 4: them build capacity for data driven decisions as part of
Speaker 4: building research capacity. Do you have recommendations for those of
Speaker 4: us who are coordinating administering large networks of primary care
Speaker 4: practice to help them develop capacity for data quality.
Speaker 1: Yeah, so that is a really big issue the pbr
Speaker 1: ns and pbr in like sites that I've worked with.
Speaker 1: Instead of saying sure, Meredith will send you that, let
Speaker 1: me get you in the queue, they say, Meredith, if
Speaker 1: you want that data, come down here yourself and get it.
Speaker 1: You're going to have to email it to yourself. Now,
Speaker 1: not that we would email it, but you get the picture.
Speaker 1: It's more of a self service kind of thing because
Speaker 1: they're usually their EHR is hosted and someone else does
Speaker 1: all the work on it for them. They wouldn't know
Speaker 1: how to go in and extract the data many of
Speaker 1: those sites if they had to. The other thing I'll
Speaker 1: say is that there's sort of a fundamental principle in
Speaker 1: data quality that information that's used is of better quality.
Speaker 1: So your point about if we can just get them
Speaker 1: the kool aid and get them hooked on the power
Speaker 1: of being able to make data driven decisions, then this
Speaker 1: will be one. The case of data quality will be one. Unfortunately,
Speaker 1: it takes an awful lot of work to get the
Speaker 1: skills in data analytics, not just framing the questions, not
Speaker 1: just programming, not just formatting the reports, not just formatting
Speaker 1: the data visualizations, or even getting it to tuning an
Speaker 1: AI algorithm.
Speaker 2: Heaven forbid.
Speaker 1: That's a lot of skill, and those skills are much
Speaker 1: much less common way out in the community and in
Speaker 1: today's world. It's a hand to hand combat thing. It's
Speaker 1: a one side at a time helping them figure out
Speaker 1: what data day themselves can get out of their EHR
Speaker 1: and what questions that they have have to better manage
Speaker 1: their facility or care quality they can make based on
Speaker 1: the data that they have. Thanks for the question. We're
Speaker 1: at time, So I'm just gonna Nope. You can ask
Speaker 1: your question, and I'm just going to invite you to
Speaker 1: to condense it as much as possible.
Speaker 2: Then we'll move on to our next.
Speaker 5: Sure, this was fantastic and disturbing at the same time.
Speaker 2: Thank you.
Speaker 5: I think yours was great. I guess I'm just interested
Speaker 5: in hearing your thoughts on our industry's latest shiny penny,
Speaker 5: and that is the impact of AI on such a
Speaker 5: data set.
Speaker 1: Yeah, sure, so you know, because that's a short question. Yeah,
Speaker 1: what I'll say, and try to be short, is that
Speaker 1: the penny is really not so shiny, and AI, like
Speaker 1: any algorithm, takes work and takes proving and takes ongoing
Speaker 1: human monitoring. When we do the fire based data extraction,
Speaker 1: there's a study coordinator sitting there in the middle that
Speaker 1: looks at it and says, oh, yeah, okay, that was
Speaker 1: the data from that visit send it through the same thing.
Speaker 1: When with the AI that we're working with for adverse
Speaker 1: event identification or fact extraction out of unstructured data, we
Speaker 1: make it a human in the loop and supervised learning
Speaker 1: where the human looks at the AI ongoing and once
Speaker 1: you're confident and the data source is stable, you can
Speaker 1: back off a little and go to just things that
Speaker 1: are at a lower confidence. But yeah, it's it's not
Speaker 1: the shiny bullet short as I could be. Sorry, perfect,
Speaker 1: thank you,