WEBVTT 1 00:00:00.040 --> 00:00:03.480 I would like to now invite up meritisosis from ut 2 00:00:03.720 --> 00:00:10.439 Health San Antonio. Thanks so much for having me. So, 3 00:00:10.599 --> 00:00:13.919 you know, on using clinical research as a care option, 4 00:00:14.000 --> 00:00:18.280 it really relies on taking data from routine care and 5 00:00:18.399 --> 00:00:21.559 using it in research, and or taking the data from 6 00:00:21.600 --> 00:00:24.960 research and getting it back into routine care where it's 7 00:00:25.000 --> 00:00:27.760 relevant to routine care. And so I'm going to talk 8 00:00:27.800 --> 00:00:29.879 a little bit about the goes In's part of that, 9 00:00:30.280 --> 00:00:36.439 which is using routine care data for research. So doing 10 00:00:36.479 --> 00:00:38.079 this using routine care. 11 00:00:37.960 --> 00:00:39.359 Data for research. 12 00:00:39.439 --> 00:00:43.359 Literally has been the holy grail. It's been a dream 13 00:00:43.679 --> 00:00:48.119 since the first uses of computers in medicine, and doing 14 00:00:48.200 --> 00:00:52.119 so would hugely facilitate our ability to integrate care and 15 00:00:52.240 --> 00:00:58.159 research because both care and research are incredibly information intensive endeavors, 16 00:00:58.679 --> 00:01:01.560 and so people can't make the decisions that they need 17 00:01:01.600 --> 00:01:09.159 in either of those aspects without having the correct information. Unfortunately, 18 00:01:09.239 --> 00:01:12.719 we're not there yet, and I would say we're probably 19 00:01:12.760 --> 00:01:15.400 not nearly as close as we would like to be, 20 00:01:15.560 --> 00:01:19.799 even though there's a ton of standards and a ton 21 00:01:19.840 --> 00:01:21.599 of really good technology out there. 22 00:01:23.760 --> 00:01:25.959 So why is using. 23 00:01:25.840 --> 00:01:30.799 EHR data such a big deal in care and research? Well, first, 24 00:01:30.959 --> 00:01:33.959 is the cost of redundant data collections high. I'll never 25 00:01:34.040 --> 00:01:36.760 forget the day that I sat in a meeting at 26 00:01:36.840 --> 00:01:40.280 my old institution and one of my colleagues walked in 27 00:01:40.359 --> 00:01:41.959 and he said, oh my god, I just had a 28 00:01:42.000 --> 00:01:44.239 clinical trial visit and you're not going to believe this. 29 00:01:44.680 --> 00:01:46.760 And so all of us are clinical trials sitting around 30 00:01:46.760 --> 00:01:49.599 the table. We're like, doc, what happened. He looked around 31 00:01:49.640 --> 00:01:52.959 the table and he said, I got stuck twice. We said, 32 00:01:52.959 --> 00:01:55.599 what do you mean, you got stuck twice? Well, the 33 00:01:55.719 --> 00:01:58.480 study needed a blood draw and my doctor needed my 34 00:01:58.519 --> 00:02:01.760 annual physical and they couldn't use one piece of information 35 00:02:01.840 --> 00:02:04.760 for the other. They took two blood draws from me 36 00:02:04.879 --> 00:02:09.240 thirty minutes apart. Thus, we really want to end that 37 00:02:09.639 --> 00:02:13.280 redundant data collection and the redundant participant burden, and the 38 00:02:13.319 --> 00:02:17.439 redundant burden on sites. Other things like it's hard to 39 00:02:17.439 --> 00:02:21.199 find investigators to conduct studies. We have many, many, many, 40 00:02:21.199 --> 00:02:24.960 many questions that are still unanswered in clinical care. Evidence 41 00:02:25.000 --> 00:02:28.800 based medicine is not exactly evidence based, and I've seen 42 00:02:28.919 --> 00:02:31.360 estimates as high as about half of the cases, and 43 00:02:31.400 --> 00:02:34.919 I put some interesting examples on the bottom of the slide. 44 00:02:35.599 --> 00:02:39.360 The other is that just from a healthcare facility operations standpoint. 45 00:02:39.400 --> 00:02:40.120 I mean we care. 46 00:02:39.960 --> 00:02:44.080 About research at healthcare facility all across the spectrum. Sure, 47 00:02:44.120 --> 00:02:47.360 we want to make new therapeutics and discover new things, 48 00:02:47.599 --> 00:02:49.639 but dag on it, we also want our operations to 49 00:02:49.719 --> 00:02:53.919 perform better and to provide safer and better care. So 50 00:02:54.000 --> 00:02:57.400 we care about research in all of those aspects and 51 00:02:57.479 --> 00:03:00.319 have the same needs as others that conduct research in 52 00:03:00.360 --> 00:03:04.280 all of those aspects. So this is a big deal 53 00:03:04.360 --> 00:03:07.280 to the FDA using EHR data and research. They have 54 00:03:07.400 --> 00:03:11.280 released a slew of guidance recently to try and help 55 00:03:11.319 --> 00:03:15.280 folks do this. It's a big deal to an organization 56 00:03:15.400 --> 00:03:18.719 known as the Patient Centered Outcomes Research Institute or for COORY. 57 00:03:21.039 --> 00:03:24.560 One of its fundamental goals is this pocornet network, which 58 00:03:24.599 --> 00:03:28.159 is a network of a network of networks. In each 59 00:03:28.199 --> 00:03:31.319 of those networks is multiple health care facilities that provide 60 00:03:31.319 --> 00:03:34.800 electronic health record data into a large national data set 61 00:03:35.159 --> 00:03:38.039 that can be used to support research. It's also a 62 00:03:38.080 --> 00:03:40.800 big deal to National Institutes of Health. Not only do 63 00:03:40.879 --> 00:03:44.360 they have probably the largest single collection of EHR data 64 00:03:44.400 --> 00:03:47.439 for the COVID clinical coport, but they're also they have 65 00:03:47.520 --> 00:03:51.159 finished recently a pilot for broadening that up to all 66 00:03:51.199 --> 00:03:54.360 of EHR data, and making those data available in a 67 00:03:54.439 --> 00:03:56.800 research enclave for secondary use for research. 68 00:03:57.960 --> 00:03:59.439 So it's a big deal to a lot of people. 69 00:03:59.560 --> 00:04:04.759 Yet using EHR data in research has still very much 70 00:04:04.960 --> 00:04:09.360 eluded us from an automated, helpful standpoint. Now for years, 71 00:04:09.680 --> 00:04:11.639 we've gone over to the medical record and whether it's 72 00:04:11.680 --> 00:04:15.639 paper or electronic, looked at values, scanned through page by page, 73 00:04:15.680 --> 00:04:18.160 read the notes, get the data that we need, and 74 00:04:18.319 --> 00:04:21.680 enter it into an electronic data capture system. So everybody 75 00:04:21.839 --> 00:04:25.079 uses EHR data in that way, but dag on it. 76 00:04:25.160 --> 00:04:27.399 Not only is that expensive, but it's also the most 77 00:04:27.519 --> 00:04:31.199 error prone process in collecting and managing clinical research data. 78 00:04:31.800 --> 00:04:35.199 Error rates are common up to ten or twenty percent 79 00:04:35.240 --> 00:04:37.519 of the data, and that's high enough to flip a 80 00:04:37.519 --> 00:04:43.920 P value in a study. So we've been eluded until now. 81 00:04:44.639 --> 00:04:49.720 And I'm going to talk about two different examples ways 82 00:04:49.759 --> 00:04:54.000 of using DHR data in research and some very recent data. 83 00:04:54.399 --> 00:04:56.439 One of the studies, the data literally came out of 84 00:04:56.439 --> 00:04:59.600 the field last week. The other one is an older study. 85 00:04:59.639 --> 00:05:01.839 In the other two or three year in between. But 86 00:05:01.920 --> 00:05:05.399 I want to mention that in looking from prospective to 87 00:05:05.560 --> 00:05:08.959 retrospective studies. The gentleman that came up had a similar 88 00:05:09.000 --> 00:05:11.600 concern about asking if the policy was going to cover 89 00:05:11.639 --> 00:05:14.120 both clinical trials and observational research. 90 00:05:14.560 --> 00:05:16.079 So there's a spectrum. 91 00:05:15.560 --> 00:05:19.240 There that we need to cover, and there's different ways 92 00:05:19.399 --> 00:05:22.800 of using EHR data and thinking about use of EHR 93 00:05:22.959 --> 00:05:27.600 data when we move from observational studies to some interventional 94 00:05:27.639 --> 00:05:30.920 studies and maybe all the way forward to randomized control 95 00:05:30.920 --> 00:05:35.360 clinical studies. So the first example that I'm going to 96 00:05:35.360 --> 00:05:39.399 give this is the oldest of the studies. It was 97 00:05:39.439 --> 00:05:40.920 a seven thousand patient study. 98 00:05:40.959 --> 00:05:42.319 About fifty nine. 99 00:05:42.160 --> 00:05:45.800 Hundred of them were eligible for this data quality study, 100 00:05:46.160 --> 00:05:48.600 meaning that they had EHR data, and they also had 101 00:05:48.639 --> 00:05:53.720 participants self report data for thirty four medical conditions, a procedures, hospitalizations, 102 00:05:53.720 --> 00:05:58.319 and smoking status. And so we compared them. That was 103 00:05:58.360 --> 00:06:00.120 the goal of the study was to compare them and 104 00:06:00.199 --> 00:06:03.800 to measure the quality of patient reported data versus electronic 105 00:06:03.839 --> 00:06:04.720 health record data. 106 00:06:05.959 --> 00:06:07.079 And so when we. 107 00:06:07.040 --> 00:06:09.160 Did that, first of all, we found out that ninety 108 00:06:09.160 --> 00:06:11.279 four point five percent of the participants had one or 109 00:06:11.279 --> 00:06:15.079 more discrepancy. Okay, that's kind of high. We were hoping 110 00:06:15.120 --> 00:06:17.360 for a lower number, but you know, it's reasonable. It's 111 00:06:17.360 --> 00:06:20.560 a lot of data, right, lots of opportunities for disagreement 112 00:06:20.600 --> 00:06:23.879 in the data. What we also found was that ten 113 00:06:23.959 --> 00:06:26.920 out of the forty five assessed parameters, those thirty four conditions, 114 00:06:26.920 --> 00:06:32.319 A procedures, hospitalizations, and smoking status had less than eighty 115 00:06:32.360 --> 00:06:35.079 percent overall agreement. And I will tell you that of 116 00:06:35.439 --> 00:06:37.879 the many and we can discuss later in the hall 117 00:06:37.920 --> 00:06:40.800 if you want a more detailed discussion, but overall agreement 118 00:06:41.000 --> 00:06:45.519 is the easiest bar to get over for data quality measurement. 119 00:06:46.920 --> 00:06:49.399 So we then took six hundred and eleven of the 120 00:06:49.439 --> 00:06:54.279 participants and we interviewed them about the discrepancies to figure out, well, 121 00:06:54.519 --> 00:06:57.480 your EHR says this, and you reported this, help us 122 00:06:57.519 --> 00:07:01.240 understand the difference. And then after the inner with the participant, 123 00:07:01.639 --> 00:07:04.439 which was often quite conclusive in many of the cases, 124 00:07:04.480 --> 00:07:07.160 they're like, oh, yeah, no, that records from my healthcare 125 00:07:07.199 --> 00:07:10.639 record in Cannapolis, North Carolina, and I had them down 126 00:07:10.680 --> 00:07:14.199 in Florida five years ago. Oh okay, we get it. 127 00:07:14.279 --> 00:07:17.040 You showed up with a cannapolists er because you fell 128 00:07:17.079 --> 00:07:20.480 and you broke your ankle. So we for those six 129 00:07:20.600 --> 00:07:24.720 hundred and eleven patients, we interviewed them and in for 130 00:07:24.759 --> 00:07:28.439 the Arkansas data, and this data was collected in regions 131 00:07:28.439 --> 00:07:31.639 across two states, one in North Carolina and the other 132 00:07:31.800 --> 00:07:36.839 in the state of Arkansas. The sensitivity of the EHR 133 00:07:37.000 --> 00:07:41.399 data was less than eighty percent for thirty items. So 134 00:07:41.439 --> 00:07:46.800 sensitivity basically means the ability of the EHR to detect 135 00:07:47.199 --> 00:07:50.759 that a diagnosis, that a patient has a diagnosis, given 136 00:07:50.800 --> 00:07:55.000 that they have a diagnosis. So sensitivity of eighty percent 137 00:07:55.360 --> 00:07:58.399 means that I missed twenty percent of the diagnoses in 138 00:07:58.439 --> 00:08:02.240 the EHR, or that EHR itself miss twenty percent of 139 00:08:02.240 --> 00:08:06.160 the diagnoses. And we've so when we look at the data, 140 00:08:07.120 --> 00:08:09.519 the bars are the ninety five percent confidence intervals. 141 00:08:09.519 --> 00:08:11.040 This is the accuracy data. 142 00:08:11.120 --> 00:08:13.639 So it's only in that six hundred and eleven patient 143 00:08:13.839 --> 00:08:19.759 sample we've got the top limit of the ninety five 144 00:08:19.800 --> 00:08:21.959 percent confident center well meeting the area where we're in 145 00:08:22.040 --> 00:08:26.480 ninety five percent sure that that region covers the air 146 00:08:26.560 --> 00:08:30.319 rate that we're seeing so many of them. The top 147 00:08:30.399 --> 00:08:34.759 region of that confidence interval is under eighty percent, which 148 00:08:34.840 --> 00:08:37.559 if you talk to a clinical trialist who is writing 149 00:08:37.559 --> 00:08:41.960 a clinical studies report submitting this data to FDA, they 150 00:08:41.960 --> 00:08:44.759 would fall out of their chair if you said the 151 00:08:44.840 --> 00:08:46.840 aer rate of the data might be twenty percent. 152 00:08:49.240 --> 00:08:50.879 I also want to just point out. 153 00:08:50.720 --> 00:08:53.360 For those of you that are using hospitalization as an 154 00:08:53.360 --> 00:08:58.159 outcome man, EHR data is terrible for that. 155 00:08:58.919 --> 00:08:59.679 You have to have. 156 00:08:59.639 --> 00:09:03.799 Another option for that. So can you find the culprit? 157 00:09:05.120 --> 00:09:09.000 Why so many errors in the discrepancies? What do you 158 00:09:09.039 --> 00:09:11.440 think it could have been? Or which parameter up there 159 00:09:11.519 --> 00:09:14.240 is the culprit? Since we're running out of time, I'll 160 00:09:14.279 --> 00:09:18.279 make it easy. It's the sensitivity of the EHR data. 161 00:09:19.200 --> 00:09:21.240 But it's the sensitivity of the EHR data. 162 00:09:21.279 --> 00:09:26.039 Note North Carolina and Arkansas look very different from an 163 00:09:26.039 --> 00:09:29.759 EHR sensitivity basis, and that. 164 00:09:29.639 --> 00:09:31.399 Really caused a lot of concern. 165 00:09:31.440 --> 00:09:33.679 That kind of caused us to scratch our heads a bit, 166 00:09:34.240 --> 00:09:38.639 and that was a little painful. So the first thing 167 00:09:38.679 --> 00:09:41.279 that we concluded from that, and we did sort of 168 00:09:41.279 --> 00:09:44.399 find the smoking gun with the sensitivity for the EHR data. 169 00:09:44.440 --> 00:09:48.000 The data in Arkansas came from an integrated health system 170 00:09:48.360 --> 00:09:52.679 with community sites around the four corners of the state, 171 00:09:53.320 --> 00:09:56.519 but all the data from those community sites were integrated 172 00:09:56.600 --> 00:10:00.480 centrally at the Academic Medical Center in Arkansas, so it 173 00:10:00.519 --> 00:10:04.679 was all integrated and warehoused in. 174 00:10:03.519 --> 00:10:04.399 In North Carolina. 175 00:10:04.440 --> 00:10:08.120 It was very different. The data came primarily from community 176 00:10:08.159 --> 00:10:11.399 sites to echo another theme of reaching people out in 177 00:10:11.440 --> 00:10:16.159 the communities, and their care tended to be much more fragmented. 178 00:10:16.559 --> 00:10:18.919 There were two large health systems in the area we 179 00:10:18.960 --> 00:10:21.919 had EHR data from one. The other refused to participate 180 00:10:22.000 --> 00:10:26.279 in the study, and the care was incredibly fragmented for 181 00:10:26.399 --> 00:10:30.320 those patients. And that really for us when we look 182 00:10:30.440 --> 00:10:33.679 detailed into it. That's what we penned, given the limitations 183 00:10:33.679 --> 00:10:36.879 of an observational study, sort of what we think is 184 00:10:36.919 --> 00:10:40.559 the smoking gun is the care fragmentation and subsequent data 185 00:10:40.559 --> 00:10:42.039 fragmentation in the region. 186 00:10:44.600 --> 00:10:45.240 In general. 187 00:10:45.320 --> 00:10:48.440 The sensitivity of the self report data was higher than 188 00:10:48.519 --> 00:10:52.399 that for the EHR data, meaning maybe we should just 189 00:10:52.480 --> 00:10:56.399 ask the patients, or at a minimum, maybe we should 190 00:10:56.440 --> 00:10:59.759 ask the patients whenever we use EHR data, because asking 191 00:10:59.799 --> 00:11:02.879 thes can help increase the quality of the data. Together, 192 00:11:03.759 --> 00:11:06.559 huge benefit from us. And you know what, nobody we called, 193 00:11:06.600 --> 00:11:08.799 all those six hundred and eleven people in Arkansas and 194 00:11:08.840 --> 00:11:13.320 North Carolina, nobody we called was offended that we, you know, 195 00:11:13.399 --> 00:11:15.840 said that there was a discrepancy in their data. They 196 00:11:15.879 --> 00:11:19.480 were engaged in helping us figure it out and nobody 197 00:11:19.519 --> 00:11:21.879 that we called even for consent to participate in the 198 00:11:21.879 --> 00:11:25.960 broad study of the five nine hundred people, Nobody said, wait, 199 00:11:26.240 --> 00:11:29.279 you have my EHR data. What do you mean you're 200 00:11:29.279 --> 00:11:32.799 going to use my EHR data. The North Carolina people 201 00:11:32.840 --> 00:11:36.639 were prospectively consented for use of EHR data, and the 202 00:11:36.759 --> 00:11:39.960 Arkansas people were consented for use of HR data at 203 00:11:39.960 --> 00:11:42.919 the time of their self report collection. And so even 204 00:11:42.960 --> 00:11:45.679 in North Carolina, when that initial consent could have been 205 00:11:45.799 --> 00:11:49.960 seven years before, nobody came back and said, what I 206 00:11:50.240 --> 00:11:55.559 didn't consent to that? I don't remember that for hospitalization, 207 00:11:55.840 --> 00:12:00.519 neither source alone demonstrated good sensitivity. So for a hospitalization outcome, 208 00:12:01.039 --> 00:12:05.960 we need another option and the last sort of conclusion, 209 00:12:06.000 --> 00:12:09.080 and I put the reference for the PECORI technical report 210 00:12:09.159 --> 00:12:11.879 for the seven thousand patient study. It's our It's out 211 00:12:11.879 --> 00:12:13.440 on the web, the full report. So there's a lot 212 00:12:13.440 --> 00:12:16.960 of detail out there. But the one of the recommendations 213 00:12:16.960 --> 00:12:21.600 from the report echoes that that which has been said elsewhere, 214 00:12:21.879 --> 00:12:25.200 that the sensitivity and specificity of data in a multi 215 00:12:25.200 --> 00:12:27.879 center study from EHRs really needs to be assessed in 216 00:12:27.919 --> 00:12:32.879 each center. We saw a huge difference in the two sites. 217 00:12:32.919 --> 00:12:36.320 You remember the red bar in the were's waldo huge difference, 218 00:12:37.159 --> 00:12:40.360 and when you pool data all together, combining that the 219 00:12:40.480 --> 00:12:43.639 data all together can just wash out any of that 220 00:12:43.720 --> 00:12:48.639 and give you a very wrong answer. So then, what 221 00:12:48.759 --> 00:12:51.200 what does it mean for the use of e HR 222 00:12:51.360 --> 00:12:55.799 data to support integration of healthcare and research? And what 223 00:12:55.879 --> 00:12:58.679 about all those unanswered questions that we hoped that kind 224 00:12:58.679 --> 00:13:00.480 of answer using the e HR data was going to 225 00:13:00.480 --> 00:13:04.559 make it a little easier for us. Well incomes example 226 00:13:04.639 --> 00:13:08.720 number two, completely different situation. The first was taking EHR 227 00:13:08.840 --> 00:13:12.000 data and that it's already been collected in routine care 228 00:13:12.519 --> 00:13:15.080 and looking back over it using it for some totally 229 00:13:15.080 --> 00:13:15.759 different purpose. 230 00:13:16.799 --> 00:13:17.679 This next is. 231 00:13:17.600 --> 00:13:21.799 Something that's been proposed and pursued in clinical research trials 232 00:13:21.799 --> 00:13:25.600 in particular, but broader than trials is taking the data 233 00:13:25.759 --> 00:13:26.960 in the context of. 234 00:13:26.879 --> 00:13:28.639 A structured protocol, where. 235 00:13:28.440 --> 00:13:31.519 A site happens to enter the data in the EHR 236 00:13:31.639 --> 00:13:34.519 as the source. So the original documentation of the data, 237 00:13:35.159 --> 00:13:37.639 given that the study is done in the context of 238 00:13:37.679 --> 00:13:41.879 a structured clinical protocol, maybe those data are higher quality 239 00:13:42.240 --> 00:13:45.600 because there is that structure there. So we did this 240 00:13:46.440 --> 00:13:49.799 in two oncology studies conducted in the United States. At 241 00:13:49.799 --> 00:13:51.799 one site, it happened to be our site in Texas 242 00:13:53.399 --> 00:13:57.320 thanks to a group software known as Incartes or in 243 00:13:57.399 --> 00:14:01.799 coup and also working with these two studies were SWAG 244 00:14:02.120 --> 00:14:05.799 Studies used formerly known as the Southwest Oncology Group. It's 245 00:14:05.799 --> 00:14:08.559 an NIH funded cancer cooperative group. 246 00:14:09.000 --> 00:14:10.519 The next the. 247 00:14:10.559 --> 00:14:13.639 Single study was a study conducted at Shiba Medical Center 248 00:14:13.879 --> 00:14:17.919 in Israel, so outside the US. Didn't even use the 249 00:14:18.840 --> 00:14:23.039 healthcare the Health Level seven fire interoperability standards, so a 250 00:14:23.159 --> 00:14:26.440 very different way to even access the data and the 251 00:14:26.480 --> 00:14:30.519 electronic health records. So what we found is in all 252 00:14:30.600 --> 00:14:35.840 cases in the three separate studies, we measured a zero 253 00:14:35.960 --> 00:14:39.720 percent error rate. Literally found no errors in the data. 254 00:14:40.159 --> 00:14:43.759 And it was mainly labs, meds, vital signs, this simple 255 00:14:43.840 --> 00:14:46.279 kind of common data that was mapped, not the full 256 00:14:46.320 --> 00:14:47.679 study CRF. 257 00:14:48.840 --> 00:14:49.679 And even in. 258 00:14:49.679 --> 00:14:53.159 Just the few patients that we used, really reasonable confidence 259 00:14:53.159 --> 00:14:56.559 intervals and competence intervals that don't overlap between the error 260 00:14:56.639 --> 00:15:00.120 rate real error rate as in all all calls errors, 261 00:15:00.159 --> 00:15:03.320 all errors counted in the data compared to an adjudicated 262 00:15:03.360 --> 00:15:08.480 gold standard, very different than the EDC error rates. 263 00:15:08.519 --> 00:15:10.600 So there's an indication there. 264 00:15:10.639 --> 00:15:14.919 Granted, these are observational data quality studies themselves, but there's 265 00:15:14.919 --> 00:15:18.679 an indication that actually using the data from the EHR 266 00:15:18.879 --> 00:15:22.279 that's collected in the context of a structured protocol can 267 00:15:23.000 --> 00:15:27.399 better the data quality that we get from from today's standards. 268 00:15:29.519 --> 00:15:32.399 So comparing the examples we just did that, I'll skip 269 00:15:32.440 --> 00:15:34.399 that and save more time for questions, but it's there 270 00:15:34.399 --> 00:15:36.279 in the slides if you need it. 271 00:15:38.000 --> 00:15:39.120 Some closing thoughts. 272 00:15:39.320 --> 00:15:43.320 First, the EHR to DC and what we've seen so 273 00:15:43.440 --> 00:15:47.200 far is associated with a better error rate. The program 274 00:15:47.240 --> 00:15:51.120 that we're doing this under is called ASRWD. It's Ancillary 275 00:15:51.200 --> 00:15:54.840 Studies to evaluate real world data quality, and we're working 276 00:15:55.120 --> 00:15:59.399 with sponsors and technology providers who are willing to have their. 277 00:15:59.360 --> 00:16:00.960 Data intopend evaluated. 278 00:16:01.000 --> 00:16:03.679 They send us all the discrepancies, we adjudicate them one 279 00:16:03.720 --> 00:16:06.600 by one, go through them, call a site all sorts 280 00:16:06.639 --> 00:16:08.840 of stuff, have the site look back in the EHR, 281 00:16:10.240 --> 00:16:14.200 so really an independent evaluation, and then we adjudicate a 282 00:16:14.240 --> 00:16:18.039 gold standard based on that full medical record review at 283 00:16:18.080 --> 00:16:19.759 the site by the site study coordinator. 284 00:16:21.360 --> 00:16:23.159 And so in doing. 285 00:16:22.879 --> 00:16:27.039 That, the ACE is just starting to generate this look 286 00:16:27.360 --> 00:16:30.759 at what we can achieve with integration of care and 287 00:16:30.799 --> 00:16:34.399 research at a site where we're reducing what's a huge 288 00:16:34.399 --> 00:16:37.639 burden for a site in terms of data collection. Hopefully 289 00:16:37.720 --> 00:16:41.279 we're reducing the I got stuck twice phenomenon from our 290 00:16:41.320 --> 00:16:45.679 patients in clinical studies. So making sort of the final 291 00:16:45.960 --> 00:16:50.440 point that when all sizes, it's not all equal, right 292 00:16:50.480 --> 00:16:54.600 when you compare use of EHR data using it retrospectively 293 00:16:54.679 --> 00:16:56.720 where it was collected in the course of routine care 294 00:16:56.799 --> 00:17:00.639 outside the context of a clinical studies, completely different from 295 00:17:00.639 --> 00:17:03.519 when the site has the context of that clinical study 296 00:17:03.559 --> 00:17:07.559 protocol and may even have aspects of that protocol implemented 297 00:17:07.559 --> 00:17:10.000 in their EHR. And as we as sites all get 298 00:17:10.000 --> 00:17:13.240 better and more fascial with our EHR systems, that aspect 299 00:17:13.319 --> 00:17:15.720 is going to get even better. So we have ways 300 00:17:15.759 --> 00:17:17.920 to go to catch up with Denise and what she's 301 00:17:18.000 --> 00:17:22.880 implemented at Duke. With respect to that last thing I'll 302 00:17:22.920 --> 00:17:25.559 mention is the ASRWD program is ongoing. 303 00:17:25.599 --> 00:17:26.920 We've got two years left in it. 304 00:17:27.039 --> 00:17:30.799 We're always happy to do these independent assessments or work 305 00:17:30.880 --> 00:17:35.359 with somebody that's got both to do an independent assessment 306 00:17:35.480 --> 00:17:42.319 of the data. We're doing this so that technology providers, sponsors, regulators, 307 00:17:42.359 --> 00:17:46.119 everybody has a shared pool of information that's cross site, 308 00:17:46.200 --> 00:17:50.359 cross therapeutic area cross study, so that they can have 309 00:17:51.160 --> 00:17:54.240 much better information about what the actual quality of the 310 00:17:54.279 --> 00:17:56.960 data is moving forward, and that we can get best 311 00:17:56.960 --> 00:17:59.640 practices out of how to implement this and keep the 312 00:17:59.720 --> 00:18:04.680 error slow and uh, Thursday or I guess it's actually 313 00:18:04.720 --> 00:18:09.039 Wednesday afternoon. Sorry, we'll talk more about how we actually 314 00:18:09.079 --> 00:18:12.000 measured it and measurement methods if folks are interested in 315 00:18:12.079 --> 00:18:16.599 doing that. Did I mention ACE is still recruiting. I'm 316 00:18:16.640 --> 00:18:22.559 just saying I do in the interest of fully disclosing there. 317 00:18:23.400 --> 00:18:26.039 This work is really hard to get funded, right because 318 00:18:26.160 --> 00:18:28.400 very few people care about data quality and at this 319 00:18:28.400 --> 00:18:31.359 little detail level and want to pay extra money to 320 00:18:31.359 --> 00:18:35.000 get the studies done. So huge shout out to Cory 321 00:18:35.200 --> 00:18:38.920 for the funding for the older study, n CI National 322 00:18:38.960 --> 00:18:43.440 Institutes of Health for the cancer study, or our Cancer 323 00:18:43.480 --> 00:18:48.000 Institute at Texas, the Borough's Welcome Fund which which funds 324 00:18:48.000 --> 00:18:50.960 the Coordinating Center for the ASR w D program, UH 325 00:18:51.160 --> 00:18:54.519 in Coop and Karts Platform for e h R to 326 00:18:54.599 --> 00:18:58.160 e d C and Yonahlink for their platform and in 327 00:18:58.279 --> 00:19:02.160 kind contribution uh the data. Huge thank you to these guys. 328 00:19:03.079 --> 00:19:04.680 I'm happy for questions if we have time. 329 00:19:10.200 --> 00:19:11.480 We do have time. 330 00:19:11.559 --> 00:19:13.240 For one or two questions. 331 00:19:13.240 --> 00:19:22.440 Go ahead. I thank you for that wonderful study and presentation. 332 00:19:23.279 --> 00:19:25.799 One thing that I would add, if I might, to 333 00:19:25.920 --> 00:19:30.319 the general topic of HR data for research is the 334 00:19:30.319 --> 00:19:37.480 insecurity of vhrs, the most hacked and breached data sources 335 00:19:37.519 --> 00:19:42.480 in the United States. And I understand that the Office 336 00:19:42.480 --> 00:19:48.160 of National Coordinator is promoting TEFKA, the Trusted Exchange Framework 337 00:19:48.599 --> 00:19:52.839 and Common Agreement Program as a way of making more 338 00:19:52.920 --> 00:19:58.759 EHR data using tokens presumably to link to put together 339 00:19:58.880 --> 00:20:02.200 the pieces of patient data, so you have supposedly single 340 00:20:02.279 --> 00:20:08.279 individual data for research purposes, for pandemic use, for major 341 00:20:08.799 --> 00:20:14.200 healthcare uses. And yet what they're really doing is cascading 342 00:20:14.559 --> 00:20:17.119 the error that can come in, not just the kind 343 00:20:17.160 --> 00:20:19.759 of errors that you were talking about, but from hackle 344 00:20:20.039 --> 00:20:24.079 actual hacking. And I think there's another The White House 345 00:20:24.119 --> 00:20:28.079 Office of National Security does have a major concern about 346 00:20:28.079 --> 00:20:33.240 that as a national security risk, because TEFKA is really 347 00:20:33.279 --> 00:20:39.359 just expanding the contact surface and the opportunity for bad 348 00:20:39.400 --> 00:20:43.279 actors to throw wrong data in. But what could be 349 00:20:43.359 --> 00:20:47.720 happening even now, as I understand it, is that bad 350 00:20:47.759 --> 00:20:52.480 actors are doing precisely that they can get in and 351 00:20:53.240 --> 00:20:56.400 without anyone knowing it, just alter data here and there, 352 00:20:57.839 --> 00:21:01.720 and so researchers have really no sense of what they're what. 353 00:21:01.960 --> 00:21:06.720 It might seem accurate, but anyway, that's just another concern 354 00:21:07.119 --> 00:21:08.160 the whole security question. 355 00:21:08.359 --> 00:21:10.960 So yet, yes and no, I mean definitely there are 356 00:21:11.000 --> 00:21:12.799 those of us who have been hacked who don't even 357 00:21:12.880 --> 00:21:16.640 know it yet, and there are hundreds of thousands of 358 00:21:16.759 --> 00:21:19.680 hacking tries, if the numbers that low across our country 359 00:21:19.680 --> 00:21:22.759 today in EHR systems. I will say though, that the 360 00:21:23.039 --> 00:21:27.160 hacker's primary interest is selling the data or getting getting 361 00:21:27.200 --> 00:21:29.880 funds for the data, and they don't get much for 362 00:21:30.000 --> 00:21:34.200 going in and playfully changing values in our EHRs. It's 363 00:21:34.240 --> 00:21:38.039 when they sell the identifiers on the record. So there's 364 00:21:38.039 --> 00:21:42.680 a lack of a financial motivation so much from. 365 00:21:43.079 --> 00:21:47.519 Concern of state actors deliberately trying to create problems in 366 00:21:47.519 --> 00:21:48.400 the United States. 367 00:21:50.400 --> 00:21:54.119 Yep. Well, there's those of us that haven't been hacked unfortunately, 368 00:21:54.240 --> 00:21:57.000 will uh, we'll deal with it in the future and 369 00:21:57.000 --> 00:21:59.599 and have you know, be hacked at some point most 370 00:21:59.599 --> 00:21:59.920 of us. 371 00:22:02.279 --> 00:22:02.920 So thank you. 372 00:22:03.000 --> 00:22:03.599 That was great. 373 00:22:04.440 --> 00:22:08.200 So my question relates to practice based research networks, and 374 00:22:08.279 --> 00:22:11.960 I just got a big thirty million dollar fund we're 375 00:22:12.000 --> 00:22:12.799 all going after. 376 00:22:14.720 --> 00:22:17.160 My question really. 377 00:22:16.880 --> 00:22:21.559 Relates to the quality of data in clinical practice sites 378 00:22:22.240 --> 00:22:29.079 family medicine, internal medicine, primary care, general pediatrics. What we've 379 00:22:29.160 --> 00:22:33.839 found with our PBRN is that smaller sites that aren't 380 00:22:33.839 --> 00:22:38.440 necessarily part of a larger health system actually need help 381 00:22:39.279 --> 00:22:44.400 understanding how to make data driven decisions. And so if 382 00:22:44.519 --> 00:22:49.440 data is not all that accurate going into their electronic 383 00:22:49.480 --> 00:22:53.640 health record to start with, and we're trying to help 384 00:22:53.680 --> 00:22:58.160 them build capacity for data driven decisions as part of 385 00:22:58.200 --> 00:23:03.000 building research capacity. Do you have recommendations for those of 386 00:23:03.119 --> 00:23:10.519 us who are coordinating administering large networks of primary care 387 00:23:10.559 --> 00:23:16.559 practice to help them develop capacity for data quality. 388 00:23:16.680 --> 00:23:21.599 Yeah, so that is a really big issue the pbr 389 00:23:21.799 --> 00:23:26.039 ns and pbr in like sites that I've worked with. 390 00:23:26.640 --> 00:23:29.000 Instead of saying sure, Meredith will send you that, let 391 00:23:29.079 --> 00:23:31.839 me get you in the queue, they say, Meredith, if 392 00:23:31.880 --> 00:23:34.119 you want that data, come down here yourself and get it. 393 00:23:34.519 --> 00:23:36.519 You're going to have to email it to yourself. Now, 394 00:23:36.559 --> 00:23:39.000 not that we would email it, but you get the picture. 395 00:23:39.519 --> 00:23:41.640 It's more of a self service kind of thing because 396 00:23:41.680 --> 00:23:47.319 they're usually their EHR is hosted and someone else does 397 00:23:47.359 --> 00:23:49.960 all the work on it for them. They wouldn't know 398 00:23:50.000 --> 00:23:52.160 how to go in and extract the data many of 399 00:23:52.160 --> 00:23:54.839 those sites if they had to. The other thing I'll 400 00:23:54.839 --> 00:23:57.599 say is that there's sort of a fundamental principle in 401 00:23:57.799 --> 00:24:01.240