Speaker 1: Up next, we have Ben Galen, portfolio leader from Roche,
Speaker 1: and Ken Getz, MBA, Director and Professor Tuft's Center for
Speaker 1: the Study of Drug Development at Tufts University, and they
Speaker 1: will be presenting the Patient Voice and Protocol Planning and
Speaker 1: Execution for Long Term Business Sustainability.
Speaker 2: Good afternoon, everyone. I'm so sorry the person who asked
Speaker 2: about measuring patient burden isn't here, because that's exactly what
Speaker 2: we're going to talk about now. I have the pleasure
Speaker 2: of co presenting with Ben Galan. We've been collaborating on
Speaker 2: an effort to think through strategic concepts and the application
Speaker 2: of an approach to measuring a patient burden based on
Speaker 2: protocol design characteristics, and then to use that input to
Speaker 2: inform decision making about the protocol itself and ultimately the
Speaker 2: execution of the clinical trial. So I'm going to start
Speaker 2: by providing a little bit of context for you. Many
Speaker 2: of you have seen this data before, but I want
Speaker 2: to sort of set the stage by just showing you
Speaker 2: how complex our protocol designs have actually become and this
Speaker 2: is something we measure routinely. And then I'll move into
Speaker 2: specifically how we are able to systematically introduce and integrate
Speaker 2: patient input based on the characteristics of the protocol itself.
Speaker 2: It's scientific characteristics as well as the executional elements of
Speaker 2: the design. And then I'm going to turn it over
Speaker 2: to Ben, who's going to really talk about how you
Speaker 2: apply this and you determine ways to integrate patient input
Speaker 2: in to really drive sustainable value for the organization overall,
Speaker 2: and how to use those insights to inform cross functional
Speaker 2: decision making. First, the context on the left hand side,
Speaker 2: you see some data that was published a few years ago,
Speaker 2: just to really give you a flavor for some of
Speaker 2: the most common elements in a protocol, scientific and operational
Speaker 2: or executional. And I'm showing you really just two time
Speaker 2: periods here. We've been measuring growth in the prevalence of
Speaker 2: these variables for quite a long time. If you look
Speaker 2: at the total number of endpoints, for example, this is
Speaker 2: on average, looking across multiple therapeutic areas, we've seen nearly
Speaker 2: a doubling. Eligibility criteria has remained relatively flat, but that's
Speaker 2: in part a function of the way we count inclusion
Speaker 2: and exclusion criteria. Some companies group multiple criteria and counted
Speaker 2: as a single as a single inclusion criterion. For example,
Speaker 2: total procedures to support those endpoints has grown dramatically, And
Speaker 2: you can look across any element of the design, the
Speaker 2: number of sites, number of countries, virtually, they have all
Speaker 2: grown dramatically. Look at that bottom data point. Total data
Speaker 2: collected by the protocol has risen far more dramatically than
Speaker 2: any other area, nearly four times the level we observed
Speaker 2: in twenty ten on average. On the right hand side,
Speaker 2: I'm showing you just another measure of customization and complexity,
Speaker 2: and that is the number of intermediaries involved in supporting
Speaker 2: any clinical trial, both the design, but more so the
Speaker 2: execution of the study and the data collection. About four
Speaker 2: or five years ago, total spending by pharma on contract
Speaker 2: service providers technology services, study conduct services, cro services surpassed
Speaker 2: the total amount that is spent each year just supporting
Speaker 2: internal infrastructure. And the key takeaway with all of this
Speaker 2: is complexity is associated with poor with poorer performance. It's
Speaker 2: correlated inversely with poor performance. So the higher complexity rises,
Speaker 2: the more performance suffers. Our recruitment rates are lower, our
Speaker 2: retention rates are worse, our timelines are longer. The number
Speaker 2: of protocol amendments that we observe actually increases the more
Speaker 2: complex our designs. On the right hand side, I'm showing
Speaker 2: you just one measure of site burden to administer our protocols,
Speaker 2: and we have countless measures that really show how difficult
Speaker 2: it is for sites to manage and administer our protocols today.
Speaker 2: Here's just one measure showing you site enrollment achievement, the
Speaker 2: percentage of sites that were activated that ultimately achieve target enrollment,
Speaker 2: and that number has been declining steadily every year for
Speaker 2: more than a decade. Here are just two other men
Speaker 2: measures that now bring it down to the patient burden level,
Speaker 2: and we have many others that I could share with you.
Speaker 2: The percentage of patients who drop out prematurely due to
Speaker 2: their own choice, not the choice of the site personnel,
Speaker 2: or due to a response to the study drug has skyrocketed.
Speaker 2: Nearly two thirds of patients in recent studies are dropping
Speaker 2: out out of their own choice first, so they're pre
Speaker 2: empting the site or they're choosing to drop out because
Speaker 2: of the burden of participation. On the right hand side,
Speaker 2: these are now exit interviews that were conducted with patients
Speaker 2: who completed their participation and we asked what did you
Speaker 2: like least about being in the study, and four of
Speaker 2: the five areas all relate to the burden itself, the
Speaker 2: location of the research center. The study visits were so
Speaker 2: time consuming, there were too many procedures, they were cumbersome,
Speaker 2: and I love that one compensation was not enough. You
Speaker 2: could not pay me enough for the kind of requirements
Speaker 2: that this protocol post. We have been looking to simplify
Speaker 2: protocol design for a long time. You know, I've been
Speaker 2: measuring as an outsider looking in as an academic for
Speaker 2: nearly thirty years. We've been looking at protocol design behaviors.
Speaker 2: Early on, one of the most common approaches was to
Speaker 2: use economic input into protocol design. The idea that if
Speaker 2: you cut out a procedure, or you cut out a
Speaker 2: certain number of times that the procedure is performed, would
Speaker 2: save the study budget some expense. That was a really
Speaker 2: highly prevalent approach in the early nineties, a period where
Speaker 2: we started to see a lot of companies look for
Speaker 2: tools that would help them manage the economics and the
Speaker 2: financial requirements of a lot of their activity. You might remember,
Speaker 2: for example, the PIKS database, which was a tool that
Speaker 2: gave pharma companies the ability to understand the typical pricing
Speaker 2: for an investigative site to perform a particular protocol. When
Speaker 2: we shifted in the two thousand twenty ten period, this
Speaker 2: was one of the early phases of sponsors looking to
Speaker 2: establish a more meaningful and more effective relationship with their sites,
Speaker 2: and so in that twenty twenty ten period we saw
Speaker 2: height and focus on executional feasibility, and you may recall
Speaker 2: that a lot of the feasibility assessments that were introduced
Speaker 2: started to get larger and larger and larger as more
Speaker 2: data was collected by the site. We then shifted our
Speaker 2: focus and we're still in this environment where we're looking
Speaker 2: to reduce the number of endpoints. We're looking to prioritize
Speaker 2: our designs so that we can minimize avoidable amendments, one
Speaker 2: of the most disruptive experiences that we have in any
Speaker 2: clinical trial, and that's still a very very important way
Speaker 2: that we look to simplify protocol design. And the patient
Speaker 2: engagement movement has introduced and elevated our sensitivity, our awareness,
Speaker 2: and our interest in integrating patient input into design and
Speaker 2: using the patient's own perceived a sense of burden, the
Speaker 2: relevance of the trial their own concerns about the convenience
Speaker 2: of participation to help identify areas that we can modify.
Speaker 2: So these are essentially the four optimization areas. We know
Speaker 2: if many organizations that have integrated these approaches. Patient engagement
Speaker 2: and the measurement of patient burden is perhaps the newest area,
Speaker 2: and we mostly see companies using advisory boards to gather
Speaker 2: that input. These are like focus groups, and I know
Speaker 2: many organizations are doing them, but they typically appear when
Speaker 2: the protocol is nearly finalized, so they're actually coming into
Speaker 2: the planning and the design phase slightly late to the game,
Speaker 2: after many internal champions and thought leaders and others have
Speaker 2: provided input into the design of the protocol. The idea
Speaker 2: for the burden assessment is to move patient input into
Speaker 2: the earliest stages so that before we move to solicit
Speaker 2: input from thought leaders or at the same time that
Speaker 2: we're soliciting input from multiple parties or already incorporating patient input.
Speaker 2: So there are a number of approaches a number of
Speaker 2: organizations that are offering methodologies. What the Tough Center has
Speaker 2: done is created essentially an approach that organizations can ultimately
Speaker 2: build in house and a take in house so that
Speaker 2: the can do this internally. We are not specifically designing
Speaker 2: a product that can be licensed. We're developing essentially an
Speaker 2: approach that any organization can adopt and use. We've published
Speaker 2: extensively and I'm listing three of the publications here. What
Speaker 2: we did is we based on our work in protocol
Speaker 2: design and the benchmark activity that is now coming up
Speaker 2: on thirty years of experience, we were able to narrow
Speaker 2: down the sixty most common procedures that we see in
Speaker 2: every protocol, and we presented those procedures and other elements
Speaker 2: of the participation experience, distance to travel, whether a procedure
Speaker 2: might be performed on site or at a different location,
Speaker 2: the length of the visit, for example. All of that
Speaker 2: was presented to patients globally and they would rate the
Speaker 2: relative perceived bird of every one of those procedures or
Speaker 2: of those experiences related to essentially a reference point, and
Speaker 2: that reference was a routine physical exam, something that most
Speaker 2: patients are familiar with, even just by interacting with their
Speaker 2: own primary care physician practice. And we gathered ultimately responses
Speaker 2: from thirty six hundred global patients, so we have some
Speaker 2: granularity by disease conditioned by demographic characteristics, by socioeconomic status,
Speaker 2: and many others, and we're now able to use that
Speaker 2: input and its variation by all of these different subgroups
Speaker 2: to map to every protocol design. So we're looking at
Speaker 2: a number of different areas. We're looking at the procedures
Speaker 2: that are performed, we're looking at burden to adherents, we're
Speaker 2: looking at lifestyle restrictions, we're looking at other convenience factors,
Speaker 2: and all of that can be mapped based on the
Speaker 2: input that we've received from patients. To date, fourteen companies
Speaker 2: have been applying this approach and bringing it in house.
Speaker 2: And I'll just share a little bit of the high
Speaker 2: level aggregate data with you. The chart on the left
Speaker 2: here is not intuitive, and it's because we've created a
Speaker 2: burden's score, which does not have meaning in and of itself.
Speaker 2: You have to sort of compare that to benchmark data.
Speaker 2: For companies, they can compare it to their own internal
Speaker 2: burden score to see how it's changing over time. I'm
Speaker 2: showing you aggregate data here, and what's most important is
Speaker 2: it tracts so well to changes that we've seen in
Speaker 2: protocol design over time anyway, So it's sort of a
Speaker 2: way that we have helped to validate a lot of
Speaker 2: our measures. They map very well to change in design decisions.
Speaker 2: On the right hand side, I'm showing you another measure
Speaker 2: of burden. This is now the distribution of visits by
Speaker 2: their average duration. Where in twenty eleven to twenty fourteen,
Speaker 2: nearly sixty percent of all patient visits averaged about an
Speaker 2: hour or less. That has dropped down now to below half.
Speaker 2: And you can see the proportion that has grown the
Speaker 2: most are now those visits that are lasting more than
Speaker 2: two hours. This may include travel, right, we've sort of
Speaker 2: consolidated that to summarize the statistic here, But that's just
Speaker 2: another measure of burden that we can assess. And again
Speaker 2: the punchline and all this is you can start to
Speaker 2: relate burden to performance outcomes and that helps you isolate
Speaker 2: those areas of that may be most contributing to burden
Speaker 2: that might be predictive of different outcomes in the study. Here,
Speaker 2: I'm pulling together data from all of the fourteen companies
Speaker 2: that have participated so far, and what I wanted you
Speaker 2: to see is just a pattern that is emerging. Essentially,
Speaker 2: when burden is low, we typically see faster relative cycle times,
Speaker 2: fewer protocol amendments, fewer protocol deviations, and even lower dropout
Speaker 2: rates and drop out for many of you, you'll know,
Speaker 2: it's a very complicated measure because there are so many
Speaker 2: factors that play a part, so we were surprised to
Speaker 2: see a slight decline in the overall rate here in
Speaker 2: this assessment. With that, I'm going to turn it over
Speaker 2: to Ben, who's going to really show how his organization
Speaker 2: has approached trying to integrate and leverage this insight into
Speaker 2: their own protocol design activity. So Ben, I'm turning it
Speaker 2: over to you. Thanks so much, and.
Speaker 3: Thanks Ken for the introduction. Yeah, so I'm here representing
Speaker 3: ROSH and really talking about the collaboration that we've had
Speaker 3: with Ken and Tufts and how this has helped us
Speaker 3: to actually integrate a lot of the different insights across
Speaker 3: the ecosystem of our clinical trial design. So I want
Speaker 3: to start talking first about how this came to be.
Speaker 3: We have in my company, like the theme you see
Speaker 3: across many of the sponsored companies right now, what we
Speaker 3: consider to be a productivity issue in our clinical trials,
Speaker 3: and that is something that has some very serious legs
Speaker 3: behind it to try and actually uplift that. But also
Speaker 3: we have a growing need and understanding that we need
Speaker 3: to make sure that the trial designs themselves are much
Speaker 3: more patient centric, and we have the opportunity therefore to
Speaker 3: essentially bring together a mutually beneficial relationship where we can
Speaker 3: make the trials much more patient inclusive, much better for patients,
Speaker 3: but also actually for us cheaper to run, faster to run,
Speaker 3: and get to market more quickly. When we started down
Speaker 3: this path several years ago, we decided that we didn't
Speaker 3: want to reinvent the wheel for obvious reasons, so we
Speaker 3: reached out to Ken in the Tought organization to try
Speaker 3: and actually start leveraging the standard framework and methodology that
Speaker 3: they can and tough to actually brought. And what that
Speaker 3: enabled us to do very quickly was to actually have
Speaker 3: a point of comparison of where do our trials actually
Speaker 3: land against other large industry bodies, because we wanted a
Speaker 3: point of comparison that's actually sort of we run extremely
Speaker 3: complex global trials. We have a reputation for being amongst
Speaker 3: i think it's fair to say, the most complex and
Speaker 3: the most expensive and also the slowest trials amongst most
Speaker 3: of the large organizations which is not something we're terribly
Speaker 3: proud of, so we needed a framework that would actually
Speaker 3: help us actually bring this together and to directly address
Speaker 3: R and D productivity. What we identified quite quickly through
Speaker 3: this work was not just the KPI is underneath this
Speaker 3: in terms of how do we actually measure the burden
Speaker 3: of our designs and linking that to for example, the
Speaker 3: investigator burdens and the site level burdens and different feedback
Speaker 3: from patient bodies and collecting evidence around for example, the
Speaker 3: value of patient engagement for our patient engagement organizations, but
Speaker 3: actually to contextualize those scores against key drivers and determinants
Speaker 3: of burden, which allows us to then actually influence decision
Speaker 3: making and designs within our very large and complex organization
Speaker 3: across different functions that we have there. Of course, to
Speaker 3: do this you can see through the different talks earlier today,
Speaker 3: there are a large variety of different complexities involved across
Speaker 3: For example, the people that actually develop the protocols and
Speaker 3: study designs need to essentially make sure that the protocol
Speaker 3: is for purpose and actually determining the key end points
Speaker 3: of evidence it needs to actually work for the science,
Speaker 3: but then actually in operationalizing design and running the studies. Effectively,
Speaker 3: we also have a second series of stakeholders internally, and
Speaker 3: they have very different priorities and incentive structures. And then,
Speaker 3: of course we have the people that actually are participating
Speaker 3: in our trials who have a whole different range of
Speaker 3: issues and complications that we need to make sure that
Speaker 3: we're actually taking sufficiently into account. When we started presenting
Speaker 3: the results of these initial analyses and insights, we very
Speaker 3: quickly gained across pharma reaction by a cross farmer, I
Speaker 3: mean internally across our early research organizations and late research
Speaker 3: organizations and a medical affair organizations. Suddenly we started gaining
Speaker 3: this critical mass of different people across the different functions
Speaker 3: who were very motivated to actually participate in this effort
Speaker 3: and to bring together those insights. So I want to obviously,
Speaker 3: I can't talk too many specifics around the data that
Speaker 3: we're actually seeing. A lot of this is in flight.
Speaker 3: I'm going to talk about it sort of in an
Speaker 3: aggregate way, and I'm happy to answer questions offline and
Speaker 3: off recording at a later date if you would like
Speaker 3: and be interested in sort of how we're approaching this
Speaker 3: and some of the value we're seeing. But I think
Speaker 3: the suffice to say, in the initial efforts that are
Speaker 3: actually coming through with study designs that are actually changing
Speaker 3: and are in flight with patients right now, we are
Speaker 3: seeing substantial improvements in different productivity scores that can just
Speaker 3: displad improvements in our cycle times in terms of patient
Speaker 3: recruitment activities, reductions and patients dropouts and reduction reductions in
Speaker 3: protocol avoidable protocol amendments and different drivers there a value
Speaker 3: and the feedback from the patient organizations has also been positive.
Speaker 3: So we are starting to see the shift that we
Speaker 3: are looking for through this effort. And say, this effort,
Speaker 3: but it's simple an amalgamation of many, many different efforts
Speaker 3: coming together with a single lens. I'm talking a little
Speaker 3: bit generic here. We don't actually call this the triad
Speaker 3: in company, but it's more sort of how do we
Speaker 3: think about this and how are we framing it as
Speaker 3: a construct, Because what we need to do is actually
Speaker 3: make the data that Ken was talking about actionable across
Speaker 3: different functions that have competing priorities. And not only do
Speaker 3: we need to understand what performance is and what's driving it,
Speaker 3: we need to actually be able to get to the
Speaker 3: root causes of that, what specifically is driving the increase
Speaker 3: in burden and protocol complexity and how do we nip
Speaker 3: that in the bud where we can and how do
Speaker 3: we recognize what is necessary burden and what is necessary
Speaker 3: complexity that actually achieves a beneficial outcome for patients, and
Speaker 3: where can we actually start cutting back to try and
Speaker 3: make sure that we are optimizing and minimalizing all of
Speaker 3: those efforts. So I'm going to talk through, in fairly
Speaker 3: vague detail, a little bit of what we mean in
Speaker 3: each one of these different elements of the triad. In
Speaker 3: the performance sector, which is largely my part of the
Speaker 3: organization portfolio strategy, we want to understand the mutual benefit
Speaker 3: through simulations and measurements of what do we actually expect
Speaker 3: the trial to perform across key milestones and how is
Speaker 3: this affecting the budget. So essentially, how do we do
Speaker 3: the trade offs in the trial performance factors that we
Speaker 3: are looking for and how does that influence financial decision
Speaker 3: making investment decisions across our portfolio. On the other hand,
Speaker 3: we want to make sure that we are actively reducing
Speaker 3: patient burden, reducing, minimizing and managing our site patient burdened
Speaker 3: our complexity, saying before that we know that some of
Speaker 3: this is necessary will always be there. There will never
Speaker 3: be a trial that doesn't have any burden. It would
Speaker 3: be nice if we could get there, but I don't
Speaker 3: think it's going to happen. And then we want to
Speaker 3: make sure, on the other hand, that we are actually
Speaker 3: using directly patient feedback from the get go, not simply
Speaker 3: as Ken said, like after the protocol is largely designed
Speaker 3: and then we tweak it if we can, or too often.
Speaker 3: I think we know that the trial design is often
Speaker 3: sort of said and doesn't change terribly much after we
Speaker 3: get that engagement. How do we make sure that it's
Speaker 3: there from the start and actually that we have an
Speaker 3: active learning loop so that the next trial design actively
Speaker 3: incorporates what we've already understood from different patient communities, rather
Speaker 3: than sort of going back and starting a new advisory
Speaker 3: boarder or asking the same questions again. So again, that's
Speaker 3: both easier and cheaper for us to do, and it's
Speaker 3: better for the patient community because we have the feedback
Speaker 3: that they get a little bit sick of us asking
Speaker 3: the same questions every single time. It doesn't have a
Speaker 3: terribly professional appeal to it. Some of the different measures
Speaker 3: here and this is not comprehensive, and I'm mixing across
Speaker 3: some key performance indicator type data points and some drivers
Speaker 3: of what is actually what is underneath that. But here
Speaker 3: are some of the things that are actively looking at.
Speaker 3: Many of them are measurable through the Tuft's work that
Speaker 3: Ken is presented. So we're in terms of the trial
Speaker 3: design measures, the additive patient and site burden scores. What
Speaker 3: I mean by additive is contextualized, it's against standard of care.
Speaker 3: So when you looked at what Ken was presenting an
Speaker 3: aggregate in terms of the total burden score, what we've
Speaker 3: done to try and actually make it more actionable by
Speaker 3: our clinical study teams is actually put against the standard
Speaker 3: of care. So what is the additional burden above that
Speaker 3: standard of care? And then against our folio? How does
Speaker 3: that compare to other studies within the same disease area
Speaker 3: or the same therapeutic area within our portfolio, and then
Speaker 3: how does that compare against other similar studies from other
Speaker 3: sponsors within that externally, so that we can really understand
Speaker 3: is this trial adding more burden than it's worth? Do
Speaker 3: we have a sufficient understanding of the disease and the
Speaker 3: potential benefit of this that it's actually worth the additive benefit.
Speaker 3: So the additive burden, on the other hand, we have,
Speaker 3: for example, we know that we need to collect the
Speaker 3: actual patient insights around their support needs, the additive burden
Speaker 3: from their perspective, not what we have measured it as
Speaker 3: using our internal expertise and our scientific russianale, but actually
Speaker 3: what do they think it is, because that's quite different
Speaker 3: and we need to make sure that that's taken sufficiently
Speaker 3: into account. So I'm happy to answer questions around how
Speaker 3: we are using these this. There's obviously a lot of
Speaker 3: different calculations in the background about how these relate to
Speaker 3: each so for us to say that it is relatively complex,
Speaker 3: but we have started to make this work for us
Speaker 3: in terms of influencing decisions across these different data points
Speaker 3: and this informs our learning loop. Now this is in
Speaker 3: company fairly early maturity. It's not fully embedded yet, it's
Speaker 3: in process, and I can't talk too many specifics, but essentially,
Speaker 3: as I said, we are making sure that the key
Speaker 3: insights from patients and healthcare providers and sites is actually
Speaker 3: informed from the start before you start designing the protocol,
Speaker 3: and that sort of goes into a study design tool
Speaker 3: technology against our portfolio and pharma strategy to actually allow
Speaker 3: us to simulate the likely performance of this trial design
Speaker 3: as we tweak it, so you can then live make
Speaker 3: changes and see the likely results of how does that
Speaker 3: perform or how would that likely perform? And we can
Speaker 3: adjust that for value, we can adjust that for speed,
Speaker 3: and we can make different trade off decisions if we
Speaker 3: need to make this happen faster, what would it cost
Speaker 3: in addition or what would the patient trade off be?
Speaker 3: Are we satisfied with that? If we take that back
Speaker 3: to patients, how would that actually look? How would they
Speaker 3: react if we talk about regulator interactions amongst all of
Speaker 3: this as well. So ipsyn colleagues talked about how does
Speaker 3: that all fit in and how do we make the
Speaker 3: right level of informed decisions? That is actually the third
Speaker 3: point as well, so I've to covered both. Then we
Speaker 3: follow the loop, so any data point that actually goes
Speaker 3: into actual we feedback into the system so we can
Speaker 3: understand did the scenario play out like we expected? Why?
Speaker 3: Why not? And then we make sure that this is
Speaker 3: all sort of self reinforcing across the text, and we
Speaker 3: add that to the data set to make this actually work,
Speaker 3: and you can appreciate maybe the complexity of this Organizationally,
Speaker 3: we have an elaborate partnership across the organization of people
Speaker 3: that are actively involved in getting this across. I've highlighted
Speaker 3: our patient partnerships function and a strategic insights function, but
Speaker 3: it's far more comprehensive than that. It's across our patient
Speaker 3: centered data groups. It's across our clinical operations groups, because
Speaker 3: our clinical science groups in early and late commercial everything
Speaker 3: like that. The most important insights for actually getting this
Speaker 3: off the ground that we've noted that I wanted to
Speaker 3: highlight first of all, is transparency. So this doesn't really
Speaker 3: work if it's only a senior leadership view or if
Speaker 3: it's only a clinical science view. The way that we
Speaker 3: have actually really gained ground here is making sure that
Speaker 3: everyone can understand the implications of what choices they are making.
Speaker 3: If you change the study design in these ways and
Speaker 3: it compromises something else, if you're making a trade off decision,
Speaker 3: that is visible across functions, and we found, we have
Speaker 3: found so far that this is instrumental and actually driving
Speaker 3: sustainable change because now everyone is much more accountable to
Speaker 3: actually making this work and happen, and the other one
Speaker 3: is incentives to make sure that, for example, we can
Speaker 3: embed this in corporate goals and our sort of bonus
Speaker 3: multipliers for every employee. This is starting to happen, and
Speaker 3: that means that it's public for everyone in the company
Speaker 3: and everyone is actively incentivized to work towards this and
Speaker 3: make sure that we are acting as an organization in
Speaker 3: this route. And this is I think the most important
Speaker 3: part of actually making this sustainable is it's a lot
Speaker 3: of effort to actually help make sure that we can
Speaker 3: do all of this. There's obviously a lot of extra
Speaker 3: data collection that we need to do internally in data processing.
Speaker 3: The technology investment is not insubstantial, and then we have
Speaker 3: to make sure that we can validate all of this,
Speaker 3: that we can actually make sure that this is driving
Speaker 3: the right behaviors, that it's driving the right performance that
Speaker 3: we actually want to see, that it's not compromising anything
Speaker 3: in terms of quality for example. So the only way
Speaker 3: to then actually pay this out is to tie it
Speaker 3: back then to these different areas and make sure that
Speaker 3: we are understanding the value against any one of those
Speaker 3: factors in the triad. So far that seems to be
Speaker 3: going in the right direction for us. We are seeing
Speaker 3: sort of a trial design costs coming down, a performance
Speaker 3: going up against that effort. However, we still have a
Speaker 3: long worad ahead of us to actually making this fully
Speaker 3: embedded across the organization and fully maturing this, and yeah,
Speaker 3: I hope to actually see that through with Ken in
Speaker 3: the near future. Near future meaning the next year or so. Okay,
Speaker 3: so it didn't even get dinged, I think. So we
Speaker 3: have thirty seconds per tect