Speaker 1: We are going to now move on to our next
Speaker 1: session for accelerating patient enrollment by deploying a product model approach,
Speaker 1: and I'd like to welcome our two speakers to the stage.
Speaker 2: Hello.
Speaker 3: I am Tara Schultz. I am an Associate director in
Speaker 3: Drug Development IT at Bristonal Meyer Squib. I am the
Speaker 3: IT product manager for patient Recruitment him.
Speaker 4: My name is Chen Deep. I'm the product designer at
Speaker 4: Bristo my squip.
Speaker 1: Thank you.
Speaker 3: So there's been a lot of talk at this conference
Speaker 3: and in the industry about AI. This talk will not
Speaker 3: be about AI, will be other things in it. We're
Speaker 3: going to discuss how we implemented a product model and
Speaker 3: product thinking which helps us with a build a strong
Speaker 3: foundation so that you can take advantage of all the
Speaker 3: technology has to offer, including AI.
Speaker 4: Right. And it's also not about diversity, but it's about
Speaker 4: talking to your patients about problems that you're going to
Speaker 4: solve for them.
Speaker 3: So we're going to talk about our new website, which
Speaker 3: is BMS Clinicaltrials dot com. It is a global website.
Speaker 4: Yep, so we have a global reach. We had available
Speaker 4: globally in multiple languages, attracting a significant number of visitors
Speaker 4: comprehensive search in terms of accessibility for our patients, caregivers
Speaker 4: and healthcare professionals for BMS active recruiting trials. On our website,
Speaker 4: patients can pre screen and for any recruiting trial and
Speaker 4: then they can self refer themselves for the trial, find
Speaker 4: out the sites, and register themselves for the trial for
Speaker 4: those sites.
Speaker 3: So our current our prior website which is BMS studyconnect
Speaker 3: dot com, we got a lot of feedback. We talked
Speaker 3: to a lot of people. There was a poor website
Speaker 3: user experience. Our patients gave us feedback that they were
Speaker 3: having difficulty doing the primary things that they wanted to
Speaker 3: do on the site with inefficient SEM and virtually non
Speaker 3: existent SEO. So our search engine marketing efforts were not
Speaker 3: as effective as they should have been. We were getting
Speaker 3: a low referral rate and no one was seeing us
Speaker 3: on the seer piece. Patients were struggling to find the
Speaker 3: trials and find what they needed.
Speaker 4: And.
Speaker 3: We also didn't really appear on any search results, so
Speaker 3: we couldn't leverage what we were doing right.
Speaker 4: So to overcome these obstacles, these problems, we adopted a
Speaker 4: product thinking approach dive into what that means product thinking
Speaker 4: combines product model and design thinking principles, and what it
Speaker 4: allowed us to do is to really focus on user
Speaker 4: center design while making iterative improvements. We started with defining
Speaker 4: our product vision, what problem our product is going to solve?
Speaker 4: Is it going to solve and how is it different
Speaker 4: from other products that are existing in the market. So
Speaker 4: we looked at other competitive websites and saw what value
Speaker 4: they were offering to the patients, what kind of problem
Speaker 4: they were solving, And we also did our own user research,
Speaker 4: so we reached out to patients. We interviewed the patients
Speaker 4: and understood and mapped their journey throughout the trial and
Speaker 4: different steps about from diagnosis to finding about trials, searching
Speaker 4: for trials, and then referring themselves for the trials. At
Speaker 4: a very early stage, we showed them the new website markups,
Speaker 4: compared them with the older version, and try to find
Speaker 4: out what really works for the patient because we wanted
Speaker 4: to listen to them, We wanted to understand their pain
Speaker 4: points and we wanted to see how we can solve
Speaker 4: their problems by listening to them. And we spent most
Speaker 4: of the time defining the problem and then itating prototyping
Speaker 4: building it with our business stakeholders and with of course
Speaker 4: our patients, testing and validating getting the feedback on the
Speaker 4: improved designs. So on the right hand you see typical
Speaker 4: design thinking steps and on the left you see the
Speaker 4: product model approach. How it really couples to call it
Speaker 4: a product thinking approach. And this is an example of
Speaker 4: the patient journey that we mapped out from diagnosis to
Speaker 4: finding about trials, lanning on the website, checking their eligibility
Speaker 4: of pre screening, and then selecting a site and registering
Speaker 4: them for the trial. And for each step along the way,
Speaker 4: we identified their actions, activities, who they were talking to,
Speaker 4: what kind of interactions they were having, what might be
Speaker 4: influencing their decisions, their sentiments, thoughts and challenges. Now this
Speaker 4: is the simplified version of the patient journey. We, of course,
Speaker 4: you did it very exhaustively with our patients, and this
Speaker 4: was fundamental for our transition from BMS studyc connect dot
Speaker 4: com to BMS clinical trial.
Speaker 3: So how did what did we do differently this time
Speaker 3: that we hadn't done before. So the first thing we
Speaker 3: did was we focused on MVP. That's our minimally viable product,
Speaker 3: that is the least amount of functionality that you need
Speaker 3: that will have the greatest impact. So in order to
Speaker 3: get to the MVP, you have to ruthlessly prioritize. That
Speaker 3: means saying no to most just about everything. You really
Speaker 3: got an arrow it down. We said no a lot.
Speaker 3: We deprioritized a lot, but it did help us with
Speaker 3: the first goal, which is focusing, and everything that we
Speaker 3: did had to correlate to an actual business, a measurable
Speaker 3: business outcome. If we couldn't do it, it wasn't going
Speaker 3: to get prioritized in our release.
Speaker 4: Right. And the outcomes were impressive. We achieved three times
Speaker 4: faster delivery and significantly reduced time to market. We achieved
Speaker 4: seven times more refles than we had compared to last year.
Speaker 4: Our solution was cost effective as compared to the vendor
Speaker 4: solution at that time it was proposed, so we saved
Speaker 4: ninety percent or within that budget, and then also conducted
Speaker 4: user research within the same budget, so we were able
Speaker 4: to define value and focused on key metrics rather than
Speaker 4: website metrics. We were focusing more on patient metrics, what
Speaker 4: how many referrals we were getting. We were showcasing active
Speaker 4: recruiting trials on the website. We reduced the number of
Speaker 4: pages external or exit links, and that really significantly helped
Speaker 4: us reduce the bounds rate and increased engagement and search
Speaker 4: engine rankings, which in a way means that patients could
Speaker 4: find us higher on the search results when they would
Speaker 4: Google for a clinical trial, for example Looper's Clinical Trials.
Speaker 4: At the same time, this also made our campaigns more effective,
Speaker 4: so more ROI on each taller spent and we were
Speaker 4: continuously improving delivering iteratively within two week sprint so business
Speaker 4: could see value more quickly, so business was happy as well.
Speaker 4: And all of this didn't come without any challenges. Our
Speaker 4: approval process was the most difficult, as you would know
Speaker 4: working in FHARMA. Navigating approvals from business stakeholders to legal
Speaker 4: to IRB stakeholders and then aligning them with our product
Speaker 4: roadmap was the most challenging task. And then maintaining redirects,
Speaker 4: ensuring seamless flow of analytics for the campaigns that were
Speaker 4: already running and making sure when people are searching for
Speaker 4: those campaigns they were hitting the right URLs. When we
Speaker 4: switched to the new domain and those analytics or metrix
Speaker 4: also flow to the dashboard for our business users and
Speaker 4: of course, resource allocations is a lot of work was
Speaker 4: UI development and development on aem Adobe Experience Manager on
Speaker 4: which our website is built, so that required a small
Speaker 4: investment as well. And now Tara will sum up what
Speaker 4: it meant actually for our different stakeholders.
Speaker 3: So we had three groups of stakeholders. We have our operations, counterparts,
Speaker 3: our patients, and obviously we're in it, so we're stakeholder
Speaker 3: as well. So for the business we're we did accelerate
Speaker 3: our delivery significantly prior to the year's past. Uh, it's much.
Speaker 3: We've provided a very cost efficient solution and we increase
Speaker 3: the visibility on our search engine result pages. Now for
Speaker 3: our patients, you know, which is what this is really
Speaker 3: all about. We gave them a much much easier use
Speaker 3: to use interface, so the user experience was enhanced. We've
Speaker 3: received seven times the amount of self referrals this year
Speaker 3: versus last year, and the trend is accelerating. So this
Speaker 3: shows us that people are really coming to our site
Speaker 3: to find a trial that they can qualify and enroll in.
Speaker 3: And we had to reduce bounce rate. A bounce rate
Speaker 3: is how long someone will spend on your website, so
Speaker 3: the lower it is the better. It means that people
Speaker 3: are actually coming to your site, want to be on
Speaker 3: your site, and are engaging with your site. And from
Speaker 3: the IT perspective, we came up with a templated design,
Speaker 3: so it makes it much easier to implement, so we
Speaker 3: can keep that delivery pace of every two weeks and
Speaker 3: making sure that we're giving the business what they need
Speaker 3: when they need it. The reduction and the amount of pages,
Speaker 3: as well as the size of our site map makes
Speaker 3: it much easier to maintain. There's a lot of focus
Speaker 3: right now on IT maintenance costs, so any solution that
Speaker 3: we put in, we really had to focus on how
Speaker 3: much the maintenance was going to be around that product.
Speaker 3: So a few things that we did so we did
Speaker 3: strategic collaboration, so we worked with the business and the
Speaker 3: operations team and they give us the strategy. They left
Speaker 3: it to it to do the technical or the tactical
Speaker 3: implementation with our technical tools using the patient insights that
Speaker 3: we get aathered, you can really accelerate your product goal
Speaker 3: if you have clear high level goals. If your goal
Speaker 3: is to increase patient recruitment, that's the goal. It's one goal,
Speaker 3: it's not seven goals. It makes it much easier keeping
Speaker 3: our scope tight. This is always difficult because everybody likes everything,
Speaker 3: but we weren't able to do that. We dropped a
Speaker 3: lot of functionalities and in each priority, in each product cycle,
Speaker 3: we prioritize what could be dropped rather than added. So
Speaker 3: you really need to look at your backlog and see
Speaker 3: what it is that is really going to move the
Speaker 3: needle for your patients and your organization. And since we
Speaker 3: had so many stakeholders, we had a prioritization tech list
Speaker 3: so we could balance the goals of the different groups
Speaker 3: and assess the impact accordingly, because it doesn't always align correctly.
Speaker 3: So in this instance we have a few backlog items.
Speaker 3: We give it a number from one to five for
Speaker 3: business value, same scale for patient impact. Effort is the inverse,
Speaker 3: so we want it to be a low effort item,
Speaker 3: urgency and then the maintenance burden. It gives us a
Speaker 3: total score and that helps us prioritize our items so
Speaker 3: we can deliver more effective solution quicker.
Speaker 4: Awesome. Now it's time to showcase our product, though the
Speaker 4: links might not work, so we have put together screenshots
Speaker 4: at the last moment, so it is I see after all,
Speaker 4: Come on, yeah, I know technical issue, Bree, could you
Speaker 4: show the screenshot for the older version of the website.
Speaker 4: What I really wanted to do here is to show
Speaker 4: you the old version and how it looks like. This
Speaker 4: is our older website, and since our website is also available,
Speaker 4: we get in the transition transitioning phase, so it's available
Speaker 4: for some other countries as well. This is the Canadian version.
Speaker 4: If you see the hero image, it doesn't really connect
Speaker 4: with all the diverse patient profiles that we would like
Speaker 4: to capture, and of course it doesn't focus on the
Speaker 4: patient much. And then there's no menu on this website
Speaker 4: on the older version, so that's also not an SEOS practice.
Speaker 4: You have to have a menu so that when the
Speaker 4: search engine bought really crowls your website, it finds out
Speaker 4: the most important pages and gives the best experience to
Speaker 4: the user who are searching and finding that relevant search
Speaker 4: in Google. We can't, of course scroll it down, but
Speaker 4: this website had a long scroll and then also a
Speaker 4: lot of fasted real estate and a lot of exit
Speaker 4: links right from the homepage, which we rectified in the
Speaker 4: new version. So can we go to the new version now?
Speaker 4: So this is how the new version looks like it's modern,
Speaker 4: but it's also very minimal and it focuses on the
Speaker 4: patient more. You cannot see the carousel of pictures that
Speaker 4: we have implemented, so we have put together screenshots capture
Speaker 4: all the diverse patient profiles. These are not all the
Speaker 4: diverse patient profile spot although that we could get approved
Speaker 4: in the minimum time frame that we could. It does
Speaker 4: put our patient and focus so they are able to connect.
Speaker 4: And of course the name of the website, it's no
Speaker 4: longer called BMS Study Connect because we realized Study Connect
Speaker 4: could not connect with our patients, so BMS Clinical Trials
Speaker 4: was a more obvious choice. And please go ahead check
Speaker 4: it out on your mobile desktops and let us know
Speaker 4: your feedback. Thank you so much for your time. If
Speaker 4: there any question answered, we will be happy to take it.
Speaker 4: Don't hesitate, this is not a plant question.
Speaker 2: Full disclosure. I talked to Terra yesterday this and it's
Speaker 2: fantastic because the multiple of seven referrals is very interesting
Speaker 2: when you're in the clinical operation space. Obviously, the one
Speaker 2: of the key pieces a lot of us are interesting
Speaker 2: when you're advertising studies is the content itself. I'm just
Speaker 2: very interested in because of the basis of the functionality,
Speaker 2: it sounds like it's pretty much a keep it simple,
Speaker 2: stupid type mentality. We think in that with regard to
Speaker 2: the content thing and then presenting, because obviously the protocols
Speaker 2: themselves are extremely complex, and a lot of this conference
Speaker 2: is about how we best engage patients by making it
Speaker 2: as simple as possible. So I guess how you're generating
Speaker 2: that content, and then how you guess being able to
Speaker 2: that process of bringing it to making sure it's as
Speaker 2: understandable for patients to engage with right.
Speaker 4: So I think as part of our prioritization we focused
Speaker 4: on as part of our MVP, we started with the
Speaker 4: protocol page three design and from our patient interviews, we
Speaker 4: understood what's more important to the patient. For a patient,
Speaker 4: what's important is their disease. For sometimes when you go
Speaker 4: to any website, any clinical trial webs you see various categories,
Speaker 4: but they don't make sense to the patient. For the patient,
Speaker 4: it makes sense as their disease. So when you see
Speaker 4: the new version of the website, you would see there
Speaker 4: are two filters there, disease and location. We also understood
Speaker 4: from the patient interviews that location is the most important
Speaker 4: factor when they go ahead and select a site after
Speaker 4: they are matched to a trial. So it's important for
Speaker 4: them to know if you know, the site is closer
Speaker 4: to them, and it's also helpful later on reducing the
Speaker 4: drop rate. Right, So, considering these two important factors, considering
Speaker 4: the patient experience and what we understood from the interviews,
Speaker 4: we kept that as the most important item on our
Speaker 4: backlog and move that and push that for the MVP.
Speaker 4: And that's how our protocol pages also look like I
Speaker 4: can't show you right now, but if you go ahead,
Speaker 4: you know when you will check it out. You'll see
Speaker 4: on the top right we have the nearest recruiting site section.
Speaker 4: As soon as you've you entered the location, you'll see
Speaker 4: if there's a site closer to you or not. So yeah, really,
Speaker 4: patients driving that experience for themselves is what is the
Speaker 4: approach that we are following.
Speaker 3: Thanks,