With Benefits Like These, Who Needs the Costs?

Inevitably, as more and more companies invest in new information technologies for clinical research, they are beginning to ask: “Am I getting any benefit from this investment?” As it turns out, this is either a very simple or a very complex question. The simple answer, that many companies find to be quite legitimate, is what we call the value of necessity, i.e. “we could not have done it any other way.” The complex answer, when a company seeks to actually calculate a cost/benefit, or a formal Return on Investment (ROI) in terms accepted in the financial world, is much more difficult to arrive at, primarily because very few companies conducting clinical research understand their true costs or even how to start identifying them. If you do not understand your true costs, then obviously you will have a hard time knowing if a change has affected them positively or negatively.

Leaving financial calculations aside (a topic for another column), I have become more concerned lately about a fundamental misunderstanding about the benefits of new information technologies in clinical research. These misunderstandings can be quite dangerous, because they can create unrealistic expectations, impossible goals for managers to achieve, and lead to inappropriate process change initiatives. With benefits like these, who needs costs?

Let’s look at four benefits typically expected by adopters of electronic data capture (EDC): faster database lock; real-time data; reduced outsourcing costs; and accelerated decision-making.

Faster Database Lock?

One of the most universally accepted benefits of EDC is faster database lock (DBL): by catching eighty percent or more of common data entry errors at the time of entry, the number of errors that need to be followed up in person is drastically reduced, and the study team (monitors and data managers) can keep up with the dataflow in such a timely manner that little needs to be cleaned up at the study close. Of course many factors contribute to whether this will be achieved or not, including site training, monitor participation, back-end database issues, adverse event reconciliation processes, third-party data sources such as central laboratories, and so on. But nonetheless, many companies have now reported database lock times using EDC as certainly a matter of days (instead of weeks), and sometimes even a matter of hours.

So, this is an easy one, right? Faster DBL is a benefit of EDC. But for whom? How is this benefit manifested in a clinical development organization? The key is this: if you are not ready to start the next study in the clinical development plan at the moment you have obtained the necessary results from the just-closed EDC trial, but instead start that next study when you always would (some months later), then you have not achieved any overall drug development time savings. Faster DBL yes, timeline compression no. And without the latter, the former is only mildly satisfying and modest in dollars saved.

Yet we rarely hear anyone talk about this — either sponsors or vendors. The problem will be when someone wants to justify the EDC expense by faster DBLs. An executive with sufficient vision, beyond one trial at a time, will look back and say, “my darn drug got to the FDA no faster with EDC, so what was all the hoopla about?” Faster DBLs are not enough; your clinical development program has to change its calendar, its fundamental way of thinking about time. EDC helps you do that, but only the process change will achieve the business benefit.

Real-Time Data?

The advent of the Internet for applying information technology has given us the great opportunity of “real-time” data, or so they say. This phrase is a pet peeve of mine (along with “hybrid systems”). First, let’s ignore the implication that there is “unreal time” (perhaps only in trials of hallucinogenic drugs). The real problem with the hype about real-time data is that no one seems to ask themselves if they need the data in real time. Do you need to know what patient 2576B’s blood pressure value was within minutes of it having been taken? Would it be alright if you knew tonight instead of “right now”? How about by the end of the week? Or two weeks from now?

In some very specific examples of particular trial designs, one can imagine how real-time data might be useful (perhaps in a Phase I of a potentially dangerous compound, or to track a developing trend in Phase II that raises a safety question, or is producing such a positive outcome as to justify a compassionate use exemption). Otherwise, every time I have asked this question of groups large and small, almost no one has a reason why they need the trial data right now.

On the other hand, there are many circumstances when we need trial data just in time, and that is a very different concept. If we are trying to make a judgment about a site’s recruitment performance, perhaps to pull further patients away from them because it is taking them too long, at the time we need to make that decision we would like to have the most up-to-date information about that site. We don’t want to have to wait until we can send a CRA out there to see what the status is, but neither did we need to know, every hour since the trial started, hour by hour, how that site was doing.

This is the crucial difference between real-time data and just-in-time data. The implication is that an EDC technology which can’t get you real-time data, but can get you data more or less instantly when you ask for it, is probably just fine for your needs. That’s the expectation you want to set, and that’s the benefit you want to pay for.

Reduced Outsourcing Costs?

Nothing is more loaded for setting executive expectations than to tell them that EDC will save on outsourcing costs. Whose costs did we have in mind, exactly? Usually people mean that EDC enables the sponsor to bring more of the data handling process (or projects) “back” in house, where it is assumed to be cheaper, getting it out of the hands of those expensive CROs. Or for the more sophisticated EDC users, in-licensing the software is assumed to be cheaper than relying on an ASP-delivered EDC solution (where a vendor provides all the necessary ancillary infrastructure and process required to make an EDC trial run).

Well, we may indeed have reduced our outsourcing costs, but what have we done to our internal costs in so doing? Very few sponsors understand the cost of their routine processes today, as we have said, much less understand the myriad cost implications to internalizing the new EDC process, especially if the software has been in-licensed. The operational implications of internalizing EDC are broad, and potentially costly. We usually advise sponsors to think of EDC as “cost neutral”, at least in the short run.

This is not to say, by any means, that EDC is not cost-effective. But throwing around phrases like “EDC will reduce outsourcing”, or “EDC will save money,” without knowing how to back up those statements, creates the worse kind of expectations among management, financial staff and clinical researchers. It is highly unlikely, if someone who knows what they are doing comes in to do a financial audit, that the average EDC project could come out in the black — not because it isn’t in the black, but because those using EDC don’t know how to prove the cost differential properly.

Accelerated Decision-Making?

Another easily accepted benefit of EDC, which is incorporated into people’s expectations without much thought, is that because EDC allows for faster DBLs and more accurate, faster interim analyses through real-time data, that EDC can accelerate decision making. With the clinical data coming in faster and cleaner, we can make in-trial decisions (close that under-performing site) or program decisions (kill this equivocal drug) faster, with all the attendant cost-savings implied (eliminating future development of a drug going nowhere, or accelerating a good drug through a slimmer series of tightly knit trials).

This is indeed a potentially powerful benefit of EDC. The problem with this one is more subtle: are we, by killing off a compound faster, losing the chance to learn more about it at a more steady, thoughtful, scientific pace? Are we closing the door to serendipity, that highly productive R&D tool which has produced so many blockbusters? This strategic debate rages up and down the executive halls of most biopharmas, and does not necessarily have anything to do with software. But the extent to which EDC is “sold” on this concept, this “benefit” may backfire if/when management decides its decisions are accelerated, yes, but also too mechanical.

There are many benefits, provable objectively, to electronic data capture, so much so that few companies are seriously planning a long-term clinical development strategy which ignores EDC and sticks to NCR paper. Those seeking to speed its adoption, in an industry which needs computerization so urgently, need to use words carefully, and need to differentiate true benefits from false gods. Timeline compression is what speeds submissions, not faster database locks. Just-in-time data is what we need, not real-time data. The operational cost of using EDC may not be less than how you do research today. And accelerated decision-making, at the cost of lack of insight, is no benefit at all. Keep the true benefits in your sights, and you will mislead or disappoint no one. Instead, your investment will reap the returns it deserves.