What Kinds of Evidence do Digital Health Technology Investors Expect?
Photo by me, drawing by my son (@6.5 y). His goal was 199, without mistakes.
What kinds of evidence do investors expect?

Rock Health, the killer seed fund that supports startups in digital health technology, hosted a killer panel recently with powerhouse moderator Deborah Kilpatrick of Evidation to get at the bottom of the growing need for proof generated by digital health technology companies.

Living at the crossroads of healthcare, IT, startups and end-user adoption/outcomes, I devoured this video (1hr:15min) and am sharing the best quotes by way of instruction and time saver for others here.

Quoted are panelists Skip Fleshman (Asset Management Ventures), Lisa Suennen (GE Ventures, Amy Belt Raimundo (Kaiser Permanente Ventures), and Dave Schulte (McKesson Ventures). Here we go.

0:45         In traditional health care, … we think about evidence in a fundamental way: A) does your product do what you say it does (i.e., performance validation)? and B) does that matter (i.e., clinical benefit, economic benefit)? The does-it-matter part is actually harder than the does-it-work part. ‒Deb Kilpatrick

1:10          When I first started my career in health care, that last question, wasn’t a question. I went through the first half of my career in health care, in Silicon Valley, and never once talked about economics. Those were the good old fee-for-service days! ‒Deb Kilpatrick

2:35          My definition of evidence? Somebody pays for it. — Lisa Suennen

4:08          My definition of evidence …? In digital therapeutics/interventions: …. running a study, … showing improved outcomes. The economic piece is also there. In other areas (e.g., scalability or workflow solutions): showing ROI. — Skip Fleshman

5:38          The more clinical you are (the more clinical decision making you are affecting), the more traditional clinical outcomes are relevant. But there also is an evidence component in “Did something change because of the data you presented?” Did someone make a different decision in the implementation and use of that data, and did that have an outcome that matters? Did anybody do anything differently, … and does it justify the cost of implementationThat piece gets missed a lot in the evidenceThe cost is not the cost of the technology… It is all the downstream cost of implementing that technology.— Amy Belt Raimundo

10:20        Just because you can digitize something, does not mean it matters. There is not a whole lot of new things being done, honestly. It’s just done in a different way. That’s great, but did it change anything, that’s what needs to be proven. Is it meaningful, and countable, in large dollars, with many zeros. It’s the change quotient that is the most important part of the evidence. — Lisa Suennen

11:40        We’re reaching a day of reckoning for a lot of digital health companies. Those that meet the [stated, above] requirements, will survive, and those that do not will have to pivot, may have to find different business models, or may not survive. — Dave Schulte

12:30        Tangible example: A couple of years ago, we used to see a lot of pitches where there would be slides with charts showing engagement, or utilization of an app, how much time someone spent with an app. Now I think that is just table stakes. What I really want to see is: Have you changed behavior? Changed an outcome? Do you have customers that are willing to pay for it? — Dave Schulte

13:30        The vast majority of the companies in digital health never get very big, … 5 to 10 million in revenue may be valuable, but is a small business. Those are not appropriate companies for venture backing. People should distinguish between those things. We [VCs] need revenues of hundreds of millions of dollars. — Lisa Suennen

14:45        Randomized controlled trials is what gets them [digital health companies] in the door [with clinicians]. But then clinicians look for other things, such as long-term patient engagement, … but also how mobile can re-imagine how care is being delivered, such as reduce the number of face-to-face visits. Randomized controlled studies as the table stakes versus new alternatives that are not existent in the physical world. What are you doing with that, ….that new level of evidence? — Amy Belt Raimundo 

17:36       Value is created as you begin to publish – there is not much talk about that, and get key opinion leaders on board. — Skip Fleshman

18:00        Initially, there were a lot of tech people trying to solve problems in health care, and they were not comfortable with the data. It took forever to extract the data [from a pilot], .. and to run proper studies. I encourage entrepreneurs to bring in the health care professionals. — Skip Fleshman

20:00        The closer you are to the patient, the higher the bar for the evidence. — Amy Belt Raimundo.

20:43        There has to be a pull from somewhere. Where I am actively seeking is where our clinicians are saying “I have a problem, a significant problem, …. and I actually need to solve this.” …We are not going to pay way up for something that has not proven any evidence, that is in development, … early-stage. The stage dictates the evidence level. Unfortunately, there is not a hard and fast on that rule. — Amy Belt Raimundo.

21:51        A lot of folks from health IT or from tech that wasn’t pharma or med tech, spent a lot of energy and sometimes still do, trying to avoid FDA-approval. Avoid legitimate trials. And that is just a mistake. Yes, it costs more money and it takes more time. No doubt. But that is how you create value. You have to do that, you cannot get around it. Your failure to do that makes you look unsophisticated. — Lisa Suennen

23:17        The failure to recognize that you have entered the medical world, by many digital health companies, is a terrible mistake. — Lisa Suennen

23:37        The evidence or FDA-approval is actually a confidence builder that what you have is legitimate. And that we should spend our time on it. This is from an adoption perspective, not VC. Folks are not going to want to spend the incredible cost associated with potentially implementing, and linking all our systems. … We are only going to do that if it really matters, and you have evidence that it really matters. — Amy Belt Raimundo

26:26        It is germane to how entrepreneurs need to think, early on in their development: what evidence base do you need to develop for whom? … And this is done in parallel: you’re investing in clinical evidence all the while you’re developing an economic rationale. You have to be able to iterate between those two discussions depending on who the call point is you are trying to sell to. — Dave Schulte

What does Bad look like? We’ve spent a lot of time on what Good looks like, but what are some “Don’t-Come-In-and-Do-This”? What are things that people should avoid in 2017 and beyond [in digital health technology]? — Deb Kilpatrick H3

27:57        It cannot add any cost. It has to take away cost, whatever it is. The medical world is a zero sum game. If you cannot talk about how your evidence leads to reduced cost, – and I see a lot of this, it’s not going anywhere. — Lisa Suennen

30:22        As an entrepreneur you have got to find cost-effective ways to de-risk the question of health economics over time. In other words, do something with 5 users or 5 patients. Find ways to do small incremental pilots, or tests or studies that hold up to an initial discussion of evidence. But if you say “We have not gotten to that point yet because we don’t have the money, that suggests you have not really thought about how you are going to get there once you do have the money. — Dave Schulte

31:04:      Don’t come in and say machine learning is going to solve everything. — Dave Schulte

31:44        The ROI calculation cannot be only on the cost of the new technology. The big costs are being held by the person you are talking to as the customer… The ROI has to factor in the implementation and change management cost of executing whatever your solution is. And the number should still be positive. That is a new level of sophistication. — Amy Belt Raimundo

32:17        The other thing we see a lot is nobody has ever talked to patients. You’ve developed a product that you think solves a problem, and it may theoretically, but the only patient you built it for is your grandmother. — Lisa Suennen

32:40        One of the problems with clinical trials, even the traditional ones, is that the evidence achieved is not often consistent with the evidence desired by the end-user, the patient. How do you actually define what is good in the patient terms, which are often very different than the physician terms. Evidence of the user, the patient, is essential and often overlooked. — Lisa Suennen

What are unintended consequences of digital health [technology], in addition to physician burnout, too much stuff, too much information? — Deb Kilpatrick H3

 41:50       We are evolving to a world where some of these digital health tools have to be combined with services, and that is a new realization that we are starting to wake up to. — Dave Schulte

42:16        Healthcare is all services, that is how they [providers] make money. We have to get passed that [the notion that services are not scalable, no margins, et cetera]. It is humans that use them [the services], … and we have to keep the humanity in the services. While technology can make it scalable, more accurate, and bring data to the decision making and all that, the combination of tech and services is where it going to be at. That is how you create sustainable value. — Lisa Suennen

46:40        It [technology X] is an enabler. It is taking load off the physician. It is not adding load onto the physician. It takes away some of the burden— Amy Belt Raimundo

Food for thought… big time. In the remainder of the video, panelist answer questions from the audience. Cheers, happy weekend!

Chantal Kerssens understands how people think, make decisions, and behave. And she knows how people interact with new health technologies. Contact Chantal for all of your digital health application needs. 

Share this: