"In order for it to be commercial, IBM shouldn't be putting their hands on the data. They shouldn't have analysts in there building models for the clients; that should all be available to the client," Burghard said.
Another problem, Burghard pointed out, is Watson was trained using data from IBM development partner MSKCC.
Because the system was trained using the hospital system's data, its query results tend to be biased toward that institution's cancer treatments and is not as inclusive with data from other hospitals, such as the Mayo Clinic or other smaller facilities.
While that may work for simple cancers whose treatments are relatively generic, Burghard said complex cancers are treated differently in various facilities. And, smaller hospitals may not even have access to the same treatment options as their larger, regional counterparts.
Because of its failure to produce the results promised, Burghard said IBM has lost momentum and "there's such skepticism that unless they have some silver bullet sitting behind the curtain, I think it's just going to plug along until the stock holders say they've had enough."
In 2012, one of the first pilots of Watson for Oncology took place with the M.D. Anderson Cancer Center at the University of Texas.
The hospital used IBM's Watson supercomputer to expedite clinical decision making by matching cancer patients to clinical trials to improve outcomes "worldwide." At a final cost of $62 million, the cancer centre's Watson-backed Oncology Expert Advisor (OEA) never got off the ground and was halted after an external audit was sought by the university.
The initial scope of OEA system development was for MDS leukemia, but it was expanded in February 2013 to include five additional types of leukemia, then in December 2014 to include lung cancer.
The audit revealed the Watson Oncology system could not integrate with M.D. Anderson Cancer Centers' EPIC electronic medical record (EMR) system, so internal pilots of the OEA for Leukemia and Lung cancer were conducted using the prior medical records system (ClinicStation).
The Cancer Center and IBM Watson ceased active development in 2015. And IBM ended support for the OEA Pilot System and for the OEA Demo System effective 1 September 2016.
The system is not in clinical use and has not been piloted outside of M.D. Anderson, according to the the audit.
The IBM agreement at the time the project was halted said the system "is not ready for human investigational or clinical use, and its use in the treatment of patients is prohibited" except as needed to test and evaluate the system, according to the University of Texas audit.
When asked by Computerworld why the project failed, M.D. Anderson Cancer Center said via email: "While a variety of approaches have been examined, a final approach using [cognitive computing] to benefit patients has not been determined at this time.
"MD Anderson is committed to continuing to explore how digital solutions can accelerate the translation of research into advanced cancer care for patients."
Computerworld also reached out to Memorial Sloan Kettering Cancer Center and the Mayo Clinic, two of IBM's top development partners on Watson Health who've been cited as success stories for training Watson and using it for clinical trial matching.
Begun in 2014, Watson's job at the Mayo Clinic was to sift through thousands of medical studies and ensure that more patients are accurately and consistently matched with promising clinical trials. (IBM has announced that enrolment rates for breast cancer clinical trials at the Mayo Clinic had increased dramatically.)
A request for comment from the Mayo Clinic on Watson's effectiveness was not returned. A Mayo Clinic spokesperson said multiple attempts had been made at reaching the physician in charge of the Watson project but were unsuccessful.
A Memorial Sloan Kettering Cancer Center spokesperson referred questions to IBM, stating that IBM receives feedback on Watson for Oncology directly from its customers, and while the hospital trains Watson's AI with its data, "we do not use it here."
Another clinic touted early on by IBM is the Highlands Oncology Group (HOG), which participated in a feasibility study of IBM Watson to increase the efficiency and accuracy of the clinical trial matching.
Located in Northeast Arkansas, HOG has 15 physicians and 310 staff members working across three sites; the facility's pilot lasted 16 weeks and used data from 2,620 visits by lung and breast cancer patients.
In an initial pre-screening test, the HOG clinical trial coordinator took 1 hour and 50 minutes to process 90 patients against three breast cancer screenings. Conversely, when the Watson's clinical trial matching platform was used, that job took 24 minutes. "This represents a significant reduction in time of 86 minutes or 78%," HOG said in a statement.
Computerworld reached out to HOG about the Watson trial, and asked specifically if there were any problems during the pilot; HOG's medical director said the clinic had signed a confidentiality agreement with IBM and was not allowed to give out any information.
"So, IBM Watson would be the ones that provide you the concerns and road blocks they've run into," a HOG spokesperson wrote via email.
An IBM buying spree, and what comes next
Upon completing all three acquisitions, IBM boasted its Watson Health Cloud housed "one of the world's largest and most diverse collections of health-related data, representing an aggregate of approximately 300 million patient lives acquired from three companies."
"They all in their own right, before they were acquired, were very successful companies and had good, strong, loyal client bases and were plugging along. I think IBM thought, 'We should buy these guys and throw in some AI and really take the market by storm,'" Burghard said. "As far as I can tell, that hasn't happened."
At least one of those acquisitions, Truvan, was recently cited by IBM's Kelly as key to moving insurance provider data onto the IBM Watson Health platform now that it's going to be offered through a hybrid cloud.
In late October, IBM announced plans to seed its new hybrid cloud model for Watson by first moving data from insurance payer systems. For that, Truvan will be key.
"They [Truvan] are very big in the payer space," Kelly said. "We process payer claims and we have payer records. So, what does it cost for a certain procedure in a state or in a hospital – that is a very rich data set we can apply AI to to dramatically reduce cost."
Once payer data is moved to the hybrid cloud, the electronic medical records (EMRs) acquired through the Explorys acquisition will follow, Kelly said.