How to Compare Competing AI Vendor’s Engine Performance Prior to Purchase
NLP in healthcare for risk adjustment is among the fastest-growing AI disciplines. As with any emerging technology, however, limited actual competitive market performance history makes it difficult to distinguish what to truly expect. Highly polished sales presentations can set unrealistic expectations, so validating the true competitiveness of the options being considered prior to purchase is essential.
Given the lack of market maturity, each vendor offers their own spin on how to evaluate NLP engine performance, featuring the metrics they feel best showcase their inherently limited data. To contrast actual performance in a true apples to apples comparison, those looking to invest in NLP in healthcare for HCC Risk Adjustment would be best served to compile their own sample set of 100 to 200 member charts to test with the different vendors they are considering.
By sharing the same sample set of chart data with competing vendors, you can evaluate how they perform with your specific mix of cases. Each vendor will undoubtedly have impressive results to share with their sales materials, but you can count on those results to be hand selected to show their offering in the best possible light, which may or may not reflect your case mix. If a vendor is at all hesitant to participate in such side-by-side comparisons based on your supplied cases, that should tell you everything you need to know about what they might actually deliver.
The quality of NLP engines is determined by a combination of the AI programming sophistication and the volume of cases the engine has been applied against during development. Some have strong programming applied to only a nominal volume of cases, meaning it may perform well against limited case types. Others may produce more generic results for a wider range of cases, automating HCC Risk Adjustment efforts but not really improving HCC identification results. Ideally, you want to find an engine developed with strong programming sharpened against a very large pool of sample cases.
Contrasting HCC Risk Adjustment findings from the following workflow combinations will help you gauge the complete potential financial impact of the options being considered:
- Your coding teams original HCC Risk Adjustment results
- Your Auditing team’s additional HCC Risk Adjustment results
- Prospective vendor’s raw NLP engine’s HCC Risk Adjustment results
- Combined vendors coding team and NLP engine HCC Risk Adjustment results
The range of identified HCCs from the different vendors, both as stand along technology plug-ins or combined with coding labor, will expose the value you can expect along with the total solution costs. Vendors who are confident in their competitive capabilities will be more than happy to conduct such Proof-of-Concept exercises at no cost to earn your business.
In an effort to provide the industry with open market competitive outcomes, The University of South Carolina, in conjunction with RAAPID, are actively seeking sample chart sets to test. Those who choose to participate will be provided with de-identified competitive data for the workflow options above to see how their internal teams performed as well as the value added by the NLP engine. Participants will also be given the opportunity to contribute to the resulting white paper and be cited as co-authors with the subsequent published effort.
If you are interested in participating in this no-cost Proof-of-Concept study, please contact us.
