MIT Sloan Management Review Article on The No. 1 Question to Ask When Evaluating AI Tools

In the fast-moving and highly competitive artificial intelligence sector, developers’ claims that their AI tools can make critical predictions with a high degree of accuracy are key to selling prospective customers on their value. Because it can be daunting for people who are not AI experts to evaluate these tools, leaders may be tempted to rely on the high-level performance metrics published in sales materials. But doing so often leads to disappointing or even risky implementations.

Over the course of an 11-month investigation, we observed managers in a leading health care organization as they conducted internal pilot studies of five AI tools. Impressive performance results had been promised for each, but several of the tools did extremely poorly in their pilots. Analyzing the evaluation process, we found that an effective way to determine an AI tool’s quality is understanding and examining its ground truth. In this article, we’ll explain what that is and how managers can dig into it to better assess whether a particular AI tool may enhance or diminish decision-making in their organization.

About the Author

Sarah Lebovitz is an assistant professor at the McIntire School of Commerce at the University of Virginia. Hila Lifshitz-Assaf is a professor at Warwick University and a faculty affiliate at the Lab for Innovation Science at Harvard. Natalia Levina is a professor at New York University’s Stern School of Business.

Learn more about MIT SMR.