AI-Enhanced ETL Testing: Ensuring Data Accuracy and Integrity in Healthcare Analytics

Abstract

Healthcare organizations are increasingly integrating data-focused systems and applications like business intelligence and analytics and data warehousing solutions to support data-driven decision-making. The accuracy and integrity of data within these solutions are essential to ensure that the insights and actions derived from them can be trusted. Data profiling is critical to ensuring data quality and is often performed as part of the ETL process at the beginning and end of ETL pipelines. However, end-to-end data accuracy validation and other aspects of quality and integrity are also important. As the volume and complexity of healthcare data continue to grow, so does the need for automation. This paper explores the benefits and challenges of using a combination of human-guided, machine-assisted, and AI-enhanced approaches to test the accuracy and integrity of data warehouse ETL pipelines and the implications and opportunities specific to ETL testing in healthcare analytics.