Developing a data lakehouse for a South African government-sector training authority : implementing quality control for incremental extract-load-transform pipelines in the ingestion layer

Abstract

The Durban University of Technology is undertaking a project to develop a data lakehouse system for a South African government-sector training authority. This system is considered critical to enhance the monitoring and evaluation capabilities of the training authority and ensure service delivery. Ensuring the quality of data ingested into the lakehouse is critical, as poor data quality deteriorates the efficiency of the lakehouse solution. This chapter studies quality control for ingestion-layer pipelines to propose a data quality framework. Metrics considered for data quality were completeness, accuracy, integrity, correctness, and timeliness. The framework was evaluated by practically applying it to a sample semi-structured dataset to gauge its effectiveness. Recommendations for future work include expanded integration, such as incorporating data from more varied sources and implementing incremental data ingestion triggers.