Automated Schema Evolution Handling in Machine Learning Data Pipelines

Abstract

Schema changes are inevitable in evolving data ecosystems and pose significant risks to machine learning pipelines. This research proposes an automated schema evolution handling mechanism that minimizes pipeline failures and ensures backward compatibility. Evaluation results demonstrate improved system resilience.