A Dynamic Metadata-Driven ETL/ELT Orchestration Framework for Adaptive and Scalable Data Pipelines in Multi-Cloud Environments

Abstract

The accelerated distributed data processing has heightened the necessity to effectively orchestrate the ETL/ELT in multi-cloud settings. Nevertheless, the conventional methods of scheduling (as static) have huge disadvantages such as poor use of resources, higher execution time, and lack of responsiveness to changes in workload. In order to overcome these difficulties, this paper introduces a dynamic metadata-driven ETL/ELT orchestration framework that will allow carrying out data pipeline executions in heterogeneous multi-cloud environments in a dynamical and scaled manner. The given framework makes use of metadata-conscious scheduling algorithm that dynamically examines the features of tasks, availability of resources, and execution dependencies in order to optimize the distribution of tasks and execution process. The system has the capability to make intelligent decisions and effectively coordinate distributed pipelines through the addition of real-time metadata like the workload size, node capacity and historical performance metrics. The experimental performance in simulated multi-cloud setting has shown that the suggested method brings considerable performance gains in comparison with the traditional scheduling algorithms. In particular, the framework yields a significant reduction in the overall latency of execution, a significant improvement in throughput due to higher levels of parallelism, and a large-scale improvement in the utilization of resources distributed across distributed nodes. The findings verify that the suggested metadata-based orchestration model offers a scalable and effective system to next-generation data engineering systems and is therefore most appropriate in dynamic and large-scale multi-cloud data processing applications.