Resource-Aware Distributed Task Scheduling and Dynamic Load Balancing for High-Performance Data Pipeline Execution

Abstract

Task scheduling and load balancing are of high importance in distributed data pipeline systems where it is a common phenomenon to experience underutilization of resources and performance bottlenecks with the use of static methods. In this work, the authors suggest a resource-aware dynamic scheduling model that combines Heterogeneous Earliest Finish Time (HEFT) with flexible allocation of priorities and work-stealing load balancing scheme. The hybrid method is suggested to dynamically utilize the resources in the system, such as CPU usage and memory, and queue length, to optimize the placement and execution of tasks. The experimental analysis done on the simulated distributed environment shows that performance improvement is significant and includes; reduction of the average task latency by 28.6%, increase in throughput by 34.2% and improvement in resource utilization efficiency by 22.8% against the traditional scheduling methods Round Robin and the static HEFT. It also has enhanced distribution of load through the framework and less imbalance between nodes. These findings affirm that the presented approach is effective in terms of scaling, adaptability, and performance since the execution cost is adequately low and suitable to the application of the high-performance distributed data pipeline.