A Fault-Tolerant Microservices-Oriented Pipeline Architecture for Scalable and Resilient Distributed Data Engineering Systems

Abstract

Monolithic data pipeline systems have serious problems with large scale distributed data processing: they lack scalability, are strongly coupled, and have poor fault isolation. Such constraints usually lead to bottlenecks in performance, more downtime in the system, and less flexibility to dynamic data engineering settings. In order to solve these problems, the paper suggests a fault-tolerant microservices-oriented pipeline architecture that would improve the scalability, resilience, and modularity of distributed data engineering systems. The solution has proposed to use Directed Acyclic Graph (DAG)-based scheduling to handle task dependencies and execute them in parallel and an event-driven communication model to handle asynchronous and loosely coupled interactions among microservices. These methods enable effective workflow coordination, adaptable resource distribution as well as heightened system responsiveness. Key metrics are used to assess the performance of the recommended system, with the key metrics being throughput, latency, execution time, scalability, and Mean Time To Recovery (MTTR). Experimental evidence shows that there is a great improvement when compared to conventional monolithic pipelines. In particular, the proposed system will increase the throughput up to 83.3, reduce the latency by around 52, reduce the execution time by 66.7, and reduce the MTTR by 60. Also, the system is almost scalable with the number of computational nodes. All in all, the findings agree that the proposed architecture is effective in terms of performance, scaling, and fault tolerance, which makes it appropriate to the modern distributed data engineering application, where high reliability and efficiency are essential.