Abstract

The rapid adoption of hybrid cloud architectures has transformed modern data platforms by enabling organizations to combine the scalability of cloud computing with the control and security of on-premises infrastructure. However, maintaining reliable and continuous data ingestion across heterogeneous environments remains a critical challenge due to dynamic workloads, network fluctuations, infrastructure failures, and variations in computational resources. Existing fault-tolerant data ingestion mechanisms primarily rely on reactive strategies, including replication, checkpointing, and predefined recovery rules, which are unable to proactively identify potential failures or optimize ingestion performance under changing operating conditions. This paper proposes AIM-DIF (Artificial Intelligence-based Monitoring and Data Ingestion Framework), an AI-driven adaptive fault-tolerant framework designed for hybrid on-premise and cloud big data platforms. The proposed framework integrates continuous system monitoring, feature-based performance analysis, ensemble machine learning-based failure prediction, and adaptive recovery decision mechanisms to improve data ingestion reliability and operational efficiency. AIM-DIF models ingestion performance as a function of key system characteristics, including fault detection rate, recovery time, data throughput, and system availability. Ensemble learning techniques, including Random Forest Regression and AdaBoost Regression, are employed to predict ingestion performance and support intelligent decision-making for fault mitigation. Experimental evaluation demonstrates that the Random Forest model achieves superior predictive performance with an R² value of 0.8375, while AdaBoost Regression achieves an R² value of 0.5682, indicating that Random Forest provides better stability and accuracy for predicting hybrid cloud ingestion behavior. The proposed framework demonstrates the ability to transform conventional reactive fault tolerance mechanisms into predictive and adaptive operations by integrating machine learning with intelligent recovery strategies. AIM-DIF provides a scalable approach for improving reliability, reducing service interruptions, and enhancing performance in modern cloud-native data platforms. The proposed framework has potential applications in enterprise data lakes, cloud analytics platforms, IoT processing systems, financial transaction pipelines, healthcare data platforms, and largescale distributed computing environments. Keywords: data ingestion; hybrid cloud; fault tolerance; ensemble learning; Random Forest; AdaBoost; big data platforms; AIOps; failure prediction; adaptive recovery