AI-Driven Data Warehousing: ML Innovations for Performance, Prediction, and Cost Optimization (Published)
The rapid expansion of big data has accelerated the evolution of Data Warehousing (DWH) from static, rule-based systems to adaptive, intelligent, and automated frameworks powered by Machine Learning (ML). Traditional data warehouses face challenges in scalability, efficiency, and real-time analytics, which ML can effectively address. This paper presents an integrated ML-driven optimization framework that enhances data storage, query performance, and analytical capabilities across ingestion, transformation, and execution layers. The framework leverages supervised, unsupervised, and reinforcement learning techniques to predict query costs, optimize execution plans, detect anomalies, and forecast workloads for dynamic resource allocation. AI-powered automation further improves data integration, schema evolution, and adaptability to changing workloads. Experimental evaluations on PostgreSQL and cloud-native environments demonstrate measurable gains in query latency reduction, storage efficiency, and operational cost optimization through adaptive indexing, workload forecasting, and ML-assisted plan selection. The study also addresses emerging challenges such as computational complexity, data security, and model explainability, along with the potential of cloud-based and federated learning for distributed data management. By embedding ML intelligence within DWH operations, organizations can achieve predictive scalability, cost efficiency, and governance assurance, transforming traditional warehouses into autonomous, self-optimizing analytical systems aligned with modern business needs.
Keywords: Automation, adaptive indexing, anomaly detection, cloud data warehouse, data governance, data integration, data warehousing, federated learning, machine learning, query optimization, self-tuning database, workload forecasting