The customer, a leading global manufacturer of commercial vehicles, had a large on-prem data platform (Cloudera data lake), which had the inherent limitations of high dependency on data platform engineers, slow delivery, performance challenges, etc.
They wanted to adopt a data platform approach that:
• Is scalable.
• Provides data ownership by domain.
• Has clear data governance.
• Provides easy data availability and self-serve capabilities.
• Provides data as a product.
ClearPeaks helped in designing and implementing a Data Mesh based platform with CDP (Cloudera Data Platform) & AWS.
This included solutions for self-service provision of data workloads with Storage, Impala, Spark, PySpark, Airflow & NiFi.
We also developed data platform health monitoring solutions for long-term stability.
With this project, we have pioneered the implementation of the fourth generation EDP (Enterprise Data Platform) paradigm, data mesh, at an enterprise scale.
Our expertise was vital to the design, architecture & implementation of this complex, large-scale deployment with a strong focus on best practices in software development & DevOps, CI/CD & IaC, and Agile.
Most importantly, our self-service provisioning enables business users to make data-driven decisions with ease and confidence.
