Data, Data Everywhere…
Date:
I presented a talk on the challenges and solutions in modern data engineering, exploring how organisations can move from having abundant data to actually deriving actionable insights from it. The title “Data, Data Everywhere…” references the paradox that many companies find themselves drowning in data yet struggling to extract meaningful value—much like the Ancient Mariner surrounded by water but unable to drink.
The presentation covered the Data Science Hierarchy of Needs, illustrating how data engineering forms the foundational layer that enables analytics, machine learning, and AI. I walked through the Data Maturity Curve showing how businesses evolve from basic reporting (what happened) through predictive analytics (what will happen) to embedded intelligence (how to optimise based on predictions). I emphasized that building a data-driven culture requires investment not just in people and technology, but also in governance, data quality, and management support.
A key focus was the Data Engineering Lifecycle—covering ingestion, storage, transformation, and serving—along with the critical undercurrents of security, data management, DataOps, orchestration, and software engineering practices. I shared a practical case study of a modern data engineering infrastructure stack, including tools like Airflow for orchestration, modern testing frameworks, CI/CD platforms, and various storage and processing technologies that enable scalable batch and streaming pipelines.
The session concluded with an interactive discussion about whether the community prefers learning specific tools through hands-on workshops or understanding broader approaches and methodologies for building data systems in business contexts.