Data Engineering Podcast – Podcastlijstje

This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.

Afleveringen

Entity Resolution at Scale: Graphs, LLMs, and Real-World Data Messiness

2 november 2026 · 59 min
Afspelen

Summary
In this episode, I spoke with Steven Renwick, CEO and co-founder of Tilores, about the realities of entity-oriented data processing and the practical role of LLMs in entity resolution. We covered the confusing terminology around entity resolution, identity resolution, record linkage, named entity recognition, and disambiguation, and why these problems show up almost everywhere once organizations reach a certain scale. Steven explained why entity resolution is especially critical in domains like financial services, public sector systems, logistics, and any environment where messy, duplicated, or inconsistent records can create compliance, operational, or analytical risk.
We also dug into the question of whether LLMs can replace traditional entity resolution systems. Steven shared why the answer is only partly yes: LLMs can be very useful for discovery, entity extraction, weak-match review, onboarding, and configuration, but they struggle as the core production matching engine because of cost, scale, consistency, explainability, and state management requirements. We explored why graph structures are so useful for representing entities, how TeloRes approaches scale without relying on graph databases directly, and why teams should think carefully about build-versus-buy when tackling entity resolution. The conversation closed with lessons from real-world data quality, the growing importance of trustworthy identity layers for AI and analytics, and how clean entity data can strengthen everything from knowledge graphs to AI applications. Lees meer

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management

Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products a…
How Amazon Uses Telemetry and AI to Automate Physical Logistics

20 oktober 2026 · 50 min
Afspelen

Summary
In this episode, I spoke with Keerti Kulkarni about the data and AI systems behind physical logistics automation at Amazon. She explained how her team supports dock-door operations across more than 100 North American facilities by turning telemetry from physical workflows into actionable operational intelligence. We discussed the end-to-end data path from CloudWatch, Firehose, and S3 into Redshift and AI-powered analysis, along with the realities of handling petabyte-scale event streams, late and duplicate data, reconciliation, cost controls, and the need to treat physical telemetry as a first-class signal rather than noisy exhaust. Kirti also shared how these systems are designed to protect worker privacy while still surfacing the right information for safety and operational decisions.
We also dug into the role of LLMs and agentic workflows in helping operations teams diagnose failures quickly through conversational interfaces. Kirti described how AI-generated summaries help identify root causes such as authorization problems, hardware issues, or connectivity failures, while human feedback loops, structured outputs, and strict vocabulary controls help manage hallucination risk. Beyond troubleshooting, we explored how the platform is evolving toward proactive maintenance, training insights, per-site cost attribution, and broader modular agent architectures that could be shared across logistics teams. It’s a detailed look at what it takes to combine data engineering, real-time operations, and AI in a large-scale physical environment. Lees meer

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management

Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core product…
Why Metadata Still Matters: From Data Catalogs to AI-Ready Semantic Hubs

7 oktober 2026 · 1 uur 25 min
Afspelen

Summary Lees meer

In this episode, I spoke with Christian Bremeau and Simon Dynin about their decades-long work in metadata management and how that experience shaped the evolution of Meta Integration Technology and its direct-to-customer MetaKarta platform. We explored the recurring challenges of fragmented data ecosystems, why every tool inevitably creates its own metadata silo, and why organizations still need a vendor-neutral system to stitch lineage, semantics, and change history across databases, ETL/ELT pipelines, BI tools, and cloud platforms. Christian and Simon explained why deep lineage, versioning, configuration management, and deterministic understanding of transformations matter far more than a polished catalog UI when teams need to trust numbers, assess impact, or satisfy audits.

We also dug into the growing importance of metadata in the age of AI. Christian and Simon shared their view that metadata management is becoming foundational for both BI and agentic systems, especially as organizations struggle with semantic drift, tool sprawl, and non-deterministic AI behavior. They introduced Metacarta’s new Semantic Hub as a way to reconcile business logic across legacy and modern tools, compile shared semantics into downstream systems, and provide consistent answers for both human analysts and AI agents. Along the way, we discussed metadata ops, the organizational processes needed to make metadata programs succeed, and the practical situations where a cross-platform metadata system is—or isn’t—the right fit.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management

Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products are a suite of APIs for retri…
Reducing Data Debt with Agile Ledger Architecture

25 september 2026 · 50 min
Afspelen

Summary
In this episode Christopher Doidge talks about his Agile Ledger Architecture (ALA) approach to data warehousing and how it aims to reduce data debt while shortening the path from raw data to trustworthy business insight. Christopher explained that ALA is not a replacement for existing warehouse patterns like medallion architecture, star schemas, or other modeling approaches, but a complementary discipline focused on pushing business definitions upstream, enforcing cleaner ledger-style transformations, and producing gold-layer tables that stakeholders can actually use without relying on analysts to repeatedly rebuild the same logic. He also discussed his “15-minute litmus test” for time-to-insight, the importance of durable documentation through a data dictionary, and why undocumented business logic living in analyst scripts is often one of the biggest hidden forms of data debt. Overall, this was a thoughtful conversation about designing warehouse systems that serve not just analysts, but the broader business as well.
Announcements Lees meer

Hello and welcome to the Data Engineering Podcast, the show about modern data management

Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto-optimal quality, cost, and speed. Whether you work on voice agents that need 200 millisecond latency, chat bots that balance speed, depth, and quality, or long-horizon agents to do thorough, overnight research for you, Parallel is a single platform for all your agentic research. Get started for free at dataengineeringpodcast.com/parallel

Your host is Tobias Macey and today I'm interviewing Christopher Doidge about a data warehousing ap…
What Context Really Means in Data Engineering and AI

15 september 2026 · 53 min
Afspelen

Summary
In this episode Soham Mazumdar, co-founder and CEO of Wisdom.ai, talks about what “context” really means in data engineering and AI systems. He explores why context has become such an overloaded term, spanning everything from semantic layers and data catalogs to tribal knowledge, query logs, dashboards, and even agent memory. Soham explained that the big shift is that context is no longer being prepared primarily for human analysts, but for LLMs and agents that can’t reliably fill in missing gaps on their own. That change raises the bar for how context is represented, validated, benchmarked, and maintained so that AI systems can produce trustworthy outcomes. Lees meer

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management

Your host is Tobias Macey and today I'm interviewing Soham Mazumdar about what "context" actually means in data engineering

Interview

Introduction

How did you get involved in the area of data management?

One of the perennial challenges of engineering in all forms is building a shared understanding of what a given word means. "Context" is one that is being used for an increasing number of purposes with the introduction of AI agents. Can you start by sharing some of the ways that this terminology overload has caused problems in your own experience?

Data engineering has arguably always been about context engineering, but at the scale of human consumers. What are the substantive changes that AI/agentic consumers bring to the discipline?

While we all understand the notion of "context", turning it into a useful and re-usable component is a different matter entirely. What are some of the ways that "business context" or "technical context" manifests as a tangible artifact?

This also brings up the question of data modeling. What are some of the key attributes that are necessary when storing, enriching, evolving, and joining into that context?

One could argue that the entire history of data warehousin…
Specialized AI for Data Engineers: Inside Astronomer’s Otto

27 augustus 2026 · 46 min
Afspelen

Summary
In this episode Yetunde Dada discusses Otto, Astronomer’s AI agent for Airflow, and the broader challenge of making agentic tooling actually useful for data engineers. She explored why generic coding assistants often fall short in data workflows, how Otto adds the missing context around Airflow, Astro, upgrades, and troubleshooting, and why Astronomer focused first on high-leverage use cases such as DAG authoring, investigation of pipeline failures, version migrations, and legacy scheduler modernization. She also discussed the practical realities of introducing agents into engineering teams: model choice, security boundaries, vendor lock-in concerns, validation of generated code, and the need for agents to fit into existing workflows rather than forcing users into new ones. Overall, this conversation offers a detailed look at how specialized AI agents can support data engineers today, and where Astronomer is headed next with a vision for self-healing pipelines that keep humans in control while automating more of the operational burden. Lees meer

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management

Your host is Tobias Macey and today I'm interviewing Yetunde Dada about Otto, Astronomer's expert Airflow agent

Interview

Introduction

How did you get involved in the area of data management?

Can you describe what Otto is and the story behind it?

What are the core problems that you are trying to solve with Otto and for whom?

What was your process for identifying the scope of activities that Otto should be incorporated into?

Orchestration engines are a rich source of information. What are the aspects of Airflow that lend themselves to extending with this agentic context?

What are the other supporting systems that are necessary to enable Otto to work effectively, especially in mixed orchestration environments? (e.g. metadata platforms)

One of the explicit capabilities that you invested in is code review for Airflow…
Why Multi-Agent Systems Need Shared State, Graph Semantics, and Governance

3 augustus 2026 · 1 uur 2 min
Afspelen

Summary
In this episode Ragnor Comerford talks about OmniGraph, a lakehouse-native graph storage layer designed around the needs of agentic systems. He explores how graphs are primarily a semantic model for representing the world, rather than just a specialized engine for traversal workloads, and how that perspective shaped OmniGraph’s design on top of object storage, Lance, Arrow, and DataFusion. Ragnor explained the motivation for combining graph semantics with Git-style branching and merging so that teams can manage probabilistic writers such as AI agents with stronger governance, shared context, and safer collaboration patterns. He also dug into the practical tradeoffs of building a graph engine for multi-agent coordination instead of traditional graph analytics use cases. He closed with a look at emerging use cases such as company “brain” systems, software development lifecycle graphs, research workflows, and event-driven agent orchestration, along with a broader conversation about composability, and sovereign AI infrastructure. Lees meer

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management

Your host is Tobias Macey and today I'm interviewing Ragnor Comerford about OmniGraph, a lakehouse-native graph storage layer with git semantics

Interview

Introduction

How did you get involved in the area of data management?

Can you describe what OmniGraph is and the story behind it?

What was the original problem that you were trying to solve by creating it?

There are numerous graph engines available, what are the properties of OmniGraph that differentiate it from the competition?

Cypher (GQL) and Gremlin are all established languages with years of examples to work from. What are the benefits of developing a new and more constrained query interface for an agentic audience?

How does that change the potential applications of OmniGraph? (e.g. general knowledge graph, fraud detection, SIEM, etc.)

Can you describe the architect…
Building the Context Flywheel for AI Data Agents

6 juli 2026 · 1 uur 1 min
Afspelen

Summary
In this episode Prukalpa Sankar, co-founder of Atlan, talks about what it takes to build a “context flywheel” for AI agents in data-intensive organizations. She explained why model intelligence alone isn’t enough to make AI useful in production, and how real performance depends on contextual intelligence: institutional knowledge, semantic meaning, procedural know-how, and access to the right tools. She also dug into how metadata catalogs are evolving into broader context layers that serve both humans and agents, and why agentic systems are changing the economics of metadata and governance work. Prakulpa shared Atlan’s perspective on bootstrapping context from existing systems such as warehouses, BI tools, query logs, and SaaS applications, then using simulation, traces, and human governance loops to improve agent accuracy over time. Lees meer

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management

Your host is Tobias Macey and today I'm interviewing Prukalpa Sankar about strategies for building a context flywheel for your data agents

Interview

Introduction

How did you get involved in the area of data management?

You have spent several years working in the metadata catalog space with Atlan. What are the notable changes in scope, adoption, and application that you have seen since we last spoke (June 2022)?

The recurring theme since the start of 2026 has been agentic augmentation of all engineering workflows, including data. How do you differentiate between data catalogs, semantic layers, agent memory, context layers, etc. when architecting an AI-powered data-oriented system?

One of the perennial problems with data catalogs, business glossaries, master data management, etc. is the up-front investment required to get a real-world impact. How can agents help reduce the activation energy needed to get to that return on effort?

One of the perennial problems in data engineering is fragmentation and siloing of data. Thi…
Holding Kafka Right: Product-Friendly Streaming with TypeStream

19 juni 2026 · 50 min
Afspelen

Summary
In this episode Jevin Maltais talks about the practical realities of building reliable, product-focused streaming systems with Kafka. Jevin shares lessons from roles at Zapier, Humi, and Clio, where real-time synchronization, customer data unification, and document sync at scale highlighted both the strengths and common misuses of Kafka. He digs into using events as the source of truth, materialized views with KTables, and how schema registries and type safety prevent downstream breakage. Jevin explains why teams often reach for heavyweight Kafka clusters without leveraging Streams, Connect, or interactive queries—and how his project, TypeStream, aims to make those capabilities accessible via config-as-code while keeping a thin abstraction and clear escape hatches. He also explore trade-offs across Kafka-compatible alternatives, CDC with Debezium in the real world, and where abstractions should stop so teams can scale responsibility as complexity grows. Lees meer

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management

This episode is sponsored by DataDriven.io, the free data engineering interview prep platform built by data engineers for data engineers. Ever walked into a data engineering interview and gotten a question that has nothing to do with real data engineering work? Interviewing is its own skill, separate from the job. Watch your code execute live, inspect Spark internals, and whiteboard your data models and pipelines and defend your decisions. Unlike SQL-only or Python-only practice, DataDriven.io covers the full interview loop: star schemas, slowly changing dimensions, grain and fact table design, idempotency, watermarks, dead letter queues, change data capture, and backpressure. Every question comes from real Data Engineer interview loops at Google, Amazon, Meta, Stripe, Databricks, Netflix, and Airbnb. Go to dataengineeringpodcast.com/datadriven today to start practicing.

Your host is Tobias Macey and to…
Text to Data Products: Kaarvi’s End-to-End AI for Ingestion, Quality, and Dashboards

8 juni 2026 · 53 min
Afspelen

Summary
In this episode Shravan Gunda, founder and CEO of Kaarvi AI, talks about building an AI-native, agent-driven data platform designed to eliminate the janitorial work that consumes most data teams. He explores Kaarvi’s multi-agent architecture that runs queries across seven LLMs in parallel for reliability, its synthetic data generator that mirrors source schemas for quick testing, and “Hey Kaarvi” chat for text-to-SQL, text-to-transformations, and text-to-dashboard workflows. He also digs into on-prem versus SaaS deployments, domain-specialized agents for privacy and accuracy, code blocks for custom Python/SQL, and the roadmap for a marketplace and desktop assistant. Shravan highlights how Kaarvi compresses weeks of work into hours and bridges the gap between business users and data engineers by turning AI into a dependable force multiplier. Lees meer

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management

This episode is sponsored by DataDriven.io, the free data engineering interview prep platform built by data engineers for data engineers. Ever walked into a data engineering interview and gotten a question that has nothing to do with real data engineering work? Interviewing is its own skill, separate from the job. Watch your code execute live, inspect Spark internals, and whiteboard your data models and pipelines and defend your decisions. Unlike SQL-only or Python-only practice, DataDriven.io covers the full interview loop: star schemas, slowly changing dimensions, grain and fact table design, idempotency, watermarks, dead letter queues, change data capture, and backpressure. Every question comes from real Data Engineer interview loops at Google, Amazon, Meta, Stripe, Databricks, Netflix, and Airbnb. Go to dataengineeringpodcast.com/datadriven today to start practicing.

Your host is Tobias Macey and today I'm interviewing Shravan Gunda about building an agent-driven data platform at Kaarvi

Interview

Introduction
…