Big Data

A mobile request can fail without the server-side work failing. An iOS app may time out, lose the response after a POST has reached the service, or retry after connectivity changes while the original execution is still progressing. Apple explicitly distinguishes safe retry behavior by HTTP method and notes that URLSession can retry requests in some connection-loss cases, waitsForConnectivity can also cause the system to continue a request when connectivity returns. The dangerous state is therefore not “request failed,” but “completion is unknown.” If that request starts an agent that charges an account, reserves inventory, sends a message, or invokes an MCP tool, a second submission can become a second side effect. The Retry Boundary Is the Real Transaction Boundary “Exactly once” is too strong for a workflow crossing an iPhone, HTTP, an agent runtime, an MCP server, Kafka, a database, and an external API. Kafka can provide exactly-once guarantees within defined Kafka processing boundaries, but those guarantees do not atomically include arbitrary remote tool effects. The practical target is effectively-once behavior, and retries are expected, but every effect is guarded by a stable operation identity and converges on one committed outcome. Kafka’s idempotent producer suppresses duplicate records caused by producer retries, while transactional producers can atomically publish across Kafka partitions; the producer documentation also limits idempotence guarantees to a producer session and requires read_committed consumers for end-to-end transactional visibility. The operation identity must exist before the first network attempt. An iOS client can create an operationId when an action becomes durable local intent, persist it, and reuse it across transport retries. Transport material such as a server challenge may change, but the business ID must not. The server treats (subjectId, operationId) as a uniqueness boundary and stores a canonical payload hash with it. PostgreSQL unique constraints enforce row uniqueness, while INSERT ... ON CONFLICT provides an atomic conflict path under concurrency. SQL INSERT INTO agent_operation(subject_id, operation_id, payload_hash, status) VALUES (:subject, :operationId, :payloadHash, 'ACCEPTED') ON CONFLICT (subject_id, operation_id) DO NOTHING; A conflict with the same payload hash returns the existing operation; a different hash rejects key reuse. The record should exist before agent execution starts, and the accepted response should expose the durable operation identity. Let LangGraph Resume Without Repeating Effects LangGraph persistence is useful precisely because durable execution can replay code. With a checkpointer, LangGraph saves state at super-step boundaries; if execution resumes after a failure, an affected node can run again from the beginning. Official guidance consequently requires idempotent node logic, and task results can be checkpointed so completed task work can be reused during resume instead of recomputed. Replaying from an earlier checkpoint can also re-trigger later LLM calls and API requests. A stable business operation should therefore map to a stable LangGraph thread, while every effectful tool boundary receives the same operation ID. Python config = {"configurable": {"thread_id": operation_id} result = graph.invoke( {"operation_id": operation_id, "command": command}, config ) Checkpointing reduces recomputation but does not replace downstream idempotency. A reservation can succeed before its task result is durably checkpointed. LangGraph’s functional API therefore recommends idempotent tasks because an incomplete task can execute again during resume. Python @task def reserve_inventory(operation_id, sku, quantity): return mcp.call_tool("reserve_inventory", { "operationId": operation_id, "sku": sku, "quantity": quantity }) The significant property in this snippet is not the decorator. The important part is that the business identity crosses the graph boundary and reaches the tool implementation. A downstream inventory service can then use that identity to return a previously committed reservation rather than creating another one. MCP Tasks Are Durable Handles, Not Deduplication Keys The current MCP Tasks design is especially relevant to long-running agent tools. In the July 28, 2026 protocol revision, Tasks moved into the io.modelcontextprotocol/tasks extension. A server can return a durable task handle, and the client can poll with tasks/get, provide input with tasks/update, or request cancellation with tasks/cancel. The task is durably created before its handle is returned, which allows polling after a disconnect. That durability solves result retrieval after task creation, but it does not by itself deduplicate the request that creates the task. The task ID is server-generated. If the server creates task A, the response disappears, and the original tools/call is sent again, a naïve implementation can create task B. Therefore, the business operationId must be part of the tool arguments or equivalent application metadata, and task creation must first look up an existing operation. This follows directly from MCP’s server-generated task-ID model combined with retry ambiguity at the HTTP boundary. The MCP server can return an existing task handle for the same authenticated subject, operation ID, and payload hash, and later return the stored terminal result. Cancellation should also be idempotent because MCP defines it as cooperative rather than a guarantee that underlying work stops immediately. Keep Kafka Guarantees Inside Kafka Kafka is most valuable after the operation has been claimed. A database transaction can persist operation state with an outbox row carrying the same ID. Kafka producer idempotence protects against duplicates caused by producer retries, while consumers can still use the operation ID for application-level deduplication. Kafka transactions can atomically cover Kafka writes, but they do not extend over an MCP server or payment API. The event contract should preserve causality rather than inventing a new identity at each hop. JSON { "operationId": "8E7B6D9E-...", "type": "AgentToolCompleted", "tool": "reserve_inventory", "status": "SUCCEEDED" } A consumer can enforce uniqueness on (consumerName, operationId, eventType) or make the state transition conditional. Kafka delivery guarantees and application idempotency then reinforce each other instead of being treated as interchangeable. Bind Retry Identity to App Attest Without Blocking Legitimate Retries App Attest addresses a different failure mode: whether a request comes from a legitimate app instance and whether signed request material has been replayed or altered. Apple’s current guidance uses a server-provided challenge for assertions and requires the server to validate a strictly increasing assertion counter; that counter is specifically an anti-replay signal. Assertions are generated locally on the device after key attestation. The App Attest assertion must not become the business idempotency token. A legitimate retry should obtain fresh challenge material and generate a fresh assertion while retaining the original operation ID. The data hashed for the assertion can bind the server challenge, operation ID, and canonical payload hash together. Swift let payloadHash = SHA256.hash(data: body) let clientData = challenge + operationID.data + Data(payloadHash) let clientDataHash = Data(SHA256.hash(data: clientData)) let assertion = try await service.generateAssertion( keyID, clientDataHash: clientDataHash ) Apple recommends server-controlled challenges, server-side validation, and assertion-counter tracking as assertions are generated on demand without a round trip to Apple’s servers. The server verifies App Attest, checks that the challenge binds the operation ID and payload, then performs the idempotency lookup. A fresh assertion can retry the same operation; a replayed assertion fails anti-replay validation; an altered payload fails the hash check. Effectively-Once Behavior Is a Composition Property Reliable agent execution does not come from asking iOS to retry less often or from labeling a Kafka pipeline “exactly once.” It comes from carrying one durable business identity across every retry and every boundary, claiming that identity atomically before execution, making LangGraph effects idempotent under resume, using MCP Tasks as durable result handles rather than creation-time deduplication keys, restricting Kafka’s exactly-once guarantees to Kafka’s transactional domain, and using App Attest to prove request integrity without confusing anti-replay state with business deduplication. When those boundaries align, a lost mobile response can cause another HTTP attempt, another graph invocation, or another poll, but it does not cause another business effect. That is the operational meaning of effectively once.

By Uthej Mopathi CORE

Every major AI vendor now supports the Model Context Protocol. The framing is almost always the same: MCP is the universal connector for AI agents in the enterprise. That framing sets up a false choice. MCP, REST/HTTP APIs, and Apache Kafka are not alternatives. They solve different problems at different layers of the architecture. Treating them as competing options produces systems that are fragile exactly where they need to be reliable. These three technologies can and do coexist in the same architecture. The question is not which one to pick. It is which one belongs where, and what the tradeoffs are when more than one could technically do the job. This article maps that decision: what each technology is built for, where the boundaries are, and where the genuine gray areas lie. 1. What Is MCP and What Is It Built For? Anthropic introduced the Model Context Protocol in November 2024 as an open standard for connecting AI assistants to external tools and data sources. Before MCP, every AI model required a custom connector to each external system. Three models, ten systems: thirty custom integrations to build and maintain. MCP collapses that to one standard interface. Any compliant client talks to any compliant server without prior coordination. OpenAI adopted MCP in March 2025. Google DeepMind confirmed support in April 2025. By December 2025, MCP had reached over 97 million monthly SDK downloads across Python, TypeScript, Java, Kotlin, C#, and Swift. Anthropic donated the protocol to the Agentic AI Foundation under the Linux Foundation, with AWS, Google, Microsoft, Bloomberg, and OpenAI as platinum members. MCP is no longer a developer experiment. Signals of enterprise maturity are arriving quickly: AI agents paying for API access autonomously, cross-SDK interoperability between Anthropic and OpenAI converging on MCP Resources, composable enterprise workflows where agents read tool signatures and compose cross-system flows without predefined paths, and an official MCP Registry launched in late 2025 as the community-driven server directory. The 2026 roadmap focuses on scalable transport, agent-to-agent communication, governance maturation, and enterprise readiness covering audit trails and SSO-integrated authentication. MCP handles tool access: how an agent calls an external capability. It does not handle agent-to-agent coordination, which is the domain of protocols like Google's Agent-to-Agent (A2A). MCP and A2A are complementary and address different layers of agentic architecture. The moment MCP is asked to do more than tool access, the architecture starts to break. Security Maturity Is Still Catching Up With Adoption Most incidents disclosed in 2025 and early 2026 are implementation failures, not protocol flaws. An Endor Labs analysis of 2,614 MCP implementations found 82% use file system operations prone to path traversal and 67% use APIs related to code injection. Enterprise-grade authentication with OAuth 2.1 and SAML/OIDC is on the 2026 roadmap but still in progress. The practical controls for today: apply least privilege, limit MCP server access to only the systems and data each tool requires, and monitor tool definitions for unexpected changes. 2. MCP vs. REST/HTTP API MCP and REST/HTTP APIs serve different consumers and should not be treated as interchangeable. REST is an architectural style built on HTTP, widely adopted but with no fixed conventions for discovery, error formats, or method naming. Well-designed REST APIs backed by OpenAPI specifications work well for direct, programmatic data access when a native SDK or versioned API already exists and teams know how to operate it. MCP enforces consistency at the interface level because the consumer is an AI model that cannot tolerate creative API interpretation. MCP standardizes how a tool is called. It does not standardize what the tool returns, how fresh that data is, or whether two agents calling the same tool simultaneously see the same state. For direct data access to vector stores, databases, or business application APIs, a well-governed REST API, native SDK, or Kafka Connect integration is almost always the better choice: lower latency, no protocol overhead, mature tooling. For giving AI agents standardized, discoverable access to a broader set of tools across vendors and frameworks, MCP is the right layer. The two are complementary, not competing. Tool Design Matters as Much as the Protocol Choice One important nuance on tool design: mapping one-to-one from existing APIs to MCP tools rarely works well. What matters is tool granularity, smart metadata, and thoughtful assembly of the MCP layer. An MCP server that exposes well-structured, semantically rich tools lets an AI agent reason about capabilities and compose workflows. This is reminiscent of the composability questions from the enterprise SOA (Service-oriented Architecture) era. SOA promised flexible service composition but delivered integration chaos when governance, metadata quality, and service granularity were treated as afterthoughts. MCP faces the same risk. The protocol is sound; what determines success is the discipline applied to how tools are defined, documented, and assembled. What MCP Does Not Do What MCP does not do matters as much as what it does. It does not manage data, guarantee message delivery, enforce governance, or guarantee consistency across systems. It is an interface layer, not a data pipeline. That boundary becomes even clearer when looking at what Kafka does, which is structurally different from both MCP and REST. 3. Apache Kafka: Event Broker, Decoupling, and the Backbone Role Operational data is the live data that runs business processes: order states, inventory levels, transaction records, customer accounts, risk scores. It originates in systems like SAP, Salesforce, Oracle, and mainframes, and it changes continuously. Kafka is architecturally different from both HTTP and MCP in one way that matters most: it decouples producers and consumers through a persistent, ordered, append-only log. With HTTP or MCP, the caller and the callee are coupled at request time. Every integration is point-to-point. If the target system is slow or unavailable, the caller is directly affected. Kafka breaks that coupling entirely. A producer writes an event once. Any number of consumers read it independently, at their own pace, using their own communication paradigm. One consumer processes records in real time. Another runs nightly batch analytics over the same events. A third powers a stream processing pipeline. A fourth writes results to a data lake via Apache Iceberg. All of them consume the same underlying data product. None of them affects the others. Kafka supports three consumption patterns from a single event stream: streaming, request-response, and batch. The event exists once; each consumer is independent. This is the pub/sub event broker model, and it is what makes Kafka the integration backbone between operational and analytical systems. The diagram below shows this decoupling: a single Kafka topic serving real-time applications, HTTP-based consumers, batch analytics, and MCP agent interfaces simultaneously. Stream Processing With Kafka Streams and Apache Flink Stream processing is a core complement to Apache Kafka, extending the platform from event transport into real-time data processing and decisioning. Kafka Streams is a lightweight Java library embedded in applications. It is well-suited for streaming ETL and simple to medium stateful stream processing without requiring a separate cluster. It integrates closely with existing JVM-based services. Apache Flink is a distributed stream processing engine designed for more complex workloads. It supports Java, Python, and SQL APIs, making it accessible to both application developers and data engineers. Flink runs as a dedicated cluster or in managed environments and is built for high-scale scenarios such as multi-stream joins, event-time processing, large state management, exactly-once semantics, Complex Event Processing (CEP), real-time analytics, and AI model inference. Both approaches extend Kafka with processing capabilities. The choice depends on workload complexity, required deployment model, and preferred programming language, not on replacing Kafka’s role as the event streaming backbone. A detailed comparison is available in the post Apache Kafka and Apache Flink: A Match Made in Heaven. Operational and Analytical Integration, Including the Data Lakehouse Kafka is not only for operational data integration. It serves as the ingestion layer into data lakes, feeds real-time analytical pipelines, enables stream processing with embedded AI models, and connects business applications bidirectionally. A governed data streaming platform provides schema registry, lineage tracking, role-based access control, and exactly-once delivery semantics across all of that. It serves both operational and analytical use cases and acts as the bridge between those two worlds. For how streaming and the lakehouse converge via Apache Iceberg, see Data Streaming Meets Lakehouse. Kafka's append-only commit log is the foundation of data consistency across the enterprise. Every downstream consumer sees the same data in the same order. That is not just a performance feature. It is what prevents the architecture where every system has its own version of the truth. 4. The Tradeoffs: It Is Not Black and White The choice between MCP, REST/HTTP APIs, and Kafka is rarely clean. All three can play a role in the same architecture. REST/HTTP APIs work well for operational data access when volume is moderate and a well-governed API already exists. A REST API backed by a Kafka-derived serving layer can return consistent, current data. The API is the interface; the streaming platform is what makes the data trustworthy behind it. A financial services firm exposing account balances via REST is not doing it wrong, as long as those balances are derived from a governed, consistent data source rather than pulled directly from a source system on every request. Kafka becomes the clear choice when data is high-volume or high-velocity, when multiple consumers need the same events, when ordering and exactly-once delivery matter, or when the same events need to feed operational applications, analytical pipelines, and AI agents simultaneously. MCP fits best when access is supplementary, loosely coupled, and low-frequency. A support agent looking up a ServiceNow ticket before drafting a response, or a sales assistant pulling the latest slide deck from Google Drive before a call, are good fits. The key test is simple: does it matter if the data the agent receives is a few seconds or minutes old? If yes, MCP should not own that responsibility. If no, MCP is the right interface. SAP: Clean Separation Between ERP Integration and Developer Tooling The boundary between MCP and REST is not a choice between two equivalent options for the same integration. SAP is the clearest example of a clean separation. SAP exposes extensive REST and OData APIs for ERP integration: order management, finance, supply chain, procurement, and HR data flowing bidirectionally between SAP and other enterprise systems. SAP's MCP servers serve an entirely different purpose: developer tooling for ABAP code generation, CAP application development, UI5 and Fiori assistance, and operational tasks like transport validation and incident management. An architect connecting SAP order events to downstream systems uses OData and Kafka Connect. A developer asking an AI coding assistant to generate ABAP code uses the SAP MCP server. Different consumers, different use cases, different data. No overlap. Salesforce and ServiceNow: Same Data, Different Consumer Salesforce and ServiceNow follow a different pattern. Their MCP servers wrap the same underlying REST APIs and expose the same underlying data, but for a different consumer. A developer-written integration calls the Salesforce REST API directly with known endpoints and hardcoded logic. An AI agent calls the Salesforce MCP server, which wraps that same API to make it discoverable and stateful for an agent that cannot read documentation or manage its own session state. The data is identical. The access path differs based on who is consuming it. This is not a free choice between equivalent options. It is the same system serving two different client types through two different interface layers. REST vs. Kafka for Operational Data: The Harder Call The harder boundary is between REST and Kafka for operational data. Both can technically serve it, and that is where the real architectural decision lies. REST is simpler to start with but introduces point-to-point coupling, integration spaghetti at scale, and consistency risks when the same data needs to reach multiple consumers. Kafka is more complex to operate but provides the decoupling, consistency, and governance that enterprise architectures require when the same data needs to reach many consumers reliably. The two are not mutually exclusive. A common and well-proven pattern combines both: Kafka handles the event backbone, decoupling, and consistency, while a REST layer sits on top for synchronous request-response access, API management integration, or compatibility with systems that cannot speak the native Kafka protocol. This is particularly common in mobile applications, legacy system integration, and API gateway architectures. For a detailed look at how REST and Kafka complement each other in practice, see Request-Response with REST/HTTP vs. Data Streaming with Apache Kafka. 5. Decision Framework: MCP, REST/HTTP, or Kafka? Choosing between MCP, REST/HTTP, and Kafka is not a single decision but a set of tradeoffs that depend on data volume, consumer type, consistency requirements, and what is already in production. The comparison table below makes those tradeoffs concrete across eight dimensions. When to Use Which: A Guide to the Decision Tree The decision tree below walks through the same logic as a series of questions, routing to the right choice based on the integration's actual requirements. Use MCP when the integration is supplementary and tool-like: Slack, Google Drive, ServiceNow tickets, internal knowledge bases. The agent needs context to act, not a stream of events to react to. Eventual consistency is acceptable. Apply least privilege, monitor tool definitions for changes, and isolate MCP servers from production systems. Use a REST/HTTP API or native SDK when a well-documented API or SDK already exists and the engineering team knows how to operate it. The access pattern is direct, moderate-volume, and latency-sensitive. REST is also a reasonable choice for operational data when the backend is a governed Kafka-derived serving layer and consistency properties are inherited, not assumed. Use Apache Kafka when data is high-volume or high-velocity, when multiple consumers need the same events, when ordering and exactly-once delivery matter, or when governance, lineage, and auditability are non-negotiable. Kafka is also the right choice when the same data needs to feed operational applications, real-time analytics, data lakes, and AI agents simultaneously. Use the real-time context engine when an AI agent needs current, consistent operational context for autonomous decisions. Kafka and Flink govern the data. MCP provides the agent interface. The consistency guarantee comes from the streaming layer, not from MCP. The practical question is not which protocol to choose. It is whether the data architecture underneath the agents can be trusted. Agents making autonomous decisions about inventory, risk, or customer service are only as reliable as the data they act on. 6. Where MCP and Kafka Work Together: The Real-Time Context Engine There is one pattern where MCP and data streaming complement each other directly: the real-time context engine. Kafka and Flink process and govern the data: ingesting from operational systems, applying transformations and filters, producing real-time materialized views. Those views are then exposed to AI agents through a standardized MCP interface. The streaming platform owns the data, its freshness, and its consistency guarantees. MCP owns the interface to the agent. Neither layer bleeds into the other's responsibility. Data consistency is not delegated to MCP. The streaming platform enforces it upstream before the MCP interface comes into play. The agent calls a tool and receives context that is current, governed, and consistent, not because MCP guarantees it, but because the streaming platform does. Any compliant AI agent, whether Claude, ChatGPT, Amazon Bedrock, LlamaIndex, or CrewAI, can call the context engine and receive current context from operational systems without needing to understand Kafka topics, Flink jobs, or schema evolution. An agent routing shipments from yesterday's inventory, approving transactions against a risk score from three hours ago, or reading an account balance that has not propagated: none of these is reliable. A real-time context engine eliminates this class of error at the source, reduces hallucinations, lowers inference cost, and anchors decisions to current operational reality. From Data Freshness to Agent Governance Enterprise readiness for this pattern also depends on how agents are governed once deployed. Trust, control, and accountability become central once agents start chaining decisions across domains. The context engine is the data layer of that answer. Governance of the agents themselves, covering what they are permitted to do, under what conditions, and with what audit trail, is the other half. This is the dimension enterprise buyers are actively evaluating when selecting agent orchestration platforms. The diagram below shows how the three layers fit together: the streaming platform as the data backbone, the context engine as the governed serving layer, and MCP as the clean interface to agents. 7. Conclusion: One Protocol, One Job MCP has earned its place in the enterprise architecture stack. What it has not yet earned is the role of universal integration layer, and understanding that distinction is what this article has been about. The broader architecture this sits inside connects three interdependent pillars. Event-driven data integration, with Kafka as the backbone, moves data reliably between operational and analytical systems and delivers governed data products to every consumer. Process intelligence is the orchestration layer that determines which decisions to automate, in what sequence, and under what conditions, giving agentic workflows the structure and governance they need to be trustworthy. Trusted agentic AI is where MCP plays its role: the standardized, governed interface through which agents access external tools and context, anchored to real data by the streaming layer beneath it. For a vendor-by-vendor analysis of trust and lock-in across the major AI platforms, see the Enterprise Agentic AI Landscape 2026. For a deeper look at how the three pillars fit together as an enterprise architecture framework, see The Trinity of Modern Data Architecture: Process Intelligence, Event-Driven Integration, and Trusted Agentic AI. One protocol, one job. That is the right way to use MCP.

By Kai Wähner CORE

Enterprise AI is moving beyond isolated prompt-response calls and toward systems that observe events, preserve state, invoke tools, and publish decisions back into operational workflows. In that setting, event streaming is not simply middleware. It becomes the record of how intelligent behavior unfolds over time. Kafka is designed to read, write, store, and process streams of events across distributed systems, while Kafka Streams adds joins, aggregations, windowing, event-time processing, and exactly once support for stateful stream applications. At the same time, modern agent runtimes have shifted toward durable execution, persistence, and human-governed control flows rather than single-turn prompting alone. That convergence makes Kafka a strong coordination layer for autonomous agents that need to react continuously instead of responding once and disappearing. That architectural change also alters the role of the model. In an API-centric design, the model is often treated as a synchronous dependency behind a request. In an event-driven design, the model becomes one participant in a larger decision pipeline. Observations arrive as events, context is assembled from topics and state stores, agent steps are logged, and decisions are emitted as new events for downstream systems. Because Kafka topics can be replayed and reprocessed, the same stream can feed planners, validators, enrichment services, audit consumers, and human-review workflows without creating hard coupling between those components. The resulting system is easier to inspect, easier to recover, and easier to evolve than a chain of tightly bound remote calls. Turning Kafka Into the Coordination Layer The most important benefit is not only scale. It is the replacement of brittle request chains with an append-only coordination layer. A payment event, support ticket update, equipment alarm, or fraud signal can be published once and then consumed independently by retrieval components, compliance checks, planners, and execution agents. Kafka consumer groups divide partitions across consumers in the same group, and each partition is consumed by a single consumer within that group, which preserves ordering at the partition level while still allowing horizontal scale. For agentic systems, that detail is central. If all events for the same case, customer, or device are keyed consistently, one partition becomes the serialized timeline for that entity, and the agent no longer has to reconstruct order from racing HTTP callbacks. The event log also becomes a durable memory boundary. Kafka log compaction retains the latest value for each key, which makes compacted topics useful for task state, policy snapshots, approval status, or tool metadata that must survive restarts and recover quickly. On the runtime side, agent frameworks persist checkpoints and thread-scoped state so interrupted flows can resume from a saved step instead of starting over. Used together, those layers create a pragmatic split of responsibilities, such as Kafka preserves externally visible state transitions, and the agent runtime preserves internal execution context between steps, pauses, and failures. That is exactly the kind of separation needed when autonomous behavior must remain observable without being reduced to stateless prompt calls. Designing Agent Loops Around Events Once Kafka becomes the backbone, the agent loop changes shape. The entry point is no longer a prompt alone. It becomes a domain event that is enriched, correlated, and converted into a bounded task. Research on ReAct showed the value of interleaving reasoning and acting, and current agent frameworks translate that idea into practical workflows with durable execution, interrupts, and resumable state. The production version of an autonomous agent is therefore less like a chat session and more like a state machine that reasons, uses tools, emits intermediate facts, and pauses when a policy boundary requires approval. A concise stream processor can prepare that task before the model loop begins: Java builder.stream("order-events", Consumed.with(Serdes.String(), orderSerde)) .selectKey((key, event) -> event.customerId()) .join(customerTable, this::mergeContext) .mapValues(this::toAgentTask) .to("agent-tasks"); This pattern keeps context assembly close to the log instead of scattering it across synchronous service calls. Records are keyed by stable business identity, joined with the latest customer state, and emitted as small agent-tasks messages that the runtime can consume directly. Kafka Streams is explicitly intended for stateful processing with joins, event-time semantics, and exactly-once guarantees, so the enrichment stage remains deterministic, replayable, and independent from the model-serving layer. The execution boundary can remain equally narrow: Java @KafkaListener(topics = "agent-tasks", groupId = "claims-agent") @Transactional public void handle(AgentTask task) { AgentDecision decision = agentRuntime.run(task); kafkaTemplate.send("agent-decisions", task.taskId(), decision); } A compact runtime method can express the control flow without hiding it: Java public AgentDecision run(AgentTask task) { AgentState state = stateStore.load(task.taskId()); PlanStep step = planner.next(state, task); if (step.requiresApproval()) return AgentDecision.pause(task.taskId(), "manual-review"); ToolResult result = toolExecutor.execute(step.tool(), step.arguments()); return planner.complete(task, state, result); } This arrangement matters because the runtime receives a prepared task and emits an explicit decision event instead of mutating external systems invisibly. When transactions are enabled, Spring for Apache Kafka supports exactly-once semantics for the read-process-write sequence, and Kafka itself uses idempotent producers plus transactions so retries do not create duplicate log entries. External side effects still need idempotent design when they happen outside Kafka, but the event pipeline itself becomes much more predictable and auditable. Reliability and Control in Production Reliability in event-driven AI systems is usually lost at the edges rather than inside the model call. Kafka’s exactly-once features matter because an autonomous agent often emits decisions that trigger downstream actions, compensations, or audits. Kafka Streams supports exactly-once v2, and exactly-once flows configure consumers with read_committed isolation so aborted transactions do not leak into downstream processing. The event contract matters just as much as the delivery contract. Schema Registry centralizes schemas, validates them, and enforces compatibility modes so producers and consumers can evolve independently. In practice, a stable AgentDecision schema with explicit action type, confidence, explanation reference, and approval status is usually more valuable than a loosely structured JSON envelope because it can be consumed safely by analytics jobs, rule engines, operational systems, and auditors maintained by different teams. Operational control also has to assume malformed input, tool failure, and policy limits. Kafka Connect supports dead letter queues for records that cannot be processed successfully, and Spring Kafka supports dead-letter handling for repeated listener failures. Kafka also supports SASL-based authentication and ACL-driven authorization, which matters when planners, tool executors, and audit services must have different permissions over topics and consumer groups. Combined with interrupt-driven approval workflows from modern agent runtimes, those controls allow autonomous agents to operate inside explicit safety and governance boundaries instead of as opaque background processes. Where This Architecture Fits Best This architecture is strongest when work is asynchronous, stateful, and externally observable. Fraud triage, claims handling, supply chain exception management, field-service coordination, and security operations are better fits than chat-only assistance because the hard problem is not generating a sentence. The hard problem is reacting to a changing stream of facts, correlating them by entity and time, and making bounded decisions with replayable outcomes. Event-driven AI systems with Kafka and autonomous agents are compelling because they treat intelligence as part of an operational stream rather than as an isolated endpoint. The most effective implementations keep the log authoritative, keep schemas explicit, keep agent state durable, and keep irreversible actions observable and governable. That combination produces systems that are not only responsive, but also replayable, auditable, and resilient enough for enterprise use, which is ultimately the threshold that separates a convincing demo from a production architecture.

By Uthej Mopathi CORE

The modern enterprise generates and consumes unprecedented volumes of data across operational systems, customer interactions, partner ecosystems, cloud applications, IoT devices, and AI platforms. At the same time, AI systems are becoming major consumers of enterprise data, making decisions, generating content, recommending actions, and automating workflows. Poor data quality is no longer just a reporting issue; it is also an AI issue. Inaccurate, incomplete, or poorly governed data can produce biased outcomes, regulatory violations, AI hallucinations, and flawed business decisions. Traditional data governance programs were primarily designed to support business intelligence and regulatory compliance. However, the AI era introduces new requirements around model governance, explainability, lineage, ethical AI, data observability, and autonomous decision-making. Organizations must therefore evolve toward a unified data and AI governance model that ensures data can be trusted not only by humans but also by machines. Poor data governance can result in hallucinating AI systems, biased model outcomes, regulatory violations, security breaches, increased operational costs, customer trust erosion, and incorrect business decisions. Data governance has therefore evolved from a compliance function into a strategic business capability. Enterprises that establish trusted, governed, and accessible data foundations will be better positioned to scale AI initiatives, accelerate innovation, and create sustainable competitive advantages. The basic objectives of data governance are: Enhance the agility of data-informed business decisionsFacilitate seamless knowledge sharing across the enterpriseEliminate ambiguity and foster trust in data assetsIncrease data trust, better decision-making, and faster innovation cyclesImprove compliance posture, reduce data duplication, and increase business agility To fully comprehend these objectives, it is essential to first recognize the critical role that data governance plays within an enterprise's broader data management strategy. This white paper explores the challenges, next-generation data capabilities, modern data architecture, and strategic considerations required to build an AI-ready data foundation. Industry Trends of Data Governance According to Gartner, “Any organization in any industry, especially those with very large amounts of data, can use AI for business value.” According to Statista, by 2027 the global market for big data will be worth $103 billion. According to Gartner, 60% of organizations will fail to realize the value of their AI initiatives due to weak data governance frameworks. By 2028, enterprises will increasingly adopt autonomous, AI‑driven governance systems capable of automated policy enforcement, continuous data quality scoring, and real‑time anomaly detection. Gartner forecasts that AI‑driven automation will reduce manual data stewardship tasks by 40% by 2027. Governance models will shift from centralized to federated and hybrid, ultimately evolving toward autonomous domain‑driven governance. Gartner reports that over 60% of enterprises will adopt federated governance by 2027. The rise of AI‑augmented data mesh as a dominant architecture by 2028 (Thoughtworks). AI Trust, Risk, and Security (AI TRiSM) will become the top governance investment area as organizations confront risks related to hallucinations, bias, and regulatory compliance. Gartner predicts that enterprises implementing AI TRiSM will reduce AI‑related risk incidents by 50% by 2026. With the rapid expansion of IoT, 5G, and edge AI, governance must operate in real time. IDC estimates that 30% of enterprise data will be processed at the edge by 2027. AI platforms will embed governance natively, enabling governed prompt engineering, model access, and data contracts. Gartner predicts that 75% of AI platforms will include built‑in governance controls by 2027. Databricks Mosaic AI, Snowflake Cortex, and Microsoft Azure AI’s Responsible AI Dashboard exemplify this trend. Synthetic data will become a regulated and essential component of AI training. Gartner projects that synthetic data will overshadow real data in AI training by 2030. McKinsey estimates that 50% of AI training datasets will include synthetic data by 2028. Global regulations will mandate transparency, lineage, and automated audits. Gartner states that regulatory pressure will be the top driver of data governance investments through 2030. Data contracts will replace traditional API documentation, enforcing schema, SLAs, lineage, and quality. Gartner predicts that data contracts will reduce integration failures by 40% by 2027. Challenges in Data Governance As enterprises expand into multi-cloud environments and increasingly adopt generative AI, governance challenges continue to multiply. The most common data governance challenges faced by enterprises today are, Data explosion: Data exists across multiple, diverse systems throughout the enterprise. Data is spread across structured data, semi-structured data, unstructured content, streaming data, IoT telemetry, computer vision assets, agent-generated content, and AI-generated outputs. Traditional governance frameworks often lack the scalability and automation required to manage such diversity.Data silos: Data is segmented across various platforms, channels, tools, and business units, making it challenging to access across the enterprise. Most data resides across ERP systems, CRM platforms, legacy applications, cloud-native platforms, data warehouses, data lakes, and SaaS applications. This leads to inefficiency, data duplication, and data inconsistency.Data accuracy, completeness, and timeliness: Ensuring data accuracy, completeness, and timeliness remains a challenge.Data quality: Poor oversight of the quality of data coming into an enterprise, as well as its usage throughout the organization, can lead to poor data quality. Common quality challenges include missing values, duplicate records, outdated information, inconsistent definitions, incomplete lineage, and data drift.Regulatory complexity: Managing regulatory compliance, data security, and data privacy presents significant challenges. Enterprises must comply with GDPR, HIPAA, CCPA, PCI-DSS, the EU AI Act, and industry-specific regulations.Data management: Poor data management strategies can result in an enormous amount of data in a completely unmanageable format.Data leakage: Sensitive business information or customer data may be exposed or leaked, leading to misuse. Unsecured data originating from different data sources can lead to data breaches.AI-specific risks: New AI-era governance concerns include algorithmic bias, explainability requirements, training data provenance, prompt governance, LLM hallucinations, and autonomous agent controls. Next Generation Data Capabilities Governance alone does not create value. Enterprises need enterprise data capabilities that make governance operational while enabling innovation and AI adoption. Modern data ecosystems require intelligent platforms capable of discovering, understanding, protecting, and serving data on a scale. Data processing techniques: Unstructured processing covers entity extraction, concept extraction, sentiment analysis, NLP, ontology, etc. To automate portions of the extraction process, Machine Learning techniques are leveraged. Data intelligent platform: It enables natural language queries, AI-powered recommendations, intelligent search, and context-aware discovery.Data products: Data products provide ownership, accountability, defined SLAs, reusability, and business value measurement.On-demand data services: Provide virtualized access to data across the enterprise through way of composable on-demand data services for both online and offline use. It should provide the ability to query in a federated fashion for both online and offline access.Intelligent metadata management: Digital throws data into enterprise systems at a rate that doesn’t allow SMEs to look at data structures and extract metadata. Automated metadata extraction based on ontology is critical. Modern metadata platforms provide automated discovery, classification, catalog generation, lineage tracking, and semantic enrichment.Data fabric: It provides unified data access, cross-platform integration, federated governance, and policy automation.Data mesh: It enables domain ownership, distributed accountability, product-centric thinking, and decentralized governance. Data observability: It focuses on data health monitoring, pipeline performance, anomaly detection, drift identification, and SLA compliance.Real-time analytics: Multi-channel applications and decision management systems are used to capture interactions for digital processes in real-time scenarios. Data archival: Compliance and performance requirements drive the need for archival of both structured and unstructured data. Principles of Data Governance Architecture principles provide a baseline for decision-making across the enterprise. To guide implementation, enterprise data governance principles are categorized into three strategic domains: Value and ownership, security, privacy and ethics, and architecture and quality. Value and Ownership Data as an asset: Data is an enterprise asset with specific, measurable value to the enterprise and must be managed accordingly.Data is shared: Users have access to the data necessary to perform their duties; therefore, data is shared across enterprise functions and business units. Data stewardship: Governance structure must define the owner and those accountable for data-related decisions that are cross-functional. Define the personnel accountable for leadership activities and assign responsibilities to individual contributors or groups of data handlers. Data trustee: Each data element has an assigned trustee accountable for its quality, lifecycle, and compliance. Security, Privacy & Ethics Principles Data security: Data is protected from unauthorized use and disclosure. Data privacy: Privacy and data protection are considered throughout the entire life cycle of the data. All data sharing will conform to relevant regulatory and business requirementsData integrity: Each party to data must be aware of, and abide by, their responsibilities regarding the provision of source data and the obligation to establish and maintain adequate controls over the use of personal or other sensitive data. Data transparency: Governance decisions, policies, and lineage must be transparently documented and clearly communicated across the enterprise. All data-related decisions must be explained clearly to all personnel how, when, and why they are introduced. Architecture & Quality Principles Common vocabulary and data definitions: Data definitions are consistent across the enterprise and understandable to all users.Fit for purpose: Next-generation information ecosystem needs to have fit-for-purpose tools, as no one technology will satisfy all the workloads and processing techniques - E.g., Text Processing, Data Discovery, Dynamic Data Services, High-Performance Analysis, Streaming Analytics, etc.Data metrics: Critical Data Elements (CDEs) of the Business are managed through a lifecycle-oriented data governance process to ensure data quality, with clear metrics and dashboards. As data will reside in many repositories, integrated metadata lineage and PII protection are important. Key Components of Data Governance In the modern era, data management covers both technical requirements and strategic assets for businesses. Efficient data management Strategies help enterprises make informed decisions, improve customer experiences, and drive innovation. Data governance covers the automation of policies, guidelines, principles, and standards for managing data assets. It ensures data quality, accuracy, and compliance with regulatory requirements, building trust in the data. Data governance must be aligned with EA Governance at the enterprise level to realize the business objectives. Some of the open-source data governance tools are Amundsen, DataHub, Apache Atlas, Magda, Open Metadata, Egeria, and TrueData. These tools offer features like Metadata Management, Data Cataloging, and Collaboration to manage data assets effectively. The major components of data governance are: Data qualityData stewardshipData policies and procedures Data security Metadata management Master data management Data storageData privacy and complianceData metrics The following figure depicts the key components of data governance: Figure 1: Key Components of Data Governance Data Quality It helps ensure the accuracy, completeness, and consistency of data. Data quality management involves identifying and correcting errors, standardizing formats, and maintaining a high level of data integrity. Some of the top open-source data quality tools are: Cucumber, Deequ, dbt Core, MobyDQ, Great Expectations, and Soda Core. These tools help automate data validation, data cleaning, and monitoring. Data Stewardship It is about assigning roles and responsibilities related to data management. Data stewards are designated individuals or teams entrusted with overseeing the appropriate use, integrity, and secure storage of enterprise data. They serve as a vital bridge between IT and business units, ensuring that data conforms to the enterprise’s established quality and consistency standards. Key responsibilities include defining and standardizing data elements, monitoring data quality, and collaborating with IT to resolve any technical challenges. Other key data roles are: Chief data officers (CDOs) lead the data strategy, ensuring data is treated as a valuable business asset. Their goal is to drive executive investment in data compliance, risk reduction, and value creation as data becomes a trusted driver of business outcomes.Data protection officers (DPOs) ensure organizational compliance with data privacy laws like GDPR and CCPA. They oversee the protection of personal data, such as that of customers or suppliers, processed during daily operations. DPOs must have direct access to senior leadership to fulfill regulatory requirements.Data architects design robust yet flexible data foundations that empower users to manage and enhance their own datasets. They ensure data is meaningful, business-driven, and aligned with organizational goals. Their priorities often reflect measurable business outcomes.Data engineers and developers design and maintain data pipelines, ensuring data quality and flow across complex systems. They aim to empower business users while managing access, security, and data product governance.Data scientists extract value from data pipelines to deliver actionable insights. They solve complex problems using statistics, mathematics, and computer science. Their expertise often includes data mining and predictive analytics.Business analysts identify trends, assess risks, and gauge business performance using BI tools like Tableau, Power BI, and Looker. They extract trusted insights from data pipelines and present them through clear, actionable dashboards. Data Policies and Procedures It establishes and enforces policies for how data is collected, stored, shared, and used. As enterprise central data management, Prescribes permitted and prohibited practices at every stage of the data lifecycleEnsures compliance with internal standards and external regulationsAssigns accountability for data stewardship and risk mitigationAligns day-to-day data handling with strategic business objectives Data Security Establishing proper security protocols helps in reducing the risk of data breaches and threats. It also safeguards sensitive information. Implementation of Encryption, access controls, authentication, and intrusion detection systems helps in protecting data across the lifecycle. Top open-source data security tools that are widely used include: Metasploit, OSSEC, OpenVAS, Snort, KeePass, ClamAV. These tools can be integrated into various security strategies to protect against a wide range of cyber threats. Metadata Management It helps in keeping track of data definitions, relationships, and structures. It’s essentially data about data. Metadata functions as the contextual glue that transforms isolated data points into coherent, actionable assets. It captures essential attributes covering: Creation timestampAuthorship and ownershipSource provenanceRelationships to other data elements Metadata strategy should: Adopt a centralized metadata catalog (e.g., Apache Atlas, Collibra)Automate metadata harvesting and lineage trackingIntegrate metadata-driven data quality checks into your pipelinesEstablish governance policies for metadata stewardship and versioningMonitor metadata KPIs like catalog adoption rate and lineage coverage to drive continuous improvement Leading open-source metadata management tools are Apache Atlas, Amundsen, Metacat Data Catalog, Open Metadata, and Marquez. Master Data Management Master data management is a process for ensuring the accuracy, consistency, and completeness of critical data elements, such as customer data and product data, etc. master data is standardized, matched, merged, enriched, and validated according to governance rules. Some of the open-source key players in the MDM area are Talend Open Studio for MDM, AtroCore, and Pimcore. Data Storage It helps determine where and how data will be stored within the enterprise data repository. It covers both structured and unstructured data sources, which include databases, data warehouses, and data lakes. The factors that determine data storage are Performance, scalability, and data retrieval requirements. Some of the key open-source players in the data storage area are Hadoop, LakeFS, Cassandra, and Neo4j. These tools provide scalability, robustness, and performance in managing large data and analyzing large datasets in various applications. Data Privacy and Compliance It ensures adherence to regulations and ethical considerations. Privacy implements controls to prevent unauthorized access and provides control over individuals' personal data. Regulatory frameworks such as the European Union’s General Data Protection Regulation (GDPR) and California’s Consumer Privacy Act (CCPA) impose stringent requirements on how businesses collect, process, and safeguard personal data. Data Metrics Management Defining and implementing robust business metrics and key performance indicators (KPIs) to quantify the enterprise-wide impact of data governance is critical to its success. These measures should be clearly articulated, inherently quantifiable, tracked longitudinally, and applied each year consistently to ensure comparability, accountability, and continuous improvement. Some of the metrics monitoring activities are, Aligning KPIs to strategic goals (e.g., data-quality gains, reduced time-to-insight, compliance rates, cost savings)Leveraging real-time dashboards for ongoing visibilityConducting annual KPI reviews to recalibrate targets and processes as the organization evolves Modern Data Architecture for AI A modern data architecture provides capabilities necessary for analytics, machine learning, generative AI, and autonomous systems. It enables enterprises to manage data as a strategic asset while ensuring governance, security, and scalability. The architecture is a unified, governed, AI-ready data foundation that enables trusted insights, intelligent automation, and autonomous decision-making through reusable data products, continuous observability, and embedded governance controls. The architecture is organized into two structural categories. The first five layers form the primary pipeline, the path data travels, from the moment it is created in a source system to the moment it produces a business outcome. The remaining three layers are cross-cutting disciplines that are applied continuously, at every stage, from ingestion through consumption. A modern AI-ready data architecture provides the infrastructure necessary for analytics, machine learning, generative AI, and autonomous systems. It enables organizations to manage data as a strategic asset while ensuring governance, security, and scalability. Figure 2: Enterprise Data Architecture For AI Data Sources This layer represents the full surface area of enterprise data — every system, channel, partner relationship, and unstructured artifact that generates information the organization can use. This layer groups the ecosystem into four major categories: Operational systems: The systems of record that run the business day-to-day: ERP, CRM, domain platforms, billing, and HR, etc. These remain the backbone of structured, transactional data.Digital channels: Web, mobile, API, and customer portal through which customers and employees interact directly with the enterprise. These channels are not purely a source; they also receive personalized or real-time data back through APIs.Partner ecosystems: B2B integrations, data exchanges, and marketplaces that bring external, third-party data into the enterprise's view.Unstructured and knowledge: Documents, email, video, knowledge bases, and ontologies. This category has grown in strategic importance because it is precisely the content that large language models and retrieval-augmented generation (RAG) pipelines depend on. Ingestion, Integration, and Orchestration This helps to move data from source into the platform reliably, securely, and in the right cadence, like batch, streaming, or on-demand. This layer comprises four capability areas, Data pipelines and orchestration: Engines that sequence and monitor data movement, paired with pipeline observability so failures and delays are visible before they become business problems.API management: Gateways, throttling, versioning, and security policy enforcement for every API-based integration, ensuring that data movement through APIs is controlled rather than ad hoc.Streaming and events: Event hubs and pub/sub infrastructure (e.g., Kafka-style platforms) that support event-driven integration for use cases where near-real-time movement is required.Data virtualization: Query federation that lets consumers query across multiple heterogeneous stores without first physically consolidating the data, reducing duplication and latency for enterprise usage. Core Data Platform (Analytics + AI) This is the heart of the architecture that acts as an AI-ready layer. It provides a unified storage and serving layer. This is the place where data lives and is made available for both traditional analytics and AI workloads from a single, governed foundation. Lakehouse and warehouse: It combines the flexibility of a data lake with the performance and semantic structure of a warehouse, including reusable semantic models that give consistent business meaning to raw tables.Operational data stores (ODS): Supports near-real-time reporting for use cases that cannot wait for a batch cycle.Vector and knowledge layer: Vector databases and ontologies that power agentic AI and semantic search are foundational to GenAI.Feature and model stores: Reusable features, a model registry, and model artifact storage, enabling machine learning models to be built, versioned, and reused consistently rather than recreated per project.Content and document stores: A repository that supports GenAI applications operating directly over enterprise content (contracts, policies, knowledge articles). AI, Analytics, and Decision Intelligence In this layer, the governed data is converted into insight, prediction, and increasingly autonomous action. Descriptive and diagnostic: BI, dashboards, and self-service analyticsPredictive and prescriptive: Machine learning models, optimization, and simulation GenAI and agentic AI: Copilots, task-oriented agents, and RAG pipelines that generate content to take bounded actions on the enterprise's own dataDecision intelligence: Composite decision flows that blend rules engines, analytics, and AI models into a single decision path Data Management and Semantics Layer Makes data trustworthy, findable, and consistently defined. This is applied continuously across every stage of the pipeline rather than as a single processing step. Enterprise data catalog: Technical and business metadata plus a data marketplace, such that stakeholders and systems can discover what data exists and what it means.Business glossary: Shared definitions, metrics, and domain vocabularies that prevent the classic problem of different business units calculating "revenue" or "active customer" differently.MDM and reference data: Golden records for core entities such as provider or product, eliminating duplication and conflicting versions of the truth.Data quality and profiling: Rules, scoring, and remediation workflows that continuously monitor and improve data fitness for use.Lifecycle management: Retention, archival, tiering, and deletion policies that keep the data estate compliant and cost-efficient over time. Agentic AI Governance, Security, and AI TRiSM Protects data and models with policy, privacy, identity, and full traceability. Policy-as-Code: Codified policies that are enforced programmatically rather than documentedLeast-privilege tool scope: Agents should operate with scoped function definitions rather than open-ended enterprise API access. Tools exposed to agents must enforce fine-grained parameter constraints AI TRiSM (Trust, Risk, and Security Management): Model risk assessment, explainability, fairness testing, and ongoing monitoring, addressing the risks introduced by AI/ML modelsIdentity delegation and impersonation: Enterprise agents must pass user identity context (OAuth 2.0 Token Exchange/On-Behalf-Of flow) down to underlying APIs. The agent must never inherit broader database permissions than the initiating user.Privacy and protection: PII/PHI classification, masking, and tokenization to limit exposure of sensitive data.Access and identity: RBAC/ABAC, fine-grained entitlements, and a Zero Trust posture, ensuring access is granted on a least-privilege basisLineage and observability: End-to-end lineage across data, models, and promptsPrompt/Context provenance and non-determinism audit: Every dynamic branch decision made by an agent must log its inputs, system prompts, retrieved context chunks, and seed parameters. This ensures that non-deterministic outputs can be audited post-hoc for compliance, debugging, and root-cause analysis during hallucinations or incorrect tool dispatches.Lineage granularity for vector and RAG workflows: Lineage models must extend beyond tabular source-to-target paths to map vector embedding lineage, tracing an agent’s final action back through the vector search embeddings, semantic chunking boundaries, and original unstructured document versions. Platform Engineering and MLOps/DataOps Dedicated engineering discipline. DataOps: CI/CD for data pipelines, including automated testing and deployment, bringing software-engineering rigor to pipeline changes.MLOps: CI/CD for models, including drift detection and automated retraining, so model performance is managed as an ongoing operational concern rather than a one-time deployment event.Platform engineering: Self-service portals, templates, and guardrails that let data and AI teams provision what they need quickly while staying within approved patterns.Infrastructure layer: Serverless compute, storage tiering, and cost management, ensuring the platform scales economically as usage grows. Business Consumption and Experience In this layer, the value is realized. The components and agents in this layer call back into the AI/Analytics layer in real time to inform what gets built upstream. Line-of-business applications: Domain applications, operations tooling, and customer service platforms through which employees and customers experience the businessCopilots and agents: Embedded copilots and agents inside applications and communication channelsAutomation and orchestration: Business process management (BPM), robotic process automation (RPA), and event-driven automation that act on insight without requiring manual interventionKPIs and value realization: OKRs, business outcome tracking, and benefit tracking that close the loop, measuring whether the solution is delivering value Benefits of Data Governance Enterprises with mature governance capabilities experience higher AI model accuracy, increased data trust, better decision-making, faster innovation cycles, improved compliance posture, reduced data duplication, and greater business agility. It also helps in: Ensuring consistent, uniform data across the enterprise, empowering smarter, more comprehensive decision supportEstablishing data integrity, data accuracy, completeness, trustworthiness, and dependability to achieve higher quality business decisionsHelping teams gain comprehensive decision support by enabling strong governance across the enterpriseDefining clear protocols for evolving data workflows; data governance helps in establishing agility and scalability for both the business and ITReducing duplication of effort and improving productivityMaking better-informed decisions with accurate and reliable dataIncreasing efficiency by introducing the ability to reuse data and data processesLowering the expenses of data management by implementing centralized control mechanisms and reducing the risk of data breachesReducing the volume of data collected and retained, optimizing data storage, and improving data management practicesEnhancing trust in the accuracy of data and the documentation of data-related proceduresEnsuring adherence to data regulations and supporting compliance with the EU’s GDPR, California Consumer Privacy Act (CCPA), Health Insurance Portability and Accountability Act (HIPAA), and the Payment Card Industry Data Security Standard (PCI-DSS) Conclusion Data governance is not a one-time activity, but it’s a journey. It is not optional but mandatory. It enables insight generation and informed decision-making. Effective data governance is a collection of processes, people, policies, standards, and metrics that ensure the efficient and effective use of data, enabling an enterprise to achieve its goals. It helps streamline operations, minimize data risks, enhance decision-making, drive innovation, create data policies, maximize data usage, and improve business efficiency and competitiveness. The modern AI data architecture is a unified, governed, and AI-ready foundation that turns enterprise data into trusted decisions and measurable business outcomes, with governance and AI risk management built in from the first byte rather than added at the end. By implementing data governance best practices, enterprises can ensure that they are managing their data to maximize its value. Acknowledgements The authors would like to thank Tricon Solutions LLC and Gspann Technologies, Inc for giving the required time and support in many ways in bringing up this article. Disclaimer The views expressed in this article/presentation are those of the authors, and Tricon Solutions LLC and Gspann Technologies, Inc. do not subscribe to the substance, veracity, or truthfulness of the said opinion.

By Dr Gopala Krishna Behara CORE

In many analytics platforms, there are performance issues that do not always come from complex transformations. Sometimes the bottleneck is much simpler: the same large datasets are being read repeatedly from remote storage. This pattern is common in shared analytics environments. A data engineering job reads a curated dataset to build aggregates. A BI refresh reads the same table again. A data science notebook filters the same records during exploration. Another scheduled workflow joins against the same reference data several times during the day. Each workload may be valid on its own, but together they create repeated remote reads. Over time, this can increase query latency, consume unnecessary infrastructure resources, and make interactive analytics feel slower than expected. Databricks disk cache is designed to help with this type of workload. It stores copies of remote Parquet data files on the local storage of worker nodes so that repeated reads can be served locally instead of fetching the same files again from cloud object storage. This article walks through a practical use case for using Databricks disk cache to improve repeated analytics workloads. The focus is not simply on enabling a feature, but on understanding when disk cache helps, where it fits in a pipeline, and what tradeoffs teams should consider before relying on it. The Use Case: Repeated Reads From Curated Analytics Tables Consider a common analytics setup. A team maintains a curated dataset that is used by multiple downstream workloads. The table is stored in cloud object storage and accessed through Databricks. It is already cleaned, standardized, and partitioned by date. Several jobs and users access this table throughout the day. The dataset supports different types of work: dashboard refreshesscheduled aggregationsexploratory notebooksfeature preparation jobsad hoc analysisdownstream transformation pipelines. The problem is not that the table is poorly designed. The problem is that the same files are repeatedly scanned from remote storage. In this situation, the first read of the data still needs to fetch files from remote storage. However, after the data is cached locally on worker nodes, repeated reads can avoid some of that remote access. For workloads that repeatedly query overlapping data, this can make a noticeable difference. This use case is especially relevant when teams work with large Parquet or Delta tables where the same filtered slices are accessed multiple times. Where Disk Cache Fits in the Pipeline Disk cache is not a replacement for good data modeling, partitioning, or query optimization. It works best as an acceleration layer for workloads that already read reasonably structured data. A practical architecture may look like this: Data Architecture Pipeline With Cache Layer The important point is that disk cache usually adds the most value after data has already been curated. If raw data is messy, unpartitioned, or constantly changing, caching alone will not solve the deeper performance problem. A better pattern is to first create reliable curated datasets and then use disk cache to improve workloads that repeatedly read those datasets. Why Repeated Reads Become Expensive Cloud object storage is highly scalable, but repeatedly reading the same large files still introduces overhead. A query may need to: locate filesread metadatafetch data over the networkdeserialize columnar datascan partitionsapply filterspass data into downstream transformations When one workflow performs this operation, the cost may be acceptable. When several workloads read the same dataset repeatedly, the overhead becomes more visible. This is especially noticeable in interactive analytics. A user may run one query, adjust a filter, run another query, and continue exploring. If every query repeatedly fetches the same underlying files from remote storage, the user experience can degrade quickly. Disk cache helps by keeping frequently accessed data closer to the compute layer. Disk Cache vs Spark Cache One source of confusion is the difference between Databricks disk cache and Apache Spark cache. Spark cache is usually applied manually to a DataFrame or table. It is useful when a specific intermediate result will be reused within the same job or notebook. However, Spark cache requires the developer to decide what to cache and when to unpersist it. Databricks disk cache behaves differently. It works at the file-read level and stores remote Parquet data files locally on worker nodes. When the same data is read again, Databricks can serve it from local disk instead of fetching it again from remote storage. A simple way to think about the difference is this: Spark Cache Developer-controlledApplied to DataFrames or RDDsUseful for reused intermediate resultsRequires explicit cache management. Databricks Disk Cache Managed by DatabricksApplied to remote Parquet/Delta file readsUseful for repeated reads from storageUses local worker disk. In practice, these two caching approaches solve different problems. Spark cache is useful when the same transformed DataFrame is reused multiple times inside a workload. Disk cache is useful when workloads repeatedly scan the same remote Parquet or Delta files. Using the wrong caching strategy can lead to unnecessary memory pressure, unstable performance, or no real improvement. A Practical Example Without Making It Industry-Specific Assume an organization maintains a large curated events table. The table contains activity records from different systems and is used for reporting, operational analytics, and product usage analysis. Several teams query this dataset daily. One dashboard refresh reads the last 30 days of activity. A transformation job reads the same table to calculate weekly aggregates. Analysts use notebooks to filter the data by region, product, and time period. Another pipeline reads the same table to prepare downstream metrics. Even though the consumers are different, many of them repeatedly access the same recent partitions. Without disk cache, these workloads repeatedly read files from remote storage. With disk cache, frequently accessed Parquet files can be stored locally on workers after the first read, allowing later reads to avoid repeated remote fetches. This is not a dramatic redesign of the pipeline. It is an optimization layer that improves workloads with repeated access patterns. When Disk Cache Helps Disk cache is most useful when workloads repeatedly read the same data files. Good candidates include: frequently queried Delta or Parquet tablesdashboard refreshes that scan the same recent partitionsexploratory notebooks that repeatedly filter the same datasetshared reference tables used across multiple joinsiterative analytics workflowsrepeated batch jobs using overlapping input data. The key pattern is repeated access. If every job reads a completely different dataset, disk cache will have limited benefit. If data is accessed once and never reused, the first read still has to fetch the files from remote storage. Disk cache is most effective when the same data is accessed more than once by workloads running on the same or similar compute resources. When Disk Cache May Not Help Much Caching is not a universal performance solution. Disk cache may provide limited improvement when: workloads read data only oncetables change constantlyqueries scan entirely different partitions each timetransformations are CPU-bound rather than I/O-boundjoins and shuffles dominate execution timeclusters are frequently restartedworker nodes are frequently replaced. This last point matters in elastic environments. If workers are decommissioned, local cache data on those workers is lost. The next workload may need to reread data from remote storage. This does not make disk cache unreliable. It simply means teams should understand its behavior before treating it as a guaranteed performance layer. How To Evaluate Whether Disk Cache Is Helping A common mistake is assuming that caching is helping just because it is enabled. A better approach is to compare workload behavior before and after repeated reads. Useful evaluation questions include: Does the second run complete faster than the first run?Are repeated queries reading overlapping data?Is the workload I/O-bound or shuffle-bound?Are the same partitions being scanned repeatedly?Are clusters stable long enough for cache reuse?Are users querying curated tables or constantly changing raw data? Teams should also compare job execution stages. If most time is spent reading remote files, disk cache can help. If most time is spent in large joins, aggregations, or shuffles, caching file reads may only improve part of the workload. Performance tuning should start with measurement, not assumptions. Designing Pipelines To Benefit From Disk Cache To get value from disk cache, the pipeline should be designed in a way that encourages reusable reads. One practical pattern is to separate raw ingestion from curated analytical datasets. Raw data may be inconsistent, frequently updated, and can be less suitable for repeated consumption, while curated datasets are usually cleaner, more stable, and more likely to be accessed repeatedly. A stronger design looks like this: Designing Pipelines for Disk Cache Optimization This design allows disk cache to work on datasets that are already optimized for downstream use. Partitioning also matters. If tables are partitioned in a way that matches query patterns, repeated workloads are more likely to access the same files, if partitioning is poorly aligned with usage patterns then queries may scan too much unnecessary data which would reduce the benefit of caching. For example, if most users query recent data, organizing the table around time-based access patterns can make repeated reads more efficient. Disk cache should be viewed as part of a broader performance strategy, not as a substitute for table design. Operational Considerations There are a few operational details teams should consider before depending heavily on disk cache. First, disk cache depends on local storage on worker nodes. Choosing worker types with local SSD storage can improve caching effectiveness. Second, cache behavior is tied to the lifecycle of the compute environment. If clusters restart frequently, cached data may not persist long enough to benefit repeated workloads. Third, disk cache works best when workloads have predictable reuse patterns. Highly random access patterns are less likely to benefit. Fourth, teams should monitor whether performance improvements are consistent. If query times vary significantly, the issue may not be remote reads alone. The bottleneck may be skewed partitions, insufficient cluster resources, poor join strategy, or inefficient transformations. Finally, caching should not be used to hide poor pipeline design. If a table is too wide, poorly partitioned, or filled with unnecessary historical data, disk cache may improve repeated reads but will not fix the underlying design problem. Avoiding Common Mistakes A few mistakes appear frequently when teams start relying on caching. The first mistake is caching too early in the pipeline. Raw datasets are often unstable and less useful for repeated analytical access. Caching is more valuable after data has been cleaned, standardized, and organized for consumption. The second mistake is confusing disk cache with Spark cache. Spark cache is useful for reused intermediate DataFrames. Disk cache is better suited for repeated reads of remote Parquet or Delta files. The third mistake is ignoring cluster behavior. If compute resources are short-lived, cache reuse may be limited. The fourth mistake is measuring only one query run. Since disk cache is useful for repeated reads, teams should compare cold-read and warm-read behavior rather than judging performance from a single execution. The fifth mistake is treating disk cache as a substitute for optimization. Good partitioning, file sizing, query filtering, and transformation design still matter. Practical Checklist Before depending on disk cache, teams should ask: Are the same datasets read repeatedly?Are workloads reading Parquet or Delta data?Are the tables curated and reasonably stable?Are query patterns predictable?Are clusters stable enough for cache reuse?Are bottlenecks related to file reads rather than shuffles?Are partitions aligned with common access patterns?Are performance gains measured across repeated runs? If the answer to most of these questions is yes, disk cache is likely worth evaluating. If the answer is no, teams should first investigate table design, query plans, file layout, and transformation logic. Conclusion Databricks disk cache can be a useful optimization for analytics workloads that repeatedly read the same Parquet or Delta data from remote storage. It is especially helpful for curated datasets used by dashboards, notebooks, scheduled jobs, and downstream analytics workflows. However, disk cache should not be treated as a general solution for every performance issue. It works best when data access patterns are repeated, compute resources remain stable, and the underlying tables are already designed reasonably well. The biggest lesson is that caching should be intentional. Teams should understand where repeated reads happen, measure cold-read and warm-read behavior, and combine disk cache with good table design, partitioning, and pipeline structure. When used in the right context, disk cache can reduce repeated remote reads and make analytics workloads more responsive. When used without understanding the workload, it becomes just another configuration setting with unclear impact. Reliable analytics performance comes from knowing which bottleneck is actually being solved.

Running Apache Flink on a mainframe sounds odd at first. A modern stream processing engine on a platform most people call legacy? But take a closer look. It is not only possible. It might be a smart move for some of the largest financial institutions in the world. This post explores why some enterprises want Apache Flink on the mainframe, how it could work, and whether it is a brilliant innovation or a technical detour. Disclaimer: The views and opinions expressed in this blog are strictly my own and do not necessarily reflect the official policy or position of my employer. Mainframes Will Still Matter in 203X! A few months ago, I wrote about integrating Apache Kafka with mainframe systems. The blog covered various real-world examples across industries. The key message: Mainframes are still in use. In many organizations, they are not going away. They remain a central part of IT strategy, especially in banking, insurance, and the public sector. But they are not just legacy systems. Modern mainframes such as the IBM z17 offer the latest Telum II processor and support up to 64 terabytes of system memory. The z17 enables very large in‑memory workloads and faster processing for analytics and real‑time use cases. These systems also integrate on‑chip AI acceleration and optional AI‑focused hardware to support machine learning and real‑time decisions directly where mission‑critical data resides, while running modern Linux environments and container platforms. Some companies are still on the mainframe because they cannot easily migrate. But many others do not want to move away. Instead, they modernize around the mainframe. Apache Kafka and Flink play a key role in this journey. They enable a real-time data foundation that connects core systems with modern applications across environments. In future hybrid cloud strategies, this becomes even more critical. Kafka acts as the central nervous system, delivering the right data and context at the right time between on-prem mainframes and cloud-based AI services, including agentic AI and large language models. An event-driven architecture with hybrid streaming replication ensures business-critical decisions are made on fresh, reliable, and contextual information. Mainframe Migration Has Not Happened Ask any architect or CTO in banking. Mainframe migration has been on the roadmap for over two decades. Full replacement of core systems is still rare. However, it is important to distinguish between migration and offloading. Mainframe migration means shutting down mainframe workloads entirely and moving all applications and data to a new platform. There are many reasons: Risk is too highOrganizational resistance is strongMainframe skills are still needed but hard to findSystems are complex and deeply integratedThese applications run reliably and perform well Mainframe offloading, on the other hand, is much more common. It means moving selected workloads, queries, or processing tasks off the mainframe to more flexible and scalable platforms. This reduces load and cost on the mainframe while enabling innovation elsewhere. I have shared several real-world examples of offloading in action, using Kafka, IBM MQ, and Change Data Capture (CDC) tools like IBM IIDR or Precisely to synchronize and replicate data between mainframe systems and the cloud or distributed infrastructure in real time: Mainframe Offloading and Integration Examples. Because of this, many firms choose mainframe integration and a slow lift-and-shift leveraging the Strangler Fig design pattern over migration. Kafka is already helping. Flink is the next step. Apache Flink Meets the Mainframe: Unlikely Combo, Real Potential At first glance, Apache Flink and the mainframe seem like technologies from two different worlds. But combining them can unlock surprising value. What Is Apache Flink? Apache Flink is the leading open-source stream processing engine. It is designed to process high volumes of data continuously and in real time, rather than in batches. Flink is widely used to support use cases like fraud detection, customer personalization, operational monitoring, and data transformation at scale. Many of the largest tech companies and digital natives rely on Flink to process billions of events per day with low latency and high throughput. It supports both event streaming and batch workloads, but its true strength lies in real-time use cases. Here is an example of continuous stream processing leveraging Apache Flink together with OpenAI for Generative AI in real-time: Flink is built for modern environments. It runs natively on Kubernetes, integrates with Apache Kafka for real-time data ingestion, and is commonly deployed in public cloud, private cloud, or hybrid architectures. This makes it an ideal fit for enterprises looking to build fast, intelligent applications on fresh and contextual data. How to Run Apache Flink on the Mainframe? Yes, Apache Flink can run on the mainframe. In fact, it already does. I have already seen this deployed in a real-world environment. A large global financial institution is preparing to invest massively to expand its use of Apache Flink. Running Flink on IBM LinuxONE is a central part of that strategy. This is NOT a lab experiment, but a production-focused initiative. This bank already uses Kafka and Flink in production. Now they want to move Flink compute workloads onto the mainframe. The reason is simple. They already have unused compute on LinuxONE. Running Flink there is cheaper and easier to scale (for some companies) than scaling out other systems. The architecture is modern. IBM LinuxONE runs OpenShift. IBM LinuxONE is a high-performance, enterprise-grade server built on IBM Z architecture. It is designed to run Linux workloads with extreme reliability, scalability, and security. Unlike traditional mainframes focused on COBOL and legacy apps, LinuxONE is optimized for modern Linux applications. Flink is deployed in containers inside OpenShift's Kubernetes infrastructure, just like in any other cloud or data center. From a technical perspective, you need to build Docker images for the s390x architecture to run Apache Flink on IBM LinuxONE. In addition, components like RocksDB, which is used as a state backend in Flink, must be compiled for s390x to ensure full functionality. Why Put Apache Flink on the IBM Mainframe? This is a very valid question! Nobody would buy a mainframe just to run Flink on it. However, this approach offers several benefits for organizations that already own and operate mainframe infrastructure: Available compute resources on the mainframe.Consume data directly from mainframe sources (such as IBM MQ or other integration interfaces) and process it directly on the mainframe; or consume data from external sources such as a Kafka cluster running on x86 infrastructure, enabling flexible integration across hybrid environments.Lower total cost of ownership (TCO) regarding hardware and license costs compared to adding new external x86 servers and bi-directional integration pipelines.Simplified operations within a single, familiar environment.Benefit from IBM actively promoting LinuxONE and driving more workloads onto the platform. Mainframes are not only still alive. They are growing. IBM’s infrastructure business, which includes the mainframe, is doing very well. In Q3 2025, IBM reported 3.6 billion dollars in revenue for the infrastructure segment. That is 17 percent growth. IBM Z alone grew 61 percent. In Q2 2025, infrastructure revenue was 4.14 billion dollars, beating expectations by a wide margin. This is not legacy tech in decline. It is a platform in transformation. A New Chapter for Stream Processing and Mainframes Apache Flink running on the mainframe may sound unusual at first, but it reflects a broader shift in how enterprises think about modernization. The mainframe is not just a legacy system to replace. IBM Mainframe can fit hybrid cloud strategies, especially in highly regulated industries like banking and insurance. Apache Flink brings real-time intelligence. The mainframe brings performance, reliability, and unmatched security. Together, they offer a powerful combination for building fast, contextual, and mission-critical applications, without abandoning existing infrastructure. With Kafka as the backbone and Flink as the engine for real-time processing, organizations can connect mainframe systems with cloud innovation, including advanced AI workloads. This is not just about preserving the past. It is about extending and reusing trusted systems to meet the demands of the future. Enterprises that embrace this model can reduce risk, increase agility, and unlock new value from the heart of their operations.

By Kai Wähner CORE

Dashboards are everywhere. Business and IT teams use them to track metrics, visualize trends, and make decisions. But when working with real-time data from Apache Kafka, it’s not obvious how to connect dashboards to the stream or whether you should at all. The conversation often jumps to technical options like Flink SQL, Kafka Streams Interactive Queries, or Confluent's TableFlow. Others try to build interactive dashboards directly on top of Kafka topics using a JDBC connector into a database and a Business Intelligence tool. But that only makes sense once the actual goal is clear. What is the business trying to do with the data? Dashboards are not always the right tool. Automation, smart agents, or process intelligence often deliver more value. Let’s unpack the bigger picture. This blog post breaks down the different types of queries on Apache Kafka data, when dashboards make sense, and why a context engine often plays a key role. Why Dashboards — And When Not To Use Them Dashboards give people visual access to data. They support decisions, reporting, and oversight. But not all data needs to be visualized. Dashboards make sense when: Business users want a regular view of changing dataTeams need to investigate operational metricsThere is a requirement for manual filtering and inspection But in many cases, dashboards are not the best answer. For example A machine overheating should trigger an alert, not wait for someone to look at a graphA fraud detection system should act instantly, not visualize the anomalyAn AI agent monitoring supply chains should get structured context, not a dashboard snapshot In these scenarios, dashboards are a fallback. The real need is action or automation, not visualization. This is where agentic AI and process intelligence come into play. AI agents require structured, fresh context. They do not use dashboards. They consume streaming data, apply logic or reasoning, and trigger downstream actions. Dashboards might still be used to audit what happened but not to drive the process itself. So before jumping into dashboard tools, first ask: Is this data for a human to observe or a system to act on? Foundations First: Apache Kafka, Event Streaming, Data Products, and Governance Apache Kafka is the core of modern event-driven architecture. It enables systems to stream events in real time, such as customer interactions, machine signals, backend transactions, or system logs. Unlike batch pipelines, event streaming allows continuous data flow across the business. This supports responsive applications, automation, and real-time analytics. But fast data is not enough. Real-time value depends on reliable data. That’s why many teams now treat Kafka topics as data products. Each stream should have a clear owner, a defined schema, and a contract between producers and consumers. Schemas must be versioned and validated. Metadata must be consistent and available. Lineage, access control, and quality checks are critical to avoid downstream errors. Without this foundation, queries will return incorrect results, and automation may act on bad signals. Governance, schema control, and product thinking are not extras. They are required to build trustworthy systems on streaming data. Three Kinds of Queries for Apache Kafka Events If a dashboard is needed, the next step is understanding the type of query behind it. This helps define the right technical setup. Operational Queries These are fully automated. They respond to events and trigger actions. Think of them as the nervous system of an application. They are built directly into stream processing applications using Apache Flink or Kafka Streams. The logic is reactive and runs continuously. These systems are part of mission-critical operations. They must be highly available, fault-tolerant, and operate with minimal latency. Any downtime or delay can disrupt core business processes. A modern data streaming platform that augments Kafka and Flink with on-the-fly table serving, snapshot queries, and a context engine helps close this gap between streaming and interactive exploration. Example use cases include raising alerts on thresholds, aggregating orders for reporting, or triggering workflows. These systems should not rely on dashboards. Explorative Queries These are used by people to explore the data. They are ad hoc, flexible, and interactive. This type of query is difficult to support directly on Apache Kafka. Kafka is optimized for high-throughput event streaming and acts as an immutable event log. It provides a durable persistence layer and decouples producers from consumers, which makes it ideal for data pipelines and ensuring data consistency across real-time and batch systems. However, it is not designed for indexed lookups or ad hoc filtering across large datasets. Kafka does not offer queryable storage, secondary indexes, or snapshot consistency, all of which are essential for interactive exploration. Flink can process the data, but it does not offer indexed access. That makes joins or drilldowns inefficient without an external engine. Exploratory queries are often run in SQL workbenches, BI tools like Superset, or analytical engines like Druid and ClickHouse. They are useful for finding anomalies, trying out new logic, or investigating correlations. They require indexing, snapshot consistency, and historical access. Example use cases include joining marketing and sales events to find conversion patterns, analyzing user journeys through digital platforms, or testing new business rules across historical data. These queries typically require interactive tools and should not rely on stream processing systems alone. Monitoring Dashboards This use case is simpler but more common. The goal is to display filtered, consistent, and up-to-date data to end users. It does not involve complex joins or deep exploration. Instead, dashboards show metrics from recent data, business KPIs, or precomputed aggregations. Tools used here include Power BI, Grafana, or custom frontends connected to Flink or TableFlow. Dashboards in this case should be thin and rely on upstream systems for logic. Example use cases include showing live production status on a factory screen, displaying transaction volumes in a finance dashboard, or visualizing the health of streaming pipelines for operations teams. These dashboards are read-only and should not contain business logic. What Businesses Really Need Today While use cases vary, a few patterns repeat across industries. These needs can guide architecture decisions. Lightweight dashboards with filtering but no complex joins: Power BI and Grafana are the most common tools. Used for message tracing, monitoring, and status overviews. Users prefer querying externally instead of importing data.Real-time data that stays up to date: Dashboards refresh automatically. Data is pushed from Flink or precomputed topics. Materialized views support this, but changing schema can cause frontend problems.Business logic belongs upstream: Dashboards should not do computation. Flink or Kafka Streams handle the logic and prepare the data.Integration with ML models and agents: Dashboards may show results from predictions or scoring models. These are often trained ML models, not LLMs. Model drift monitoring is gaining interest. LLMs are still early stage in these setups.Protocol-agnostic connectors: REST, WebSocket, MQTT, JDBC — all needed. Most organizations expect flexible integration. Sink connectors alone are often not enough. APIs with query parameters are common requests. The Context Engine: Serving Dashboards and AI Agents from Apache Kafka Events A powerful pattern is the context engine. It connects Kafka streams to dashboards and AI systems by offering real-time, structured, and indexed access to data. It works like this Flink or Kafka Streams process raw Kafka topicsOutput data flows to context topicsA service builds indexed views of relevant business objectsDashboards and agents query those views through an API This setup creates a reliable source of truth. Business logic stays in the stream. The context engine focuses on enrichment, access control, and exposing views. For AI agents, this API layer usually follows the Model Context Protocol (MCP), which is becoming the de facto interface for connecting agents to structured enterprise data. Dashboards, in contrast, are typically served from materialized views in cache or in-memory databases, or directly through REST APIs optimized for low-latency reads. Agentic AI systems benefit directly. They consume these views as context to make decisions in real time. Instead of querying raw data or relying on stale batches, they get structured signals. Generative AI also benefits, using the same views as grounding data. Dashboards and AI agents both rely on fresh, accurate context. A context engine provides that bridge. Start With the Use Case, Not the Tool The right dashboard architecture does not start with a tool choice. It starts with business needs. Ask the right questions: What decisions or actions should this data support?Is the goal observation or automation?Does the user need filtering, drilldowns, or live KPIs?How fresh must the data be?Can the logic run upstream, or must it remain flexible? These answers will guide the setup. Sometimes a simple Power BI dashboard is enough. Other times a context engine or Flink job is required. In many cases, a dashboard is just the user interface to something much more powerful running behind the scenes. Of course, even when the focus is on business outcomes, a tool still has to be selected. That decision should follow the use case, not drive it. There are many options. Some teams prefer code-driven frameworks that give full control and allow deep integration with APIs and AI agent interfaces. Others choose no-code or low-code tools with prebuilt widgets so business users can create interactive views quickly. Each option comes with trade-offs in flexibility, governance, scalability, and integration. Exploring these tooling choices in depth would fill an entire chapter on its own. The key message here is simple: start with the outcome. The tool is an implementation detail. Build for the decision, not for the visualization. That is how streaming data creates real business value.

By Kai Wähner CORE

For the past two decades, most enterprise data engineering systems have been built on one default assumption: People understand the system. The system executes the pipeline. Engineers understand the business context, break a requirement into steps, write SQL, Spark jobs, shell scripts, synchronization tasks, and scheduling workflows, and then let the system run them. The scheduler does not need to understand the business. The sync engine does not need to understand the metric. It only needs to execute the predefined flow reliably. That model supported the era of data warehouses, data lakes, BI reporting, and batch scheduling very well. But now that assumption is starting to break down. Enterprise data systems are becoming more complex in every direction: More data sources.Longer pipelines.Stronger real-time requirements.Faster business changes.More conflicting metric definitions.More AI application data, model feedback data, vector indexes, and unstructured content. In this environment, enterprises do not just need more pipelines, and they do not just need a better Copilot that can write SQL faster. They increasingly need a Data Engineering Agent that can understand the system, plan tasks, call tools, validate outcomes, and accumulate experience over time. In that shift, Apache SeaTunnel becomes especially important. Because in the agent era, it is not enough for a system to "think." It also has to connect to real data sources, capture changes, execute synchronization, process incremental updates, preserve consistency, and move data to target systems in a reliable and cost-effective way. In other words: The agent understands the goal and plans the action. SeaTunnel turns that action into real, reliable, and recoverable data movement. That is why SeaTunnel is well positioned to become a core execution foundation in the evolution from ETL, ELT, and EtLT to agent-driven data engineering. ETL to ELT: The First Major Shift Traditional ETL is straightforward: Extract data from the source.Transform it in an intermediate layer.Load the processed result into the target system. This model fit the early data warehouse era well. At that time, data sources were relatively limited, the pipeline was easier to understand, and compute resources were more centralized. Enterprises wanted to clean the data, standardize the structure, and define the core logic before loading data into the warehouse. At its core, ETL is a deterministic pipeline model. Its key assumption is: People define the process in advance. The system executes the process. Later, with the rise of cloud warehouses, data lakes, lakehouse architectures, and elastic compute, ELT became more popular. ELT changed the order: ExtractLoadTransform inside the target platform. Instead of transforming everything before loading, enterprises started moving raw or near-raw data into a unified storage layer first, then using the target platform's compute power for downstream modeling and analytics. ELT solved several ETL limitations: It reduced upfront processing complexity.It preserved more original data.It gave analysts and modeling teams more flexibility later. But ELT also created a new problem. If all transformation is delayed until after loading, then dirty source data, schema drift, type mismatches, CDC events, privacy fields, and format inconsistencies all arrive directly in the target system. That might be acceptable in simple batch scenarios. It becomes much more expensive in real-time synchronization, CDC, multi-table sync, lakehouse ingestion, SaaS API ingestion, and AI-oriented data engineering. That is where a third pattern becomes more useful: EtLT. Why EtLT Matters EtLT is not just a compromise between ETL and ELT. A more useful way to understand it is: Extract -> lightweight transform -> Load -> semantic Transform That means: Extract the data.Apply the minimum engineering transformations required to make the data usable.Load it into a unified data foundation.Apply business-level and semantic transformation later. The key idea is the distinction between lowercase t and uppercase T. Lowercase t is not heavy business modeling. It is the engineering work that must happen before data enters the platform safely and consistently, such as: Field projectionType mappingFormat normalizationPrimary key or partition field handlingSensitive field maskingCDC event conversionMulti-table routingSchema evolution handlingPre-ingestion quality validationOne-read, multi-write patternsRate limiting and parallelism control. These transformations should not always be postponed to the target system. Otherwise, the lakehouse or warehouse becomes full of inconsistent, weakly governed, and semantically unclear raw data. At the same time, lowercase t should not try to absorb all business logic. Complex business definitions, KPI semantics, subject-area modeling, and cross-domain aggregation still belong to uppercase T, which should happen in the warehouse, lakehouse, semantic layer, or metric layer. That is the value of EtLT: Standardize the data engineering layer before loading, then apply business semantics after loading. This is exactly the place where SeaTunnel fits naturally. Its Source, Transform, and Sink architecture is well suited for the lowercase t in EtLT. It can connect heterogeneous systems, apply lightweight transformation during movement, handle CDC, adapt schemas, route multiple tables, and write the result into the target platform. In an EtLT architecture, SeaTunnel is not just a data mover. It becomes the data integration runtime that prepares data before it enters the unified data foundation. Why Traditional ETL Starts to Struggle Traditional ETL is built for relatively stable pipelines. You write the rules, draw the DAG, schedule the tasks, and fix failures when they happen. But modern enterprise data environments are no longer that simple. Today a single enterprise may operate across: OLTP databasesKafka streamsCDC pipelinesSaaS APIsObject storageLogs and eventsLakehouse platformsReal-time OLAP systemsVector databasesAI interaction logsModel output datasets. The problem is not only that there is more data. The data is also more fragmented, more heterogeneous, and more real-time. Pipeline length is another issue. A single business metric may depend on dozens of tables, multiple layers of wide tables, several business domains, and a long chain of definition changes. At that point, many enterprises no longer struggle with "Can we build the workflow?" They struggle with "Can anyone still explain the whole pipeline end to end?" This is where traditional ETL shows a structural limitation. One renamed field can break hundreds of tasks.One changed enum can silently shift multiple core metrics.One incorrect incremental logic branch can pollute an entire downstream analysis chain. The scheduler can tell you that a task failed. It usually cannot tell you why that task matters. The sync tool can move the data. It usually cannot tell you which business metric is now at risk. The engineer can fix the script. But only if that engineer can first rebuild the missing context. So the real weakness of traditional ETL is not just performance or reliability. It is that: It can execute the process, but it does not understand the system. Why Copilot is Not Enough Many teams first bring AI into data engineering through Copilot-style workflows: Generate SQLComplete Spark codeDraft YAMLProduce test samples. These capabilities are useful. They improve local productivity. But they do not solve the deepest problem in enterprise data engineering. Because the hardest part of data engineering is rarely just code generation. It is system understanding. Copilot can help generate a SQL statement, but it does not know the real business meaning of the field. It can help draft a synchronization task, but it does not know which downstream metrics will be affected by a schema change. It can help generate a scheduler config, but it does not know whether the change breaks historical consistency or recovery semantics. What enterprises actually struggle with includes: Lineage reasoningDependency analysisSemantic understandingMetric governanceRisk estimationImpact analysisIncremental recovery. These are not just autocomplete problems. So enterprises do not only need an AI IDE. They increasingly need an agentic data engineering system that can understand the target, decompose tasks, call engineering tools, and verify the result. The Real Shift: From Pipeline to Agent If we keep only one conclusion, it is this: Traditional ETL is "people define the process, systems execute the process." Agentic data engineering is "people define the goal, systems generate the process." That is not a slogan. It is a change in how work is organized. In the traditional model, engineers design the task chain first, configure Source, Transform, and Sink, and then let the scheduler execute the pipeline. The system faces a fixed process. In the agent model, the input may only be a business goal. For example: Add a new gross margin metric for orders and keep it aligned with the finance definition. Traditionally, the engineer must: Identify relevant data sources.Read table schemas.Inspect lineage.Design transformation logic.Configure sync and scheduling jobs.Add quality checks.Run regression validation. In an agent-oriented workflow, the system should be able to generate a sequence of actions around the goal: Identify the affected business entities.Discover candidate data sources.Analyze upstream lineage.Decide whether the job belongs to ETL, ELT, or EtLT.Generate or update the SeaTunnel synchronization task.Configure full-load or CDC mode.Apply lightweight transformation.Write the result into the warehouse or lakehouse.Trigger data quality validation.Evaluate downstream impact.Present the result for human confirmation. That is the real difference. The breakthrough is not "AI wrote a SQL statement for me." The breakthrough is: The system starts generating engineering actions from a business goal. But this immediately raises a critical question: When the agent plans a data action, who executes it reliably? That is exactly where SeaTunnel becomes essential. Apache SeaTunnel in the Agent Era: The Data Integration Execution Layer An agent cannot stop at reasoning and recommendations. If a Data Engineering Agent decides that a table should be synchronized, a CDC job should be adjusted, a broken pipeline segment should be replayed, or a data slice should be reloaded into the target system, it needs a stable and observable execution layer to carry out that decision. That execution layer needs several core capabilities. 1. It Must Connect to Many Kinds of Data Sources Enterprise data systems are inherently heterogeneous. An agent cannot live in a world with only one database or one file system. It needs to connect to MySQL, Oracle, PostgreSQL, SQL Server, Kafka, Hive, Iceberg, Doris, ClickHouse, StarRocks, Elasticsearch, S3, HDFS, MongoDB, and many other systems. SeaTunnel's connector architecture is designed for exactly this kind of environment. It abstracts Source, Transform, and Sink through a consistent plugin model so heterogeneous systems can be integrated in a unified way. 2. It Must Support Batch, Streaming, CDC, and Large-Scale Synchronization The agent era does not run on a single data movement pattern. It needs: One-time full migrationContinuous CDCOffline batch movementReal-time synchronizationSingle-table syncMulti-table or database-level sync. SeaTunnel is valuable here because it is not just a script wrapper for ETL. It is a real data integration runtime that can support full load, incremental sync, real-time processing, CDC, and multi-table movement in the same ecosystem. 3. It Must Handle the Lowercase t in EtLT Agentic systems do not need every business transformation to happen inside the sync layer. But they do need the sync layer to complete the minimum engineering transformation required to make the data trustworthy and usable before it lands in the platform. SeaTunnel's Transform layer is a strong fit for: Field mappingType conversionFilteringColumn projectionData maskingRoutingSimple reshaping. That is exactly the role of the lowercase t in EtLT: Do not overload the movement layer with heavy business modeling, but make the data governable and ready for the next stage. 4. It Must Provide Consistency, Fault Tolerance, and Recovery An agent can decide that a broken link should be replayed. But replay only matters if the underlying system can recover correctly. The execution layer still needs checkpointing, failure recovery, state handling, restart behavior, and strong delivery guarantees where needed. A reasoning layer without a reliable execution layer becomes a planner without hands. That is why execution quality still matters as much as intelligence. What the Future Stack Starts to Look Like If we look one step ahead, enterprise data engineering increasingly resembles a layered operating system rather than a collection of disconnected pipelines. In that stack: The semantic layer defines the business model.Metadata provides structure and context.Memory accumulates operational experience.The planning layer turns goals into actions.The execution layer performs synchronization, CDC, movement, replay, and recovery. SeaTunnel belongs to this execution layer. That placement is important. The future is not "put a large language model on top of ETL." The future is a coordinated system where reasoning and execution are separated clearly: The agent decides what should happen.SeaTunnel ensures that it actually happens in a reliable way. The Evolution in One Sentence ETL built data pipelines. ELT moved raw data into a unified platform first. EtLT rebalanced pre-load engineering standardization and post-load semantic modeling. The agent era pushes data engineering one step further: From fixed pipelines to goal-driven systems. In that world, SeaTunnel is not just a synchronization tool. It becomes a practical execution foundation for agentic data engineering. Agents make data systems understand goals. EtLT makes ingestion more controllable. SeaTunnel turns those goals into reliable data engineering actions. That is the deeper change now happening across enterprise data engineering.

I learned this lesson the hard way. We had a critical data pipeline running for over 3 hours every single day. The logic was perfectly clean. The overarching schema was explicitly right. There were absolutely no obvious memory leaks, and absolutely nothing looked fundamentally broken in the raw PySpark transformations. Then I finally checked the physical query plan. Under the hood, Apache Spark was quietly executing a massive Sort-Merge Join to merge a multi-terabyte fact table with a dimensional lookup table that was barely 50MB in total size. One single line of code changed—wrapping that exact tiny lookup table cleanly in a broadcast() hint—and the exact same analytic job plummeted from 3 hours to just 18 minutes. That was it. One word saved us hours of daily compute and massive underlying cloud FinOps costs. Wrong join types are financially devastating directly because they are completely silent. Spark will not throw an aggressive exception. Your pipeline will not explicitly fail. It will simply execute your logic confidently 10× slower than it architecturally ever needs to. Here is the exact mental model I exclusively use now every single time I write a distributed join in Apache Spark. TL;DR: Silent shuffle bottlenecks kill massive Spark performance. Always explicitly broadcast small tables (< 200MB), default natively to Sort-Merge for massive dual-sided joins, and actively, aggressively leverage AQE skew joins to fundamentally prevent heavy task skew. Check your physical plans! 1. The Small Table: Always Broadcast When actively joining a massive fact table heavily against a tiny dimension table (like cleanly mapping a primitive status_id logically to a status_name), globally shuffling the multi-terabyte fact table wildly across the distributed cluster is architectural suicide. The rule: If a table is reliably under 200MB, rigorously physically force a Broadcast Hash Join.The mechanism: Spark intelligently bypasses the massive network shuffle entirely. It simply naturally copies the tiny 50MB table directly into the RAM of every single native worker node, allowing them to map data logically and locally. The Implementation Python from pyspark.sql.functions import broadcast # Wrapping the small lookup table strictly natively in a broadcast hint enriched_df = massive_fact_df.join( broadcast(small_lookup_df), "customer_id", "left" ) 2. Both Sides Massive: Default to Sort-Merge If you are systematically actively joining two massive, multi-terabyte tables accurately together (e.g., dynamically merging historical transactions cleanly with historical web_sessions), you physically cannot organically broadcast data without instantly dynamically triggering brutal Out-Of-Memory (OOM) driver exceptions. The Rule: Default heavily unconditionally to the Sort-Merge Join.The Mechanism: This is Spark's absolute most robust, incredibly stable joining algorithm physically built for massive scale. Spark heavily and organically shuffles the massive data wildly across the cluster so that precisely matching keys uniquely land physically on the exact same nodes, fundamentally and strictly sort them, and actively, efficiently, and accurately merge them natively. It is technically slower than a pure broadcast, but it is incredibly beautifully resilient inherently at petabyte scale. The Implementation Python # No explicit hints structurally required. Spark will cleanly natively default seamlessly to Sort-Merge for massive large datasets. final_df = massive_transactions_df.join( massive_sessions_df, "user_id", "inner" ) 3. Highly Skewed Data: Enable AQE Skew Join In heavy enterprise datasets, physical data is rarely organically distributed perfectly evenly. Imagine an active e-commerce platform where the default "Guest Customer" cleanly accounts for physically 60% of all universal platform transactions. If you intelligently execute a naive Sort-Merge Join broadly on customer_id, one single isolated Spark executor will physically be forced systematically to exclusively process the entire massive 60% "Guest" chunk. The other 199 regular executors will efficiently and cleanly finish in seconds and sit completely idle while that one node globally grinds to a halt. The rule: Actively, safely leverage Adaptive Query Execution (AQE) dynamically to natively, beautifully split heavily skewed partitions dynamically.The mechanism: AQE actively, dynamically, and securely detects massively skewed partitions directly mid-flight, accurately splitting them cleanly into optimally smaller, incredibly uniform, reliable sub-partitions so they can be effectively and seamlessly processed rapidly and cleanly in parallel. The Implementation Python # Ensuring AQE and Skew Join optimization are physically aggressively cleanly enabled locally in the specific cluster config spark.conf.set("spark.sql.adaptive.enabled", "true") spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true") The Silent Hero: Adaptive Query Execution (AQE) The part most data engineers fundamentally miss: AQE has actually been turned ON by default since Apache Spark 3.2. This means Spark is reading actual statistical data mid-job. If you execute a Sort-Merge Join on a massive table that unexpectedly shrinks to 40MB after an aggressive .filter() clause, AQE will actively intercept the job mid-flight and organically auto-switch the execution directly into a lightning-fast Broadcast Join. You technically don't have to code anything for this to happen. It organically just happens. But you do have to verify it. You must explicitly verify that spark.sql.adaptive.enabled is active in your environment. You must actively understand exactly what it is doing—because when AQE occasionally guesses wrong (usually due to stale table statistics), you need to know precisely how to aggressively override it with manual hints. Conclusion Performance tuning in distributed compute engines fundamentally comes down to actively understanding the physical network shuffle. Check your explicit joins. Aggressively read your physical query plans (using .explain()). And never blindly trust default configurations at the enterprise level. What is the absolute worst join performance bottleneck you have ever hit in production? Let me know in the comments below!

Change data capture (CDC) pipelines look straightforward on paper: capture database changes, publish them to Kafka, and update downstream systems. The difficulty starts when events are duplicated, consumers restart, projections drift, or a team needs to replay months of history without corrupting the state it is trying to recover. A reliable CDC design has to account for those failure modes from the beginning. That means combining Kafka and Debezium with idempotent writes, deterministic projections, controlled replay workflows, reconciliation checks, and enough recovery evidence to explain what happened when something goes wrong. The architecture: The goal is not only to move inventory changes quickly. The goal is to make replay safe enough that operators can rebuild and explain the derived state after failure. This article builds one concrete pattern: The important detail is that replay safety is not a single feature. It is the result of several boring decisions lining up correctly. Data Model The data model should separate the aggregate state, the classification state, and the transaction history. PLSQL CREATE TABLE inventory_stock_on_hand ( sku VARCHAR(64) PRIMARY KEY, stock_on_hand BIGINT NOT NULL, updated_at TIMESTAMP NOT NULL ); CREATE TABLE inventory_bucket ( sku VARCHAR(64) NOT NULL, bucket_type VARCHAR(32) NOT NULL, location_id VARCHAR(64) NOT NULL, quantity BIGINT NOT NULL, updated_at TIMESTAMP NOT NULL, PRIMARY KEY (sku, bucket_type, location_id) ); CREATE TABLE inventory_transaction ( event_id VARCHAR(128) PRIMARY KEY, sku VARCHAR(64) NOT NULL, seller_id VARCHAR(64) NOT NULL, delta_quantity BIGINT NOT NULL, event_time TIMESTAMP NOT NULL, accepted_at TIMESTAMP NOT NULL ); CREATE INDEX idx_inventory_transaction_sku_time ON inventory_transaction (sku, event_time); CREATE INDEX idx_inventory_bucket_sku_bucket ON inventory_bucket (sku, bucket_type); The transaction table is the recovery anchor. If the availability projection drifts, the system needs a history to explain the projection. Do not rely only on the mutable aggregate table. inventory_stock_on_hand is useful for fast reads, but it is not enough for recovery. If the aggregate is wrong, it cannot explain how it became wrong. The accepted transaction history gives replay something durable to reason from. Ingestion Event Use an event ID that can survive retries and replay. JSON { "event_id": "mkt-evt-8f11a", "sku": "1231241", "quantity": 100, "operation": "I", "event_time": "2026-06-19T18:23:11Z", "seller_id": "seller-42" } The consumer should perform an idempotent write. One pattern is to insert the transaction first using event_id as the primary key. If the insert fails because the event already exists, skip the duplicate and emit a duplicate-suppression metric. Java public InventoryWriteResult apply(InventoryEvent event) { try { transactionRepository.insert(event.toTransactionRow()); } catch (DuplicateKeyException duplicate) { metrics.increment("inventory.duplicate_event"); return InventoryWriteResult.duplicate(event.eventId()); } stockRepository.incrementStockOnHand(event.sku(), event.quantity()); bucketRepository.incrementBucket(event.sku(), "SELLABLE", event.quantity()); return InventoryWriteResult.accepted(event.eventId()); } In production, the accepted transaction insert and the aggregate updates should be part of the same database transaction. A useful shape is: PLSQL BEGIN; WITH accepted AS ( INSERT INTO inventory_transaction ( event_id, sku, seller_id, delta_quantity, event_time, accepted_at ) VALUES ( :event_id, :sku, :seller_id, :delta_quantity, :event_time, now() ) ON CONFLICT (event_id) DO NOTHING RETURNING sku, delta_quantity ) INSERT INTO inventory_stock_on_hand (sku, stock_on_hand, updated_at) SELECT sku, delta_quantity, now() FROM accepted ON CONFLICT (sku) DO UPDATE SET stock_on_hand = inventory_stock_on_hand.stock_on_hand + EXCLUDED.stock_on_hand, updated_at = now(); COMMIT; That ON CONFLICT clause is not just a database convenience. It is part of the replay contract. It ensures that retrying the same business event does not apply the same inventory delta twice. Debezium Configuration Enable PostgreSQL logical decoding and configure Debezium to emit CDC topics for the inventory tables. JSON { "name": "postgres-inventory-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "<POSTGRES_HOSTNAME>", "database.port": "5432", "database.user": "<POSTGRES_USER>", "database.password": "<POSTGRES_PASSWORD>", "database.dbname": "<POSTGRES_DBNAME>", "topic.prefix": "inventory_source", "plugin.name": "pgoutput", "slot.name": "debezium_inventory_slot", "publication.autocreate.mode": "filtered", "table.include.list": "public.inventory_stock_on_hand,public.inventory_bucket,public.inventory_transaction", "snapshot.mode": "initial", "heartbeat.interval.ms": "10000", "tombstones.on.delete": "false", "key.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "true", "value.converter.schemas.enable": "true" } } Debezium gives you history, but not recovery confidence. The confidence comes from how you key, project, replay, and reconcile that history. For replay work, track these connector facts in your runbook: Connector name and versionReplication slot namePublication name and included tablesSnapshot mode used for initial loadTopic prefixLast processed LSNConnector lagSchema history topic When a connector interruption happens, those details tell you whether you can resume normally, need a bounded replay, or need a new snapshot plus downstream reconciliation. Partition-Aware Routing The partition key should be chosen from the business ordering boundary. Java public class SkuPartitioner implements Partitioner { @Override public int partition( String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) { InventoryEvent event = (InventoryEvent) value; String orderingKey = event.getSku(); int partitionCount = cluster.partitionCountForTopic(topic); return Math.floorMod(orderingKey.hashCode(), partitionCount); } } Partitioning is not merely a throughput setting. If the projection depends on entity-local ordering, the entity belongs in the key. Kafka Streams Topology A simplified topology might rekey CDC records by SKU, materialize source tables, and compute availability. Java StreamsBuilder builder = new StreamsBuilder(); KTable<String, StockOnHand> stock = builder.table("inventory_source.public.inventory_stock_on_hand", Consumed.with(Serdes.String(), stockSerde)); KTable<String, InventoryBuckets> buckets = builder.table("inventory_source.public.inventory_bucket", Consumed.with(Serdes.String(), bucketSerde)); KTable<String, AvailabilityProjection> availability = stock.join( buckets, (stockRow, bucketRows) -> AvailabilityProjection.compute(stockRow, bucketRows), Materialized.<String, AvailabilityProjection, KeyValueStore<Bytes, byte[]>>as("availability-store") .withKeySerde(Serdes.String()) .withValueSerde(availabilitySerde) ); availability .toStream() .filter((sku, projection) -> projection.isPublishable()) .to("inventory.availability.v2", Produced.with(Serdes.String(), availabilitySerde)); The projection function should be deterministic. If replaying the same accepted history does not produce the same projection, the topology is not replay-safe. Recovery Contract Attach a Recovery Contract to the flow. YAML recovery_contract: flow: inventory-availability-projection tuple: "<H, O, I, F, S, Q, E>" history: source: - inventory_transaction - debezium.inventory_transaction order: key: sku idempotency: key: event_id duplicate_policy: skip_and_report function: name: compute_sellable_availability deterministic: true scope: supported: - by_sku - by_time_window - by_partition checks: - stock_on_hand_matches_transactions - sellable_quantity_non_negative - projection_event_time_valid evidence: - replay_scope - events_processed - duplicates_skipped - projections_changed - reconciliation_failures - confidence_status Treat this file as executable architecture documentation. A service should fail fast if the contract is incomplete for a critical flow. Java public final class RecoveryContractValidator { public void validate(RecoveryContract contract) { requireNonEmpty(contract.flow(), "flow"); requireNonEmpty(contract.history().source(), "history.source"); requireNonEmpty(contract.order().key(), "order.key"); requireNonEmpty(contract.idempotency().key(), "idempotency.key"); requireNonEmpty(contract.function().name(), "function.name"); requireTrue(contract.function().deterministic(), "projection must be deterministic"); requireNonEmpty(contract.scope().supported(), "scope.supported"); requireNonEmpty(contract.checks(), "checks"); requireNonEmpty(contract.evidence(), "evidence"); } private void requireNonEmpty(Object value, String field) { if (value == null || value.toString().isBlank()) { throw new IllegalArgumentException("Missing recovery contract field: " + field); } } private void requireTrue(boolean value, String message) { if (!value) { throw new IllegalArgumentException(message); } } } That validator does not make the system correct by itself. It prevents a more common failure: discovering during an incident that nobody defined the replay scope, idempotency key, or reconciliation checks. Replay Workflow Replay should be treated as a controlled workflow. Plain Text 1. Identify incident scope. 2. Select replay scope by SKU, time window, or partition. 3. Read authoritative history. 4. Rebuild deterministic projection. 5. Run reconciliation checks. 6. Emit recovery evidence. 7. Republish only if checks pass. The output should be an evidence report. JSON { "recovery_id": "rec-2026-06-19-001", "flow": "inventory-availability-projection", "events_processed": 1842, "duplicates_skipped": 17, "projection_rows_changed": 11, "reconciliation": { "stock_on_hand_matches_transactions": true, "sellable_quantity_non_negative": true, "projection_event_time_valid": true }, "confidence_status": "trusted" } A replay runner can keep the workflow explicit: Java public RecoveryEvidence replay(ReplayRequest request) { RecoveryContract contract = contracts.load(request.flow()); validator.validate(contract); ReplayScope scope = scopeResolver.resolve(request, contract); List<InventoryEvent> history = historyReader.read(contract.history(), scope); ReplayResult result = projector.rebuild(history, contract.function()); ReconciliationResult reconciliation = reconciliationRunner.run(contract.checks(), scope, result); RecoveryEvidence evidence = RecoveryEvidence.builder() .recoveryId(UUID.randomUUID().toString()) .flow(request.flow()) .scope(scope) .eventsProcessed(history.size()) .duplicatesSkipped(result.duplicatesSkipped()) .projectionsChanged(result.changedRows()) .reconciliation(reconciliation) .confidenceStatus(reconciliation.passed() ? "trusted" : "review_required") .build(); evidenceStore.write(evidence); if (request.publish() && reconciliation.passed()) { publisher.publish(result.projections()); } return evidence; } The replay runner should support dry runs. Dry runs let operators answer "What would change?" before republishing availability, billing, or detection outputs. Operational Metrics Track ordinary health and recovery confidence separately. Ordinary health: Consumer lagConnector lagTask restartsDLQ countEnd-to-end latency Recovery confidence: Replay durationReplay scope sizeDuplicate suppression countProjection rows changedReconciliation failuresConfidence status Example metric names: Plain Text inventory_ingest_events_total{result="accepted|duplicate|rejected"} inventory_cdc_connector_lag_seconds{connector="postgres-inventory-connector"} inventory_stream_projection_lag_seconds{topology="availability"} inventory_replay_duration_seconds{flow="inventory-availability-projection"} inventory_replay_events_processed_total{flow="inventory-availability-projection"} inventory_replay_duplicates_skipped_total{flow="inventory-availability-projection"} inventory_reconciliation_failures_total{check="stock_on_hand_matches_transactions"} inventory_recovery_confidence_status{status="trusted|review_required|failed"} Alert on disagreement, not only lag. A good pipeline can be caught up and still be wrong. YAML alerts: - name: InventoryProjectionReconciliationFailure expr: inventory_reconciliation_failures_total > 0 severity: page - name: InventoryReplayRequiresReview expr: inventory_recovery_confidence_status{status="review_required"} > 0 severity: ticket - name: InventoryConnectorLagHigh expr: inventory_cdc_connector_lag_seconds > 300 severity: ticket Reconciliation Queries Reconciliation should be executable, not just a diagram in a runbook. Start with invariants that are simple enough to automate. Example: Stock-on-hand should match accepted transaction deltas for a replay window. PLSQL WITH accepted_delta AS ( SELECT sku, SUM(delta_quantity) AS expected_delta FROM inventory_transaction WHERE accepted_at BETWEEN :from_time AND :to_time GROUP BY sku ), actual_delta AS ( SELECT sku, stock_on_hand - :baseline_stock_on_hand AS observed_delta FROM inventory_stock_on_hand WHERE sku = :sku ) SELECT a.sku, a.expected_delta, b.observed_delta, (a.expected_delta = b.observed_delta) AS matches FROM accepted_delta a JOIN actual_delta b ON a.sku = b.sku; Example: Sellable inventory should never be negative. PLSQL SELECT sku, location_id, quantity FROM inventory_bucket WHERE bucket_type = 'SELLABLE' AND quantity < 0; These queries are not academically exciting, but they are operationally powerful. They turn "the replay finished" into "the replay finished and the invariants passed." Replay Endpoint Sketch A replay workflow should be explicit and permissioned. One possible internal API: HTTP POST /internal/recovery/replay Content-Type: application/json { "flow": "inventory-availability-projection", "scope": { "type": "sku_and_time_window", "sku": "1231241", "from_event_time": "2026-06-19T18:00:00Z", "to_event_time": "2026-06-19T19:00:00Z" }, "dry_run": false, "requested_by": "sre-oncall", "reason": "projection drift after stream task restart" } The response should not just say 200 OK. JSON { "recovery_id": "rec-2026-06-19-001", "status": "trusted", "events_processed": 1842, "duplicates_skipped": 17, "projections_changed": 11, "reconciliation_failures": 0, "evidence_uri": "<RECOVERY_EVIDENCE_URI>" } The response is the operational artifact. It gives the team something to attach to an incident timeline and something to compare against later recovery runs. Tests for Replay Safety Replay safety should be tested before production incidents. Java @Test void replayingSameHistoryDoesNotChangeProjectionTwice() { List<InventoryEvent> history = List.of( event("evt-1", "SKU-1", 10), event("evt-2", "SKU-1", -2), event("evt-1", "SKU-1", 10) // duplicate ); AvailabilityProjection first = projector.replay(history); AvailabilityProjection second = projector.replay(history); assertThat(first).isEqualTo(second); assertThat(first.sellableQuantity()).isEqualTo(8); assertThat(first.duplicatesSkipped()).isEqualTo(1); } Also test late events, schema versions, partition rebalance, connector restart, and partial replay by entity. If replay is part of your recovery model, it deserves the same test discipline as the happy-path pipeline. Add failure injection tests that mirror production recovery: Java @Test void lateEventTriggersReviewWhenItChangesPublishedAvailability() { ReplayScope scope = ReplayScope.forSkuAndWindow( "SKU-1", Instant.parse("2026-06-19T18:00:00Z"), Instant.parse("2026-06-19T19:00:00Z") ); history.append(event("evt-1", "SKU-1", 10, "2026-06-19T18:01:00Z")); history.append(event("evt-2", "SKU-1", -3, "2026-06-19T18:59:00Z")); history.appendLate(event("evt-3", "SKU-1", -2, "2026-06-19T18:30:00Z")); RecoveryEvidence evidence = replayRunner.replay( ReplayRequest.dryRun("inventory-availability-projection", scope) ); assertThat(evidence.eventsProcessed()).isEqualTo(3); assertThat(evidence.projectionsChanged()).isGreaterThan(0); assertThat(evidence.confidenceStatus()).isEqualTo("review_required"); } Failure Injection Matrix Use a small matrix before every major release of the pipeline. Duplicate Event Injection: Send the same event_id twice.Expected evidence: duplicates_skipped > 0; no double-counted stock.Late Event Injection: Delay event arrival until after the projection has already published output.Expected evidence: late event count, changed projections, and review status if the output changes.Connector Pause Injection: Stop the Debezium connector for several minutes.Expected evidence: connector lag, replay scope, and reconciliation status.Offset Rewind Injection: Reprocess a known event range.Expected evidence: deterministic replay agreement.Schema Change Injection: Replay old and new schema versions.Expected evidence: schema versions recorded in the recovery evidence.Bad projection deploy Injection: Publish an incorrect derived state, then replay.Expected evidence: projections changed; reconciliation passes after rebuild. The point is not to create chaos for its own sake. The point is to practice the exact recovery motion before a real incident. Production Hardening Checklist Before relying on replay in production, confirm: The authoritative history has retention longer than the largest expected recovery window.The idempotency key is stable across producer retries.The Kafka partition key matches the business ordering boundary.The projection function is deterministic for the supported replay scope.The contract names every source topic, source table, check, and evidence field.The replay endpoint supports dry runs.Republish requires reconciliation success.Evidence is written to durable storage.Evidence records include schema versions and replay input bounds.Operators can find the runbook from the alert.The DLQ is treated as an input to recovery, not as the recovery plan itself. For high-value flows, make this checklist part of the architecture review. It is much cheaper to define replay semantics while designing the pipeline than to invent them under pressure. Common Mistakes Treating CDC topics as transient integration messages instead of durable recovery history.Choosing partition keys for infrastructure convenience rather than business ordering.Allowing stream processors to perform hidden non-idempotent side effects.Measuring lag but not correctness.Resetting offsets without a reconciliation plan.Assuming exactly-once semantics removes the need for recovery evidence. Conclusion Replay-safe CDC pipelines require more than Kafka, Debezium, and stream processing. They require explicit recovery semantics. Recovery Contracts give teams a compact way to define those semantics. Confidence-carrying replay gives operators evidence that the recovered state can be trusted. That is the difference between a pipeline that resumes and a platform that actually recovers.