Coding

The first warning sign wasn't an outage. It was a boring pull request. We changed one App Service setting. It was the sort of change that should have resulted in a small plan and a quick review. Instead, Terraform refreshed networking, private endpoints, DNS, Key Vaults, storage accounts, app services, and monitoring before showing what would actually change. Nothing was broken; that was the point. Terraform did exactly what it was designed to do: account for everything represented in state before calculating change. The problem was that our Terraform state had become a single, platform-sized boundary that every small change had to pass through, and one no team could fully own. If you have run a landing zone as a single Terraform configuration, you have probably had a version of that pull request. The instinct afterward is to blame size: the configuration has grown too large, so break it up. That instinct is wrong, or at least incomplete. Size is uncomfortable, but coupling is what actually hurts. Nothing in the change touched networking, DNS, or those key vaults. They were dragged into the plan because everything was bound together through one state. At first, that coupling just means slow plans and noisy reviews. Later, it raises a harder question: who actually owns this? Where the Coupling Shows Up Start with the plan. In a monolith, Terraform has to account for everything represented in the state before it can tell you what changed. You can target a single resource, but that is an escape hatch, not a way to run a platform. So the wait scales with the size of the estate, not your change. Both a one-line edit and a fifty-resource migration get stuck behind the same refresh before the diff appears. Provider upgrades show the same problem. A single root configuration pins one set of provider versions, so you cannot move networking to a newer azurerm version and leave everything else behind. Every upgrade becomes all-or-nothing, which means it keeps losing to smaller, safer priorities. Ours sat on azurerm 2.97 and only moved to the 4.x line once the upgrade could no longer be put off. The monolith had made the jump too big to schedule any sooner. The bigger concern is blast radius. One state file, one lock, one plan. A bad apply, a corrupted state, a destroy that catches more than you aimed at: whatever goes wrong can reach more of the platform than the change was ever meant to touch, because nothing in the layout is there to contain it. The dependency graph suffers too. Unrelated resources get sequenced together just because they share a graph. A network change might wait on unrelated compute, DNS on policy. The graph ends up reflecting accidental grouping rather than real dependencies. The result is clear. There is no small change. You cannot ship a DNS record or a new Key Vault without running the entire configuration through plan and apply. Every change is a platform change, carrying platform risk and requiring review, no matter how minor. These look like separate problems, but all come from the same design choice: too many unrelated concerns tied into one Terraform boundary. Where Coupling Becomes Ownership It is easy to call these operational annoyances: slow plans, awkward upgrades, risky applies, the tax you pay for a big configuration. But the same coupling appears in review and approval, where it stops being just an operational problem. Once too many concerns share the same state, pipeline, and approval path, the question is no longer only "how long did the plan take?" It becomes "who is accountable for the boundary this change is crossing?" Take private connectivity. A single private endpoint on Azure isn't handled by just one team. The application team owns the service behind it. The platform team manages the landing zone, subnet, and endpoint placement. Private DNS zones might be managed centrally or by another team. Security or governance may require the service to be private. How these map to teams varies, but in a monolith, everything ends up in the same state, pipeline, and plan. So "who owns this?" rarely has a clear answer. However you split teams, they are coupled through a single configuration that none can truly own. When the application team changes its service, the same config still carries platform connectivity and governance controls. You cannot draw ownership along your real organizational boundaries, because the code does not have them. Both slow plans and unclear ownership trace back to the same issue: shared concerns treated as if they belong to just one team. Figure 1: When Terraform boundaries stop matching ownership boundaries. The monolith gives Terraform one boundary. Organizations have several. The pain comes when small changes have to cross boundaries that no team fully owns. Reach for the Coupling, Not the Size The reflex now is to split the state and move on. But splitting a landing zone poorly can be worse than leaving it alone. If you split along the wrong lines, you trade one blast radius for tangled cross-state dependencies. You also lose the single plan that at least showed the whole graph in one place. For example, splitting private endpoints into one state and private DNS zones into another may look clean on paper. But if different teams deploy them without a clear agreement, every new endpoint becomes a coordination headache, not a smaller change. Moving files into separate folders does nothing if the same pipeline, credentials, and approval path still govern everything. Decomposition should follow actual coupling, not just line count. So the next question is not "how many states should we create?" It is "which boundaries are real enough for teams to own, deploy, and recover independently?" If your Terraform monolith hurts, do not start by counting files or resources. Look at what is actually being coupled. Slow plans and unclear ownership are both signs that your Terraform boundaries no longer match your real ownership boundaries.

Enterprise applications commonly face multiple data challenges. Some data requires transactional integrity and relationships, while other data prioritizes fast, predictable access. Sessions, counters, rate limits, temporary state, often-accessed objects, and coordination data may not benefit from the complexity of a relational model. In these cases, a key-value database's simplicity becomes an architectural advantage. This simplicity is especially valuable in distributed and cloud-native systems, where latency, throughput, plus scalability directly shape user experience and infrastructure costs. A key-value database offers a focused approach: identify data by a key and retrieve or update it efficiently. The challenge is selecting a technology that delivers this performance while meeting the operational maturity, ecosystem support, and governance standards required for enterprise applications. Valkey meets these needs successfully. Originating from the Redis OSS lineage and developed as a vendor-neutral open-source project under the Linux Foundation, Valkey delivers a high-performance key-value platform suitable for caching, application state, messaging, and primary data storage. Beyond being another database option, it lets organizations explore how key-value persistence fits into modern enterprise architecture and lets Java applications use its benefits without tightly coupling to a specific datastore. Why Key-Value Databases Matter Key-value databases use a simple data model in which each unique key identifies a value. This simplicity is effective when applications can directly locate the required data. By enabling direct reads and writes, key-value databases typically deliver low latency, high throughput, and a horizontally scalable operational model. In enterprise systems, this model suits scenarios such as distributed sessions, caching, counters, rate limiting, feature flags, shopping carts, temporary workflow state, idempotency keys, leaderboards, and frequently accessed application data. These workloads prioritize fast access by identifier over joins, ad hoc queries, or complex relational constraints. The main architectural advantage of key-value databases is their specialization for specific access patterns, rather than universal speed or simplicity. When the primary requirement is to retrieve the current value for a given key, adding a more complex persistence model can introduce unnecessary overhead. As part of a polyglot persistence strategy, key-value stores enable architects to align the database model with the workload, rather than forcing all workloads into a single database. Putting Valkey Into Practice With Jakarta NoSQL A key advantage of using Valkey in enterprise Java is that it does not require a new programming model. With Jakarta NoSQL and Eclipse JNoSQL, Valkey serves as another key-value implementation behind a consistent API and mapping model. Domain annotations remain unchanged, so switching between key-value databases usually involves only updating the driver and its configuration, not rewriting the application. This abstraction is valuable architecturally. The application relies on the Jakarta NoSQL contract, while Eclipse JNoSQL manages integration with the database. Although database-specific features may introduce some coupling, applications that use the portable API can switch key-value implementations with minimal impact. For this article, we will use a simple Java SE example. This persistence layer can later support a REST API, messaging consumer, scheduled process, or other enterprise architecture without altering the core database interaction. Starting Valkey The first step is to make a Valkey instance available. Docker provides a convenient way to start one locally: Shell docker run --name valkey-instance \ -p 6379:6379 \ -d valkey/valkey:latest With Valkey running, add the Eclipse JNoSQL Valkey driver to the Jakarta NoSQL infrastructure, which includes CDI, Eclipse MicroProfile Config, and Jakarta JSON Processing. XML <dependency> <groupId>org.eclipse.jnosql.databases</groupId> <artifactId>jnosql-valkey</artifactId> <version>${jnosql.version}</version> </dependency> Configure the connection externally: Properties files jnosql.keyvalue.database=developers jnosql.valkey.port=6379 jnosql.valkey.host=localhost Since Eclipse JNoSQL integrates with Eclipse MicroProfile Config, you do not need to hard-code these values. They can be provided through configuration sources such as environment variables, in line with the Twelve-Factor App methodology. Mapping an Entity The mapping model for a key-value database is intentionally simple. Identify the class as an entity and specify the field that represents its key: Java @Entity public class User { @Id private String userName; private String name; private List<String> phones; // constructors, getters, setters... } Importantly, @Entity and @Id are part of the mapping abstraction, not Valkey itself. The domain model does not require Valkey-specific annotations. Using Jakarta NoSQL Eclipse JNoSQL provides KeyValueTemplate, a specialization of the Jakarta NoSQL Template API for key-value databases. This allows direct persistence and retrieval of entities: Java User user = User.builder() .phones(Arrays.asList("234", "432")) .username("username") .name("Name") .build(); KeyValueTemplate template = container.select(KeyValueTemplate.class).get(); User userSaved = template.put(user); System.out.println("User saved: " + userSaved); Optional<User> userFound = template.get("username", User.class); System.out.println("Entity found: " + userFound); For applications that prefer a repository abstraction, Eclipse JNoSQL integrates with Jakarta Data: Java @Repository public interface UserRepository extends CrudRepository<User, String> { } This approach allows the application code to focus more directly on domain operations: Java User user = User.builder() .phones(Arrays.asList("234", "432")) .username("username") .name("Name") .build(); UserRepository repository = container .select( UserRepository.class, DatabaseQualifier.ofKeyValue() ) .get(); repository.save(user); Optional<User> userFound = repository.findById("username"); System.out.println("User found: " + userFound); Notably, this code includes no Valkey-specific API in the entity or repository. Valkey is an infrastructure choice, while Jakarta NoSQL and Jakarta Data remain the application-facing abstractions. This separation guarantees the architecture remains reusable if the underlying key-value technology changes. Conclusion Key-value databases are highly effective for workloads that require direct access, low latency, and high throughput, rather than complex queries or relational navigation. This article examined how this model fits within enterprise architecture and how Valkey can integrate via Eclipse JNoSQL, allowing applications to avoid direct dependencies on vendor-specific APIs. By maintaining consistent entity mapping and using Jakarta NoSQL or Jakarta Data abstractions, switching key-value implementations becomes mainly a matter of infrastructure and configuration. This shift reflects a broader evolution in enterprise Java, as the platform expands its persistence capabilities beyond traditional relational databases. With Jakarta Persistence, Jakarta Data, Jakarta NoSQL, and tools like Eclipse JNoSQL, architects can choose the best data model for each workload while keeping familiar programming abstractions. Valkey enhances this ecosystem by providing a robust key-value option, making polyglot persistence both feasible and practical.

By Otavio Santana CORE

A mobile request can fail without the server-side work failing. An iOS app may time out, lose the response after a POST has reached the service, or retry after connectivity changes while the original execution is still progressing. Apple explicitly distinguishes safe retry behavior by HTTP method and notes that URLSession can retry requests in some connection-loss cases, waitsForConnectivity can also cause the system to continue a request when connectivity returns. The dangerous state is therefore not “request failed,” but “completion is unknown.” If that request starts an agent that charges an account, reserves inventory, sends a message, or invokes an MCP tool, a second submission can become a second side effect. The Retry Boundary Is the Real Transaction Boundary “Exactly once” is too strong for a workflow crossing an iPhone, HTTP, an agent runtime, an MCP server, Kafka, a database, and an external API. Kafka can provide exactly-once guarantees within defined Kafka processing boundaries, but those guarantees do not atomically include arbitrary remote tool effects. The practical target is effectively-once behavior, and retries are expected, but every effect is guarded by a stable operation identity and converges on one committed outcome. Kafka’s idempotent producer suppresses duplicate records caused by producer retries, while transactional producers can atomically publish across Kafka partitions; the producer documentation also limits idempotence guarantees to a producer session and requires read_committed consumers for end-to-end transactional visibility. The operation identity must exist before the first network attempt. An iOS client can create an operationId when an action becomes durable local intent, persist it, and reuse it across transport retries. Transport material such as a server challenge may change, but the business ID must not. The server treats (subjectId, operationId) as a uniqueness boundary and stores a canonical payload hash with it. PostgreSQL unique constraints enforce row uniqueness, while INSERT ... ON CONFLICT provides an atomic conflict path under concurrency. SQL INSERT INTO agent_operation(subject_id, operation_id, payload_hash, status) VALUES (:subject, :operationId, :payloadHash, 'ACCEPTED') ON CONFLICT (subject_id, operation_id) DO NOTHING; A conflict with the same payload hash returns the existing operation; a different hash rejects key reuse. The record should exist before agent execution starts, and the accepted response should expose the durable operation identity. Let LangGraph Resume Without Repeating Effects LangGraph persistence is useful precisely because durable execution can replay code. With a checkpointer, LangGraph saves state at super-step boundaries; if execution resumes after a failure, an affected node can run again from the beginning. Official guidance consequently requires idempotent node logic, and task results can be checkpointed so completed task work can be reused during resume instead of recomputed. Replaying from an earlier checkpoint can also re-trigger later LLM calls and API requests. A stable business operation should therefore map to a stable LangGraph thread, while every effectful tool boundary receives the same operation ID. Python config = {"configurable": {"thread_id": operation_id} result = graph.invoke( {"operation_id": operation_id, "command": command}, config ) Checkpointing reduces recomputation but does not replace downstream idempotency. A reservation can succeed before its task result is durably checkpointed. LangGraph’s functional API therefore recommends idempotent tasks because an incomplete task can execute again during resume. Python @task def reserve_inventory(operation_id, sku, quantity): return mcp.call_tool("reserve_inventory", { "operationId": operation_id, "sku": sku, "quantity": quantity }) The significant property in this snippet is not the decorator. The important part is that the business identity crosses the graph boundary and reaches the tool implementation. A downstream inventory service can then use that identity to return a previously committed reservation rather than creating another one. MCP Tasks Are Durable Handles, Not Deduplication Keys The current MCP Tasks design is especially relevant to long-running agent tools. In the July 28, 2026 protocol revision, Tasks moved into the io.modelcontextprotocol/tasks extension. A server can return a durable task handle, and the client can poll with tasks/get, provide input with tasks/update, or request cancellation with tasks/cancel. The task is durably created before its handle is returned, which allows polling after a disconnect. That durability solves result retrieval after task creation, but it does not by itself deduplicate the request that creates the task. The task ID is server-generated. If the server creates task A, the response disappears, and the original tools/call is sent again, a naïve implementation can create task B. Therefore, the business operationId must be part of the tool arguments or equivalent application metadata, and task creation must first look up an existing operation. This follows directly from MCP’s server-generated task-ID model combined with retry ambiguity at the HTTP boundary. The MCP server can return an existing task handle for the same authenticated subject, operation ID, and payload hash, and later return the stored terminal result. Cancellation should also be idempotent because MCP defines it as cooperative rather than a guarantee that underlying work stops immediately. Keep Kafka Guarantees Inside Kafka Kafka is most valuable after the operation has been claimed. A database transaction can persist operation state with an outbox row carrying the same ID. Kafka producer idempotence protects against duplicates caused by producer retries, while consumers can still use the operation ID for application-level deduplication. Kafka transactions can atomically cover Kafka writes, but they do not extend over an MCP server or payment API. The event contract should preserve causality rather than inventing a new identity at each hop. JSON { "operationId": "8E7B6D9E-...", "type": "AgentToolCompleted", "tool": "reserve_inventory", "status": "SUCCEEDED" } A consumer can enforce uniqueness on (consumerName, operationId, eventType) or make the state transition conditional. Kafka delivery guarantees and application idempotency then reinforce each other instead of being treated as interchangeable. Bind Retry Identity to App Attest Without Blocking Legitimate Retries App Attest addresses a different failure mode: whether a request comes from a legitimate app instance and whether signed request material has been replayed or altered. Apple’s current guidance uses a server-provided challenge for assertions and requires the server to validate a strictly increasing assertion counter; that counter is specifically an anti-replay signal. Assertions are generated locally on the device after key attestation. The App Attest assertion must not become the business idempotency token. A legitimate retry should obtain fresh challenge material and generate a fresh assertion while retaining the original operation ID. The data hashed for the assertion can bind the server challenge, operation ID, and canonical payload hash together. Swift let payloadHash = SHA256.hash(data: body) let clientData = challenge + operationID.data + Data(payloadHash) let clientDataHash = Data(SHA256.hash(data: clientData)) let assertion = try await service.generateAssertion( keyID, clientDataHash: clientDataHash ) Apple recommends server-controlled challenges, server-side validation, and assertion-counter tracking as assertions are generated on demand without a round trip to Apple’s servers. The server verifies App Attest, checks that the challenge binds the operation ID and payload, then performs the idempotency lookup. A fresh assertion can retry the same operation; a replayed assertion fails anti-replay validation; an altered payload fails the hash check. Effectively-Once Behavior Is a Composition Property Reliable agent execution does not come from asking iOS to retry less often or from labeling a Kafka pipeline “exactly once.” It comes from carrying one durable business identity across every retry and every boundary, claiming that identity atomically before execution, making LangGraph effects idempotent under resume, using MCP Tasks as durable result handles rather than creation-time deduplication keys, restricting Kafka’s exactly-once guarantees to Kafka’s transactional domain, and using App Attest to prove request integrity without confusing anti-replay state with business deduplication. When those boundaries align, a lost mobile response can cause another HTTP attempt, another graph invocation, or another poll, but it does not cause another business effect. That is the operational meaning of effectively once.

By Uthej Mopathi CORE

Every major AI vendor now supports the Model Context Protocol. The framing is almost always the same: MCP is the universal connector for AI agents in the enterprise. That framing sets up a false choice. MCP, REST/HTTP APIs, and Apache Kafka are not alternatives. They solve different problems at different layers of the architecture. Treating them as competing options produces systems that are fragile exactly where they need to be reliable. These three technologies can and do coexist in the same architecture. The question is not which one to pick. It is which one belongs where, and what the tradeoffs are when more than one could technically do the job. This article maps that decision: what each technology is built for, where the boundaries are, and where the genuine gray areas lie. 1. What Is MCP and What Is It Built For? Anthropic introduced the Model Context Protocol in November 2024 as an open standard for connecting AI assistants to external tools and data sources. Before MCP, every AI model required a custom connector to each external system. Three models, ten systems: thirty custom integrations to build and maintain. MCP collapses that to one standard interface. Any compliant client talks to any compliant server without prior coordination. OpenAI adopted MCP in March 2025. Google DeepMind confirmed support in April 2025. By December 2025, MCP had reached over 97 million monthly SDK downloads across Python, TypeScript, Java, Kotlin, C#, and Swift. Anthropic donated the protocol to the Agentic AI Foundation under the Linux Foundation, with AWS, Google, Microsoft, Bloomberg, and OpenAI as platinum members. MCP is no longer a developer experiment. Signals of enterprise maturity are arriving quickly: AI agents paying for API access autonomously, cross-SDK interoperability between Anthropic and OpenAI converging on MCP Resources, composable enterprise workflows where agents read tool signatures and compose cross-system flows without predefined paths, and an official MCP Registry launched in late 2025 as the community-driven server directory. The 2026 roadmap focuses on scalable transport, agent-to-agent communication, governance maturation, and enterprise readiness covering audit trails and SSO-integrated authentication. MCP handles tool access: how an agent calls an external capability. It does not handle agent-to-agent coordination, which is the domain of protocols like Google's Agent-to-Agent (A2A). MCP and A2A are complementary and address different layers of agentic architecture. The moment MCP is asked to do more than tool access, the architecture starts to break. Security Maturity Is Still Catching Up With Adoption Most incidents disclosed in 2025 and early 2026 are implementation failures, not protocol flaws. An Endor Labs analysis of 2,614 MCP implementations found 82% use file system operations prone to path traversal and 67% use APIs related to code injection. Enterprise-grade authentication with OAuth 2.1 and SAML/OIDC is on the 2026 roadmap but still in progress. The practical controls for today: apply least privilege, limit MCP server access to only the systems and data each tool requires, and monitor tool definitions for unexpected changes. 2. MCP vs. REST/HTTP API MCP and REST/HTTP APIs serve different consumers and should not be treated as interchangeable. REST is an architectural style built on HTTP, widely adopted but with no fixed conventions for discovery, error formats, or method naming. Well-designed REST APIs backed by OpenAPI specifications work well for direct, programmatic data access when a native SDK or versioned API already exists and teams know how to operate it. MCP enforces consistency at the interface level because the consumer is an AI model that cannot tolerate creative API interpretation. MCP standardizes how a tool is called. It does not standardize what the tool returns, how fresh that data is, or whether two agents calling the same tool simultaneously see the same state. For direct data access to vector stores, databases, or business application APIs, a well-governed REST API, native SDK, or Kafka Connect integration is almost always the better choice: lower latency, no protocol overhead, mature tooling. For giving AI agents standardized, discoverable access to a broader set of tools across vendors and frameworks, MCP is the right layer. The two are complementary, not competing. Tool Design Matters as Much as the Protocol Choice One important nuance on tool design: mapping one-to-one from existing APIs to MCP tools rarely works well. What matters is tool granularity, smart metadata, and thoughtful assembly of the MCP layer. An MCP server that exposes well-structured, semantically rich tools lets an AI agent reason about capabilities and compose workflows. This is reminiscent of the composability questions from the enterprise SOA (Service-oriented Architecture) era. SOA promised flexible service composition but delivered integration chaos when governance, metadata quality, and service granularity were treated as afterthoughts. MCP faces the same risk. The protocol is sound; what determines success is the discipline applied to how tools are defined, documented, and assembled. What MCP Does Not Do What MCP does not do matters as much as what it does. It does not manage data, guarantee message delivery, enforce governance, or guarantee consistency across systems. It is an interface layer, not a data pipeline. That boundary becomes even clearer when looking at what Kafka does, which is structurally different from both MCP and REST. 3. Apache Kafka: Event Broker, Decoupling, and the Backbone Role Operational data is the live data that runs business processes: order states, inventory levels, transaction records, customer accounts, risk scores. It originates in systems like SAP, Salesforce, Oracle, and mainframes, and it changes continuously. Kafka is architecturally different from both HTTP and MCP in one way that matters most: it decouples producers and consumers through a persistent, ordered, append-only log. With HTTP or MCP, the caller and the callee are coupled at request time. Every integration is point-to-point. If the target system is slow or unavailable, the caller is directly affected. Kafka breaks that coupling entirely. A producer writes an event once. Any number of consumers read it independently, at their own pace, using their own communication paradigm. One consumer processes records in real time. Another runs nightly batch analytics over the same events. A third powers a stream processing pipeline. A fourth writes results to a data lake via Apache Iceberg. All of them consume the same underlying data product. None of them affects the others. Kafka supports three consumption patterns from a single event stream: streaming, request-response, and batch. The event exists once; each consumer is independent. This is the pub/sub event broker model, and it is what makes Kafka the integration backbone between operational and analytical systems. The diagram below shows this decoupling: a single Kafka topic serving real-time applications, HTTP-based consumers, batch analytics, and MCP agent interfaces simultaneously. Stream Processing With Kafka Streams and Apache Flink Stream processing is a core complement to Apache Kafka, extending the platform from event transport into real-time data processing and decisioning. Kafka Streams is a lightweight Java library embedded in applications. It is well-suited for streaming ETL and simple to medium stateful stream processing without requiring a separate cluster. It integrates closely with existing JVM-based services. Apache Flink is a distributed stream processing engine designed for more complex workloads. It supports Java, Python, and SQL APIs, making it accessible to both application developers and data engineers. Flink runs as a dedicated cluster or in managed environments and is built for high-scale scenarios such as multi-stream joins, event-time processing, large state management, exactly-once semantics, Complex Event Processing (CEP), real-time analytics, and AI model inference. Both approaches extend Kafka with processing capabilities. The choice depends on workload complexity, required deployment model, and preferred programming language, not on replacing Kafka’s role as the event streaming backbone. A detailed comparison is available in the post Apache Kafka and Apache Flink: A Match Made in Heaven. Operational and Analytical Integration, Including the Data Lakehouse Kafka is not only for operational data integration. It serves as the ingestion layer into data lakes, feeds real-time analytical pipelines, enables stream processing with embedded AI models, and connects business applications bidirectionally. A governed data streaming platform provides schema registry, lineage tracking, role-based access control, and exactly-once delivery semantics across all of that. It serves both operational and analytical use cases and acts as the bridge between those two worlds. For how streaming and the lakehouse converge via Apache Iceberg, see Data Streaming Meets Lakehouse. Kafka's append-only commit log is the foundation of data consistency across the enterprise. Every downstream consumer sees the same data in the same order. That is not just a performance feature. It is what prevents the architecture where every system has its own version of the truth. 4. The Tradeoffs: It Is Not Black and White The choice between MCP, REST/HTTP APIs, and Kafka is rarely clean. All three can play a role in the same architecture. REST/HTTP APIs work well for operational data access when volume is moderate and a well-governed API already exists. A REST API backed by a Kafka-derived serving layer can return consistent, current data. The API is the interface; the streaming platform is what makes the data trustworthy behind it. A financial services firm exposing account balances via REST is not doing it wrong, as long as those balances are derived from a governed, consistent data source rather than pulled directly from a source system on every request. Kafka becomes the clear choice when data is high-volume or high-velocity, when multiple consumers need the same events, when ordering and exactly-once delivery matter, or when the same events need to feed operational applications, analytical pipelines, and AI agents simultaneously. MCP fits best when access is supplementary, loosely coupled, and low-frequency. A support agent looking up a ServiceNow ticket before drafting a response, or a sales assistant pulling the latest slide deck from Google Drive before a call, are good fits. The key test is simple: does it matter if the data the agent receives is a few seconds or minutes old? If yes, MCP should not own that responsibility. If no, MCP is the right interface. SAP: Clean Separation Between ERP Integration and Developer Tooling The boundary between MCP and REST is not a choice between two equivalent options for the same integration. SAP is the clearest example of a clean separation. SAP exposes extensive REST and OData APIs for ERP integration: order management, finance, supply chain, procurement, and HR data flowing bidirectionally between SAP and other enterprise systems. SAP's MCP servers serve an entirely different purpose: developer tooling for ABAP code generation, CAP application development, UI5 and Fiori assistance, and operational tasks like transport validation and incident management. An architect connecting SAP order events to downstream systems uses OData and Kafka Connect. A developer asking an AI coding assistant to generate ABAP code uses the SAP MCP server. Different consumers, different use cases, different data. No overlap. Salesforce and ServiceNow: Same Data, Different Consumer Salesforce and ServiceNow follow a different pattern. Their MCP servers wrap the same underlying REST APIs and expose the same underlying data, but for a different consumer. A developer-written integration calls the Salesforce REST API directly with known endpoints and hardcoded logic. An AI agent calls the Salesforce MCP server, which wraps that same API to make it discoverable and stateful for an agent that cannot read documentation or manage its own session state. The data is identical. The access path differs based on who is consuming it. This is not a free choice between equivalent options. It is the same system serving two different client types through two different interface layers. REST vs. Kafka for Operational Data: The Harder Call The harder boundary is between REST and Kafka for operational data. Both can technically serve it, and that is where the real architectural decision lies. REST is simpler to start with but introduces point-to-point coupling, integration spaghetti at scale, and consistency risks when the same data needs to reach multiple consumers. Kafka is more complex to operate but provides the decoupling, consistency, and governance that enterprise architectures require when the same data needs to reach many consumers reliably. The two are not mutually exclusive. A common and well-proven pattern combines both: Kafka handles the event backbone, decoupling, and consistency, while a REST layer sits on top for synchronous request-response access, API management integration, or compatibility with systems that cannot speak the native Kafka protocol. This is particularly common in mobile applications, legacy system integration, and API gateway architectures. For a detailed look at how REST and Kafka complement each other in practice, see Request-Response with REST/HTTP vs. Data Streaming with Apache Kafka. 5. Decision Framework: MCP, REST/HTTP, or Kafka? Choosing between MCP, REST/HTTP, and Kafka is not a single decision but a set of tradeoffs that depend on data volume, consumer type, consistency requirements, and what is already in production. The comparison table below makes those tradeoffs concrete across eight dimensions. When to Use Which: A Guide to the Decision Tree The decision tree below walks through the same logic as a series of questions, routing to the right choice based on the integration's actual requirements. Use MCP when the integration is supplementary and tool-like: Slack, Google Drive, ServiceNow tickets, internal knowledge bases. The agent needs context to act, not a stream of events to react to. Eventual consistency is acceptable. Apply least privilege, monitor tool definitions for changes, and isolate MCP servers from production systems. Use a REST/HTTP API or native SDK when a well-documented API or SDK already exists and the engineering team knows how to operate it. The access pattern is direct, moderate-volume, and latency-sensitive. REST is also a reasonable choice for operational data when the backend is a governed Kafka-derived serving layer and consistency properties are inherited, not assumed. Use Apache Kafka when data is high-volume or high-velocity, when multiple consumers need the same events, when ordering and exactly-once delivery matter, or when governance, lineage, and auditability are non-negotiable. Kafka is also the right choice when the same data needs to feed operational applications, real-time analytics, data lakes, and AI agents simultaneously. Use the real-time context engine when an AI agent needs current, consistent operational context for autonomous decisions. Kafka and Flink govern the data. MCP provides the agent interface. The consistency guarantee comes from the streaming layer, not from MCP. The practical question is not which protocol to choose. It is whether the data architecture underneath the agents can be trusted. Agents making autonomous decisions about inventory, risk, or customer service are only as reliable as the data they act on. 6. Where MCP and Kafka Work Together: The Real-Time Context Engine There is one pattern where MCP and data streaming complement each other directly: the real-time context engine. Kafka and Flink process and govern the data: ingesting from operational systems, applying transformations and filters, producing real-time materialized views. Those views are then exposed to AI agents through a standardized MCP interface. The streaming platform owns the data, its freshness, and its consistency guarantees. MCP owns the interface to the agent. Neither layer bleeds into the other's responsibility. Data consistency is not delegated to MCP. The streaming platform enforces it upstream before the MCP interface comes into play. The agent calls a tool and receives context that is current, governed, and consistent, not because MCP guarantees it, but because the streaming platform does. Any compliant AI agent, whether Claude, ChatGPT, Amazon Bedrock, LlamaIndex, or CrewAI, can call the context engine and receive current context from operational systems without needing to understand Kafka topics, Flink jobs, or schema evolution. An agent routing shipments from yesterday's inventory, approving transactions against a risk score from three hours ago, or reading an account balance that has not propagated: none of these is reliable. A real-time context engine eliminates this class of error at the source, reduces hallucinations, lowers inference cost, and anchors decisions to current operational reality. From Data Freshness to Agent Governance Enterprise readiness for this pattern also depends on how agents are governed once deployed. Trust, control, and accountability become central once agents start chaining decisions across domains. The context engine is the data layer of that answer. Governance of the agents themselves, covering what they are permitted to do, under what conditions, and with what audit trail, is the other half. This is the dimension enterprise buyers are actively evaluating when selecting agent orchestration platforms. The diagram below shows how the three layers fit together: the streaming platform as the data backbone, the context engine as the governed serving layer, and MCP as the clean interface to agents. 7. Conclusion: One Protocol, One Job MCP has earned its place in the enterprise architecture stack. What it has not yet earned is the role of universal integration layer, and understanding that distinction is what this article has been about. The broader architecture this sits inside connects three interdependent pillars. Event-driven data integration, with Kafka as the backbone, moves data reliably between operational and analytical systems and delivers governed data products to every consumer. Process intelligence is the orchestration layer that determines which decisions to automate, in what sequence, and under what conditions, giving agentic workflows the structure and governance they need to be trustworthy. Trusted agentic AI is where MCP plays its role: the standardized, governed interface through which agents access external tools and context, anchored to real data by the streaming layer beneath it. For a vendor-by-vendor analysis of trust and lock-in across the major AI platforms, see the Enterprise Agentic AI Landscape 2026. For a deeper look at how the three pillars fit together as an enterprise architecture framework, see The Trinity of Modern Data Architecture: Process Intelligence, Event-Driven Integration, and Trusted Agentic AI. One protocol, one job. That is the right way to use MCP.

By Kai Wähner CORE

Oracle Database 23ai introduced the powerful DBMS_DEVELOPER package, giving developers and database administrators a streamlined way to access database object metadata in JSON format. This feature represents a significant advancement in how we interact with database schemas, offering a more structured and programmatic way to extract and analyze metadata compared to traditional dictionary views or the older DBMS_METADATA package. In this article, we'll explore the capabilities of DBMS_DEVELOPER, focusing on its GET_METADATA function through detailed examples and practical implementation scenarios. Understanding DBMS_DEVELOPER The DBMS_DEVELOPER package was designed specifically for modern application development patterns, where JSON has become a universal data exchange format. Rather than returning metadata as DDL statements (like DBMS_METADATA), this package returns structured JSON documents that can be easily parsed, processed, and integrated into applications or DevOps workflows. Key Benefits Structured data format: Returns metadata as JSON objects that can be easily parsed Programmatic access: Perfect for integration with applications and automation scripts Versioning capabilities: Built-in ETag mechanism for tracking object changesConfigurable detail levels: Ability to retrieve basic, typical, or comprehensive metadata Setting Up Our Environment Let's set up a sample schema to demonstrate the package functionality: SQL CREATE TABLE customers ( customer_id NUMBER(10) CONSTRAINT pk_customers PRIMARY KEY, first_name VARCHAR2(50) NOT NULL, last_name VARCHAR2(50) NOT NULL, email VARCHAR2(100) CONSTRAINT uk_customer_email UNIQUE, join_date DATE DEFAULT SYSDATE, status VARCHAR2(10) DEFAULT 'ACTIVE' ); CREATE INDEX idx_customer_name ON customers(last_name, first_name); CREATE OR REPLACE VIEW active_customers AS SELECT customer_id, first_name, last_name, email FROM customers WHERE status = 'ACTIVE'; GET_METADATA Basics The core function of the DBMS_DEVELOPER package is GET_METADATA, which returns metadata about database objects in JSON format. Let's start with a basic example: SQL -- Using JSON_SERIALIZE for formatted output SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA(name => 'CUSTOMERS') PRETTY) AS metadata; The result is a structured JSON document containing comprehensive information about the table, including: Table name and schema Column definitions with data types and constraints Primary key, unique key, and foreign key information Index definitions An etag value representing the current state of the object This structured format makes it significantly easier to extract specific information programmatically compared to parsing DDL statements. NAME and SCHEMA Parameters The NAME and SCHEMA parameters work together to identify the specific database object. These parameters are case-sensitive and must match the object definition in the data dictionary. SQL -- Explicitly specifying schema SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'CUSTOMERS', schema => 'FINANCE') PRETTY) AS metadata; -- Using current schema (implicit) SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA(name => 'CUSTOMERS') PRETTY) AS metadata; When the SCHEMA parameter is omitted, the function uses the current schema. This behavior provides flexibility when working with objects across different schemas in your database environment. OBJECT_TYPE Parameter The OBJECT_TYPE parameter allows you to explicitly specify the type of object you're retrieving metadata for. While often optional (as the database can infer the object type from the name), it becomes necessary in cases where name resolution alone is insufficient. Currently, `DBMS_DEVELOPER` supports three object types: TABLEINDEXVIEW Let's examine metadata for our index and view: SQL -- Retrieving index metadata SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'IDX_CUSTOMER_NAME', object_type => 'INDEX') PRETTY) AS metadata; -- Retrieving view metadata SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'ACTIVE_CUSTOMERS', object_type => 'VIEW') PRETTY) AS metadata; The OBJECT_TYPE parameter becomes particularly important when dealing with objects that share the same name but have different types, such as packages and package bodies. LEVEL Parameter The LEVEL parameter controls the amount of detail included in the JSON output. Oracle provides three levels: BASIC: Minimal informationTYPICAL: Standard level of detail (default)ALL: Comprehensive metadata This flexibility lets you balance concise output with detailed information based on your needs. SQL -- Basic level metadata SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'IDX_CUSTOMER_NAME', level => 'BASIC') PRETTY) AS metadata; -- All details SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'IDX_CUSTOMER_NAME', level => 'ALL') PRETTY) AS metadata; The output at the ALL level includes additional attributes such as segment information, compression settings, and physical storage details that aren't present at the BASIC level. ETAG Parameter One of the most powerful features of DBMS_DEVELOPER is the etag mechanism, which provides version tracking for database objects. The etag value changes whenever the object definition changes, making it invaluable for change detection. SQL -- Store the current etag value DECLARE v_metadata CLOB; v_etag VARCHAR2(100); BEGIN v_metadata := DBMS_DEVELOPER.GET_METADATA(name => 'ACTIVE_CUSTOMERS'); SELECT JSON_VALUE(v_metadata, '$.etag') INTO v_etag FROM dual; DBMS_OUTPUT.PUT_LINE('Current etag: ' || v_etag); END; / -- Modify the view CREATE OR REPLACE VIEW active_customers AS SELECT customer_id, first_name, last_name, email, join_date FROM customers WHERE status = 'ACTIVE'; -- Check if the object has changed using the stored etag SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'ACTIVE_CUSTOMERS', etag => 'A1B2C3D4E5F6G7H8I9J0') -- Previous etag value PRETTY) AS metadata; When you pass an ETag value that matches the current state of the object, the function returns an empty JSON document {}. If the object has changed, it returns the complete metadata with a new ETag value. Practical Scenario: Database Migration and Documentation Let's consider a practical scenario where DBMS_DEVELOPER proves invaluable: a large-scale database migration project with continuous schema changes. The Challenge You're leading a project to migrate a critical application database from on-premises to Oracle Cloud. The development team continues to make schema changes during the migration process, and you need to: Document the current state of all database objectsTrack changes between migration wavesValidate that objects were created correctly in the target environmentGenerate comprehensive documentation for compliance requirements The Solution Using DBMS_DEVELOPER, you can create a robust metadata management system: SQL CREATE TABLE schema_versions ( object_name VARCHAR2(128), object_type VARCHAR2(30), object_schema VARCHAR2(128), capture_date TIMESTAMP, etag VARCHAR2(100), metadata CLOB ); -- Procedure to capture all tables in a schema CREATE OR REPLACE PROCEDURE capture_schema_metadata(p_schema VARCHAR2) AS v_metadata CLOB; v_etag VARCHAR2(100); CURSOR c_objects IS SELECT object_name, object_type FROM all_objects WHERE owner = p_schema AND object_type IN ('TABLE', 'INDEX', 'VIEW'); BEGIN FOR obj IN c_objects LOOP BEGIN v_metadata := DBMS_DEVELOPER.GET_METADATA( name => obj.object_name, schema => p_schema, object_type => obj.object_type ); SELECT JSON_VALUE(v_metadata, '$.etag') INTO v_etag FROM dual; INSERT INTO schema_versions (object_name, object_type, object_schema, capture_date, etag, metadata) VALUES (obj.object_name, obj.object_type, p_schema, SYSTIMESTAMP, v_etag, v_metadata); COMMIT; DBMS_OUTPUT.PUT_LINE('Captured metadata for ' || obj.object_type || ' ' || p_schema || '.' || obj.object_name); EXCEPTION WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE('Error capturing ' || obj.object_type || ' ' || p_schema || '.' || obj.object_name || ': ' || SQLERRM); END; END LOOP; END; / This solution provides several key benefits: Efficient change tracking: Using etags to identify exactly which objects have changedStructured documentation: Storing metadata in JSON format for easy extraction of specific attributesHistorical record: Maintaining snapshots of schema evolution over timeValidation capabilities: Comparing source and target schemas during migration During migration, you can extend this system to compare environments: -- Procedure to compare object between environments CREATE OR REPLACE PROCEDURE compare_object( p_name VARCHAR2, p_type VARCHAR2, p_source_schema VARCHAR2, p_target_schema VARCHAR2, p_target_db VARCHAR2 ) AS v_source_metadata CLOB; v_target_metadata CLOB; v_source_etag VARCHAR2(100); v_target_etag VARCHAR2(100); BEGIN -- Get source metadata v_source_metadata := DBMS_DEVELOPER.GET_METADATA( name => p_name, schema => p_source_schema, object_type => p_type ); -- Get target metadata via database link EXECUTE IMMEDIATE 'SELECT DBMS_DEVELOPER.GET_METADATA( name => :1, schema => :2, object_type => :3 ) FROM dual@' || p_target_db INTO v_target_metadata USING p_name, p_target_schema, p_type; -- Extract etag values SELECT JSON_VALUE(v_source_metadata, '$.etag') INTO v_source_etag FROM dual; SELECT JSON_VALUE(v_target_metadata, '$.etag') INTO v_target_etag FROM dual; -- Compare and report IF v_source_etag = v_target_etag THEN DBMS_OUTPUT.PUT_LINE('Objects match exactly'); ELSE DBMS_OUTPUT.PUT_LINE('Objects differ - detailed comparison needed'); -- Further JSON comparison logic could be implemented here END; END; / Conclusion The DBMS_DEVELOPER package represents a significant advancement in Oracle's metadata management capabilities. By providing metadata in JSON format, Oracle has created a more developer-friendly interface that aligns with modern application architecture patterns. Key takeaways include: JSON-based metadata is more programmatically accessible than traditional DDL statements The etag mechanism provides a reliable way to track object changes Multiple detail levels allow you to retrieve just the information you need The package is particularly valuable for documentation, migration, and change tracking While currently limited to tables, indexes, and views, the DBMS_DEVELOPER package has tremendous potential for expansion in future Oracle releases. Database architects and developers should consider integrating this powerful tool into their workflows, particularly for projects involving schema documentation, migration, or programmatic metadata access. As databases continue to evolve toward more autonomous and programmable systems, tools like DBMS_DEVELOPER will become increasingly central to efficient database management practices.

By arvind toorpu CORE

After a decade of building and debugging large-scale data pipelines across financial services, payments processing, and analytics platforms, I can tell you that almost every slow Spark job I've investigated had the same root cause — and it wasn't the one the team thought it was. The default response when a Spark job is slow is to add more executor memory, increase the number of executors, or bump spark.sql.shuffle.partitions. Sometimes that helps. Usually it doesn't. What I've found, consistently, is that the real problems are structural — a join strategy mismatch that silently multiplies your intermediate dataset by ten times, a single slow task on a degraded node that holds an entire stage hostage, or a decrypt chain that re-reads source data six times when it only needed to read it once. This article is organized around five patterns I keep seeing across teams. Each one looks different on the surface but traces back to a misunderstanding of how Spark actually executes your code. For each pattern, I'll describe what it looks like, when it bites you, the failure mode, and how to fix it. Pattern 1: The OR Join That Quietly Multiplies Your Data What It Looks Like A join condition with an OR clause. Usually introduced when a business requirement adds a secondary matching rule — match on primary card number, or if the transaction is a virtual card transaction, match on the underlying physical PAN. The SQL looks reasonable. The engineer tests it on a sample, and it returns the right rows. When It Bites You At scale. With 100 million transaction rows and 50 million account rows, this query starts running for hours. The output size is also wrong — much larger than expected before DISTINCT trims it down. The Failure Mode Spark cannot use a hash join or sort-merge join when the join condition contains OR. It falls back to BroadcastNestedLoopJoin — for every row in the left table, scan every row in the right table. That's O(n x m). On real datasets, this produces an intermediate result in the hundreds of GB before any downstream filter runs. I've watched a pipeline that should produce 8 GB of output generate 400 GB of intermediate data because of exactly this pattern, taking a 20-minute job to 4 hours. You can verify this in 30 seconds: run df.explain(formatted) and look for BroadcastNestedLoopJoin in the physical plan. If you see it on a join involving any table over a few million rows, it's almost certainly unintentional. The Fix Split the join into two equi-join legs and UNION ALL the results: SQL -- Leg 1: primary match (equi-join — uses SortMergeJoin or BroadcastHashJoin) SELECT txn.*, acct.* FROM transactions txn JOIN accounts acct ON txn.card_number = acct.card_number UNION ALL -- Leg 2: fallback match, filtered scope only SELECT txn.*, acct.* FROM transactions txn JOIN accounts acct ON txn.fpan = acct.physical_pan WHERE txn.transaction_type = 'VIRTUAL' Each leg is a proper equi-join. Apply DISTINCT at the end to deduplicate rows that matched both. The performance difference is routinely an order of magnitude. Pattern 2: The Straggler Task That Nobody Notices Until It's Too Late What It Looks Like A stage that should take 10 minutes takes 3 hours. The Spark UI shows nearly all tasks completed quickly. One or two tasks are still running with a disproportionately long duration. When It Bites You Jobs running on shared YARN or cloud infrastructure where any node can have a bad disk, a noisy neighbor, or degraded network throughput. Also common in stages that call external services per partition — one slow API response can cause a single partition's tasks to take 100x longer than the others. The Failure Mode A stage doesn't complete until the last task completes. Not the median. Not p95. The absolute last one. If 2,200 tasks finish in under 2 minutes and one takes 3 hours and 7 minutes, the stage takes 3 hours and 7 minutes. The other 2,199 executors sit idle. This is the straggler problem, and it's distinct from data skew. The diagnostic: in the Stage detail view, check the task duration distribution. If MAX is dramatically higher than p99, that's a straggler (hardware or external service issue). If p75 is already much higher than p50, that's skew (data distribution issue). They require different fixes, and many teams treat them identically. The Fix For stragglers caused by degraded infrastructure, enable Spark speculation: Properties files spark.speculation=true spark.speculation.multiplier=3 # task must be 3x slower than median spark.speculation.quantile=0.9 # wait for 90% completion before speculating Speculation re-launches slow tasks on a different executor and uses whichever copy finishes first. The caveat: don't use this on stages that write to non-idempotent sinks. For read-heavy or compute-heavy stages — including external decryption calls — it's often the single most impactful config change you can make. Pattern 3: The df.rdd Decrypt Chain That Recomputes Everything Six Times What It Looks Like A pipeline that calls an external encryption or decryption service per record, implemented as a series of df.rdd.mapPartitions() calls, one per column that needs to be processed. When It Bites You When you have multiple columns to decrypt. Each .rdd call creates a new computation starting from the original DataFrame — Spark re-reads from source, re-executes all upstream joins and filters, and then runs the decryption for that column. With six columns to decrypt, you're doing that six times. The Failure Mode Two distinct sub-problems compound each other. First, going to RDD bypasses Catalyst entirely — no predicate pushdown, no column pruning, no Tungsten execution. Second, without a persist checkpoint before the chain, every decrypt call lineages all the way back to the source. I've seen this double the runtime of a job compared to the same pipeline with a single persist() before the decrypt chain. On top of that, the external call latency per partition is dominated by the number of HTTP round trips, not the payload size. Cutting your batch size in half doubles your request count and roughly doubles your wall-clock time for that stage. Most teams set an initial batch size and never revisit it. The Fix Two changes, applied together: Persist the input DataFrame before starting the decrypt chain. This means the join and filter logic runs once, and each decrypt call reads from the cached result.Increase the batch size for external calls. Test at several sizes — going from 20,000 to 40,000 records per batch often cuts stage time by 30-50% with no change to correctness. Scala val base = rawDf.filter(...).join(key1, ...).persist(StorageLevel.MEMORY_AND_DISK) val step1 = decryptColumn(base, secret1) // reads from cache val step2 = decryptColumn(step1, secret2) // reads from cache val step3 = decryptColumn(step2, secret3) // reads from cache Without persist, step2 re-executes everything step1 did from source. With persist, each step reads from the in-memory result of the previous. Pattern 4: The shuffle.partitions Setting That Nobody Updates What It Looks Like A job that works fine in staging — where data volumes are 10% of production — but runs slowly, spills to disk, or produces thousands of tiny output files in production. When It Bites You When the default spark.sql.shuffle.partitions=200 is left unchanged. 200 partitions made sense as a default for medium datasets but is almost always wrong at production scale — either too few (huge partitions, memory pressure) or too many (tiny partitions, scheduling overhead, small files problem). The Failure Mode Too few partitions means each executor handles a disproportionately large chunk of data. With 200 partitions on a 1 TB shuffle, each partition is 5 GB. That will spill to disk. Too many partitions means thousands of 1 MB tasks — the scheduling overhead becomes significant, and your output has thousands of tiny files that hurt downstream readers. With Adaptive Query Execution (AQE) enabled in Spark 3.2+, this problem largely manages itself. AQE merges small post-shuffle partitions automatically and can handle modest skew. But AQE can't help if it's disabled, and it can't fix the upstream causes of extreme skew. The Fix Enable AQE if you're on Spark 3.2+: Properties files spark.sql.adaptive.enabled=true spark.sql.adaptive.coalescePartitions.enabled=true spark.sql.adaptive.skewJoin.enabled=true If you need to set shuffle.partitions manually, target roughly 128-256 MB per partition post-shuffle. For a 500 GB shuffle, that means 2,000-4,000 partitions. Set it high and let AQE coalesce down — that's cheaper than setting it low and getting OOM errors. Pattern 5: The Incremental Job That Degrades Silently Over Time What It Looks Like A job that runs in 15 minutes when first deployed and runs in 4 hours six months later. No code changes. No obvious data quality issues. The team attributes it to data growth. When It Bites You When the job fails a few times in a row, and the recovery accumulates multiple windows' worth of data. Or when the watermark logic was designed for small windows but nobody anticipated that the underlying join tables would grow significantly. The Failure Mode Two separate causes, often confused. First, if the watermark is a single timestamp and the job has been failing, recovery runs can accumulate large backlogs. A job that normally processes 2 hours of data may need to process 48 hours on first successful recovery, with no change to the resource configuration. Second, growth in reference data (like an accounts table or lookup table used in a join) increases the size of every run regardless of whether the incremental input grew. I've seen a 30-minute job become a 3-hour job purely because the accounts table grew from 10 million rows to 80 million rows over 18 months, while the OR join condition (see Pattern 1) meant that growth was amplified into the intermediate result. The Fix Two design principles that pay off over the lifetime of the pipeline: Track processed partitions explicitly rather than using a single timestamp watermark. This makes recovery granular — you can replay specific missing partitions without re-processing everything after them.Add a fast-path no-op check before initializing the full Spark session. Check whether any new partitions exist first. A 5-second check that exits early is much better than a 2-minute executor startup that discovers there's nothing to process. For the reference table growth problem: if your lookup table grows significantly, revisit whether it can be broadcast (small enough to fit in executor memory) or whether the join itself needs to be redesigned. Quick Diagnostic Reference Use this table to map what you observe in the Spark UI to the likely pattern and first action to take: WHat you observeLikely patternconfirm withfirst action MAX task duration >> p99 Straggler (Pattern 2) Task timeline in Stage UI Enable spark.speculation p75 >> p50 task duration Data skew Input bytes per task Repartition on join key; AQE skewJoin BroadcastNestedLoopJoin in explain() OR join (Pattern 1) df.explain( formatted) Rewrite as UNION of equi-joins Stage runtime grows week on week; no code change Incremental accumulation or reference table growth (Pattern 5) Input bytes trend in History Server Audit watermark logic; check reference table size OOM errors or heavy disk spill Too few shuffle partitions (Pattern 4) Spill metrics in Stage UI Enable AQE or increase shuffle.partitions The Common Thread Every pattern here traces back to the same underlying issue: Spark is executing something different from what the engineer intended. The OR join was intended as a flexible matching rule; Spark turned it into a nested loop. The decrypt chain was intended as six independent transformations; Spark turned it into six full re-reads of source data. The incremental job was intended to process one window of data; without proper watermark design, it occasionally processes twelve. The Spark UI has everything you need to see this — task distribution, input and output sizes, physical plans, spill metrics. Most teams open it when something breaks and close it once they find the obvious error. Opening it proactively, forming a hypothesis, and then confirming or refuting it in the metrics is the practice that separates engineers who consistently improve pipeline performance from those who add executor memory and hope for the best. The mistake isn't choosing the wrong config. It's not understanding what Spark is actually doing with your code.

A federated gateway provides secure, policy-aware access to tool servers. The thing that made me stop and rethink our whole approach to agentic tooling was a text file. An engineer on one of our platform teams had wired an AI coding assistant up to our internal source control. To do it, they had pasted a personal access token into a local MCP server config in their home directory. It worked. That also meant a long-lived credential with broad repository scope sat in plaintext in a file the agent could read, on a laptop, with no audit trail and no expiry. Multiply that by every engineer who wants their assistant to see internal code, artifacts, docs, and warehouse tables, and you have hundreds of copies of your crown-jewel credentials distributed across endpoints you do not control. That is the real problem with Model Context Protocol adoption in an enterprise. MCP itself is a good protocol. The failure mode is topological: the default deployment story puts the server, the credentials, and the client on the same machine, which is exactly where you least want them in a network-isolated environment. What we built instead was a federated control plane. One gateway, many backend tool servers, and a thin local connector that holds no secrets at all. The Three-Hop Topology The pattern is simple to state, and most of the engineering effort goes into the seams: Plain Text Connector -> Gateway -> Server The connector runs locally next to the IDE or agent. It speaks stdio to the client, because that is what most assistants expect, and streamable HTTP outbound to the gateway. It is deliberately dumb. It knows one URL and how to complete a browser-based login. It stores no client secret, no API key, no PAT. The gateway is the control plane. It terminates authentication, brokers OAuth on the user's behalf, resolves which backend server should handle a given request, enforces policy, and emits telemetry. It is the only component that ever touches a credential. The backend servers are the actual MCP implementations: source control, artifact repository, documentation search, static analysis, browser automation, warehouse metadata. Each is a separate deployment with its own least-privilege identity. They live in-cluster, on the internal network, with no default egress to the public internet. The property that matters is that the trust boundary sits at the gateway, not at the laptop. A compromised developer machine yields a session, not a credential. The Gateway as an OAuth Broker This is the part people underestimate. The gateway does not proxy the user's token; it exchanges an authenticated session for a narrowly scoped downstream credential, per backend, per request. Concretely, when a request arrives, the gateway resolves the caller's identity from the session, looks up the target server, and mints or fetches a downstream token with only the scopes that server is registered to need: Python async def broker(request: MCPRequest, session: Session) -> MCPResponse: server = registry.resolve(request.server_id) if server is None: raise PolicyError("unregistered_server") if not policy.allows(session.principal, server, request.method): audit.deny(session.principal, server.id, request.method) raise PolicyError("not_permitted") # Client secrets are held by the gateway only; never sent downstream # to the connector and never written to a client-side config. token = await broker_pool.token_for( principal=session.principal, provider=server.auth_provider, # e.g. saml_scm, google scopes=server.least_privilege_scopes, # e.g. ["repo:read"] ttl_seconds=900, ) return await transport.forward(server, request, bearer=token) Two design choices are worth calling out. First, least_privilege_scopes is a property of the registered server, not of the user's login. A developer authenticating once through the gateway does not thereby grant every backend the union of their permissions. A documentation server gets read scope on docs and nothing else, even if the same human has admin rights elsewhere. Second, we deliberately started with a static client registration model backed by the platform's own secret store, with a migration path to Dynamic Client Registration. DCR is where this should end up, but shipping a working broker with rotating short-lived tokens beat waiting for the spec ecosystem to settle. Secrets are created by CI/CD from a managed secret store; no human hands a production secret to a running workload. Guardrails Against Tool Poisoning Once agents can call tools, tool descriptions become an attack surface. A malicious or compromised server can return a tool definition whose description instructs the model to exfiltrate context, or can silently mutate a description after initial approval. Rate limiting alone does not help here. We enforce validation at the gateway in both directions of the exchange: Python POISON_PATTERNS = [ r"ignore (all )?(previous|prior) instructions", r"do not (tell|inform|mention to) the user", r"<\s*(system|assistant)\s*>", ] def validate_tool_manifest(server_id: str, manifest: dict) -> None: for tool in manifest["tools"]: blob = f"{tool['name']} {tool.get('description', '')}" for pattern in POISON_PATTERNS: if re.search(pattern, blob, re.IGNORECASE): quarantine(server_id, tool["name"], reason=pattern) raise PolicyError("suspect_tool_description") # Descriptions are pinned at review time. Drift requires re-approval. if sha256(blob) != registry.approved_digest(server_id, tool["name"]): raise PolicyError("manifest_drift") The digest pinning is the load-bearing control. Pattern matching catches the naive cases; pinning catches the case where an approved server changes its behavior after review. Any drift takes the tool out of rotation until a human re-approves it. On top of that: per-principal and per-server rate limits, an explicit allow/block list of methods, and argument validation before forwarding. We mapped these controls to published guidance for AI system risks so the security review had something concrete to assess rather than a narrative. Observability Is Not Optional Here When something goes wrong in an agentic workflow, the user's report is usually "the assistant got confused." That is not debuggable. Centralizing traffic through one gateway means you get, for free, the telemetry that makes it debuggable: latency percentiles per server and per method, error rates by status code, MCP method distribution, transport breakdown between stdio and streamable HTTP, and per-principal activity. Two things surfaced from that data that we would never have found otherwise. One backend was returning successful responses with empty payloads for a large share of calls, which looked healthy on an error-rate dashboard and terrible to users. And tool usage was heavily concentrated: a small number of servers and a small number of engineers accounted for most traffic, which told us where to spend reliability effort instead of guessing. Making It Self-Service, or It Dies A control plane that requires a platform engineer in the loop becomes the bottleneck it was meant to remove. The onboarding path we settled on is a scaffolded repository from an internal portal, image build and promotion through CI, infrastructure-as-code deployment via pull request, automated vulnerability scanning, and auto-registration into the gateway registry on merge. New server idea to registered production service is one pull request and two approvals. The lesson I would pass on: solve the credential topology first, then the ergonomics. Teams that start with developer convenience end up retrofitting security onto a distributed pile of local configs, and that retrofit is far more expensive than getting the trust boundary right on day one.

A sidecar is a container that runs alongside another container as part of the same deployment unit. Just because two containers are in the same cluster or deployed around the same time doesn't make one a sidecar. There are two things that make a sidecar. First is that they share a network namespace, so they can reach each other over localhost rather than a network address. Second, they share a lifecycle. This means that they are created together, scaled together, and by default torn down together. Neither container has an existence independent of the other. The problem it solves is giving a specific concern its own boundary. For example, it can have its own filesystem, its own memory space, and often its own permissions or dependency set, without giving up the simplicity of deploying and operating one unit. You get isolation without paying for the operational overhead of running and coordinating a fully separate service. The test that defines the pattern across all of these is this: does it live and die with its partner container as one unit of deployment? If yes, it's a sidecar. If you have to reach it by hostname, through service discovery, or via a queue, it isn't one anymore. That is a separate service that happens to sit next to the first. That test matters because two adjacent patterns get called "sidecar" when they aren't: Decoupled worker/microservice. A separately deployed container, reached over the network, scaled on its own. A web application offloading work to Celery workers via Redis is a common instance of this: the app enqueues a job (send this signup email), a pool of workers pulls jobs off the queue independently, and neither side shares a network namespace or a lifecycle with the other. The workers scale on queue depth, not on how many web replicas are running, and a web app restart doesn't take queued or in-flight jobs down with it. n8n has its own version of the same shape: "queue mode," where a main node accepts webhooks and separate worker nodes pull jobs off a Redis queue. It's tempting to call either of these a sidecar relationship since the worker and the web app do feel paired, but neither qualifies: they don't share a deployment unit, and killing one doesn't touch the other.Ambassador/adapter. A container that proxies or translates traffic on its parent's behalf, like the Envoy example above, is actually this, more precisely. Structurally it's still a sidecar; it just gets a more specific name for what it does. Using n8n to Understand It What n8n Is n8n is a workflow automation platform like Zapier, but self-hostable and node-based rather than form-based. A handful of components make up a running instance: The editor/UI, where workflows are built visually as a graph of nodes.The main process, which serves that UI, listens for webhooks, and orchestrates workflow execution. The workflow execution decides what runs next, passing data between nodes and recording results.Nodes, the individual units of a workflow: trigger nodes (a webhook arrives, a schedule fires), action nodes (call an API, write to a database, send an email), and the Code node. The code node lets you drop in arbitrary JavaScript or Python to transform data however the built-in nodes can't. The code node is relevant in this article. The database, where workflow definitions, credentials, and execution history persist. In this article, Postgres is used. For most of what n8n does, the main process is the only thing doing work: routing a webhook, calling an API, writing a database row. The exception is the Code node, and that exception is the whole reason task runners exist. The Task Runner Feature and Its Use Case By default, a Code node's JavaScript or Python executes inside n8n's main process. This main process holds the database connection, the encryption key, and every credential stored in every workflow you've built. That's fine for trusted, well-understood scripts. It becomes a real problem the moment the code in that node is untrusted, third-party, or arbitrary enough that you can't fully audit it before it runs. By the way, that is how most Code nodes are used in practice. Task runners exist to solve exactly that use case: run Code node logic somewhere the main process's credentials and connections aren't reachable from it, without turning "write some JavaScript to reshape this JSON" into a separately deployed microservice every time. Going Deep on the Task Runner Feature n8n ships two modes for this: Internal mode (the default) runs Code nodes inline, in-process. No isolation. This is the fastest to set up, but the weakest boundary.External mode moves execution into a separate runner process entirely. That process connects back to the main n8n instance over a broker (an authenticated connection the main process listens on) and receives individual tasks to execute rather than having any standing access to n8n's internals. The runner never touches the database connection, the encryption key, or stored credentials directly; it only ever sees the specific input data for the task it's been handed. External mode goes further than just "a different process," too. The runner's own configuration (the n8n-task-runners.json file built in Phase 4) sets explicit allowlists — which environment variables the runner process can see at all, and which JavaScript built-ins or Python modules it's permitted to import, standard library and third-party tracked separately. So the boundary isn't just "different memory space," it's "different memory space, plus a declared, auditable list of exactly what this process is allowed to touch." That's a specific concern (arbitrary code execution) given its own boundary, without turning it into a fully independent service you have to deploy, discover, and monitor separately. It's the sidecar problem, stated exactly: external mode gives you the isolation; running the external runner as its own container in the same task definition is what makes that isolation a sidecar rather than just a separate process sharing a machine. Why This Needs to Scale Independently and Why "In the Same Container" Isn't Enough Most n8n deployment guides run n8n with task runners in internal mode, or with the external runner living inside the same container as the main process. For example, you will see guides about deploying n8n on a single EC2 instance, Render, DigitalOcean, or any platform's basic tier. That gets you the process isolation, which solves the security half of the problem. It doesn't solve the other half, which is that a runner sharing a container with the app can't be scaled, resourced, or restarted independently of it. That stops mattering the moment Code-node execution becomes the actual bottleneck rather than webhook handling or UI traffic. Imagine workflows doing heavy data transformation in Python, running numpy/pandas operations across large payloads, or executing many Code nodes concurrently. If the runner is bundled into the main container, giving it more CPU means giving the entire n8n instance more CPU, whether the UI and webhook layer need it or not. There's no way to say "the runner needs 2 more vCPUs, n8n itself is fine". Why AWS Fargate's Task Definition Is the Right Fit A Fargate task definition lets each container in the task carry its own CPU and memory reservation, its own health check, and its own essential flag governing what happens if it fails while still keeping every container in the task on one shared network interface. That's the sidecar promise made literal: isolation and independent resourcing for the runner, without losing the operational simplicity of one task, one deploy, one thing to scale as a unit when you do want to scale both together. The rest of this guide deploys exactly that: one Fargate task, two containers, wired together the way the definition above requires. Each infrastructure decision below gets tied back to a specific part of what's laid out here, so that by the end, the concept isn't something read once at the top, but it's something built. Prerequisites AWS account with billing enabledA domain you control, with DNS accessDocker installed locally, with docker buildx availableAWS CLI configured (aws configure) with permissions for ECR, ECS, RDS, ACM, and IAMThe runner image source (Dockerfile + n8n-task-runners.json) — built in Phase 4 Architecture Markdown User's Browser (HTTPS) | [Application Load Balancer] <- Certificate Manager (SSL Cert) | (Port 5678, HTTP internal) [ECS Fargate Task] |-- Container: n8n (main) <-- shared network namespace --> Container: n8n-runner (sidecar) | (Port 5432, PostgreSQL) [RDS PostgreSQL Database] The load balancer and RDS layers are ordinary AWS plumbing. The box in the middle is where the sidecar relationship actually lives. There is one task and two containers, each with its own resourcing. Phase 1: RDS PostgreSQL RDS Console → Create database → Standard create → Engine: PostgreSQLDB instance identifier: n8n-db. Master username: postgres. Generate and save a strong master password.Instance size: db.t4g.microStorage: 20 GB gp3, autoscaling on, max 100 GBConnectivity: the VPC you'll use throughout. Public access: No. New security group: n8n-db-sg, left empty for now.Additional configuration → Initial database name: n8n. Skip this and n8n fails on first connect with "database does not exist" — the DB instance identifier names the server, this field names the database inside it.Create, wait for "Available," copy the endpoint from Connectivity & security. Phase 2: ACM Certificate n8n requires HTTPS for webhooks to function Certificate Manager, in the same region you'll deploy the Load Balancer in → Request a public certificateDomain name: n8n.yourdomain.comValidation method: DNS validationCreate the CNAME record ACM provides at your registrar. If your registrar auto-appends your domain to the Host field, paste only the portion before your domain — the full string duplicates it and validation never completes.Wait for status: Issued Phase 3: Security Groups Two connections need rules: Security groupInbound rulePurposen8n-alb-sg443 from 0.0.0.0/0Public HTTPSn8n-ecs-sg5678 from n8n-alb-sgALB → n8n containern8n-db-sg (edit existing)5432 from n8n-ecs-sgn8n container → RDS Phase 4: Build and Push the Runner Image Dockerfile: Dockerfile FROM n8nio/runners:1.121.0 USER root RUN cd /opt/runners/task-runner-javascript && pnpm add moment uuid adm-zip RUN cd /opt/runners/task-runner-python && uv pip install numpy pandas pydantic requests boto3 certifi COPY n8n-task-runners.json /etc/n8n-task-runners.json ENV N8N_RUNNERS_CONFIG_FILE=/etc/n8n-task-runners.json USER runner It starts from n8n's own n8nio/runners base (containing the launcher and both runner processes), adds only the dependencies workflows actually need, and drops back to a non-root user once the root-only install steps finish. n8n-task-runners.json is where the isolation described above stops being architectural and becomes enforced: JSON { "task-runners": [ { "runner-type": "javascript", "health-check-server-port": "5681", "allowed-env": ["PATH", "GENERIC_TIMEZONE", "NODE_OPTIONS"], "env-overrides": { "NODE_FUNCTION_ALLOW_BUILTIN": "crypto,zlib", "NODE_FUNCTION_ALLOW_EXTERNAL": "moment,uuid,adm-zip" } }, { "runner-type": "python", "health-check-server-port": "5682", "env-overrides": { "N8N_RUNNERS_STDLIB_ALLOW": "json,zipfile,io,base64,datetime,re,math,random,statistics", "N8N_RUNNERS_EXTERNAL_ALLOW": "numpy,pandas,pydantic,requests,boto3,certifi" } } ] } allowed-env restricts which environment variables the runner process can see; N8N_RUNNERS_STDLIB_ALLOW / EXTERNAL_ALLOW restrict which Python modules it can import, stdlib and third-party separately. One container, two runner processes — the launcher inside n8nio/runners spawns both. Build and push: Shell docker buildx build -t n8nio/runners:custom . aws ecr create-repository --repository-name n8n-runners --region us-east-1 aws ecr get-login-password --region us-east-1 \ | docker login --username AWS --password-stdin <account-id>.dkr.ecr.us-east-1.amazonaws.com docker tag n8nio/runners:custom <account-id>.dkr.ecr.us-east-1.amazonaws.com/n8n-runners:custom docker push <account-id>.dkr.ecr.us-east-1.amazonaws.com/n8n-runners:custom --username AWS is a fixed literal, not your actual username — ECR auth always uses it. The password piped via --password-stdin is a short-lived token generated by the CLI, not your account password. Phase 5: The Task Definition This is where the two containers become an actual sidecar pair, and where the independent-resourcing argument from the introduction becomes a real field rather than a claim. JSON { "family": "n8n-task", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "cpu": "1024", "memory": "2048", "executionRoleArn": "arn:aws:iam::<account-id>:role/n8n-task-execution-role", "containerDefinitions": [ { "name": "n8n", "image": "n8nio/n8n:1.121.0", "essential": true, "entryPoint": ["sh", "-c"], "command": [ "mkdir -p /home/node/certs && wget https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem -O /home/node/certs/rds-ca.pem && /docker-entrypoint.sh" ], "portMappings": [{ "containerPort": 5678, "protocol": "tcp" }], "environment": [ { "name": "DB_TYPE", "value": "postgresdb" }, { "name": "DB_POSTGRESDB_HOST", "value": "<rds-endpoint>" }, { "name": "DB_POSTGRESDB_PORT", "value": "5432" }, { "name": "DB_POSTGRESDB_DATABASE", "value": "n8n" }, { "name": "DB_POSTGRESDB_USER", "value": "postgres" }, { "name": "DB_POSTGRESDB_SSL_CA", "value": "/home/node/certs/rds-ca.pem" }, { "name": "DB_POSTGRESDB_SSL_REJECT_UNAUTHORIZED", "value": "false" }, { "name": "WEBHOOK_URL", "value": "https://n8n.yourdomain.com/" }, { "name": "GENERIC_TIMEZONE", "value": "Africa/Lagos" }, { "name": "N8N_RUNNERS_ENABLED", "value": "true" }, { "name": "N8N_RUNNERS_MODE", "value": "external" }, { "name": "N8N_RUNNERS_BROKER_LISTEN_ADDRESS", "value": "0.0.0.0" }, { "name": "N8N_RUNNERS_BROKER_PORT", "value": "5679" } ], "secrets": [ { "name": "DB_POSTGRESDB_PASSWORD", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/db-password" }, { "name": "N8N_ENCRYPTION_KEY", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/encryption-key" }, { "name": "N8N_RUNNERS_AUTH_TOKEN", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/runners-auth-token" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/n8n-task", "awslogs-region": "<region>", "awslogs-stream-prefix": "n8n" } } }, { "name": "n8n-runner", "image": "<account-id>.dkr.ecr.<region>.amazonaws.com/n8n-runners:custom", "cpu": 512, "memory": 1024, "essential": false, "dependsOn": [{ "containerName": "n8n", "condition": "START" }], "environment": [ { "name": "N8N_RUNNERS_TASK_BROKER_URI", "value": "http://localhost:5679" } ], "secrets": [ { "name": "N8N_RUNNERS_AUTH_TOKEN", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/runners-auth-token" } ], "healthCheck": { "command": ["CMD-SHELL", "curl -f http://localhost:5680/healthz || exit 1"], "interval": 30, "timeout": 5, "retries": 3, "startPeriod": 20 }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/n8n-task", "awslogs-region": "<region>", "awslogs-stream-prefix": "n8n-runner" } } } ] } Five fields here map directly back to the introduction: Per-container cpu/memory on n8n-runner. This is the independent-resourcing argument made literal. The runner gets its own 512 CPU units and 1024 MB, carved out of the task total, separate from whatever n8n is allotted. If Code-node execution turns out to be the bottleneck, this is the number you raise without touching the main container's allocation at all. That's the exact thing a same-container runner can't offer you. networkMode: awsvpc is the mechanical basis of "shared network namespace." Every container in the task gets one elastic network interface between them. This is the setting that makes Phase 3's missing security group rule make sense. There's one network surface, not two. N8N_RUNNERS_TASK_BROKER_URI: http://localhost:5679 only works because of the line above. The runner reaches n8n over localhost because they are the same task. If this pointed anywhere else, you would have built the decoupled-worker pattern from the introduction instead, no matter what you called the container. A shared N8N_RUNNERS_AUTH_TOKEN, pulled from Secrets Manager by both containers. Sharing a network namespace means the runner is reachable by anything else in the task. The isolation the whole pattern exists for still needs a trust boundary at the process level, not just the network level. A plaintext token here would defeat that, since task definitions are readable by anyone with ecs:DescribeTaskDefinition. essential: false on the runner. This governs how tightly the two containers' lifecycles are actually coupled. essential: true would mean a runner crash tears down the whole task, main container included. false means the runner can crash and recover independently: Code-node executions fail until it's back, but the UI and webhooks keep serving. The pattern doesn't mandate one answer; it just means this has to be a decision, not a default you inherited. The health check on port 5680 hits the launcher's own endpoint, separate from the per-runner-type ports (5681 JS, 5682 Python) set in Phase 4's config file. ECS is checking the supervisor, not each runner process individually. Register it: aws ecs register-task-definition --cli-input-json file://n8n-task-def.json Phase 6: Cluster, Service, and Load Balancer ECS → Create cluster → n8n-cluster → Infrastructure: AWS FargateCreate a service inside it: Task definition: n8n-task, latest revisionDesired tasks: 1Networking: your VPC, at least two subnets across AZs, security group n8n-ecs-sg, public IP onLoad balancing: Application Load Balancer, listener on 443 using the Phase 2 certificateTarget group: HTTP, port 5678, health check path /healthzCreate, wait for steady state. Notice the target group and health check only ever reference the n8n container. It did not mention n8n-runner at all. The n8n-runner container doesn't get a port that maps to the load balancer, doesn't get its own listener, doesn't get its own DNS entry. Everything that makes it reachable from outside the task goes through n8n . Phase 7: DNS At your registrar, add a CNAME: Host n8n, Value = your Load Balancer's DNS name. Confirm with nslookup n8n.yourdomain.com once it propagates. Verifying the Sidecar Relationship Visiting https://n8n.yourdomain.com and completing owner setup confirms the main container and database are working. To confirm the runner specifically: Create a workflow with a Code node (JavaScript or Python), and run it.Pull CloudWatch logs for both streams (/ecs/n8n-task, prefixes n8n and n8n-runner). The n8n-runner stream should show the launcher starting both runner processes and reporting a broker connection. The n8n stream should show the Code node's execution dispatched out rather than run inline. If the workflow completes but nothing appears in n8n-runner's logs, check N8N_RUNNERS_MODE=external on the main container first. That's the setting that actually hands execution off instead of running it in-process regardless of what else is configured.

Context engineering is becoming essential as AI agents take on more software development work. An agent can plan, code, test, investigate incidents, trigger CI, and help deploy software. But none of that matters if it is operating without the right information. This is the main problem I keep seeing. We connect an LLM to a few tools, give it a good prompt, and expect magic. Then the agent has to figure out which service we mean, who owns it, what repository it belongs to, whether it is healthy, what incidents are open, and whether a deployment is safe. That is a lot of disconnected information to reconstruct every single time. Context engineering is the discipline of structuring, surfacing, and governing the information an AI agent needs to act reliably. It is how we give agents the right facts, rules, tools, and boundaries so they can make better decisions without hallucinating or wasting time hopping between systems. Key Takeaways Context engineering gives AI agents structured access to instructions, knowledge, memory, examples, tools, and guardrails.A context layer reduces tool switching and prevents agents from wasting effort interpreting disconnected SDLC data.Service catalogs, reusable skills, and human approval gates make agentic workflows more reliable and governable.Deployment recommendations should be grounded in visible evidence such as ownership, health, test coverage, runbooks, and incidents. Step 1: Understand What Context Engineering Actually Means AI agents are powered by LLMs. The LLM is basically the brain, but it does not automatically know the current state of your engineering organization. It does not know your service ownership, deployment history, runbooks, incident status, infrastructure, or internal policies unless you provide that information. That is where Context Engineering comes in. Instead of leaving an agent to guess, I give it access to relevant, organized context. This helps it plan properly, use tools properly, and take actions with much more accuracy. A simple way to think about it is this: Without context: An agent guesses what a service is, where its data lives, and what action is safe.With context: An agent can retrieve the service record, ownership, health, repository, runbook, deployment data, and guardrails before responding or acting. Context engineering is not just about putting more tokens into a prompt. It is about making the right information accessible at the moment an agent needs it. The goal is grounded actions, not longer conversations. Step 2: Identify Why Your Engineering Team Needs a Context Layer Most engineering ecosystems are distributed by design. Source code might live in GitHub, documentation in Notion, incidents in PagerDuty, conversations in Slack, infrastructure in AWS, observability in Datadog, and deployments in Kubernetes. Each tool is useful. The issue is that the knowledge is fragmented. For a developer, that fragmentation creates constant context switching. To understand one service, I may need to open a repository, find its owner, inspect deployment history, search incident records, locate the runbook, and check infrastructure health. That slows down delivery and increases the chance of missing something important. For an AI agent, the problem becomes even bigger. If I ask it to analyze bottlenecks, delivery velocity, quality gaps, and patterns across the SDLC, it may need to fetch and interpret data from every one of those disconnected systems. It spends tokens trying to understand the environment before it can solve the actual task. A context layer sits between the engineering ecosystem and the agent. It connects services, teams, workflows, documentation, policies, and operational data in one place. With that layer in place, context engineering can improve: Deployment speed and confidenceAccuracy in agent responses and actionsSecurity and policy enforcementOperational reliabilityCollaboration across teamsDeveloper productivity by reducing tool switching The point is not to replace every engineering tool. The point is to let humans and agents access the relevant context without manually rebuilding the story every time. Step 3: Fix Engineering Chaos Before It Turns Into Agentic Chaos The software development lifecycle has many stages: planning, coding, building, testing, securing, deploying, operating, learning, and improving. Teams commonly introduce specialized tools at every stage. Over time, that creates tool sprawl, duplicate data, fragmented workflows, and unclear ownership. I call that engineering chaos. It hurts quality, security, compliance, productivity, and operational excellence. Now add AI agents on top of that environment. If every agent is independently connected to different tools and given incomplete instructions, the chaos gets multiplied. Agents may have no shared visibility, no human approval points, inconsistent decisions, and no meaningful safeguards. This is why context engineering should begin with a simple question: What does an agent need to know before it can safely answer or act? For example, if I ask an agent, “Analyze my SDLC data and surface bottlenecks, velocity, quality gaps, and interesting patterns,” the agent needs more than a prompt. It may need: Repository and pull request data from GitHubInfrastructure context from AWSIncident data from PagerDutyOperational discussions from SlackDeployment state from Kubernetes Without a unified context model, the agent must interpret isolated facts from every system. That consumes tokens and can lead to weak, incomplete, or incorrect conclusions. Context Engineering gives that agent a better starting point. Step 4: Build the Six Types of Agent Context When I design context for an AI agent, I think in six categories. Each category answers a different part of the agent’s decision-making problem. 1. Instructions Instructions define rules, goals, and boundaries. They tell an agent what its job is and what it should not do. For example, an incident investigation agent may be instructed to gather evidence, summarize findings, and avoid triggering production actions. 2. Knowledge Knowledge includes documents, architecture diagrams, service metadata, domain data, repositories, and runbooks. This is the factual material an agent needs to understand the environment. 3. Memory Memory holds session logs, previous decisions, and persistent state. It lets an agent maintain continuity across multi-step workflows rather than treating every action as a completely new task. 4. Examples Examples provide short demonstrations and reference patterns. They show an agent what a useful output or a correct workflow looks like. This is especially useful when a task needs a consistent format. 5. Tools Tools include APIs, scripts, CI systems, and external services. Tools turn an agent from a chat interface into something that can retrieve current data and execute approved tasks. 6. Guardrails Guardrails are the hard constraints: safety rules, checklists, policy requirements, and approval gates. They are critical when an agent can do more than just answer a question. Instructions, knowledge, and memory are generally more static forms of context. Examples, tools, and guardrails are dynamic because they can change with the workflow, the service, and the current situation. Effective Context Engineering brings all six together instead of relying on a single prompt. Step 5: Separate Prompt Engineering From Context Engineering Prompt engineering and context engineering work together, but they solve different problems. Prompt engineering is about what to say. It focuses on the instructions and examples used to guide an interaction. It is useful for optimizing a single request or response. Context engineering is about what the agent gets to see. It focuses on managing accessible information across a workflow: the service data, connected tools, policies, history, ownership, and real-time status the agent needs. A great prompt cannot compensate for missing operational facts. If an agent does not know the owning team, service tier, runbook, open incidents, or deployment policy, no clever wording will make its production decision trustworthy. Step 6: Create a Service Catalog That Gives Agents Grounded Context To make Context Engineering practical, I need a system that represents the services in my environment and connects their information. In the demo, I use Port.io as a context layer for an agentic SDLC. A service catalog can hold details such as: Service name and identifierEnvironment, such as staging or productionOwning teamRepository associationRunbook URLSlack channelService tier and visibilityObservability linksOn-call rotation status Once this context is registered, an agent can answer a question like “Share everything about the shipment service” by retrieving a unified service overview. In the example, that overview includes the owning team, language, repository, branch, recent code activity, runbook, on-call status, health information, deployments, pull requests, and scorecard data. This is the practical value of context engineering. Instead of manually gathering facts from several tools, I can ask once and get a contextual answer built from the connected service record. Step 7: Turn Repeated Agent Instructions Into Reusable Skills Agents often perform repeated tasks: investigate an incident, assess deployment risk, review a pull request, measure DORA metrics, run CI, or deploy a service to production. Repeating the full instructions every time is not scalable. That is where agent skills are useful. A skill packages the context and logic needed for a repeatable operation. For example, I can define skills for: Incident responsePort readiness checksRunning CIDeploying a serviceDeploying to production When I ask an agent to run CI for the shipment service, it can load the relevant CI skill and combine it with the shipment service context. The agent is not starting from zero. It knows the service, the intended workflow, and the constraints around execution. This makes Context Engineering reusable. Skills reduce repeated setup work, standardize workflows, and help agents perform the same task in a predictable way across services. Step 8: Add Human Gates to Agentic SDLC Workflows Automation does not mean removing human control. In an agentic SDLC workflow, agents can gather requirements, plan work, generate code, test changes, and run continuous integration. But important actions should still include approval or rejection points. For example, a workflow can fetch service context first, then proceed through: Requirements gatheringPlanningCodingTestingContinuous integrationHuman approval before sensitive actions Human gates are part of good context engineering because they provide governance. The agent can recommend, prepare, and trigger approved workflows, but a person can still decide whether a proposed action should proceed. Step 9: Use Context to Make Better Deployment Decisions The final demo makes the value very clear. A simple application loads context for a selected service and gives a production-readiness verdict. For a healthy payment service, the context shows a clear picture: ownership is assigned, the Slack channel is configured, the runbook is documented, on-call rotation is active, test coverage is 94%, health status is healthy, the service was deployed recently, and there are no open incidents. Based on that connected information, the service is marked ready to deploy. For another service, the result is completely different. It is marked as not ready because key context is missing. There is no owning team, no runbook, and several other readiness requirements are incomplete. The system identifies the gaps instead of making a blind recommendation. That is what a production decision should look like. Not “yes” or “no” based on a vague prompt, but a verdict grounded in explicit evidence: Identity and ownershipHealth and operational statusRunbook availabilityOn-call coverageTest coverageRecent deployment historyOpen incidentsRequired scorecard checks When the context indicates risk, the result can say to proceed with caution and explain why. This is far more useful than an agent giving an unverified deployment recommendation. Step 10: Treat Context Engineering as an Engineering Discipline Context engineering is important because AI agents are only as reliable as the environment they can understand. If an agent has scattered data, unclear ownership, missing policies, and unrestricted tools, it will struggle no matter how advanced the model is. The practical path is straightforward: Map the tools and data sources that define your SDLC.Define the service-level context agents need to retrieve.Centralize ownership, health, repositories, runbooks, incidents, and policies.Create reusable skills for common workflows.Use tools for live data and approved execution.Add guardrails and human approvals around consequential actions.Make agent verdicts explainable through visible context. That is how I move from disconnected AI experiments to reliable agentic engineering workflows. Context Engineering reduces unnecessary token use, reduces confusion, and gives agents the facts they need to help build, test, operate, and deploy software with more control. Context Engineering FAQs What Is Context Engineering for AI agents? Context Engineering is the practice of organizing and governing the information an AI agent can access, including instructions, service data, memory, tools, examples, and safety constraints. It helps the agent make grounded decisions rather than guessing. How Is Context Engineering Different From Prompt Engineering? Prompt engineering focuses on how to phrase instructions for an interaction. Context Engineering focuses on the information the agent can retrieve and use throughout a workflow, such as ownership, repositories, incidents, deployment data, and policies. What Context Should an SDLC Agent Have? An SDLC agent should have the context needed for its task, which can include service ownership, repository details, environment, runbooks, on-call status, deployment history, test coverage, incident status, relevant tools, and hard safety rules. Why Are Human Approval Gates Important for AI Workflows? Human gates preserve control over consequential actions. Agents can retrieve context, prepare work, and recommend or trigger an approved workflow, while a person retains the ability to approve or reject sensitive changes.

By Pavan Belagatti CORE

Enterprise AI is moving beyond isolated prompt-response calls and toward systems that observe events, preserve state, invoke tools, and publish decisions back into operational workflows. In that setting, event streaming is not simply middleware. It becomes the record of how intelligent behavior unfolds over time. Kafka is designed to read, write, store, and process streams of events across distributed systems, while Kafka Streams adds joins, aggregations, windowing, event-time processing, and exactly once support for stateful stream applications. At the same time, modern agent runtimes have shifted toward durable execution, persistence, and human-governed control flows rather than single-turn prompting alone. That convergence makes Kafka a strong coordination layer for autonomous agents that need to react continuously instead of responding once and disappearing. That architectural change also alters the role of the model. In an API-centric design, the model is often treated as a synchronous dependency behind a request. In an event-driven design, the model becomes one participant in a larger decision pipeline. Observations arrive as events, context is assembled from topics and state stores, agent steps are logged, and decisions are emitted as new events for downstream systems. Because Kafka topics can be replayed and reprocessed, the same stream can feed planners, validators, enrichment services, audit consumers, and human-review workflows without creating hard coupling between those components. The resulting system is easier to inspect, easier to recover, and easier to evolve than a chain of tightly bound remote calls. Turning Kafka Into the Coordination Layer The most important benefit is not only scale. It is the replacement of brittle request chains with an append-only coordination layer. A payment event, support ticket update, equipment alarm, or fraud signal can be published once and then consumed independently by retrieval components, compliance checks, planners, and execution agents. Kafka consumer groups divide partitions across consumers in the same group, and each partition is consumed by a single consumer within that group, which preserves ordering at the partition level while still allowing horizontal scale. For agentic systems, that detail is central. If all events for the same case, customer, or device are keyed consistently, one partition becomes the serialized timeline for that entity, and the agent no longer has to reconstruct order from racing HTTP callbacks. The event log also becomes a durable memory boundary. Kafka log compaction retains the latest value for each key, which makes compacted topics useful for task state, policy snapshots, approval status, or tool metadata that must survive restarts and recover quickly. On the runtime side, agent frameworks persist checkpoints and thread-scoped state so interrupted flows can resume from a saved step instead of starting over. Used together, those layers create a pragmatic split of responsibilities, such as Kafka preserves externally visible state transitions, and the agent runtime preserves internal execution context between steps, pauses, and failures. That is exactly the kind of separation needed when autonomous behavior must remain observable without being reduced to stateless prompt calls. Designing Agent Loops Around Events Once Kafka becomes the backbone, the agent loop changes shape. The entry point is no longer a prompt alone. It becomes a domain event that is enriched, correlated, and converted into a bounded task. Research on ReAct showed the value of interleaving reasoning and acting, and current agent frameworks translate that idea into practical workflows with durable execution, interrupts, and resumable state. The production version of an autonomous agent is therefore less like a chat session and more like a state machine that reasons, uses tools, emits intermediate facts, and pauses when a policy boundary requires approval. A concise stream processor can prepare that task before the model loop begins: Java builder.stream("order-events", Consumed.with(Serdes.String(), orderSerde)) .selectKey((key, event) -> event.customerId()) .join(customerTable, this::mergeContext) .mapValues(this::toAgentTask) .to("agent-tasks"); This pattern keeps context assembly close to the log instead of scattering it across synchronous service calls. Records are keyed by stable business identity, joined with the latest customer state, and emitted as small agent-tasks messages that the runtime can consume directly. Kafka Streams is explicitly intended for stateful processing with joins, event-time semantics, and exactly-once guarantees, so the enrichment stage remains deterministic, replayable, and independent from the model-serving layer. The execution boundary can remain equally narrow: Java @KafkaListener(topics = "agent-tasks", groupId = "claims-agent") @Transactional public void handle(AgentTask task) { AgentDecision decision = agentRuntime.run(task); kafkaTemplate.send("agent-decisions", task.taskId(), decision); } A compact runtime method can express the control flow without hiding it: Java public AgentDecision run(AgentTask task) { AgentState state = stateStore.load(task.taskId()); PlanStep step = planner.next(state, task); if (step.requiresApproval()) return AgentDecision.pause(task.taskId(), "manual-review"); ToolResult result = toolExecutor.execute(step.tool(), step.arguments()); return planner.complete(task, state, result); } This arrangement matters because the runtime receives a prepared task and emits an explicit decision event instead of mutating external systems invisibly. When transactions are enabled, Spring for Apache Kafka supports exactly-once semantics for the read-process-write sequence, and Kafka itself uses idempotent producers plus transactions so retries do not create duplicate log entries. External side effects still need idempotent design when they happen outside Kafka, but the event pipeline itself becomes much more predictable and auditable. Reliability and Control in Production Reliability in event-driven AI systems is usually lost at the edges rather than inside the model call. Kafka’s exactly-once features matter because an autonomous agent often emits decisions that trigger downstream actions, compensations, or audits. Kafka Streams supports exactly-once v2, and exactly-once flows configure consumers with read_committed isolation so aborted transactions do not leak into downstream processing. The event contract matters just as much as the delivery contract. Schema Registry centralizes schemas, validates them, and enforces compatibility modes so producers and consumers can evolve independently. In practice, a stable AgentDecision schema with explicit action type, confidence, explanation reference, and approval status is usually more valuable than a loosely structured JSON envelope because it can be consumed safely by analytics jobs, rule engines, operational systems, and auditors maintained by different teams. Operational control also has to assume malformed input, tool failure, and policy limits. Kafka Connect supports dead letter queues for records that cannot be processed successfully, and Spring Kafka supports dead-letter handling for repeated listener failures. Kafka also supports SASL-based authentication and ACL-driven authorization, which matters when planners, tool executors, and audit services must have different permissions over topics and consumer groups. Combined with interrupt-driven approval workflows from modern agent runtimes, those controls allow autonomous agents to operate inside explicit safety and governance boundaries instead of as opaque background processes. Where This Architecture Fits Best This architecture is strongest when work is asynchronous, stateful, and externally observable. Fraud triage, claims handling, supply chain exception management, field-service coordination, and security operations are better fits than chat-only assistance because the hard problem is not generating a sentence. The hard problem is reacting to a changing stream of facts, correlating them by entity and time, and making bounded decisions with replayable outcomes. Event-driven AI systems with Kafka and autonomous agents are compelling because they treat intelligence as part of an operational stream rather than as an isolated endpoint. The most effective implementations keep the log authoritative, keep schemas explicit, keep agent state durable, and keep irreversible actions observable and governable. That combination produces systems that are not only responsive, but also replayable, auditable, and resilient enough for enterprise use, which is ultimately the threshold that separates a convincing demo from a production architecture.

By Uthej Mopathi CORE