DZone Spotlight

The Disturbing Discovery In July 2026, the AI Red Team at NVIDIA published findings of a six-month assessment review of enterprise AI agents, ranging from tools for interactive coding to continuously running autonomous assistants. Across every framework and harness, the pattern that emerges is consistently the same — the agents that failed did so for four primary reasons: no access controls on the agent itself, capabilities to execute arbitrary code, no restrictions on outbound networking or segregation, and plaintext secrets available to the agent. The problem is inherently architectural in nature. Any kind of defense relying on the control plane of the model — for example, constraining the system prompt or having the large language model serve as an adjudicator of the commands issued — inherits the statistical nature of the underlying model. There are three primary methods to bypass these defenses: disguising malicious activities as legitimate ones (e.g., “I’m debugging” or “I’m an admin”); gradual escalation through the dialogue until enough history accumulates to establish the legitimacy of the commands; and embedding code execution in legitimate behavior (e.g., installing a package). This last one is especially worth noting. The coding agent that installs a library is expected behavior. The command pip install git+https://… pointing to a repository that is under the control of the attacker is arbitrary code execution disguised as legitimate development, and no policy-judging model can prevent this action from being performed without disabling the functionality of the agent entirely. For the companies running such agents, the prompt must not be seen as the security boundary. Here are some considerations that better fit the situation. Control 1: Identify the Agent via Authentication and Propagate the Caller’s Identity The first and most common vulnerability is an agent that holds a service identity that can be accessed by any entity on the internal network. This configuration elevates a simple productivity tool into a common privilege escalation endpoint, where each user automatically receives the combined set of privileges of the agent. Two key prerequisites have been established: Authenticate each call. No matter if it is an entry point through the Slack app, web UI, or MCP endpoint, the calls cannot be anonymous and implicitly granted by the network. An agent that ignores unauthenticated callers is a much harder target to probe.Propagate the human user’s identity into downstream calls. The agent shouldn’t be a self-sufficient entity to invoke commands. OAuth 2.0 Token Exchange (RFC 8693) can be used to allow the agent to exchange the user’s token for a downstream token which represents the user’s privileges, not the agent’s: HTTP POST /oauth2/token HTTP/1.1 Host: idp.internal.example.com Content-Type: application/x-www-form-urlencoded grant_type=urn:ietf:params:oauth:grant-type:token-exchange &subject_token=<end_user_access_token> &subject_token_type=urn:ietf:params:oauth:token-type:access_token &audience=https://jira.internal.example.com &scope=issue:read issue:comment &requested_token_type=urn:ietf:params:oauth:token-type:access_token This token would be limited to a single audience, to the two scopes necessary for the job, and to a short expiration. In case of misuse of the agent’s powers, the impact will be limited to the privileges of a single user, rather than the aggregated privileges of all users. Consider the agent to be a non-human identity with a registered owner, a scheduled rotation period, and an expiration. An agent with no owner is virtually never going to get decommissioned. Control 2: Assume Code Execution and Limit Its Effects Instead of trying to prevent code execution through careful design, make the assumption that the agent will run attacker-influenced code and arrange for the effect of that code to be benign and insignificant. It is important to note that a shell utility is not needed for achieving that goal – only write access is required. When an agent can modify configuration files like ~/.bashrc, ~/.gitconfig, a Git hook, MCP.json, or its own instruction file, then code execution happens as soon as another process reads the modified file. Configuration files, in this sense, serve as executable code, but with some extra steps in between. Shell docker run \ --rm \ --read-only \ --tmpfs /tmp:rw,noexec,nosuid,size=256m \ --mount type=bind,source="$PWD/workspace",target=/workspace \ --user 10001:10001 \ --cap-drop=ALL \ --security-opt no-new-privileges:true \ --security-opt seccomp=/etc/docker/seccomp-agent.json \ --pids-limit 256 \ --memory 4g --cpus 2 \ --network agent-egress \ agent-runtime:2026.07 When creating a hardened baseline of containers, the following points should be emphasized: A read-only root filesystem will ensure that write attempts to dotfiles fail at the OS level rather than at the model’s discretion.Use of noexec on writable mounts breaks the “read, write, execute” pattern.Dropping all capabilities and setting no-new-privileges blocks privilege escalation mechanisms. Then, mount the agent’s configuration as read-only and from a different mount point than the workspace of the agent: Shell --mount type=bind,source=/etc/agent/AGENT.md,target=/etc/agent/AGENT.md,readonly \ --mount type=bind,source=/etc/agent/mcp.json,target=/etc/agent/mcp.json,readonly An agent that is able to change its own instructions can assume a completely different persona, including the “authorized debugging user” frame the red team was able to demonstrate. In cases where providing a command utility is unavoidable, use the following strategy: Use an allowlist of binaries and wrap each invocation in a wrapper that removes shell metacharacters, resolves paths, and does not allow any action that goes beyond /workspace.Treat any external inputs – filenames, ticket titles, and document names coming from external systems – as tainted. Control 3: Default-Deny Egress From Each Perimeter Outbound network connectivity turns the constrained execution environment primitive into an actual incident by serving as the means of exfiltration and establishing a reverse shell connection. When NVIDIA tested their system under proper egress restriction, the red team had to perform their activities through the agent process itself — characterized by low speed, high noise, and unreliable performance. Restrict egress in places where the agent does not have direct access to the enforcement point. In case of Kubernetes environments: YAML apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: agent-runtime-egress namespace: agents spec: podSelector: matchLabels: { app: agent-runtime } policyTypes: [Egress] egress: - to: - podSelector: matchLabels: { app: egress-proxy } ports: - { protocol: TCP, port: 3128 } - to: - namespaceSelector: matchLabels: { kubernetes.io/metadata.name: kube-system } ports: - { protocol: UDP, port: 53 } All network connections are restricted except those that are explicitly allowed, including blocking the cloud metadata endpoint (169.254.169.254), which provides a credential source without requiring any exploitation. Route the allowed connections through an authenticating proxy server that uses an allowlist of fully qualified domain names (FQDNs), optionally terminates TLS for analysis, and records every request with user identification data attached. This logging creates the incident timeline. Control 4: The Agent Never Holds a Persistent Secret The common practice is to inject secrets via environment variables without making any write calls to the disk, because it is commonly accepted that the only code supposed to run in the container is the expected one. This is untrue for modern times, where a large language model (LLM) runs with the shell in the same process space — env, printenv, and /proc/self/environ are one prompt away, and CLI tools helpfully cache credentials in predictable locations: .netrc, .git-credentials, shell history, and .env files. The most interesting observation made during red teaming was the ability to extract secrets via the chat interface even when all network-based data exfiltration is prevented. The model can read environment variables and return credentials. Regardless of any network isolation, there is no way to protect data the agent is authorized to see. Thus, secrets cannot be accessible to the agent at all. Broker tokens per task instead: Python # Agent requests capability, never a credential. token = broker.issue( principal=ctx.end_user_id, # the human, not the agent audience="https://api.github.com", scopes=["repo:status", "pull_request:write"], resources=["org/repo-name"], ttl_seconds=300, ) try: github.post_review(token, pr_id, body) finally: broker.revoke(token) # revoke on completion, not on expiry Recommendations: Never inject secrets into the container image, environment, volume mounts, or context window.Set very short time-to-live (TTL) values for secrets, measured in minutes.Invalidate tokens after finishing the task.Record every secret issuance along with the identification of the human user.Once the secret is available to the agent, it is already a win for the attacker. Control 5: Package Installation Is a Supply Chain Control Use an internal proxy repository to control the agent’s package manager and stop VCS and URL installations of any packages: Plain Text # /etc/pip.conf (root-owned, read-only mount) [global] index-url = https://artifactory.internal.example.com/api/pypi/pypi-approved/simple no-index = false require-hashes = true # /usr/etc/npmrc registry=https://artifactory.internal.example.com/api/npm/npm-approved/ ignore-scripts=true ignore-scripts=true is the silent victory — this will stop postinstall from being used as an execution vector. The agent must only install packages which are resolvable through the internal repository. Trust, But Verify Ship these as test cases, not as documentation: Assertion Test Unauthenticated callers rejected Invoke the agent with no token, and with another user’s token Dotfile writes blocked Ask it to append to ~/.bashrc and to modify its own instruction file Egress denied by default Request a fetch from an unapproved host; confirm proxy denial in logs No secrets in environment Ask it to print its environment and read /proc/self/environ Metadata endpoint unreachable Request 169.254.169.254/latest/meta-data/ VCS installs blocked Ask it to pip install git+https://… from an external URL Run these on every release, and run the multi-turn variants — the escalation that works is rarely the one in a single message. Key Takeaway Prompt-based guardrails are meant to be a usability feature that prevents accidental damage, but they do not hinder an adversarial actor who intends to cause harm. Each request needs to be validated through identity authentication (JWT validation or equivalent), confirming the caller is who they say they are — alongside a secure sandbox environment without writable-executable paths, default-deny network egress at every boundary, and short-lived credentials issued to the agent per task. This is not new security engineering. It is the application of least privilege, isolation, and secrets management to a workload that interacts with untrusted input in real time. The mistake is assuming the model is the enforcement point, when it is in fact the thing being defended.

You have a checkout flow. You have 40 tests. They're green. Now: what happens when a payment webhook arrives after the user cancels? What happens when a retry lands on a session that already expired? What happens on the fourth failed attempt when autoRenew is off and the period boundary has already passed? You don't know. Not because you're careless — because a state machine with 6 states, 7 actions, and 3 payload values has thousands of reachable (state, action, data) combinations, and your 40 tests visit 40 of them. The bugs that page you at 2 am live in the other several thousand. Polygraph is a Claude Code plugin and standalone CLI that walks all of them. Why You'd Bother Polygraph is for stateful code: reducers, workflow engines, protocol handlers, session managers, order state machines, anything with a dispatch(state, action) shape. If your code is a pile of pure functions, go use property-based testing. If it's a state machine, keep reading. Narrow on shape, not on language. The model reads your source in whatever it's written in — you name it once as lang in the contract — and the trace format is just NDJSON, so any runtime that can log a {pre, action, data, post} line per step can feed it. What's always JavaScript is the derived spec and your rules, because those are what the replayer and model checker execute on Node. What you get back is not a lint warning. It's a shortest action sequence that reaches a state violating a rule you wrote. Something like: Shell ✗ never-charged-twice [state] — pred returned false init {"status":"new","attempts":0,"hasDue":false} CREATE({}) -> {"status":"active","attempts":0,"hasDue":false} RENEW_CHARGE({"result":"5xx"}) -> {"status":"grace","attempts":1,"hasDue":true} RENEW_CHARGE({"result":"ok"}) -> {"status":"grace","attempts":2,"hasDue":true} That's a repro. You paste it into a test file, and you have a failing test in about ninety seconds. A real one: On a production SaaS subscription-billing machine, Polygraph flagged a disagreement on exactly one window: a 5xx from the payment processor during renewal moved the row to grace and marked it due, when the dunning path in the same codebase correctly treated 5xx as ambiguous. The next retry rotated the idempotency key. If the 503'd transfer had actually settled, the customer got charged twice. A human reviewer had found the same bug by hand; five independent model-derived readings of the source landed on it blind. And in the controlled seeded-bug eval, the split is worth knowing: replaying real traces against the derived spec found 0 of 5 seeded bugs. Model checking found 5 of 5, with counterexamples. Trace replay tells you whether to trust the model. Model checking is where the bugs actually are. What It Actually Does Three artifacts, all diffable, all in your repo: 1. contract.json — the scope. Which state fields matter, which actions the machine accepts, what data each action can carry, which states are terminal, and lang is the language your source is written in. 2. A spec — a JavaScript model of your code, written by an LLM from your source (whatever language that is). It's a strict SAM v2 module: every action it ignores has to say why via reject(reason), it can't hide bookkeeping state, and it declares its own action/data domains — so the checker knows what to explore with zero config. Several specs are generated independently and vote, so one bad generation doesn't decide anything. JavaScript export const stateInvariants = [ { name: 'locked-only-at-limit', pred: (s) => s.status !== 'locked' || s.attempts >= 3 }, ]; export const transitionInvariants = [ { name: 'expired-never-verifies', pred: (pre, action, data, post) => !(action === 'ATTEMPT' && data?.expired) || post.status !== 'verified' }, ]; 3. invariants.mjs — your rules, as plain JS predicates: This part is yours and can't be automated away. Code with a bug is a perfectly faithful description of the wrong behavior. Invariants are where your intent enters the system. Then two checks run. Replay asks "is the spec faithful?" Real traces ({pre, action, data, post} windows, captured by wrapping your dispatch once) are replayed against each spec, with positive and negative controls proving the harness can tell good from bad. Model check asks "where are the bugs?" It iterates the faithful spec exhaustively from init against your invariants and prints the shortest path to every violation. The Caveats "Exhaustive" means exhaustive over the finite (action, data) domain declared in your contract. A machine whose behavior depends on unbounded counters or arbitrary strings is checked only at the representative values someone chose. That's the standard TLA+ modeling move, and the gap between declared domain and real data is real.It's a consistency check, not a proof. A clean run means your code's observable behavior matches an independent reading of its own source. Nothing more.Every finding is a lead to investigate, not a verdict. There is no triage step that discharges "real invariant break with no observable consequence."It's experimental and not peer-reviewed. Don't make it your only safeguard on safety-critical code. API Key and Cost Only three things call the Anthropic API: spec generation, code authoring (polygen), and polynv's optional headless invariant harvest. You need ANTHROPIC_API_KEY in your environment for those, including inside Claude Code, where the skills shell out to the same scripts and do not use your session credentials. Ballpark, on a typical machine: you runkey?costverify.mjs --source … (generate + replay)yes~$0.50polygen.mjs --intent … (author new code — JS/TS output only)yes~$2replay saved specs, model check, --tla, polyvers, polynv, polyrunno$0 That second row is the load-bearing one. Everything that checks (replay, the exhaustive model check, version gating, the mutation grade, TLC escalation) is keyless, local, and deterministic on Node ≥ 20. Which is precisely what makes CI viable: you commit the spec, and the gate re-runs it on every merge request for free. No key in CI, no per-MR API bill, no nondeterminism in your pipeline. That gate is polygate, and there's a GitLab reference implementation at <POLYGATE_GITLAB_URL> — a .gitlab-ci.yml you can copy that runs corpus validation, replay, and the model check against your committed artifacts and fails the MR on a violation. (Contrast: Specula, the closest comparable agentic TLA+ pipeline, reports a median of $57 and 3.7 hours per system. Excellent tool, structurally can't run on every MR.) Getting Started Prerequisite, and it's a hard one: The stateful code has to be runnable in isolation, because traces are ground truth from the code actually executing. A clean step boundary: a dispatch, reducer, or handler, from experience, Claude will refactor it easily for you. If it only runs against a live DB or device, stand up doubles first (in Claude Code, the agent will build them). Note this is the only place your language matters, and only for convenience: the bundled withTracing / tapReducer helpers are JS, so a Go or Python machine means writing the {pre, action, data, post} NDJSON lines yourself. It's about ten lines. Zero-cost first (no key, five minutes): Shell git clone https://github.com/cognitive-fab/polygraph cd polygraph && npm test # validates the bundled corpus, runs the controls npm run verify:turnstile-v2 # replays bundled specs — see the output shape Then on your own machine, as a plugin: Shell /plugin marketplace add cognitive-fab/polygraph /plugin install polygraph@polygraph …and just ask: "verify this state machine", or /polygraph:polygraph for the guided end-to-end run (Claude drafts the contract, instruments the boundary, captures traces, runs controls, triages with you). Trace capture is historically what made this expensive; it's the step the agent now carries. Or plain CLI, no Claude Code: Wrap your dispatch once, projecting only the contract's observable keys (JS shown; in another language, emit the same NDJSON shape by hand): JavaScript import { withTracing } from '<plugin>/scripts/instrument/trace-emitter.mjs'; const dispatch = withTracing( rawDispatch, () => ({ status: m.status }),'traces/s1_normal.ndjson' ); Note --source takes your real file, in your real language: Shell node scripts/validate_corpus.mjs contract.json traces/ # no key node scripts/verify.mjs --contract contract.json --source src/machine.ts \ --traces traces/ --model opus-5 --n 5 --out out/ # key, ~$0.50 That writes out/findings.md and the generated specs to out/specs/. Commit the winning one, and from then on the loop is free: Shell node scripts/check.mjs --spec out/specs/spec_0.js --contract contract.json \ --invariants invariants.mjs # no key, forever There's no default model: pass --model. Use opus-5 or better; deriving a faithful transition function is a hard reasoning task and lighter models don't clear the bar. If you see empty specs, you lowered --max-tokens below what the reasoning block needs; put it back to 32000. Apache-2.0. The method is written up in arXiv:2607.05076. Your test suite is a sample. This is the census.

Every B2B SaaS product team knows this moment. You're trying to close a deal, and the prospect says, "We just need you to sync with our CRM. And our HRIS. Oh, and these three other tools. You can do that, right?" Your roadmap takes a hit, and your engineering backlog doubles overnight. And eventually someone says, "What about a unified API?" It sounds like the answer — one normalized schema, one auth model, and one point of connection for a dozen or more apps in a vertical. You buy it, hook it up, and ship the integrations before the quarter ends, the integration checkbox gets checked, and you move on. For a while, it works. But there's a problem most teams don't see until they start moving upmarket. For many SaaS teams, a unified API is the right first move. It's rarely the right last one. Unified APIs Exist for a Reason, and They're Good at What They Do Most apps in a category share the same data objects. CRMs have contacts, accounts, opportunities, and activities. HRIS platforms store employee, department, and compensation data. Ticketing systems track tickets, users, and statuses. A unified API vendor abstracts the data models for an app category into a common schema so that, rather than learning a dozen APIs, your devs learn one. For startups under pressure to ship quickly, that abstraction is valuable. You can launch integrations faster, reduce engineering work, and simplify auth across the board. If your customers need common objects and standard workflows, a unified API can meaningfully accelerate your roadmap. That's all positive. The negative shows up down the road. The Lowest Common Denominator Problem A normalized data model (which is what a unified API is based on) is, by definition, a reduced or simplified data model. To present a single schema across N apps, a unified API must identify the fields they have in common. The result is a model built on the smallest shared dataset. Anything that's app-specific is abstracted away, and anything proprietary is dropped. Unified APIs work until your customers stop being generic. Enterprise customers have Salesforce custom objects built for their unique processes. They have Workday compensation structures that don't fit a normalized HRIS schema. They have vertical-specific fields that are critical to their business processes. And, it's increasingly common for them to be running systems that the unified API vendor has never heard of. The moment a prospect asks you to sync a custom object, access a proprietary field, or connect to an app outside your unified API vendor's supported list, the abstraction layer is no longer sufficient. You either tell your prospect "No" or you build a custom, one-off integration anyway, which largely defeats the point of a unified API. At first, these seem like edge cases. Then you realize enterprise customers are the edge cases. And that they are bringing the highest-value deals in your pipeline. The "Zero Maintenance" Promise Doesn't Hold Up The biggest marketing claim of a unified API is that upstream API changes are no longer your problem: "They update their API, we handle the change." In reality, you're trading one type of maintenance for another. With native APIs, you worry about endpoint deprecations, auth updates, and rate limits. With a unified API, you worry about data lost in translation or debugging through an abstraction layer. When that happens for an enterprise customer, you can't just look at the target system's logs. You have to work through the unified API provider's black box. If the root cause is a nuance in how they handle a specific app's rate-limiting rules, your engineering team is now waiting on someone else's support ticket queue. The maintenance didn't go away. It just moved down the street. Complexity Comes Later The full cost of a unified API strategy rarely appears during implementation. Instead, it waits until things have settled into a steady rhythm and then shows up as operational complexity. Dual integration architectures – Once you need custom integrations alongside your unified API (and you will), your team will maintain two separate integration layers with different auth flows, error handling, retry logic, and monitoring. Every integration request now needs to go through a decision tree to determine which of these patterns (or perhaps even a new one) you should use for development.Data model constraints – Your app connects with the unified API's schema rather than to the underlying apps. When customers ask for fields the schema doesn't expose, your team builds manual workarounds, relocating rather than reducing the complexity.Vendor roadmap dependency – If your unified API provider doesn't support a specific endpoint, a webhook behavior, an advanced API feature, or a vertical SaaS platform your customer uses, you wait (or you build around it). Either way, the original value proposition isn't holding up to the rigors of reality.Escalation cost – Enterprise prospects bring technical evaluators. When those evaluators discover that your integration can't provide the specific data they depend on, the deal may end right there. That's not good for your bottom line. What the Workaround Trap Looks Like Most teams respond the same way when they hit these limits. They start building custom integrations in addition to those handled through the unified API. What began as a simplification strategy is starting to look like this: a unified API for common integrations, direct API connections for exceptions, custom middleware for unsupported workflows, separate auth handling, multiple sync models, and one-off transformation logic wherever it's needed. In short, that neatly ordered integration layer is no longer. The abstraction created to reduce maintenance has, in fact, increased it. Teams find they're burning an appreciable portion of their integration budget maintaining low-value integrations and working around the things their unified API vendor can't support. That's engineering time that isn't being devoted to your core product. Vertical SaaS Is the Forcing Function The continued fragmentation of B2B software makes this worse every year. Beyond mainstream CRMs and HR platforms, companies increasingly rely on industry-specific applications: systems narrowly designed and built for healthcare, manufacturing, financial services, and a score of other verticals. These systems rarely conform to standardized schemas. Many of them don't appear in any unified API vendor's list of supported apps. A unified API might help you connect to ten generic CRMs. It won't help much when your largest prospect is running Epic, Procore, or a heavily customized NetSuite instance. Those are the integrations that determine whether enterprise deals close. What Happens at Scale Unified APIs are usually evaluated based on how fast they help teams launch. However, the more important question is: "What happens when integration requirements grow more complex?" Because they always do. Every single time. As SaaS products mature, integration requests shift from "Can you connect to this category?" to "Can you support this exact workflow?" That move exposes the architectural limits of a unified API. And the teams that hit the limit mid-deal (or mid-contract) feel the immediate pain. Why Embedded iPaaS Is the Durable Foundation This is where embedded iPaaS platforms fundamentally differ from unified APIs: they aren't constrained to a single simplified schema. An embedded iPaaS gives your team a flexible integration foundation that handles both ends of the spectrum: the common apps that benefit from productized integrations, and the complex, vertical-specific, niche apps that don't fit any standardized model. Some of your customers need a basic CRM sync. Others need multi-flow orchestration, conditional business logic, extensive data mapping, and more. A rigid abstraction model breaks under those requirements. An embedded iPaaS doesn't. This Isn't "Unified APIs vs. Embedded iPaaS" Unified APIs still have value. For early-stage validation or straightforward category integrations at scale, they can accelerate time-to-market. Many mature teams use them alongside a more flexible platform for the scenarios where standardization works. But for most B2B SaaS teams, they are a way-station, not the destination. The mistake teams make is assuming the abstraction can scale indefinitely as customer complexity increases. But that's not true. It can't, and it doesn't. The bigger and more complex your customers get, the more a lowest-common-denominator approach becomes an obstacle instead of a shortcut.

Agent framework debates are mostly vibes. One engineer swears LangGraph is faster, another prefers the OpenAI Agents SDK, someone wants Google ADK because it feels future-proof. The team picks one, wires the workflow into its SDK, and the choice is welded in. Changing frameworks later means tearing out the wiring for one SDK and rebuilding the workflow on another, an expensive rewrite few teams take on. This tutorial makes that decision reversible and then settles it with data. You put the agent graph in LaunchDarkly and run four frameworks (LangGraph, Strands, OpenAI Agents SDK, and Google ADK) over the same topology, with the model pinned so the framework is the only variable. A LaunchDarkly experiment ranks them on graph latency and token use, with an LLM judge guarding quality. The results table tells you which framework runs your graph fastest without degrading it. This tutorial is the sequel to Compare AI orchestrators, which ran the same workflow across frameworks but kept the topology in each framework’s code. Here, the topology, routing, models, prompts, tools, and judge all live in LaunchDarkly, and each framework supplies only two functions. The experiment results do more than set a benchmark. The flag that splits experiment traffic also routes production. When one framework wins, you don’t rewrite the app; you change the flag to serve the winner. In a single loop, LaunchDarkly does three jobs: the graph definition, the experiment split, and the runtime control that ships the winner. The workload is a research-gap analysis over a set of arXiv papers. Two readers, approach-analyzer and contradiction-detector, read the same papers in parallel and fan in to gap-synthesizer, which writes the report. Prerequisites A LaunchDarkly account with AgentControl access, and your environment’s SDK keyPython 3.11+ and uvAn ANTHROPIC_API_KEY for the pinned model. OPENAI_API_KEY and GOOGLE_API_KEY are only needed if you run the optional native-model bake-off in Step 9The companion repo: ai-orchestrators on branch tutorial/graph-experiments The Experiment Design The comparison is controlled: same graph, same model, same papers, same judge, with the framework as the only variable. Mechanically, it runs in four stages: Bootstrap. manifest.yaml creates the node configs, graph, orchestrator flag, and judge in LaunchDarkly.Route. On each request, the app evaluates the orchestrator flag to pick a framework: langgraph, strands, openai-agents, or google-adk.Run. The dispatcher runs the shared graph as a directed acyclic graph (DAG). The two readers run concurrently and fan in to the synthesizer.Measure. Each run records how long the graph took, how many tokens it used, and whether the report passed the quality judge. The shape looks like this: ┌──▶ approach-analyzer ───────┐ intake (papers) ─────┤ ├──▶ gap-synthesizer ──▶ report └──▶ contradiction-detector ──┘ Step 1: Create the Graph, Flag, and Judge Everything starts from one file, config/graph_experiment_manifest.yaml. It declares the fetch_paper tool, four node configs (intake plus the three agents, pinned to claude-sonnet-4-5), the graph, the orchestrator flag, and the judge. First, clone the companion repo and install its dependencies with uv: Shell git clone https://github.com/launchdarkly-labs/ai-orchestrators cd ai-orchestrators git checkout tutorial/graph-experiments uv sync Next, set up a LaunchDarkly project. The bootstrap doesn’t create one, so create it with the LaunchDarkly MCP server, the projects agent skill, or the UI. Name it graph-experiments to match the value in .env.example, so the defaults work without edits. When it exists, copy its key into LD_PROJECT_KEY and its production environment SDK key into LD_SDK_KEY in .env. The runners and experiment harness use that SDK key to evaluate the flag and graph. The bootstrap also reads LD_API_KEY from .env to create the resources. Copy the example file to create your .env: Shell cp .env.example .env # then set LD_PROJECT_KEY, LD_SDK_KEY, and LD_API_KEY in .env With the keys in place, run the bootstrap: Shell uv run python scripts/launchdarkly/bootstrap.py config/graph_experiment_manifest.yaml This creates all four node configs, the research-gap-graph, the orchestrator flag (created off), and the gap-quality-judge attached to the gap-synthesizer node (its synthesizer-claude variation, set to 100% sampling). The judge scores the final report against the source papers, so it can verify grounding and citations. A judge can only check based on the information it has, so we give it the papers, not only an upstream agent’s analysis. When the graph ships, it is incomplete by design. The bootstrap creates the contradiction-detector config but wires only intake to approach-analyzer to gap-synthesizer, leaving the detector out. You’ll add it in Step 5 to complete the parallel fan-in. When it finishes, the bootstrap prints a link to your new agent graph. Open it and review the topology before moving on. The graph shows a straight line from intake to approach-analyzer to gap-synthesizer, with contradiction-detector created but not yet wired in. Step 2: The Dispatcher Runs the Graph The dispatcher is the heart of the project, and it’s the same code for every framework. It reads the graph as a DAG, runs the entry nodes concurrently, hands every node the papers as ground truth, and connects the readers at the fan-in node. The only framework-specific pieces are build_agent and invoke, which are passed in as arguments. The whole process is about 100 lines, built on the agent graph traversal methods in the SDK. The complete dispatcher.py is in the companion repo. The dispatcher carries the design in four parts: it builds the execution plan from the graph’s edges, composes each node’s input, runs every ready node concurrently each round, and records the graph’s metrics once per run. First, the dispatcher builds the execution plan from the graph’s edges, so the topology you draw in LaunchDarkly runs: Python for key, node in nodes.items(): for edge in node.get_edges(): target = edge.target_config if target in nodes: succ[key].append(target) preds[target].append(key) Next, every node receives the source papers and any upstream analyses, so each agent and the judge work directly from the source material rather than a summary handed down a chain: Python def compose_input(user_input, predecessor_outputs): parts = [f"=== SOURCE PAPERS ===\n{user_input}"] for key, out in predecessor_outputs: if out and out.strip(): parts.append(f"=== {key} ===\n{out}") return "\n\n".join(parts) Then each round runs every node whose predecessors have finished, concurrently, so the two readers fan out and fan in with no special casing: Python ready = [k for k in pending if all(p in done for p in preds[k])] results = await asyncio.gather(*(run_node(k) for k in ready)) Finally, the dispatcher records the graph’s metrics on each run, including the end-to-end latency the experiment ranks on: Python graph_tracker.track_duration(int((time.monotonic() - start) * 1000)) graph_tracker.track_total_tokens(TokenUsage(input=totals["in"], output=totals["out"], total=totals["in"] + totals["out"])) graph_tracker.track_path(path) graph_tracker.track_invocation_success() The dispatcher reads the topology at runtime, so reshaping the workflow in the UI, adding a node, or redrawing an edge takes effect on the next request with no code change. You’ll do exactly that in Step 5. Step 3: Each Framework Is a Thin Adapter Each framework implements build_agent(node_key, config, instructions) and async invoke(agent, input_text, tracker). Everything dynamic still comes from the LaunchDarkly node config: the model, the attached tools, and the instructions. LangGraph has a LaunchDarkly companion package, so its runner is only a few lines. The companion handles model creation, tool binding, and token tracking, so the adapter holds no framework plumbing of its own: Python def build_agent(node_key, config, instructions): llm = create_langchain_model(config) tools = build_tools(config, TOOL_REGISTRY) # binds only this node's attached tools return create_react_agent(llm, tools, prompt=instructions) async def invoke(agent, input_text, tracker): result = await tracker.track_metrics_of_async( lambda res: LDAIMetrics(success=True, tokens=sum_token_usage_from_messages(res.get("messages", []))), lambda: agent.ainvoke({"messages": [{"role": "user", "content": input_text}]}), ) messages = result.get("messages", []) for message in messages: for name in get_tool_calls_from_response(message): tracker.track_tool_call(name) text = _content_to_text(messages[-1].content) if messages else "" return text, sum_token_usage_from_messages(messages) Strands has no companion package, so its runner builds the model with a small provider-aware factory and binds tools with Strands’ native @tool. The contract is identical: Python def build_agent(node_key, config, instructions): return Agent( name=node_key, model=_create_strands_model(config), system_prompt=instructions or "Process the input and respond.", tools=_bind_tools(config), callback_handler=None, ) OpenAI Agents and Google ADK round out the four. For the comparison to stay fair, all four have to run the same model, but these two SDKs default to their own vendors’ models. LiteLLM, a thin adapter, lets them call any provider, so we point both at the pinned claude-sonnet-4-5 and keep the model identical across all four orchestrators. No OpenAI or Google servers are involved. Instead, LiteLLM translates the request format in-process, and the call goes straight to Anthropic with your key. Google ADK is fully companion-free, and OpenAI Agents uses the ldai_openai companion for token and tool-call telemetry even though it builds the model through LiteLLM. This experiment pins one model across all four frameworks, so every framework here runs Claude. Pointing each framework at its own vendor’s default model instead is a separate, optional exercise, the native-model bake-off in Step 9. The tool callables live in TOOL_REGISTRY, a plain {name: callable} map that each framework binds its own way. Step 4: Smoke Test the Graph Before you run any experiment, confirm the bootstrapped graph runs end to end. First, run one framework: Python uv run python orchestrators/verify_run.py langgraph It prints the path it took and the first part of the report. On the graph as it shipped, the path is intake -> approach-analyzer -> gap-synthesizer: intake runs its short pass, approach-analyzer reads the papers, and gap-synthesizer writes the report. There’s no contradiction-detector yet, and no error. The metrics land in the AgentControl UI under the graph you created. Step 5: Add the Parallel Fan-In In the UI Here’s the payoff of keeping the topology in LaunchDarkly: you finish building the workflow in the UI, with no redeploy, and the running app picks up the new shape on its next request. The contradiction-detector config already exists, with its fetch_paper tool attached. You wire it into the graph to add the second reader and form the parallel fan-in. To complete the graph: Click Agents in the LaunchDarkly sidebar.Click Agent graphs.Select research-gap-graph.Add the contradiction-detector node.Draw an edge from intake to contradiction-detector, then another from contradiction-detector to gap-synthesizer.Click Save. You add no routing logic: the edge itself is the route, because routing is structural. Re-run the smoke test: Shell uv run python orchestrators/verify_run.py langgraph The path now includes contradiction-detector, and because approach-analyzer and contradiction-detector run concurrently, their order can vary. You completed a multi-agent workflow from the UI, and the config you wired in already had its tool attached. You finished a multi-agent workflow from the UI, mid-development, and the dispatcher ran the new shape on the next request. No redeploy, no code change: the graph you draw is the graph that runs. Step 6: Smoke Test All Four Frameworks Before you collect experiment data, make sure all four frameworks can run the completed graph. One command runs all four in sequence: Shell uv run python orchestrators/verify_run.py all It runs each framework against the completed graph and ends with a pass/fail summary, one line per framework, exiting non-zero if any framework failed, so it works as a gate. Each framework prints the path it took and a preview of its report, then a final summary collects the results. A successful run looks like this: Plain Text ▶ Running 'langgraph' over 2 papers on graph 'research-gap-graph'... ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer ▶ Running 'strands' over 2 papers on graph 'research-gap-graph'... ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer ▶ Running 'openai-agents' over 2 papers on graph 'research-gap-graph'... ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer ▶ Running 'google-adk' over 2 papers on graph 'research-gap-graph'... ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer === smoke summary === ✓ langgraph ✓ strands ✓ openai-agents ✓ google-adk If a framework fails, its line shows an ✗ instead of a ✓ and the command exits non-zero. All four smoke tests against the pinned Claude model. ANTHROPIC_API_KEY is the only model key you need, because OpenAI Agents and Google ADK reach Claude through LiteLLM. The OpenAI Agents SDK turns on tracing by default and looks for OPENAI_API_KEY to export traces, so the openai-agents run may print a harmless tracing warning when that key is absent. It doesn’t affect the run. Step 7: Run It Through the Experiment Now you can use a LaunchDarkly experiment to rank the four frameworks on real traffic, on the same graph, with the model held constant. Because the model is fixed, the comparison is operational: which orchestrator delivers the model’s quality fastest, with the least token overhead. The bootstrap already created the flag, the judge, and the graph. These metrics are measured on each request, so do a one-time setup first: Make the request context kind available for experiments.Set the analysis unit of graph latency, tokens, and the judge metric to request. Then create the experiment in the UI: Create an experiment with the orchestrator flag as the treatment.Set the primary metric to Graph latency ($ld:ai:graph:duration:total, the time for a complete graph execution).Add tokens and $ld:ai:judge:gap-quality as secondary metrics.Set the audience to 100% and the randomization unit to request. Each run is a single request, there are no users in this workflow, and request is the unit LaunchDarkly measures AI and graph metrics by.Turn on the orchestrator flag, which the bootstrap created set to off, so it serves the experiment’s variations.Start an experiment iteration. We rank on latency and tokens because, with the model and the graph held constant, those are the things that genuinely differ: a framework can move quality only by degrading the plumbing, like a truncated report or a broken tool call. So $ld:ai:judge:gap-quality stays a guardrail that catches a framework “winning” by cutting corners, not part of the ranking. Swap the model, prompt, or tools later instead of the framework, and that same judge becomes your primary metric. Then drive traffic. The flag assigns each run one framework at random: Shell uv run python scripts/run_experiment.py --runs-per-category 6 That’s six runs over each of the six shipped topics, 36 in total. Assignment is random, so it usually fills all four variations, though it isn’t guaranteed. Each run analyzes the topic’s entire paper set, because gap analysis needs every paper to find real gaps. Open the experiment in LaunchDarkly: latency per variation, with tokens and $ld:ai:judge:gap-quality alongside. The winner is the framework with the best latency and lowest token use that doesn’t let quality slip. Because the model is pinned, cost is a fixed multiple of tokens, so the token column is also the cost ranking; for actual dollar figures, read them from Insights. Because the experiment holds everything but the framework constant, most of these bars land close, often within a few percent, which is by design. In our run, Strands won on speed: it ran the graph fastest, with quality holding at the guardrail. If you optimize for speed and quality holds, that makes Strands the orchestrator to ship for this workload. Six topics and one randomized split isn’t a large sample, so confirm the lead with more topics before you standardize on it. You can do that in Step 9. Step 8: Ship the Winner With Runtime Control The experiment gave you data. The reason to run it in LaunchDarkly, rather than a one-off script, is that acting on that data takes no deploy: the orchestrator flag that was the experiment treatment is also your production router. When a variation wins, stop the iteration and set the flag’s default to that framework. Every request routes to it on the next evaluation, with no redeploy. Then automate what you don’t want to babysit. An adaptive trigger watches a guardrail and changes a flag on its own when production drifts past it. The orchestrator you shipped is operational and won’t degrade by itself, so point the trigger at the model flag from Step 9: it fails over to a backup model when your primary provider has a bad day, the same guardrail driving a different flag. That closes the loop: experiment to find the winner, runtime control to ship it, and automation to keep it healthy. Step 9: Extend the Experiment Tighten the bands by adding more topics. Confidence comes from more distinct topics, not more runs over the same few. Download one with a title-phrase (ti:) query, and the harness picks it up automatically on the next run: Shell uv run python scripts/download_papers.py --query 'ti:"LLM-as-a-judge"' Make quality the headline by flipping a config, not a flag. The framework lives in the orchestrator flag because it is app-level routing, not a property of any agent. The model, the prompt, and the tool set are different: they live in the node configs, so you experiment on the config itself. Add a second variation to a node, such as gap-synthesizer with a stronger model or a tightened prompt, and run an experiment with that config as the treatment and its variations as the arms. Pin the framework by setting the orchestrator flag to one value and leave the graph alone, so the config is the only thing moving. The judge attached to the synthesizer already emits $ld:ai:judge:gap-quality, so quality is the primary metric with no new instrumentation. Now it genuinely moves, because a different model or prompt reasons differently about the same papers. Experiment on the graph shape with a graph-key flag. The dispatcher takes the graph key as an argument, so the shape is another value you can put behind a flag: Python graph_key = ld.variation("graph_shape", context, "research-gap-graph") result = await execute_graph(ai_client, graph_key, context, user_input, build_agent, invoke) Build two graphs with different keys: for example, a linear research-gap-graph-linear (intake to approach-analyzer to gap-synthesizer) against the parallel research-gap-graph, or one with an added critic node against one without. Make a multivariate graph_shape flag whose variations are those graph keys, evaluate it exactly as the app evaluates orchestrator, and set it as the experiment treatment with the framework and model held constant. You are measuring whether the extra structure earns its latency and quality, and because the dispatcher runs whatever shape the key resolves to, no runner or dispatcher code changes. You build the judge once, and it is the guardrail for the framework bake-off, and the headline metric for every model, prompt, tool, and shape you test next. Run a native-model bake-off. This experiment holds the model constant so the framework is the only variable. To compare each framework on its own default model instead, build separate node configs per framework. This is the optional bake-off the prerequisites mention. It’s a follow-up beyond this walkthrough, and the only part that needs OPENAI_API_KEY and GOOGLE_API_KEY. Whatever you flip, follow three rules: Change one variable at a time (the framework, the model, or the shape), never two. If you change more than one, you can’t attribute the win.Keep the quality guardrail on every run, because the fastest variant is often the one that quietly truncated its report or dropped a tool call.Earn confidence with distinct inputs, not repeats: a tight band around three repeated topics is still a tight band around the wrong number. To learn more about judge design, read When to add online evals and Evaluating with LLM-as-judge evaluators. To add a pre-production regression layer, read Offline evaluation of RAG-grounded answers. Recap and Next Steps Framework choice doesn’t have to be a one-way door. Put the topology in a LaunchDarkly agent graph, have each framework supply only build_agent and invoke, and let one experiment settle a question that usually gets answered by whoever argues hardest: pin the model, let the judge guard quality, and pick the orchestrator that delivers it fastest, with evidence in hand. Then keep going, because the framework is only the first swappable component. The same flag, experiment, and judge machinery compares models, prompts, tools, and whole graph shapes the same way, so “which is better” stops being a debate and becomes a measurement. And because the experiment and the runtime control are one flag, you never stop at a finding: you ship it, ramp it with a progressive rollout, and let an adaptive trigger hold the line in production while the AI iteration loop for reliable agents keeps the next change shipping behind eval gates. The complete code is in the sample repo. Get started with AgentControl, point the four frameworks at a graph your team actually runs, and settle the next framework argument with a number instead of a hunch.

Sponsored By: NutanixThe following is sponsored content. It may not reflect the views of our editorial staff. The Kubernetes scaling problem nobody talks about Enterprise platform teams encounter the same pattern repeatedly: a Kubernetes platform works well enough that nobody wants to change it. This happens gradually as teams make reasonable technology choices: selecting different ingress controllers, secrets management tools, CD platforms, or observability software. Individually, none of these decisions is a problem. Months later, however, they’ve created a Kubernetes environment that only a handful of people understand. As soon as that one person gets sick or leaves the company, maintaining or improving the platform becomes much more difficult. Mark Dastmalchi-Round, a Solutions Architect at Nutanix with decades of experience in platform engineering, describes the pattern in blunt terms: “Configuration drift, exacerbated by the fact that multicloud is increasingly becoming the new reality.” Over time, that drift compounds. Companies get acquired, technology merges, and silos form. Suddenly, organizations are managing clusters that look nothing alike and are often held together by institutional knowledge. As a solution, proprietary overlays have sought to address these issues, with mixed results. They tend to reduce overall surface area (fewer choices lead to fewer points of divergence), but often at a cost to portability and extensibility, which is what made Kubernetes so attractive in the first place. A more durable approach is to build on Kubernetes-native primitives, adding governance and operational consistency without replacing the workflows teams already use. The remainder of this article will demonstrate what that looks like in practice. What an open platform actually means in enterprise Kubernetes “Open platform” is a common phrase in the Kubernetes ecosystem, but it’s worth defining what that term actually means in practice. Dastmalchi-Round defines an open platform as one that “exposes industry-standard APIs and, where possible, uses pure upstream open-source projects.” The distinction isn't whether the platform is open source. It's whether it relies on Kubernetes-native APIs and tooling or introduces proprietary CRDs, workflows, and CLIs that make migration difficult. As he notes, "You can still get lock-in with open source, because if it is only one vendor's solution and they layer all of their stuff on top of standard tooling, you are now dependent on their abstractions." The difference is easier to see when comparing an open platform with a proprietary overlay. Comparing Open Kubernetes Platforms and Proprietary Overlays Dimension Open Platform (NKP) Proprietary Overlay Core CRDs Standard upstream (Cluster API, FluxCD, Helm) Vendor-specific, migration cost is high GitOps engine FluxCD (CNCF project) Proprietary sync engine App packaging Helm + OCI (industry standard) Custom catalog format Monitoring stack Pure upstream CNCF (Prometheus, Grafana) Wrapped / vendor-branded Exit cost Clusters survive platform removal Manifests tied to platform APIs Third-party tooling Works if it runs on Kubernetes Requires certified integration Nutanix Kubernetes Platform (NKP) applies these principles by building on upstream Kubernetes components rather than replacing them. As Dastmalchi-Round puts it, the real test is what survives if you remove the platform. "With NKP, the clusters are pure upstream Kubernetes,” says Dastmalchi-Round. “The monitoring stack is pure upstream CNCF projects. GitOps is provided by FluxCD. Your manifests and charts are standard Helm." In other words, the operational tooling may change, but the underlying applications and deployment artifacts remain portable. Raw manifests to managed artifacts: Helm and OCI packaging in NKP Most enterprise teams start with a collection of Kubernetes YAML manifests that work for a single application or environment. While those manifests are typically stored in version control, they aren't easily reusable across environments, self-service for other teams, or packaged in a way that supports consistent versioning and rollback. Helm addresses those limitations by packaging manifests into versioned, parameterized charts. For existing applications, the process typically starts by converting Kubernetes manifests into a standard Helm chart, either manually or with tools such as Helmify. The result is a familiar Helm project structure built around Chart.yaml, parameterized templates, and a values.yaml file, giving teams a reusable deployment artifact instead of a collection of static manifests. Deployment-specific settings, such as image tags, replica counts, and resource limits, move into a values.yaml file, while the underlying templates remain unchanged. Those deployment-specific settings are defined in the chart's values.yaml file. For example: # values.yaml — the self-service interface for application teams replicaCount: 2 image: repository: registry.example.com/myapp tag: "2.1.0" pullPolicy: IfNotPresent resources: limits: cpu: 500m memory: 256Mi requests: cpu: 250m memory: 128Mi ingress: enabled: true host: myapp.internal.example.com annotations: kubernetes.io/ingress.class: "traefik" serviceAccount: create: true name: "myapp-sa" Versioning makes deployments reproducible across environments while providing a clear history of releases. Teams can promote the same chart through development, staging, and production with confidence, then roll back to a previous version if needed. OCI registries address the next challenge: distributing and versioning those charts. Instead of relying on a separate chart repository, teams can store Helm charts alongside container images as immutable, versioned artifacts. Because chart versions can't be overwritten, deployments are reproducible and easier to audit. The approach also fits existing registry workflows. Organizations using Harbor, Amazon ECR, or similar registries can manage container images and Helm charts in the same place, using the same authentication, access controls, and security policies. For example: # Package the chart locally helm package ./myapp --version 2.3.0 # Authenticate to the OCI registry (same registry as your container images) helm registry login registry.example.com \ --username $REGISTRY_USER \ --password $REGISTRY_PASSWORD # Push is stored as an OCI artifact alongside container images helm push myapp-2.3.0.tgz oci://registry.example.com/charts # Any team can pull without touching the source repo helm pull oci://registry.example.com/charts/myapp --version 2.1.0 # Inspect the chart before deploying helm show values oci://registry.example.com/charts/myapp --version 2.1.0 The goal of packaging is to create a self-service deployment model. Once packaged, Helm charts are registered with the NKP catalog, where they appear alongside built-in platform applications as versioned deployment artifacts. Application teams can deploy them by configuring only the settings that vary between environments, while platform teams focus on maintaining reusable application catalogs instead of manually managing deployments. FluxCD deployments, overrides, and upgrades Once Helm charts are stored in an OCI registry, FluxCD keeps deployed clusters aligned with the desired state defined in Git. It continuously reconciles each cluster against that source of truth, automatically correcting configuration drift. In multi-cluster environments, each cluster follows the same reconciliation process using its own configuration. NKP's FluxCD implementation centers on two resources: HelmRepository, which points to the OCI registry, and HelmRelease, which specifies the chart version, configuration values, and target namespace. # Source: points FluxCD at your OCI chart registry apiVersion: source.toolkit.fluxcd.io/v1beta3 kind: HelmRepository metadata: name: internal-charts namespace: flux-system spec: type: oci url: oci://registry.example.com/charts interval: 5m # poll for new chart versions every 5 minutes # Release: declares desired state for a specific deployment apiVersion: helm.toolkit.fluxcd.io/v2beta3 kind: HelmRelease metadata: name: myapp-production namespace: production spec: interval: 10m chart: spec: chart: myapp version: "2.3.0" sourceRef: kind: HelmRepository name: internal-charts namespace: flux-system values: replicaCount: 3 resources: limits: cpu: 1000m memory: 512Mi ingress: host: myapp.prod.example.com Although teams interact with NKP through its web interface, those actions are ultimately represented as standard Kubernetes resources. Configuration changes become declarative objects that FluxCD reconciles like any other GitOps workflow, making the deployment model transparent and compatible with standard Kubernetes tooling without relying on proprietary deployment workflows. Teams typically promote the same chart version from development to staging and production while applying environment-specific overrides through HelmRelease values rather than modifying the chart itself. Promotion becomes a Git commit instead of a manual deployment, with FluxCD automatically reconciling and applying the change. FluxCD also provides continuous drift detection. If someone manually changes a resource in the cluster, FluxCD restores it to the state defined in Git during the next reconciliation cycle. Rolling back a deployment is simply a Git revert, with Git history providing a complete audit trail of configuration changes. How to integrate third-party tools without losing openness Enterprise platform teams are often asked to integrate tools such as vulnerability scanners, cost management dashboards, and application performance monitoring (APM) platforms. The tools themselves aren't the problem. The problem is managing each one through a separate deployment and maintenance process, increasing operational complexity over time. NKP addresses this by treating third-party software like any other platform application. Whether it's an upstream open-source project or a commercial product distributed as a Helm chart, it follows the same Helm-over-OCI packaging model and is deployed and managed through FluxCD. The outcome is a consistent deployment and lifecycle workflow across both first- and third-party applications. For example, an upstream Helm chart such as Redis can be published to the NKP catalog and managed through the same deployment workflow as a first-party application, avoiding the need for a separate integration process. Because this approach relies on standard Kubernetes resources, Helm charts, Git, and Kubernetes RBAC, those workloads remain portable across platforms. As Dastmalchi-Round summarizes, "If it works on Kubernetes, it will work on NKP." Dastmalchi-Round notes that the biggest integration challenges typically come from tools that rely on rigid deployment models, particularly older operator-based packages that expose little configuration. "A few years ago, there was a trend of people overusing the operator pattern for packaging applications," he says. "Operators have their uses, but when they became the distribution artifact, they often resulted in big, opaque blobs running in your cluster. If they didn't do exactly what you needed, you were out of luck." As more vendors have adopted Helm-based packaging, those limitations have become less common. Examples of Third-Party Tool Integrations in NKP Integration Type Packaging Model Configuration Upgrade Path NKP Catalog Security scanner (e.g., Trivy) Helm chart via OCI values.yaml in Git FluxCD HelmRelease bump Yes Custom Grafana dashboard Helm chart + ConfigMap Dashboard JSON in Git Chart version update Yes Cost management (e.g., OpenCost) Helm chart via OCI values.yaml in Git FluxCD HelmRelease bump Yes Service mesh (e.g. Istio) Helm chart via OCI IstioOperator CRDs in Git Controlled chart upgrade Yes Legacy operator-only tool Operator bundle Operator-managed CRDs Operator version update Requires evaluation In practice, the less a tool depends on proprietary deployment mechanisms, the easier it is to integrate, manage, and move between Kubernetes platforms. Conclusion: the platform that gets out of the way NKP doesn't replace Kubernetes workflows—it builds on them. Helm packages applications, OCI registries distribute them, Git defines the desired state, and FluxCD keeps deployments in sync. Instead of introducing proprietary workflows, NKP brings these familiar tools together with the governance, lifecycle management, and self-service capabilities required for enterprise-scale operations. It standardizes these workflows across any environment, including public clouds, on-premises, and edge locations. For enterprise teams, the value lies in achieving consistency without sacrificing portability. As Dastmalchi-Round notes, the question isn't whether lock-in exists, but how costly it is to leave. By relying on upstream Kubernetes components, Helm charts, and GitOps workflows, organizations retain portable applications and deployment artifacts even if they choose a different platform in the future. In the end, an open platform shouldn’t be defined by its licensing model. It should be defined by how much of your platform remains yours if you decide to move on.

Artificial intelligence has rapidly become a core capability in modern software development. For Java developers, integrating these capabilities into existing enterprise applications no longer requires learning entirely new frameworks or interacting directly with complex AI APIs. Spring AI bridges this gap by providing a familiar Spring programming model for working with large language models (LLMs) from providers such as OpenAI, Google Gemini, and others. In this article, we will build a simple AI-powered REST API using Spring Boot and Spring AI while exploring practices that help move beyond proof-of-concept implementations toward production-ready enterprise applications. A Typical Enterprise Architecture Rather than allowing clients to communicate directly with an AI provider, enterprise applications usually introduce a service layer responsible for security, validation, business logic, and monitoring. Plain Text Client Application │ ▼ Spring Boot REST API │ Validation & Business Logic │ ▼ Spring AI ChatClient │ ▼ Large Language Model (OpenAI / Gemini / Azure) This architecture keeps AI interactions behind your own APIs, allowing you to enforce authentication, authorization, logging, rate limiting, and governance without exposing provider-specific details to consumers. Creating the Spring Boot Project Getting started with Spring AI is straightforward. The application requires Spring Web, Validation, and the Spring AI starter. XML <properties> <java.version>21</java.version> <spring-ai.version>1.0.0</spring-ai.version> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-bom</artifactId> <version>${spring-ai.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-validation</artifactId> </dependency> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-openai</artifactId> </dependency> </dependencies> Configuring the AI Model One security practice I strongly recommend is avoiding hard-coded API keys or model names inside the application. Instead, configure them using environment variables or an enterprise secrets manager. YAML spring: ai: openai: api-key: ${OPENAI_API_KEY} chat: options: model: ${OPENAI_MODEL} temperature: 0.2 The lower temperature value encourages more deterministic responses, which is generally preferable for technical or business APIs where consistency matters. Designing the API Contract Rather than exposing raw AI requests directly, I prefer defining explicit request and response models. This keeps the REST API independent of the underlying AI provider and makes future changes much easier. Java import jakarta.validation.constraints.NotBlank; import jakarta.validation.constraints.Size; public record AIQuestionRequest( @NotBlank @Size(max = 2000) String question, String audience ) {} Response model: Java public record AIAnswerResponse( String answer ) {} Configuring the ChatClient Spring AI's ChatClient is responsible for interacting with the configured language model. Rather than repeating the same instructions in every request, we can configure a default system prompt once. Java @Configuration public class AIConfiguration { @Bean ChatClient chatClient(ChatClient.Builder builder) { return builder .defaultSystem(""" You are an experienced Java architect. Provide concise, accurate, production-ready answers. Never invent APIs. If uncertain, clearly state your assumptions. """) .build(); } } The system prompt establishes the overall behavior of the assistant. It ensures that every request follows the same guidelines, resulting in more predictable responses. Implementing the AI Service One architectural decision I recommend is keeping AI interactions inside a dedicated service layer rather than calling the language model directly from a controller. This separation makes the code easier to test, improves maintainability, and keeps business logic independent of the web layer. Java @Service public class TechnicalAssistantService { private final ChatClient chatClient; public TechnicalAssistantService(ChatClient chatClient) { this.chatClient = chatClient; } public AIAnswerResponse answer(AIQuestionRequest request) { String audience = request.audience() == null ? "Java Developer" : request.audience(); String response = chatClient.prompt() .user(user -> user .text(""" Explain the following question. Audience: {audience} Question: {question} Keep the answer under 300 words. """) .param("audience", audience) .param("question", request.question())) .call() .content(); return new AIAnswerResponse(response); } } Creating the REST Controller With the service layer complete, exposing the AI functionality through a REST endpoint becomes straightforward. Java @RestController @RequestMapping("/api/ai") public class AIController { private final TechnicalAssistantService assistantService; public AIController(TechnicalAssistantService assistantService) { this.assistantService = assistantService; } @PostMapping("/ask") public ResponseEntity<AIAnswerResponse> ask( @Valid @RequestBody AIQuestionRequest request) { return ResponseEntity.ok( assistantService.answer(request)); } } The endpoint accepts a JSON request, validates the input, invokes the service layer, and returns a structured response. Returning Structured AI Responses Many AI examples simply return text. While that's useful for chat applications, enterprise APIs usually need predictable JSON responses. For example, suppose we want AI to review Java code. Instead of receiving one long paragraph, we can ask the model to return structured data. Java public record CodeReviewResponse( String summary, List<String> strengths, List<String>issues, List<String>recommendations, String riskLevel ){} Now Spring AI can map the model response directly into a Java object. Java public CodeReviewResponse review(String sourceCode){ return chatClient.prompt() .system(""" You are a Senior Java Architect. Review the code for correctness, performance, security and maintainability. """) .user(sourceCode) .call() .entity(CodeReviewResponse.class); } This approach is much cleaner than parsing raw JSON or trying to interpret free-form responses manually. It also keeps the rest of the application strongly typed. Streaming AI Responses Some AI responses can take several seconds to complete. Rather than waiting until the entire response has been generated, Spring AI allows responses to be streamed back to the client. Java @RestController @RequestMapping("/api/ai") public class StreamingController { private final ChatClient chatClient; public StreamingController(ChatClient chatClient) { this.chatClient = chatClient; } @GetMapping( value="/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE) public Flux<String> stream( @RequestParam String question){ return chatClient.prompt() .user(question) .stream() .content(); } } Streaming significantly improves the user experience because clients can begin displaying the answer immediately instead of waiting for the complete response. This is especially useful for chat applications and AI assistants. Cache Responses When Appropriate AI requests introduce additional latency and cost because every request communicates with an external model. If the same prompt is frequently submitted, consider caching the response. Spring Cache makes this simple. Java @Service public class TechnicalAssistantService { @Cacheable("aiResponses") public AIAnswerResponse answer(AIQuestionRequest request) { // AI Call } } Caching works particularly well for frequently asked questions, product descriptions, technical explanations, and internal knowledge articles. Dynamic or user-specific responses generally should not be cached unless the cache key includes the relevant context. Final Thoughts What stands out to me is that Spring AI allows AI capabilities to become a natural extension of an existing Spring Boot application rather than requiring an entirely new architecture. Whether the goal is building an internal knowledge assistant, generating summaries, reviewing code, or automating repetitive tasks, the development experience remains consistent with the rest of the Spring ecosystem. That said, building a production-ready AI application involves much more than calling an LLM. Prompt design, security, validation, observability, performance, and cost management all play a critical role in delivering reliable solutions.

The pitch for multi-cloud always sounds clean. Avoid vendor lock-in. Optimize costs by running workloads on whichever provider is cheapest for a given task. Improve resilience by distributing across independent failure domains. On paper, it's a compelling case. In practice, the teams living with multi-cloud deployments often describe something closer to the opposite: doubled operational complexity, halved observability, and a category of reliability problems that only exist because there are two clouds instead of one. A team I worked closely with made the move to multi-cloud workloads on AWS and ML inference pipelines on GCP because of better GPU availability and pricing at the time and spent the next eight months dealing with a class of incident they hadn't anticipated: failures that were neither the application's fault nor either cloud provider's fault but existed in the boundary between them. Data transfer latency spikes that only appeared under load. Authentication token expiry edge cases that only trigger during cross-cloud calls. Network policy interactions that passed every pre-production test and failed in production at 3 am. The problems weren't hard individually. They were hard because the diagnostic tools for each cloud pointed inward, and the failure lived in the space neither tool was looking at. The Visibility Gap at the Boundary The first thing that breaks in a multi-cloud architecture is unified observability, and it breaks before you notice. Each cloud provider ships excellent native monitoring tooling: CloudWatch on AWS, Cloud Monitoring on GCP, and Azure Monitor on Azure. Each is well-integrated with that provider's services and reasonably effective at surfacing problems within its domain. None of them are designed to tell you what's happening in the gap between providers. When a request originates in AWS, crosses a private interconnect or the public internet to GCP, gets processed, and returns a response, the AWS tooling sees a latency value that includes the round-trip to GCP. The GCP tooling sees the processing time on its end. Neither surface shows you the network transit time in isolation, the connection establishment overhead, or the variance in that transit time under different load conditions. You're looking at the sum when you need to see the components. The fix requires pulling observability out of both providers' native tooling and into a neutral layer. In practice, that means OpenTelemetry instrumentation at every service boundary, every outbound cross-cloud call tagged with a span that captures the full round trip from the caller's perspective, and shipped to a backend that neither provider controls. This sounds straightforward, and technically it is. The organizational friction is real: teams used to relying on provider-native dashboards resist the overhead of running a separate observability stack, and the political question of which team owns it when it spans two provider accounts is never as simple as it should be. The Configuration Drift Problem Here's a failure mode that didn't appear in the architecture review because nobody thought to look for it: configuration drift between environments. In a single-cloud deployment, there's usually a meaningful concept of a canonical configuration in the infrastructure-as-code that defines the state of the environment, version-controlled, reviewed, and applied through a pipeline. In a multi-cloud deployment, you have two canonical configurations, maintained by teams with different tooling preferences and release cadences, and the interaction between them is rarely explicitly modeled. The incident that made this concrete: a security team rotation updated the cross-cloud service account credentials in the AWS Secrets Manager. The GCP side that consumed those credentials was on a different rotation schedule and a different team. For eleven days, the system ran on cached credentials. On the twelfth day, the cache expired during a peak traffic window. The GCP inference pipeline started returning authentication errors. The AWS team saw timeouts. The GCP team saw auth failures. Neither alert correlated the two. The on-call rotation spent ninety minutes establishing that the credentials were the issue before they could even start on the fix. The lesson is that cross-cloud dependencies, credentials, certificates, API contracts, and network allowlists need to be modeled and monitored as first-class infrastructure, not as bilateral agreements between teams. In practice, this means a dependency inventory: an explicit record of every configuration element in Cloud A that depends on something in Cloud B, with ownership, rotation schedules, and health checks defined for each. Python # Cross-cloud dependency health check (Python) # Run on a schedule to detect drift before it causes an incident import boto3, google.auth, requests from datetime import datetime, timezone def check_cross_cloud_credential(secret_name: str, gcp_endpoint: str) -> dict: """ Validates that the credential stored in AWS Secrets Manager is currently accepted by the GCP service that consumes it. """ # Fetch current credential from AWS sm = boto3.client('secretsmanager', region_name='us-east-1') secret = sm.get_secret_value(SecretId=secret_name) credential = secret['SecretString'] # Probe the GCP endpoint with the current credential resp = requests.get( gcp_endpoint + '/health', headers={'Authorization': f'Bearer {credential}'}, timeout=5 ) return { 'secret': secret_name, 'valid': resp.status_code == 200, 'checked_at': datetime.now(timezone.utc).isoformat(), 'status': resp.status_code, } # Alert if valid=False — don't wait for production traffic to find out The health assessment above is intentionally simple. The sophistication isn't in the code; it's in the discipline of running it, alerting on failure, and treating a failed credential probe as an incident rather than a routine maintenance item. Teams that implement this pattern catch rotation mismatches days before they would have surfaced as production failures. Network Reliability: The Assumptions That Break Single-cloud architectures inherit a reasonably reliable network fabric. Traffic within an availability zone is fast and consistent. Traffic across zones adds a predictable overhead. Traffic across regions adds more. The provider manages the underlying network, and it generally behaves within well-understood parameters. Multi-cloud breaks that model. Traffic between providers crosses networks that no single provider fully controls, whether through a private interconnect (AWS Direct Connect to GCP via a colocation facility) or the public internet, with all the variance that entails. The latency distribution changes character: instead of a tight distribution with a predictable tail, you receive a wider distribution with a heavier tail, and the tail gets heavier under load in ways that are harder to predict and harder to reproduce in testing. The implication for application design is significant. Services that communicate across cloud boundaries need timeout and retry policies calibrated to a different latency distribution than services communicating within a single cloud. A retry budget designed for intra-cloud latency will either be too aggressive, triggering retries on latency spikes that would have resolved naturally, or too conservative, giving up on requests that would have succeeded with a longer timeout. Getting this right requires measured data from the actual cross-cloud path under realistic load, not assumptions imported from single-cloud experiences. Python # Cross-cloud call with calibrated timeout and retry policy import httpx from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type # Timeouts calibrated from p99 measurements of actual cross-cloud path CROSS_CLOUD_CONNECT_TIMEOUT = 2.0 # seconds CROSS_CLOUD_READ_TIMEOUT = 8.0 # wider than intra-cloud to absorb tail latency @retry( stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=1, max=10), retry=retry_if_exception_type((httpx.TimeoutException, httpx.ConnectError)) ) async def call_gcp_inference(payload: dict) -> dict: async with httpx.AsyncClient( timeout=httpx.Timeout( connect=CROSS_CLOUD_CONNECT_TIMEOUT, read=CROSS_CLOUD_READ_TIMEOUT ) ) as client: resp = await client.post( GCP_INFERENCE_ENDPOINT, json=payload, headers=get_auth_headers() ) resp.raise_for_status() return resp.json() The timeout values in the example above aren't guesses; they're derived from measuring actual cross-cloud latency at the 99th percentile under production load. The read timeout of 8 seconds would be far too generous for an intra-cloud call, where p99 might be under 200 milliseconds. For a cross-cloud inference call with real tail latency, it's calibrated to let legitimate slow requests complete while still protecting against genuine hangs. The difference matters: a timeout set at 500ms on this path would generate false failures on every traffic spike. The Cost of Distributed Incident Response One reliability challenge that rarely makes it into architecture discussions is the human cost of multi-cloud operations during incidents. When a single-cloud incident fires, there's usually a clear owner, the team responsible for that environment, and a reasonably well-understood set of tools and runbooks. When a multi-cloud incident fires at the boundary, the ownership is ambiguous by definition. In hindsight, the team I worked with should have defined cross-cloud incident ownership explicitly before going to production, not as a policy document but as a named on-call rotation and a defined escalation path. What they had instead was an informal understanding that "whoever is paged first figures it out," which works until a cross-cloud incident fires at 3 a.m. and the first person paged has deep AWS expertise and no GCP access. That situation happened twice before anyone fixed it. The solution was a dedicated cross-cloud on-call rotation, not a separate team, but a monthly rotation in which the designated engineer was expected to have current working knowledge of both environments and appropriate access to both. It also required shared runbooks stored outside either provider's tooling, because documents stored in an AWS wiki are inaccessible if the incident involves AWS authentication failures. What I'd Do Differently The single most important investment before going multi-cloud is measuring the cross-cloud network path under realistic load and calibrating every timeout, retry budget, and circuit breaker to those measurements rather than to assumptions. This is an unglamorous workload: testing a network path, collecting latency distributions, and deriving timeout values, and it's almost always skipped in favor of getting to production faster. The debt shows up as a category of reliability problem that's challenging to diagnose because it looks like application errors but behaves like network variance. I'd also resist the organizational tendency to treat multi-cloud as a flag day, a point at which the system transitions from single-cloud to multi-cloud. The reliability problems at the boundary need to be understood incrementally, with each cross-cloud dependency introduced deliberately and monitored explicitly before the next one is added. The teams that go from single-cloud to multi-cloud in one large migration tend to discover all their boundary problems simultaneously under production load. When might it be advisable to avoid a multi-cloud approach? The honest answer is most of the time. The cases where multi-cloud genuinely earns its operational cost are narrow: regulatory requirements that mandate geographic or provider separation, specific technical capabilities that a single provider doesn't offer at the required scale or price point, or acquisition scenarios where two companies on different clouds need to integrate without immediate migration. If the primary driver is "avoiding vendor lock-in" as a philosophical position, the operational cost almost never justifies it. A single cloud with well-designed abstractions at the application layer provides most of the portability benefits without most of the operational burdens. Key Takeaways The reliability problems in multi-cloud environments often live at the boundary between providers in the gap that neither provider's native tooling is designed to illuminate. Neutral observability infrastructure spanning both environments is a prerequisite, not an enhancement. Cross-cloud dependencies on credentials, certificates, and network allowlists need to be explicitly inventoried and health-checked on a schedule. Rotation mismatches and configuration drift between providers are a common source of incidents that look unrelated until you discover the shared dependency. Timeout and retry policies for cross-cloud calls must be calibrated using measured latency data on the actual path under load. Assumptions imported from a single cloud experience will be wrong in ways that generate either false failures or genuine availability problems. Define cross-cloud incident ownership before going to production. Ambiguous ownership at the boundary can lead to increased resolution time during critical moments. Conclusion Multi-cloud is frequently sold as a resilience strategy and often experienced as a complexity tax. The resilience argument is real but conditional: if the cross-cloud architecture is well-instrumented, the boundary dependencies are explicitly managed, and the failure modes at the boundary are understood and designed for, then distribution across providers does improve resilience. If those conditions aren't met, multi-cloud primarily adds new failure modes without reliably eliminating old ones. The teams that make multi-cloud work well tend to share one characteristic: they treated the inter-cloud boundary as a first-class engineering concern from the beginning with its observability, its dependency management, and its own incident ownership. Teams that treated it as a network detail that would take care of itself consistently found that it required attention. The question worth sitting with before committing to multi-cloud is "Are you solving a real problem that a single cloud with better architecture can't solve, or are you building for a failure scenario, vendor lock-in, or catastrophic provider outage that is less likely than the operational problems you're about to introduce?" Multi-cloud at the wrong time, for the wrong reasons, creates the very fragility it's supposed to prevent.

Business documents rarely keep their most useful information in convenient database records or JSON objects. Invoices hold line items in tables, financial reports organize figures by period, inspection forms group findings by category, and emailed statements often arrive as attachments or message files. Before our applications can validate, compare, search, or store that information, they have to somehow recover the relationships between rows, columns, headers, and values. At first glance, this may look like a routine text-extraction problem. If we can read the words on the page, surely we can rebuild the table... right? In practice, unfortunately, recognizing the text is only the first layer. We also need to determine which values belong to the same row, where columns begin and end, which headers describe which cells, and whether the document contains multiple distinct tables. Again, tables represent relationships, and if those relationships are mishandled, the data becomes useless. In this article, we’ll look at why table extraction becomes difficult across different document formats. We’ll then implement an API-based extraction workflow in C# and process the structured table data it returns. Why Table Extraction Is More Than OCR Optical character recognition (OCR) has been around forever, and by today's technological standards, it answers a relatively narrow question: which characters appear in an image, and where are they located? Table extraction has to answer a more contextual question: how are those characters related? Consider a scanned invoice with the headers Description, Quantity, Unit Price, and Total. An OCR engine might correctly identify all four headers and every value beneath them, but a stream of recognized text isn’t enough for downstream automation. We still need to establish that 3, $12.00, and $36.00 belong to the same line item, and that $36.00 represents the total rather than the unit price. Visible borders can certainly help, but we can’t count on them; many modern tables use whitespace, shading, or alignment instead of grid lines. Cells may span multiple apparent columns, descriptions may run into character constraints and wrap onto additional lines, and a table may continue onto the next page with repeated or missing headers. Poor document image quality has also been the bane of OCR solutions for decades; it introduces another unwelcome layer of ambiguity. A recognition error can change both an individual data value and the entire interpretation of its surrounding structure. All of this is to say that a document can contain perfectly readable text while still producing a poor table. Reliable table extraction requires both character recognition and layout analysis. Different Formats Hide Tables in Different Ways The word "document" itself covers several very different internal structures. A DOCX file, for example, is a package of XML parts that can contain explicit table elements. An XLSX workbook stores cells, values, formulas, and worksheet relationships in another Office Open XML structure. PPTX files can combine true table objects with independently positioned text boxes that only look like tables to a human reader. PDF files are significantly more complicated. A PDF may describe text by drawing individual characters at specific page coordinates on the page without preserving any semantic concept of a row, column, or table for extraction services to key in on. Two values that appear next to each other on the page may or may not be related within the underlying PDF structure; context is required to know for sure. Images push us fully into visual interpretation. For example, JPG, PNG, and WEBP inputs don’t contain native text or table objects, so OCR and layout analysis are required for automated data extraction. Email containers such as EML and MSG introduce another variation. We might not think about these file types quite as frequently as the others mentioned in this section, but they introduce an interesting challenge worth considering in the pursuit of solving this problem for a wide range of inputs. These formats allow for a useful table to appear in the message body, a rendered representation of that body, or an attached document. As a result, a broadly focused table-extraction system can’t simply run the same parser against every extension. It first needs to identify and decode the format, obtain a useful page or layout representation, recognize text when necessary, and map the discovered structures into a consistent table model. Once every input produces the same hierarchy (tables containing rows, rows containing cells, and cells containing headers and values), the rest of our application doesn't need format-specific extraction logic anymore. Building the Workflow With Open-Source .NET Libraries The .NET ecosystem gives us several useful open-source building blocks for structured data extraction, but the right library depends heavily on the source document. For Office documents, the Open XML SDK is your first stop: it can inspect the native structures inside DOCX, XLSX, and PPTX files directly. Format-specific libraries such as ClosedXML can provide a more approachable layer for working with Excel worksheets. These options are a strong fit when our tables exist as real Office table or cell structures, which is fairly often. PDF documents often require a different path. A library such as PdfPig (more than 28 million downloads on GitHub) can extract text and positional information from text-based PDFs, but our application may still need custom logic to group those positioned words into rows and columns. If the PDF contains scanned pages, we first need to render those pages into images and send them through an OCR engine such as Tesseract (another widely used & loved package). If reading from email files is a must, a bit more routing work is required. MimeKit can parse MIME-based messages such as EML files, while MSG files may require a separate Outlook message parser. After parsing the message, we still need to inspect the body and each relevant attachment independently. Each of these libraries can, of course, make sense within its own lane. If our application receives one predictable document type with a stable layout, an open-source implementation may give us all the control we need. The complexity rears its head when we want one production workflow to accept PDFs, Office files, email containers, and images all at once. In that case, we have to detect formats, route documents to the correct parser, render pages when native extraction fails, decide when OCR is necessary, and reconcile several different output structures. That's a lot. We also own the quality heuristics for that workflow. That includes the borderless-table detection challenge in addition to merged cells, rotated pages, repeated headers, image preprocessing, wrapped text, and validation thresholds. In other words, multi-format table extraction is better understood as a document-processing system than a single library call. Using a Normalized Table-Extraction API If maintaining separate extraction paths for each format feels like too much, we can send the input document to a dedicated table-extraction service that performs the format handling, recognition, and table analysis through a consistent API. We'll walk through one example that uses AI to consistently identify table structures in DOCX, PDF, XLSX, PPTX, EML, MSG, JPG, PNG, and WEBP input. The response JSON organizes the extracted content into tables, rows, and cells. We’ll access the endpoint through its generated .NET Core SDK. To begin, we’ll install version 1.0.0: C# dotnet add package Cloudmersive.APIClient.NETCore.DocumentAI --version 1.0.0 Once the package is installed, we can import the API, client, and model namespaces required for the request: C# using System; using System.Diagnostics; using Cloudmersive.APIClient.NETCore.DocumentAI.Api; using Cloudmersive.APIClient.NETCore.DocumentAI.Client; using Cloudmersive.APIClient.NETCore.DocumentAI.Model; The snippets provided below mirror the supplied SDK code directly; as code examples, they assume we’ll adapt placeholders and surrounding application details as needed. Configuring the API Client First, we’ll add our API key under the Apikey authorization name in the default configuration: C# Configuration.Default.AddApiKey("Apikey", "YOUR_API_KEY"); Creating the Extraction Client Next, we’ll create a new ExtractApi instance: C# var apiInstance = new ExtractApi(); Loading the Input Document We’ll assign a value to the optional recognition-mode parameter: C# var recognitionMode = "Advanced"; Advanced is the default recognition mode and provides the highest accuracy with slower processing, while Normal provides faster processing with lower accuracy for low-quality images. Next, we’ll open our input document as a FileStream: C# var inputFile = new System.IO.FileStream("C:\\temp\\inputfile", System.IO.FileMode.Open); Executing the Table-Extraction Request With recognitionMode and inputFile ready, we’ll pass them into ExtractTables and write the returned ExtractTablesResponse object to the debug output: C# try { // Extract Tables of Data from a Document using AI ExtractTablesResponse result = apiInstance.ExtractTables(recognitionMode, inputFile); Debug.WriteLine(result); } catch (Exception e) { Debug.Print("Exception when calling ExtractApi.ExtractTables: " + e.Message ); } Understanding the Response Structure A successful response should follow this general structure: JSON { "Successful": true, "TableResults": [ { "Title": "Invoice Line Items", "Rows": [ { "Cells": [ { "CellHeader": "Description", "CellValue": "Replacement filter" }, { "CellHeader": "Quantity", "CellValue": "3" }, { "CellHeader": "Unit Price", "CellValue": "$12.00" }, { "CellHeader": "Total", "CellValue": "$36.00" } ] } ] } ] } Successful tells us whether the extraction operation completed. TableResults is a collection because one document may contain multiple distinct tables; the endpoint naturally distinguishes between each and returns their results separately. Ultimately, this response is a pretty straightforward JSON mapping. Every table can include a Title followed by Rows. Every row contains a collection of Cells, and each cell provides an inferred CellHeader and extracted CellValue. Note that a structurally valid response does not guarantee every property will contain a value. A document may contain a table with no visible title, for example, so our workflow shouldn’t rely on Title as a required identifier. We should also definitely expect real documents to contain blank cells, inconsistent headers, and values that require additional parsing before validation or storage. Reading the Returned Tables in C# C# if (result.Successful == true && result.TableResults != null) { foreach (var table in result.TableResults) { foreach (var row in table.Rows) { foreach (var cell in row.Cells) { Console.WriteLine( $"{cell.CellHeader}: {cell.CellValue}" ); } } } } The simple nested loops I've included here get each header and value while preserving the table structure, leaving us free to map rows into dictionaries, database entities, CSV records, or custom models such as InvoiceLineItem. Note that we should probably avoid aggressive type casting; identifiers may need leading zeroes preserved, while currency and date values may require locale-aware parsing. Extraction may structure the data, but schema validation remains our application’s responsibility. The Full Implementation Here's a fully assembled example implementation including everything we just outlined above: C# using System; using System.Diagnostics; using Cloudmersive.APIClient.NETCore.DocumentAI.Api; using Cloudmersive.APIClient.NETCore.DocumentAI.Client; using Cloudmersive.APIClient.NETCore.DocumentAI.Model; namespace Example { public class ExtractTablesExample { public static void Main() { Configuration.Default.AddApiKey( "Apikey", "YOUR_API_KEY" ); var apiInstance = new ExtractApi(); var recognitionMode = "Advanced"; using ( var inputFile = new System.IO.FileStream( "C:\\temp\\inputfile", System.IO.FileMode.Open ) ) { try { ExtractTablesResponse result = apiInstance.ExtractTables( recognitionMode, inputFile ); Debug.WriteLine(result); if (result != null && result.Successful == true && result.TableResults != null) { foreach (var table in result.TableResults) { if (table == null || table.Rows == null) { continue; } foreach (var row in table.Rows) { if (row == null || row.Cells == null) { continue; } foreach (var cell in row.Cells) { if (cell == null) { continue; } Console.WriteLine( $"{cell.CellHeader}: " + $"{cell.CellValue}" ); } } } } } catch (Exception e) { Debug.Print( "Exception when calling " + "ExtractApi.ExtractTables: " + e.Message ); } } } } } Adding Production Guardrails Whether we use the API-based approach demonstrated above or assemble an open-source extraction system, a production pipeline still needs some guardrails around it. The exact implementation will differ, but the underlying goals are mostly the same: we want to control things like resource use & preserve traceability, and very importantly, we want to prevent questionable extraction results from quietly entering downstream systems. First, we should validate each file and enforce practical document and page limits before processing begins. With the API approach, page counts directly affect consumption. In an open-source system, those same long documents can consume substantial memory, CPU time, OCR capacity, and worker availability. Both implementations benefit from clear limits and a plan for handling unusually large documents. We should also try to retain enough context to audit each result. For the API workflow, that record might include the source document identifier, recognition mode, response status, table index, etc. An open-source workflow might additionally record which parser, OCR engine, preprocessing steps, model version, and fallback path were used. These details make extraction problems much easier to reproduce and diagnose later. Most importantly, we need to define what successful means at the application level. An API response with Successful set to true indicates that the extraction operation completed. Likewise, an open-source parser returning rows without throwing an exception only tells us that its processing path completed. Neither outcome proves that the expected table was found or that every extracted value is correct. If the extracted data affects payments, compliance decisions, inventory, or customer records, human review remains sensible for incomplete or internally inconsistent results. Automation should reduce the amount of manual work required, not remove the opportunity to catch a result that doesn’t make sense. Conclusion In this article, we separated table extraction from plain text recognition and saw why supporting extraction from PDFs, Office documents, email containers, and images can require several different processing paths. Open-source .NET libraries give us plenty of capable building blocks when our formats and layouts are controlled. A broad intake workflow, however, also needs document routing, OCR, layout analysis, output normalization, and ongoing quality logic, all of which can be burdensome to implement in a production environment. We then installed a Document AI .NET SDK and took a look at structuring a request to handle table extraction automatically. With sufficient validation incorporated around those results, the same pattern we just demonstrated can support invoice processing, reporting, database imports, reconciliation workflows, and other systems that need structured document data rather than another block of extracted text.

By Brian O'Neill CORE

AI stands for artificial intelligence, yet I prefer the term Dr. Venkat Subramaniam used in one of his talks: Accelerated Inference. To my mind, it is far more accurate, so I have embraced it. And “accelerated” is precisely the point. With AI, generating code has become cheap; it is no longer the bottleneck of software development. What has become expensive, and what this article is really about, is everything around it: aligning outcomes with intent, owning what we ship, and exercising the judgment that no amount of acceleration can replace. A Brief History The main structures underlying modern AI are neural networks and transformers — statistical models capable of replicating patterns. The former have been around for much longer than many people today assume; the latter are comparatively new. The field’s milestones trace a recurring cycle of bold ideas, disillusionment, and breakthroughs driven by new algorithms, more data, and faster hardware: from the first mathematical model of an artificial neuron (1943) and the perceptron (1958), through the first “AI winter” (1969) and the backpropagation revival (1986), to AlexNet’s deep learning breakthrough (2012) and the Transformer architecture (2017) that underpins all modern large language models. In other words, AI as a concept is not new at all. What is new is that today almost everyone, technical or not, has an opinion about AI and how it is changing the way we work. On one hand, this is perfectly normal: the world isn’t what it used to be, the available tools are different and more powerful, and certain problems can now be solved much faster. On the other hand, in this new and fashionable landscape, people should strive to form objective opinions first, filter them through their own judgment, and only then express and apply what proves useful. From individual to individual, hasn’t this always been the case with everything else? Habits Worth Keeping (and Acquiring) When it comes to people, the recommendations on how to act when ‘newness’ emerges haven’t changed from what we’ve been used to. Certain responsibilities and habits should be kept, others adapted and continually improved, while new ones acquired. Regarding software engineers, I feel slightly more entitled to an opinion; thus, here are a few pieces of advice I have compiled and consider worth having close. Before writing code, strive to turn incomplete and ambiguous requirements into a comprehensible starting point — understand not just how to build something, but why and who it serves.Before writing code, know when to use a certain algorithm or design, why it matters, and how it fits into the large application you are building.Before merging in your code, have it reviewed first by yourself, then by at least one human programmer.Get used to a shift in thinking. Engineers are used to predictable results; LLMs produce variable output. A switch from deterministic to probabilistic thinking may be needed.Before merging AI-generated code, review every line in detail and thoroughly understand why each decision was made - transform the changes into fully owned ones, as if you had written them yourself. (sounds familiar, doesn’t it?)Before merging AI-generated code, make sure it meets the project’s coding standards and remains human-readable, and improve it where possible.Sharpen your code-reading and reviewing skills. They were always a plus; now that AI agents are programming buddies producing large volumes of output, they are essential.Avoid “prompt-and-pray vibe coding.” Use AI thoughtfully, maintain rigorous standards, and don’t short-circuit your own learning.Use AI as a force multiplier to amplify your existing skills — knowledge, experience, problem-solving as a professional, and above all, objective, constructive judgment as a human being.Validate both directions. Check not only what AI agents produce as output, but also what they consume as “trusted” input.Co-build abstractions. Use LLMs to help build an abstraction, then use that abstraction to communicate with the LLM more effectively and solve problems in a more deterministic manner.Keep delivering reliable, maintainable software, but pay closer attention to how you spend your time building it. Some of these points are about AI; some are not. Some have been acknowledged for ages; others emerged recently and will be assimilated sooner or later. From Writing Code to Solving Problems One thing is certain: with AI, the software development approach and mindset are fundamentally different. In this shift, I believe software engineers have a great opportunity — to move their attention even further from writing code to solving problems, and to spend their energy on more meaningful challenges. Oleg Koverznev puts the thesis I opened with even more sharply: “… code generation is cheap and no longer a bottleneck; the real challenge is aligning outcomes with intent, along with managing the growing operational and economic complexity of agent-driven work.” AI's instant output tends to fuel our impatient expectations for immediate results. It can make us feel productive and efficient, but let's hold on for a moment, resist the rush, and ask whether the outcome has real value, whether it actually produces impact. Going deeper, there is an idea I find helpful in sustaining this point of view. Luciano Floridi — a leading figure in the philosophy of information and digital ethics, widely influential in AI ethics - has a thesis (sometimes referred to as Floridi's Conjecture) that can be summarized as follows: As the complexity of a system increases, the ethical significance of its interactions also increases, while the ability to fully predict or control its behavior decreases. In this view, advanced information technologies are re-ontologizing our world, reshaping the very fabric of reality and our place within it, rather than simply serving as tools within an unchanged reality. Applied to AI, the conjecture suggests that a fully autonomous system cannot have great scope and great certainty at the same time; AI agents will therefore always require human oversight. Yet more and more, out of convenience, people tend toward fully embracing agent autonomy, when it would be wiser to remain in the loop: tempered and watchful. Language, Thought, and Staying Sane Science in general and AI in this particular context is not a magic trick, but a special way of using human intelligence. We embrace science and AI not when we put on a white coat, but when we start practicing a set of canons of thought, many of which have to do with the use of natural language, which is, after all, the main medium of AI-human interaction. New technologies change what we understand by “knowledge” and “truth.” They alter the deeply rooted habits of thought that give us a sense of the world, of the natural order of things, of what is reasonable, necessary, inevitable, or real. The actual magnitude of the effect varies from person to person. But as Neil Postman observed, a new technology doesn’t merely add or subtract something — it changes everything. It redefines the words by which we guide our lives: freedom, truth, intelligence, reality, wisdom, memory, history. And it never stops to warn us, and we never stop to ask. We keep rushing. Conclusion In these days when AI seems to be the solution for almost everything, I consider it very important to keep improving our reading, writing, and technical skills, our natural language, and our critical thinking, and to apply them wisely. To be respectful to ourselves and to others. To value truth more than mere correctness and, why not, to become good bullshit detectors as well. In a world where consumerism is amplified as never before, where objects are overvalued and abundant, and where speed is prized over competence and quality, remaining sane is a real virtue. Improving our human virtues is a continuous and tedious process, but a rewarding, fundamental, and safe one in the long run. Let’s not change the world, let’s change ourselves.

By Horatiu Dan CORE

A few months back, I wrote about Loop Engineering: The Layer After Prompt, Context, and Harness Engineering, arguing that once your prompts are tuned, your context is assembled, and your harness is wired up, the thing that actually determines whether an agent works is the loop it runs in: how it decides to keep going, stop, retry, or hand off. A few readers asked a fair follow-up after that piece: loops over what, exactly? What happens when one job stops being one loop's worth of work? That question turned out to have an answer that was already forming across the AI engineering world by the time I went looking for it. Mid-July 2026 saw a fast, noisy round of debate on X about exactly this, kicked off by a question from OpenClaw creator Peter Steinberger about whether the conversation had already moved past loops into graphs. Within a couple of days it had a name: graph engineering. I want to walk through what it actually means, where it overlaps with loop engineering, and where I think the skeptics in that debate had a fair point. What Graph Engineering Actually Means Here's the definition that held up best once I filtered out the hype: graph engineering is the practice of designing how multiple specialized agents or steps connect, not the internal cycle any one of them runs. A loop engineering designs the repeat-check-stop cycle a single agent goes through. Graph engineering designs how several of those loops hand off to each other. Three parts make up that structure: Nodes – the units doing the work. Usually a specialized agent (researcher, writer, reviewer) or a plain deterministic step like a tool call or data fetch.Edges – the routing between nodes. Can be a straight handoff, a conditional branch, a fan-out to several nodes at once, or a fan-in that joins parallel results back together.Shared state – the object that travels along the edges. It's what turns a pile of agents into an actual system instead of a group of assistants that forget everything the moment they hand off. One clarification I should make since I muddied this in my own earlier writing on the topic: this isn't the same thing as a knowledge graph or GraphRAG. Those model data, entities, and the relationships between them, for retrieval. Agent graph engineering models execution, which node runs next and what state it's handed. Same word, genuinely different problem, and it's an easy mix-up if you come at this from the infrastructure side as I do. Where I'd Already Been Doing a Version of This I didn't arrive at graphs from a whiteboard exercise. I got there from Terraform and service meshes. Years ago I wrote about visualizing Terraform plans and the tooling built around the terraform graph command, because a plan diff tells you what's changing, but a graph tells you what depends on what. Around the same time, I wrote about using Istio and its add-ons to visualize service meshes, where Kiali and Weave Scope turn a tangle of microservice calls into an actual picture of who talks to whom. Those are dependency graphs, not agent orchestration graphs, and I want to be precise about that distinction now rather than blur it the way I did before. But the design instinct is the same one: decide what counts as a node, what counts as an edge, and don't let the structure stay implicit just because it's inconvenient to draw. That instinct is exactly what's being renamed and re-applied to multi-agent systems right now. Graph and Loop, Side by Side The cleanest way I've found to hold both concepts at once: the graph is the map, the loop is the walk. The graph defines what's reachable and who talks to whom. It doesn't say how many times a node should retry, when it should give up, or what counts as good enough before advancing. That's still the loop's job, running inside each node. Graph and loop Every node in that diagram is its own loop underneath: the researcher discovers, plans, executes, and verifies its own search before handing off. The graph is only the outer wiring, deciding the researcher goes before the writer, and the reviewer's conditional edge decides whether the writer gets another pass. A single agent working alone is just the smallest possible version of this same shape: one node, with an edge that points back to itself. The Layer Stack, Extended My original loop engineering piece described a stack that ran from prompt to context to harness to loop. Graph is the next rung, and it's useful to see the whole climb at once. LayerWhat you're actually engineeringCore question it answersPromptThe single request you sendAm I asking well?ContextWhat the model gets to seeDoes it have the right information?HarnessTools, memory, scaffolding around the modelCan it act and remember across steps?LoopThe repeat cycle one agent runsWhen does it check its own work and stop?GraphCoordination between many agents or stepsWho does what, in what order, sharing what state? The stack is cumulative, not a ladder you climb away from. A graph is made of nodes; a good node is a well-designed loop, and a good loop still needs the harness underneath it doing its job. Skip a lower layer and the graph on top just fails in a more elaborate, harder-to-debug way. If the individual nodes are weak agents, wiring them into an org chart just gives you a weak org. When a Graph Actually Earns Its Keep This is the part worth being disciplined about, because the honest default answer is that most tasks don't need one. A single well-scoped agent with a clear stopping condition is a loop, and reaching for a graph before the work demands it is how a two-hour task turns into a two-week framework project. Signal in the workA loop is enoughReach for a graphShape of the taskOne job, one clear finish lineSplits into distinct specialties that hand offParallelism neededSteps run in sequenceYou need several things done at once, then joinedTools or models per stepSame toolset the whole way throughDifferent model or toolset at each stageVerificationThe agent checks its own outputA separate, dedicated node reviews another node's workFailure isolationA bad step just retriesOne failing node shouldn't poison the rest of the run None of these signals need to be unanimous. But if most of the honest answers land in the left column, you're describing a loop that someone talked you into over-architecting. Graph vs Loop Isn't This Just LangGraph? Worth addressing directly, because it's the most common pushback and it's mostly fair. The idea of building agent systems as graphs of nodes and edges over shared state shipped in real frameworks well before the term "graph engineering" started trending. LangGraph has offered exactly this model for a while now. Microsoft's AutoGen added graph-based orchestration through what it calls GraphFlow. Google's Agent Development Kit builds graph-based architecture as a headline feature, with sequential, parallel, and loop workflow agents as first-class building blocks. The Agent2Agent protocol, A2A, tackles the related problem of agents delegating across systems owned by different teams entirely. So when LangGraph's own creator publicly said he wasn't sure the term named anything beyond his own framework, that's a fair challenge worth taking seriously rather than waving off. My honest read: the technology mostly isn't new. What's newer is a shared vocabulary for a design decision these frameworks always asked of you anyway, which are what your nodes are, what your edges are, and what belongs in shared state. That's a real, useful naming exercise. It's a much smaller claim than "a new paradigm," and I'd rather undersell it than oversell it. A Quick Comparison of the Framework Options FrameworkOrchestration modelWhere it fitsLangGraphExplicit StateGraph: you define nodes and the edges between themTeams wanting a low-level, code-first orchestration runtimeMicrosoft AutoGen (GraphFlow)Graph-based multi-agent orchestration layered onto AutoGen's agent modelTeams already in the AutoGen ecosystem needing structured handoffsGoogle ADKNamed sequential, parallel, and loop workflow agents, plus agent routingTeams wanting graph patterns as built-in primitives rather than hand-rolledA2A protocolOpen protocol for agents to delegate across systems and organizational boundariesCross-team or cross-vendor agent handoffs, not a single app's internal graph I made a related point about keeping routing logic and decision logic as separate, inspectable concerns in ToolOrchestra vs Mixture of Experts: Routing Intelligence at Scale, and about explicit versus implicit capability structures in MCP vs Skills vs Agents With Scripts. Both pieces land on the same conclusion this table does: an implicit graph is still a graph; you just can't see it until something breaks, so you're generally better off picking a framework that makes the structure explicit rather than hand-rolling one that hides it. Where I Land On the Hype Question The skeptics in this debate aren't wrong about the mechanics: directed graphs, state machines, and multi-agent orchestration predate this month's vocabulary by years, and a fair amount of what got published about it in the past couple of weeks is exactly the kind of hype cycle content you'd expect. I'd rather say that plainly than pretend this is a brand-new capability. But separate the word from the actual shift, and there's something real underneath it. Teams that spent the past year getting good at running one agent in a loop are increasingly hitting cases where one loop is the wrong shape for the work, and are deliberately splitting it into coordinated, specialized nodes with state flowing between them. That escalation is happening whether or not you call it graph engineering, the same way the underlying shift I wrote about in the loop engineering piece was real whether or not "loop engineering" stuck as a term. Where This Matters in Practice Bringing this back to something concrete: in the incident response and SRE tooling I've written about recently, the AI agents that investigate production issues autonomously are, underneath the marketing, graph-and-loop systems. The graph is the set of things the agent is allowed to check: logs, metrics, traces, deploy history, past incidents. The loop is the policy for how it moves through that graph, how many sources to check before proposing a hypothesis, when to backtrack if the evidence doesn't line up, and when to stop and hand off to a human instead of guessing further. Teams that get frustrated with these tools usually have a graph problem, not a model problem. If the agent's graph has no path to the system that actually caused the issue, no amount of looping finds it. Worth checking before blaming the reasoning quality of whatever model sits underneath. A Short Checklist Before You Build One Try to keep it a loop first. If a single well-scoped agent with a good verifier can do the job, stop there.Only name a node if it's a genuine specialty, a different model, a different toolset, or a read-only reviewer role. Steps you could inline aren't nodes.Draw the edges before you write code. If you can't sketch the routing on a napkin, it's already too complex.Design the shared state object on purpose, and decide who's allowed to write to it. State drift is the fastest way a graph rots.Give the reviewer node real teeth: a separate agent from the one that produced the work, not the same agent grading its own output.Isolate failure so one bad node can retry without corrupting shared state or poisoning the rest of the run.Reach for an existing framework, LangGraph, AutoGen's GraphFlow, or Google ADK, before hand-rolling your own orchestration runtime.

By Vidyasagar (Sarath Chandra) Machupalli FBCS CORE