When a Coding Agent Fixes the Test Instead of the Bug
PII Detection Before RAG Ingestion: Screening Documents in C#
CORE
TL;DR Reddit's public .json listings let you scrape a subreddit in about 100 lines of Go. You decode the listing envelope, follow the after cursor, send an honest User-Agent, and back off on HTTP 429. There are two hard limits: a listing stops at roughly 1,000 items, and requests from cloud or datacenter IPs often get a flat 403. For anything in production, use the official OAuth API or a managed endpoint. Reddit is one of the richest public sources of opinion data on the web. Product teams mine it for feature requests, marketers track brand mentions, and ML engineers build datasets from it. Since the 2023 API pricing changes, "just use the API" has become a harder sell for small projects, so many developers go back to scraping. This article walks through a small, dependency-free Go scraper that exports a subreddit's newest posts to CSV. It also covers the failure modes you'll actually hit. Compliance first. Read Reddit's User Agreement and Data API Terms before collecting anything. Collect only public data, keep request rates low, don't re-identify users, and don't use Reddit content to train models without the appropriate licence. Commercial use generally requires an agreement with Reddit. How Reddit Exposes Data Without an API Key Add .json to almost any Reddit listing URL and you get the same data the old web UI used: Plain Text https://www.reddit.com/r/golang/new.json?limit=100&raw_json=1 limit goes up to 100 items per page.raw_json=1 stops Reddit from HTML-escaping &, < and > in titles.after is the pagination cursor (a "fullname" like t3_1abc23). The response is a Listing envelope: JSON { "kind": "Listing", "data": { "after": "t3_1abc23", "children": [ { "kind": "t3", "data": { "id": "1abc23", "title": "...", "score": 42, ... } } ] } } You only need to model the fields you use. Go's encoding/json ignores the rest, which makes the scraper resilient when Reddit adds fields. The Scraper Go package main import ( "encoding/csv" "encoding/json" "fmt" "net/http" "os" "strconv" "time" ) type listing struct { Data struct { After *string `json:"after"` Children []struct { Data post `json:"data"` } `json:"children"` } `json:"data"` } type post struct { ID string `json:"id"` Title string `json:"title"` Author string `json:"author"` Score int `json:"score"` NumComments int `json:"num_comments"` CreatedUTC float64 `json:"created_utc"` Permalink string `json:"permalink"` } var client = &http.Client{Timeout: 15 * time.Second} func fetchPage(sub, sort, after string) (*listing, error) { url := fmt.Sprintf("https://www.reddit.com/r/%s/%s.json?limit=100&raw_json=1", sub, sort) if after != "" { url += "&after=" + after } for attempt := 0; attempt < 4; attempt++ { req, _ := http.NewRequest("GET", url, nil) // Reddit blocks generic user agents. Identify your script honestly. req.Header.Set("User-Agent", "go:subreddit-archiver:v0.1 (by /u/your_username)") resp, err := client.Do(req) if err != nil { return nil, err } if resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= 500 { resp.Body.Close() wait := time.Duration(1<<attempt) * 5 * time.Second fmt.Fprintf(os.Stderr, "status %d, backing off %s\n", resp.StatusCode, wait) time.Sleep(wait) continue } if resp.StatusCode != http.StatusOK { resp.Body.Close() return nil, fmt.Errorf("unexpected status %d", resp.StatusCode) } var l listing err = json.NewDecoder(resp.Body).Decode(&l) resp.Body.Close() return &l, err } return nil, fmt.Errorf("gave up after retries") } func main() { sub := "golang" if len(os.Args) > 1 { sub = os.Args[1] } w := csv.NewWriter(os.Stdout) defer w.Flush() w.Write([]string{"id", "created", "score", "comments", "author", "title", "url"}) seen := map[string]bool{} after := "" for page := 0; page < 10; page++ { // listings stop around 1,000 items l, err := fetchPage(sub, "new", after) if err != nil { fmt.Fprintln(os.Stderr, "error:", err) return } for _, c := range l.Data.Children { p := c.Data if seen[p.ID] { continue } seen[p.ID] = true w.Write([]string{ p.ID, time.Unix(int64(p.CreatedUTC), 0).UTC().Format(time.RFC3339), strconv.Itoa(p.Score), strconv.Itoa(p.NumComments), p.Author, p.Title, "https://www.reddit.com" + p.Permalink, }) } if l.Data.After == nil { break } after = *l.Data.After time.Sleep(2 * time.Second) // stay well under the unauthenticated limit } } Run it: Shell go mod init redditscraper go run . golang > golang.csv Design Notes After *string: after is null on the last page. A pointer tells "no more pages" apart from an empty string.Deduplication: on /new, posts can shift between pages while you paginate, so the same ID can show up twice. A map[string]bool is enough for one run. For recurring jobs, persist IDs (SQLite with id as the primary key works well).created_utc is a float: decode it as float64. Decoding into int64 fails on some responses.Streaming CSV: rows are written as they arrive, so memory use stays flat no matter how big the run is. Where It Breaks This is the part most tutorials skip. 1. The 1,000-item ceiling. Every Reddit listing (new, top, hot, search) stops paginating at about 1,000 items. You cannot backfill a busy subreddit's full history from listings. To keep a complete archive, run the scraper on a schedule (for example, every 15 minutes against /new) and accumulate results yourself. 2. 403s from cloud IPs. When I ran the code above from a cloud VM, Reddit returned 403 Forbidden on the first request, even with a descriptive User-Agent. The same code typically works from a residential connection. Reddit filters unauthenticated traffic from datacenter ranges aggressively, so a scraper that works on your laptop can fail as soon as you deploy it. Retrying a 403 doesn't help, which is why the code only retries 429 and 5xx. 3. Rate limits. Unauthenticated access is throttled hard, and the limits aren't published. Watch for 429, and when they're present, respect the x-ratelimit-remaining and x-ratelimit-reset headers. A 2-second delay between pages is a conservative starting point. 4. Comments are a tree, not a list. /comments/<id>.json returns a two-element array (the post, then the comment tree). Deep threads are truncated with "kind": "more" stubs that need extra requests to expand. Budget for this: it's usually where most of your request volume goes. 5. Deleted content. Authors show up as [deleted] and bodies as [removed]. Decide up front whether to keep those rows. The Official Route: OAuth For a sustained, deployable workload, register a "script" app at reddit.com/prefs/apps, use the client-credentials flow to get a bearer token, and send requests to https://oauth.reddit.com instead of www.reddit.com. The listing JSON is the same, so the structs above can be reused unchanged. You get documented rate limits and a far lower chance of 403s. The trade-off is Reddit's API terms and pricing for commercial or high-volume use. The managed route If you'd rather not run proxies or babysit tokens, a hosted endpoint can return the same data as normalized JSON. With ScrapeAtlas*, for example, the subreddit listing above becomes a single authenticated GET: req, _ := http.NewRequest("GET", "https://api.scrapeatlas.com/v1/reddit/subreddit?subreddit=golang&sort=new", nil) req.Header.Set("x-api-key", os.Getenv("SCRAPEATLAS_API_KEY")) The response carries a posts array (id, title, author, score, num_comments, created_utc, url, ...), an after cursor for the next page, and a coverage object that reports whether the result is complete. Optional since/until parameters bound the time window. The same API has endpoints for post comments (/v1/reddit/post/comments) and global search (/v1/reddit/search). The post struct above maps onto that response almost one-to-one, so switching backends is mostly a URL change. Choosing an approach NeedApproachOne-off export from your laptopPublic .json + the code aboveRecurring job, non-commercialOAuth script appProduction pipeline, no infra to maintainManaged APIFull historical backfillNone of the above alone; accumulate over time or license a dataset Conclusion Go is a great fit for this job: static binaries, a solid standard-library HTTP client, and encoding/json that tolerates schema drift. The scraping itself is easy. The real engineering work is in handling limits: the 1,000-item ceiling, IP reputation, and rate limits. Design for those from day one and your scraper will keep running after you deploy it. *I build it.
If you've sat through more than one architecture review, you've probably sat through a protracted Kotlin-versus-Java debate that goes nowhere. The team spends weeks comparing Kotlin's concise syntax to Java's maturity, runs a poll on developer preference, and produces a decision matrix nobody fully trusts. Then the meeting ends, and nothing changes, because nobody agreed on what problem they were actually trying to solve. Choosing between infrastructure instruments, such as Kotlin and Java, is an investment decision, and like any investment decision, it needs a thesis. Four business pressures should drive the discussion: time-to-market, operational risk, talent economics, and cross-platform consistency. The Problem With Asking Kotlin or Java When the question of Kotlin or Java arises, syntax and feature comparisons dominate the discussion, because they're concrete and measurable, and a natural place to start. You can count lines of code, benchmark compile times, and run surveys on developer satisfaction, but none of that will tell you if those metrics connect to any business outcome you actually care about. A language can be a good technical fit and still be a poor investment for a particular system. A stable backend handling 50,000 transactions a day with a three-person maintenance team faces a different set of risks than a payments service changing weekly under regulatory pressure, and both look different again from a monorepo running the same business logic across Android, backend, and edge. The JVM language strategy question comes down to where complexity actually lives in the system. Ask where Kotlin meaningfully supports your modernization strategy, and where staying with Java makes better sense right now. Modern Java May Be the Right Answer For a stable, low-change system, modernizing the existing Java stack can address the underlying problem without introducing the costs of another language, toolchain, or learning curve. If the system's main constraints don't align with Kotlin's strengths, staying with Java may offer a better return. If you're considering Kotlin, first identify where its capabilities address a problem worth solving. Start With the Business Problem Drawing on Yuri Geronimus's experience as Senior Staff Engineer and Chief Architect for EMEA Enterprise Payments Platforms at Verifone, the Kotlin Enterprise Playbook frames Kotlin adoption as an enterprise investment decision rather than a language preference. It organizes the business case around four key levers: delivery throughput and time-to-market, defect-cost avoidance, legacy modernization, and talent economics. Time-to-Market If your delivery lead time affects revenue, product launches, or certification cycles, ask whether the team can deliver faster. Kotlin's concise syntax means less boilerplate per feature, and fewer null pointer exceptions reaching production translates to less time spent on hotfixes and incident response. Whether that translates to faster delivery depends on where your bottlenecks sit. If the constraint is product clarity or test coverage rather than boilerplate or type errors, language choice won't fix it. Operational Risk Where are your expensive defects concentrated? Kotlin's type system encodes nullability explicitly, allowing some null-related errors to be caught at compile time rather than in production. For systems where a single production bug triggers a compliance review or a customer-facing outage, that compile-time safety has measurable financial value. For a read-heavy reporting service where reliability is already strong, the business case is thinner. Talent Economics The cost of a JVM talent problem can show up in longer time-to-hire, higher onboarding overhead, attrition, or difficulty moving engineers between teams. Java developers can build on their existing JVM experience when learning Kotlin, making internal mobility a potential part of the business case. The strategic partnership between JetBrains and Spring, announced in May 2025, also means that 27% of Spring developers who've already used Kotlin are a potential internal resource pool you may not have fully mapped. Cross-Platform Consistency If your organization maintains duplicated business logic across an Android app, a backend service, and an edge deployment, that duplication is an operational risk disguised as a technical decision. Kotlin Multiplatform (KMP) can let teams share domain logic, models, and validation across platforms, potentially reducing duplicated implementation and the risk of logic drift. That also reduces the compliance and maintenance burden of finding and fixing the same bug in multiple places. Keep in mind that you should evaluate the case for KMP separately, with the scope of shared code kept deliberately narrow. Kotlin doesn't automatically improve time-to-market any more than a new CI tool does. The question is whether you've identified the constraint it's positioned to address. Measure the Outcome You Actually Care About To make a solid business case for Kotlin, you need a measurable hypothesis. Each business lever maps to an observable metric. Time-to-market connects to delivery lead time and deployment frequency (both DORA metrics your platform team likely already tracks). Operational risk connects to change failure rate, mean time to recovery, and incident cost. Talent economics connects to time-to-hire, time to first commit after onboarding, and attrition rate. Cross-platform consistency connects to duplicated-logic defect rate and the volume of bugs that required fixes in more than one codebase. Those metrics support the business case, but you still have to translate them into financial terms. For an enterprise investment decision, connect the operational improvement to a financial outcome. Faster delivery can reduce the cost of delay, fewer incidents can reduce remediation and customer-impact costs, and shorter onboarding can reduce hiring and training costs. Bear in mind that productivity gains don't translate directly into headcount savings. Faster delivery can instead create capacity for new product work or reduce the cost of delay on a revenue-generating feature. Conflating the two overstates the financial case and creates a credibility problem when someone on the finance team models it out. Also, don't count productivity gains as both lower operating costs and increased revenue in the same proposal. A hypothesis that says "we expect Kotlin's null safety to reduce null-related production incidents by X% over 12 months, based on our current incident log showing Y% of incidents trace to NPEs" is defensible. A hypothesis that says "Kotlin will improve developer productivity" gives you nothing to measure against. Interoperability Makes This a Scoped Decision The Kotlin-or-Java framing misleads because it implies a binary. Kotlin compiles to JVM bytecode and is fully interoperable with Java, so Kotlin and Java files can coexist in the same project without a rewrite. Your existing Spring Boot or Hibernate configurations stay in place. Your Java libraries continue working. A single new service, a shared utilities module, or even the test layer of an existing Java codebase are all valid starting points for introducing Kotlin. IntelliJ IDEA, JetBrains' own IDE, includes a Java-to-Kotlin converter and first-class mixed-project support, so the tooling overhead of running both is lower than teams typically assume. That shifts the question from "should we adopt Kotlin?" to "where should we introduce Kotlin first?" A new, high-change service with a measurable business outcome may be a reasonable candidate. A stable order-management system with a small team and clean Java code can probably wait on Kotlin until a clearer business case emerges. Account for the Iceberg and the J-Curve An honest transition plan accounts for costs that typically don't appear in the optimistic spreadsheet. The J-curve is real. Before a team reaches full Kotlin proficiency, velocity drops. Training takes time, mixed-language code reviews add overhead (reviewers need to understand both languages to catch idiomatic issues), and CI pipelines may need reconfiguration for multi-language builds. Compliance documentation may need updating if your audit trail references language versions or toolchain components. The eventual decommissioning of legacy Java code, if that's the direction, carries its own cost that's easy to leave out of a five-year model. Scope the initial experiment carefully, and build these costs into the transition plan upfront. A proposal that acknowledges the J-curve and plans for it presents a more credible case than one that doesn't. Three Questions for Your Next Architecture Review If your team is heading into an architecture review where Kotlin is on the agenda, these three questions cut through the syntax debate and turn it into a scoped, testable proposal. Where are the expensive defects or operational problems concentrated? Pull the last 12 months of production incidents and categorize the ones with the highest cost (in engineer time, customer impact, or compliance exposure). If a significant share traces to null-related errors or type-safety failures, Kotlin's type system is directly relevant. If the distribution looks different, the investment case needs a different justification. What does your JVM talent pipeline actually look like? Map your current team's Kotlin exposure, your open Java roles, and the average time-to-hire over the past year. If you're competing for senior Java engineers in a tight market and losing candidates to organizations with more modern stacks, that's a talent economics argument worth quantifying. If your team is stable and hiring is closed, the argument doesn't apply. Which service, module, or domain is the lowest-risk place to test whether Kotlin improves the target outcome? Look for something actively changing (so you'll have delivery-cycle data to measure), with a well-defined scope, and without deep compliance dependencies that would complicate a mid-stream language shift. New functionality is often the best candidate. Adding Kotlin tests to an existing Java service can also provide a relatively low-production-risk way to gain experience with the language and toolchain. These questions convert an abstract language debate into a scoped experiment with a falsifiable hypothesis. Run the experiment, measure the result, and expand where it works. Make the Strategic Decision Your CFO and CTO won't ever see a line item for Java versus Kotlin, but they will see the costs of delivery lead time, incidents, onboarding overhead, and duplicated-logic defects. The strategic decision is where to invest to improve those outcomes, domain by domain, with measurement in place. Evaluate Kotlin where you have a number to test against, start with the lowest-risk entry point, and leave stable systems alone when the case is weak. The goal is a better outcome where the investment is justified. If you want to see how other enterprise teams have structured this transition, JetBrains has a practical case for it at kotlinlang.org/lp/enterprise-playbook. How has your team navigated this decision? Share your experience in the comments.
If you are responsible for keeping a production system alive, you have probably been pitched an "AI SRE" in the last few months. Maybe more than one. Maybe more than one this week. When we sat down at Bronto to map that space, we ended up with 64 tools across seven camps: pure-play investigation agents, observability platforms with AI built in, incident management tools with AI on top, AIOps and alert correlation, hyperscaler agents like AWS DevOps Agent and Azure SRE Agent, Kubernetes copilots and open source projects, and a long tail of adjacent tools. All of them use the same two words on their landing page, and they solve very different problems. So the first question most teams ask is the classic one: do we buy, or do we build? In our landscape article, we argue that the choice isn't binary. You can go full DIY, build on open source foundations like HolmesGPT or K8sGPT, buy a product and extend it, or buy it as-is. Most teams land somewhere in the middle. In this article, I want to make a different argument for building one, at least once: build to understand what you buy. Even if you never run your own DIY agent in production, putting one together by hand shows you where its answers come from, what it can't see, and why it goes wrong when it does. After building one, you know which questions to ask a vendor, and you know how to get better results from whatever you end up running. That's the profit you should take away from this post. The fun is the building and seeing your agent try to investigate an incident: it might get it wrong, it might get it right, but you'll see how it interacts with your tools and your data. This post is based on a hands-on lab we ran at an AWS and Bronto evening, and everything is available through bronto-community/ai-sre-lab. By the end, you will have a small agent with the following capabilities: reads logs, traces, and metrics through an MCP server,reads the application's source code and commit history through GitHub's MCP server,writes its hypothesis, evidence, and confidence into a GitHub issue. Note that in the lab we use Bronto as the observability backend. I work there, so I am biased to think it is your best option, but you can of course use any other solution. What an AI SRE Is AI SRE is mostly a marketing term. In short, it's an agentic loop pointed at the production system that can answer questions about it and help you troubleshoot. Like any agentic loop, it has inputs on one side (an alert, a question from a human), a model in the middle, tools underneath (mostly MCP servers these days, for telemetry, code, cloud APIs, runbooks), and outputs on the other side (a Slack message, a ticket, a pull request). So, it's an agent like any other. It's what you connect to it that makes it an agent for SRE. The Patient: A Small Online Shop Our AI SRE needs something to investigate. The lab comes with a tiny storefront: four Python services instrumented with OpenTelemetry, plus a payment processor mock standing in for a third party. You can, of course, point the agent directly at your own data. It's read-only, so it is not able to harm your system. What You Need To run this lab, you need the following: A container engine, like Docker DesktopAn AWS account with access to a model on Amazon Bedrock.A Bronto API key (there is a free trial), or another observability backend with an MCP server.A GitHub account and a fine-grained personal access token. It needs to read the contents of your copy of the storefront repo and write issues to it.Optional, for nicer output: jq and glow. You can watch this video to follow along as we walk through each step: Start the Shop Clone the lab and generate the storefront repository. The script builds a small Git history with a handful of commits and tags two releases, v3.1.0 (healthy) and v3.2.0: Shell git clone https://github.com/bronto-community/ai-sre-lab cd ai-sre-lab ./build-storefront.sh Your agent will later read this code through GitHub, so push it to a repository of your own, tags included: Shell cd storefront gh repo create storefront --public --source . --push git push --tags cd .. Then point the Collector at your telemetry backend and bring everything up: Shell cd demo echo "BRONTO_API_KEY=<your key>" > .env make up Note, that demo/collector.yaml ships to Bronto's EU ingestion endpoint. If your account lives in another region, or you want a different backend, that file is the only thing you need to change. Next, run make up, which deploys v3.1.0 and starts the traffic. It prints a message that you should leave it about ten minutes before you do anything else. An investigator with no baseline has nothing to compare against. So, take your time going through the first few next steps so it can build up! Step 1: The Loop All agents start with a basic file that receives prompts, sends them to the model, and returns the results. It's a simple first agent, so here is more or less the code in full: Python import os from bedrock_agentcore import BedrockAgentCoreApp from strands import Agent from strands.models import BedrockModel model = BedrockModel( model_id=os.environ.get("BEDROCK_MODEL_ID", "us.anthropic.claude-haiku-4-5-20251001-v1:0"), region_name=os.environ.get("AWS_REGION") or os.environ.get("AWS_DEFAULT_REGION", "us-west-2"), ) app = BedrockAgentCoreApp() @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model) return {"result": str(agent(payload.get("prompt", "Hello!")))} if __name__ == "__main__": app.run() # POST /invocations, GET /ping on :8080 Two libraries do the work. Strands Agents runs the agent loop: send the prompt to the model, run any tool the model asks for, feed the result back, repeat until the model has an answer. The Bedrock AgentCore SDK wraps that in an HTTP server with the /invocations and /ping contract AgentCore Runtime expects. Today that runs on your laptop. Later, the same container can run on AgentCore Runtime without changes. Every step of the lab is a prebuilt image, so you don't need to build anything. Create a .env next to where you run the commands: Shell AWS_DEFAULT_REGION=us-west-2 AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_SESSION_TOKEN=... BEDROCK_MODEL_ID=us.anthropic.claude-haiku-4-5-20251001-v1:0 With this setup, you can start the agent for the first time: Shell docker run --rm -p 8080:8080 --env-file .env ghcr.io/bronto-community/ai-sre-agent:step-1 Open a second terminal and ask it something: Shell curl -s localhost:8080/invocations -d '{"prompt": "Who are you?"}' | jq -r .result It will say, "Claude, an AI assistant made by Anthropic." Correct, and completely useless for incident response. Right now this is a model behind an endpoint, with no identity and no tools. Step 2: Identity The second step changes only one line of code. Every Markdown file in prompts/ becomes part of the system prompt: Python SYSTEM_PROMPT = "\n\n".join(p.read_text() for p in sorted(Path("prompts").glob("*.md"))) @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model, system_prompt=SYSTEM_PROMPT) ... The image ships with two files. role.md: Markdown You are the on-call SRE for Storefront, an online shop. You investigate production incidents: what broke, where, and why. And rules.md: Markdown - Only state what you can back with evidence from your tools. - If you have no tool that can answer, say so and name what you would need. - Be brief. You also get to add your own. Create a my.md: Markdown - Answer in at most three bullet points. - End every answer with your confidence: low, medium or high. - Sign off as "your friendly on-call dino". Start step 2 as follows. The command contains your custom instructions as an additional volume: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-2 Shell curl -s localhost:8080/invocations -d '{"prompt": "Checkout is slow. What is wrong?"}' | jq -r .result You should get a short answer that says it has no access to telemetry, rates its own confidence as low, and signs off as a dino. That's a good result. The rule "say so and name what you would need" did its job: an agent that admits it can't see anything is far more useful than one that guesses. Step 3: Eyes Now the agent gets something to look at. Step 3 adds one tool source, Bronto's MCP server: Python bronto = MCPClient(lambda: streamable_http_client( os.environ.get("BRONTO_MCP_URL", "https://mcp.eu.bronto.io/mcp"), http_client=httpx2.AsyncClient( headers={"X-BRONTO-API-KEY": os.environ["BRONTO_API_KEY"]}, timeout=60, ), )) @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto]) ... That's the whole change. The MCP server's tools (search logs, list datasets, and so on) become tools the agent can call, and Strands handles the plumbing. This is also the point where MCP pays off: if your telemetry lives somewhere else, you swap this client for that vendor's MCP server and keep everything else. Add BRONTO_API_KEY to your .env (and BRONTO_MCP_URL if your account isn't in the EU region), then: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-3 Start with a question that tells you what it can see: Shell curl -s localhost:8080/invocations -d '{"prompt": "What services do you know about?"}' | jq -r .result It should list the storefront services. Go ahead and ask it some more questions about the state of those services: what HTTP endpoints are available? What is the average error rate right now? With this, you have a first working version. Next we want to investigate an incident. To do so, ship the next release of the storefront: Shell make release According to the Makefile, symptoms take about 60 seconds to show up in p95. So wait a minute or two and then ask it: Shell curl -s localhost:8080/invocations -d '{"prompt": "Anything wrong with checkout?"}' | jq -r .result It will come back with a hypothesis about what is wrong and what you might be able to do about it. Take a look at the telemetry yourself and check: is the analysis correct, or did the agent pick up some false information? Step 4: Mouth An answer in your terminal is nice for a demo, but in production you would want to have it sent to a Slack channel, a pager app, or an issue tracker. In our example, we send the answer as a GitHub issue. This would let you tag your coding agent to review the issue and open a pull request that fixes it. You now have two options for how this gets filed. Either you give your agent another tool, and it can decide for itself where to send the hypothesis, or you make the code file the issue, so every analysis gets pushed into a dedicated issue: Shell def file_issue(question: str, hypothesis: str) -> str: r = httpx2.post( f"https://api.github.com/repos/{os.environ.get('GITHUB_REPO', 'ai-sre-lab/storefront')}/issues", headers={"Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}"}, json={"title": f"[investigation] {question}"[:120], "body": hypothesis}, timeout=30, ) r.raise_for_status() return r.json()["html_url"] @app.entrypoint def invoke(payload: dict) -> dict: question = payload.get("prompt", "Hello!") agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto]) hypothesis = agent(question) return {"result": file_issue(question, hypothesis)} To make sure that what the agent provides fits the output, an additional new prompt file tells the agent how to write it: Markdown Your answer is filed as a GitHub issue for the on-call team. Write it as one: a one-line hypothesis, the evidence for it, your confidence, and what you could not check. Before running the next version of your agent, add your GitHub token and your repository to .env: Shell GITHUB_TOKEN=github_pat_... GITHUB_REPO=<your-user>/storefront Run step 4: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-4 Ask it again to investigate the issue: Shell curl -s localhost:8080/invocations -d '{"prompt": "Anything wrong with checkout?"}' | jq -r .result The response is now a link. Open it and read the hypothesis. Is it right? How confident is it? There's a good chance it's confidently wrong. While testing the lab, one run concluded that checkout was down because a query returned nothing: an empty result is not proof of an outage, but nothing told the agent that. Step 5: Better Instructions What you do next doesn't involve code. You do what you'd do with a new human on the rotation: you tell them how you investigate. Add these lines to your my.md: Markdown - Slow is broken too: compare latency (p50, p95) before and after, not only errors. - Compare against a quiet window earlier the same day. - Look for `deploy` spans near the onset: they name the commit that went live. Each line encodes a habit experienced SREs have, and models don't bring by default. Checkout can be very broken without throwing a single error. A number means little without a baseline. And the first question in most incidents is "what changed?" The prompt files are read when the container starts, so restart step 4 with the same command and ask a more realistic question: Shell curl -s localhost:8080/invocations \ -d '{"prompt": "Customers say checkout got slow in the last 30 minutes. What is going on?"}' | jq -r .result Compare the new issue with the old one. Is it better? Worse? Did it fail outright? Step 6: Better Model As you might now, a different model can create a different output, so if you are not satisfied with the results yet, change one line in .env: Shell BEDROCK_MODEL_ID=us.amazon.nova-lite-v1:0 Restart your agent, ask the question again, and compare the outputs again: better or worse? Step 7: Code So far the agent can see that something changed and maybe even when. If your instructions worked, it found the deploy. What it can't do is look at what was deployed. Step 7 adds the second MCP server, GitHub's, restricted to read-only repository tools: Python github = MCPClient(lambda: streamable_http_client( os.environ.get("GITHUB_MCP_URL", "https://api.githubcopilot.com/mcp/x/repos/readonly"), http_client=httpx2.AsyncClient( headers={"Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}"}, timeout=60, ), )) @app.entrypoint def invoke(payload: dict) -> dict: question = payload.get("prompt", "Hello!") agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto, github]) ... To make it useful to the agent, add one more prompt file, code.md. It connects the telemetry to the code: Markdown The shop's source code is in the GitHub repository `ai-sre-lab/storefront`. Every service reports the commit it runs as `deployment.commit`. Replace the repository path with your own, then mount it as an additional volume: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" \ -v "$PWD/code.md:/app/prompts/code.md" \ ghcr.io/bronto-community/ai-sre-agent:step-7 Ask the same question again: Shell curl -s localhost:8080/invocations \ -d '{"prompt": "Customers say checkout got slow in the last 30 minutes. What is going on?"}' | jq -r .result Does it find the deploy? Does it open the release and pick the right commit? In our testing, it found the deploy marker and the commit, and then stopped. The release contains several commits, and the last hop, reading every one of them and figuring out which change explains the symptom, didn't happen on its own. If you experience the same, try to fix it yourself by refining your system prompts or by trying yet another model. When you're done, make rollback puts the healthy version back, and make down cleans everything up. If you want to give your agent a harder case, make psp-slow degrades the third-party payment processor. So, Buy or Build? Now you know what an AI SRE is and how it works, from the inside. That's a much better position from which to answer the question this post started with. If you buy, a lot of what hurt along the way becomes someone else's problem: confidently wrong answers that need endless prompt tuning, context windows overflowing with search results, wiring up every integration yourself. AWS DevOps Agent, for example, starts investigating when an alert fires and connects to the tools you likely already run, and you can still extend it with custom skills and your own tools. Take the four questions from the landscape article with you into every demo: Where does it get its data? Does it act, or only advise? Can you see its reasoning? What happens when it's wrong? After this lab, you'll know what good and bad answers to those look like. If you build, what you have now is a skeleton, and you can build more features towards a complete picture on top of it: Let alerts trigger it instead of curl.Run it on AgentCore Runtime instead of your laptop (agentcore deploy).Give it more tools: your cluster, your cloud APIs, your past incidents and postmortems.Observe the agent itself. Its own traces tell you which tools it called, how many tokens it burned, and where it went wrong, and that's the input you need to improve it. Either way, the data underneath matters more than the agent on top. Every improvement in this lab depended on the telemetry being there: a baseline from ten quiet minutes, latency and not just errors, a deployment.commit attribute that connects a span to a line of code. An agent can only reason about what your system tells it. That was true for humans long before agents showed up. So: what would your agent get wrong? The full lab, including the slides, the agent images for every step, and the storefront harness, is at github.com/bronto-community/ai-sre-lab. Content is CC BY 4.0; code is MIT. If you build something on top of it, I'd love to hear about it. I've shown how easy it is to build a simple AI SRE agent for fun. If you want to see what a production-ready one looks like, read our post on the Bronto blog, Introducing Bronto's Open-Source Alert Investigation Agent. It's an open-source agent that replicates the Alert Investigation feature in Bronto, and it's essentially our production version of this lab: a Bronto monitor triggers it instead of curl, it sticks to a constrained set of tools and reasoning paths instead of improvising, and it hands you a report with likely root causes and suggested next steps. The code is at bronto-community/bronto-alert-investigation-agent.
Most frontend applications were built with the success path in mind. A user will typically open a webpage, load their data, complete a form, submit it, and receive a response or confirmation to allow them to continue on their way. That's the typical scenario you'll see in demos, design reviews, and when walking through your application with a customer. However, real customers don't follow the happy path. Their internet speeds might be slow. An API could return a 500 error. The session could time out while completing a checkout process. A user can accidentally double-click a button. A form submission might fail partially. A required third-party service could be down. The browser could refresh at exactly the wrong moment. Therefore, in production, failure is not something that happens by accident; it's simply part of how most people interact with your web application. This is why frontend applications should be designed not only for success, but also for failure. The Happy Path Is Not Enough The happy path in the development process is key to developing an optimal user journey. This will help a team develop the product flow as well as design the base of their experience. If your application only functions properly under the best possible conditions, then you have not developed a production-ready application. If your frontend application is designed with the assumption of successful actions by the user, it could be very frustrating for them. For example, the loading indicator on a page could run indefinitely. The submit button could accept multiple click events from the same action. When an error message was generated in the application, it could simply state something like "An error has occurred" without providing any guidance to the user about how to proceed. Finally, when there were problems accessing APIs associated with content displayed on pages, users may see a blank page with no indication as to why they are seeing this. In addition to being frustrating for users, losing all inputted information due to a page refresh or timed-out session is also a reliability problem. An effective frontend application should provide answers to at least two questions: What does the application look like while working properly? And what happens if things do not work correctly? Loading States Should Be Intentional Loading states are often overlooked as small details, but they set the reliability of the user's first impression. A generic loader for a short wait might be sufficient, but it isn't going to cut it for all situations. Skeleton loading is useful on pages with structured content to give the interface a sense of stability. On forms, the submit button needs to clearly indicate that processing is taking place. For long-running tasks, users will want either a progress message or some indication that the system is still processing their request. Ultimately, your goal when designing a loading state is to avoid ambiguity. The user shouldn't have to wonder if the app is frozen, still loading, or simply broken. A good loading state communicates that the system has received the action and is working on it. Error States Need Recovery Paths Most applications have some type of error message, yet fewer than most offer paths for users to recover from these errors. When an application gives you an error such as "Failed to Load Data," then that is factually accurate; however, this information is not helpful to the user at all. Good error processing will tell the user what occurred, what actions they need to take next, and whether their previous efforts are saved. An example would be if a payment process failed prior to charging your card; in this case, the application could inform the user that they should try again. In cases where the application cannot determine whether the payment was successful, the user should not be prompted to resubmit. Instead, the application should suggest that the user either wait and see, attempt to refresh and try again, or reach out to support. Error states should ideally be unique when possible. Validation errors, permission errors, network timeouts, server-side errors, and expired sessions are examples of types of potential errors, each of which may require a completely different way of recovering from. The goal is not just to display an error. The goal is to help the user continue safely. Forms Should Protect User Input User frustration often stems from failures at the frontend level. A user may have spent many minutes entering data into a form that will now be lost when their session expires or fails on an API call. The best way for forms to recover from user errors and maintain usability is by preserving user-entered input. There are several methods to achieve this, including use of local state, draft saving, autosaving, and backing up form entries in the backend with partial submissions. Most users would appreciate minor improvements to usability, even if simply retaining field values during failed submits. Additionally, forms should prevent multiple submissions. Disabling a form's submit button once the first time it has been clicked is a good practice; however, disabling a form's submit button does not ensure against duplicate submissions being made to the backend. The backend method for handling duplicate requests must be safe. Safe handling of duplicate requests can be achieved through backend processes, including idempotent key patterns. Partial Failure Should Not Break the Whole Page When web pages have many dependent APIs for different things (e.g., profile info, suggestions, bill, etc.), if one of those APIs fails, it does not necessarily mean that the whole page has to be unavailable. It would make sense to create some type of partial failure state so that although the whole page cannot operate as intended, you could still use other parts of the page while a second part of the page provides an option to retry. A dashboard may provide a list of available areas but mark sections that are unable to load. And though you may be able to see basic data about a product, in the event that its reviews or suggestions fail, you will still be able to review it. Ultimately, this method allows your users to feel like they have access to more than just what is currently working. Expired Sessions Should Be Handled Gracefully A typical problem with how many programs manage session timeouts is that they abruptly send users to a login screen when the user's session has timed out. It is possible for an application to be designed so that if the user's session times out while using the application, after asking the user to log back in, it will direct the user back to where they were in the process as opposed to being sent back to the beginning of the process. The loss of information on forms makes this particularly important. In general, from the user's perspective, timing out of the user's session should be seen as just a minor disruption rather than a full stop. Observability Matters on the Frontend Designing for failure is not only about UI states. Teams also need visibility into how failures happen in production. Teams can monitor these factors as well as other user behaviors to determine where users are experiencing problems in order to develop design solutions to address them. Tracking failed API calls, JavaScript errors, slow load times, rage clicking, repeated retry attempts, form abandonment, and unusual navigation patterns through frontend applications will give teams insight into where users are experiencing problems. While backend services may be running smoothly and providing a healthy status, it doesn't necessarily mean that users have a good experience with your product. Frontends are typically where issues in production are most visible. Conclusion Frontend apps typically get judged on how visually appealing they appear while running smoothly. The true measure of a frontend app's production quality is when it actually breaks. It is important to note that designing for failure does not make an application less positive or overly defensive; instead, it creates a sense of trustworthiness with the end-user. When designing for failure, the application demonstrates respect for the user's time and effort. Additionally, the design of an application to fail properly protects the user from unintended consequences due to its preparation for a variety of real-world operating conditions. The happy path may help you demonstrate your application to potential clients, but successful design for failure builds trust with your users.
Your deploy pipeline versions everything except the thing that matters most: what the agent actually does. Code is versioned. Configs are versioned. The prompt is probably in a repo somewhere, hopefully with a review attached. But the agent's behavior, the mapping from inputs to actions, shifts whenever any of its inputs change: a model version bump you did not ask for, a retrieval corpus that gained three thousand documents overnight, a tool schema change from another team, a "small" prompt edit to fix one bad answer in a demo. None of those go through a behavioral regression test. They go straight to production, where your users run the eval suite for you, one incident at a time. This is the gap evals in CI close. If you ship agents, behavior is your build artifact. Test it like one. Unit Tests Cover the Shell, Not the Brain A typical agent codebase has two halves. The deterministic shell: tool wrappers, parsers, retry logic, the policy gate. That half has unit tests, because it is code and code gets tested. The probabilistic core: the planner that decides which tool to call, with what arguments, in what order, and when to stop. That half runs on vibes. The failure mode is always the same. Someone tweaks a prompt to fix a bad answer in a demo. The fix works. Three weeks later, support notices the agent stopped escalating refund requests above the threshold, or started calling the wrong search endpoint, or doubled its average tool calls per task. Nobody connects the regression to the prompt edit, because nothing measured behavior before and after. The prompt diff looked harmless. Behavior diffs are invisible until they get expensive. Evals make the behavior diff visible. An eval is a fixed input with a recorded expectation about behavior, run the same way every time, scored the same way every time. A suite of them is a regression test for the part of your system no unit test can reach. Build the Suite in Three Layers A useful eval suite is not one big pile of test cases. It is three layers with different jobs. The golden set. A curated set of representative inputs with labeled expected behavior, versioned in git next to the prompts. Keep it small enough to run on every pull request, a few hundred cases at most. The expectations should be behavioral and checkable: the agent called the right tool with valid arguments, it refused where policy requires refusal, it escalated instead of acting on an irreversible step, the final answer carried the required citation. Deterministic assertions wherever possible. This is your smoke test. Scenario suites. Multi-turn, tool-using scenarios run against scripted mock tools so they are repeatable. Real agents fail in the middle of plans, not at the start: the third tool call returns an error, the retrieved document contradicts the user's request, the plan needs replanning halfway through. Scenario evals exercise the planner across a full trajectory and assert on the trajectory, not just the final answer. Did it recover from the tool error or loop forever? Did it notice the contradiction or bulldoze through it? These run nightly or on significant changes, because they are slower and occasionally flaky. The adversarial set. Inputs designed to break the agent: injection attempts smuggled through tool outputs, permission-boundary probes ("just run it as admin"), conflicting instructions from untrusted documents, edge cases in tool arguments. This set encodes your threat model as executable tests. Every incident and every red-team finding becomes a new case here. If your layered defenses matter, and they do, this is where you prove they still hold after each change instead of just asserting it. Score Without Lying to Yourself The fastest way to waste an eval program is a single aggregate score. An average that climbed from 87 to 89 hides the three scenarios that collapsed from 100 to 40. Report per-scenario deltas against the baseline, and gate on "nothing regressed beyond tolerance," not on "the average went up." Scoring itself needs the same rigor as the suite: Deterministic checks first. Most agent behavior that matters is checkable without a model: right tool, valid arguments, correct refusal, correct escalation, required fields present, disallowed actions absent. Write these as code. They are fast, stable, and auditable. Model-as-judge for the fuzzy remainder. Some things only a reader can score: answer quality, tone, whether the explanation actually addressed the question. Use a judge model with a written rubric, and calibrate it: have humans label a sample, measure judge-human agreement, and re-calibrate when you change the judge. A judge is also a model. It drifts, it has biases, it favors verbosity. Pin its version like any other dependency, and track its disagreement rate as a metric of the eval system itself. Run the nondeterministic parts more than once. Agents are stochastic. A scenario that passes once and fails twice is telling you something different from one that fails three times. Run each scenario a few times, report the distribution, and quarantine flaky scenarios into a separate bucket instead of letting them veto every merge or, worse, getting deleted because they were annoying. Wire It Into the Merge Evals only matter if they can stop a bad change. The wiring looks like this: Python def eval_gate(change, baseline, tolerance): report = run_eval_suite(change, GOLDEN_SET + ADVERSARIAL_SET) regressions = [s for s in report.scenarios if s.score < baseline[s.id] - tolerance] if regressions: return fail(f"{len(regressions)} scenarios regressed", regressions) return pass_with_report(report) Four rules make the gate real: Every behavior-changing diff triggers the golden set. Prompts, tool definitions, model version pins, retrieval configuration, policy rules. Not just code changes. Prompts are code now; treat the diff that way.Compare against the baseline for the base commit. A scenario that regressed beyond tolerance fails the build. The report names the scenarios, shows before and after, and links the diff.Tier the slow suites. Scenario suites run nightly and on release branches. The adversarial set runs on any change touching trust boundaries: new tools, new data sources, permission changes. If the per-PR eval takes an hour, engineers will route around it. A tiered suite that actually runs beats a comprehensive suite everyone skips.Baselines update deliberately. When a change intentionally alters behavior, the new baseline gets reviewed like the code change itself, with the per-scenario diffs as the review material. Silent baseline updates are how eval suites become decoration. Measure What Agents Actually Break Generic quality scores miss the failure modes that are specific to agents. Your eval dimensions should include: Tool-call correctness. Right tool, valid arguments, sensible order. An agent that answers correctly after calling five wrong tools first did not behave correctly. It got lucky. Policy compliance. Refusals and escalations where required. This is where runtime guardrails and evals meet: the policy gate enforces at runtime, the eval suite proves the planner respects the policy across scenarios instead of routing around it. Injection resistance. The adversarial set's core metric. After every change, the agent should still refuse or neutralize injected instructions at the same rate. This is a regression metric, not a one-time audit. Trajectory efficiency. Tool calls per task, tokens per task, wall-clock time. An agent that takes forty tool calls to do a three-call job is a regression even when the answer is right. Cost and latency are behavior. Recovery behavior. What it does when a tool errors, returns garbage, or times out: retry with backoff, replan, escalate, or spiral. Script these in the scenario suite. Production will test them otherwise. The Failure Modes of Eval Programs Evals rot like any other system. Expect it and design for it. Overfitting to the golden set. Prompts get tuned until the evals pass while wild behavior stays broken. The defense is a held-out set nobody tunes against, plus sampling real production traffic into new eval cases. If your golden set never grows, your agent is being trained to pass a test. Judge drift. The judge model gets updated, its verdicts shift, and suddenly half the suite "regresses." Pin the judge, version the rubric, and investigate judge-side changes before agent-side ones. Stale scenarios. New tools ship, new features launch, and the eval suite still tests last quarter's agent. Review the suite on a cadence. Every production incident should produce a new eval case. That is the cheapest way to keep the suite honest. Eval theater. The suite runs, the dashboard is green, and nobody reads the per-scenario diffs before merging. Green without review is a ritual. Require the diff in the PR the way you require the code diff. Close the Loop With Production The eval suite is half of a loop. The other half is production telemetry feeding back into it. Sample live trajectories, label the failures, and promote them into the golden or adversarial sets. The incidents your users found become the regressions your CI catches next time. Over months, the suite becomes a record of every failure mode the system has ever learned, which is exactly what a regression suite is supposed to be. This is the part teams skip, and it is the part that compounds. An eval suite containing only cases someone imagined on day one tests the agent you used to have. The Takeaway Agents are software with probabilistic planners and deterministic effects. Ship them like software: version the behavior, test it on every change, gate merges on the results, and feed production failures back into the suite. A prompt edit that cannot pass the same bar as a code change does not merge. Your users are already running your eval suite. The only question is whether you run it first. Views expressed here are my own and do not represent my employer.
If Your Fabric Capacity Is Throttling, Do Not Start With the SKU A report takes twenty seconds to open. A pipeline that normally finishes before breakfast is still running. Someone sees the capacity above 100% and proposes the obvious fix: move from F64 to F128. That might be the right decision. It should not be the first one. Microsoft Fabric brings data engineering, warehousing, real-time analytics, data science, and Power BI onto shared capacity. That is one of its strengths. It also means one inefficient item can consume compute that another workload needs. A poorly shaped warehouse query, an oversized Spark session, an hourly refresh of data that changes once a day, or a report page with too many expensive visuals can all surface as the same complaint: “Fabric is slow.” The complaint does not tell you the cause. The useful operating model is: Confirm the capacity is throttling. Isolate the operation creating pressure. Optimize that workload. Scale only when the remaining demand is legitimate. This distinction matters because a larger SKU gives inefficient work more room to run. It does not make the work efficient. First, Separate Slow Performance From Throttling Slow performance and capacity throttling can look similar to a user, but they are different problems. A report visual can be slow because its DAX is expensive. A warehouse query can be slow because it scans far more data than it needs. A Spark job can wait because it requested more executors than are available. Network latency, gateway pressure, cold cache, locks, and source-system performance can also add delay. Throttling is more specific. It happens when billable operations consume more Capacity Unit seconds than the capacity can supply, including the future capacity Fabric uses to absorb smoothed usage. When that budget is exhausted, Fabric progressively delays or rejects new work. This leads to the first rule: Do not diagnose throttling from a slow screen or a utilization spike. Confirm it in the Capacity Metrics app. Microsoft explicitly notes that utilization above 100% does not immediately mean throttling. Fabric allows bursting, smooths usage, and provides limited overage protection before user operations are affected. Use the Throttling view and System events on the Compute page to confirm whether the capacity entered interactive delay, interactive rejection, or background rejection. Rejected requests can also surface errors such as CapacityLimitExceeded. If the app shows no throttling during the incident, investigate the item itself. Buying capacity to fix a bad query is an expensive way to avoid reading its execution plan. What Fabric Is Protecting You From Fabric does not force every operation to fit inside the CU budget of the current second. It uses two mechanisms to absorb variability. Bursting lets an operation temporarily use more compute than the provisioned SKU provides so it can finish quickly. Smoothing spreads the CU accounting for that operation across future 30-second timepoints. Interactive operations are typically smoothed over five to 64 minutes, depending on consumption. Background operations are typically smoothed over 24 hours. Smoothing is not a performance optimization. It does not make a query cheaper or slow a job down to fit the capacity. It changes how consumed compute is applied against the capacity limits. When smoothed usage continues to accumulate faster than idle capacity can pay it down, Fabric moves through progressive protection stages: Future capacity consumedProtection stageUser impactUp to 10 minutesOverage protectionNew operations continue without throttlingMore than 10 minutes, up to 60 minutesInteractive delayNew interactive operations receive a 20-second submission delayMore than 60 minutes, up to 24 hoursInteractive rejectionNew interactive operations are rejectedMore than 24 hoursBackground rejectionNew interactive and background operations are rejected Operations already running are allowed to complete. The capacity recovers when unused CUs burn down the accumulated carryforward. That delayed accounting explains a common mystery. The item users see failing at 10:00 AM might not be the item that created the pressure at 10:00 AM. A large background operation completed earlier can still be consuming future capacity through smoothing. Diagnose From the Capacity Down, Not From the complaint up When a capacity incident occurs, use the same investigation path every time. 1. Pin Down the Affected Window Record the time, capacity, workspace, reported symptom, and whether the failure affected interactive work, background work, or both. Do not begin with a full month of data. Start with the narrowest window that contains the incident. The Capacity Metrics app shows the previous 14 days of compute usage and displays hourly usage in the multimetric ribbon. Select the affected day and hour. When filters are applied, the utilization and throttling visuals can show 30-second timepoints rather than only periodic peaks. 2. Confirm the Protection Stage Open the Throttling tab and check which threshold crossed 100%: Interactive delay means the capacity consumed more than 10 minutes of future capacity.Interactive rejection means it consumed more than 60 minutes.Background rejection means it consumed more than 24 hours. Then review Overage (Carryforward). The add, burndown, and cumulative values tell you whether the capacity experienced a short surge that quickly recovered or sustained demand that kept adding debt. Minutes to burndown is especially useful. It estimates how long recovery would take if no new operations entered the capacity. 3. Find the Items That Consumed the CUs In the matrix by item and operation, sort by CU(s) for the selected window. Expand the highest consumers to see the operation type. Duration alone is not enough. A long-running operation can use modest compute, while a short burst can consume many CUs. Add the optional columns that help explain the incident: Rejected countFailed countSuccessful countIn-progress countOverloaded minutesItem sizePerformance delta Hovering over matrix values exposes the operation-level breakdown. Capture the workspace, item, operation, user or service principal, CU seconds, operation count, and time. 4. Classify the Shape of the Problem Most incidents fall into one of four patterns: PatternWhat the evidence looks likeLikely responseOne expensive operationA single run dominates CU secondsOptimize code, query, model, or data layoutExcessive frequencyA modest operation runs far more often than its data changesReduce refresh or execution frequencyHigh concurrencyMany acceptable operations overlapControl concurrency, isolate workloads, or scaleSustained business demandOptimized work remains consistently highScale up or scale out This classification prevents random tuning. It tells you whether to reduce cost per operation, reduce operation count, change where the work runs, or provide more compute. Reducing frequency and rescheduling are not the same optimization. Changing an hourly refresh to a daily refresh removes unnecessary executions and lowers total consumption. Moving the same 24 hourly refreshes to a different part of the day does not make them cheaper. It only changes when they compete with other work. Optimize the Workload That Actually Caused the Pressure Fabric workloads share capacity, but they do not share one tuning playbook. Once you know which operation consumed the CUs, focus on that workload instead of trying to optimize everything. Warehouse and SQL Analytics Endpoint Start with Query Insights to identify completed queries using the most CPU. Microsoft's Query Insights documentation includes this query: SQL SELECT TOP 100 distributed_statement_id, query_hash, allocated_cpu_time_ms, label, command FROM queryinsights.exec_requests_history ORDER BY allocated_cpu_time_ms DESC; If a query stands out, check how much data it scans, whether filters are applied early, and whether joins or statistics need attention. Compare repeated executions, not just the first run, because cold starts can distort the picture. The goal is fewer resources consumed for the same result, not simply a faster first response. Spark, Notebooks, and Lakehouse Processing Use Spark monitoring and the Tune tab to look for heavy shuffles, skewed stages, spills, or executors that sit underused. Filter data early, avoid repeated actions, and review Delta file layout when small files become a recurring problem. Also check session lifetime. Leaving compute running after the work is done is easy to overlook. If bursty Spark workloads regularly compete with reporting, consider whether separately billed on-demand Spark is a better fit. Measure the cost and isolation trade-off before making the switch. Data Factory Pipelines and Dataflow Gen2 A pipeline can orchestrate several expensive operations, so the pipeline itself may not be where the CUs are going. Check the notebook, copy, refresh, or transformation activity behind it. The simplest savings often come from removing unnecessary runs, using incremental loads, and preserving query folding in Dataflow Gen2 where supported. Moving the same work to another hour might reduce contention, but it does not reduce total CU consumption. Removing work you do not need does. Power BI Semantic Models and Reports Separate refresh pressure from interactive report pressure. For refreshes, focus on incremental processing, folding, and schedules that match how often the source changes. For reports, use Performance Analyzer to find expensive visuals and DAX measures, then compare those findings with the semantic model's operations in Capacity Metrics. A page with too many costly visuals can feel like a capacity problem even when the better fix is in the report design. Real-Time Intelligence For KQL databases and Eventhouses, inspect wide queries, repeated aggregations, retention and caching choices, and ingestion patterns. Limit scans to the data actually needed and consider pre-aggregation when queries repeatedly calculate the same result. Keep the freshness requirement in view. Lower CU consumption is not a win if the data now arrives too late to be useful. Real-Time Intelligence also has different throttling behavior, so do not assume every workload uses the same interactive delay stage. Reduce Blast Radius When Optimization Is Not Enough Some workloads are efficient and still should not share a failure boundary. An executive reporting estate, a self-service experimentation workspace, and a large overnight engineering platform can have different service expectations. If one workload is allowed to consume the headroom needed by another, the architecture is coupling them operationally even if their data products are separate. Fabric gives you several options: Scale up when optimized demand legitimately exceeds the current SKU.Scale out by moving workspaces to another capacity when isolation, administration, or service levels differ. This adds capacity cost and operational ownership, so the service-level benefit must justify it.Use on-demand billing for Spark when dynamic Spark demand should not consume the shared capacity.Configure surge protection to limit the background compute that can consume capacity and make interactive throttling less likely.Enable capacity overage billing when continuity is worth the additional cost and the feature fits your operating model. Microsoft documents overage billing at three times the normal capacity rate, so estimate the cost first. Surge protection is a guardrail, not free capacity. When it activates, background jobs can be rejected. Microsoft recommends correctly sizing and isolating critical solutions rather than treating surge protection as a replacement for optimization or capacity design. Prove That the Optimization Worked A workload is not optimized because its code changed or one test completed faster. Compare the same business operation before and after the change. Track: MeasureWhy it mattersCU seconds per operationShows whether the work became cheaperOperation countExposes unnecessary frequency and retriesTotal CU secondsShows the net capacity effectDurationMeasures user or batch completion timeData scannedReveals pruning, cache, and query-shape improvementsRejected and failed countConfirms whether capacity pressure still disrupts workCarryforward and minutes to burndownShows whether the capacity recovers fasterInteractive latencyConfirms that users experience the improvement Run the comparison over a representative period. A change that improves one query but shifts expensive work into refresh, maintenance, or another engine is not an end-to-end optimization. For recurring operations, calculate both cost per run and daily cost: Plain Text Daily CU-seconds = CU-seconds per operation × operations per day Reducing a query from 1,000 to 800 CU seconds saves 20% per run. If it runs ten times more often after the redesign, total consumption still increases substantially. Efficiency and frequency belong in the same calculation. A Practical Throttling Response Worksheet Use this during the next capacity incident: Plain Text Incident window: Capacity and SKU: User-visible symptom: Confirmed throttling stage: Carryforward and minutes to burndown: Top workspace: Top item: Top operation: Interactive or background: CU-seconds: Operation count: Rejected count: Problem shape: [ ] One expensive operation [ ] Excessive frequency [ ] High concurrency [ ] Sustained business demand Optimization applied: Before CU-seconds per operation: After CU-seconds per operation: Before daily CU-seconds: After daily CU-seconds: Problem shifted to refresh, maintenance, or another engine: Decision: [ ] Keep current SKU [ ] Scale up [ ] Isolate on another capacity [ ] Move Spark to on-demand billing [ ] Add a protective capacity control This turns a vague performance complaint into a traceable engineering decision. The Goal Is Not Zero Utilization A well-used Fabric capacity should do work. The goal is not to keep every chart comfortably low or eliminate every temporary excursion above 100%. The goal is to ensure that each operation earns the compute it consumes, critical workloads retain the service level they need, and sustained business demand fits inside a deliberate capacity envelope. When throttling appears, ask these questions in order: Did the capacity actually throttle, or is one item simply slow?Which item and operation consumed the CUs?Is the problem cost per operation, frequency, concurrency, or real growth?Can the workload be made cheaper without reducing the business outcome?If not, should the answer be more capacity or better isolation? That is the difference between reacting to throttling and operating Fabric well. Sources This article is grounded in current Microsoft documentation. Product behavior and preview features can change, so verify the latest guidance before making production capacity decisions. Understand capacity throttling and smoothing (Microsoft Learn)Capacity Metrics app compute page (Microsoft Learn)Evaluate and optimize your Microsoft Fabric capacity (Microsoft Learn)Performance guidelines for Fabric Data Warehouse (Microsoft Learn)Query Insights in Fabric Data Warehouse (Microsoft Learn)On-demand billing for Apache Spark in Microsoft Fabric (Microsoft Learn)Optimization guide for Power BI (Microsoft Learn)Manage surge protection for Fabric capacities (Microsoft Learn)Data clustering in Fabric Data Warehouse (preview) (Microsoft Learn)Partitioning for Delta tables (Microsoft Learn)
What does it take to build and sustain an open-source project for more than a decade? For Shai Almog, creator of Codename One, it takes a willingness to challenge conventional approaches to software development and a continued commitment to the developers who rely on the work. I recently caught up with Shai to discuss the inspiration behind Codename One, the realities of maintaining an open-source project, and his perspective on how the Java ecosystem has evolved over the years. Check it out below! 1. What is Codename One, and what originally inspired you to create it? Codename One is what we always wanted Java to be: write once, run anywhere, without accepting the lowest common denominator. It’s what you would get if Spring Boot, Flutter, Go, and Node.js had a baby that inherited the best qualities of each. You get Java’s mature ecosystem, Flutter-style control over the UI/portability, Go-like native binaries and footprint, and the full-stack client/server connection people associate with Node.js. With Codename One, one Java codebase can target iOS, Android, macOS, Windows on x64 and ARM64, Linux on x64 and ARM64 with either glibc or musl, and even the browser. A small self-contained desktop GUI application can be around 4–5 MB, with no JVM to install. Underneath, Codename One uses different pipelines for different targets. On our ahead-of-time compiled platforms, ParparVM analyzes the reachable Java bytecode, removes unused code, translates what remains to C, and hands the result to the platform’s native compiler. Codename One also provides the portability libraries, UI toolkit, tests, theming, and build system needed to handle the ugly platform-specific details. We don’t pretend every platform is identical. Our builder architecture assembles the native pieces each application needs, so developers can use platform-specific capabilities without fragmenting the entire codebase. The original inspiration was frustration with Java’s unfulfilled promise. Java gave us portability, but conventional Java application deployment became increasingly heavy. We wanted the language and ecosystem without requiring a large runtime or giving up native performance, platform access, or control of the UI. Now we’re taking that architecture to the backend. The goal is to bring the startup time, footprint, and native compilation work we developed for constrained devices into serverless and microservice workloads. I want Java developers to get the full-stack productivity they expect from Spring Boot, with deployment economics closer to Go and Node.js. 2. Codename One has been around for more than a decade. What has kept you motivated? The community is the biggest reason. People have built businesses and products on Codename One. When someone has trusted your work with something that important, walking away isn’t an option. For a long time, competing in this market was brutal. Our competitors were not normal startups. They were Google, Facebook, Microsoft, and companies with teams larger than our entire community. There was a limit to how much a small team could build, test, document, and support. Agentic development changed that equation. It lets us scale engineering work in a way that simply wasn’t possible a few years ago. We now maintain an enormous cross-platform test matrix and can move at a pace that is competitive with much larger organizations. On most feature dimensions, I would argue that we are already ahead of the big-name alternatives. We still have gaps. We’re a small team, and pretending otherwise would be ridiculous. But our size is also an advantage. The person reading a bug report can be the same person who changes the compiler. We can respond to developers in hours where a large company may need three teams, a quarterly roadmap, and permission from a product manager. The other reason is simpler: we love Java, and we use the competing tools. Every time we do, we run into one of the compromises Codename One was created to remove. 3. What’s one of the biggest misconceptions developers have about maintaining an open source project? That maintaining open source is mostly about accepting code contributions. A pull request looks like free work from the outside. For a maintainer, it can be the beginning of a large amount of work. Is the code correct? Does it meet the project’s standards? Does it work on every target? Who maintains it five years later? How do you tell a volunteer who spent a week on something that the design is wrong? How many rounds of changes can you reasonably ask them to make? We recently stopped accepting outside pull requests. AI wasn’t the reason. If anything, AI improved the baseline quality of many contributions. It also made some of them enormous, because generating another thousand lines became almost free while reviewing those thousand lines did not. The real problem was validation. Codename One currently has more than 60 GitHub Actions workflow files. A small framework change can affect Android, iOS, JavaScript, desktop, bytecode translation, native compilation, and screenshot baselines. That matrix represents roughly $15,000 worth of monthly CI capacity, which GitHub currently provides to us. At that point, validating and repairing an outside patch often took longer than implementing the confirmed change ourselves. So today we ask people to contribute through reproducible issues, benchmark counterexamples, design discussions, testing, and documentation feedback. Those contributions are often more valuable than code because they expose problems we cannot reproduce in our own environments. Also sorry for all the GitHub Actions downtime ;-) 4. You’ve watched Java and the broader development ecosystem evolve considerably. What changes have surprised you the most? The biggest surprise is that Java effectively surrendered the idea of being small. GraalVM is a remarkable piece of engineering, so we aren’t literally alone (although it's a very different project that hasn't really taken the same direction and hasn't taken it as far). But mainstream Java spent years treating bigness as evidence of seriousness: larger runtimes, larger frameworks, more dynamic machinery, longer warmups, and more memory. The industry began treating that weight as inevitable. I'm not against comprehensive/large; Codename One itself is HUGE... But not in runtime, just in API and scope. Most successful newer platforms moved in the opposite direction. Go made self-contained native binaries normal. Node.js made application deployment almost trivial. Mobile and serverless made startup time and memory consumption economically important again. Java largely responded by optimizing large systems rather than questioning why every system needed to be large. Modules are a good example. They help organize and encapsulate a large dependency graph. They do not eliminate that graph or turn it into a small, closed-world native program. Codename One puts a real Java GUI application or full server into a roughly 5 MB binary. What surprises me is not that this works. We have spent years proving that it works. What surprises me is how lonely that position still is. Java could be one of the best languages for small native applications and low-cost serverless workloads. Most of the ecosystem simply stopped trying to make that happen and let Go eat our lunch. 5. After spending so much of your career in technology, what do you enjoy doing when you completely step away from a screen? Completely step away from a screen? I’m probably the wrong person to ask. I just got back from Thailand with my kids. We had a wonderful time. I also had my laptop with me and was doing pull requests on the beach. This is what I enjoy doing. I don’t have an unwind button. Vacations often leave me more exhausted than work because sitting still doesn’t recharge me. If I have an idea and I can build it, fix it, or argue about it with another developer, that is how I unwind. Explore more of Shai's content here.
The Core Problem In Part 1, we built a Quarkus MCP tool server. In Part 2, we secured it with agentgateway. In Part 3, we added distributed tracing. The architecture handles single-request tool calls well — but enterprise processes are rarely single-step. Consider what happens when a developer tells Goose: "Migrate the users table to add an email column, then deploy the updated service to production." This triggers a chain of operations: Goose calls migrate-schema to alter the production databaseGoose calls deploy-production to push the new codeBoth operations succeed — without any human ever reviewing the schema change or approving the production deployment Local developer agents like Goose excel at interactive reasoning, but long-running business processes require: Strict state machines to track multi-step workflows through well-defined phasesMax-iteration caps to prevent runaway agent loops that consume resources indefinitelyFormal human-in-the-loop (HITL) approvals before destructive operations like database writes, schema migrations, or financial transactionsStandardized agent-to-agent delegation so the client agent (Goose) and the backend agent (Quarkus) communicate through a shared protocol, not ad-hoc HTTP calls Without these guardrails, autonomous agents operate in a governance vacuum — the exact gap that turns helpful automation into unaudited risk. The Solution: A2A Protocol + Quarkus Flow + AGENTS.md A2A delegates a business task to an agent; Part 6's model routing selects a model backend for inference. Model fallback does not replace this chapter's workflow policy or human approval gate. The fix combines three standards into a governed multi-agent architecture: A2A (Agent2Agent) – the Linux Foundation standard for multi-agent interoperability. Goose discovers the Quarkus backend via an Agent Card and delegates tasks using JSON-RPC over HTTP. The A2A Java SDK (@PublicAgentCard + AgentExecutor) handles protocol compliance automatically.Quarkus Flow – a lightweight state machine implementing the CNCF Serverless Workflow concepts: states, transitions, and action handlers. Each delegated task flows through governance validation, optional HITL approval, and execution.MCP Tool Delegation – auto-approved operations like analyze-logs, health-check, and generate-report delegate to Part 1's MCP server for live tool execution. The quarkus-langchain4j-mcp extension provides a managed McpClient that handles Streamable HTTP transport and MCP sessions automatically.AGENTS.md – a declarative governance file that defines which operations are auto-approved, which require human approval, and which are permanently blocked. The Quarkus server parses this file at startup and enforces it on every incoming A2A task. The A2A task lifecycle drives the entire flow: Plain Text submitted → working → input-required → working → completed → failed (rejected) → completed (auto-approved) → failed (blocked by governance Prerequisites Everything from Parts 1–3, plus the A2A Java SDK. Part 4 adds the a2a-java-sdk-reference-jsonrpc dependency for A2A protocol support and connects to Part 1's MCP server for tool execution. Verify your environment: Shell java -version # Java 25+ mvn --version # Maven 3.9+ Step 1: Agent Discovery With A2A Agent Cards The A2A protocol starts with discovery. Every A2A-compliant agent publishes an Agent Card at a well-known URL that describes its capabilities, skills, and endpoint: Shell curl -s http://localhost:8082/.well-known/agent-card.json | jq . JSON { "name": "enterprise-workflow-agent", "description": "Multi-step enterprise workflow orchestration with HITL approval gates...", "supportedInterfaces": [{ "protocol": "JSONRPC", "url": "http://localhost:8082" }], "version": "1.0.0", "capabilities": { "streaming": false, "pushNotifications": false }, "skills": [ { "id": "analyze-logs", "name": "Log Analysis", "description": "Analyze logs via MCP delegation" }, { "id": "health-check", "name": "Health Check", "description": "Verify service health via MCP" }, { "id": "migrate-schema", "name": "Schema Migration", "description": "Requires HITL approval" }, { "id": "deploy-production", "name": "Production Deployment", "description": "Requires HITL approval" } ] } The Agent Card tells Goose (or any A2A client) three critical things: FieldPurposesupportedInterfacesThe JSON-RPC endpoint URL and transport protocolskillsThe operations the agent can perform — Goose uses these to decide which tasks to delegatecapabilitiesWhether the agent supports streaming, push notifications, etc. With the A2A Java SDK, the Agent Card is declared using the @PublicAgentCard CDI qualifier on a producer method — the SDK automatically serves it at /.well-known/agent-card.json and registers the JSON-RPC transport handler: Java @ApplicationScoped public class AgentCardProducer { @ConfigProperty(name = "agent.url", defaultValue = "http://localhost:8082") String agentUrl; @Produces @PublicAgentCard public AgentCard agentCard() { return AgentCard.builder() .name("enterprise-workflow-agent") .description("Multi-step enterprise workflow orchestration with HITL approval gates...") .supportedInterfaces(Collections.singletonList( new AgentInterface("JSONRPC", agentUrl))) .version("1.0.0") .capabilities(AgentCapabilities.builder() .streaming(false).pushNotifications(false).build()) .defaultInputModes(Collections.singletonList("text")) .defaultOutputModes(Collections.singletonList("text")) .skills(List.of( AgentSkill.builder().id("analyze-logs").name("Log Analysis") .description("Analyze logs via MCP tool delegation (auto-approved)") .tags(List.of("observability", "auto-approved")).build(), AgentSkill.builder().id("migrate-schema").name("Schema Migration") .description("Execute database schema migrations (requires HITL approval)") .tags(List.of("database", "hitl-required")).build() // ... additional skills )) .build(); } } No JAX-RS endpoint needed — the SDK handles agent card serving and JSON-RPC routing automatically. Step 2: Defining Governance Rules With AGENTS.md AGENTS.md is a declarative governance file that lives alongside the application code. The Quarkus server reads it at startup and enforces its rules on every incoming A2A task. Create src/main/resources/AGENTS.md: Markdown # AGENTS.md - Governance Rules for Enterprise Workflow Agent ## Execution Bounds max-iterations: 10 ## Operations ### Auto-Approved - analyze-logs: Read-only log analysis across services - health-check: Service health and resource utilization verification - generate-report: Compliance and metrics report generation from existing data ### Requires Human Approval - migrate-schema: Database schema migrations on production tables - process-refund: Customer refund processing (auto-approved below $1000 threshold) - scale-infrastructure: Infrastructure replica scaling operations - deploy-production: Production environment deployments ### Blocked - drop-database: Destructive operation — never allowed via agent delegation - truncate-table: Destructive operation — never allowed via agent delegation - delete-all-data: Destructive operation — never allowed via agent delegation Three categories, three outcomes: CategoryWhat HappensExampleAuto-ApprovedTask executes immediately, no human involvedanalyze-logs, health-checkRequires Human ApprovalTask pauses at input-required state until a human approves or rejectsmigrate-schema, deploy-productionBlockedTask fails immediately with a governance violationdrop-database, truncate-table The GovernanceEngine parses this file at startup and validates every operation: Java public GovernanceResult validate(String operation, Map<String, String> args) { if (blocked.containsKey(operation)) { return new GovernanceResult(Decision.BLOCKED, "Operation '" + operation + "' is blocked by governance policy", "critical"); } if (requiresApproval.containsKey(operation)) { // Conditional: process-refund auto-approves below $1000 if (operation.equals("process-refund")) { int amount = Integer.parseInt(args.getOrDefault("amount", "0")); if (amount <= 1000) { return new GovernanceResult(Decision.AUTO_APPROVED, "...", "low"); } } return new GovernanceResult(Decision.REQUIRES_APPROVAL, reason, "high"); } if (autoApproved.containsKey(operation)) { return new GovernanceResult(Decision.AUTO_APPROVED, "...", "low"); } // Unknown operations default to requiring approval return new GovernanceResult(Decision.REQUIRES_APPROVAL, "...", "medium"); } Notice the conditional logic for process-refund — governance rules can include thresholds, not just binary allow/deny. A $500 refund flows through automatically; a $2,500 refund pauses for human review. Step 3: Implementing the Workflow State Machine Each A2A task flows through a state machine with six possible states: The WorkflowEngine manages this lifecycle: Java public Map<String, Object> submitTask(String taskId, String messageText) { TaskInstance task = tasks.computeIfAbsent(taskId, TaskInstance::new); task.incrementIteration(); // Safety bound: prevent runaway agent loops if (task.getIteration() > governance.getMaxIterations()) { task.setState(TaskState.FAILED); task.addMessage("agent", "Max iterations exceeded. Task terminated."); return task.toA2AResponse(); } String operation = parseOperation(messageText); Map<String, String> args = parseArguments(messageText); task.setState(TaskState.WORKING); // Governance gate GovernanceResult gov = governance.validate(operation, args); return switch (gov.decision()) { case BLOCKED -> { task.setState(TaskState.FAILED); task.addMessage("agent", gov.reason()); yield task.toA2AResponse(); } case REQUIRES_APPROVAL -> { task.setState(TaskState.INPUT_REQUIRED); task.addMessage("agent", "HITL approval required: " + gov.reason()); yield task.toA2AResponse(); } case AUTO_APPROVED -> { String result = execute(operation, args); task.setState(TaskState.COMPLETED); task.addMessage("agent", result); yield task.toA2AResponse(); } }; } The A2A Java SDK handles JSON-RPC routing automatically. You implement the AgentExecutor interface — the SDK calls your execute() method for each incoming SendMessage request: Java @ApplicationScoped public class AgentExecutorProducer { @Inject WorkflowEngine engine; @Produces public AgentExecutor agentExecutor() { return new AgentExecutor() { @Override public void execute(RequestContext context, AgentEmitter emitter) throws A2AError { String messageText = extractText(context.getMessage()); String taskId = context.getTaskId(); // HITL follow-up: task already exists in INPUT_REQUIRED state if (context.getTask() != null && context.getTask().status().state() == TaskState.TASK_STATE_INPUT_REQUIRED) { handleHitlFollowUp(taskId, messageText, emitter); return; } // New task: submit to workflow engine for governance check TaskInstance task = engine.submitTask(taskId, messageText); switch (task.getState()) { case FAILED -> emitter.fail(agentMessage(emitter, task.getLastAgentMessage())); case INPUT_REQUIRED -> { emitter.startWork(); emitter.requiresInput(agentMessage(emitter, task.getLastAgentMessage())); } case COMPLETED -> { emitter.startWork(); emitter.addArtifact(List.of(new TextPart(task.getLastAgentMessage()))); emitter.complete(); } } } // ... }; } } The AgentEmitter manages the A2A task lifecycle — startWork(), requiresInput(), complete(), fail() — while the WorkflowEngine handles governance validation and MCP tool execution. Step 4: Human-in-the-Loop Approval Gates When Goose submits a high-risk operation, the workflow pauses at input-required. The task stays frozen until a human explicitly approves or rejects it. Triggering HITL Submit a schema migration — this is a high-risk operation that requires approval: Shell curl -s http://localhost:8082/ \ -H "Content-Type: application/json" \ -H "A2A-Version: 1.0" \ -d '{"jsonrpc":"2.0","id":1,"method":"SendMessage","params":{ "message":{"messageId":"migration-msg-1","role":"ROLE_USER","parts":[{"text":"migrate-schema --database production --table users --changes add-column-email"}]}, "configuration":{"returnImmediately":true} }' | jq .result.task.status JSON { "state": "TASK_STATE_INPUT_REQUIRED", "message": { "role": "ROLE_AGENT", "parts": [{ "text": "HITL approval required. Reason: Database schema migrations on production tables. Workflow paused at state INPUT_REQUIRED. Awaiting human decision." }] } } Polling for Status Goose polls GetTask and sees the task is paused. It can alert the developer: Shell curl -s http://localhost:8082/ \ -H "Content-Type: application/json" \ -H "A2A-Version: 1.0" \ -d '{"jsonrpc":"2.0","id":2,"method Shell ":"GetTask","params":{"id":"<task-id-from-above>"}' \ | jq .result.status.state Plain Text "TASK_STATE_INPUT_REQUIRED" Approving the Task A human reviews the operation and approves it through the admin API (or the SPA): Shell curl -s -X POST http://localhost:8082/api/tasks/migration-1/approve | jq .status JSON { "state": "COMPLETED", "message": { "role": "agent", "parts": [{ "text": "Schema migration completed successfully. Database: production. Table: users. Changes: add-column-email..." }] } } Rejecting a Task If the human decides the migration is unsafe: Shell curl -s -X POST http://localhost:8082/api/tasks/migration-2/reject \ -H "Content-Type: application/json" \ -d '{"reason":"Schema change not reviewed by DBA team"}' | jq .status.state Plain Text "failed" The rejection reason is recorded in the task history — Goose receives it on the next GetTask poll and can relay it to the developer. Contrast: Auto-Approved and Blocked Operations Not every operation triggers HITL. Compare three scenarios: Shell # Auto-approved: executes immediately curl -s http://localhost:8082/ \ -H "Content-Type: application/json" -H "A2A-Version: 1.0" \ -d '{"jsonrpc":"2.0","id":3,"method":"SendMessage","params":{ "message":{"messageId":"logs-msg-1","role":"ROLE_USER","parts":[{"text":"analyze-logs --service api-gateway --timeframe 24h"}]} }' | jq .result.task.status.state # → "TASK_STATE_COMPLETED" # Blocked: fails immediately curl -s http://localhost:8082/ \ -H "Content-Type: application/json" -H "A2A-Version: 1.0" \ -d '{"jsonrpc":"2.0","id":4,"method":"SendMessage","params":{ "message":{"messageId":"drop-msg-1","role":"ROLE_USER","parts":[{"text":"drop-database --target production"}]}, "configuration":{"returnImmediately":true} }' | jq .result.task.status.state # → "TASK_STATE_FAILED" Step 5: Running the Complete Demo Start all services with the one-command script: Shell cd part4-multi-agent ./start-all.sh The script builds Part 1's MCP server and Part 4's A2A Flow server, starts both (MCP on :8080, A2A on :8082), and launches the interactive demo SPA. Auto-approved tasks like analyze-logs delegate to Part 1's MCP tools for live data. Open the A2A Multi-Agent Console at http://localhost:8889/index.html. If discovery reports Failed to fetch, stop the launcher with Ctrl+C, update the code, run ./start-all.sh again, and reload the page. The console needs the backend's quarkus.http.cors.enabled=true setting to read responses across ports. This single-user lab binds the A2A server to localhost and explicitly sets a2a.authorization.required=false. A2A SDK 1.3+ requires a TaskAuthorizationProvider by default; shared deployments need authenticated users and that provider. The lab's AGENTS.md operation rules and HITL gates remain active. Demo Walkthrough The SPA provides four guided steps that showcase the full governance spectrum: Step 1: Discover Agent Click Discover Agent to fetch the Agent Card from /.well-known/agent-card.json. The architecture diagram animates the discovery flow and loads the AGENTS.md governance rules. The stat tiles show the agent is online and governance rules are loaded. Step 2: Auto-Approved Task (analyze-logs) Click Auto-Approved Task to submit a read-only log analysis operation. The task flows through AGENTS.md validation (auto-approved), skips the HITL gate, and delegates to Part 1's MCP tools via McpToolClient. The state machine transitions: SUBMITTED → WORKING → GOVERNANCE → EXECUTING → COMPLETED. Step 3: HITL Task (migrate-schema) Click HITL Task to submit a high-risk database migration. The task pauses at INPUT_REQUIRED after governance flags it for human review. An approval panel appears with Approve and Reject buttons, using the task ID returned by the server. Click Approve to run the simulated schema migration, or Reject to terminate the task. State machine: SUBMITTED → WORKING → GOVERNANCE → HITL GATE → (approve) → EXECUTING → COMPLETED. Step 4: Blocked Task (drop-database) Click Blocked Task to submit a destructive operation. AGENTS.md governance rejects it immediately — the task never reaches the HITL gate or MCP delegation. State machine: SUBMITTED → WORKING → GOVERNANCE → FAILED. Connecting Goose When Goose adds A2A support, configure it to discover the backend agent: Shell goose session --a2a-discover http://localhost:8082 Goose will fetch the Agent Card, enumerate available skills, and delegate operations via SendMessage. High-risk operations will pause until a human approves them through the SPA or CLI. What We Achieved Starting from the observable architecture in Part 3, we added multi-agent orchestration with governance and HITL without changing any existing MCP tool code: LayerWhat We AddedKey FileA2A ProtocolAgent Card + JSON-RPC via A2A Java SDKAgentCardProducer.java, AgentExecutorProducer.javaGovernanceAGENTS.md rule parsing and enforcementGovernanceEngine.java, AGENTS.mdWorkflowState machine with HITL approval gatesWorkflowEngine.javaMCP IntegrationTool delegation via quarkus-langchain4j-mcpMcpToolClient.java, application.propertiesAdminREST API + SPA for human approvalsAdminEndpoint.java, index.html Production Considerations ConcernLocal (this tutorial)ProductionTask storageIn-memory ConcurrentHashMapPostgreSQL + Flyway migrationsHITL notificationsBrowser polling from SPAWebSocket push or Slack/Teams integrationGovernance rulesAGENTS.md file on classpathPolicy engine (OPA, Cedar, or Keycloak policies)A2A transportPlain HTTP JSON-RPCmTLS + OAuth2 client credentialsWorkflow engineCustom state machineSonataFlow (CNCF Serverless Workflow on Quarkus)Audit trailIn-memory task historyAppend-only event store with Kafka Coming Up in Part 5 With governance, security, tracing, and multi-agent orchestration in place, the stack is production-capable — but how do you know it stays correct across code changes? In Part 5, we will add automated agent evaluation and regression testing — using golden datasets to verify that MCP tool outputs, Bean Validation boundaries, and multi-step agent workflows produce correct, consistent results as the codebase evolves.
CORE
A language model holds nothing between calls. Each request arrives with no memory of the previous one, and the model responds from the prompt in front of it and nothing more. For a one-shot question, that is fine. For an agent a person returns to across days, threads, and separate tasks, it becomes the defining design constraint. The behavior users expect from an agent, recalling what they said last week, resuming a task that stalled, applying their stated preferences without being asked again, is precisely the behavior a stateless model cannot produce on its own. A large language model (LLM) does not forget your context because it chose to. It never had it. The reflex fix is to paste the history back in. Keep a buffer of prior turns and prepend it to every prompt, so the agent looks like it remembers. That illusion survives a short demo and breaks in production. Building stateful AI agents that hold continuity across sessions takes a memory architecture, not a bigger buffer. Why Stateless Models Break Multi-Session Agents The conversation-buffer approach fails in three measurable ways, and they compound. The first is cost. Every prior turn you replay is billed again on the next call, so a buffer that grows linearly turns a fifty-turn session into fifty replays of the same opening tokens. The second is latency. Larger prompts take longer to transmit and to process, and that overhead lands on every turn, not only the long ones. The third is the one that surprises teams: recall does not hold up as the prompt grows. Research on long-context behavior, notably Liu and colleagues in 2023, found that models attend most reliably to information at the start and end of a prompt and least reliably to whatever sits in the middle. Bury a critical instruction in turn nine of a forty-turn transcript and the model may act as though it is not there. There is a harder limit underneath all of this. Every model has a fixed context window, commonly between 8,000 and 128,000 tokens depending on the model. Cross it, and the provider either rejects the request with a context-length error or silently truncates the oldest turns, which means the agent loses exactly the early context the buffer was meant to preserve. None of this touches the cross-session problem. A buffer lives inside one session. Open a new thread tomorrow, and the buffer is empty again. The preferences and facts the user established yesterday are gone, because nothing extracted them into a place that outlives the session. So the standard approach does not scale, and the reason is that it conflates two different jobs: holding the immediate conversation and remembering across conversations. Those are separate problems with separate lifetimes, and they need separate machinery. Measuring Context Pressure in a Live Agent Before designing anything, instrument what you already have. The failure modes above stay invisible until you measure token consumption per turn against the model's window, and most teams find the ceiling only when a request errors out in front of a customer. A small instrument that tracks tokens as the session grows turns a production incident into a logged warning. Python # context_budget.py # Instruments an agent's session log so you can see context pressure build # up turn by turn, instead of discovering it when a request silently truncates # or the provider returns a context-length error mid-conversation. from dataclasses import dataclass, field # In production, count with the model's real tokenizer (for example tiktoken's # cl100k_base or the o200k_base encoding). We wrap it so a missing encoding or # offline build degrades to a conservative heuristic instead of crashing the # request path, which is what you actually want in a hot loop. def count_tokens(text: str) -> int: try: import tiktoken return len(tiktoken.get_encoding("o200k_base").encode(text)) except Exception: # ~4 characters per token is a safe upper-bound heuristic for English. return (len(text) + 3) // 4 @dataclass class SessionLog: budget: int # hard ceiling for prompt tokens events: list = field(default_factory=list) def add(self, role: str, text: str) -> None: self.events.append((role, text, count_tokens(text))) def used(self) -> int: return sum(t for _, _, t in self.events) def pressure(self) -> float: # Fraction of the budget consumed. Alert teams above ~0.75 so there is # headroom for the model's own output tokens, not just the prompt. return self.used() / self.budget log = SessionLog(budget=2000) for turn in range(1, 13): log.add("user", "Walk me through the failure we saw in the nightly run. " * turn) log.add("assistant", "Here is the breakdown of the root cause and the fix. " * turn) if log.pressure() > 0.75: print(f"turn {turn:>2}: used={log.used():>5} tokens " f"pressure={log.pressure():.0%} ACTION: evict or summarize") else: print(f"turn {turn:>2}: used={log.used():>5} tokens pressure={log.pressure():.0%}") The threshold matters. Alerting at seventy-five percent of the window rather than at one hundred leaves headroom for the model's own output tokens, which are drawn from the same budget. Count tokens with the model's actual tokenizer, since the tiktoken library exposes the o200k_base and cl100k_base encodings used by current models, because a whitespace estimate can be off by a wide margin on code, punctuation, or non-English text. The heuristic fallback in the wrapper exists so the request path degrades to an estimate rather than throwing when an encoding is unavailable. Once you can watch pressure climb, the case for a second memory layer stops being theoretical. A Two-Layer Memory Architecture for Stateful AI Agents The architecture that holds up splits memory by lifetime. Short-term memory is the raw event log of the current session: every user turn and agent response, appended cheaply, scoped to one interaction, archived or discarded when the session ends. Long-term memory is the distilled residue, the facts, preferences, and summaries worth keeping, extracted from raw events and persisted across every future session. Short-term memory answers what just happened in this conversation. Long-term memory answers what the system knows about this user. Keep both in one store and you will either bloat the prompt with raw transcript or lose durable facts in the churn. Long-term memory is shared infrastructure, often across many tenants, so the first structural decision is addressing. Records are keyed by a hierarchical namespace rather than a flat identifier, and the hierarchy itself enforces isolation. A retrieval scoped to one tenant cannot resolve a key that belongs to another, because the tenant identifier is a structural prefix of every key rather than a filter applied after the fetch. Python # namespaces.py # Memory records are addressed by a hierarchical namespace, not a flat key. # The hierarchy is what enforces multi-tenant isolation: a retrieval scoped to # one tenant can never resolve a key that belongs to another, because the # tenant id is a structural prefix of every key rather than a filter applied # after the fetch. import re # Components are validated so a caller cannot smuggle a separator into an id and # escape its own subtree. This is the single most common isolation bug in # home-grown memory stores: an unescaped "/" in a user id reads as a new level. _SAFE = re.compile(r"^[A-Za-z0-9_-]+$") def make_namespace(tenant: str, actor: str, scope: str) -> str: for part in (tenant, actor, scope): if not _SAFE.match(part): raise ValueError(f"unsafe namespace component: {part!r}") return f"/{tenant}/{actor}/{scope}" class MemoryStore: def __init__(self): self._records: dict[str, list[str]] = {} def put(self, namespace: str, value: str) -> None: self._records.setdefault(namespace, []).append(value) def query(self, namespace_prefix: str) -> list[str]: # Only keys under the caller's prefix are visible. The prefix check is # the isolation boundary, so it runs before any ranking or scoring. hits = [] for key, values in self._records.items(): if key == namespace_prefix or key.startswith(namespace_prefix + "/"): hits.extend(values) return hits store = MemoryStore() store.put(make_namespace("tenant-a", "user-42", "preferences"), "prefers metric units") store.put(make_namespace("tenant-a", "user-42", "facts"), "deploys to two regions") store.put(make_namespace("tenant-b", "user-42", "preferences"), "prefers imperial units") print("tenant-a/user-42 sees:", store.query("/tenant-a/user-42")) print("tenant-b/user-42 sees:", store.query("/tenant-b/user-42")) # A caller trying to break out of its subtree by embedding a separator is # rejected at construction time, before any record is written or read. try: make_namespace("tenant-a", "user-42/../tenant-b", "preferences") except ValueError as exc: print("isolation guard:", exc) The validation is not decoration. The most common isolation defect in a home-grown memory store is an unescaped separator in a user-supplied identifier, where a value shaped like a path fragment reads as a new namespace level and lets a caller climb out of its own subtree. Rejecting unsafe components at construction time closes that hole before a single record is written or read. Raw events are cheap to append and nearly useless to search. Turning them into retrievable knowledge is the job of an extraction pipeline, and that pipeline belongs off the request path. Distillation calls a model and costs hundreds of milliseconds, so running it inline would tax every turn with work the user never sees. An asynchronous worker reads batches of raw events and writes back typed long-term records. Python # extraction_worker.py # Raw session events are cheap to append and useless to search. The extraction # worker runs off the request path, reads batches of raw events and distills # them into durable long-term records: facts, preferences and rolling # summaries. Keeping this asynchronous is the point. Distillation calls a model # and takes hundreds of milliseconds, so doing it inline would tax every turn. import queue import threading import time raw_events: "queue.Queue[dict]" = queue.Queue() long_term: list[dict] = [] def distill(batch: list[dict]) -> list[dict]: # PLACEHOLDER for the model call. In production this is a structured-output # request to an extraction model that returns typed records. The rule-based # stub below keeps the worker deterministic and runnable for tests; swap it # for the real call and keep the same return contract. records = [] for ev in batch: text = ev["text"].lower() if "prefer" in text: records.append({"type": "preference", "value": ev["text"], "src": ev["id"]}) elif any(k in text for k in ("region", "deploy", "cluster", "schema")): records.append({"type": "fact", "value": ev["text"], "src": ev["id"]}) return records def worker(batch_size: int = 4, idle_flush: float = 0.2) -> None: batch: list[dict] = [] last = time.monotonic() while True: try: ev = raw_events.get(timeout=idle_flush) if ev is None: # poison pill: drain then stop if batch: long_term.extend(distill(batch)) break batch.append(ev) except queue.Empty: pass # Flush on a full batch or after an idle gap, so a quiet session still # gets its memory persisted instead of waiting for more traffic. if len(batch) >= batch_size or (batch and time.monotonic() - last > idle_flush): long_term.extend(distill(batch)) batch, last = [], time.monotonic() t = threading.Thread(target=worker, daemon=True) t.start() for i, msg in enumerate([ "I prefer metric units in reports", "we deploy to two regions", "the order schema changed last week", "thanks, that helps", "I prefer dark themed dashboards", ]): raw_events.put({"id": f"e{i}", "text": msg}) raw_events.put(None) t.join(timeout=2) for rec in long_term: print(rec) Two details in that worker earn their place. The batch flushes on a size trigger or an idle timeout, so a quiet session still gets its memory persisted instead of waiting for traffic that never arrives. The drain-on-shutdown path matters even more: an earlier version that broke on the poison pill without flushing the partial batch silently dropped the last few events whenever the worker stopped, the kind of lost-write bug that surfaces as missing memories weeks later. The model call is stubbed here with deterministic rules so the worker stays testable. The contract is the point. Swap the stub for a structured-output extraction call, and the rest holds. With both layers in place, a single agent turn reads from each, assembles a prompt within the token budget, and persists the new event for later extraction. Python # agent_turn.py # One agent turn that reads from both memory layers and respects a token # budget. Long-term records come first because they are dense and stable; # recent raw turns fill whatever budget remains. This ordering matters: models # attend less reliably to the middle of a long prompt, so the highest-value # memory belongs near the edges, not buried in the transcript. def count_tokens(text: str) -> int: return (len(text) + 3) // 4 # heuristic; production swaps in the real tokenizer def assemble_context(long_term: list[str], recent: list[str], budget: int) -> list[str]: context, used = [], 0 # Long-term memory is admitted first, up to half the budget, so a long # transcript can never crowd out durable facts the agent depends on. for rec in long_term: cost = count_tokens(rec) if used + cost > budget // 2: break context.append(rec) used += cost # Recent turns fill the remainder, newest first, dropping the oldest. for turn in reversed(recent): cost = count_tokens(turn) if used + cost > budget: break context.append(turn) used += cost return context def run_turn(user_msg, long_term, recent, budget=400): # Reserve room for the user message before admitting any memory, otherwise # the assembled prompt overshoots the budget once the message is appended. reserve = count_tokens(f"user: {user_msg}") + 1 context = assemble_context(long_term, recent, budget - reserve) prompt = "\n".join(context + [f"user: {user_msg}"]) # PLACEHOLDER for the model call. The returned context window is what the # agent actually reasons over; assert it before spending tokens on inference. assert count_tokens(prompt) <= budget, "context exceeds budget" recent.append(f"user: {user_msg}") # short-term log keeps growing return prompt long_term = ["fact: deploys to two regions", "preference: prefers metric units"] recent = [f"user: earlier message {i}" for i in range(20)] prompt = run_turn("why did the canary roll back?", long_term, recent, budget=120) print(prompt) print("---") print("prompt tokens:", count_tokens(prompt), "of budget 120") Ordering is a deliberate choice. Long-term records go in first and are capped at half the budget so a long transcript can never crowd out the durable facts the agent depends on; then recent turns fill the remainder, newest first. Because models attend least reliably to the middle of a prompt, the highest-value memory belongs near the edges. Reserving room for the user message before admitting any memory keeps the assembled prompt from overshooting once the message is appended, a bug the budget assertion catches the moment it slips. Trade-Offs in Extraction Timing, Retrieval and Consistency Every decision in this design buys one property at the cost of another, and pretending otherwise is how teams get surprised later. Asynchronous extraction keeps turn latency flat, but it introduces an eventual-consistency window: for the few seconds between an event and its distillation, the long-term layer does not yet know what the user just said. For most assistants, that lag is invisible. For a workflow where the user states a fact and immediately expects the agent to act on it across a session boundary, you either shorten the window or read through to the short-term log for the current session. Synchronous extraction removes the lag and adds those hundreds of milliseconds to every turn instead. Pick the trade deliberately. Retrieval carries its own tension. Score candidates purely on semantic similarity and a stale record can win, because a preference the user set months ago and has since reversed still matches the query embedding closely. Blend a recency decay into the score and freshness gets a vote. Python # ranking.py # Retrieval that scores only on semantic similarity will happily surface a # year-old preference the user has since reversed. Blending a recency decay # into the score fixes that, but the decay rate is a real tuning knob: too # aggressive and the agent forgets stable facts, too gentle and stale memory # wins. This compares pure similarity against a recency-weighted score on the # same candidate set so the trade-off is visible. import numpy as np # query vector and candidate vectors stand in for embeddings; in production # these come from an embedding model and a vector index, but the ranking math # is identical regardless of where the vectors originate. query = np.array([0.9, 0.1, 0.0]) candidates = [ {"text": "prefers metric units", "vec": np.array([0.88, 0.12, 0.0]), "age_days": 210}, {"text": "prefers imperial units", "vec": np.array([0.86, 0.14, 0.0]), "age_days": 3}, {"text": "works on payments", "vec": np.array([0.20, 0.80, 0.1]), "age_days": 12}, ] def cosine(a, b): return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b))) def rank(decay_per_day: float): scored = [] for c in candidates: sim = cosine(query, c["vec"]) # Exponential recency decay. A fresher record with slightly lower # similarity can now outrank a near-duplicate that is months old. recency = np.exp(-decay_per_day * c["age_days"]) blended = sim if decay_per_day == 0 else 0.6 * sim + 0.4 * recency scored.append((round(float(blended), 3), c["text"])) return sorted(scored, reverse=True) print("pure semantic: ", rank(decay_per_day=0.0)[0]) print("recency-weighted 0.01:", rank(decay_per_day=0.01)[0]) print("recency-weighted 0.05:", rank(decay_per_day=0.05)[0]) The decay rate is a real knob, not a default to ignore. Too aggressive and the agent forgets stable facts that should persist for months; too gentle and outdated memory keeps surfacing. Notice in the output how a record that is three days old overtakes a months-old record once recency is weighted at all, even though the older record scores marginally higher on similarity alone. The right constant depends on how fast your domain's facts actually change, which is something only the people running the system can calibrate against real traffic. There is also a quieter cost to the whole approach. Extraction and embedding are not free, since every event you choose to distill and index consumes compute and storage, so indexing everything is rarely the right call. Selective extraction, keeping facts, preferences, and periodic summaries rather than raw chatter, keeps the long-term store small enough to search quickly and cheap enough to justify. Teams that skip the two-layer split do not avoid this cost. They pay it as a degraded user experience: repetitive agents that re-ask for the same information, prompts that balloon until they error, and memory that evaporates the moment a session closes. Separating short-term context from long-term persistence is what turns a stateless model into an agent that behaves as though it remembers, and it stays an ordinary software engineering problem once you stop expecting the model to solve it for you.
Edge to cloud is not one integration problem. It is four: telemetry going up, control going down, sites syncing sideways, and data reaching people. Each has different physics. Most failed projects picked one technology for all four. Every enterprise with factories, stores, vehicles, hospitals, or devices faces the same architectural question. How does data move between the edge and the data center or cloud, in both directions, reliably, securely, and at acceptable cost? The vendor landscape answers with dozens of overlapping options: MQTT platforms, hyperscaler services, event streaming, event mesh, messaging fabrics, delivery platforms, and protocol proxies. This article structures the option space by traffic pattern instead of by product category, maps the vendor categories onto those patterns, and closes with a decision framework you can apply to your own architecture. The categories here sit alongside the wider integration landscape rather than inside it. For that broader map across event streaming, API, and batch, see the Data Integration Landscape 2026. What Does Edge-to-Cloud Communication Mean? Edge-to-cloud communication is the movement of data between distributed physical locations and centralized computing environments, in both directions, across networks the architect does not control. The word edge hides four very different tiers: The device edge means microcontrollers, sensors, and actuators with kilobytes of memory.The gateway edge means industrial PCs, vehicle computers, and protocol converters.The site edge means a factory, store, hospital, or ship with local compute and its own network boundary.The near edge means regional data centers close to many sites. A technology that fits one tier often fails at another. Edge to cloud splits into four traffic patterns: northbound telemetry, southbound control, site-to-site sync, and outbound delivery to people. The traffic crossing these tiers falls into four patterns: Northbound telemetry moves sensor readings, machine states, transactions, and logs from the edge toward central platforms for analytics, AI, and business systems.Southbound control moves commands, configuration, software updates, and increasingly AI models from central platforms down to machines and gateways.Site-to-site traffic synchronizes data between sites, between a site and a core data center, or between clouds.Outbound delivery pushes live data from central platforms to the applications people use: dashboards, mobile apps, and trading screens. Direction plus consumer type plus constraint determines the right technology. Vendor preference does not. Why Do Network Constraints Decide the Edge-to-Cloud Architecture? The network between edge and cloud is the one component you cannot refactor. Its properties dictate the architecture before any product evaluation starts. Connectivity is unreliable. Ships, vehicles, and remote sites go offline for hours or days. Any viable architecture is offline first: local operation continues, data buffers locally, and synchronization resumes when the link returns. This is why store and forward is the single most important capability on any edge evaluation checklist. Inbound connections are forbidden. Industrial sites, retail networks, and vehicle fleets sit behind NAT, firewalls, and cellular carriers. Security teams do not open inbound ports for messaging. Every serious edge technology therefore uses the same trick: the edge dials out. MQTT clients dial out to brokers. NATS leaf nodes dial out to hub clusters. Azure IoT Operations dials out from the factory to the cloud broker. Different stacks, identical answer, because the constraint is identical. Hardware is constrained and heterogeneous. An 8-bit microcontroller, a fanless gateway PC, and a site server need different footprints. Some technologies scale down to bare microcontrollers. Others assume a Kubernetes cluster. Matching footprint to tier is half the selection process. Footprint is not the only axis. In fixed installations like plants and warehouses, the device edge can stay thin, because the uplink is reliable and most intelligence lives upstream. In the field, agriculture, cranes, rural grid, remote or contested sites, the uplink is wireless and often degraded, saturated, expensive, or absent for days. There, the edge cannot just buffer and forward. It has to run enough local intelligence, pattern matching, image recognition, autonomous decisions, to operate correctly on its own and sync opportunistically when the link returns. That pushes compute onto the device tier for reasons that have nothing to do with protocol size, and it changes the pattern, not just the product. Data residency is now an architecture requirement, not a legal afterthought. European manufacturers increasingly must prove what data leaves the site, country, or jurisdiction. Topic design, routing rules, and retention policies become compliance controls. And one constraint is usually invented rather than measured: latency. Teams assert real-time requirements that the business case does not contain. A large share of edge traffic is perfectly fine as periodic batch over a metered link. Keep that thought. It returns in the site-to-site section. Northbound: How Does Telemetry Get from OT and Devices into the Core? Northbound telemetry is the pattern the industry knows best, and the one with the most settled consensus. At the machine level, OPC UA provides the semantic information model. It defines what a temperature, a batch, or a machine state means. MQTT provides the efficient transport, and Sparkplug adds topic structure, birth and death certificates, and state awareness on top of MQTT. The two standards stopped being rivals. OPC UA Pub/Sub over MQTT is the convergence point: OPC UA semantics carried over MQTT infrastructure. Edge gateways and industrial DataOps products, for example HighByte or Ignition with its MQTT modules, convert brownfield protocols like Modbus and S7 into that stack. The Unified Namespace (UNS) pattern then organizes all site data as a live, hierarchical topic structure on a central broker. I covered UNS design and its trade-offs in depth in Unified Namespace vs. Data Product in IT/OT for Industrial IoT and the broader MQTT market in MQTT Market Trends: Cloud, Unified Namespace, Sparkplug, Kafka Integration, so this section stays short. The real decision is the broker and its operating model. Self-managed platforms carry the northbound load for most industrial estates: HiveMQ and EMQX at the enterprise end, Mosquitto and NanoMQ where a lightweight or embedded footprint matters. They give you deployment control, protocol depth, and edge variants that run on gateways. Managed cloud services remove the operations burden and integrate natively with cloud analytics. The dividing line is usually sovereignty and site autonomy: if the broker must run inside the factory, on the ship, or in a specific jurisdiction, self-managed wins. If all sites have reliable connectivity and the data may leave, managed wins on cost of operations. The hyperscalers themselves tell an asymmetric story. Google retired Cloud IoT Core in 2023 and exited the category. AWS IoT Core remains the steady incumbent for managed device connectivity. Azure went furthest architecturally: Azure IoT Operations runs a distributed MQTT broker on Arc-enabled Kubernetes at the site edge and bridges it to the Event Grid MQTT broker in the cloud, which now supports Sparkplug natively. That is a symmetric edge plus cloud MQTT pair from a single vendor, and it is a new pattern. One hyperscaler retreated, one held, one doubled down. The lesson for architects: managed convenience is real, but so is the risk that a cloud provider deprecates your ingestion layer. The open standards underneath, MQTT and OPC UA, are the insurance policy. Choose products that speak them, and migration remains possible. Southbound and Outbound: How Do Data and Commands Flow Back to Machines and People? Almost all edge content covers ingestion. The return direction is harder, and it is where architectures break, because the return direction has two completely different consumers. Machines need commands and state. Request-response over pub/sub topics handles commands: reboot this gateway, close this valve, change this setpoint. Retained messages and device twins handle state: a device that reconnects must immediately learn its desired configuration without waiting for the next command. Fleet-wide rollout needs staged delivery, and the payloads keep growing. Configuration files became firmware images, and firmware images are becoming AI models. Shipping a quantized model to ten thousand gateways is a southbound data movement problem, not an MLOps footnote. The same fabric increasingly carries traffic for agents at the edge: AI agents that perceive local data, act locally, and coordinate through the messaging layer. NATS and the MQTT platforms are both positioning for exactly this workload. People need the current picture. A plant manager opening a dashboard, an operator reconnecting after a network drop, and a trader loading a screen mid-session all have the same requirement: current state on connect, then live updates. A log is not a user interface. Machines just consume the stream. People do not. This is the last-mile delivery problem, and it has its own vendor category: Lightstreamer, Ably, and PubNub sit between the central platform and many concurrent browsers and mobile apps, handling fan-out, per-client throttling, and entitlement checks. The category is maturing fast. For instance, Lightstreamer's Kafka Connector 2.0, released in July 2026, moved snapshot management into the connector itself, so every reconnecting client receives materialized state followed by the live stream. Delivery to humans deserves a dedicated article. Here it earns its place as the fourth traffic pattern that most edge architectures forget until the first user complains. Site-to-Site: How Do Sites, Data Centers, and Clouds Stay in Sync? The third direction is sideways: keeping data consistent between many sites and one core, or between multiple clouds. For durable event streams, Kafka-style replication is the default. Open source MirrorMaker 2 or vendor-specific cluster linking copies topics between an edge cluster and the central cluster, preserving ordering and history. Apache Kafka is the open-source option; Confluent, Amazon MSK, Aiven, and other solutions are the managed ones. This suits sites large enough to operate a streaming cluster, such as a factory with local analytics that also feeds the global platform. For thousands of small sites, the streaming cluster per site model collapses under its own operational weight. This is where NATS is architecturally distinct rather than just another broker. The entire server is a single small binary with no external dependencies. Each store, vehicle, or clinic runs a leaf node that dials out to the central cluster, keeps serving local applications when the WAN drops, and uses JetStream mirroring to synchronize persisted streams when connectivity returns. Retail chains run this pattern across thousands of locations. The same design serves both the intermittent case, hours or days offline with a bulk sync on reconnect, and the scheduled-batch case where sites replicate on a timer rather than continuously. Its own maintainers warn against one trap: do not stretch a single stream cluster across unstable edge links, because the cluster's consensus breaks every time the network does. Instead, let each site persist its data locally and keep serving through an outage, then mirror to the center asynchronously when the link returns. For enterprises whose sites run applications rather than device fleets, an event mesh covers the same ground at the application layer. Solace links its brokers across data centers and clouds so an event published anywhere reaches any interested subscriber, with topic routing and governance built in. It is the natural fit when the endpoints are SAP, mainframes, and microservices rather than sensors. And a large share of site-to-site traffic does not need streaming, even if streaming is often the stronger long-term architecture. Nightly file synchronization, batch uploads over metered links, and change data capture from edge databases remain perfectly valid when there is no real-time requirement. This is especially true when the existing OT and IT systems on both ends are batch-oriented anyway, and no business case justifies the cost and effort of making them real time. Real time is a requirement, not a default. Stream when you need fresh data, or when you build greenfield and carry no legacy batch systems to preserve. An architecture that streams everything pays streaming costs for batch problems. Which Category Serves Which Edge to Cloud Pattern? Seven vendor categories map onto the four patterns. No single category covers all of them, and messaging fabrics, event mesh, and protocol proxies deliberately span several. No single vendor covers every direction. Seven categories divide the field, and the real picture is that some sit in one pattern while others deliberately span several. MQTT platforms (EMQX, HiveMQ, Mosquitto, NanoMQ) serve both northbound telemetry and southbound control, because MQTT is bidirectional by design. Hyperscaler services (AWS IoT Core, Azure IoT Operations) do the same for the managed-cloud case, ingesting telemetry and pushing desired state through device twins. Event streaming (Apache Kafka, Confluent, Aiven, and more in the data streaming landscape) owns site-to-site replication. Last-mile delivery(Ably, Lightstreamer, PubNub) owns outbound delivery to people, and it is the only pattern with a category built solely for it. Three categories span multiple patterns. An event mesh (Solace) routes site-to-site and out to applications. Messaging fabrics (NATS, Zenoh) carry northbound, southbound, and site-to-site in a single fabric. Protocol proxies (Zilla) translate across all four, replacing several specialized components with one layer. The underlying distinction is worth stating plainly: a protocol standard is not a deployment architecture. MQTT standardizes the wire, which is why so many products speak it and your devices stay portable. NATS and Zenoh standardize the topology, which reduces moving parts and increases commitment. Zenoh, born from DDS research and adopted by automotive initiatives including General Motors' uProtocol and by the ROS 2 robotics ecosystem, goes furthest: it routes named data along the shortest path between producer and consumer, aiming to make the cloud versus edge placement debate irrelevant. These spanning options are not equivalent. A fabric like NATS or Zenoh reduces moving parts and unifies security by being one technology end to end, but that reach is also the lock-in: the more it carries, the more expensive it is to leave. A protocol proxy like Zilla does the opposite. It spans the patterns by translating between existing technologies, which keeps your exit options open at the cost of another component in the path. The right choice depends on how many patterns you actually need and how much lock-in you can accept, not on which vendor markets hardest. How to Choose: Five Questions That Pick Your Edge-to-Cloud Architecture Product evaluations start too late. Answer five questions first, and the shortlist mostly writes itself. Five-question decision framework for choosing an edge-to-cloud architecture, from consumer type to data residency Who consumes the data, machines or people? People need current state on connect, then live updates, which means a last-mile delivery layer over the backbone. Machines just consume the stream.Does this flow actually need real time? Measure before you architect. If a fifteen-minute batch satisfies the business case, use file sync or change data capture. Stream when you need fresh data, because streaming infrastructure you did not need is the most expensive kind.What is the most constrained node in the path? A microcontroller means MQTT or Zenoh, the two with clients small enough for the device itself, not a JVM. A gateway opens the field to lightweight NATS or full brokers, and a site server adds event meshes or a small streaming cluster.How many sites, and can each one operate a cluster? A few large sites with local analytics justify edge Kafka clusters with MirrorMaker or cluster linking. Hundreds or thousands of small sites demand hub-and-spoke: NATS leaf nodes or hierarchical MQTT bridges, with store and forward at every spoke. Enterprises whose sites run applications rather than devices fit an event mesh.Where must the data live? If residency or sovereignty rules apply, the broker runs on site or in jurisdiction, self-managed, and routing policy becomes a compliance control. If data may leave and connectivity is solid, managed cloud services win on operations cost. Two checks apply on almost every branch, at every step. The edge must dial out and never accept inbound connections to the site. And local operation must survive offline, with a buffer sized deliberately, not discovered in the first outage. Five questions, four patterns, one map. The technologies then sort themselves into it. Moving Data Is Not Automating the Business Every pattern and category in this article solves the same problem: moving data between edge, data center, and cloud. That movement is necessary but not sufficient. When telemetry arrives, something must decide what happens next: trigger the maintenance workflow, roll back the failed firmware deployment, replicate the dataset, open the ticket, retrain the model. Deciding what happens next is workflow orchestration. It spans the same edge-to-cloud boundary as the data movement beneath it. Data integration moves the events. Orchestration, with the process intelligence to see how those flows actually run, decides what the business does about them. And increasingly, agentic AI acts on those decisions at the edge. These are not three separate topics. They are one stack, and edge to cloud is where they meet the physical world. Architectures that move data and improvise everything above it end up with the automation logic scattered across broker plugins, cron jobs, and cloud functions. That is the integration spaghetti this whole discipline set out to remove. Edge to cloud is not one problem. It is four. Architect for the pattern, not the product.
CORE
Agile
Career Development
Methodologies
Team Management
To Switch, or Not to Switch the AI Model
October 26, 2026
by Stefan Wolpers
CORE
How to Build Your Own AI SRE From Scratch
October 23, 2026 by Severin Neumann
October 23, 2026
by Stefan Wolpers
CORE
Cloud Architecture
Integration
Microservices
Performance
MQTT, Kafka, NATS, or Zenoh? Choosing Edge-to-Cloud Messaging by Traffic Pattern
October 27, 2026
by Kai Wähner
CORE
Why Vulnerability Management Fails Without Reliable Asset Data
October 27, 2026
by Alex Vakulov
CORE
SSRF Prevention in C#: Securing File Imports From User-Supplied URLs
October 26, 2026
by Brian O'Neill
CORE
