DZone Spotlight

Every app that accepts photo uploads meets HEIC sooner or later. iPhones save in it by default. Neither Java nor Rust can read it out of the box, and the usual answer, libheif, is a C library: native packages in every container image, a C toolchain in the build, and a parser written in C reading files that strangers upload. I had the problem twice, in a Java product and in a Rust one. So I went the other way: decoders written in the language itself, built and improved by AI coding agents, with me steering. Here is how it went. 1. Make It Work, In Each Project One agent per codebase, each starting from an existing open-source decoder in its own language: Openize.HEIC for Java, HEIC decoder for Rust. The first versions lived inside each app. They read the photos, and that was enough to ship. They were ready in no time, given there were upstream libraries to start from, even if some features had to be added to match libheif. It was tempting to stop there. Nobody complains about an upload that works. 2. Extract and Iterate Next, I had each decoder pulled out as a standalone library, open source, with its own tests and releases, aligned with the best practices for each language (and mine). Then I asked each agent to iterate on its own: measure, improve, measure again. That alone went a long way. The Java decoder went from 19 seconds to 3.2 for our ten test photos. 3. Compare Everything With Everything Working alone, each agent eventually ran out of obvious ideas. What changed the game was putting both decoders next to libheif on the same machine with the same photos, and then comparing them stage by stage. Every gap became a lead. If another implementation did a step fast, there was a way to do it even faster, and the size of the gap told me whether it was worth chasing. A gap is a hint with a number attached. The comparison also kept us honest. It showed where we were ahead, and it showed that part of libheif's lead came from doing less work: it copied color samples where our decoder interpolated them. 4. Make the Agents Talk Here is the part that led to further improvements. Each agent, working in its own session, was told to talk with the other agent. I had the Rust agent brief the Java agent on what it had found, and the other way around. A gap one side had already closed became a recipe for the other. The Java decoder took the Rust decoder's entropy decoding approach; the Rust decoder took the Java decoder's parallel tiles and transform tricks. Later, the two agents, who were then aware of each other, even coordinated on their own, communicating which CPU cores each could benchmark on. Connecting them was the most useful step in the project. 5. Repeat On the Gaps That Remain Then we ran the same loop again: compare, find the largest remaining gap, close it, then measure. One more round on the Rust side closed the last single-core gap with libheif. The Java side tried the JVM's vector instructions and gained another 9 to 14%. Not every idea survived. The trick the C library is famous for, wide vector instructions, gained almost nothing in Rust: its slow parts were elsewhere. We measured it, wrote it down, and dropped it. Be Strict About Evidence A computer is never idle. Browsers, chat apps, other agents: everything competes for the CPU, and the numbers move with it. When we measured the very same code twice, back to back, the second run came out about 0.4% slower. Noise with a direction looks exactly like progress, or like a regression. So I asked for proof, not impressions: count CPU work instead of wall-clock time, run old and new in alternating order, split results by type of photo, and drop any "gain" smaller than the noise. Several changes didn't make it. One that looked good overall turned out to slow down rotated photos, which are most of what phones upload, until the agent found why. Where They Stand Today Measured together on the same machine, the same ten photos, all three doing the same work: 8 cores: Rust 0.46 s, Java 0.55 s, libheif 0.81 s1 core: Rust 1.38 s, libheif 1.45 s, Java 1.96 sRust can also interpolate color for sharper edges and still matches libheif: 1.45 s Memory is on par with libheif or below, and a long way from where the two decoders started. What I Take Away Iterating alone gets you far. Comparing gets you further. The gap between two implementations is the best to-do list.Cross-pollinating agents helps a lot. The ideas were there, in two separate sessions. Making them talk to each other triggered more optimizations.Be strict about evidence. Ask for measurements that hold up against noise, and be ready to throw away a change that doesn't.Two implementations in two languages beat one. Each made the other faster. Getting a third as reference (libheif) helped even more.Java is fast, too. “Java is slow” is a meme, not a measurement. Optimized the right way, it comes close to the C library on one core and beats it on eight. Both decoders are open source: Rust: github.com/Valuya/heic-decoder-rustJava: github.com/Valuya/openize-heic-java

Authentication Isn’t a Feature, It’s Frontend Architecture Whereas the authentication process can typically be implemented as an afterthought for any development team through just creating a login interface that will be connected to some sort of identity management solution, it will not be as straightforward when there is a need to develop a fully functional enterprise solution with the use of Angular and Spring Boot. From implementing single sign-on and multi-factor authentication in such cases with the use of such a front-end/back-end combination, it became evident that authentication is more of a constraint than a feature. Why SSO Changes More Than the Login Screen However, the idea of single sign-on appears to be fairly straightforward – you visit the identity provider and get your token, then that’s all. But that implies that the application will have to start working differently from within. From now on, the routes should have a guard to check if the user is authorized before any component gets initialized. Deep links have to be maintained during the redirect and return from the identity provider, which essentially means that the app needs to maintain the intended destination state in a process it does not control. Plus, since the token comes from an external provider on their own schedule, it needs to be treated by the frontend as an ordinary and expected situation where a token expires. In our platform, we implemented route guards checking token validity before resolving a protected route and token refreshing in the background mode without any interruptions to the user’s workflow. This part was more important for the whole implementation than the single sign-on flow itself, since a technically correct single sign-on flow that constantly interrupts users is still considered broken. The refresh mechanism brought its own nuances. In case multiple requests failed due to token expiration, an oblivious interceptor would spawn multiple refresh actions, while the majority of identity providers will revoke all previous refresh tokens upon the generation of a new one. It was resolved by having a request queue, where the first error will cause a single refresh action to be fired, while the rest of the pending requests would wait for this result and get re-fired with a fresh token. Should the refresh action fail, the queue would be cleared, and the user would go through the entire authentication process with his destination saved. Where Multi-Factor Authentication Adds Real Complexity One of the features MFA brings to the table is an additional level of state that the frontend has to handle properly: users might be authenticated but not yet fully verified. Confusing these two states is what makes most MFA setups break: they allow access to routes only available to fully authenticated users or require users to go through a verification step that they have already done in the current session. We considered the authentication state to be a state machine rather than a mere boolean value due to the existence of different states like the state of unauthenticated, the state of authenticated and waiting for verification, and the state of authenticated. Different permissions were applied in each of those states, and different transitions were possible between the different states. It allowed us to easily predict the results of the authentication code analysis and especially testing since we could test each transition between the different states. Step-up verification introduced yet another complication. Certain activities, like processing an important transaction, needed more recent verification than merely logging in. We simulated this by modeling it as a level of verification tied to a period of time within the session, thus allowing the back-end to request re-verification during certain important tasks while retaining the user’s work at the front-end. The Spring Boot Side of the Contract All of that required the backend to implement the exact same model that the frontend expected. The validation of tokens was done by our Spring Boot services on every single request instead of relying only on the frontend, and we were sending different and clearly recognizable error messages for token expiry, invalid tokens, and insufficient verification level instead of simply sending 401 all the time. And these differences are extremely important on the frontend – token expiry causes silent refreshing of tokens, invalid tokens cause the full authentication process, and insufficient verification level causes the user to be forced to pass through the MFA challenge once again, and not start the whole authentication from scratch. It took us way more coordination and planning to get this contract right than either side expected initially. The backend team could have considered all types of authentication errors to be the same thing and the frontend team could have thought that the error codes would translate perfectly into frontend states without any further checks. We recognized this problem at the very early stages mainly thanks to our SDET testing this boundary explicitly. What This Means for Compliance-Heavy Environments As the banking system is a highly regulated one, authentication was not only about the quality of the user experience; it was also an audit problem. With every state transition done, we needed to be able to explain and justify our choice of actions rather than just copying what worked before. This way we had to move toward using the explicit state machine rather than implicit, since the latter was difficult to prove. The Takeaway It is the choices about the design of the authentication procedure, the model of the state, semantic error management in the frontend-backend communication, token refreshing, and many more that will define the level of opposition that our team needs to overcome in order to make the app survive. The fact that our approach to the design of the authentication procedure as part of the frontend architecture from the very beginning was the reason for our success.

Recap: Where This Fits Part 1 of this series covered exact-match caching, and Part 2 covered semantic caching. Both sit in front of the model and can skip the call entirely on a hit. This article covers what happens to the requests that still reach the model: reusing computed model state, either inside your own inference engine or inside the API provider's system, plus the production details that turn a caching plan into something a team can actually run and trust. Layers 3 and 4 are two versions of the same basic idea, reusing computed model state, split by who runs the model server. If you run your own GPUs, layer 3 applies. If you call a hosted API, layer 4 applies. You rarely need both at once for the same model call. 3. Inference-Engine KV Cache Reuse (Self-Hosted Serving) This layer gets left out of most write-ups because it lives below the application, inside the model-serving system itself, but it is where prefix caching really started, and it is the standard answer for any team running open-weight models on their own GPUs. What Is Actually Being Cached When a model processes a prompt, it computes a key and value tensor for every token, at every layer, during the first pass over the input. That stored state is what the model reads from while it writes its answer. If two requests share the first 3,000 tokens (a shared system prompt, tool list, or conversation history up to a point), redoing the work for those 3,000 tokens on every request is pure waste. Inference engines keep that state and reuse it right on the GPU. This still runs work for the new tokens; it just skips redoing the tokens it has already seen. The Two Main Approaches vLLM, using PagedAttention. vLLM manages this stored state the way an operating system manages memory, in fixed-size pages handed out as needed. Its prefix caching reuses state at the page level whenever a new request starts with pages that match an earlier request exactly. Prefix caching is on by default on recent vLLM V1 engines. Turn it off with --no-enable-prefix-caching for testing. Older versions worked the other way, needing --enable-prefix-caching to turn it on, so check which one your deployed version uses.SGLang, using RadixAttention. SGLang keeps all its stored state in a tree structure built from token sequences, and finds the longest shared starting point for any new request, at any depth, even across branching conversations where several replies split off from a common point. It works by default at the token level. Turn it off with --disable-radix-cache if you want a no-cache baseline. Because the match works at the token level, not a fixed block size, SGLang tends to find more reuse on deep, branching, or unevenly-growing conversations, which is exactly the shape of most agent workflows. The diagram above shows the core difference: vLLM's blocks reuse cleanly when requests share a prefix that lines up with block edges, while SGLang's tree finds the longest shared path at any point, which is why it tends to win on deeply branching agent sessions, where different requests split off at different points, not just at the very end. A Minimal Setup Shell # vLLM: prefix caching is on by default in V1, shown here for clarity vllm serve meta-llama/Llama-3.3-70B-Instruct \ --enable-prefix-caching \ --port 8000 # SGLang: RadixAttention runs by default, no extra flag needed python -m sglang.launch_server \ --model-path meta-llama/Llama-3.3-70B-Instruct \ --port 8000 Both give you an OpenAI-compatible /v1/chat/completions endpoint. Your application code, including the exact-match and semantic cache layers from Parts 1 and 2, does not change at all. Where It Really Pays Off Reported hit rates of 60 to 85% on agent loops, multi-tenant apps, repo Q&A, and long-document RAG, wherever a large shared prefix repeats across many requests.Per-call cost drops of roughly 5 to 12x on a hit, since the first-pass work for the shared prefix is skipped entirely rather than just discounted.On prefix-heavy workloads, SGLang's RadixAttention has been shown to reach several times vLLM's throughput, though the gap narrows on unique, non-overlapping prompts. Trade-Offs and Things to Watch This is a GPU memory problem as much as a caching problem. Stored state competes with model weights and the active batch for GPU memory, and past a few hundred thousand tokens of context, it can use more memory than the model weights themselves. State that does not fit gets removed, and the removal policy (vLLM roughly follows LRU at the block level, SGLang removes along the tree) decides what survives under pressure.Hit rate depends on the prefix staying the same, the same way provider prefix caching does. Reordering tool definitions, or adding a timestamp early in the prompt, breaks it just as easily here as on a hosted API.LoRA adapters, quantization, and speculative decoding all quietly become part of the cache key. Two requests with the exact same text but a different adapter ID or a different quantization level do not share cached state, because the underlying computed values are genuinely different. Fewer, more shared setups cache better than many narrow ones.Keeping tenants apart. Two tenants sharing one inference engine could, in theory, end up reusing each other's cached state. vLLM handles this with a cache_salt setting, a hidden value added to the block hash so only requests with the same salt can share cached blocks. This also closes a timing leak where an attacker could guess cached content just from how fast a response comes back. Set a per-tenant salt on any multi-tenant self-hosted setup. 4. Provider-Side Prompt/Prefix Caching When a request has to reach the model because there was no exact or semantic hit, provider-side prefix caching reuses the expensive first-pass work on long, unchanging parts of the prompt: system prompts, tool lists, RAG documents, few-shot examples. On the Claude API, a cache read costs roughly one-tenth of the normal input price, while cache writes cost more than a normal input token. Anthropic offers two TTL options, a 5-minute default and a longer 1-hour option, and moved the default down from 1 hour in early March 2026. Any setup built before that change, that does not explicitly set ttl on the cache_control block, will see its hit rate drop on workloads with gaps longer than 5 minutes. Check cache_read_input_tokens in your usage data to confirm you are still getting hits. The shortest prefix length that qualifies for caching depends on the model, generally a few thousand tokens, and cache breakpoints depend on order: put unchanging content first and changing content last, so the unchanging part forms a stable prefix. Reordering tool definitions is a common way to silently break the cache without any error showing up. Provider Comparison at a Glance The numbers below shift often and are reported inconsistently across vendor pages. Treat this as a starting point, not something to plan a budget on directly, and confirm every figure on the provider's live docs first. ProviderHow it's triggeredTypical read discountWrite costDefault TTLMinimum prefixAnthropic (Claude API)Explicit cache_control markerAbout 90% off input1.25x at 5-min TTL, 2x at 1-hour TTL5 minutes (1-hour option available)A few thousand tokensOpenAIAutomatic, no code changeVaries by model, historically 50%, up to about 90% on top-tier modelsUsually none5 to 10 min of inactivity; up to 24h on some tiersAbout 1,024 tokensGoogle GeminiAutomatic (implicit), or set up by the developer (explicit)Reported inconsistently, roughly 25% to 90% depending on model and mode, check per modelImplicit: none. Explicit: yes, plus a storage feeImplicit: short. Explicit: set by the developerImplicit: about 1K-4K tokens. Explicit: tens of thousands on some modelsAmazon BedrockExplicit cache-point markersFollows the underlying hosted model's own pricingDepends on the modelDepends on the modelDepends on the model Every provider now offers something in the 50 to 90% read-discount range, but the default behavior is quite different: an explicit marker with a short default TTL, a fully automatic system, or a split between an automatic and a developer-managed mode. Assuming one provider's defaults carry over to another is a common way to lose savings after a provider switch. Keeping the Cache Warm A session that pauses for 6 minutes while a person reviews a tool result pays full price on the next call, under Anthropic's 5-minute default. Two fixes: send a cheap keep-alive request just before the TTL would run out, if your traffic has gaps you can predict, or pay for the longer TTL option where gaps regularly run past the short default. The trade-off is simple: the extra write cost of the longer tier against the input-token cost of the misses it would otherwise cause. If a system prompt gets reused more than about 5 to 7 times within the window, the longer TTL usually wins out. Best-of-Breed Production Scenarios ScenarioMain LayersNotesFAQ/support botExact + SemanticLots of reworded repeats; use a strict semantic threshold and a short TTLAgentic workflows, hosted APIExact + Provider prefixCache tool definitions and the system promptAgentic workflows, self-hostedExact + Inference-engine KV cacheSGLang usually works better for deep or branching agent stateDocument Q&A / RAGProvider prefix or Inference-engine KV cache + SemanticCache the long document prefix, match the question with semantic searchForm-driven / templatedExact onlySafest and lowest latency choiceMulti-tenant appAll layers + tenant filtersFilter by tenant at the vector search step and the KV cache step, not just in app codeTime-sensitive (pricing, stock)Exact/Semantic, short TTL + clear on eventsA stale price is worse than a cache missSelf-hosted deployment (any use case)Inference-engine KV cache is the baselineRuns automatically once you pick the engine, add exact and semantic caching on top Cost and Latency Expectations Exact hit: basically free, usually under 10 ms.Semantic hit: an embedding call plus a vector search, tens of milliseconds, no model cost.Inference-engine KV cache hit: roughly 5 to 12x lower cost per call on the matched part of the prompt, with hit rates of 60 to 85% on agent and RAG workloads.Provider prefix hit: roughly a 90% cut on the cached part of the input tokens, though write cost and TTL choice affect how much of that saving you actually keep.Combined setups commonly report 50 to 90% lower overall cost, with templated traffic at the high end and open-ended chat at the low end. Observability: What to Actually Watch "Log hits, misses, and similarity scores" is correct, but too vague to act on: Hit rate for each layer. Exact, semantic, and inference-engine or provider cache hit rate (cache_read_input_tokens on Anthropic, the matching field elsewhere, or the engine's own stats on vLLM/SGLang), tracked apart from each other. A drop in one layer can easily hide behind stability in another.False-positive rate on semantic hits, checked on an ongoing basis. This is the metric most often skipped, and the one most likely to catch a wrong answer before a user complains about it.Stampede rate and wait time, from the request coalescing pattern in Part 1. A rising wait time under steady load is an early sign the model call itself is getting slower.GPU memory use for stored state, on self-hosted setups. A hit rate that drops under load is often explained by state being removed under memory pressure, not by a change in traffic.Cost per request, broken down by layer. This tells you which layer to spend more time on next. If you can only add one thing first, make it hit rate per layer. Everything else is used to work out why that number moved. A Worked Cost Example Take a support chatbot with an 8,000-token system prompt plus tool definitions, a Claude Sonnet-class model at roughly $3 per million input tokens, and 1 million requests a month. Without caching, that prefix alone costs roughly $24,000 a month, before a single user token is even counted. Exact and semantic caching catch 60% of requests as either identical or reworded repeats, which is common on a support FAQ. Those never reach the model. Of the remaining 400,000 requests, 80% of the prefix tokens hit the provider's prefix cache at roughly $0.30 per million instead of $3 per million. That works out to (400,000 x 8,000 x 0.8 x $0.30/M) + (400,000 x 8,000 x 0.2 x $3/M), which is about $2,688, against $9,600 without caching for that same slice of traffic. Total prefix cost across all traffic lands around $2,688, against a $24,000 uncached baseline: about an 89% cut, matching the higher end of the range mentioned earlier. That number depends entirely on the 60% and 80% hit-rate guesses used here. A very repetitive FAQ can genuinely land here. An open-ended assistant will land closer to the 50% floor. Run this same math against your own measured hit rates from the metrics above, instead of assuming a headline number from a vendor blog matches your own traffic. Implementation Checklist Clean up prompts before hashing. Include every setting that changes the output in the exact-match key.Pick the embedding model and distance measure on purpose, and test it against real traffic before trusting a threshold in production.Set TTLs and size limits on every cache layer, including semantic. Keep semantic TTLs shorter than exact-match TTLs by default.Log hits, misses, and similarity scores broken down by layer, not as one combined number.Guard against personal data and cross-tenant leaks at the vector search and KV cache steps directly, not just in app code.Handle repeated concurrent requests with request coalescing, instead of letting a stampede hit the model.Prefer built-in provider prefix caching, or your inference engine's default KV cache reuse if self-hosted, and set TTLs by hand.Keep application-level caching on top of model-side reuse. They solve different problems, and neither one replaces the other.If you self-host, pick vLLM or SGLang based on your workload (throughput on batches vs. deep multi-turn agent state), not just habit.Start with exact match. Add semantic caching only after you measure the rate of reworded queries and false positives on your own traffic. Closing The right way to think about this, across all three parts of this series, is to get the exact-match case right first, then add semantic matching on top while being honest about the trade-offs: some error, extra latency, and a harder invalidation problem. Under both of those, whatever reaches the model gains from reusing computed state, whether that happens inside your own inference engine or inside the provider's system. The pattern that works is layered: exact first, semantic second, then reuse on the model side for whatever is left, with careful thresholds, TTLs, and monitoring at every step. None of this is a one-time job. Provider defaults change, traffic patterns shift, and a threshold tuned to last quarter's queries will not necessarily hold up this quarter. Build the exact-match layer today, watch the traffic you actually have, and let the measured rate of reworded queries and false positives tell you whether semantic caching's extra hit rate is worth the added work for your own workload. Further Reading Anthropic, "Prompt caching," Claude Platform DocsOpenAI, "Prompt caching," OpenAI API DocsGoogle, Gemini API context caching documentation (implicit and explicit modes)Amazon Bedrock prompt caching documentationvLLM documentation, automatic prefix caching and PagedAttentionSGLang documentation, RadixAttention

By Ram Ghadiyaram CORE

Production code gets scrutiny. Builds are watched, libraries are checked for known vulnerabilities, and someone is responsible when a service fails at night. The job that pulls in outside data usually escapes all of this. It began as a quick script, got scheduled, and has run untouched ever since. It has no tests, its only error handling is a catch-all exception block, and its author may have left the team long ago. The failures are rarely loud. A scraper does not crash when a site stops cooperating. It keeps running and keeps writing rows, and the rows quietly get worse. This article covers where pipelines actually break, how to respond to each kind of failure, and how to decide between maintaining the collection layer yourself and licensing it. The Block Page Is the Last Signal A CAPTCHA is the most visible sign of trouble, but sites test requests long before showing one. Several quieter checks decide whether the client looks like a real browser: TLS fingerprinting. Methods such as JA3 and the newer JA4 profile a client from its handshake. A stock Python or Node HTTP library is recognizable before any header is read. Changing the User-Agent string does nothing here, because the handshake gives the client away first.Rate limiting. Limits can attach to an IP, a session, or a client fingerprint. Some sites return a clean 429 with a Retry-After header. Many return a 403, a CAPTCHA, or an ordinary page with the data removed.Interstitials. Consent prompts and “checking your browser” pages often return status 200, as do regional redirects. A naive client logs a success without ever seeing the content.Honeypots. Links hidden with CSS are invisible to people. A crawler that follows every anchor walks straight into them, and the site can flag the session on the spot. Because these checks fail silently, the response code is the wrong thing to trust. Treat a 200 as a claim that needs verification. Compare each response against what that source normally returns. A page that usually weighs 150 KB and suddenly weighs 3 KB deserves a flag, whatever the status says. So does any page containing phrases typical of challenge screens, such as “verify you are human” or “checking your browser.” Both checks are cheap, and together they catch most soft blocks. Thresholds and phrase lists need tuning per target, since every site words its challenges differently. Even a pipeline that passes every check today will break when the target changes its markup. The checks have to keep running after launch. Different Failures Need Different Responses A scraper that reacts to every problem the same way will either hammer a site or give up too early. Each signal calls for its own handling: A 429 means slow down. Wait for the interval the server names in Retry-After, then resume.A persistent 403 means stop and investigate. More requests will not change the answer. The cause is usually a fingerprint, a policy, or an IP reputation problem that needs a human decision.A 200 with unexpected content means validate. Run the size and phrase checks, quarantine the response, and do not count it as a success.A 404 means record it and move on. A second attempt adds load and log noise without changing the outcome. For transient failures, retry with exponential backoff. Wait a second after the first failure, two after the second, four after the third, up to a sensible ceiling. Add random jitter to every wait. Without it, many workers retrying on the same schedule hit the site at the same moment and trigger the limit again. Rendering Adds a Second Problem More sites now deliver content through client-side JavaScript. A plain HTTP request returns an empty shell with a script tag and nothing to parse. Before reaching for a browser, open the network tab. Many pages load their data from a JSON endpoint that the client can call directly. That path is usually faster and cheaper than parsing rendered HTML, and it often changes less when the page design changes. It comes with a caveat. Undocumented endpoints carry no stability promise, so they can change or disappear without notice and need the same monitoring as any other target. Check the site’s terms and robots rules as well before depending on one. When no usable endpoint exists, a headless browser driven by Playwright or Puppeteer is the usual answer. It solves rendering and creates new costs: Each browser instance uses far more memory and CPU than an HTTP client, so throughput per machine drops sharply.Page loads take seconds instead of milliseconds, and flaky waits produce intermittent empty results.Detection systems look for automation signals such as navigator.webdriver, inconsistent canvas output, and unusual viewport settings. One habit prevents a large share of rendering bugs. Wait for a specific element that proves the content has loaded, not for a fixed number of seconds, and fail the job when that element never appears. A fixed sleep hides rendering failures. An explicit wait turns them into errors someone can see. Why the Pieces Are Harder Together Than Apart Proxy rotation, retry policy, and browser rendering are each manageable alone. Combined, they interact. A rotated IP changes the apparent location, which changes the consent banner, which changes the page structure your selector expects. A slow browser session can outlive the proxy connection it started on. Retries multiply browser launches and the cost that comes with them. Keeping all of this stable beyond a few requests a day is an engineering discipline of its own, separate from whatever the data is used for afterward. AI Workloads Raise the Bar A scraper built for a nightly batch load rarely suits AI use. A retrieval system needs a site’s structure mapped before it can decide what to embed. An agent that calls a scraper as a tool needs output it can parse immediately, returned fast enough to fit inside one tool call instead of a background job. Retrofitting a batch scraper for interactive use tends to expose every shortcut taken in the original build. Output format becomes a real requirement for these consumers. Clean Markdown or structured JSON saves a cleanup step that otherwise sits between collection and an embedding pipeline. Raw HTML still has a place when an application needs the full page structure. No format removes the need to validate content. Agents also introduce a security problem. When a model or an outside user can supply the URL that a scraper fetches, the scraper becomes a way to reach internal systems. This is server-side request forgery. A request aimed at a private address or a cloud metadata endpoint can expose data that was never meant to leave the network. Production deployments should restrict which destinations are allowed, resolve each hostname and check the resulting address before connecting, and apply the same checks to every redirect. Build or License Building means running several parts at once: Rotating proxy pool with enough geographic spread for the targets you needRetry queue with per-domain rate limitsBrowser cluster for JavaScript-heavy sitesParsing layer that changes whenever a target doesMonitoring that tells you when any of the above has quietly stopped working Each part is well documented. The difficulty is keeping them reliable together, which looks less like writing a script and more like operating a small production service with its own on-call rotation. Proxy coverage is the piece teams most often underestimate. Reaching residential and mobile addresses across many regions is a procurement and operations problem, not a coding one. For lighter needs, building may be more than the job requires. An official API, a published feed, or a scheduled HTTP client is often enough, and none of them needs this much machinery. Licensing moves some of that work to a provider. The category has moved well beyond proxy resale. Some managed services combine scraping, search, site mapping, and crawling under one API key. For example, a web data API can return HTML or Markdown, with structured formats such as JSON and CSV available for supported targets. This can reduce conversion work before the data reaches a database, a BI tool, or an embedding pipeline. Mapping and crawling endpoints also help with scoping. A map request can list a large share of a domain’s URLs up front, which lets a team estimate size and pick what deserves a full crawl. A crawl job can then collect thousands of pages in one pass, enough to cover a typical documentation site or product catalog. Licensing adds its own obligations. Check how the provider reports failures, whether you can see the raw response, how pricing changes at volume, where its IPs are located, and how much control you have over sessions and output. A managed service also does not take over responsibility for data quality, security, or compliance. When collecting personal data, consider why it is needed and whether its use is consistent with the original purpose of collection. Your application still has to detect unusable responses and decide what is permitted to collect. An opaque API that fails silently is the same problem you started with, moved to someone else’s servers. Plan for Recovery Whichever route you choose, assume jobs will fail midway. Failed work should produce diagnostics that name the target, the failure type, and whether it looks temporary. Collection should be restartable for the affected pages only, without repeating everything that already succeeded. Engineers also need to tell network errors from access restrictions, rendering errors and extraction errors. A dashboard of successful HTTP responses cannot do that when the records inside those responses are incomplete. Decide With a Pilot Which route is right depends on whether data collection is the product or only an input to it. If scraping is core to what you sell, owning the stack gives you control where it counts. If it feeds something else, every hour spent on proxies and fingerprints is an hour not spent on the actual product. The quickest way to find out is to run one representative target through both approaches at small scale for a few weeks. Track these for each: Engineering hours spent building and fixingShare of requests that produce usable data, measured by content and not by status codesCost per thousand usable pagesLatency per requestTime to recover after the target changes The number that matters is not how many requests the system completes. It is how reliably the system delivers usable data at a cost the team can sustain. A pilot on a production target settles that better than any vendor claim or internal estimate.

By Alex Vakulov CORE

An agent built in Microsoft Foundry makes many decisions in a single run. It calls a model, selects a tool, reads the tool result, and calls the model again. When the answer is slow or wrong, the cause sits somewhere inside that sequence, and the final response gives no hint about which step failed. Foundry records this sequence as OpenTelemetry traces. By default, a hosted agent sends them to Application Insights, which is a good place to start. Two gaps remain for a team that builds agents every day. The first gap appears during development. A developer who changes a prompt or a tool wants to see the spans of that one run within seconds, on a laptop, without a deployed project. The second gap appears after deployment. Agent owners want a shared dashboard with run counts, latency percentiles, token use, tool latency, and errors, and they want to query raw spans with the same language they use for the rest of their data. This article builds a debugging dashboard that covers both gaps with one OpenTelemetry Collector configuration. In development, the collector runs in Docker Compose next to the Aspire dashboard, which displays each trace as a waterfall. In production, the same collector forwards spans to a Microsoft Fabric Eventstream. An Eventhouse stores them, and a Real-Time Dashboard reads them with KQL. Teams that already run Grafana can read the same Eventhouse there as well, and a later section shows that option. A note on scope before the build starts. The collector configuration and the Compose file in this article were checked for syntax only, and the Fabric steps follow the product documentation. Run each stage against a real project and compare the result with the screenshots described here. The dashboard images in this article use illustrative data. What a Foundry Agent Emits Foundry follows the OpenTelemetry semantic conventions for generative AI. These conventions currently have Development status, so attribute names can change between releases. The agent tracing documentation names several span types, which the following table summarizes together with the attributes this article reads. Span nameWhat it coversAttributes used in this articleinvoke_agentOne complete run of an agentgen_ai.agent.name, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, error.typeexecute_toolOne tool call made by the agentgen_ai.tool.call.arguments, gen_ai.tool.call.resultplan and invoke_workflowPlanning and workflow stepsOperation specific The Foundry documentation does not list every attribute that the service writes on every span. For that reason, the KQL later in this article is a starting point. Open one real span in the Aspire dashboard first, read its attribute names, and adjust the queries if they differ. Message content is the second point of attention. The environment variable OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT controls whether prompts and responses are written into spans. Enable it on a laptop when the content is needed to debug a prompt. Leave it disabled for anything that leaves the laptop, because spans sent to a provider other than Microsoft can contain prompt and tool content. Trace One Agent Run Locally The client-side tracing path uses the azure-ai-projects package at version 2.0.0 or later, together with the OpenTelemetry SDK and an exporter. Tracing for generative AI is an opt-in feature, so the environment variable AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING must be set to true before the instrumentor starts. Shell pip install "azure-ai-projects>=2.0.0" azure-identity opentelemetry-sdk \ azure-core-tracing-opentelemetry opentelemetry-exporter-otlp The documentation shows a console exporter and describes the OTLP exporter as a replacement without a full sample. The script below applies that replacement. It creates a prompt agent, sends one request through the Responses API, and exports the spans to localhost:4317, which is the port that the collector will expose. Python import os os.environ["AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING"] = "true" os.environ["OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"] = "false" from azure.identity import DefaultAzureCredential from azure.ai.projects import AIProjectClient from azure.ai.projects.models import PromptAgentDefinition from azure.ai.projects.telemetry import AIProjectInstrumentor from opentelemetry import trace from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter provider = TracerProvider(resource=Resource.create({"service.name": "foundry-agent-local"})) provider.add_span_processor( BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)) ) trace.set_tracer_provider(provider) AIProjectInstrumentor().instrument() tracer = trace.get_tracer(__name__) project = AIProjectClient( endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"], credential=DefaultAzureCredential(), ) with tracer.start_as_current_span("local-agent-run"): agent = project.agents.create_version( agent_name="trace-demo-agent", definition=PromptAgentDefinition( model=os.environ["FOUNDRY_MODEL_NAME"], instructions="Answer in one sentence.", ), ) client = project.get_openai_client() response = client.responses.create( input="What does a scheduler heartbeat tell an operator?", extra_body={"agent": {"name": agent.name, "type": "agent_reference"}, ) print(response.output_text) provider.force_flush() Set FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL_NAME in the shell, and sign in with the Azure CLI so that DefaultAzureCredential can find an identity. The script runs correctly before the collector exists, but it prints an export warning because nothing listens on port 4317 yet. Start the Collector and the Aspire Dashboard The Aspire dashboard is a standalone container that accepts OTLP and displays traces, logs, and metrics. It listens for OTLP over gRPC on container port 18889 and serves its interface on port 18888. The collector forwards to it, so the same spans can later go to a second destination. YAML services: otel-collector: image: otel/opentelemetry-collector-contrib:latest command: ["--config=/etc/otelcol/config.yaml"] environment: FABRIC_KAFKA_BOOTSTRAP: ${FABRIC_KAFKA_BOOTSTRAP:-localhost:9093} FABRIC_KAFKA_TOPIC: ${FABRIC_KAFKA_TOPIC:-agent-traces} FABRIC_KAFKA_CONNECTION_STRING: ${FABRIC_KAFKA_CONNECTION_STRING:-unset} volumes: - ./otel-collector-config.yaml:/etc/otelcol/config.yaml:ro ports: - 4317:4317 - 4318:4318 depends_on: - aspire-dashboard aspire-dashboard: image: mcr.microsoft.com/aspire/dashboard:latest environment: ASPIRE_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS: "true" ports: - 18888:18888 # Optional. Start with: docker compose --profile grafana up -d # The Azure Data Explorer plugin is a Grafana Enterprise plugin and needs a license. grafana: profiles: ["grafana"] image: grafana/grafana-enterprise:latest environment: GF_INSTALL_PLUGINS: grafana-azure-data-explorer-datasource GF_ENTERPRISE_LICENSE_TEXT: ${GRAFANA_LICENSE_TEXT:-} EVENTHOUSE_QUERY_URI: ${EVENTHOUSE_QUERY_URI:-} AZURE_TENANT_ID: ${AZURE_TENANT_ID:-} AZURE_CLIENT_ID: ${AZURE_CLIENT_ID:-} AZURE_CLIENT_SECRET: ${AZURE_CLIENT_SECRET:-} volumes: - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro ports: - 3000:3000 The anonymous access variable removes the login prompt, which is acceptable for a dashboard that is reachable only from the local machine. The grafana service belongs to a Compose profile, so it does not start until a later section asks for it. The collector configuration defines two pipelines, one for each destination. YAML receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: timeout: 5s exporters: # Local trace viewer otlp/aspire: endpoint: aspire-dashboard:18889 tls: insecure: true # Fabric Eventstream, through its Kafka-compatible custom endpoint kafka/fabric: brokers: - ${env:FABRIC_KAFKA_BOOTSTRAP} traces: topic: ${env:FABRIC_KAFKA_TOPIC} encoding: otlp_json tls: insecure: false auth: sasl: mechanism: PLAIN username: $$ConnectionString password: ${env:FABRIC_KAFKA_CONNECTION_STRING} service: pipelines: traces/local: receivers: [otlp] processors: [batch] exporters: [otlp/aspire] traces/fabric: receivers: [otlp] processors: [batch] exporters: [kafka/fabric] Several details in this file deserve attention. The exporter name otlp/aspire points at the service name aspire-dashboard on the Compose network, and TLS is disabled for that hop only. The SASL username contains a literal dollar sign, so the collector configuration escapes it as $$ConnectionString. The encoding value otlp_json makes each Kafka message a readable JSON document, which the Eventhouse can ingest without a custom decoder. Start the stack with docker compose up -d, then run the Python script. A new trace appears at http://localhost:18888 under the service name foundry-agent-local. Select it to see the waterfall, with invoke_agent at the top and its child spans below. The first run is the moment to record the real attribute names for the KQL stage. Without Fabric values in the environment, the Kafka pipeline fails to connect, and the collector logs retry messages. The local pipeline continues to work, because each pipeline has its own queue. Comment out the traces/fabric pipeline if the log noise is distracting. Create the Fabric Eventstream and Eventhouse The Fabric side has three items. An Eventstream receives the spans, an Eventhouse stores them, and a Real-Time Dashboard displays them. The Eventstream accepts Kafka clients through a custom endpoint source, which is the reason the collector can use its standard Kafka exporter. Create an Eventhouse named agent-telemetry. Fabric creates a default KQL database with the same name.Create an Eventstream and add a Custom endpoint source.Open the source, select the Kafka tab, and choose SAS Key Authentication. The page shows the bootstrap server, the connection string for the primary and secondary keys, and the topic name.Copy these values into a .env file next to the Compose file. Plain Text # Copy to .env and fill in from the Kafka tab of the Eventstream custom endpoint. FABRIC_KAFKA_BOOTSTRAP=<bootstrap-server-from-sas-key-authentication-page> FABRIC_KAFKA_TOPIC=<topic-name-from-the-kafka-tab> FABRIC_KAFKA_CONNECTION_STRING=<connection-string-primary-key> The Fabric documentation describes the authentication settings as security.protocol of SASL_SSL, sasl.mechanism of PLAIN, a username equal to the literal text $ConnectionString, and the connection string as the password. The collector configuration above maps each of these to its own keys. Copy the bootstrap server exactly as the page shows it, including the port. Before the Eventstream can write to the Eventhouse, the tables must exist. Run the following script in the KQL query set of the database. It creates a landing table, a typed table for analysis, a function that flattens the OTLP structure, and an update policy that runs the function for every ingested batch. Plain Text .create table RawTraces (resourceSpans: dynamic) .create table AgentSpans ( Timestamp: datetime, TraceId: string, SpanId: string, ParentSpanId: string, Service: string, SpanName: string, Operation: string, AgentName: string, Model: string, DurationMs: real, InputTokens: long, OutputTokens: long, ErrorType: string ) .create-or-alter function ExpandAgentSpans() { RawTraces | mv-expand rs = resourceSpans | mv-apply a = rs.resource.attributes on ( where a.key == "service.name" | project Service = tostring(a.value.stringValue) ) | mv-expand ss = rs.scopeSpans | mv-expand sp = ss.spans | mv-apply a = sp.attributes on ( summarize attrs = make_bag(bag_pack(tostring(a.key), a.value)) ) | project Timestamp = unixtime_nanoseconds_todatetime(tolong(sp.startTimeUnixNano)), TraceId = tostring(sp.traceId), SpanId = tostring(sp.spanId), ParentSpanId = tostring(sp.parentSpanId), Service, SpanName = tostring(sp.name), Operation = tostring(attrs["gen_ai.operation.name"].stringValue), AgentName = tostring(attrs["gen_ai.agent.name"].stringValue), Model = tostring(attrs["gen_ai.request.model"].stringValue), DurationMs = (tolong(sp.endTimeUnixNano) - tolong(sp.startTimeUnixNano)) / 1000000.0, InputTokens = tolong(attrs["gen_ai.usage.input_tokens"].intValue), OutputTokens = tolong(attrs["gen_ai.usage.output_tokens"].intValue), ErrorType = tostring(attrs["error.type"].stringValue) } .alter table AgentSpans policy update @'[{"IsEnabled": true, "Source": "RawTraces", "Query": "ExpandAgentSpans()", "IsTransactional": false, "PropagateIngestionProperties": false}]' .alter-merge table RawTraces policy retention softdelete = 1d Each Kafka message holds one OTLP export request, so each row in RawTraces holds one resourceSpans array. The function expands that array level by level, reads the service name from the resource attributes, and converts the span attributes into a property bag. OTLP JSON stores every attribute as a typed object, for instance {"intValue": "412"}, which explains why the function reads .intValue and .stringValue separately. The retention policy on RawTraces keeps raw data for one day, because the typed table holds the long-term copy. Back in the Eventstream, add the Eventhouse as a destination. Select direct ingestion, choose the RawTraces table, and set the data format to JSON. When the mapping page appears, map the top-level resourceSpans field to the single column of the same name. Publish the Eventstream, then restart the collector so that it picks up the .env values. Python docker compose up -d --force-recreate otel-collector python agent_trace_demo.py Within a minute, the query AgentSpans | take 10 should return rows. When it does not, query RawTraces | take 1 first. An empty landing table points to the Kafka connection, and a populated landing table with an empty AgentSpans points to the update policy or to an attribute name that differs from the one in the function. Build the Real-Time Dashboard Create a real-time dashboard in the same workspace and add the Eventhouse database as a data source. Each tile runs one KQL query. The queries below cover the five questions that agent owners ask most often. The first compares load between agents, and the second shows whether latency has changed. The third and fourth show cost and tool health, and the last lists failures. Plain Text // Agent runs per 15 minutes AgentSpans | where Operation == "invoke_agent" and Timestamp > ago(6h) | summarize Runs = count() by bin(Timestamp, 15m), AgentName | render timechart // Run latency, median and 95th percentile AgentSpans | where Operation == "invoke_agent" and Timestamp > ago(6h) | summarize p50 = percentile(DurationMs, 50), p95 = percentile(DurationMs, 95) by bin(Timestamp, 15m) | render timechart // Token use per hour AgentSpans | where Operation == "invoke_agent" and Timestamp > ago(24h) | summarize InputTokens = sum(InputTokens), OutputTokens = sum(OutputTokens) by bin(Timestamp, 1h) | render columnchart // Tool calls and their latency AgentSpans | where Operation == "execute_tool" and Timestamp > ago(6h) | summarize Calls = count(), p95 = percentile(DurationMs, 95) by SpanName | order by Calls desc // Errors by type AgentSpans | where isnotempty(ErrorType) and Timestamp > ago(24h) | summarize Errors = count() by ErrorType | order by Errors desc Two choices in these queries are worth explaining. Latency uses percentiles, because the mean hides slow runs when most runs are fast. Token counts are summed from invoke_agent spans only, so a model call inside a run is not counted twice. Set the dashboard to refresh every 30 seconds. A tile that shows errors can also be connected to a Fabric Activator rule, which sends a message to a Teams channel when the count exceeds a threshold. That step is optional, and it works on the same AgentSpans table. Read the Same Data in Grafana The Real-Time Dashboard is the shortest path, because it needs no extra software. Many teams already operate Grafana for other services, and they prefer to keep agent panels next to those. Grafana can query the Eventhouse with the Azure Data Explorer data source plugin, because an Eventhouse accepts the same KQL queries as an Azure Data Explorer cluster. One condition applies before any setup. The plugin is a Grafana Enterprise plugin. Its installation page states that it needs an Enterprise license for self-managed Grafana, or a Pro or Advanced plan on Grafana Cloud. If no license is available, the Real-Time Dashboard from the previous section remains the way to read the data. The Compose file already contains a grafana service in a profile. It uses the Enterprise image, installs the plugin at startup through GF_INSTALL_PLUGINS, and reads a provisioning file that defines the data source. YAML apiVersion: 1 datasources: - name: Eventhouse type: grafana-azure-data-explorer-datasource access: proxy jsonData: clusterUrl: ${EVENTHOUSE_QUERY_URI} tenantId: ${AZURE_TENANT_ID} clientId: ${AZURE_CLIENT_ID} defaultDatabase: agent-telemetry secureJsonData: clientSecret: ${AZURE_CLIENT_SECRET} version: 1 The data source authenticates with an app registration. Create one in Microsoft Entra ID and add it as a viewer on the agent-telemetry KQL database in Fabric. Copy the Query URI from the Eventhouse page into EVENTHOUSE_QUERY_URI, and add the tenant ID, client ID, and client secret to the .env file. Whether a service principal can reach an Eventhouse depends on the Fabric tenant settings, so ask a Fabric administrator to confirm that setting before the first attempt. Shell docker compose --profile grafana up -d Open http://localhost:3000, select the Eventhouse data source, and use Save and test. A panel then takes the same queries as the Real-Time Dashboard. Replace the fixed ago(6h) filter with the Grafana time macro so that the dashboard time picker controls the range, for example where $__timeFilter(Timestamp). Check the macro names against the plugin documentation for the installed version. Both dashboards read one table, so the numbers agree. The choice between them depends on where the team already works. The Real-Time Dashboard fits a team that works in Fabric, and Grafana fits a team with an existing alerting and on-call setup. Send Spans From a Hosted Agent A hosted agent in Foundry sends telemetry to Application Insights through the connection string that the platform injects. It also exports OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is set. The related variables are OTEL_EXPORTER_OTLP_PROTOCOL, which accepts http/protobuf or grpc, OTEL_EXPORTER_OTLP_HEADERS, and OTEL_TRACES_SAMPLER. These variables belong in the environment_variables section of the agent definition. YAML environment_variables: OTEL_EXPORTER_OTLP_ENDPOINT: https://<collector-host> OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf OTEL_EXPORTER_OTLP_HEADERS: ${{connections.otel-secrets.credentials.otlp_headers} The headers value usually carries a credential, so the documentation stores it in a custom keys connection and references it, as the last line shows. Check the hosted agent telemetry page for the exact agent definition schema before the first deployment. Several constraints shape the design. Environment variables are fixed for each agent version, so a change to the endpoint requires a new version. The service.name value always equals the agent name. OTLP export does not disable Application Insights, and project monitoring must be turned off for an OTLP-only setup. The collector must also be reachable from Foundry. A container app with HTTPS ingress is one option, and the same otel-collector-config.yaml works there with the receiver on port 4318 and the Aspire exporter removed. Add authentication to the receiver before exposing it, because an open OTLP endpoint accepts spans from any sender. Where This Leaves You The local stack answers the question a developer asks after every change: what happened inside this run. The Fabric stack answers the question an owner asks every morning: how the agent behaved overnight. Both read the same spans from the same collector, so a pattern found on the dashboard can be reproduced on a laptop with a matching input. Two limits apply. The attribute names come from a convention still in development, so the flattening function is the only place to adjust after a platform update. Eventstream and Eventhouse also consume Fabric capacity, so review the sampler setting for high-volume agents. References Trace an agent in Microsoft Foundry (client-side tracing)Configure telemetry for hosted agents in Microsoft FoundryAgent tracing concepts in Microsoft FoundryConsume events from a Fabric Eventstream with the Kafka endpointOpenTelemetry Collector Contrib: Kafka exporterAspire standalone dashboardAzure Data Explorer data source for Grafana: installationAzure Data Explorer data source for Grafana: configuration

By Jubin Soni, FBCS CORE

These are failure modes that actually page you — and the boring fixes that stop them. A Kafka consumer is a stateful client built on a membership protocol. It joins a group, accepts a partition assignment, processes records in order, and proves it is still alive on a fixed cadence of heartbeats and poll calls. Every guarantee the group offers — ordered processing, shared load, exactly one owner per partition — rests on that membership staying stable while the work flows through it. Kubernetes starts from the opposite assumption. A pod is disposable. The platform restarts pods, throttles CPU, rolls deployments, and reclaims nodes on a schedule the consumer group never agreed to, and each of those routine events reaches the group protocol as a member leaving, arriving, or going silent. Most consumer incidents on EKS are not Kafka bugs. They are the collision between a stateful, membership-based client and an orchestrator doing exactly what it was designed to do. In this post, I start with the baseline configuration that prevents half of these problems outright, then walk through the eight failure modes that account for nearly everything else — symptom, mechanism, and fix, which is usually a config line or a lifecycle hook rather than a redesign. I close with the design principle that ties the eight together and the debug routine that names a failure mode in minutes. The Core Tension A standalone consumer fails when its code fails or when its broker fails. A consumer on EKS carries a third failure source that never sleeps: the platform itself. Rescheduling a pod is normal platform behavior. Enforcing a CPU limit is normal platform behavior. Terminating a container during a rollout is normal platform behavior. The group protocol holds no category for these events except membership change, so it reacts to all of them with the tools it has — rebalances, session expirations, revoked partitions — whether the event was an accident or a planned deploy. That framing changes how the whole problem gets solved. A restart stops being an exception to recover from and becomes a permanent condition to design around: stable identities, generous timing, and a shutdown path the group can see coming. The eight modes below are what happens when a design skips that framing and assumes the laptop's stability on the platform's terms. Start With the Baseline Half of these issues never happen with a sane baseline: Properties files session.timeout.ms=10000 heartbeat.interval.ms=3000 max.poll.interval.ms=300000 max.poll.records=500 partition.assignment.strategy=CooperativeStickyAssignor enable.auto.commit=false auto.offset.reset=earliest Three decisions in that block do most of the work. Manual commits put offset progress under the application's control, so a restart replays records instead of silently skipping or duplicating them. Cooperative-sticky assignment keeps partitions with their current owners through a rebalance wherever it can, instead of revoking everything and starting the assignment from zero. A poll interval with real headroom gives slow processing room to finish before the group concludes the member has died. Everything below is what still goes wrong on top of that baseline — and what to do when it does. The Eight Failure Modes 1. Pod Restart and CrashLoop Symptoms: The pod keeps restarting; the consumer leaves and rejoins the group constantly. Cause: An application error or an OOMKill; a long startup; uncaught exceptions inside the poll loop. The poll loop is the consumer's heartbeat in the broad sense — the thread that polls is the thread that processes, so an exception that escapes it takes the whole member down, and the group sees a departure to rebalance around. Fix: Fix resource limits, handle exceptions in the loop, and commit sync or async with idempotent processing, so a restart replays safely instead of corrupting progress. Idempotency is the load-bearing half of that fix: a consumer that can replay a batch without harm treats a restart as a routine event, while a consumer that cannot turns every CrashLoop into a data problem on top of an availability problem. 2. Session Timeout — Heartbeats Missed Symptoms: Logs report "member has failed"; frequent rebalances; consumption stalls. Cause: GC pauses or CPU throttling starve the background heartbeat thread. The consumer is alive — it just cannot prove it in time. The heartbeat runs on its own thread precisely so that processing cannot block it. That separation holds only while the scheduler gives the thread its share of CPU; a throttled container breaks it quietly, and the first sign is the group expelling a member that was never down. Fix: Raise session.timeout.ms, fix CPU and memory limits so the heartbeat thread gets scheduled, and keep poll() being called regularly. Raising the timeout buys tolerance; fixing the limits removes the cause. The timeout increase without the limits change leaves a consumer that fails more slowly instead of one that stops failing. 3. max.poll.interval.ms Exceeded Symptoms: "Consumer is not polling" — the member is kicked out mid-processing. Cause: Processing per batch takes longer than the poll interval allows, usually because of slow database or API calls inside the loop. The poll interval is the group's second liveness check: it measures not whether the member exists but whether the member is still making progress through its assignment, and slow downstream calls stretch a single batch past the limit without anything being wrong with the consumer itself. Fix: Raise max.poll.interval.ms, shrink batches with max.poll.records, or hand work to worker threads with pause/resume so the poll loop keeps moving. The three options trade against each other — a longer interval tolerates slower batches, smaller batches bound the work per cycle, and worker threads decouple processing from polling at the cost of more moving parts. 4. Rebalance Storms Symptoms: The group spends more time reassigning partitions than processing records. Cause: Pods scaling up and down, rolling restarts, short timeouts, and an eager assignor that revokes everything on every change. A rebalance is not free: assignments pause while partitions move, in-flight work gets discarded or replayed, and a group that rebalances in a loop spends its capacity on coordination instead of consumption while lag climbs behind it. Fix: Use cooperative-sticky assignment, tune the HPA carefully, and stabilize the pod lifecycle — no random restarts. Cooperative assignment revokes only the partitions that actually need to move, and sticky ownership keeps the rest where they are, so a single pod event stops being a group-wide interruption. Autoscaling against a consumer group needs that softness underneath it, because every scale event is a membership event. 5. Offset Commit Failures Symptoms: CommitFailedException in the logs; records reprocessed after a restart. Cause: A stale generation after a rebalance, or the coordinator moved or is unreachable. A commit is a claim about progress made under a specific generation of the group, and the coordinator rejects claims from a generation that no longer exists — the consumer processed the records, and the group has already moved on without recording that fact. Fix: Commit only inside the poll loop, handle CommitFailedException and retry, and use commitAsync with callbacks carefully. Committing inside the loop keeps the commit attached to the generation that did the work. The reprocessing that follows a rejected commit is the group working as designed; the goal is making replays safe and rare. 6. Resource Limits — CPU and Memory Symptoms: High CPU throttling, long GC pauses, OOMKilled pods, missed heartbeats. Cause: Limits set too low for the batch sizes in use; JVM heap not matched to container memory. This mode sits underneath several of the others — throttling starves the heartbeat thread of mode 2, stretches the batch processing of mode 3, and turns the memory pressure of mode 1 into an OOMKill. A consumer's batch size is also a memory decision, since every record in the batch lives in the heap until the batch finishes. Fix: Set requests and limits thoughtfully, tune the JVM (-Xmx against container memory, not host memory), right-size message batches, and watch it all with Prometheus and Grafana. The host-memory mistake is the subtle one: a JVM sized for the node it lands on will be killed by a container limit it never knew existed, and the pod restart that follows presents as a consumer problem with a platform cause. 7. DNS and Bootstrap Servers Symptoms: UnknownHostException; the consumer cannot connect at startup; intermittent failures after that. Cause: Wrong bootstrap addresses, CoreDNS issues, or private DNS without the VPC setup to back it. This is the one mode on the list that fails before the group protocol is even involved — the consumer cannot reach a broker to join anything — and the intermittent version is worse than the hard failure, because it presents as flakiness instead of a name-resolution problem with a fixed address. Fix: Use the correct bootstrap DNS names, verify CoreDNS in the EKS cluster, and check VPC DNS settings and private hosted zones. Every layer of that chain is platform surface, and all of them have to agree before the consumer's own configuration gets a chance to matter. 8. Ungraceful Shutdown Symptoms: Every deploy triggers a full rebalance; partitions get reassigned the slow way. Cause: Kubernetes kills the pod with no chance to leave the group or commit offsets. From the group's side, a terminated member and a crashed member look identical — both simply stop responding — so the group waits out the session timeout and then reassigns everything, on every single deploy, even though the shutdown was planned in advance. Fix: Add a PreStop hook that calls close(), commit offsets before shutdown, and set terminationGracePeriodSeconds generously enough for the close to finish. A clean close() is the consumer announcing its departure: the group rebalances immediately and narrowly, offsets land before the pod dies, and a deploy stops looking like an outage. The grace period is the budget that makes the announcement possible, and it has to be long enough for the close to actually complete. Designing for the Platform None of these eight modes is exotic, and that is the point. Read together, they reduce to a short list of design commitments. Timing gets budgeted as a first-class resource, sized for a platform that throttles and pauses things rather than a quiet laptop. Shutdown becomes a feature with its own code path instead of an abrupt disappearance the group has to time out. Identity stays as stable as the platform allows, so sticky assignments have something to stick to. And processing is written to survive replay, because replays are the group's recovery mechanism and no configuration makes them fully disappear. The tradeoffs stay honest. Generous timeouts tolerate platform noise and also delay detection of a genuinely dead member. Cooperative-sticky assignment calms rebalances and still cannot hold a partition whose owner is gone. Bigger resource limits end throttling and cost money on every pod, every hour. The design point is choosing these tensions deliberately, not eliminating them. Debug Fast Shell kubectl get pods -n kafka-consumers kubectl logs <pod> kubectl get events kafka-consumer-groups.sh --describe --group <group> Lag plus member state tells you which issue you are in. Name the issue first; the fix follows from the name. The pod list and events stream show the platform's half of the story — restarts, kills, throttling — and the group description shows the protocol's half: who owns what, how far behind each partition sits, and whether membership is stable or churning. What Changes in Practice With the baseline in place and the eight modes handled, the operational picture changes in specific ways. Deploys stop triggering full rebalances, because the shutdown path announces the departure and the group reassigns narrowly. Restarts replay records safely instead of corrupting progress, because commits are manual and processing is idempotent. Timing stops being the hidden cause behind pages that present as broker problems. And diagnosis gets faster: lag and member state from kafka-consumer-groups.sh --describe identify which of the eight modes is active in minutes, and the fix follows from the name. Takeaways On EKS, the platform is part of the consumer's failure model: restarts, throttling, and shutdowns are normal events, not exceptions.Most consumer pages trace back to timing — heartbeats, poll intervals, and shutdown grace — not to broker problems.Cooperative-sticky assignment and manual commits remove the two loudest sources of churn before tuning even starts.A graceful shutdown path is a feature: PreStop hook, a final commit, and a grace period long enough to use them.Debug by symptom against the eight modes above; lag and member state from kafka-consumer-groups.sh --describe identify the mode in minutes. What's Next The same failure-model thinking extends to every stateful client the platform hosts. Any process that joins a group, holds an assignment, and proves liveness on a cadence will meet its own version of these eight modes, and the fixes generalize with it: budget the timing, announce the shutdown, keep the identity stable, and make the work replayable. The consumer is simply where the collision shows up first, because the group protocol writes its assumptions down in configuration — and configuration can be reviewed line by line. The platform is part of the failure model. Design for it there, in the open, and the pages stop.

Git plays a key role in modern software development. However, most Git GUIs still focus on the same basic operations: stage, commit, branch, merge, pull, and push. These actions matter, but they cover only part of what developers do every day. When working on a real project, developers also need to answer questions such as: What changed across several repositories?When did this piece of code or configuration exist?How do I compare two versions?Where did a particular change disappear?How do I review a change without switching between several tools? Gitember is an open-source desktop Git client designed to address these issues while supporting the full cycle of daily Git workflows. The most interesting aspect here is not just another interface for executing git commit, but the functionality built around Git. Of Course, Gitember Is a Complete Everyday Git Client Gitember covers the typical Git workflow, focusing on the following areas: Repository and remote operations – cloning/initializing, fetch, pull, push, working with remote repositories, and Git LFS.Changes and branches – staging changes, creating commits, editing and undoing commits, branch management, merging, rebasing, and interactive rebasing.History and repository management – commit history, modified files, comparing versions and branches, reflog, Git notes, worktrees, and submodules.Conflict resolution and security – stash management, three-way merging, commit signing, and signature verification. The goal is simple: day-to-day Git work shouldn't require constantly switching between the command line, a diff viewer, a merge tool, and a browser. However, it is not these features that make Gitember particularly interesting. Workspaces: Working With Several Repositories as One Project Modern applications are often distributed across multiple repositories. A single product might consist of a frontend and backend, a set of services, APIs and their implementations, and so on. Git treats these as independent repositories, which is the right approach from a version-control perspective. The challenge arises because developers often view them as a single, unified system. Workspaces in Gitember provide an abstraction layer over individual repositories. A workspace allows you to group multiple independent Git repositories and view their aggregate state. Git operations — such as pull, push, fetch, and managing branches and commits — can be performed across all these repositories simultaneously. It is important to note that this does not turn them into a monorepo; each repository retains its own history and remote configurations. There are also deliberate limitations: operations within a workspace are executed sequentially, and a commit spanning multiple repositories is not an atomic transaction. Thus, a workspace serves more as a coordination mechanism than a distributed transaction system. Submodules are supported as well. Search Git History by Content, Not Only by Commit Metadata Typically, searching Git history is done by commit messages, authors, dates, or file paths. But in most cases, you remember the content, not the commit itself. For example: “There was a configuration value somewhere around two years ago.” Or: “At some point we had this API endpoint.” Or: “Which version contained this particular text?” The Extended Search function in Gitember indexes the selected repository history and allows searching for file content in the history. This changes the workflow from: Find the commit and then inspect its contents. to: Find the content and then navigate to the commit. Search is not limited to source code. The documentation describes searching within the content of documents in various supported formats, including Office files, CAD files, archives, audio recordings, and other materials. The Git history effectively transforms into a searchable array of historical data. Integrated Diff and Merge Workflow Another common developer workflow involves jumping between several applications: Git client -> diff tool -> merge tool -> Git client Gitember tries to keep this workflow together. It provides: side-by-side diffsversion comparisonbranch comparisonfile and folder comparisonvisual three-way merge The three-way merge editor is particularly useful when conflicts need to be understood rather than simply resolved by choosing “ours” or “theirs.” The file and folder comparison function can be used independently. Repository Statistics The Git commit graph shows exactly what happened. Statistics, however, help answer other questions: How active is this repository?How has development activity changed over time?Who is making the biggest contribution?What does the repository's history look like in terms of trends? Gitember includes tools for analyzing statistics, allowing you to view the repository not just as a collection of Git objects, but as a dataset regarding development activity. This is particularly useful when exploring an unfamiliar repository. Instead of immediately browsing through hundreds of commits, you can first get a general overview of the project. Local AI Instead of Another Cloud Dependency Gitember includes experimental AI-powered features that utilize locally running large language models (LLMs). The described use cases include: Detecting potential secrets in staged files;Generating descriptions of differences between branches;Creating commit messages. A key architectural feature is that the AI model runs locally. This means that using the AI does not require adding another mandatory cloud service to the developer's workflow. This feature is intentionally positioned as an auxiliary tool rather than a replacement for comprehensive security solutions. Why Build This in Java? Gitember is a Java desktop application using Swing and JGit. One reason this is practical is that the Java ecosystem already provides many of the building blocks required for this type of application: JGit, mature search/indexing technologies, document extraction libraries, etc. The result can be a fairly self-contained developer tool rather than a thin GUI that depends on a collection of external applications. There is also an interesting misconception about Java desktop applications and memory consumption. Java is not automatically the biggest memory consumer simply because it is Java. Many modern desktop developer tools are Electron-based and bring a browser runtime with them. In practical measurements, Gitember can consume less memory than GitHub Desktop despite providing a broader set of Git functionality. The important metric is therefore not simply the implementation language, but how efficiently the complete application performs its workload. Location Gitember is open source under GPL-3.0 and available for Windows, macOS, and Linux. Project: https://gitember.orgSource code: https://github.com/iazarny/gitember

AI is changing the way we code. Last week, a friend told me he completed a user story to add new columns to a table and tested it in under an hour. Upstream teams are now shipping changes in days that once took months. For data pipelines that sit downstream of these fast-moving systems, this speed is a new design constraint. One of the potential solutions is to build dynamic data pipelines that are built to keep up. What Is a Dynamic Data Pipeline Dynamic data pipelines are data processing workflows that adapt to changing data and business logic. They use parameterized templates and config files to run custom pipelines on the go as per data requirements. These pipelines are better suited to today’s fast-changing data environment. staticdynamic Hard-coded, fixed tasks Configuration-driven, flexible logic Breaks on schema changes Adapts automatically to schema changes Requires coding and redeployment to handle business logic changes Requires just config changes to handle business logic changes High maintenance with constant updates to prevent failure Low maintenance, may require some config changes from time to time Quick to build the pipeline Takes more time upfront for initial coding but once built, it’s easy to add more tasks Sample Dynamic Data Pipeline Architecture I'll walk you through a sample architecture to show how dynamicity is configured into a data pipeline. Sample dynamic data pipeline architecture 1. Configuration The key configurations of a dynamic data pipeline can be stored in Git (YAML/JSON) and a database. Git stores the pipeline logic, and a PostgreSQL/MongoDB database stores the dynamic values used when the pipeline runs. For example, you may want to build a standardized data ingestion workflow; the logic for it will be in Git, but the list of tables to be ingested can be stored in the database. So when an ingestion workflow initiates, it triggers multiple workflows for each table entry. Let’s understand it with code: YAML config of dynamic data ingestion pipeline: YAML pipeline: name: dynamic_data_ingestion_pipeline version: “1.0” source: type: "{{ source_type }" connection: host: "{{ source_host }" database: "{{ source_database }" table: "{{ source_table }" schema: on_new_column_detected: “add” # automatically add new columns on_column_removal_detected: “ignore” # ignore column removal on_type_change_detected: “cast” # cast the datatype on change transformations: name: “if_filter_null” enabled: “{{ if_filter_null }” column: “{{ null_filter_column }” target: type: “{{ target_type }” dataset: “{{ target_dataset }” table: “{{ target_table }” runtime: schedule: “{{ cron_schedule }” retry_attempts: 3 alert_on_failure: true Database entry of dynamic data ingestion pipeline: Markdown -- Table: pipeline_config | id | source_type | source_host | source_database | source_table | if_filter_null | null_filter_column | column_mapping | target_type | target_dataset | target_table | cron_schedule | |----|-------------|------------------------|-----------------|--------------|----------------|--------------------|----------------|-------------|----------------|-----------------------|---------------| | 1 | postgres | db.customers.prod.com | customers_db | customers | true | customer_id | true | bigquery | analytics | customers_details | 0 * * * * | | 2 | s3 | s3.amazonaws.com | transactions_db | transactions | false | null | true | redshift | finance | transactions_details | 0 0 * * * | These configurations are what make the pipeline dynamic. Instead of hardcoding column names or schema expectations into the pipeline code, the script reads the schema at runtime and adapts automatically, handling new, removed, or renamed columns without breaking. And whenever any changes happen upstream, all that's needed is a config table update. No pipeline rework, no redeployment. Tip: Make sure the database fetch step for dynamic pipelines happens during execution inside a task, not during DAG parsing. Because DAG parsing occurs on every scheduler heartbeat, it can crash your orchestrator. 2. Orchestrator After those configurations are ready, you can design the dynamic pipeline in an orchestrator (such as Airflow, Prefect, or Dagster). It generates pipelines based on configurations defined in YAML/database and dynamically creates the required tasks. Sample flow in orchestrator: Plain Text Reads YAML template ↓ Loops over DB config table ↓ One sub DAG per row with all tasks Dynamic DAG skeleton in Airflow: Python with open("/opt/airflow/configs/pipeline_config.yaml", "r") as f: raw_yaml_template = f.read() def get_pipeline_settings(): connection = psycopg2.connect( host="company_db.com", database="metadata_db", user="sys_admin", password="******" ) cur = connection.cursor() cur.execute(""" SELECT id, source_type, source_host, source_database, source_table, if_filter_null, null_filter_column, target_type, target_dataset, target_table, cron_schedule FROM pipeline_config """) rows = cur.fetchall() field_names = [col[0] for col in cur.description] cur.close() connection.close() return [dict(zip(field_names, row)) for row in rows] all_pipeline_settings = get_pipeline_settings() all_configs = [] for pipeline_settings in all_pipeline_settings: resolved_yaml = Template(raw_yaml_template).render(**pipeline_settings) pipeline_config = yaml.safe_load(resolved_yaml) all_configs.append(pipeline_config) def task1 (): #logic def task2 (): #logic def task3 (): #logic task1= PythonOperator( task_id="task1", python_callable=task1, dag=ingestion_dag ) task2= PythonOperator( task_id="task2", python_callable=task2, dag=ingestion_dag ) task3= PythonOperator( task_id="task3", python_callable=task3, dag=ingestion_dag ) # ========================================================== # DAG FLOW # ========================================================== task1>> task2>> task3 3. Executor Your pipeline logic will usually run on compute engines such as Apache Spark, BigQuery, Snowflake, or Presto/Trino, depending on your infrastructure and pipeline configuration. The output will be stored in cloud systems such as Amazon S3, Google Cloud Storage, or Azure Data Lake Storage. 4. Monitoring Depending on your environment, built-in logging platforms are available, such as AWS CloudWatch, Google Cloud Monitoring, or Azure Monitor. For advanced monitoring, tools like Logstash and Elasticsearch help to aggregate logs and perform analytics. Should Every Data Pipeline Be Dynamic No, not every pipeline needs to be dynamic. It will become very tough to maintain all dynamic pipelines. And then it doesn’t make sense to write this logic for every workflow. In some cases, explicit hardcoded logic is actually simpler to read and maintain. Dynamic pipelines are a good choice if: You have 20+ pipelines with similar logic that differ only in a few configuration valuesYour data source schema changes frequentlyYour pipelines have frequent business rule changesYou can handle pipeline failures in isolation, meaning one bad config cannot break all pipelines at once.You can easily monitor DAGs even after the shift to dynamicYou have non-engineers such as analysts, PMs, or business users who need to make changes to DAG configurations Best Practices to Maintain Dynamic Data Pipelines Dynamic data pipelines are still a relatively new concept. Here are some best practices to help you maintain them. Avoid Doing Too Much in One DAG One single DAG generating 1000s of sub-DAGs will be a maintenance nightmare. Also, if it’s hitting any API or shared services, it can cause overload. Limit how many sub-DAGs one parent DAG can initiate. How to implement this? Group pipelines based on business function or sources. For example, you can have one parent DAG for all tables from the same source. Alternatively, you can group pipelines by business function, such as marketing, finance, or reporting. Maintain Version History Given the dynamic nature of sources and pipelines, schemas and sources can change over time. The DAG must maintain version history, and any backfill must respect the configuration that was active when the partition originally ran. How to implement this? Always store the configuration associated with each run so you can trace it for any backfills. You can also design your pipeline config table to maintain history by adding "valid_from" and "valid_to" columns with the exact timelines when that config was active. Monitor at Sub-DAG Level Dynamic pipelines are essentially sets of tasks grouped together and generated multiple times based on configuration. For example, an ingestion pipeline is a set of tasks to extract, transform, and load data for different tables. Monitoring must be done at the sub-DAG level rather than the individual task level. That means ingestion must be tracked separately for each source. How to implement this? Maintain separate logs and tracking for each sub-DAG. For example, you can push a new entry for every sub-DAG with metadata like source name, config ID, row count, and duration to a monitoring table or tool like Datadog or Grafana. One Sub-DAG Should Not Block the Rest A dynamic pipeline can trigger hundreds of sub-DAGs, but each sub-DAG must operate as an independent entity. A failed or skipped sub-DAG should not affect other sub-DAGs generated from the same pipeline. How to implement this? Set failure rules at the sub-DAG level to prevent one failed task from blocking the rest. For example, configurations like on_failure_callback in Airflow can be set at the sub-DAG level to isolate failures and let the rest of the sub-DAGs run fine. Dynamic Data Pipelines Are No Longer a Nice-to-Have With AI, companies will soon be building high-velocity products. That means new features will get added at a pace that leaves rigid downstream systems struggling to keep up. The pipelines that run smoothly won't be the ones built for today's schema/rules. It’ll be the ones built to adapt to whatever comes next.

Most teams treat a broken web scraper as a chore. Someone notices bad data, opens the page in a browser, patches a selector, and redeploys. That works when you have one scraper and someone who looks at its output every day. It stops working when you have a dozen, and the breakage surfaces a week later as a strange number in a downstream report. In this article, I walk through how I built scrapers that are cataloged, health-checked, traced, and repaired like any other production service, with an LLM proposing fixes and plain Python deciding whether those fixes ship. The project came out of the WeMakeDevs hackathon, where it took first place. The product itself is small: it pulls about twelve days of TV guide data across three streaming providers, ranks the top ten upcoming movies and top ten upcoming shows, and lets you record any of them on Philo with one click. I am going to skip the product and focus on the plumbing: the drift rules, the state model, the verifier, and the code behind each. Those pieces carry over to any team that runs scrapers, or to any team that lets an LLM modify a live system. The architecture follows one split. Diagnosis is probabilistic; acceptance is not. The LLM figures out what broke and proposes a repair. A deterministic diff against a frozen sample decides whether that repair is correct, and nothing gets promoted without passing it. Why Do Scrapers Fail, and Why Old Fixes Are Not Enough The usual way to build a scraper is with structural selectors. Someone opens the page, finds that the airtime lives in div.card > span:nth-child(3), and hard-codes it. This is fragile by nature because it describes where something is, not what it is. Rename a CSS class and it fails. Reorder elements and it fails. Wrap the time in a new tag and it still fails. There is a newer, AI-based approach: you write what the data means, and a model finds it in the webpage DOM. For my project, I tried Bright Data's Scraper Studio, which creates a collector from a natural-language prompt. When you build the channel-schedule scraper, the prompt example reads like the following: Python PROMPT = ( "This page lists a TV channel's upcoming schedule as program cards. Extract every " "program card. For each: the program title as title, the episode title if shown as " "episode_title, the airing date exactly as displayed (like Sat, Aug 22) as date_raw, " "the start time exactly as displayed (like 2:00 PM) as start_raw, the time range text " "if shown (like Sat, Aug 22 2:00 PM - 2:30 PM) as time_range_raw, the category text " "as category. Include every card.") Notice what is not in there: no class names, no element positions, no DOM paths. The prompt describes everything. That matters for two reasons. First, a semantic description survives a redesign far better than a selector does, and second, this is the part that took me a while to appreciate: the prompt is the healing artifact. When the scraper drifts, the repair tool reasons over that same description plus a plain-language symptom report. A prompt that said "the third span inside each grid cell" would give the healer nothing to work with. Also notice date_raw and start_raw. I always capture the raw displayed string alongside any parsed value. If start_raw is "8:00 PM" but start_utc parsed to null, that is not a data quirk. That is a drift signal, and one of the detection rules below keys on exactly that. AI extraction alone does not solve the problem, though. It reduces the breakage rate, but it cannot eliminate it, and it introduces a new risk. When you ask a model to repair a scraper, how do you know the repair is correct? A model will confidently produce output for a broken extraction. Two rows that look plausible are still two rows when there should be thirty. Scrapers as Cataloged Services With Health States The first design decision was to stop treating scrapers as scripts and start treating them as services with an on-call rotation. Each scraper is a cataloged entity in an internal developer portal in Port IO, another vendor in the hackathon, with a health property that moves between healthy, drifting, and broken. Every scrape run is its own entity with a status of success, drift, or error. Every repair attempt is a heal_event with its trigger being the drift description. Once scrapers are entities with state, the rest of the platform falls out. Workflows react to entity changes. A scrape_run landing in success triggers publish. A scrape_run landing in error or drift triggers heal. Scheduled workflows kick off the daily scrape at 08:00 UTC and the catalog refresh at 07:30. A scorecard grades each scraper from Basic through Bronze, Silver, and Gold based on whether it verified today, whether verification is passing, and whether it is currently healthy. An operator dashboard shows all the scrapers by health, the recent health events, and the current ranked output. None of this is new. It is what you would do for any microservice. The point is that scrapers rarely get this treatment, and they need it more than most services because their upstream is a third party that ships changes without telling you. Detecting Drift Deterministically Before you can repair anything, you have to notice what is broken, and you have to notice it without a human staring at the output. I wrote seven deterministic drift rules that run on every scrape in plain Python, with no model involved. The thresholds live in one place: Python REQUIRED_FIELDS = ("channel", "title", "start_utc") NULL_RATE_LIMIT = 0.10 EMPTY_PAGE_LIMIT = 0.20 # >20% of channel pages returning nothing is breakage CHANNEL_COLLAPSE = 0.80 # <80% of the requested channels yielding rows is breakage LINEUP_DELTA_LIMIT = 0.10 # >10% channel-set churn needs two confirming runs The row band sets how many rows a single channel page should yield. A page returning 3 cards for a channel that airs around 40 programs a day is broken, not news; so is one returning thousands. Python PAGE_ROW_BANDS = (15, 800) Alongside the row band, the rules check the null rate of required fields, the case where a raw time is present but the parsed time is null, channel collapse (fewer than 80 percent of requested channels yielding rows), the empty-page rate, a change in the schema hash of returned keys, and lineup disappearance. That last one deserves a note because it exposes the hardest problem in drift detection. A broken scraper and a legitimately changed source look identical in a single sample. If 15 percent of channels vanish from a provider's lineup, that could be a scraper failure, or it could be a real carriage change, which happens regularly with streaming lineups. I handle it two ways. Lineup churn requires two consecutive runs agreeing before it is accepted as real. A single run showing the change is flagged as unconfirmed drift. If the scraper still reproduces the expected output from a frozen HTML page, the scraper is fine, and the source actually changed. Python delta = _lineup_delta(prev["rows"], rows) if delta > LINEUP_DELTA_LIMIT: signature = f"lineup:{_channel_sig(rows)}" seen = state.confirm_change(state_key, signature) if seen < 2: drift.add("lineup_change", f"channel set moved {delta:.0%} (unconfirmed, run {seen}/2)") One problem I encountered: a single collector serves all three providers because every channel page on the guide site has the same structure. But drift state has to be keyed per collector:provider pair because comparing one provider's channel set to another provider's snapshot reads as mass channel loss. When drift is detected, two things happen. The pipeline immediately sets the scraper's health to drifting in the catalog (the fast path, within seconds). It also increments a scraper.drift.detected metric that an alert rule in the observability stack (SigNoz, the third hackathon sponsor) watches, firing a webhook roughly five minutes later (the safety net, and the only path that catches a pipeline that died entirely and never ran the inline detector). Both paths hit the same idempotent /heal endpoint. Whichever fires first does the work; the other finds the scraper already healthy and exits. The Repair Loop and the Gate Here is the heal flow, with the part I care most about in the middle: Plain Text /heal → scraper heal (semantic symptom description) → preview_result | Verifier: diff vs frozen fixture | ┌───────────┴───────────┐ pass fail | | approve reject | | healthy broken | human queue The heal itself is delegated. Bright Data's scraper /heal command takes a collector ID and a plain-language description of what broke, reasons over the original semantic prompt, and returns a proposal in an awaiting_approval state with a preview_result of what the repaired scraper would extract. In my system, a Triage agent in the developer portal authors that symptom description by reading the run and heal history. That is judgment, and I was skeptical of handing it to a model, so I decided to introduce a verifier. The verifier is deliberately not an agent: Python """The Verifier -- deliberately NOT an agent. Judgment goes to the LLM agent; proof stays deterministic. A verification gate you cannot trust deterministically is not a gate, so this is plain code: normalise both sides, diff, return a verdict. It is the only thing allowed to authorise a promotion. """ COMPARE_FIELDS = ("title", "date_raw", "start_raw") def verify(provider: str, rows: list[dict]) -> Verdict: want, got = _normalise(expected(provider)), _normalise(rows) if want == got: return Verdict(True, []) diff = [] missing = [w for w in want if w not in got] extra = [g for g in got if g not in want] if len(want) != len(got): diff.append(f"row count {len(got)}, expected {len(want)}") for item in missing[:3]: diff.append(f"missing {item}") for item in extra[:3]: diff.append(f"unexpected {item}") return Verdict(False, diff) It normalizes both sides (strip, lowercase, round floats), sorts, and compares. If the diff passes, the healer approves the proposal and sets health back to healthy. If it fails, the healer explicitly rejects the proposal, sets health to broken, writes a heal_event with outcome rolled_back, and leaves the scraper in a human's queue. The fixture that backs this is three files: channel.html, a real captured channel page trimmed to 30 program cards; channel.expected.json, the hand-checked correct output for that exact page; and channel.mutated.html, the same 30 programs with every structural hook deliberately broken. Classes are renamed, card order is shuffled, and airtimes sit inside extra wrapper elements. That file lets the tests and the demo trigger drift on demand. A selector-based scraper fails on it; a semantic one should not. I enforce two rules about fixtures, and I wrote them into the repo's agent instructions so the coding agent working on the codebase would respect them too. Fixtures are frozen inputs (hardcoded schema) never regenerated to make a test pass, and a rejected bad repair is a success, not a failure. That second one is counterintuitive until you think about what the alternative is. A repair that gets promoted without proof is exactly how a broken scraper silently poisons everything downstream. Walking Through a Rejected Repair The case this design exists for is a repair that looks fine and is wrong. Here is the state sequence from a real end-to-end run where that happened: A fault-injected run produced a scrape_run entity with status error.The event workflow fired, and the Triage agent wrote a drift description from the run and heal history.The /heal endpoint sent that description to the vendor and got back a proposal in awaiting_approval, with a preview_result.The verifier diffed the preview against channel.expected.json and returned row count 2, expected 30.The healer rejected the proposal, set health to broken, wrote a heal_event with outcome rolled_back, and queued it for a human.The next clean scheduled run passed every drift rule and moved health back to healthy. The proposal in step 3 was well-formed and confidently produced. Without step 4, it would have been promoted as the fix, and the ranking would have been built from 2 programs instead of 30 while the system recorded the repair as a success. Observability To make every heal cycle inspectable after the fact, the whole pipeline runs under OpenTelemetry auto-instrumentation with traces, metrics, and logs shipped to SigNoz (self-hosted locally and exposed over ngrok). The trace for a scrape run has one scrape.fetch span per channel page (tagged with provider, channel ID, row count, and fetch mode) followed by scrape.extract, scrape.validate, and when relevant scrape.heal.reprompt and scrape.verify.fixture. Because the FastAPI service itself is instrumented, even the developer portal's automation calling /heal shows up as a span. The drift description, the heal prompt, and the verifier's diff are all logged inside the relevant span, so trace_id is attached automatically. The developer portal's own action and workflow runs, and the scraping vendor's remaining budget, are not things those systems emit anywhere, so a small background watcher samples both and re-emits them as metrics. Runs are de-duplicated in SQLite so a restart never double-counts. Applying the Same Split to Shipping Code Once the pattern worked for scraper repair, I applied it to the software change loop, too. A feature request submitted through the portal becomes a requirement entity, which triggers a local coding agent (Claude Code, running on my machine with file tools only) to edit a scratch clone. Then deterministic code takes over: branch, commit, push, open a PR with a fixed first-line contract that the deploy workflow parses later. GitHub Actions runs pytest, a TypeScript type check, and the Next.js build. A human reviews and merges. Only then does the deploy workflow ship it and flip the requirement to deployed. The agent is forbidden from touching fixtures, the verifier module, and the drift thresholds, and the run aborts if the resulting diff touches any of them. The model gets to write the feature. Implementation Checklist If you build something similar, these are the details that mattered most in practice: Store start_raw next to start_utc, and follow the same pattern for every parsed field. A raw value that is present alongside a null parsed value was the earliest and most dependable drift signal in this build. If one scraper serves multiple providers, key drift state by collector:provider, not by collector. Otherwise, one provider's channel set gets compared to another's snapshot and reads as mass channel loss. Route the inline detector and the alert webhook to the same /heal endpoint, and make it a no-op when the scraper is already healthy. With two triggers, duplicate calls are a certainty, not an edge case. Require two consecutive runs to agree before accepting a lineup change as real. A single sample cannot tell a broken scraper from a changed source. Write the "fixtures are frozen" rule down where both humans and coding agents will read it. The temptation to regenerate expected output to make a red test green is strong, and an agent will do it without hesitation if you have not told it not to. Draw a hard line between diagnosis and acceptance. Diagnosis can be fuzzy. Acceptance should be code short enough to read on one screen and trust completely. Limitations To be fair about what this is and is not, the factory runs on the operator's machine, reached from the cloud through a tunnel, so the portal operates it but does not host it. The catalog scraper has no fixture because the catalog legitimately changes daily. There is no retry or backoff on vendor calls. And the Triage agent's reasoning over long run histories is shallow. It authors a good symptom description, but it is not yet doing root-cause analysis. But the core loop — detect deterministically, repair with judgment, promote only on proof — has now survived both the happy path and a real rejected repair, and I think it is the right shape for anyone letting an agent touch production data.

There is a moment during a rolling update when everything looks fine. The old pods are draining. The new pods are coming up. The readiness probes are passing. The deployment controller is happy. You watch the replica counts shift in your terminal and think this is working exactly as designed. Then your on-call phone buzzes. In a 5G signaling deployment I worked on, this happened during what should have been a routine update. Every infrastructure signal was green. The Kubernetes event log was clean. The new pods passed their readiness checks within a few seconds of starting up. Session establishment failures jumped 18%. The pods were running. They just weren’t ready. And our probes had no way to tell the difference. The Problem With HTTP Probes Readiness probes are one of Kubernetes’ better ideas. Before they existed, you’d route traffic to a pod and hope the application had finished initializing. Probes let you say: Don't send traffic until the pod tells you it’s ready. The default implementation looks something like this: Plain Text readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 Kubernetes hits /healthz. Gets a 200. Marks the pod ready. Routes traffic. For a stateless web service, this is fine. The HTTP endpoint responding and the application being ready to handle requests are essentially the same thing. Load a config file, spin up a thread pool, and you’re good. The assumption breaks the moment your application has to integrate with something before it can serve real traffic. What Integration Actually Means In telecom environments, network functions don’t just start — they register. A SIP proxy needs to complete handshakes with upstream session border controllers. A Diameter node needs to establish peer connections with authentication servers. A GTP function needs to synchronize state with its neighbors. These aren’t instantaneous. Depending on the environment, full registration can take anywhere from ten seconds to over a minute after the process has started. During that window, here’s what happens: Pod starts./healthz returns 200 (process is up).Readiness probe passes.Kubernetes routes traffic to the pod.Pod receives a request that requires upstream registration.Registration isn’t complete yet.Request fails. From Kubernetes’ perspective, everything worked. The probe passed. Traffic was routed. From your user’s perspective, their session failed. The problem isn’t telecom-specific. Any of these patterns hit the same issue: gRPC services that need to warm connection pools before handling loadMessage queue consumers that need to establish broker connections and claim partitionsDatabase-backed services that need to run migrations or seed connection poolsService mesh participants that need sidecar handshakes before traffic can flow The HTTP probe can only tell you the first thing. It says nothing about the second. What the Incident Actually Looked Like In the 5G deployment, the readiness probe was checking a lightweight internal endpoint that returned 200 as soon as the main process thread started. Two seconds after startup, at most. What it wasn’t checking: whether the pod had completed SIP registration with the upstream proxy and re-established session state with the session management function. That took closer to thirty seconds. Sometimes longer under load. So for that thirty-second window during each pod’s rollout, we were routing live signaling traffic to pods that were alive but unregistered. Requests that hit those pods during registration would fail. The failures were silent—no crash, no obvious error, just session establishment attempts that didn’t complete. The 18 percent failure rate during the update window wasn’t from a bug in the new code. It was from routing traffic to pods before they were actually ready to handle it. The first time it happened, it resolved on its own, and we wrote it off as a fluke. The second time, it happened during a peak traffic period, and the impact was hard to ignore. The third time never happened because we’d fixed the probe by then. How to Fix It The fix is to replace the HTTP probe with something that checks actual readiness — not process readiness, but integration readiness. For the signaling service, that meant an exec probe that called a script checking registration state: YAML readinessProbe: exec: command: - /bin/sh - -c - "check-registration-status.sh && exit 0 || exit 1" initialDelaySeconds: 15 periodSeconds: 5 failureThreshold: 18 The script did two things: it verified that the SIP registration with the upstream proxy had been completed and confirmed that the session management peer connection was established. Until both returned healthy, the pod stayed out of rotation. The tuning matters: initialDelaySeconds: 15 – gives registration time to begin before Kubernetes starts checkingperiodSeconds: 5 – checks frequently enough to catch completion quicklyfailureThreshold: 18 – allows up to 90 seconds for registration to complete before Kubernetes gives up and restarts the pod For non-telecom workloads, the same pattern applies. A gRPC service might check whether its connection pool is warmed: YAML readinessProbe: exec: command: - /bin/sh - -c - "grpc_health_probe -addr=:9090 && pool-ready-check.sh" initialDelaySeconds: 10 periodSeconds: 5 failureThreshold: 12 The principle is always the same: don’t ask Kubernetes to decide your application is ready based on evidence that doesn’t reflect what ready actually means for your workload. One More Thing Worth Knowing If you fix your readiness probe and your rolling updates get slower, that’s correct behavior—not a regression. Before the fix, pods were being marked ready before they actually were. Updates appeared fast because Kubernetes was moving quickly through the rollout. After the fix, updates take longer because Kubernetes now waits for real readiness before routing traffic. The slowdown is the probe working. Your update velocity was an illusion. The Short Version If your readiness probe is an HTTP check against a /healthz endpoint, ask yourself what that endpoint actually knows about your application’s readiness state. If the answer is "it knows the process started," you have a process-up probe, not a readiness probe. The distinction matters only when something goes wrong during a rollout — exactly when you need your probes to be accurate. Related Research The production autoscaling findings referenced in this article informed peer-reviewed research on workload-aware Kubernetes autoscaling for telecom infrastructure, published in IEEE Xplore: https://ieeexplore.ieee.org/document/11501289