DZone Spotlight

I've spent years fixing broken Glue jobs and Lake Formation permissions as an AWS support engineer, then moved to pre-sales architecture for banking data platforms. One pattern shows up in almost every migration: teams build IAM policies and a data catalog as two separate workstreams, run by two separate teams, that never talk to each other. Then six months post-launch, someone asks, "Who can see the SSN column in the gold layer?" and nobody can answer without a week of manual auditing. That gap isn't a process problem. It's an architecture problem. This article walks through why classification and access control need to be the same system, with the actual policy constructs to make that work on AWS, and where teams get this wrong. The Failure Mode A typical lakehouse migration goes like this: Data engineering builds the pipeline: source → Bronze → Silver → Gold.A separate governance initiative (often 6-12 months behind) builds a catalog: Collibra, DataHub, Purview, whatever.Security writes IAM policies against S3 prefixes and Glue databases — because that's what exists at go-live. The catalog knows column cust_ssn is PII. IAM doesn't. So access control is granted at the table or prefix level, which means anyone with table access sees every column, including the ones the catalog explicitly flagged as sensitive. The classification work becomes documentation instead of enforcement. This is exactly the gap that tag-based access control was built to close, and most teams don't use it because it requires governance and security to share a data model, which is an org problem more than a technical one. Tag-Based Access Control: The Actual Mechanism On AWS, Lake Formation's LF-Tags let you attach classification metadata directly to catalog resources — databases, tables, even columns — and write IAM-style policies against the tags instead of the ARNs. Tagging a column as sensitive: Shell aws lakeformation add-lf-tags-to-resource \ --resource '{ "TableWithColumns": { "DatabaseName": "credit_risk_gold", "Name": "customer_360", "ColumnNames": ["cust_ssn", "cust_dob"] } }' \ --lf-tags '[{"TagKey": "data_classification", "TagValues": ["pii_restricted"]}]' Granting access by tag, not by table name: Shell aws lakeformation grant-permissions \ --principal '{"DataLakePrincipalIdentifier": "arn:aws:iam::111122223333:role/RiskAnalystRole"}' \ --permissions "SELECT" \ --resource '{ "LFTagPolicy": { "ResourceType": "TABLE", "Expression": [ {"TagKey": "data_classification", "TagValues": ["public", "internal"]} ] } }' The RiskAnalystRole above gets SELECT on anything tagged public or internal — automatically, across every current and future table with that tag. It gets nothing on pii_restricted columns unless a separate grant explicitly adds it. New tables inherit policy from their tags at creation time, not from a security team manually reviewing each new dataset. Databricks Unity Catalog does the equivalent with attribute-based access control and dynamic views: SQL CREATE VIEW customer_360_masked AS SELECT cust_id, CASE WHEN is_account_group_member('risk_pii_readers') THEN cust_ssn ELSE 'REDACTED' END AS cust_ssn, cust_dob, account_balance FROM customer_360_raw; Same principle: the masking rule reads from group membership, which is governed centrally, not hardcoded per consumer. Where This Actually Breaks in Practice Three failure points I've seen repeatedly on migration engagements: Classification lag. Auto-classification tools (Macie, Purview's classifiers) run periodically, not on write. A new column with SSNs can sit unclassified — and therefore unprotected by tag policy — for hours or days. Mitigation: enforce classification as a CI/CD gate on schema changes, not a background scan. Derived data breaks lineage. A gold-layer aggregate table built from a PII source doesn't automatically inherit the source's tags unless your ETL explicitly propagates them. I've seen teams tag Bronze meticulously and leave Gold completely untagged because "it's aggregated, so it's safe" — until someone adds a customer_email passthrough column six months later during a "quick fix." Vector stores are an ungoverned blind spot. RAG pipelines embed source documents into vector DBs. The embeddings are technically an obfuscation, not encryption, and can leak source content back out under adversarial querying. If your governance program stops at the relational catalog, your embedding store has none of these controls at all — this is the fastest-growing gap I see on client engagements right now. What to Actually Do About It If you're an architect on a migration, the ordering matters more than the tooling: Classification runs before or during pipeline design, not after go-live. Build it into the migration assessment.One tag taxonomy, enforced everywhere — Bronze, Silver, Gold, and vector stores alike. If your classification schema doesn't apply to your RAG pipeline, it's incomplete.Policy grants reference tags, never table/prefix names. This is the difference between a policy that scales and one that needs manual review for every new table.Audit logs (CloudTrail, Unity Catalog audit, Lake Formation permissions history) get mapped to your actual compliance controls (SR 11-7, GDPR Article 30, DPDP) as a standing report, not a one-time audit exercise. None of this is exotic. It's mostly discipline about doing classification first and wiring access policy to read from it, instead of treating them as two projects run by two teams on two timelines. References AWS Documentation – Lake Formation Tag-Based Access ControlDatabricks Documentation – Unity Catalog attribute-based access control and dynamic viewsNIST SP 800-53 Rev. 5 – Security and Privacy Controls for Information SystemsFederal Reserve SR 11-7 – Guidance on Model Risk ManagementDAMA International – DMBOK2

The project AngularAndSpring has been refactored from Spring Reactor to Spring blocking APIs using JDK 25 and virtual threads. The Old Architecture The reason AngularAndSpring was built using Spring Reactor was primarily the efficient use of operating system threads and memory under high load. Spring Reactor had a limited thread pool (usually the number of available CPU cores) and used OS threads only when the CPU had work to do. It did not let the operating system threads sleep. This used CPU resources much better and required far fewer OS threads, each of which needs around a megabyte of memory. The trade-off was a reactive programming style, which can become harder to read as the complexity of the implemented functionality grows, along with stack traces that are very difficult to read. These hard-to-read stack traces make finding and fixing errors much harder and significantly more time-consuming. How Virtual Threads Change the Trade-Off Before JDK 25, the choice was between efficient hardware use with Spring Reactor and easy-to-read code with easier debugging and readable stack traces. Spring's support for Java virtual threads offers the use of very efficient virtual threads for request processing. With virtual threads, operating system threads are used only when processing is needed; virtual threads are unmounted from the OS thread, freeing the OS thread to run a different virtual thread; and virtual threads have minimal memory requirements. That means a limited number of operating threads is able to handle a very large number of virtual threads. With JDK 25 and the virtual threads update, the choice has changed: JDK 25 with virtual threads uses hardware resources efficiently, and you can keep the code easy to read and debug with readable stack traces. There are limitations for virtual threads when native code is called, but the same limitations apply for Spring Reactor too. The main virtual threads blocker: The missing synchronized support was fixed in JDK 25. That means the main advantages of Spring Reactor are now provided by the JVM with its virtual threads. The New Architecture The basic building blocks of the new architecture can be found in the backend pom.xml. It now uses these starters: XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-mongodb</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-restclient</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webmvc</artifactId> </dependency> The virtual threads are switched on with this in the application.properties: Properties files spring.threads.virtual.enabled=true The spring-boot-starter-webmvc is used to provide REST endpoints with blocking APIs. The spring-boot-starter-restclient is used to provide the new REST client for calling the blocking REST endpoints of other systems. The spring-boot-starter-data-mongodb is used to provide repositories and APIs for accessing MongoDB; all of these APIs are blocking. The ‘spring.threads.virtual.enabled=true’ enables the Spring Boot support for virtual threads provided by the JVM. The REST Endpoint The blocking REST endpoints now look like this in BitstampController: Java @RestController @RequestMapping("/bitstamp") public class BitstampController { private final BitstampService bitstampService; public BitstampController(BitstampService bitstampService) { this.bitstampService = bitstampService; } @GetMapping("/{currpair}/orderbook") public String getOrderbook(@PathVariable String currpair) { return this.bitstampService.getOrderbook(currpair); } ... } These endpoints do not return the reactor wrapper types. It just returns the intended return type. The ‘BitstampService’ methods now also return the intended return types without any wrappers. These endpoints just use plain Java. The Restclient The new blocking RestClient is now the Spring RestClient and looks like this in ScheduledTask: Java @Component public class ScheduledTask { private static final Logger LOG = LoggerFactory.getLogger(ScheduledTask.class); private static final String URLBS = "https://www.bitstamp.net/api"; private static final String URLCB = "https://api.coinbase.com/v2"; private static final String URLBF = "https://api.bitfinex.com"; private final BitstampService bitstampService; private final BitfinexService bitfinexService; private final CoinbaseService coinbaseService; private final MyUserService myUserService; private final RestClient.Builder restClientBuilder; public ScheduledTask(BitstampService bitstampService, MyUserService myUserService, BitfinexService bitfinexService, CoinbaseService coinbaseService, RestClient.Builder restClientBuilder) { this.bitstampService = bitstampService; this.bitfinexService = bitfinexService; this.coinbaseService = coinbaseService; this.myUserService = myUserService; this.restClientBuilder = restClientBuilder; } ... @Async("clientTaskExecutor") @Scheduled(fixedRate = 60000, initialDelay = 3000) @SchedulerLock(name = "BitstampQuoteBTC_scheduledTask", lockAtLeastFor = "PT50S", lockAtMostFor = "PT55S") public void insertBitstampQuoteBTC() { String currPair = "btceur"; insertBsQuote(currPair); } private ClientHttpRequestFactory customRequestFactory() { HttpClientSettings settings = HttpClientSettings.defaults() .withConnectTimeout(Duration.ofSeconds(4)) .withReadTimeout(Duration.ofSeconds(5)); return ClientHttpRequestFactoryBuilder.detect().build(settings); } private void insertBsQuote(String currPair) { LocalTime start = LocalTime.now(); try { QuoteBs quote = this.restClientBuilder.requestFactory(this.customRequestFactory()).build().get() .uri(String.format("%s/v2/ticker/%s/", ScheduledTask.URLBS, currPair)) .accept(MediaType.APPLICATION_JSON).retrieve().body(QuoteBs.class); quote.setPair(currPair); quote = this.limitPrecision(quote); this.bitstampService.insertQuote(quote); this.logDuration("Bitstamp", currPair, start); } catch (Exception e) { LOG.warn(String.format("Bitstamp data import failed for: %s", currPair), e); } } ... } The RestClient is created using the RestClientBuilder to allow connect and read timeouts to be configured. The timeouts are created in the customRequestFactory() method and set as a parameter of the RestClientBuilder. The RestClient uses the .uri(...) method to set the URL to call. The .accept(...) method sets the MediaType in the HTTP header that is expected from the REST client. The .retrieve() method executes the REST call, and the .body(...) method specifies the object into which the JSON response should be mapped. For debugging purposes, it can be useful to map the response body to a String first and analyze the response to ensure that it can be mapped correctly to the appropriate DTOs. The MongoDB Client For accessing the database, Spring Data MongoDB is used. The per-minute quotes are in the QuoteBs, QuoteBf, QuoteCb entities. The per-hour quotes are in the QuoteHourBs, QuoteHourBf, QuoteHourCb entities, and the per-day quotes are in the QuoteDayBs, QuoteDayBf, QuoteDayCb entities. The different entities enable Spring Data MongoDB support for each entity with its own repository. This is an example of QuoteBsMongoRepository: Java public interface QuoteBsMongoRepository extends MongoRepository<QuoteBs, ObjectId>, QuoteBsRepository { ... } The Interface ‘MongoRepository<QuoteBs, ObjectId>’ provides standard interfaces and enables the generated CRUD methods in QuoteBsRepository. Java public interface QuoteBsRepository extends QuoteRepository<QuoteBs>, QuoteMongoRepository<QuoteBs> { ... QuoteBs insert(QuoteBs quote); List<QuoteBs> findByPairAndCreatedAtAfterOrderByCreatedAtAsc(String pair, Date date, Limit limit); @Query("{ 'createdAt': { '$gt': ?0, '$lt': ?1 } }") List<QuoteBs> findByCreatedAtGreaterThanAndCreatedAtLessThan(Date from, Date to); } This interface uses the by-method-name-generated CRUD methods that Spring Data MongoDB provides and uses the ‘@Query’ annotation for a finder that can not be generated only by a method name. The entity QuoteBs looks like this: Java @Document public class QuoteBs implements Quote,QuotePairBs { @Id private ObjectId _id; @NotBlank @Indexed @JsonProperty private String pair; @NotNull @Indexed(name = "QuoteBs-createdAt") @JsonProperty private Date createdAt = new Date(); private BigDecimal high; private BigDecimal last; private Date timestamp; private BigDecimal bid; private BigDecimal vwap; private BigDecimal volume; private BigDecimal low; private BigDecimal ask; private BigDecimal open; The ‘@Document’ annotation marks the class as a MongoDB document. The ‘QuotePairBs’ interface is used to be able to access the common properties of minute, hour, and day entities of the same source. The ‘@Indexed’ annotation makes sure an index is created for this property in MongoDB for faster access and ordering. The precision of the Java BigDecimal type is higher than the precision of the MongoDB number type. Because of that, this method in ScheduledTask is used: Java private <T> T limitPrecision(T quote) { var fields = quote.getClass().getDeclaredFields(); for(Field field : fields) { if(field.getType().isAssignableFrom(BigDecimal.class)) { field.setAccessible(true); try { var value = (BigDecimal) field.get(quote); value = value.round(new MathContext(30, RoundingMode.HALF_UP)); field.set(quote, value); } catch (IllegalAccessException e) { throw new RuntimeException(e); } } } return quote; } The precision is limited to 30 decimals for all properties with the type BigDecimal. This is enough for all use cases in this application. Thread Pools If the hour and day quotes are recalculated from a large number of minute quotes, this can take several minutes. To avoid competing with the thread pool handling the user requests, a separate thread pool is created for this task in SchedulingConfig: Java @Bean(name = "clientTaskExecutor") public Executor threadPoolTaskExecutor() { ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor(); executor.setVirtualThreads(true); executor.setThreadNamePrefix("clientTask-"); return executor; } This creates a separate TaskExecutor that can be used in the ‘@Async(“clientTaskExecutor”)’ annotations. That makes sure that no user requests wait in the queue for a free thread while long-running threads are at work. Conclusion With virtual threads in JVM 25 and Spring’s support for them, many reasons for using reactive programming in Spring have disappeared. The synchronized issue with virtual threads has been fixed in JDK 25, and the remaining causes of thread pinning are not relevant anymore for most use cases. The JVM now provides scalability and efficiency of reactive frameworks out of the box, while the blocking programming model is easier to implement and debug. Spring Data MongoDB support makes database access much easier with generated CRUD methods and the ‘@Query(…)’ support. It is time for projects still using reactive programming to move to an easier, faster programming model for developers.

A confession first: I have spent the last few months building things that require actual thought. Turicum is a programming language. Perag is a personal RAG system that reads your documents so your AI doesn't have to. mdship manipulates Markdown files with the seriousness of a tool that knows people will build CI pipelines on top of it. All good, all real, all serious work with the hope that some will use them and make value. Sometimes you just want a brown bag project. The kind you can start, finish, and ship before your lunch sandwich gets soggy. So I built babbler. The Problem: Humans are Bad at Remembering Numbers With 9 Digits and No Personality Quick, what's your tax file reference number? Not the one you can look up. The one you can recall, right now, from memory, while a clerk on the phone is waiting and your insurance is somehow involved. Modern life hands you an endless parade of numbers exactly like this: Your health insurance claim is CLM-88213047.Your tax office reference is 2026/AT/0041938.Your car insurance policy is POL-773190-B.The ticket you opened with customer support, which you will need to reference again in three weeks, is #40921177. None of these are memorable. None of them are meant to be memorable. They're auto-increment counters or UUID fragments that some database handed out. The human is expected to somehow carry that around in the head or, more realistically, in a purse full of paper never to find. Nobody has ever said "oh yes, claim eight-eight-two-one-three-oh-four-seven, of course" out loud without sounding like they're reading a hostage's proof-of-life number. What humans are good at remembering are names. So: what if instead of 88213047, your claim was called Talo Buno? Instead of 88213048, it's Patu Luta. Two little pronounceable words instead of eight forgettable digits. You'd still write the number down somewhere official, but you could talk about "the Talo Buno claim" the way you'd talk about a pet, and actually remember which one it was. That's it. That's the whole idea. It's not going to save the world. It might save you one mildly awkward phone call. The Algorithm (Short Version, I Promise) The core idea: every counter-number gets converted into two pronounceable syllable-pairs, like Talo Buno or Patu Luta. Always four syllables, always two capitalized words, always the same result for the same number. Under the hood, it's mercifully simple — no cryptography, no hashing, no network calls, no "AI" (yes, I checked myself at the door on that one): Take 36 consonant-vowel syllables (ba, bo, bu, da... up to zu) — no h, no confusing look-alikes, just clean sounds that work across languages.Combine them two at a time to build words, and quietly remove the ones you wouldn't want on an official document (more on that in a second).Run the input number through a small, fixed bit of math — a bijection, meaning every number maps to exactly one name and every valid name maps back to exactly one number. No two different claim numbers ever accidentally get the same nickname. No retries, no collisions, no "sorry, try again" — just guaranteed-unique output, by construction, every time.Format the result as two capitalized words and hand it back. I went through a "hash it with MD5 and hope for the best" version first, because that's the obvious first instinct. Turns out "hope for the best" produced actual name collisions for about a third of all numbers, which is a spectacular way to guarantee two different insurance claims share a name. The fixed version replaced "hope" with actual math, which is generally a good trade. The MD5 version never even made it into a commit — some mistakes are too brief to leave fossils. The Blacklist (The Fun Part) Here's the part of this project that made me grin more than a numbering scheme has any right to: since the syllables get combined somewhat randomly, you occasionally get combinations you do not want stamped on someone's official tax document. So there's a small blacklist of forbidden two-syllable words the generator is guaranteed to never produce — permanently filtered out before any number is ever assigned, not caught after the fact. The list is a delightful little multilingual minefield: Some are baby-talk for things nobody needs on a government form, one is a slang word for police (and also, unrelated, poop, in Spanish), one means "senile," a couple is just international slang for excrement in three different languages, and two are, let's say, anatomically specific in a way that would not go over well in a customer support ticket. All tastefully avoided. Meanwhile, — and this delighted me — Mama, Papa, and Nana all made it through completely clean. Sometimes the universe hands you a nice coincidence. Try It Yourself Plain Text 10000 → "Talo Buno" 10001 → "Patu Luta" Every number from 0 up to about 1.64 million gets its own guaranteed-unique, guaranteed-(major EU languages plus Hungarian)inoffensive two-word name. One-way only — it makes names, it doesn't decode them back, because the actual point was never secrecy or reversibility, it was just "give me a name I can remember instead of a number I can't." MIT Licensed, Available in Java, Go, and Python The code — encoder, tests, docs, the works — is MIT-licensed, so grab it and use it however you like: https://github.com/verhas/babbler. It ships with matching, fully tested implementations in Java, Go, and Python — pick your poison; they all produce byte-identical names for the same number. And by "Java, Go, or Python," I obviously also mean any other language you like, because with three clean, well-documented reference implementations sitting right there, any reasonably competent LLM today can knock out a Rust, Kotlin, or — sure, why not — COBOL port before you've finished reading this sentence twice. Consider that a stretch goal I'm delegating to the machines. Brown bag project, delivered. Back to the serious stuff after the weekend. Disclosure: This article and its visuals were created with LLM assistance, but every claim, argument, and technical detail was thoroughly reviewed, fact-checked, and approved by the author before publication. No "slop" here. To be clear about what this means: I used an AI as a tool for composition and iteration—much like using a typewriter instead of painstakingly handwriting each page with a goose feather, let alone carving it into stone. The tool speeds up the mechanical work; it doesn't replace the thinking. The tone, the structure, the emphasis — all human judgment. This is how AI tools should be used in knowledge work: as amplifiers of clarity and productivity, not as replacements for expertise or accountability. A human — specifically, I stand behind every word.

By Peter Verhas CORE

AWS Secrets Manager has a lovely feature: automatic rotation. You attach a rotation Lambda, set a schedule, and every 30 days your RDS password is rotated for you. Secure by default, no human involved. There's just one problem, and it's the one nobody mentions in the demo: your application has no idea it happened. Your Go service read SecretString once, at startup, and stashed the password in a struct. Thirty days later Secrets Manager quietly rotates it, the old value stops working, your connection pool starts throwing auth errors, and you find out via a pager alert at 3 am. The "fix" everyone reaches for is to redeploy on a schedule, or restart pods so the app re-reads the secret — which turns a security feature into an operational chore. mamori is a Go library I built to close exactly this gap. It watches AWS Secrets Manager (and SSM Parameter Store) and reconciles a rotation into your running process — typed, validated, redacted — so a rotation triggers a connection-pool rebuild instead of a restart. This is the AWS-focused companion to the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go. Read that first for the full model; here we go deep on AWS. The Two AWS Schemes The providers/aws module registers two ref schemes: RefReads fromaws-sm://<name>[#json-key]Secrets Manageraws-ps://<name>[#json-key]SSM Parameter Store A few real refs — note the triple slash on Parameter Store, because SSM parameter names are absolute paths that start with /: Plain Text aws-sm://prod/api-key # whole secret value aws-sm://prod/db#password # one key out of a JSON secret aws-ps:///myapp/log-level # SSM parameter (triple slash!) aws-ps:///myapp/db#password # one key out of a JSON SSM parameter The rule of thumb: secrets (passwords, tokens, keys) belong in Secrets Manager (aws-sm), and plain, non-sensitive config (log levels, worker counts, feature toggles) belongs in the cheaper Parameter Store (aws-ps). mamori lets you mix both in the same struct. Setup Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/aws Register it with a blank import for zero-config (it uses the standard AWS credential chain — environment, shared config, IRSA on EKS, EC2 instance role): Go import _ "github.com/xavidop/mamori/providers/aws" Or, when you need to pin a region or pass a custom client, construct the provider explicitly and hand it to Load/Watch with WithProvider — this takes precedence over the registry for that call: Go import awsprov "github.com/xavidop/mamori/providers/aws" cfg, err := mamori.Load[Config](ctx, mamori.WithProvider(awsprov.NewSecretsManager(awsprov.WithRegion("eu-west-1"))), ) Your runtime's IAM role needs the obvious permissions, scoped tightly: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "secretsmanager:GetSecretValue", "Resource": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:prod/*" }, { "Effect": "Allow", "Action": "ssm:GetParameter", "Resource": "arn:aws:ssm:eu-west-1:123456789012:parameter/myapp/*" } ] } Load, Then Watch the Rotation Here's a service that pulls its database password from Secrets Manager and its operational knobs from Parameter Store: Go package main import ( "context" "log" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/aws" ) type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` APIKey secret.String `source:"aws-sm://prod/api-key"` LogLevel string `source:"aws-ps:///myapp/log-level" default:"info" validate:"oneof=debug info warn error"` Workers int `source:"aws-ps:///myapp/workers" default:"4" validate:"gte=1,lte=256"` } func main() { ctx := context.Background() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(60*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { // A rotation just landed — rebuild the pool, no restart. pool.Rotate(ev.New.DBPassword.Reveal()) log.Println("database password rotated and applied live") } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() log.Printf("started with %d workers at %s", cfg.Workers, cfg.LogLevel) } DBPassword and APIKey are secret.String, so they redact themselves everywhere — log.Printf("%+v", cfg) prints [REDACTED], json.Marshal emits "[REDACTED]", and slog logs them as [REDACTED]. The plaintext only ever escapes through an explicit .Reveal(). mamori even ships a go vet analyzer that fails the build if you put an aws-sm:// value into a plain string instead of a secret.String. Polling Done Right (Jitter, Batching, Atomicity) Secrets Manager has no push notification for value changes, so mamori polls — but it polls the way you'd want a fleet to poll: Jittered intervals. The poll interval is spread by ±20% by default, so 200 pods don't all hit GetSecretValue on the same tick and blow through your API rate limit. You can tune the spread with WithJitter.Change detection is cheap. mamori compares the secret's version (Secrets Manager's VersionId) before doing anything, so an unchanged secret costs one lightweight check, not a full re-decode.Batching. The AWS provider resolves multiple refs from the same account in as few API calls as possible, automatically — so a struct with ten aws-sm:// fields doesn't mean ten times the API traffic.Atomic + validated. When a rotation is detected, mamori re-validates the entire struct and only swaps it in if it's valid. OnChange never fires with a half-updated or invalid config. Compare this to the AWS Secrets Manager caching client: it caches and refreshes one secret at a time, with no struct composition, no validation, no diff-aware callback, and no config sources beyond Secrets Manager. mamori gives you the whole config surface — secrets and Parameter Store and env and files — in one typed struct with one reconciliation loop. Tolerate a Blip, Page on a Real Outage Secrets Manager occasionally throttles or has a transient error. You don't want that to page anyone — a blip should just retry with backoff and keep serving the last good value. But if Secrets Manager is genuinely unreachable for ten minutes, that's worth waking someone up. WithStale draws that line: Go w, err := mamori.Watch[Config](ctx, mamori.WithStale(10*time.Minute), mamori.OnError(func(err error) { var stale *mamori.StaleError if errors.As(err, &stale) { pager.Page("AWS config stale >10m", stale) return } metrics.Inc("aws_transient_error") // blips: just a metric }), ) Mixing Sources in One Struct The real payoff shows up when your config isn't all in one place. mamori doesn't care that these live in four different systems — it's one struct: Go type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` // Secrets Manager Workers int `source:"aws-ps:///myapp/workers"` // Parameter Store LogLevel string `source:"env:LOG_LEVEL" default:"info"` // environment TLSCert []byte `source:"file:///etc/tls/tls.crt"` // local file } Wrapping Up Automatic rotation is only half a feature if your app has to restart to notice. mamori supplies the other half: it watches AWS Secrets Manager and Parameter Store and reconciles a rotation into your running Go service — typed, validated, redacted, no bounce. Shell go get github.com/xavidop/mamori/providers/aws Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and at v0.1.0 — early days, so feedback and issues are genuinely welcome. Happy coding!

By Xavier Portilla Edo CORE

API testing is the practice of verifying that an API behaves as expected. In Agile environments, teams often perform it early in the SDLC as part of shift-left testing, enabling faster feedback and helping them identify and fix defects early. Playwright is a modern test automation framework that’s popular among testers and developers. Built by Microsoft, it makes it easy to automate both web and API testing across multiple programming languages, all within a single tool. In this tutorial, we will focus on testing APIs using Playwright TypeScript. By the end of this article, you should have a clear understanding of writing automation tests for PUT, PATCH, and DELETE API requests with Playwright TypeScript. Check out the previous articles to learn how to test GET API request and verify response data with Playwright TypeScript. What are PUT Requests? PUT requests are commonly used to update an existing resource by replacing its current data with the data provided in the request body. They are similar to POST requests; the Content-Type header plays an important role because it tells the server how to process the data being sent. When a PUT request is successful, it typically returns a 200 status along with the updated resource in the response. However, this can vary by API design; some APIs may return only a status code without a response body. How to Test PUT API Requests With Playwright TypeScript Playwright offers a request API that lets us create and manage HTTP request contexts easily. We’ll use the PUT /updateOrder API that updates the available order using its order ID. This API is a part of the RESTful e-commerce application available on GitHub. This API requires an authentication token to validate the request and authorize the update; if the token is missing or invalid, the request will fail. The order ID needs to be passed as a path parameter so the correct order can be identified, and the updated details should be included in the request body in JSON format. Since this is a PUT request, you must send the entire order object, not just the field being modified. Even for a small change, the full set of order data is required. We’ll use the following test scenario to demonstrate testing the PUT API request with Playwright TypeScript: Markdown ## Test Scenario Title: Update the existing orders in the system. ### Pre-condition: Valid orders are available in the system #### Test 1. Update all the order details of order_id "1" 2. Verify that the Status Code 200 is returned in the response 3. Assert that the order details have been updated correctly To implement this test scenario, we’ll need to perform the following steps: Hit the Auth API and extract the token.Write a test to send the PUT API request and pass the token extracted in step 1. Step 1: Create a Fixture to Generate and Extract the Token We’ll need the auth token for PUT, PATCH, and DELETE API requests, so let’s create a fixture that hits the POST /auth API and extracts the token for use across other tests. Let’s create a new fixture file auth.fixture.ts, in the fixtures folder in the project’s root directory. TypeScript import { test as base, expect } from "@playwright/test"; import dotenv from "dotenv"; dotenv.config(); type MyFixtures = { token: string; }; export const test = base.extend<MyFixtures>({ token: async ({ request }, use) => { const response = await request.post("http://localhost:3004/auth", { data: JSON.stringify({ username: process.env.USERNAME, password: process.env.PASSWORD, }), headers: { "Content-Type": "application/json", }, }); expect(response.status()).toBe(201); const responseBody = await response.json(); await use(responseBody.token); }, }); export { expect } from '@playwright/test'; A custom Playwright test setup is created by extending the base test object and adding a reusable token fixture. It uses the dotenv library to load environment variables like USERNAME and PASSWORD, ensuring sensitive data is not hardcoded, and defines a TypeScript type to make the token available with proper type safety. A .env file is created containing the credentials and placed in the project’s root folder. This ensures best practices are followed and sensitive information, such as usernames and passwords, is not exposed. As this is a demo API, the credentials used with the POST /auth endpoint are provided below. In a real-time project, the .env file should be added to the .gitignore file to ensure it is not committed to the remote repository. Plain Text #.env USERNAME=admin PASSWORD=secretPass123 Inside the fixture, an authentication API is called to generate a token; the response is validated, and the token is extracted and passed to the tests using the use() function. Finally, expect is re-exported so that both test and expect can be imported from this single fixture file, keeping test imports clean and consistent. Step 2: Updating the Order With a PUT Request and Verifying the Response Let’s create a new spec file to update the order using the PUT /updateOrder API request. TypeScript # 5_put-test.spec.ts import { test, expect } from "../fixtures/auth.fixture.js"; test("Update Order using PUT API Request", async ({ request, token }) => { const updatedOrder = { user_id: "89", product_id: "124", product_name: "Lindt Choclate", product_amount: 50, qty: 3, tax_amt: 10, total_amt: 160, }; const id = 1; const response = await request.put( `http://localhost:3004/updateOrder/${id}`, { data: JSON.stringify(updatedOrder), headers: { "Content-Type": "application/json", Authorization: `${token}`, }, }, ); const responseBody = await response.json(); expect(response.status()).toBe(200); expect(responseBody).toEqual( expect.objectContaining({ message: "Order updated successfully!", order: expect.objectContaining({ product_id: "124", product_name: "Lindt Choclate", }), }), ); }); Code Walkthrough Using the auth fixture for getting the token: The test imports a custom test from the auth fixture, which automatically provides the token as part of the test context. This token is generated before the test runs and injected via Playwright fixtures, eliminating the need to manually authenticate in each test.Request body: The updatedOrder object represents the payload sent in the request to update an existing order. It contains the required fields for the order update request body. This object is later serialized into JSON format before being sent to the API.Sending the PUT request: The request.put() function is used to send an HTTP PUT request to update the order with a specific ID. The request includes the serialized request body and headers such as Content-Type and Authorization, where the token is passed for authentication. This step performs the actual API interaction to update the order in the system.Dynamic endpoint handling: The order ID is stored in a variable and dynamically appended to the API endpoint using template literals. This makes the test more maintainable and reusable.Verifying the response: The response is first converted to JSON, and the status code is validated to be 200, indicating a successful update. Then, assertions are made using expect.ObjectContaining that partially validates the response body, ensuring that the success message and updated order details match expected values. Test Execution Before running the test, the RESTful e-commerce application should be started using Docker by running the following command from the terminal: Plain Text docker compose -f docker-compose-ecommerce.yml up -d The following command should be run from the terminal: Plain Text npx playwright test tests/5_put-test.spec.ts The following test output shows that the test was executed successfully: What Are PATCH Requests? A PATCH request is used when we want to update only part of a resource, not the whole thing. It’s similar to a PUT request, but the main difference is that PUT expects the entire payload, whereas PATCH lets us send only the fields we want to change. How to Test PATCH API Requests With Playwright TypeScript We’ll use the PATCH /partialUpdateOrder API from the RESTful e-commerce application that updates the available order using its order ID. Similar to the PUT API request discussed in the earlier section of this article, the PATCH API also needs an authentication token to update the order. As it is a PATCH request, we don’t need to send the full payload. Only the fields that require updates should be included in the request body. Test Scenario: Markdown ## Test Scenario Title: Partially update an existing orders in the system. ### Pre-condition: Valid orders are available in the system #### Test 1. Update the user_id and product_id for the order ID - 1 2. Verify that the Status Code 200 is returned in the response. 3. Assert that the order details have been updated correctly. The PATCH API request requires an authorization token. We’ll use the token from the auth.fixture.ts file that we generated in the earlier section while discussing the PUT API request. Let’s create a new spec file to partially update an order using the PATCH /partialUpdateOrder API. TypeScript # 6_patch-test.spec.ts import { test, expect } from "../fixtures/auth.fixture.js"; test("Update Partial Order using PATCH API Request", async ({ request, token, }) => { const updatedOrder = { user_id: "98", product_id: "214", }; const id = 1; const response = await request.patch( `http://localhost:3004/partialUpdateOrder/${id}`, { data: JSON.stringify(updatedOrder), headers: { "Content-Type": "application/json", Authorization: `${token}`, }, }, ); const responseBody = await response.json(); expect(response.status()).toBe(200); expect(responseBody).toEqual( expect.objectContaining({ message: "Order updated successfully!", order: expect.objectContaining({ user_id: "98", product_id: "214", }), }), ); }); Code Walkthrough Using the auth fixture for getting the token: The test uses a custom fixture that automatically provides a token, so we don’t have to call the POST /auth API every time. The token is generated before test execution and injected into the test context, ensuring secure and reusable authentication across multiple tests.Request body: The updatedOrder object contains only the fields user_id and product_id that need to be modified. As it is a PATCH request, there is no need to send the complete order payload; this makes the request lightweight and focused on partial updates.Sending the PATCH request: The request.patch() function is used to send a partial update request to the API endpoint, with the order ID dynamically added to the URL. The request includes the JSON-stringified body along with the Content-Type and Authorization headers. The token is passed in the Authorization header.Verifying the response: The response is parsed into JSON and validated to ensure a 200 status code is returned after the PATCH API is hit successfully. This verification ensures that the operation was successful. Then, expect.objectContaining is used to validate the response body, confirming that the success message and updated fields match the expected values. Test Execution To run the test, the following command should be executed from the terminal: Plain Text npx playwright test tests/6_patch-test.spec.ts The following screenshot of the test execution confirms that the test was run successfully: What Are DELETE Requests? A DELETE API request is used for removing a specific resource from the server. In most cases, it doesn’t return a response body. The resource is identified using a URI, and once the request is processed, it’s permanently deleted. Because of this, DELETE operations can have side effects, like removing records from a database. A DELETE request should be handled carefully because: Data deleted through a DELETE request is permanently removed and cannot be restored.It directly impacts the server data; hence, it is not considered a safe method.It may lead to inconsistencies if not managed appropriately.DELETE isn’t always idempotent in real-world scenarios. For example, the first DELETE request might return a 204 (No Content) indicating success, while calling it again on the same resource could return 404 (Not Found) because the resource has already been removed. How to Test DELETE API Requests With Playwright TypeScript We’ll use the DELETE /deleteOrder API endpoint from the RESTful e-commerce application for this demonstration. This API requires an order_id as a path parameter to identify the order to delete. A valid authentication token must be included in the headers. On successful execution, the order is removed, and a 204 (No Content) status is returned. Want to learn more about API Testing? Check out this tutorial to get hands-on API Testing Test Scenario: Markdown ## Test Scenario Title: Delete existing order from the system. ### Precondition - There should be some valid orders available in the system. #### Test Steps: 1. Delete the order with order_id "6" using the DELETE request. 2. Verify that 204 Status Code is returned in the response. To implement this test scenario, we need valid orders in the system, which can be created using the POST /addOrder API. Refer to the earlier tutorial on testing POST requests with Playwright TypeScript to explore different ways to generate request payloads and test the POST API request. Test Implementation Let’s create a new spec file to delete an order using the DELETE /deleteOrder API. TypeScript import { test, expect } from "../fixtures/auth.fixture.js"; test("Delete an order using DELETE API Request", async ({ request, token }) => { const id = 6; const response = await request.delete( `http://localhost:3004/deleteOrder/${id}`, { headers: { Authorization: `${token}`, }, }, ); expect(response.status()).toBe(204); }); Code Walkthrough Using the auth fixture for getting the token: The test imports a custom test from the auth fixture, which automatically provides a valid authentication token. This token is generated before the test runs and injected into the test context, so there’s no need to call the POST /auth API explicitly before the DELETE API.Sending the DELETE request: The request.delete() function is used to send a DELETE request to the API endpoint, with the ID dynamically passed in the URL to identify the order to be deleted. The request includes the Authorization header, where the token is sent to authenticate the request. No request body is required for this operation.Assertion: The test verifies that the response status is 204, which indicates that the request was successfully processed and the resource was deleted. Since DELETE APIs often don’t return a response body, validating the status code is sufficient to confirm success. Test Execution To run the test, run the following command from the terminal: Plain Text npx playwright test tests/7_delete-test.spec.ts The following screenshot confirms that the test ran successfully: A simple, practical approach is to call the GET API after running a DELETE request to confirm the resource has been removed from the system. Summary Testing PUT, PATCH, and DELETE APIs with Playwright and TypeScript helps ensure that the application handles updates and deletions correctly. While challenges such as managing test data, handling authentication, and maintaining test independence can arise, Playwright’s fixtures and built-in request handling make it easier to create clean and reusable tests. The main advantage is the ability to validate end-to-end API behavior with clear assertions and strong TypeScript support. In my experience, the tests should be isolated, avoid unnecessary dependencies between them, and validate both status codes and response data to make the API tests more robust and maintainable. Happy testing!!

By Faisal Khatri CORE

Every few weeks, a new name shows up in security news. Project Glasswing. Daybreak. Lightwell. CodeMender. Big Sleep. Each one is presented as the next answer to AI-driven cyberattacks. It is easy to read this as a race between AI labs, cloud providers, and security vendors, and to ask which one will win. That is the wrong question. Each of these programs strengthens a different part of the same system. Some improve discovery. Some help engineers understand which bugs actually matter. Some generate patches. Some govern who is allowed to use powerful cyber tools at all. None of them, on its own, is a complete answer. The useful question for a security team is not which vendor we should bet on. It is which part of our own system is weakest, and which of these pieces fills that gap. This article lays out the evidence for why the pace of the problem has changed, walks through the lifecycle every vulnerability goes through, maps the current tools, programs, and rulebooks onto that lifecycle, and then treats AI agents themselves as a new class of identity that needs the same governance as any other. Along the way, I point to my own earlier articles where they cover the underlying engineering discipline in more depth. Why the Old Patch Cycle No Longer Works The pressure is not hypothetical. A few figures, each traceable to its source, show the shape of the problem. SignalWhat it showsSourceA record 48,185 CVEs were published in 2025, a 263 percent rise from 2020The volume of vulnerabilities to sort through has grown far faster than security teams haveCloud Security Alliance, "The Collapsing Exploit Window"Mandiant's M-Trends 2026 puts mean time to exploit at roughly negative seven days for high-value targetsAttacks now often begin before a fix is even publicMandiant M-Trends 2026, Google CloudCISA's Binding Operational Directive 26-04 requires a three-day fix for vulnerabilities that meet four risk criteriaThe US federal government has formally shortened its own patch deadlines because of AI-accelerated exploitationCISA, Binding Operational Directive 26-04The average KEV catalog patch deadline in 2026 is 14.4 days, down from 19.7 the year beforeDeadlines across the board are compressing, not just for the most severe bugsFederal News Network, May 2026AI agents found and helped fix 1,072 Chrome security bugs across two releases in June 2026, more than the previous 23 releases combinedDiscovery tooling has genuinely gotten faster, not just louder in marketingGoogle, "Stronger with every update" None of this changes how many hours are in your team's week. The number that actually matters is the one CISA is trying to compress by regulation: the time between a fix exists and the fix is live in production. Every program in this article is, in one way or another, trying to shrink that number. The Lifecycle Every Vulnerability Goes Through Before comparing tools, it helps to agree on the steps a single weakness goes through, from discovery to a proven fix. Lifecycle of a vulnerability or a bug Discovery has become cheap. Deciding, fixing safely, and rolling out remain slow, mostly because they need judgment, testing, and coordination across teams. That is where most vulnerability management programs still struggle, and it is where the new wave of AI security tools is trying to help. This lifecycle is also a loop, not a straight line. A better way to see the whole program is as a continuous cycle that feeds back into itself. Vulnerability management Discovery finds what might be wrong. Context asks whether the vulnerable component is actually reachable, deployed, and exposed. Prioritization ranks it against everything else competing for the same engineering hours. Remediation turns a finding into an engineering change: a dependency bump, a code fix, a rotated credential, a corrected IAM policy. Governance makes sure the whole process is reviewable, approved where it needs to be, and evidenced. Learning feeds what was found back into better defaults so the same class of issue shows up less often. I covered the review and testing discipline that sits behind the remediation and governance stages in Code Review Core Practices, which stays relevant even when the patch is drafted by an AI agent rather than a person. Three Programs, Three Different Jobs Three named programs currently dominate the conversation. Each one is strongest at a different point in the lifecycle above. Project Glasswing: Discovery at the Infrastructure Layer Anthropic launched Project Glasswing in April 2026. Launch partners include Amazon Web Services, Apple, Cisco, CrowdStrike, Google, JPMorganChase, the Linux Foundation, Microsoft, NVIDIA, and Palo Alto Networks, working with Claude Mythos Preview on defensive tasks. Anthropic committed up to 100 million dollars in usage credits and 4 million dollars in donations to open source security groups, and later extended access to roughly 150 more organizations across more than fifteen countries (Anthropic, Project Glasswing). The program targets critical infrastructure and widely used open source software, giving trusted defenders early access to a strong model before similar capability becomes broadly available. Mozilla has reported that the underlying model helped it find and patch 271 vulnerabilities in a recent Firefox release. OpenAI Daybreak: Tiered Access for Defenders OpenAI announced Daybreak on May 11, 2026, describing it as bringing together its frontier models, Codex Security, and a network of security partners to accelerate cyber defense and build security into software from the design stage rather than only patching after the fact (OpenAI, Daybreak). Access is split into two tiers that require separate approval. Daybreak's tiered access In September 2026, OpenAI added a subsidized access tier for organizations that keep essential services running, describing it as billions of dollars in access over roughly six months. Project Lightwell: Fixing Open Source Without Breaking Production IBM and Red Hat announced Project Lightwell on May 28, 2026, as a five-billion-dollar commitment backed by a global force of more than 20,000 engineers, and moved it to commercial launch on July 8 (IBM Newsroom, May 28, 2026; IBM Newsroom, July 8, 2026). It targets a specific and common problem. A library has a known bug. The official fix is a newer version, but the newer version breaks a certified production setup. Lightwell Network launched with a catalog of more than 6,500 remediated, signed, and certified dependencies across ecosystems such as Java and Python, giving teams a patched version of the release they already run instead of forcing a risky upgrade. Lightwell Clearinghouse Premier, in limited availability, adds coordinated patch embargo handling. In August 2026, IBM and Red Hat extended free access to more than 185 universities and 100 NGOs and think tanks that lack the engineering capacity to keep pace on their own (IBM Newsroom, August 4, 2026). Comparing the Three GlasswingDaybreakLightwellLed byAnthropic and partnersOpenAI and partnersIBM and Red HatCore questionWhere are the bugs in critical software?How can defenders use strong AI safely?How do we fix open source bugs without breaking production?Strongest lifecycle stepsFind, patchFind, check, proveFix, test, roll outAccess modelInvited partners, expandingApproved defenders, two tiersPaid subscription, with free access for universities and NGOsPrimary scopeCritical and open source infrastructureAn organization's own code and systemsOpen source dependencies in productionMain limitationMostly invite-basedSeparate approval for each tierCovers dependencies, not custom code Tools That Find and Patch Code Directly Alongside the three named programs, a set of tools now work directly inside the discovery and fix stages. ToolOwnerFunctionLifecycle stepSourceBig SleepGoogle DeepMind and Project ZeroSearches for memory safety bugs in widely used codeFindGoogle, Chrome security blogCodeMenderGoogle DeepMindPatches new bugs and rewrites existing code to remove whole classes of bugsFind, fixDeepMind, "Introducing CodeMender"Codex SecurityOpenAIBuilds an editable threat model from a repository and generates and tests patchesFind, checkOpenAI, DaybreakClaude SecurityAnthropicPublic beta for organizations outside GlasswingFind, checkAnthropic, Project Glasswing DeepMind is explicit that every CodeMender patch is reviewed by a person before it ships. That single fact matters more than the raw bug counts. It matches a habit I described in Ansible Code Scanning and Quality Checks With SonarQube: automated scanning finds candidates, but a human still signs off before anything reaches production. There is also support work happening around the big three programs, aimed at a second-order problem: faster discovery means more reports than open source maintainers can review. Anthropic has partnered with the Open Source Security Foundation's Alpha-Omega project to help maintainers triage incoming reports, and OpenAI's Daybreak commitment includes direct funding for groups such as Trail of Bits. Palo Alto Networks has also worked with IBM and Red Hat so its network-level protections can block attacks while a Lightwell fix is tested and rolled out, a practice known as virtual patching. It buys time. It does not replace the fix. What This Sets Up Discovery is now genuinely fast. Three well-funded programs and a handful of dedicated tools are throwing real capability at the find-the-bug and build-the-fix steps of the lifecycle. But finding more bugs faster does not, by itself, get you a faster or safer program. It just moves the bottleneck. The next problem is knowing which of those findings actually matter, making sure the AI agents doing this work are governed like the privileged systems they are, and having a plan that does not depend on an invitation to any of these programs. That is where Part 2 picks up. Further Reading Code Review Core PracticesAnsible Code Scanning and Quality Checks With SonarQube

By Vidyasagar (Sarath Chandra) Machupalli FBCS CORE

Context The incident occurred in the production backend infrastructure of a mobile application. External HTTP traffic was handled by Traefik, deployed on Docker Swarm manager nodes. Traefik ran in swarmMode, used providers.docker, was configured to use the mgp_default network, and had lbswarm disabled so that routing used the required overlay network rather than the Swarm VIP. One of the critical services was mgp_router-app. It was an entry-point service with two replicas deployed to preserve availability if one of them failed. The service used placement constraints, max replicas per node = 1, and a Traefik health check configured for /actuator/health on port 8080. This meant we relied on external health checking and strictly controlled replica placement across nodes. Swarm-level load balancing was intentionally disabled. We needed routing control at the Traefik layer, along with the ability to quickly remove unhealthy backends based on their health check status. For this production setup, that provided more flexibility than the built-in Swarm load-balancing logic. What Happened The issue was neither a widespread outage nor a failure of the entire service. One replica became unreachable to Traefik over the overlay network: a Traefik instance on one of the manager nodes could not establish a connection to that replica’s backend endpoint. The second replica continued to work normally, so the service as a whole remained available. For regular traffic, the incident did not affect service availability: Traefik continued routing requests to the healthy replica and removed the unavailable backend from routing. During targeted manual testing of the unhealthy replica’s backend endpoint through Traefik, requests ended in a timeout or HTTP 503. The container appeared healthy, but only locally. The service responded inside the container, and /actuator/health returned UP. However, other containers and Traefik instances on remote nodes could not establish a connection to that replica through its IP address in mgp_default. Initial Checks The container had the same DNS name in both networks. This ruled out name mismatch as an independent cause of the unavailability: the issue occurred specifically at the network-connectivity level to the replica IP in mgp_default. First, we identified the problematic container and inspected its state: Shell docker ps --filter "name=mgp_router-app" --q docker inspect 6bd15b62f891 Next, we checked the networks connected to the container: Shell docker inspect 6bd15b62f891 --format '{{json .NetworkSettings.Networks}' | jq The container was attached to two networks: ingress with IP 10.22.3.137/22;mgp_default with IP 10.29.72.232/13. We then entered the container and checked whether the application was listening on the expected port: Shell docker exec -it 6bd15b62f891 bash ss -tuln The health endpoint also responded correctly: Shell curl "http://10.29.72.232:8080/actuator/health" The expected response was: Shell {"status":"UP"} At this point, it was clear that the issue was not the application itself, its startup, or its bind address inside the container. The service was alive, but that alone said nothing about its reachability from outside the container. Checking Connectivity Between Nodes The next step was critical: We checked whether another node in the cluster could reach the container using its overlay IP. Shell docker exec -it 373354916bfc bash ping 10.22.3.137 This was the IP from the ingress network, and the test confirmed ICMP reachability of the address in that network. That indicated the issue did not look like a complete loss of inter-node connectivity. The situation was different for the mgp_default IP: pinging it from other nodes failed, even though this overlay network was supposed to provide inter-container communication in Docker Swarm. This distinction was important. ICMP reachability of the address in ingress remained available, while connectivity to the replica through mgp_default was broken. Therefore, the problem was not the container itself or a complete network failure; it was a specific break in the network path used to reach that replica. What Docker Logs Showed The dockerd logs on the affected node provided another useful signal. They contained repeated messages related to Swarm network synchronization: Shell Jun 02 11:08:42 node-router2 dockerd: time="2025-06-02T11:08:42.462965233Z" level=warning msg="bulk sync to node 091d01df89ae failed: failed to send a TCP message during bulk sync: dial tcp 10.170.0.145:7946: i/o timeout" Jun 02 11:17:02 node-router2 dockerd: time="2025-06-02T11:17:02.581936915Z" level=error msg="Bulk sync to node 091d01df89ae timed out" The logs also contained NetworkDB stats entries: Shell Jun 02 11:07:08 node-router2 dockerd: time="2025-06-02T11:07:08.012509772Z" level=info msg="NetworkDB stats node-router2(2642aeb5172b) - netID:q9athb3moj967oxmb3y8xw4lj leaving:false netPeers:58 entries:13563 Queue qLen:2 netMsg/s:7" These messages do not prove a specific bug or root cause, but they indicate problems with exchanging network state between nodes. Combined with the endpoint being unreachable through mgp_default, they strengthened the working hypothesis of an overlay-networking issue. This was an important diagnostic signal: the Docker daemon was running, and the container was started, but the cluster’s network state was not ideal and was likely becoming inconsistent during node-to-node state exchange. Investigating the Hypotheses Hypothesis 1: The container itself was the problem. This hypothesis was not confirmed. A curl request inside the container to http://10.29.72.232:8080/actuator/health returned {"status":"UP"}, and ss -tuln showed that the service was listening on the required port. Hypothesis 2: The problem was port publishing or binding. This hypothesis was not confirmed. Even though port 10005 was reachable externally, that did not explain why Traefik could not reach port 8080 over the overlay network. The container worked locally, while the issue occurred on the network path between nodes. Hypothesis 3: Traefik itself was the problem. Probably not. Traefik was not the source of the failure; it was the first component that detected that the backend was unavailable from its node. Hypothesis 4: The Docker Swarm overlay network was the problem. This became the primary working hypothesis. The replica was reachable from inside its container but was not properly reachable from other nodes through its overlay IP. Shell curl http://10.29.72.232:8080 nc -vz 10.29.72.232 8080 The curl request worked locally, while nc -vz 10.29.72.232 8080 failed from other nodes. This indicated that traffic was not being routed correctly through the overlay network. An MTU/VXLAN issue was also considered. This is a plausible class of problems for Docker overlay networks, especially when there is an MTU mismatch or a Path MTU Discovery issue. In this case, however, it remained only a hypothesis rather than a confirmed root cause. Recovery We restored the problematic replica by forcing Docker Swarm to recreate the task through a rolling update: Shell docker service update --force \ --update-parallelism 1 \ --update-order start-first \ --update-delay 30s \ --detach=true \ mgp_router-app --force triggered an update even though neither the image nor the service configuration had changed. Swarm recreated the task for the affected replica and reattached it to mgp_default; after that, Traefik could reach it again over the overlay network. The rollout parameters were selected to avoid affecting service availability: --update-parallelism 1 – update one replica at a time;--update-order start-first – start the new task before stopping the old one;--update-delay 30s – give the new task time to start and pass its checks. Restarting Docker on the node and placing the node into drain mode were considered fallback measures, but they were not required to resolve this incident. Why It Happened The exact upstream root cause cannot be stated without additional evidence. However, the most well-supported explanation is that a local network-state inconsistency occurred in the Docker Swarm overlay network. As a result, one replica remained operational inside its container, but became unreachable through its overlay IP. The most plausible explanations include: the overlay network on the node was not fully ready when the container started;an issue in Docker libnetwork;an outdated VXLAN routing table;an iptables issue or a race condition while updating network rules;a node restart without restarting the Docker daemon;an issue with a Swarm endpoint that was not properly advertised or connected. These situations are particularly unpleasant because Docker continues to consider the container running and its internal health check green, while the network path to the replica is already partially broken. Key Takeaways The main conclusion is simple: a healthy container does not necessarily mean a reachable replica. This is critical in production because: a local curl inside the container does not prove external reachability;an internal health check does not guarantee that Traefik can reach the backend;Swarm may remain formally healthy even when one replica has dropped out of normal network connectivity. Therefore, in such clusters, it is necessary to check not only container state but also inter-node reachability over the overlay network. What to Monitor To prevent similar incidents, it is useful to monitor not only service health but also the cluster’s network plane. The minimum set of checks includes: reachability of container overlay IPs from other nodes;health checks through Traefik;bulk sync and NetworkDB errors in Docker logs;recurring issues on the same node;consistency between a container’s local health and its external reachability. It is also useful to have a script that specifically checks connectivity between nodes through the overlay network, not just through ingress or localhost. Conclusion This case demonstrated several important points. First, a container can be healthy locally while remaining unreachable externally. This is especially important for services that handle user traffic. Second, Traefik is a good early indicator of network problems. It can reveal that a backend on one node has stopped responding. Third, Docker Swarm overlay networking can degrade locally. In that case, the cluster may appear operational, while one replica has already dropped out of normal routing. Fourth, such incidents must be diagnosed along the network path, not only through application state. First, check the container from the inside, then verify reachability from other nodes, then inspect Docker logs, and only after that draw conclusions about the root cause.

Visual regression testing has a dirty secret: most of the failures it reports are not bugs. They are anti-aliasing differences, sub-pixel layout shifts, font-hinting changes between OS versions, a shadow that moves three pixels, or a GPU driver that rounds a gradient differently. Every one of those trips a pixel-diff, a human triages it, marks it "not a real defect," and re-baselines the golden image. Multiply that by every screen, every platform, every appearance mode, and the maintenance cost of the test suite quietly exceeds the cost of the bugs it catches. I have spent years leading rendering-quality engineering for systems that ship across iOS, macOS, visionOS, and CarPlay, and the conclusion I keep arriving at is that pixel comparison is the wrong abstraction for validating a rendered scene. The question we actually care about is not "is this frame byte-identical to the golden image?" It is "does this frame show the right thing?" Those are different questions, and the gap between them is where semantic visual validation lives. Why Pixel-Diffing Fails at Scale Pixel-level comparison — whether raw MSE or a perceptual metric like SSIM — measures signal similarity, not meaning. SSIM (Wang et al., 2004) was a real improvement over mean-squared error because it models luminance, contrast, and local structure the way the human visual system roughly does. But it is still a low-level statistic. It will happily flag a difference no user could ever perceive, and it has no concept of whether the content of the scene is correct. Two Structural Problems Make This Worse as You Scale The False-Positive Tax A rendering pipeline is nondeterministic at the pixel level across environments. Anti-aliasing, text rasterization, sub-pixel positioning, and GPU-dependent gradient rounding all vary by device and OS build without any defect being present. A strict pixel-diff treats all of this as failure. Loosen the threshold to suppress the noise, and you start missing the defects you actually care about. There is no threshold that separates "shadow rounded differently" from "the bridge failed to render" — because to a pixel-diff, both are just pixels that changed. Combinatorial Baseline Explosion A single view does not need one golden image. It needs one per platform, per resolution, per light/dark appearance, per dynamic-type size, per locale. For a 3D map surface — with a Mercator-to-globe transition, buildings, terrain, labels, and navigation overlays — the baseline matrix is enormous, and every intentional design change invalidates all of it at once. The suite spends more engineering time being re-baselined than finding bugs. The insight is that these are not tuning problems. They are a category error: we are using a similarity metric to answer a semantic question. The Architecture of Semantic Validation Semantic visual validation replaces "compare pixels to a golden image" with "extract meaning from the frame and check it against intent." A multimodal model looks at the rendered frame and answers structured questions about what it contains, and the test asserts on those answers. The pipeline has four stages: stageresponsibilityimplementation notes Capture Grab the rendered frame deterministically Native rendering-API hooks; capture at a fixed camera pose, seed, and time-of-day so the scene is reproducible even if pixels are not Normalize Prepare the frame for the model Downscale to the model’s input resolution; strip volatile overlays (clocks, FPS counters); redact any user data before it leaves the device Reason Extract semantic content and judge it A vision-language model (e.g., Qwen-VL) run with constrained/structured decoding, returning a typed verdict — not free text Decide Convert the verdict into a CI outcome Parse structured output; apply calibrated thresholds; emit pass / fail / uncertain The important architectural decision is at the Capture stage: you must make the scene deterministic (fixed pose, seed, lighting) even though the pixels are not. That is what lets a probabilistic model produce stable decisions. Implementation: Structured Output and CI Determinism The hardest objection to using an LLM in a test pipeline is nondeterminism — a test that returns different answers on the same input is worse than useless. You defeat this with four disciplines: pin the model version, decode at temperature 0, force structured (schema-constrained) output, and gate only on high-confidence verdicts while routing uncertainty to humans rather than failing the build. Python # Reference implementation - illustrative, not drop-in. API names follow the # Qwen-VL / DashScope pattern; adapt to your serving stack. What matters is the # structure (schema-constrained output, deterministic decoding, a calibrated # confidence gate, and a three-way verdict), not the specific client. from pydantic import BaseModel, Field MODEL = "qwen-vl-2.5" # pin the exact version; a model upgrade is a test change CONFIDENCE_GATE = 0.85 # derived from calibration (see below), not guessed class Verdict(BaseModel): passed: bool confidence: float = Field(ge=0.0, le=1.0) defects: list[str] # e.g. ["building geometry missing", "label overlap"] rationale: str def validate_frame(frame, expected_context: str) -> str: """Return 'pass', 'fail', or 'uncertain'. Never raises on ambiguity.""" result = client.infer( model=MODEL, image=frame, prompt=( f"You are validating a rendered map frame. Expected: {expected_context}. " "Report geometry defects, missing elements, and label collisions." ), response_schema=Verdict, # constrained decoding -> always valid, typed JSON temperature=0.0, # deterministic decoding seed=42, ) # Low confidence never hard-fails CI -> avoids a flaky pipeline. if result.confidence < CONFIDENCE_GATE: return "uncertain" # -> human review queue; pixel-diff tripwire still runs return "fail" if not result.passed else "pass" Two points readers should not miss. First, response_schema (constrained decoding) is what makes the output parseable every time — you are not regexing prose, you are getting a validated object. Second, even at temperature=0, token-level output is not guaranteed bit-identical across hardware and batching. That is fine, because you are not asserting on tokens — you are asserting on a decision (passed) that is stable well away from the confidence gate. Treat the model as a classifier, not a text generator. Model Evaluation and Calibration A confidence score the model emits is meaningless until you calibrate it. You build a labeled ground-truth set — frames you have manually classified as correct or defective — and measure the model against it: precision, recall, and a reliability diagram (does "0.9 confidence" actually mean right 90% of the time?). The confidence gate above is chosen from the precision-recall curve to hit a target false-negative rate, because in QA a missed defect costs more than a false alarm. Calibration is also environment-specific. A model tuned for standard daylight scenes will be over- or under-confident on high-contrast night rendering. Maintain per-environment operating thresholds, and re-run calibration whenever you change the model version, the prompt, or the rendering environment — each of those is a change to the test's behavior and deserves the same rigor as changing test code. Handling False Positives and False Negatives Semantic validation trades one failure mode for two. It largely eliminates the pixel-diff false-positive tax (perceptually-invisible differences stop failing the build). But it introduces false negatives (the model misses a subtle defect) and model false positives (the model hallucinates a defect that is not there). You manage this with layered defenses: Tiered validation: Keep a cheap pixel-diff as a tripwire for regions that must be exact — legal text, logos, brand color swatches — where semantic tolerance is wrong. Use the model for everything spatial and contextual.Three-way verdicts: 'uncertain' is a first-class outcome. It routes to a human review queue; it never fails the build. This is what keeps the pipeline non-flaky.Standing ground-truth audits: Periodically re-score the model against the labeled set and track precision/recall over time. A drop is your early warning that a model or prompt change regressed the validator itself. CI/CD Integration In the pipeline, semantic validation runs as a stage that only invokes the model on views whose rendering inputs actually changed — you do not pay for inference on untouched screens. A high-confidence 'fail' blocks the merge with the model's defects list attached to the check. 'uncertain' posts to a review queue and does not block. Every run logs the model version, prompt hash, confidence, and verdict, so a failing check is reproducible and explainable rather than an opaque "the AI said no." Cost, Latency, Privacy, and Observability Cost: Model inference is dollars-per-call, whereas pixel-diff is free. The tiered strategy is the control: cheap checks first, model only on changed or ambiguous views, batched where possible. Latency: Pixel-diff is milliseconds; VL inference is seconds. Run it asynchronously and in parallel across views so wall-clock stays flat, and reserve full semantic passes for pull requests rather than every commit. Privacy: Frame buffers can contain user content (a map centered on someone's home, a real address in a label). Normalize and redact on-device, and run inference inside your own trust boundary — never ship a user-bearing frame to an external API. This is a hard architectural constraint, not a policy checkbox. Observability: Track the `uncertain` rate, the false-positive/negative rate against the audit set, and confidence drift over time. These are the health metrics of the validator; when they move, you investigate before you trust another green check. A Migration Strategy From Pixel-Based Testing Do not flip the switch. Migrate in shadow mode: Run in parallel, gate on nothing. Keep pixel-diff authoritative; run semantic validation alongside it, logging verdicts without blocking.Mine the disagreements. Every case where the two disagree is a labeled example — pixel-diff-failed-but-semantically-fine is your false-positive tax made visible; semantically-failed-but-pixel-passed is a defect your old suite would have missed.Calibrate on that data. Set your confidence gate and per-environment thresholds from the real disagreement distribution.Flip gating for high-confidence defects only. Let the model block merges where it is confident; keep pixel-diff as the exact-match tripwire.Retire redundant baselines. As confidence holds, drop the golden-image matrix for everything except exact-match regions. At each step, you have a fallback and real data, so the migration is reversible and evidence-based rather than a leap of faith. Conclusion The move to semantic validation is a change in what we assert, not just which tool we run. Pixel-diffing asks "did the pixels change?" — a question whose answer is almost always "yes, and it doesn't matter." Semantic validation asks "is the scene correct?" — the question we actually have.

A Kafka recovery often looks successful before it actually is. The replay job exits. Consumer lag drops. Error rate returns to normal. The dashboard is green enough for everyone to want the incident closed. Then somebody checks the downstream projection, account balance, inventory state, enrichment table, or audit report and finds the real problem: data moved, but state did not converge. That is the failure this article is about. Consumer lag is a progress signal, not a recovery verdict. It proves position movement. It does not prove terminal outcomes, idempotent side effects, owned failures, or source-to-target convergence. If an incident review ends with “lag went back to zero,” the team has proved movement, not recovery. Declare the Recovery Cohort First The first observability decision is not which metric to graph. It is what work the recovery run is supposed to prove. A useful cohort names: the topic, partitions, and offset rangethe event types included in the runthe business-key scopethe expected record countthe recovery reasonthe ownerand the stop conditions A compact contract object makes that scope reviewable: YAML recovery_run: id: rec-2026-09-05-01 reason: projection_drift_after_consumer_restart owner: billing-platform-oncall run_type: incident_repair source: topic: billing-events event_types: - InvoicePosted - CreditApplied offsets: - partition: 8 from: 492100 to: 492881 - partition: 9 from: 388010 to: 388522 business_scope: key_type: account_id selector: incident_account_hashes_2026_09_05 expectations: expected_record_count: 1294 reconciliation: source_of_truth: billing_ledger checks: - account_day_balance_matches - no_missing_decision_events - no_unowned_final_failures safety: dry_run_required: true max_retry_attempts: 3 stop_on_reconciliation_mismatch: true The object does not have to be beautiful. It has to be unambiguous. During an incident, ambiguity is expensive: “replay the billing topic” is not a scope, and “check the dashboard” is not evidence. The cohort also forces a design decision: do you need partition/offset proof, business-key proof, or both? Offsets are excellent for Kafka lineage. Business keys are what downstream systems usually reconcile. A serious recovery plan connects them instead of pretending one replaces the other. Count Terminal Outcomes, Not Processed Records Processed count is a dangerous comfort metric. A record can be processed and still be wrong. It can be skipped correctly. It can be suppressed as a duplicate. It can be retried. It can land in a dead letter queue. It can be quarantined because replaying it would violate a business rule. The recovery run should account for outcomes explicitly: OutcomeTerminal?MeaningAPPLIEDYesThe expected side effect was performed.UPDATEDYesExisting target state was corrected or brought current.SKIPPEDYesThe record was intentionally ignored under a documented rule.DUPLICATE_SUPPRESSEDYesIdempotency prevented a repeat side effect.FAILED_RETRYABLENoProcessing failed and remains inside retry policy.FAILED_FINALYesProcessing failed and needs owner action.QUARANTINEDYesThe record is retained but should not be replayed without domain review. The key equation is simple: expected = applied + updated + skipped + duplicate_suppressed + failed_final + quarantined Retryable failures should not count as terminal until the retry policy is exhausted or the run is stopped. If the math does not close, the recovery has not been proven. Keep Metrics Bounded Recovery runs create tempting labels: run IDs, account IDs, order IDs, offset ranges, incident IDs. Put those blindly into metrics and the observability system becomes the next incident. Use metrics for bounded dimensions and put high-cardinality detail in decision logs, traces, audit tables, or exemplars. MetricBounded labelsPurposekafka_recovery_records_expectedtopic, event_type, run_typeDefines the denominator.kafka_recovery_records_unaccountedtopic, event_type, run_typeShows records not yet terminal.kafka_recovery_records_terminal_totaloutcome, event_type, run_typeShows how records completed.kafka_consumer_duplicate_suppressed_totalconsumer_group, event_type, run_typeProves replay did not duplicate side effects.kafka_recovery_reconciliation_mismatchescheck_name, run_typeBlocks false recovery success. That split matters. Metrics tell you that recovery is unhealthy at a class level. The decision record tells you which run, offsets, business keys, and owners are involved. Emit Decisions You Can Inspect A useful recovery log does not just say that a handler ran. It says why a record took a path. JSON { "recoveryRunId": "rec-2026-09-05-01", "eventId": "evt-1001", "businessKeyHash": "bkh_7f3a", "topic": "billing-events", "partition": 8, "offset": 492881, "eventType": "InvoicePosted", "outcome": "DUPLICATE_SUPPRESSED", "reason": "idempotency_key_exists", "schemaVersion": "v4", "consumerGroup": "usage-aggregator", "runType": "incident_repair", "decidedAt": "2026-09-05T18:42:31Z" } Notice what is missing: raw payloads, customer identifiers, tokens, and private business data. Recovery logs should explain decisions without becoming a second data-leak surface. The Java side can stay small. The following Java 17/Spring/Micrometer sketch is illustrative, not a complete production service. It assumes repository, meterRegistry, objectMapper, and log are class fields: Java record RecoveryDecision( String recoveryRunId, String eventId, String businessKeyHash, String topic, int partition, long offset, String eventType, RecoveryOutcome outcome, String reason, String consumerGroup, String runType, Instant decidedAt ) {} void emit(RecoveryDecision decision) { repository.insert(decision); Tags tags = Tags.of( "event_type", decision.eventType(), "run_type", decision.runType(), "outcome", decision.outcome().name() ); meterRegistry .counter("kafka_recovery_records_decision_total", tags) .increment(); if (decision.outcome() != RecoveryOutcome.FAILED_RETRYABLE) { meterRegistry .counter("kafka_recovery_records_terminal_total", tags) .increment(); } log.info("kafka_recovery_decision={}", toJson(decision)); } String toJson(RecoveryDecision decision) { try { return objectMapper.writeValueAsString(decision); } catch (JsonProcessingException error) { return "{\"serialization_error\":\"recovery_decision\"}"; } } That emitter is not the recovery system. It is the instrumentation boundary. It makes the record decision visible in the durable decision log, the metric stream, and the incident timeline. Reconciliation Is the Verdict Layer The hardest part of recovery observability is deciding what “correct” means. For some domains, a count is enough. For most serious domains, it is not. A count can match while the wrong keys, amounts, versions, or timestamps remain wrong. The source of truth must be named. The tolerance window must be named. Late arrivals, clock skew, eventually consistent sinks, and manual exceptions must be handled explicitly. Otherwise reconciliation becomes another dashboard panel that looks precise while hiding the policy decision. For example, if the recovery cohort is supposed to repair account/day billing state, compare the source ledger to the target projection for the same cohort. This is PostgreSQL-style SQL; use an equivalent pattern for your database: PLSQL WITH expected AS ( SELECT account_hash, business_date, SUM(amount_cents) AS expected_amount_cents FROM billing_ledger WHERE recovery_run_id = :recovery_run_id GROUP BY account_hash, business_date ), actual AS ( SELECT account_hash, business_date, amount_cents AS actual_amount_cents FROM billing_daily_projection WHERE recovery_run_id = :recovery_run_id ) SELECT COALESCE(e.account_hash, a.account_hash) AS account_hash, COALESCE(e.business_date, a.business_date) AS business_date, COALESCE(e.expected_amount_cents, 0) AS expected_amount_cents, COALESCE(a.actual_amount_cents, 0) AS actual_amount_cents FROM expected e FULL OUTER JOIN actual a ON e.account_hash = a.account_hash AND e.business_date = a.business_date WHERE COALESCE(e.expected_amount_cents, 0) <> COALESCE(a.actual_amount_cents, 0); Also check whether every selected event produced a decision: PLSQL SELECT c.event_id FROM kafka_recovery_cohort_event c LEFT JOIN kafka_recovery_decision_log d ON c.recovery_run_id = d.recovery_run_id AND c.event_id = d.event_id WHERE c.recovery_run_id = :recovery_run_id AND d.event_id IS NULL; Those queries are intentionally plain. Recovery confidence should not depend on a heroic debugging session. It should depend on executable checks that can run before anyone is asked to close the loop. Alert on Evidence Failure, Not Just Kafka Symptoms Recovery alerts should catch bad recovery, not only broken Kafka. Page when terminal outcomes do not close after the deadline, reconciliation fails after the check has completed, or an active recovery run stops making progress. Do not page simply because an in-progress recovery still has work remaining. YAML groups: - name: kafka-recovery rules: - alert: KafkaRecoveryUnaccountedRecords expr: | kafka_recovery_records_unaccounted > 0 and on(topic, event_type, run_type) kafka_recovery_deadline_reached == 1 for: 15m labels: severity: page annotations: summary: "Kafka recovery has overdue unaccounted records" - alert: KafkaRecoveryRunTooOld expr: | time() - kafka_recovery_last_progress_timestamp_seconds > 1800 and on(topic, event_type, run_type) kafka_recovery_run_active == 1 for: 5m labels: severity: ticket annotations: summary: "Kafka recovery run has stopped making progress" The alert is not the verdict. It tells the operator where to look. The verdict should still come from the recovery evidence record. Test the Evidence Path Recovery safety should not only be described. It should be testable. At minimum, test the evidence path with a small matrix: TestInjectionExpected proofDuplicate replayProcess the same event_id twice.One APPLIED, one DUPLICATE_SUPPRESSED, no second side effect.Missing decisionSelect an event into the cohort but skip emission.no_missing_decision_events fails.Projection driftChange target state after replay.Reconciliation returns mismatched keys.Poison recordSend a record that violates a domain rule.Outcome is QUARANTINED with owner and reason.Stalled runStop progress after partial terminal outcomes.Unaccounted-record or stale-progress alert fires. Run these tests in two places. First, keep a small contract-test suite around the recovery library or consumer adapter so decision emission cannot regress. Second, run a replay drill in a non-production environment with a deliberately small cohort. The drill should prove that the metrics, decision log, reconciliation query, alerts, and final verdict all agree before the same pattern is trusted during an incident. Recovery Definition of Done Before closing a Kafka recovery run, require evidence for each item: CheckEvidenceCohort declaredTopic, partitions, offsets, event types, expected count, owner, and stop conditions are recorded.Terminal math closesExpected records equal terminal outcomes, excluding retryable in-flight failures.Idempotency is visibleDuplicate suppressions and side-effect decisions are emitted as bounded metrics and durable decision records.Failure ownership existsFinal failures and quarantined records have reasons, owners, and next actions.Reconciliation passedThe named source of truth matches the target state for the approved scope, or exceptions are explicitly owned.Verdict recordedThe run ends as RECOVERED, RECOVERED_WITH_EXCEPTIONS, BLOCKED, QUARANTINED, or FAILED. This full pattern is not free. Structured decision logs add write volume. Reconciliation consumes compute. Audit tables need retention and access control. Quarantine creates human workflow. There are also cases where the full pattern is too much. A stateless consumer that only updates an internal cache may not need decision tables, reconciliation jobs, and formal verdicts. Start with the recovery definition of done, then choose the minimum evidence that proves the business risk you actually have. Kafka gives teams a powerful recovery primitive because the log can be replayed. That does not make recovery safe by itself. Lag tells you the consumer moved. Recovery evidence tells you whether the system became correct again.

The first deploy of the new pipeline shipped cleanly: metrics looked right, the deployment circuit breaker came back green, and the on-call channel stayed quiet. Two days later, a downstream consumer team pinged us. Their dashboard showed the same five-minute snapshot reported twice, by two upstream task runs that did not agree on the values. Both runs had completed normally. The problem was an assumption ECS makes about the tasks it replaces. The pipeline ran as an ECS service on Fargate with a desired count of one. The container looped on a five-minute cadence: at each boundary it pulled inputs, computed a network state snapshot for that window, wrote the result to S3, and slept until the next boundary. One iteration took about ninety seconds. The service used a textbook rolling update: minimum healthy percent of 100, maximum percent of 200, deployment circuit breaker enabled. The configuration was correct for the workload it was designed for. A rolling deploy of a stateless web server and a rolling deploy of a long-running task are different problems. For a web server, each request is a discrete unit and the load balancer routes around tasks being drained. For a task that runs ninety seconds and produces an output, a deploy that lands mid-iteration leaves two instances alive: the old one finishing under the old image and the new one starting under the new image. Both compute the same window, read the same inputs, and target the same output destination. Both exit cleanly, so the circuit breaker sees nothing. Downstream, there are now two versions of the same logical work, and the storage layer cannot tell which one is authoritative. The fix lives in the output layer rather than the deployment configuration, because ECS has no visibility into what a task's output means. Idempotent keying at the write boundary lets the system tolerate duplicate execution by collapsing duplicate outputs into a single canonical result. It adds no coordination between tasks, does not slow deployments, and holds for unplanned restarts as well as planned rollouts. Why ECS Rolling Deployments Break Long-Running Tasks Two parameters govern an ECS service deployment: minimumHealthyPercent and maximumPercent. At 100 and 200, the scheduler starts tasks under the new task definition before stopping any existing ones, and it considers the deployment complete once the new task count reaches the desired count and the old tasks have stopped. For a stateless service behind a load balancer, this is the right model. For a task that produces output, it assumes the running task can be replaced without semantic consequence, and that assumption does not hold. Here is how it played out for us. The service task, running revision N, started an iteration at the top of a five-minute window. About thirty seconds in, a deploy rolled out revision N+1. ECS started the new task and left the old one running until its stop timeout. The new task began an iteration for the current window on startup, because that is what its loop does. The old task wrote its result. The new task wrote its result. Both writes went to S3 under keys that included the task's launch timestamp, so they landed as two objects. The downstream consumer listed the window prefix, found two candidates, and picked one by listing order. Different consumers picked differently, and their metrics disagreed. This failure does not show up in load tests. Health checks pass and the deploy dashboard stays green. The bug surfaces later, somewhere downstream, as an inconsistency in metrics, a flickering chart, or a model that disagrees with itself. By the time someone correlates the symptom to a deployment window, several deploys have passed, and consumers have already mixed the duplicate writes into their reads. We found it only because the consumer team's anomaly detector tripped on the divergence. Code block 1 shows the Terraform that produces this behavior by default. The task definition and the deployment configuration are both correct. The problem is the interaction between the rolling update and the application's write semantics, which Terraform cannot see. Code block 1: ECS service and task definition with a by-the-book rolling update Plain Text # ECS service running a single long-lived task. The container loops on a # 5-minute cadence and writes one network state snapshot per window. # The deployment configuration is by-the-book and still leaves two tasks # alive, computing the same window, during every rolling deployment. resource "aws_ecs_task_definition" "network_state_job" { family = "network-state-snapshot" network_mode = "awsvpc" requires_compatibilities = ["FARGATE"] cpu = "2048" memory = "4096" execution_role_arn = aws_iam_role.task_execution.arn task_role_arn = aws_iam_role.task.arn container_definitions = jsonencode([{ name = "snapshotter" image = "${aws_ecr_repository.snapshotter.repository_url}:${var.image_tag}" essential = true environment = [ { name = "OUTPUT_BUCKET", value = aws_s3_bucket.snapshots.bucket }, { name = "JOB_VERSION", value = var.job_version }, { name = "WINDOW_SECONDS", value = "300" } ] }]) } # Rolling update: start the new task before stopping the old one. With a # 90-second iteration, the old task is usually mid-write when the new task # starts its own iteration for the same window. resource "aws_ecs_service" "network_state" { name = "network-state" cluster = aws_ecs_cluster.main.id task_definition = aws_ecs_task_definition.network_state_job.arn desired_count = 1 launch_type = "FARGATE" deployment_minimum_healthy_percent = 100 deployment_maximum_percent = 200 deployment_circuit_breaker { enable = true rollback = true } network_configuration { subnets = var.private_subnet_ids security_groups = [aws_security_group.task.id] } } Detecting Duplicate Output Without Already Knowing You Have It Detection is often harder than the fix, because by the time the symptom is loud enough to investigate, the data has already been blended downstream. Two approaches cover it. Forensic scanning looks backward. Scan output destinations for multiple writes against the same logical work identifier. For S3, correlate PutObject records from CloudTrail or S3 access logs with the task ARN that wrote them: two distinct task ARNs writing to the same window key is the signature of this failure. The first write is not the problem; the second write from a different task is. Preventive instrumentation gives a live signal. Stamp every write with the writing task's ECS task ARN and task definition revision, then scan for windows with more than one distinct task ARN. Run this hourly during and after any deployment to the service. Code block 2 shows the audit script. It lists objects under a window prefix, groups by logical window, and reports windows with writes from more than one task. Code block 2: Audit script that flags duplicate task writes per window Python """ Audit logical-key duplicates across an S3 prefix where outputs are tagged with the writing ECS task ARN. Surfaces cases where two tasks wrote to the same logical job-key window, typically during rolling deployments. Run this hourly during and after any deployment to the service. Exit code 1 if duplicates are found, 0 otherwise, so it composes with CI. """ import boto3 from collections import defaultdict from datetime import datetime, timedelta, timezone S3 = boto3.client("s3") BUCKET = "network-state-snapshots" def window_of(key: str) -> str: # Expected key shape: snapshots/<yyyy-mm-dd>/<HHMM>/... parts = key.split("/") return "/".join(parts[1:3]) def audit(hours: int = 24) -> dict[str, set[str]]: cutoff = datetime.now(timezone.utc) - timedelta(hours=hours) paginator = S3.get_paginator("list_objects_v2") dupes: dict[str, set[str]] = defaultdict(set) for page in paginator.paginate(Bucket=BUCKET, Prefix="snapshots/"): for obj in page.get("Contents", []): if obj["LastModified"] < cutoff: continue head = S3.head_object(Bucket=BUCKET, Key=obj["Key"]) task_arn = head["Metadata"].get("x-task-arn", "unknown") dupes[window_of(obj["Key"])].add(task_arn) return {w: arns for w, arns in dupes.items() if len(arns) > 1} def main() -> int: dupes = audit() for window, arns in sorted(dupes.items()): print(f"DUPLICATE window={window} writers={len(arns)}") for arn in sorted(arns): print(f" writer={arn}") print(f"scanned windows with duplicates: {len(dupes)}") return 1 if dupes else 0 if __name__ == "__main__": raise SystemExit(main()) Idempotent Keying as the Structural Fix Idempotency at the output layer means a duplicate execution cannot produce a duplicate output. Two tasks doing the same logical work resolve to the same key, so the storage layer keeps exactly one result. The mechanism has two halves: a deterministic key derived from the work identity, and a conditional write that makes the second write a no-op. A workable scheme follows three rules. First, the key derives deterministically from the logical work identity, never from the task's own attributes. Task ARNs, launch timestamps, and container IDs all change between the old and new task, so any key that includes them defeats the purpose. The window boundary, the job version, and a fingerprint over the pinned input versions are stable across both tasks. Second, the key must be content-aware enough that genuinely different inputs never collide. Pin the input versions at the window boundary and hash them into the key. Two tasks that read the same pinned inputs produce the same key. A correction that arrives later pins different input versions and lands on a different key, which is correct: it is new work, not a retry. Third, publish through a conditional write. S3's If-None-Match conditional write fails the PUT when the key already exists, which is as close to a compare-and-swap as object storage offers. The first writer wins; the second sees the precondition failure and treats it as success, because the object it would have written is byte-identical to what is already there. Code block 3 shows the Rust implementation we landed on. The key is derived from the trigger window, a job version, and a fingerprint over the pinned input table versions. Code block 3: Deterministic output key derivation Rust //! Two task executions for the same trigger window and same pinned inputs //! always produce the same key, and therefore collide at the storage layer //! where the conditional write enforces single-writer semantics. use sha2::{Digest, Sha256}; pub struct WorkIdentity { pub window_start_epoch: u64, pub job_version: u32, pub source_table_versions: Vec<(String, u64)>, // resolved as of the window boundary, never read live } impl WorkIdentity { pub fn output_key(&self) -> String { let mut h = Sha256::new(); h.update(self.window_start_epoch.to_be_bytes()); h.update(self.job_version.to_be_bytes()); let mut pinned = self.source_table_versions.clone(); pinned.sort(); for (table, version) in &pinned { h.update(table.as_bytes()); h.update(version.to_be_bytes()); } let digest = h.finalize(); format!("snapshots/{}/{:x}/state.parquet", self.window_start_epoch, digest) } } The other half is the conditional write. The deterministic key collapses duplicate logical work into a single destination, and the conditional write turns the race between the two tasks into a single winner with no coordination between them. Code block 4: Conditional write with conflict retry Rust //! Of two tasks racing toward the same key, one PutObject succeeds and the //! other sees PreconditionFailed (treated as a no-op) or, if both are in //! flight together, ConditionalRequestConflict (retried, then no-op). use std::time::Duration; use aws_sdk_s3::{Client, primitives::ByteStream}; const MAX_ATTEMPTS: u32 = 5; pub async fn publish_once( client: &Client, bucket: &str, key: &str, payload: Vec<u8>, task_arn: &str, ) -> anyhow::Result<()> { for attempt in 1..=MAX_ATTEMPTS { // If-None-Match: "*" tells S3 to fail the PUT if the key already exists. // The check is server-enforced, so it holds even when client-side // coordination is impossible, as during a rolling deployment. let result = client .put_object() .bucket(bucket) .key(key) .body(ByteStream::from(payload.clone())) .if_none_match("*") .metadata("x-task-arn", task_arn) .send() .await; match result { Ok(_) => return Ok(()), Err(e) if is_precondition_failed(&e) => return Ok(()), // already published: no-op Err(e) if is_conditional_conflict(&e) && attempt < MAX_ATTEMPTS => { tokio::time::sleep(Duration::from_millis(200 * attempt as u64)).await; continue; } Err(e) => return Err(anyhow::Error::from(e)), } } Err(anyhow::anyhow!( "conditional write to {key} did not settle after {MAX_ATTEMPTS} attempts" )) } Trade-Offs and Adoption Guidance Idempotent keying has costs. The deterministic hash adds single-digit milliseconds per job when it runs over input metadata, and more if it hashes payloads; keep the fingerprint over pinned input versions, not over data. The conditional write adds one round trip with retry backoff, which is negligible next to a ninety-second iteration. A less obvious cost is operational flexibility. Once outputs are content-keyed, you cannot rerun a job and overwrite its result by accident, which is the point, but it also means deliberate replays need a new job version. Build the job version into the key from the start so replays are a first-class operation rather than a hack around the scheme. Storage growth is the second consideration. Duplicate writes are dropped rather than overwritten, but successive job versions and new logical windows each add objects. A lifecycle policy on the output prefix keeps this bounded; the keying scheme makes objects immutable, so expiring old windows is safe. Roll this out incrementally. Start with the audit script running hourly against production to measure how often duplicate writes actually happen. Then ship the keyed write path behind the old path in shadow mode: compute the key, log it, but keep writing to the old location. When the shadow keys match the expected single-winner behavior across a few deployments, cut over. The audit script becomes your ongoing verification. The general lesson is that long-running compute under rolling deployments is a regime where the orchestrator cannot give you correctness, because the orchestrator does not understand your output. Pushing correctness down to the write boundary, where the semantics live, turns a deployment-time race into a property of the data. If you take one habit away from this, take this one. Write the property-based test for your keying scheme before you build the rest of the pipeline. Two executions with the same work identity must produce the same key; two executions with different pinned inputs must not. That test is the contract everything else hangs on. References AWS, "Amazon S3 now supports conditional writes," AWS What's New, August 2024. https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/ AWS, "Deploy Amazon ECS services by replacing tasks (rolling update)," AWS Documentation. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-ecs.html AWS, "How to prevent object overwrites with conditional writes," Amazon S3 User Guide. https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html AWS SDK for Rust, aws-sdk-s3 crate documentation. https://docs.rs/aws-sdk-s3/ proptest crate, property-based testing for Rust. https://docs.rs/proptest/