Using Middleware in LangChain: Recovering From Tool Failures
Running an OpenStreetMap Tile Server for the Whole Planet on Modest Hardware
Why Local Lift Does Not Mean System Growth A marketplace team tests a new ranking policy. On the home surface, the share of user activity rises from 30% in control to 45% in treatment: a 15-percentage-point lift. Assignment is balanced, tracking is complete, and the confidence interval excludes zero. The result looks ready to ship. Then the team expands the analysis. Search activity falls by almost 15 points, browse is unchanged, and total activity per user remains flat. The treatment did not create more engagement. It changed where that engagement happened. The home surface gained activity, but the product as a whole did not. Nothing about the local calculation is wrong. The A/B test correctly answered, “Did activity increase on home?” The mistake was treating that answer as proof that total activity increased. A precise local estimate can still support the wrong launch decision when users, suppliers, or product surfaces compete for a shared resource. Teams often respond by reaching for a switchback, cluster randomization, or separate budgets. Before changing the design, answer two simpler questions: Did the treatment increase the system total or redistribute an existing outcome?Were treatment and control isolated at the level where the shared resource operated? This article presents a practical workflow for answering both questions. It reconciles local changes against a system total using the randomized-user denominator, then screens for coupling inside the resource pools shared by the experiment arms. Why Shared Constraints Change the Meaning of Lift An ordinary user-level experiment assumes that one user’s assignment does not change another user’s outcome. This condition is part of the Stable Unit Treatment Value Assumption (SUTVA). It becomes questionable when experimental units compete through a shared constraint: The first three examples can create interference between experimental units. The fourth can create cannibalization between product surfaces even when user assignment itself is clean. These mechanisms are related, but they are not the same claim: Interference means that treatment assigned to one unit changes another unit’s outcome.Cannibalization means that a measured gain comes from a corresponding loss elsewhere in the system.Incrementality means that the treatment increases the outcome the system ultimately values, rather than changing its location or attribution. Assignment, sample-ratio, exposure, and telemetry checks still come first. Once they pass, the next review asks whether the success metric covers the system outcome and whether the randomization unit isolates the rollout effect. Use a Local-to-System Reconciliation The local-to-system reconciliation brings those checks into one review. It has five parts: Define the shared constraint and system boundary. Identify the supply, inventory, budget, attention, or capacity through which outcomes can move, and decide what must be counted for the launch decision.Measure each local change. Estimate treatment-control differences for the components inside that boundary.Reconcile the total. Aggregate those components with one denominator tied to the randomization unit.Check isolation inside shared pools. Examine whether the arms move together inside markets, regions, seller clusters, or budget groups.Classify the evidence. Treat the result as incremental, redistributive, isolation-sensitive, or inconclusive before deciding whether to ship or redesign. Figure 1: Decision flow from local lift through system reconciliation and isolation review Start with the boundary. It must correspond to the business decision, and every component inside it must use a compatible unit of analysis. A convenient total that excludes the surface likely to lose value simply moves the blind spot. Reconcile Outcomes per Randomized User Here is what the reconciliation looks like in a deterministic synthetic experiment with three mutually exclusive surfaces: The home result is real. It just does not show incremental engagement: search loses almost exactly what home gains. Calculate the treatment-control change for every affected surface using the randomized-user denominator. The system change is the sum of those component changes. The total absolute movement shows how much behavior shifted across surfaces. When movement is large, but the system change is close to zero, the treatment is primarily redistributive. The implementation preserves the randomized-unit denominator: Python arm_size = df.groupby("variant")["user_id"].nunique() surface_per_user = ( df.groupby(["variant", "surface"])["activity"] .sum() .unstack(fill_value=0) .div(arm_size, axis=0) ) system_per_user = surface_per_user.sum(axis=1) surface_delta = ( surface_per_user.loc["treatment"] - surface_per_user.loc["control"] ) system_delta = surface_delta.sum() local_movement = surface_delta.abs().sum() Counting event rows instead would let an engagement-changing treatment alter its own denominator. Per-user outcomes preserve the connection between the reported effect and the randomization design. Figure 2: Control and treatment have different surface shares while total outcome remains flat. What To Watch For Do not interpret shares without levels. A surface share can fall while its absolute activity rises if the total grows.Make the parts add up. Overlapping or incomplete surfaces do not produce a clean system total. State what is excluded, and avoid double counting.Keep the statistical review. Reconciliation is a diagnostic, not a standalone hypothesis test. Component and system estimates still need confidence intervals that respect the experiment design. Use the system total as part of the Overall Evaluation Criterion (OEC): the metric or combination of metrics used to judge experimental success and connect short-term measurement to longer-term goals (Kohavi, Tang, and Xu, Chapter 7). Home activity explains where behavior moved; it should not decide the launch when the objective is total engagement or marketplace value. Once the metric boundary is clear, the next question is whether the experiment arms were truly isolated. Check for Coupling Inside Shared Resource Pools Reconciliation can show that value moved. It cannot tell you whether treatment and control affected one another. A useful risk screen asks whether both arms remain related inside pools that share supply, inventory, or budget. Do not correlate raw market means. Large markets can have high outcomes in both arms for stable reasons unrelated to the experiment. Remove each market’s pre-period baseline and compare changes: Python market_means = ( df.groupby(["market_id", "period", "variant"])["revenue"] .mean() .unstack(["period", "variant"]) .dropna() ) control_change = ( market_means[("post", "control")] - market_means[("pre", "control")] ) treatment_change = ( market_means[("post", "treatment")] - market_means[("pre", "treatment")] ) coupling = control_change.corr(treatment_change) The synthetic data returns a correlation of 0.873. Treat that as a review signal, not an “interference detected” verdict. Common shocks, seasonality, measurement changes, and omitted market characteristics can produce the same pattern. The shared-pool coupling screen has one job: challenge the randomization unit. Define resource pools before looking at results, report the number and size of those pools, and add uncertainty or randomization inference. Even with those safeguards, the screen can identify risk in the original estimate; it cannot correct that estimate. Convert the Evidence Into a Launch Decision At this point, the team has two different forms of evidence: whether value moved across the system and whether the experiment arms may have shared the same constraint. Together, they lead to different actions: The categories should not be collapsed into a single reliability score. Redistribution and interference answer different questions. A treatment can reallocate activity across surfaces without contaminating control, or it can interfere through shared supply even when the measured system total is positive. Only an isolation problem necessarily requires a new experiment design. Match the Design to the Constraint Shared supply over time and region: Use a switchback to randomize the market state rather than individual transactions. DoorDash uses this pattern for dispatch experiments (DoorDash Engineering).Fixed campaign budgets: Separate the budgets used by treatment and control. LinkedIn describes this budget-split design for its advertising marketplace (LinkedIn Engineering).Marketplace or network spillovers: Randomize clusters rather than individuals, while accounting for the loss of power that comes with fewer independent units (Holtz et al.).Competition between listings or surfaces: Keep local metrics as diagnostics, but base the decision on incremental platform value. Airbnb describes this distinction when accounting for booking cannibalization (Airbnb Engineering). These are responses to a diagnosed constraint, not a checklist to apply to every experiment. Key Takeaways Name the shared supply, inventory, budget, or attention constraint before choosing the randomization unit.Define the system boundary before reviewing local lift.Calculate every component with the randomized-unit denominator.Compare absolute outcomes before interpreting changes in surface share.Treat shared-pool coupling as a design-review signal, not proof of interference.Redesign only when the evidence shows that treatment and control were not isolated at the level where the system operates. A p-value can tell you whether the observed treatment-control difference is compatible with a null hypothesis under a statistical model. It cannot tell you whether the experiment measured the right boundary or isolated the right unit. The final question is not whether the treatment won its local comparison. It is whether launching the treatment would make the system better. Reference Implementation The deterministic example behind the reported values is available in the runnable Python script. A broader open-source implementation of related diagnostics is available in the Expguard repository and its archived release. These implementations are optional; the workflow applies independently of either codebase. References and Further Reading Kohavi, R., Tang, D., and Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, 2020.DoorDash Engineering, Switchback Tests and Randomized Experimentation Under Network Effects.Holtz et al., Reducing Interference Bias in Online Marketplace Pricing Experiments.Li et al., Interference, Bias, and Variance in Two-Sided Marketplace Experimentation.LinkedIn Engineering, Budget-Split Testing: A Trustworthy and Powerful Approach to Marketplace A/B Testing.Liu, Mao, and Kang, Trustworthy Online Marketplace Experimentation With Budget-Split Design.Airbnb Engineering, How Airbnb Measures Future Value To Standardize Tradeoffs.
Code review is the highest-leverage activity on most engineering teams, yet it is also the most expensive in terms of senior-developer time. What if you could offload a first pass to an LLM that understands Java idioms, common anti-patterns, SOLID principles, and security pitfalls — and get back structured, actionable feedback before a human even opens the PR? In this post, we will build exactly that: a self-contained Java application that accepts a Java source snippet, sends it to an LLM (OpenAI GPT-4o or a locally running Ollama model — your choice), and returns a structured list of code-quality findings with severity levels and suggested fixes. What you will build: A CodeReviewAssistant Java service backed by LangChain4j, which analyses Java code and returns structured improvement suggestions — runnable in under 10 minutes on your laptop. Prerequisites ToolVersionNotesJava JDK17+Records & sealed types usedMaven or GradleAny recentMaven snippets shown hereLangChain4j0.36.xLatest stable at time of writingOpenAI API key or Ollama—We show both No Spring, no frameworks — this is plain Java, so you can drop it into any stack. Step 1: Add LangChain4j to Your Project LangChain4j is the Java-native counterpart of the Python LangChain library. It gives you a clean, type-safe API over multiple LLM providers — OpenAI, Anthropic, Azure, Hugging Face, Ollama, and more — with zero vendor lock-in. Add the BOM and the provider module you want: XML <!-- pom.xml --> <dependencyManagement> <dependencies> <dependency> <groupId>dev.langchain4j</groupId> <artifactId>langchain4j-bom</artifactId> <version>0.36.2</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependencies> <!-- Core --> <dependency> <groupId>dev.langchain4j</groupId> <artifactId>langchain4j</artifactId> </dependency> <!-- Choose ONE provider below --> <!-- Option A: OpenAI (GPT-4o, GPT-3.5-turbo, …) --> <dependency> <groupId>dev.langchain4j</groupId> <artifactId>langchain4j-open-ai</artifactId> </dependency> <!-- Option B: Ollama (runs locally, no API key needed) --> <dependency> <groupId>dev.langchain4j</groupId> <artifactId>langchain4j-ollama</artifactId> </dependency> </dependencies> Tip — local-first development: Start with Ollama + codellama or mistral so you can iterate without burning API credits. Switch to OpenAI GPT-4o for production quality by changing one line. Step 2: Model the Review Output as a Java Record LangChain4j's Structured Output feature can instruct the LLM to respond with JSON that maps directly onto a Java type. We define our domain model first: Java // Finding.java public record Finding( String severity, // "CRITICAL" | "HIGH" | "MEDIUM" | "LOW" | "INFO" String category, // e.g. "Null Safety", "Performance", "Security" String description, // human-readable explanation String suggestedFix // concrete code-level recommendation ) {} // CodeReviewResult.java import java.util.List; public record CodeReviewResult( String summary, List<Finding> findings ) {} Using Java Records keeps the model immutable and removes boilerplate. LangChain4j will automatically build the JSON schema from these types and include it in the system prompt. Step 3: Define the AI Service Interface LangChain4j's AI Services abstraction is the cleanest part of the library. You declare an interface, annotate it with prompts, and the library generates a proxy at runtime — no boilerplate, no manual HTTP calls. Java import dev.langchain4j.service.SystemMessage; import dev.langchain4j.service.UserMessage; import dev.langchain4j.service.V; public interface CodeReviewService { @SystemMessage(""" You are a senior Java engineer performing a thorough code review. Analyse the provided Java code for: - Correctness and potential bugs - Null safety issues - Performance anti-patterns - Security vulnerabilities (injection, insecure randomness, etc.) - SOLID principle violations - Readability and naming conventions Return ONLY a JSON object matching the CodeReviewResult schema. Be concise but actionable. Severity: CRITICAL, HIGH, MEDIUM, LOW, INFO. """) @UserMessage("Review this Java code:\n\n```java\n{{code}\n```") CodeReviewResult review(@V("code") String javaCode); } How it works: The @SystemMessage sets the LLM's persona and instructions. The @UserMessage is the per-call prompt. @V("code") is a template variable — LangChain4j substitutes it at call time. Zero boilerplate. Step 4: Wire It All Together Now we create the LLM model, register our output type, and build the AI Service proxy. The entire wiring fits in one short class: Java import dev.langchain4j.model.openai.OpenAiChatModel; import dev.langchain4j.model.ollama.OllamaChatModel; import dev.langchain4j.service.AiServices; public class CodeReviewAssistant { public static void main(String[] args) { // ── Choose your provider ────────────────────────────────────────── // Option A: OpenAI (set env var OPENAI_API_KEY) var model = OpenAiChatModel.builder() .apiKey(System.getenv("OPENAI_API_KEY")) // never hardcode keys! .modelName("gpt-4o") .responseFormat("json_object") // enforce JSON mode .temperature(0.2) // low temp = consistent output .build(); // Option B: Ollama running locally on port 11434 // var model = OllamaChatModel.builder() // .baseUrl("http://127.0.0.1:11434") // .modelName("codellama") // .build(); // ───────────────────────────────────────────────────────────────── CodeReviewService reviewer = AiServices.builder(CodeReviewService.class) .chatLanguageModel(model) .build(); // ── Sample code to review ───────────────────────────────────────── String javaCode = """ public class UserService { private static Map<String, String> cache = new HashMap<>(); public String getUser(String id) { if (cache.containsKey(id)) return cache.get(id); String sql = "SELECT * FROM users WHERE id = " + id; // ... execute query return null; } } """; CodeReviewResult result = reviewer.review(javaCode); // ── Print results ───────────────────────────────────────────────── System.out.println("Summary: " + result.summary()); System.out.println(); for (Finding f : result.findings()) { System.out.printf("[%s] %s — %s%n", f.severity(), f.category(), f.description()); System.out.printf(" Fix: %s%n%n", f.suggestedFix()); } } } Step 5: Sample Output Running the above against that deliberately flawed snippet produces output like: Plain Text Summary: The code has a critical SQL injection vulnerability, a thread-safety issue with the shared HashMap cache, and returns null without documentation. [CRITICAL] Security — String concatenation builds a raw SQL query, making the method vulnerable to SQL injection attacks. Fix: Use PreparedStatement with parameterised queries: "SELECT * FROM users WHERE id = ?" with stmt.setString(1, id) [HIGH] Concurrency — HashMap is not thread-safe. Concurrent reads/writes will cause data corruption in multi-threaded environments. Fix: Replace with ConcurrentHashMap<>() or add explicit synchronisation. [MEDIUM] Null Safety — getUser() returns null without documentation. Callers have no contract to guard against NPEs. Fix: Return Optional<String> and document the empty-case semantics. [LOW] Performance — cache.containsKey() + cache.get() performs two lookups. Fix: Use cache.computeIfAbsent(id, k -> fetchFromDb(k)) instead. Going Further: Production Hardening 1. Streaming Responses for Large Files For large classes, swap the return type of your AI Service method to TokenStream and use onNext() / onComplete() to stream findings incrementally rather than waiting for the full response. 2. Memory and Conversation History LangChain4j's MessageWindowChatMemory lets you maintain multi-turn conversations — useful if you want the developer to ask follow-up questions about a specific finding. Java CodeReviewService reviewer = AiServices.builder(CodeReviewService.class) .chatLanguageModel(model) .chatMemory(MessageWindowChatMemory.withMaxMessages(10)) .build(); 3. RAG: Ground Findings in Your Own Style Guide Use EmbeddingStore + ContentRetriever to inject excerpts from your team's coding standards into every review. The LLM will then apply your own rules rather than falling back on generic advice. 4. CI/CD Integration Wrap the service in a small CLI tool (using picocli), wire it into your GitHub Actions or Jenkins pipeline, and post findings as PR comments via the GitHub REST API. This enables a first-pass AI review before any human reviewer opens the PR. Key Takeaways LangChain4j AI Services let you express LLM interactions as plain Java interfaces — no HTTP plumbing, no JSON parsing.Structured Output (records + JSON schema) eliminates fragile string parsing; you get strongly typed results directly.Provider-agnostic design means you swap one builder line to move between OpenAI, Ollama, Anthropic, or Azure with zero business logic changes.Keep API keys in environment variables — never in source code.Start local (Ollama) to iterate fast, then promote to a hosted model for production quality. Further Reading LangChain4j official docsAI Services tutorialOllama — CodeLlama modelOpenAI Structured Outputs guide
Data federation was sold on a simple promise: query data where it already lives, across Snowflake, Databricks, and the rest of the stack, without making another copy. No more nightly ETL jobs staging the same customer table in four different warehouses. No more arguments about which copy was authoritative. Point the query at the source and get an answer. For the workloads it was designed to support, that promise mostly held. A human analyst could pull a report, or a dashboard could refresh each morning, without moving the data. The query ran. If a number looked slightly off, someone could notice, check it, or just shrug and move on. Federation was never built for an autonomous agent that accepts the answer and acts on it without a second look. That assumption is about to be tested, and it will not survive. The Copy Problem Was Never the Coherence Problem Federation solved a real issue: duplicated data creates duplicated cost, duplicated drift, and duplicated arguments about which copy to trust. Querying in place through an open table format like Iceberg, or through Snowflake's and Databricks' respective sharing layers, removes that duplication. What it does not do is guarantee that the answer coming back represents one coherent state of the world. Those are different problems, and the industry has spent the last few years solving the first while quietly assuming the second came for free. It didn't. Here's where that assumption breaks, specifically once an agent, not a person, is the one issuing the query. Five Clocks, One Query A federated join across a warehouse, a lakehouse, and a live API does not pull from one system at one moment. It pulls from three systems, each with its own refresh schedule and consistency model. None of them is coordinated with the others, and no distributed transaction aligns their data to a single point in time. The result looks like a snapshot, but it is closer to a collage assembled from whatever each source had available when the query arrived. Take a churn-risk agent deciding whether a customer needs a retention offer. It queries the CRM for account status, the support platform for open tickets, and the billing system for payment history. The CRM record was synced four hours ago. A support ticket that just closed has not propagated yet. The billing system runs on a separate pipeline and is current to the last five minutes. The agent receives one joined row and treats it as one fact: this customer is a churn risk, so act now. It has no way to know that a third of the row was already stale by the time it arrived. A human analyst who sees an unexpected number will usually pause, check the timestamp, and perhaps run the query again. An agent will not do that unless someone explicitly designed it to. It accepts the federated result as ground truth and acts on it. Figure 1. Federated queries combine data from sources with different freshness levels, creating a result that appears unified but may not represent one coherent moment. Authorization for a Session, Not a Stream of Calls Federated and shared-access models assume a human user: log in, pass an authorization check at the session boundary, and then query within that approved window. The model is intentionally coarse because a human session does not need authorization reevaluated every few seconds. An agent works differently. A fraud-review agent querying a shared Iceberg table of transaction records may use a service account that received table-level read access once, during setup. From there, it can issue several hundred narrow, autonomous queries an hour while working through flagged cases. Nothing in the architecture rechecks whether that specific task should be allowed to see each record it requests. A standing grant designed for someone opening a dashboard twice a day now supports a process capable of issuing hundreds of queries without supervision. Runtime access enforcement has to happen on every call, not just at the start of a session. Most federation architectures deployed today were not built for that level of granularity or volume. Federation makes the query possible. It doesn't make the answer agreeable. Enterprises increasingly run multiple agents, often on different vendor stacks: Databricks tooling for one and Snowflake Cortex for another. Each agent reasons independently about shared entities such as customers and orders. In this setting, the case for federated, in-place access is not efficiency. The question is whether the agents are even seeing versions of the same entity that can be reconciled. Consider a supply chain agent on one platform and a finance agent on another, both querying the same order record. One must decide whether to expedite a shipment. The other must decide whether to release a payment. The supply chain agent reads the order during a write and catches the status enum mid-transition. The finance agent reads it ninety seconds later, after the write is complete. Both queries were technically correct based on the data available when they ran. Yet the agents now hold different beliefs about the same order. Neither has any signal that the other agent, working from the same source, reached a different conclusion. Figure 2. Two agents querying the same shared data can reach conflicting conclusions when they read it at different points in time. What Actually Has to Change Three things, in order. This is the same sequencing argument behind the AI-Readiness Quadrant framework I laid out: identity first, then access enforcement, then observability. Skipping ahead doesn't work any better here; it just means patching symptoms instead of fixing what's underneath. Figure 3. Federation places particular strain on runtime access enforcement and lineage and observability within the AI Readiness Quadrant. Federated results need explicit freshness and consistency metadata that an agent can read and reason about. The system cannot assume that every source is synchronized when that synchronization was never guaranteed. If a query joins three systems, the response should report how stale each source was when the query ran, not just return the joined rows. Access enforcement also needs to happen at the level of each call. Every request should be evaluated against the data it touches instead of relying on a session that was authorized once and left open. That is a heavier operational burden than most sharing and federation layers were designed to support. It still needs to happen before agent query volumes exceed what today's coarse access models can handle. When agents from different vendors need to agree on the same entity, that agreement must be established deliberately through the identity and catalog layers. It does not emerge automatically because every system adopted the same open table format. Iceberg makes the data physically reachable by more than one engine. It does not make two independent agents agree on what that data means. The Trust Problem Federation Was Never Asked to Solve Federation got adopted to kill a cost and integration problem: too many copies, too much ETL, too much arguing about which table was current. It solved that. What it's being asked to solve now, mostly by default and without anyone deciding it deliberately, is a trust problem: whether an autonomous system acting on a federated answer is acting on something coherent. It was never built for that job. Fixing the foundation underneath agentic systems means treating that gap as the thing to solve first, before scaling the number of agents that depend on an architecture that was never asked to guarantee coherence in the first place.
OpenRewrite in a Real Enterprise Migration Recently, we modernized a 9-year-old enterprise application that involved two major changes happening together: Java 8 → Java 25Spring Boot 2.3.3 → Spring Boot 3.3.4 At the same time, we were also moving the application from a monolithic architecture toward microservices. One of the biggest challenges was the sheer volume of source-code changes. With Java 17 being the minimum required baseline for Spring Boot 3 and major namespace changes such as javax.* → jakarta.*, manually modifying a large codebase would have been time-consuming and risky. We decided to use OpenRewrite to automate the mechanical parts of the migration. The result was significant: 2,384 Java files were refactored in minutes. The manual effort that we estimated at roughly 5–7 days was reduced to automated transformation followed by code review and testing. But the most interesting part wasn't simply how much OpenRewrite changed. It was understanding what it could automate, what it couldn't, and where human judgment was still required. What Is OpenRewrite? OpenRewrite is an automated refactoring framework designed to transform source code using recipes. Instead of manually searching through thousands of files and applying the same transformation repeatedly, you define a recipe and let OpenRewrite perform the transformation consistently across the codebase. For large migrations, this is particularly useful for changes that are: RepetitiveSyntacticWell-understoodApplicable across many files The important distinction is that OpenRewrite automates code transformation. It does not understand every business rule or runtime behavior in an enterprise application. Our Migration The application was originally running on: Plain Text Java 8 Spring Boot 2.3.3 We moved toward: Plain Text Java 25 Spring Boot 3.3.4 One of the major changes introduced by Spring Boot 3 was the migration from the older Java EE namespace to Jakarta EE. For example: Java import javax.persistence.Entity; became: Java import jakarta.persistence.Entity; When you have thousands of source files, this is exactly the kind of transformation that should not be performed manually. Configuring OpenRewrite For our migration, we used the OpenRewrite Maven plugin: XML <plugin> <groupId>org.openrewrite.maven</groupId> <artifactId>rewrite-maven-plugin</artifactId> <version>5.41.0</version> <configuration> <exportDatatables>true</exportDatatables> <activeRecipes> <recipe>org.openrewrite.java.spring.boot3.UpgradeSpringBoot_3_3</recipe> </activeRecipes> </configuration> <dependencies> <dependency> <groupId>org.openrewrite.recipe</groupId> <artifactId>rewrite-spring</artifactId> <version>5.20.0</version> </dependency> </dependencies> </plugin> We then executed: Shell mvn rewrite:run That's it. The tool analyzed the codebase and applied the configured recipe. What Happened? The results were better than we expected. 2,384 files changed. OpenRewrite modified approximately 2,384 Java files. The majority of the changes were mechanical framework migration changes. Some of the important changes included: 1. javax → jakarta A large number of imports were automatically migrated. For example: Java javax.persistence.* became: Java jakarta.persistence.* This is one of the areas where automated refactoring provides tremendous value. A developer doesn't need to manually locate every affected file. 2. Spring Batch API Changes Our application also contained Spring Batch code. Older APIs such as: Java JobBuilderFactory StepBuilderFactory needed to be migrated toward the newer builder APIs: Java JobBuilder StepBuilder OpenRewrite handled a significant portion of these changes automatically. This was particularly useful because Spring Batch code was spread across the application. 3. Other Spring Boot Migration Changes The Spring Boot migration recipes can also handle many other framework-level changes, including API and configuration changes. The exact transformations depend on the Spring Boot version and the recipe version being used. The OpenRewrite Spring Boot migration documentation provides the applicable transformation set and before/after examples. How Much Time Did It Save? This was probably the biggest practical benefit. Our initial estimate for manually performing the large-scale refactoring was approximately 5–7 days. OpenRewrite completed the core source transformation in minutes. We then spent approximately 4 hours reviewing the generated changes. This is an important distinction. We did not treat the generated code as automatically correct. The workflow was: Plain Text OpenRewrite ↓ Automated transformation ↓ Code review ↓ Compilation ↓ Unit/integration testing ↓ Manual fixes This turned the migration from a large manual refactoring exercise into a much more controlled process. But OpenRewrite Did Not Migrate Everything This was the most important lesson from our experience. OpenRewrite was extremely effective for structural and mechanical changes. However, there were areas where application-specific knowledge was required. The DAO layer was one example. Our application contained a significant amount of legacy SQL code. SQL Built Using String Concatenation Some queries were constructed using direct string concatenation. For example: Java String sql = "SELECT ... WHERE customer_id = " + customerId; These situations required manual intervention. The problem wasn't simply changing Java syntax. We needed to understand the SQL, parameter handling, and application behavior. OpenRewrite could transform Java source code, but it could not safely determine the business intent behind every SQL statement. Legacy Named Parameters We also discovered another interesting problem during testing. Some legacy DAO code used parameters such as: SQL :1 :2 The code compiled successfully after the migration. However, during runtime testing, some of these parameters failed to bind correctly. We had to manually replace them with explicit names such as: SQL :customerId :employeeId This was a good example of why successful compilation does not mean a migration is complete. The problem surfaced only when the application actually executed the affected code. What About Java 25? There is another important point when planning a Java 8 → Java 25 migration. Spring Boot migration and Java migration are related, but they are not the same transformation. OpenRewrite provides Java migration recipes as well. For example, there is a dedicated recipe for migrating to Java 25: Java org.openrewrite.java.migrate.UpgradeToJava25 The Java 25 migration guide currently describes this recipe as part of the rewrite-migrate-java recipe module. The important thing is to select the recipes appropriate for the actual migration path. For example: Plain Text Java 8 ↓ Java 17 ↓ Java 21 ↓ Java 25 or combine applicable migration recipes depending on the project's starting point and the changes required. A Java 25 recipe should not be treated as a replacement for the Spring Boot migration recipe. They address different parts of the modernization. Useful OpenRewrite Recipes for Java/Spring Modernization One of the things I like about OpenRewrite is that the migration doesn't have to stop with one recipe. Depending on the application, you can combine recipes for different modernization goals. Some useful categories include: Spring Boot For example: Java org.openrewrite.java.spring.boot3.UpgradeSpringBoot_3_3 This was the recipe we used for our Spring Boot 3.3 migration. OpenRewrite also maintains Spring Boot migration recipes for newer Spring Boot versions. Java For Java modernization, OpenRewrite provides recipes such as: Java org.openrewrite.java.migrate.UpgradeToJava25 The Java migration recipes can target language/API modernization separately from Spring-specific changes. Other Framework/API Migrations The OpenRewrite recipe ecosystem also contains recipes for other common modernization scenarios. This is where OpenRewrite becomes more interesting than a simple search-and-replace tool. Recipes can encode relationships between APIs and perform structured transformations rather than blindly replacing text. What OpenRewrite Is Good At Based on our experience, I would categorize the work this way. Excellent Candidates for Automation Plain Text javax → jakarta Known API replacements Deprecated API migrations Repetitive source changes Large-scale mechanical refactoring Areas Requiring Engineering Judgment Plain Text Business logic SQL semantics Security behavior Application-specific configuration Runtime behavior This distinction is critical. You don't want to ask: "Can OpenRewrite migrate my application?" A better question is: "Which parts of my migration can be safely automated?" Automation Does Not Remove Testing One of the biggest mistakes would be to run: Shell mvn rewrite:run and assume the migration is finished. Our experience showed the opposite. The automated changes actually allowed us to spend more time on the areas that mattered. Instead of having developers spend days changing thousands of imports and APIs, we could focus our engineering effort on: Reviewing generated changesFixing application-specific issuesValidating SQL behaviorTesting Spring Batch behaviorValidating security configurationRunning integration testsInvestigating runtime failures The tool didn't remove engineering work. It moved engineering effort from repetitive refactoring to higher-value validation. Our Takeaway The most impressive part of OpenRewrite wasn't that it changed 2,384 files. It was that it allowed us to approach a large migration differently. Without automation: Plain Text Thousands of files ↓ Manual changes ↓ 5–7 days ↓ High risk of inconsistency With OpenRewrite: Plain Text Migration recipe ↓ 2,384 files transformed ↓ Minutes ↓ Code review ↓ Testing ↓ Manual fixes where required The lesson I took from the migration is simple: Automate the mechanical work. Keep human judgment for the work that requires application context. OpenRewrite doesn't eliminate the complexity of a Java/Spring modernization. But for a large legacy application, it can dramatically reduce the amount of repetitive work engineers need to perform — and make the overall migration much more controlled and repeatable.
TL; DR: Deciding What to Build Is the New Job, Now that Code Is No Longer a Constraint Agentic coding has removed the cost gate that used to stop product teams from chasing every idea at once, and the constraint has moved to product judgment: the ability to tell what is worth building. AI-Assisted product discovery can prepare that judgment by reading far more customer evidence than any product person could. Such a discovery agent analyzes three families of sources, checks eight patterns, and argues both against and for your current direction. The A3 Framework has a name for this work: Assist, the agent analyzes and prepares, and the humans decide. The framework also helps to define the agent limits with the A3 Handoff Canvas – what the agent does, what the human does, which data we use, etc. – and the AI Definition of Done – what a quality AI output “looks like” – before the first run. Thesis: This article on AI-assisted product discovery explains how product teams can use AI agents to compare customer feedback, usage data, and stakeholder requests with their product vision, product goal, product roadmap, and past decisions, flagging drift, bias, and sunk costs, while humans keep the decision under the A3 Framework’s Assist category. Disclaimer: I read Charniak/McDermott’s book on “Artificial Intelligence” decades ago; of course, I use AI for research, translations, proofreading, challenging story arcs and article structures, creating artifacts such as graphics, and summarization. It is a production tool, not a substitute for thinking. Both AI Product Camps Hand Off the Part That Just Became Scarce Product people seem to be splitting into two camps over agentic coding. The first camp is celebrating: If a coding agent can build a working version of an idea in an afternoon, why argue about which of five ideas to pursue when you can build all five and let the market tell you which one sticks? (Yes, assuming you have the credits to keep feeding tokens into your coding agents.) The second camp has drawn the opposite conclusion. It expects the next agent to take over the choosing as well, preferably with a dashboard and a confidence score attached to each recommendation. Both camps are wrong for the same reason: each hands off the one capability that has just become scarce. Throwing spaghetti at the wall has become affordable in tokens. Yet it remains as unaffordable as ever in focus, maintenance, and the attention of customers who have to make sense of five half-finished features. An agent that chooses for you, on the other hand, has taken over the part of product work that nobody can delegate without also delegating the accountability for the outcome, whatever the org chart says about who owns the product. Coding Used to Be the Gate That Forced Us to Choose For decades, writing software was expensive and slow, and that expense did a job nobody had assigned to it: It acted as a sanity check and a cost gate. When building a feature took a team several weeks, the product people had to decide which idea deserved those weeks, and a weak idea often died in a prioritization discussion before anyone wrote a line of code. The gate was crude, and plenty of bad ideas slipped through anyway (usually the ones with a senior sponsor with budget authority); nevertheless, it forced a choice. Agentic coding has removed much of that gate, and the theory of constraints predicts what happens next. Eliyahu Goldratt’s five focusing steps end with a warning that is easy to skip: Once you elevate a constraint, “the weak link may not remain weakest,” as the Theory of Constraints Institute puts it, and the constraint moves somewhere else in the system. In product development, it moves to judgment: product taste, the ability to read customer evidence, and the discipline to say no to an idea that a coding agent could build by Friday. Andrew Ng made the same observation on the No Priors podcast in August 2025, as Business Insider reported: “The bottleneck is deciding what do we actually want to build.” Photography went through a comparable shift. When every frame of film cost money and darkroom time, photographers composed carefully before they pressed the shutter, so the cost of a frame forced the judgment upfront. Digital cameras removed that cost and moved the work to selecting and editing, where shooting thousands of mediocre frames mostly buries the person who has to pick the few worth printing. Product teams now face that editor’s problem: A coding agent can produce a working version of almost any idea, and someone still has to decide which version deserves a customer’s attention. Not everyone is as gifted as Steve Jobs, even though plenty of people still try to be the product manager in charge. Every Roadmap Is a Snapshot of What You Knew Last Month If judgment is the new constraint, it helps to look at the system product teams use to exercise it. The sketch below comes from my Product Owner classes, where I use it to show the challenges product people face when they manage a product backlog. (This AI-assisted product discovery article covers its left half.) The product vision sets the direction, backed by a product strategy that describes how the organization intends to turn that vision into reality. At the operational level, the strategy becomes something more tangible and nearer-term: a product roadmap or, if your team borrows from Scrum, a Product Goal. (Plenty of product teams use elements of Scrum and would deny practicing it, which is completely fine; we are not paid to practice Scrum but to solve our customers’ problems within the given constraints while contributing to the organization’s sustainability.) Together, these artifacts form the narrow neck of a funnel: Every idea, request, and data point pours into its wide mouth, and only what serves the vision and the goal passes the neck and earns a place in the work. Below the neck, the sketch shows where the surviving ideas go: into a hypotheses backlog for validation through experiments or, when an idea does not serve the goal right now, into an anti-product backlog; both destinations depend on a human decision. Each of these artifacts is a snapshot. A product roadmap, a product goal, or a product backlog represents your understanding of your customers, the market, and your competitors at the moment you wrote it down. I like to put it this way: Based on what we know now, this is our best abstraction of how we can create customer value, and it may change as we learn more. Scrum makes the same point when it calls the Product Backlog “an emergent, ordered list” in the Scrum Guide: Its composition and order change as the team learns something, which also lets the team judge earlier decisions in hindsight and correct course. A snapshot is only useful if someone compares it with what has happened since, which raises two practical questions: Where does the new information come from, and who has the time to read it? What Pours Into the Funnel, and What the Agent Has to Remember Information reaches a product team through two kinds of sources, and both have grown faster than any team can read. Passive sources: Information arrives without the product team asking for it, for example: Messages from the contact form on your websiteTickets in the customer care department’s systemDaily traffic and user behavior data from your analytics platformProduct requirements from internal stakeholders, which arrive unrequested and usually with political weight attached. Active sources: The product team goes out and gets the information, for example: Notes from continuous user interviewsIdeas from the product people themselves, who work with the product every dayObservations of users in their natural habitat, solving the problems they chose your product to help withUser research such as user story mapping sessions with internal and external stakeholders. Nobody reads all of this every week, and that is the opening for an agent. Depending on the size of your organization, the agent runs daily or weekly, reads everything that comes in, and prepares what humans need for product decisions. However, two families of sources are not enough, and in my observation, this is where the typical “AI analyzes your customer feedback” pitch stops. An agent that only reads external evidence can tell you that customers dislike a feature; it cannot tell you that you have kept funding that feature for three quarters because of what you already spent on it. For that, the agent needs a third family. Organizational memory: The record of what the organization committed to and spent, for example: Current and previous product goals and product roadmap versionsNotes from past stakeholder reviews, and release notesThe anti-product backlog, including the reason for each rejected ideaPast stakeholder decisions, and the effort and money already spent on initiatives. The agent’s job, then, is to compare external evidence (what customers, users, and stakeholders say and do) with internal commitments (what the organization decided and spent). In most organizations, this memory lives in people’s heads, Slack threads, and Jira comments rather than in a tidy decision log, so start with what exists, for example, the current and previous product goals or product roadmaps and the raw notes of your last stakeholder reviews. Questions the agent cannot answer because records are missing are a finding in themselves, because they show where your organization doesn’t document knowledge or decisions about why it said no. Another boundary is the first question the A3 Framework asks of any task: Should AI touch it at all? Counting tickets, computing the revenue at risk from billing and CRM data, and drawing the dashboard are deterministic jobs that a BI tool or a spreadsheet already handles reliably. The agent reads those computed figures next to the unstructured text and connects the two, while the tools built for arithmetic stay with the arithmetic. Eight Questions for a Weekly AI-Assisted Product Discovery Check The agent works through eight questions in three groups, and each group needs different inputs. Drift, or has the world moved since we last looked: These questions need all three families, because drift only shows when you hold today’s evidence against yesterday’s commitments: Where does the data point to a change in customer perception or needs?Does our product roadmap or our product goal still align with what real customers say and do?Have we started moving in the wrong direction because we ignored earlier feedback from the market? (The agent can only answer this one if the organizational memory shows what the team saw at the time and decided to set aside.) Gaps and pain, or what customers keep telling us: These questions run on passive and active sources plus the figures your BI tool already computes: Which functional areas do customers bring up again and again that the product does not address?Which critical usability issues do customers complain about?Which customers threaten to cancel their subscriptions or purchases because the product disappoints them, and what is our risk exposure? (The exposure figure comes from billing data; the agent links it to what those customers wrote.) Self-deception, or whether we are fooling ourselves: These questions depend on organizational memory above all: Are the product strategy, the product roadmap, or the product goal biased, built to confirm a belief about the product’s future while the market moves in a different direction?Have we fallen victim to the sunk cost fallacy? In the self-deception group, an agent does better than a tired human reader. People usually struggle to spot bias in their own plans, and what they’ve already spent tends to argue louder than evidence of what customers want. Both patterns are old acquaintances; I described them, together with the persona-driven self-fulfilling prophecy and the habit of ignoring sales and customer support, in Product Discovery Anti-Patterns Leading to Failure. The exciting new part is that an agent can check for them every week, across every source, without getting tired or diplomatic. In AI-Assisted Product Discovery, the Agent Prepares the Decision, and the Product Team Makes It In terms of the A3 Framework, everything so far is Assist, which the original A3 Framework article sums up as “AI expands options. You own the outcome.” The agent reads the sources and prepares the briefs, and the product team decides. The moment the agent moves an idea into the hypotheses backlog, kills a roadmap item, or reorders the product backlog on its own, the work has crossed into Automate, which is the wrong category for a decision that depends on product judgment, which is the capability the shifted constraint now demands from humans. However, the preparation itself has a catch. Language models tend to agree with the person asking. Recent research suggests the problem has not gone away: Seo, Atanasova, and Augenstein tested 14 models on 42K questions and found that “all models are susceptible to user opinions regardless of size,” with models “most influenced by how confidently an opinion is stated, whom it is attributed to” (Claim Framing as a Driver of LLM Sycophancy, October 2026). A product goal handed to an agent is exactly that: a confidently stated position backed by the organization’s authority. So, I suggest running the analysis as two briefs. The first brief collects the evidence for the current direction, while the second brief starts from the assumption that the product goal is wrong and builds the strongest case against it from the same data, citing every source. The humans then read both side by side. (The study documents the agreeable behavior; the two briefs are my countermeasure, and they apply an old rule of mine: Never accept a model’s first answer.) The A3 Handoff Canvas is where the team writes this down before the first run: The inputs name the three families of sources, the outputs name the two briefs, and the stop rules say what the agent must never do, namely rank ideas, recommend a decision, or move anything. The AI Definition of Done then states what each brief must contain before anyone discusses it, and the difference is easiest to see with question 8, applied to a hypothetical custom reporting module that has been on the roadmap for a while. An Automate verdict, which is what you do not want: “The custom reporting module shows sunk cost characteristics. Recommendation: stop development.” What the product team reads instead, assembled from both briefs, which is what the AI Definition of Done asks for: Commitment: “The module has been on the product roadmap since the start of the year and received development time in every release cycle since.” (Source: roadmap versions)Spending: “Most of the recent iterations went into report templates and export filters.” (Source: release notes)Customer evidence: “Requests for the module come from long-standing accounts that want custom CSV formats; recent onboarding interviews do not mention it.” (Sources: support tickets, interview notes)Case for continuing (first brief): “Enterprise accounts up for renewal list custom reporting as an open requirement.” (Source: stakeholder review notes)Case for stopping (second brief): “Customers who canceled during the period cited friction in the core workflow, and observed users export raw data to spreadsheets instead of using the in-app reports.” (Sources: cancellation tickets, observation notes)Open question for the product team: “Can scheduled exports cover the enterprise renewals without funding another cycle of module development?” What happens next is a human call, and the options sit on the left side of the sketch: A finding can become a hypothesis in the hypotheses backlog and go to validation through an experiment; it can land in the anti-product backlog together with the reason why it does not serve the product vision or the product goal right now; it can trigger a change of the product goal itself; or the product team can dismiss it. Each of these destinations is a decision that the people accountable for the product make, preferably together. Five Ways AI-Assisted Product Discovery Goes Wrong A rule I apply to every AI initiative holds here as well: Do not apply AI on top of dysfunctional systems or processes. If a product team has no hypotheses backlog, runs no experiments, and keeps no anti-product backlog, the agent’s briefs land on nothing, and the following failure modes show what that looks like in practice: The loud-customer bias: Passive sources over-represent vocal customers, while customers who quietly cancel rarely fill out a contact form first. An agent reading tickets will give the most weight to whoever writes the most tickets, unless the product team tells it to also look at who stopped writing, for example, in usage data from accounts that stopped logging in.Sanitized records: When the notes of a stakeholder review say “stakeholders loved the demo” because nobody wanted to write down the objections, the organizational memory the agent reads has already been edited for comfort. The opposing brief can only argue from the records it receives, so a sanitized record produces a convincing case for the wrong conclusion. (Anyone who has watched a review turn into a performance knows how often this happens, or, to use a German proverb: Wer schreibt, der bleibt.)Report theater: The briefs look polished, people read the summary, and nobody opens the evidence. Polished AI artifacts quickly fool people into believing that the AI is doing the real job, and a discovery check that nobody questions may be worse than none, because it lends a bad decision the authority of data.The HiPPO override: The agent flags a misalignment, a stakeholder requirement with enough political weight wins anyway, and three weeks later, nobody reads the briefs anymore.The authority mismatch: When an agent holds customer evidence against commitments, drift at the operational level often points to a flawed assumption at the strategic level, in the product vision or the strategy. A product manager who can change the product goal but not the strategy then receives a finding without the authority to act on it, and the brief travels from inbox to inbox until it is out of date. So decide before the first run who receives which kind of finding and who takes a strategic finding to the people who own the product strategy; the task split of the A3 Handoff Canvas is the place to write that down. Amazon’s Fire Phone shows how expensive this gets when the highest-paid person is the CEO: Fast Company’s Austin Carr interviewed more than three dozen current and former employees, and a former engineering head said of the phone’s Dynamic Perspective feature, “We poured surreal amounts of money into it, yet we all thought it had no value for the customer,” adding, “Whenever anyone asked why we were doing this, the answer was, ‘Because Jeff wants it.'” (GeekWire’s summary of the Fast Company report, January 2015). Three months after launch, Amazon CFO Tom Szkutak disclosed a $170 million write-down, “primarily related to Fire Phone inventory valuation and supplier commitment costs” (GeekWire, October 2014). (The price did its share, too; Amazon’s devices chief, David Limp, later told Fortune, “We didn’t get the price right.”) AI-Assisted Product Discovery – Conclusion: Run One Discovery Check Before Your Next Stakeholder Review You do not need an elaborate setup to try this. Take the sources you already have, for example, last month’s support tickets and contact form messages, the notes from your last ten user interviews, and last quarter’s product roadmap, and run questions 2 and 7 as two briefs against your current product goal. Then check the evidence inside the product team first, in your next refinement session or a dedicated discovery check, before anyone outside the team sees it: Open the sources, look for what the agent misread, and decide which findings you are willing to defend. Only once the team owns the argument does it go to the next stakeholder review, as a token for discussion, never as a decision. And when your agent’s second brief makes a convincing case next month that customers have stopped caring about the feature your product roadmap is built around, who in your organization has the authority to act? Questions This Article on AI-Assisted Product Discovery Answers Why Does Product Judgment Become the Bottleneck When AI Agents Write Code? Agentic coding removed the cost gate that once forced product teams to choose between ideas. When building a feature took weeks, weak ideas often died in a prioritization discussion; now a coding agent can build a working version of an idea in an afternoon. As Goldratt’s theory of constraints predicts, the constraint moves elsewhere, to product judgment: deciding which idea deserves customers’ attention. Should an AI Agent Make Product Decisions? No. Under the A3 Framework, product discovery analysis belongs in the Assist category: the agent reads customer feedback, usage data, stakeholder requests, and the organization’s past decisions and prepares the evidence, while the product team decides. The A3 Handoff Canvas records the agent’s stop rules before the first run: no ranking of ideas, no recommendations, and no moving items in the product backlog. What Data Should a Product Discovery Agent Analyze? A product discovery agent needs three families of sources. Passive sources arrive unasked: contact form messages, support tickets, analytics data, and stakeholder requirements. Active sources come from the product team’s own work, such as user interviews, observations, and user story mapping. Organizational memory covers past product goals, roadmap versions, rejected ideas with reasons, and money already spent; without it, the agent cannot detect sunk costs. How Do You Stop an AI Agent From Confirming Your Roadmap? Run the analysis as two briefs. The first brief collects the evidence for the current product goal; the second assumes the goal is wrong and builds the strongest case against it from the same data, citing every source. Language models tend to agree with the person asking: Seo, Atanasova, and Augenstein (2026) found all 14 models they tested susceptible to user opinions, regardless of size. How Does AI-Assisted Product Discovery Fail? They fail in five recurring ways: they overweight vocal customers, read meeting notes sanitized for comfort, produce polished briefs nobody checks, lose to the highest-paid person’s opinion, and deliver strategic findings to people without strategic authority. Amazon’s Fire Phone shows the HiPPO override at the top: engineers kept building a feature they believed had “no value for the customer” because Jeff Bezos wanted it.
A few years back, I sat through a sprint review where a product manager and a senior developer spent twenty minutes arguing about one line in a ticket: "Users can reset their password." The developer had built exactly that. The PM expected a lockout after three failed attempts, an email confirmation, and a link that expired in an hour. Nobody was wrong, exactly. They just had two different pictures in their heads, and the ticket never forced anyone to draw theirs on paper. That gap between what people say and what they mean is why behavior-driven development exists. And it is also why picking a BDD tool is less about features and more about how your team talks. What BDD Is Really About Most people meet BDD through a tool, usually Cucumber, and assume the tool is the point. It is not. BDD is a way of agreeing on behavior before anyone writes code, using concrete examples that developers, testers, and business folks can all read. If you want a clean, no-fluff definition to share with your team, this explainer on the behavior-driven development framework is a good place to start. The framework is just what turns those agreed examples into something a machine can run. When teams skip the conversation and jump straight to writing feature files, they end up with slow, brittle tests written in a strange English dialect that only the automation engineer understands. I have seen this more than once, and it is not pretty. How a BDD Framework Works in Practice Almost every BDD framework follows the same basic shape. You describe behavior in plain language, usually with the Given, When, Then structure, and then connect each line to code that actually performs the step. Here is what that password reset scenario might look like once the PM and developer finally sat down together: Plain Text Feature: Password reset Scenario: Reset link expires after one hour Given a user has requested a password reset And 61 minutes have passed When the user opens the reset link Then they should see "This link has expired" That little block settles an argument that a vague ticket never would. Behind the scenes, each line maps to a step definition, a small function in Java, Python, C#, JavaScript, or whatever your stack uses. The framework reads the scenario, finds the matching step code, runs it, and reports back in the same readable language. Popular BDD Frameworks Worth Knowing There is no single best option. The right pick mostly depends on the language your team already writes in. Here are the ones I see most often. Cucumber Cucumber is the one everyone has heard of. It supports Java, JavaScript, Ruby, and a few other languages, and it has the largest community by a wide margin. If you get stuck, someone on Stack Overflow has almost certainly hit the same wall. The trade-off is that it can feel heavy for small projects. Reqnroll (Formerly SpecFlow) For .NET teams, SpecFlow was the go-to choice for years. After it was retired, the community fork Reqnroll picked up where it left off and keeps the familiar Gherkin workflow alive for C# projects. Migration is fairly painless for most codebases. Behave and pytest-bdd Python teams usually choose between these two. Behave is a standalone runner that feels close to Cucumber. pytest-bdd plugs into pytest, which is a big plus if your team already relies on pytest fixtures and plugins. I lean toward pytest-bdd for teams that already have a mature pytest setup, simply because it means one less tool to maintain. Godog Godog is the official Cucumber implementation for Go. It is lean, fast, and fits nicely into Go projects that want readable acceptance tests without pulling in a lot of extra machinery. How to Pick the Right One When teams ask me which framework to use, I usually walk them through a few plain questions instead of a feature comparison chart: Does it match your stack? Pick a framework that lives in the same language as your application. Asking developers to maintain step definitions in a second language is a quick way to get them ignored.Who will write the scenarios? If product managers or QA analysts will contribute scenarios, you want clean Gherkin support and decent editor plugins. If only developers will touch it, a lighter option may be enough.How good is the reporting? Readable reports matter more than people expect. The whole point is that non-developers can see what passed and what did not.Will it play nicely with CI? Check that it runs cleanly in your pipeline, supports parallel execution, and does not add minutes to every build.Is it actively maintained? A framework with an active maintainer community will save you from being stranded on an abandoned version, which is exactly what happened to plenty of SpecFlow users. Mistakes Teams Make With BDD The framework rarely causes a BDD rollout to fail. The habits around it do. A few patterns come up again and again. The first is writing scenarios like UI scripts. "When I click the blue button in the top right corner" describes clicks, not behavior. The moment the design changes, the scenario breaks, even though the business rule stayed exactly the same. The second is letting developers write every scenario alone. At that point, you have unit tests wearing a costume. The value of BDD comes from the conversation, so bring in the people who own the requirement. The third is scenario sprawl. Teams get excited and write hundreds of scenarios covering every edge case. The suite slows down, nobody reads it, and it quietly becomes a maintenance burden. Keep BDD for the behaviors that matter to the business, and push the fine-grained edge cases down into unit and integration tests. Where BDD Fits Alongside Other Testing BDD is great at capturing intended behavior, the things people agreed the system should do. What it does not do well is catch the stuff nobody thought to write down. APIs change, payloads shift, a field gets renamed, and none of your carefully written scenarios notice because they were never about that. That is why most healthy test strategies pair BDD with other layers: fast unit tests, contract checks between services, and API regression tests that reflect how the system is actually used. BDD tells you whether you built the right thing. The other layers help confirm you did not quietly break something along the way. Final Thoughts If you remember one thing, make it this: the framework is the least important part of BDD. Get your product, QA, and engineering people in a room, agree on a handful of concrete examples, and write them down. Then choose whichever tool fits your language and your team. Cucumber, Reqnroll, Behave, pytest-bdd, and Godog will all do the job if the conversations behind them are solid. And if you ever find yourself in a twenty-minute argument about what a single ticket means, that is your sign. Write the scenario. Frequently Asked Questions What is a BDD framework used for? A BDD framework turns plain-language scenarios, usually written in Gherkin, into automated tests. It lets technical and non-technical team members agree on expected behavior and then check that the software actually delivers it. Is BDD the same as TDD? Not quite. TDD focuses on writing unit tests before code, mostly from a developer's point of view. BDD builds on that idea but centers on business behavior and shared understanding across roles. Which BDD framework is best for beginners? Cucumber is usually the easiest place to start because of its documentation and huge community. Python teams often find Behave just as approachable. Do I need Gherkin to do BDD? No. Gherkin is the most common format, but BDD is a practice, not a syntax. Some teams write examples in tables or plain documents and automate them another way. Can BDD slow down development? It can if teams overuse scenarios or write them at the UI level. When kept focused on key business behaviors, BDD usually saves time by reducing rework and misunderstandings.
Accuracy is what many beginners look for when building an ML model. Imagine you have a model that detects skin diseases. Your test dataset has 10,000 patients, out of which only 100 have the disease, while the remaining are healthy. The model ends up predicting everyone as healthy. That would give a 99% accuracy, which sounds impressive, but it failed to identify a single sick patient. Now, think about deploying it in the real world while blindly trusting its accuracy. That would be disastrous, as it failed to identify even a single sick person during testing. This is why models can fail in the real world even though they gave impressive accuracy during the test phase. To actually understand the model’s behaviour in real life, we need to look at metrics beyond accuracy. The Confusion Matrix Let’s start by looking at the confusion matrix, as every binary classification problem falls into one of the four categories. It shows what your model got right and wrong. True Positive (TP): The model predicts that a patient has the disease, and the patient actually has it. The model predicts positive, and it is right.True Negative (TN): The model predicts that a patient is healthy, and the patient actually is healthy. The model predicts negative, and it is negative.False Positive (FP): The model predicts that a healthy patient has the disease. The model predicts positive, but the actual class is negative.False Negative (FN): The model predicts a sick patient as healthy. The model predicts negative, but the actual class is positive. Before proceeding, make sure you understand all four categories, as they form the foundation for the classification metrics we’ll discuss. 1. Accuracy Let’s start with the metrics everyone knows. Accuracy measures the percentage of correct predictions out of all predictions made by a model. It is mathematically measured using the formula given below. Accuracy depends on the training data. In our disease example, we saw that the model failed to detect even a single sick person. This is because the training data must have contained mostly data from healthy people, while only a very small fraction came from sick patients. As a result, it learns that guessing healthy is almost always the safest option and ends up predicting everything as healthy. This is a case of false negatives. To reduce false negatives, make sure the training data is properly balanced (having equal data for both classes, or adjusting class weights to penalize missing a sick patient). This forces the model to learn the underlying patterns in the data rather than relying on shortcuts. When to Use Accuracy? Accuracy works well when: The classes are balanced.The dataset reflects the real-world class distribution.The dataset is balanced, and a false alarm costs the same as missing a detection. If you are classifying images of dogs and cats and your dataset contains an equal number of both, then accuracy is a great metric. But if classes are imbalanced, then don’t let accuracy fool you. 2. Precision Many of us are confused between the concepts of accuracy and precision. Accuracy tells how often the model is correct, whereas precision tells how trustworthy the positive predictions are. The mathematical formula for precision is: Suppose your model predicts 100 people as sick. But after investigation, it was found that only 70 out of the 100 were sick. That would give a precision score of 70%, which means that 7 out of 10 people predicted as sick are actually sick. A model with low precision is bound to generate false positives. When to Use Precision? High precision is needed when: False positives can be costly. In the case of a spam filter, if a model wrongly classifies a legitimate email as spam, the user might miss an important message. If you want a model where a positive prediction is highly likely to be actually positive, you need high precision. 3. Recall Out of all the actual positive cases, how many accurate predictions does the model make? That's what recall measures. Imagine a model is testing 100 people who are actually sick. If the model correctly identifies 90 of them as sick, then its recall is 90%. The mathematical formula for recall is: In simple terms, recall tells us how well a model finds the positive cases that are actually there. Recall decreases when the model misses actual positive cases. When to Use Recall? Recall is used when: Missing a positive case is costly.Cases like medical screening where failing to detect a sick person can be deadly.Fraud detection, security setups, and search systems. 4. F1-Score Imagine you are training a fraud detection model to be more aggressive so its recall increases. But an overly aggressive model can sometimes flag a legitimate transaction as fraud, resulting in an increased recall but decreased precision. To avoid imbalance between precision and recall, we can use a metric called F1-Score. F1-Score is the harmonic mean of both precision and recall. We use the harmonic mean over the arithmetic mean because the harmonic mean heavily penalizes situations where one value is much lower than the other. For example, if we consider a precision of 0.90 and a recall of 0.10, the arithmetic mean would give a result of 0.50, whereas the harmonic mean would yield 0.18. The value 0.18 gives a better idea that the model is performing poorly on one side of the problem. When to Use F1-Score? F1-Score is useful when: The classes are imbalanced.Both precision and recall matter.A single metric that represents both. However, don’t always use F1-Score just because the classes are imbalanced. You should consider the type of error that matters most in your use case and choose the metrics accordingly. 5. ROC-AUC A classification model produces a probability score before prediction and uses a threshold to determine whether to classify the input as positive or negative. Consider a threshold of 0.5; any score greater than or equal to 0.5 is considered positive, whereas less than that is negative. A fixed threshold is not always suitable for every problem. Lowering the threshold helps the model find more positive cases, but it can also lead to more false alarms. Raising the threshold reduces false alarms, but the model may miss some of the actual positive cases. So instead of evaluating the model at just one cutoff, we can use a metric called ROC-AUC, which evaluates its performance across every possible threshold from 0 to 1. The ROC then plots two values: true positive rate (TPR/recall) on the y-axis and false positive rate (FPR) on the x-axis. These values are plotted across different thresholds to form the ROC curve. TPR measures how many of the actual positives the model manages to catch, whereas FPR measures how many of the actual negatives the model incorrectly flags as positive. The resulting metric (AUC) lies in the range of (0,1). If the AUC is 1, then it’s a perfect model; an AUC of 0.5 indicates a random guess, whereas a value less than 0.5 means the model’s predictions are worse than random guessing. Look at the sample ROC curve shown below. An ROC curve with an {AUC} of 0.93. The red points illustrate how shifting the operational decision threshold (T) directly navigates the trade-off between catching positive cases (TPR) and triggering false alarms (FPR). Here, in the graph above, the continuous blue curve is built by evaluating all 1,000 unique probability thresholds from the test dataset. Labeling every single cutoff would make the chart completely unreadable. Instead, the diagram highlights just three representative thresholds with red dots to provide concrete examples of the trade-off without cluttering the visual. When to Use ROC-AUC? ROC-AUC is useful when: You want to evaluate ranking performance across thresholds.The classification threshold hasn’t been finalized. The model’s overall discrimination ability is needed. It is important to note that ROC-AUC isn’t always the best choice for highly imbalanced datasets. If positive cases are rare, the false positive rate can remain small even when the model generates a large number of false positives. To overcome this problem, we can use the metric called PR-AUC. 6. PR-AUC PR-AUC stands for precision-recall area under the curve. While ROC-AUC plots true positive rate versus false positive rate, PR-AUC plots precision versus recall. This is useful when the positive classes are rare. Imagine a record of 999,000 legitimate and 1,000 fraudulent transactions. But your model raises 5000 false alarms. Here, relying on the ROC curve would give a false positive rate of approximately 0.5%. But because the number of legitimate transactions is massive, 0.5% looks almost negligible, and ROC would give a perfect score. PR-AUC solves this problem by strictly plotting precision against recall across varying classification thresholds. In simple terms, ROC compares false alarms against the huge sea of 999,000 normal transactions, whereas PR-AUC ignores the normal transactions entirely and asks: Recall: How many of the 1,000 real frauds did you catch?Precision: Out of the alerts you raised, how many were real versus false alarms (precision)? When to Use PR-AUC? PR-AUC is useful when: The positive class is rare.You’re interested in detecting the positive class.False positives matter. For imbalanced classification problems, it is recommended to use both ROC-AUC and PR-AUC rather than relying on any one. Evaluating both metrics ensures you understand global class separability while preventing high True Negative counts from masking poor precision on rare positive events. 7. Log Loss Log loss assesses how well the model’s predicted probabilities align with the true labels. This is different from metrics that evaluate whether the model’s predictions are correct. Consider two models, A and B, for the positive class. Model A shows a 55% probability, whereas B shows a 1% probability. Here, model B was not only wrong but also confident, even with the wrong prediction, and log Loss is designed to capture exactly this scenario. Its mathematical formula for binary classification is: N is the number of observations.yᵢ is the actual class, either 0 or 1.pᵢ is the predicted probability of the positive class. Log Loss rewards correct predictions and penalises confident incorrect predictions. Correct + high confidence → small lossCorrect + low confidence → larger lossWrong + high confidence → very large loss A perfectly designed classifier can approach a log loss of near zero. When to Use Log Loss? Log Loss is useful when the quality of the predicted probabilities matters, such as: Medical risk predictionCredit risk scoringInsurance risk modelsRecommendation systems Code Snippet Python from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score, average_precision_score, log_loss # Precision, Recall, F1 (Hard labels) precision = precision_score(y_true, y_pred) recall = recall_score(y_true, y_pred) f1 = f1_score(y_true, y_pred) # ROC-AUC, PR-AUC, Log Loss (Probabilities) roc_auc = roc_auc_score(y_true, y_pred_proba) pr_auc = average_precision_score(y_true, y_pred_proba) loss = log_loss(y_true, y_pred_proba) Final Thoughts: When to Choose What? In this article, we have so far discussed seven types of metrics, and the next time you build a classification model, ask yourself: "What happens when my model is wrong?" That would give a better idea of which metrics to choose for your problem.
Not every real-world application needs full Clean Architecture with its multiple layers, dependency inversion, and strict domain boundaries. Many production systems — particularly internal enterprise applications like HRMS and Payroll platforms — run successfully on a simpler, pragmatic two-layer architecture: a Controller layer and a Data Access layer. Understanding why this simpler structure works, and how it relates to the same OOP principles behind Clean Architecture, makes it easier to work confidently in either kind of codebase. The Two Layers A two-layer architecture splits an application into exactly what the name suggests: Layer 1 — Controller Layer: handles incoming HTTP requests and delegates the actual work elsewhere. C# [ApiController] [Route("api/[controller]")] public class LeaveRequestController : ControllerBase { private readonly ILeaveRequestDataAccess _dataAccess; public LeaveRequestController(ILeaveRequestDataAccess dataAccess) { _dataAccess = dataAccess; } [HttpGet("get-emp-details")] public IActionResult GetEmployeeDetails(string empId) { var result = _dataAccess.GetEmployeeDetails(empId); return Ok(result); } } Layer 2 — Data Access Layer: handles the actual database interaction — connection strings, parameters, stored procedure or package calls. C# public interface ILeaveRequestDataAccess { EmployeeDto GetEmployeeDetails(string empId); } public class LeaveRequestDataAccess : ILeaveRequestDataAccess { public EmployeeDto GetEmployeeDetails(string empId) { // ADO.NET/Oracle-specific logic lives here: // connection string, OracleCommand, package name, parameter types return employeeData; } } The Controller never touches ADO.NET directly. The Data Access layer never touches HTTP concerns like status codes or routing. Each layer has exactly one job. Why This Is Abstraction and Encapsulation in Practice This separation isn't just organizational tidiness — it's a direct, practical application of two core OOP principles. Abstraction: The Controller calls _dataAccess.GetEmployeeDetails(empId) without knowing or caring how that data actually gets retrieved. It doesn't know whether the underlying call uses a stored procedure, a package function, or raw SQL. It only needs to know that calling this method returns the data it asked for. The internal complexity of connection handling, parameter binding, and query execution is hidden behind a simple method signature. Encapsulation: The Data Access layer protects the details of how data is fetched from being scattered across the codebase. If every controller method directly opened its own OracleConnection and wrote its own SQL, any change to the database — a renamed package, a new parameter, a switched provider — would require hunting down and editing code in dozens of places. By encapsulating that logic inside one dedicated layer, a change only needs to happen once. The Practical Test: What Happens When the Database Changes A useful way to confirm this separation is working correctly is to ask: if the underlying database technology changed — say, from Oracle to SQL Server — which layer would need to change? The answer should be: only the Data Access layer. The connection strings, parameter types, and query syntax living inside LeaveRequestDataAccess would need rewriting. The Controller, which only knows about the ILeaveRequestDataAccess interface and calls GetEmployeeDetails(empId), wouldn't need to change at all — it has no idea, and no need to know, what database sits behind that interface. If a database change ever required touching Controller code too, that would be a signal the separation isn't clean — some data-access detail has leaked into a layer that shouldn't know about it. How This Relates to Clean Architecture Clean Architecture takes this same underlying idea — separating concerns so that change in one area doesn't ripple through the whole system — and extends it further, typically into more layers: a Domain layer (core business rules, independent of any framework), an Application layer (use cases, orchestration), an Infrastructure layer (database, external services), and a Presentation layer (API or UI). The core principle is identical to the two-layer setup: outer layers depend on inner layers, never the other way around, and each layer only knows what it strictly needs to know. A two-layer Controller/Data Access split is, in effect, a simplified version of this same idea — fewer layers, less ceremony, but the same underlying discipline of not letting database-specific details leak into request-handling code, and not letting request-handling concerns leak into data-access code. For many internal enterprise applications, a full Clean Architecture setup — with separate Domain, Application, and Infrastructure projects, dependency inversion containers, and strict boundary enforcement — is more structure than the project actually needs. A well-maintained two-layer architecture, with a clear interface between Controller and Data Access, delivers much of the same practical benefit — testability, easier maintenance, isolated change — without the additional overhead. When Two Layers Stop Being Enough The two-layer pattern works well until business logic starts accumulating inside the Controller itself — validation rules, calculations, workflow decisions that have nothing to do with HTTP handling or database access. When that happens, it's usually a sign that a third layer — often called a Service or Business Logic layer — is needed to hold that logic separately, keeping the Controller thin and focused purely on request/response handling. This is often the natural next step toward something closer to Clean Architecture, added incrementally as a project's complexity genuinely justifies it, rather than adopted wholesale from day one. Why This Split Also Makes Testing Easier A practical benefit of separating Controller and Data Access that often goes unmentioned until it's needed: this structure makes unit testing significantly more straightforward. Because the Controller depends on ILeaveRequestDataAccess — an interface — rather than a concrete class tied to Oracle, it becomes possible to test the Controller's logic in isolation, without touching a real database at all. A test can supply a fake or mock implementation of the interface: C# public class FakeLeaveRequestDataAccess : ILeaveRequestDataAccess { public EmployeeDto GetEmployeeDetails(string empId) { return new EmployeeDto { EmpId = empId, Name = "Test Employee", Salary = 50000 }; } } With this fake in place, a test can verify that GetEmployeeDetails on the Controller returns the correct HTTP response, handles a missing employee correctly, or triggers the right status code — all without a live Oracle connection, without test data cluttering a real database, and without the test slowing down because it's waiting on network or database round-trips. This is only possible because the Controller was written to depend on an abstraction (ILeaveRequestDataAccess) rather than directly instantiating new LeaveRequestDataAccess() inside itself. If the Controller created its own Data Access instance internally, there would be no way to substitute a fake version during testing — the real database dependency would be unavoidable. This is the same underlying idea, applied practically: depending on an interface rather than a concrete implementation isn't just an architectural preference; it's what makes isolated, fast, reliable testing possible at all. Takeaway A two-layer Controller/Data Access architecture isn't a simplified shortcut that skips "real" architecture — it's a direct, practical application of Abstraction and Encapsulation at the system level, just as those same principles apply within a single class. Understanding it this way makes the reasoning behind Clean Architecture's more elaborate layering easier to grasp too: both are solving the same underlying problem — isolating change, hiding implementation detail, and keeping each part of the system focused on exactly one responsibility — just at different levels of complexity depending on what the project actually needs.
Distributed systems usually spread data by hashing keys across nodes so no one node carries all the load. A hot key breaks that assumption. It is a key whose access volume greatly exceeds the volume of other keys during a given period. A global view counter or a popular item identifier can receive millions of reads or writes each second. Every request targets the same partition, even when the rest of the cluster has spare capacity. The result is uneven load. One node reaches its CPU, network, thread, or connection limits while other nodes remain underused. Adding nodes does not solve the problem because traffic continues to resolve to the same partition. The overloaded shard becomes the bottleneck for a system that was intended to distribute work. Hot keys also create failure risks. If a popular cache entry expires or its host fails, many requests can fall through to the backing data store at once. This cache breakdown can overwhelm the database and trigger broader failures. Teams should monitor access patterns and alert when one key represents an unusual share of traffic, latency, or bandwidth. Split a Logical Key Into Physical Buckets A common mitigation is to split one logical key into several physical buckets. Instead of updating a single counter, a writer chooses a bucket and updates that bucket. The combined value is calculated by reading and aggregating all buckets. Java public void increment(String key) { int bucket = ThreadLocalRandom.current().nextInt(NUM_BUCKETS); jedis.incr(key + ":" + bucket); } public int readTotal(String key) { int total = 0; for (int i = 0; i < NUM_BUCKETS; i++) { total += Integer.parseInt(jedis.get(key + ":" + i)); } return total; } This pattern spreads writes across hash slots and nodes. The tradeoff is more work during reads. In Redis Cluster, a native `MGET`cannot read keys from different hash slots in one command. The client must issue individual reads, use parallel reads, or provide a scatter-gather layer that collects the bucket values. The aggregated total is also not an atomic snapshot because each bucket can be read at a different instant. Bucketed counters work well when approximate totals are acceptable, such as views, metrics, or telemetry. They are less appropriate when every read requires a strictly current, exact value. The number of buckets should reflect peak write volume, expected read frequency, and the cost of aggregation. Serve Read-Heavy Hot Keys From Local Memory When a hot key is primarily read, an application cache can reduce pressure on the shared store. A short-lived in-memory cache lets many requests reuse a recently fetched value before another request reaches Redis or a database. Java private static final LoadingCache<String, String> HOT_KEY_CACHE = Caffeine.newBuilder() .expireAfterWrite(50, TimeUnit.MILLISECONDS) .build(key -> { try (Jedis jedis = JEDIS_POOL.getResource()) { return jedis.get(key); } }); With this setup, most reads can be served from memory, and cache misses fetch the current value from Redis. The short expiration bounds the staleness window. This is useful when the business can tolerate a briefly outdated response in exchange for much lower read pressure on the shared data store. Systems can also use layered caches. A content delivery network or edge cache can absorb public traffic, followed by an application cache, then a distributed cache, and finally the durable store. Each layer should have a clear expiration policy and a plan for cache invalidation. The goal is to keep repeated reads from reaching the most constrained shared component. Replicate Values When Reads Are More Important Than Writes Another option for a read-dominated hot key is replication. The system stores several copies of the logical value on different shards, then distributes reads across the copies. Java int idx = ThreadLocalRandom.current().nextInt(REPLICA_COUNT); String replicaKey = originalKey + ":" + idx; String value = jedis.get(replicaKey); Replication can increase read capacity because different nodes serve independent requests. It adds work to writes because every replica must be updated. If an update fails for one replica, callers can see different values depending on the chosen key. This approach is useful when stale or temporarily inconsistent reads are acceptable. It needs an explicit repair strategy for failed replica updates. Smooth Write Spikes With Queues and Admission Control A hot key with a high write rate may need a different approach. Instead of applying every update immediately, the service can publish updates to a queue and let workers batch or aggregate them before writing to the shared store. This smooths spikes and protects the storage layer, but it adds write latency and creates a delay before the final value is visible. Rate limiting and backpressure can protect the system when a sudden surge would otherwise overwhelm one partition. These controls should preserve important traffic first and reject or delay work that can safely wait. The correct policy depends on whether the key represents a counter, inventory, a payment state, or another domain with different consistency requirements. Choose the Tradeoff Explicitly Every hot key mitigation trades one property for another. Buckets increase read work and make aggregate snapshots non-atomic. Application caching introduces staleness. Replication increases write cost and requires reconciliation. Queues delay visibility. These tradeoffs are often preferable to letting one shard become a single point of congestion. The right design starts with the semantics of the key. Decide whether the value must be exact, how long it can be stale, whether writes can be delayed, and what happens when the shared store is unavailable. Then measure the key’s read rate, write rate, payload size, and partition utilization. A mitigation that is safe for a view counter can be incorrect for inventory or payment state. Hot keys are a normal condition in systems with uneven demand. Monitoring should detect them early, and the architecture should provide a way to redistribute load before one partition limits the entire service. With bucketed writes, local caching, replication, queues, and admission control, teams can turn a concentrated source of traffic into a managed capacity decision instead of a cluster-wide incident.
DB changes can be streamed in real time via Oracle GoldenGate. Java applications can then consume them to validate/act on/route events without polling the DB. Goal Listen to real-time transactions in the DB, act on all operations within transactions, and maintain transaction boundaries with near real-time SLAs. Component Details 1. Source DB This is the source DB from which the Java service listens for transaction changes in real time. It can be any DB supported by the GoldenGate service (e.g., MySql/ Oracle). 2. OCI Source GoldenGate Service Deployment Input: DB Redo logsOutput: Trail file The GoldenGate service has two primary components: Extract and Replicat. Extract reads changes from the source DB in the form of redo and archive logs. It commits each transaction's DML operations until either COMMIT or ROLLBACK is identified. If it encounters a COMMIT, it stores the transaction in TRAIL files. If ROLLBACK is encountered, it discards those changes. The output of this deployment is a TRAIL file with an ordered list of all transactions that occurred in the source DB. 3. OCI Target GoldenGate Big Data Deployment Input: Trail files generated by traditional GG deploymentOutput: Ordered stream of operations within a transaction This is another deployment type that can be used for transaction management. We can use the DATA STREAMS feature instead, but this uses the AsyncAPI publish/subscribe interface, whereas big data deployment is a managed integration runtime for CDC delivery to Kafka. 4. Streaming Service Input: Trail files generated by traditional GG deploymentOutput: Ordered stream of operations within a transaction Ideally, Kafka keeps up with the scale of GoldenGate and downstream scalability. Kafka provides an option to add partitions while the partitions are receiving data. This feature is particularly useful for handling traffic bursts or batch SQL jobs on the source DB. 5. Java Service Kubernetes Deployment Input – Kafka messages, in this case individual operationsOutput – Application logic A Java application instance that can scale horizontally depending on the Kafka partition backlog. Use Case Java service is interested in listening to the entire transaction in the order that it is happening for a predefined set of tables.It does not want to listen to all the source table changesIt also does not want all columns of the changed row. It is interested in listening to only specific set of columns from given list of tables. Component Setup The detailed setup guide here can be used as general guidance, but I will focus on the specific changes needed to achieve the use case above. 1. Source DB Here we want to listen to redo logs from the source DB. To enable supplemental logging on the source DB, use the command below. SQL ALTER DATABASE ADD SUPPLEMENTAL LOG DATA In our use case, we also want to make sure we always get only the columns from the table that the Java service is interested in, and not all the columns. To enable table-specific supplemental logs, use the command below. SQL ALTER TABLE HR.EMPLOYEE ADD SUPPLEMENTAL LOG GROUP emp_group_loc (EMPLOYEE_ID, EMP_NAME, EMP_DEPARTMENT_ID); 2. OCI Source GG Deployment In this step, OCI source GG deployment is configured to listen to redo logs of the source DB. As per our use case the important step is to configure only required tables in Extract's parameter file. The list of tables basically makes sure transactions happening on only those tables will be identified by GG and will be collected in trail files. You can use this article to set up Extract. Example of parameter file for extract: SQL table HR.EMPLOYEE; Along with extract, a distribution path needs to be created within the same source deployment. This distribution path makes sure the target big data deployment can access the trail file records easily. The details to create the distribution path can be found in TASK 3 of the article. 3. OCI Target GoldenGate Big Data Deployment In this deployment, a process called 'replicat' is created; replicat reads transaction data from the distribution path created in step 2 above, then publishes those transactions. Replicat works using two configuration files. Parameter File Similar to Extract, Replicat also has a parameter file. This file contains mapping information on how operations from the source DB map to the target schema and table. Since we are not replicating transactions in the DB, a JSON with the target table mapped in the parameter file gets written to kafka topic instead. Example for a complete setup guide: use this guide. SQL MAP HR.EMPLOYEE, TARGET *.*; Properties File Three important parameters that decide how the source transactions get written to the target are 1. TopicMappingTemplate – Tells replicat about the target, where source transactions will be written, 2. opMode – How the source transaction will be written. Complete transaction or one operation at a time.3. keyMappingTemplate – This parameter is the hashing key used by the replicat Kafka producer. For all the parameters that are available in the properties file, please refer to section "11.2.10.1.3.2.2 Kafka Handler Configuration" in this guide. 4. Streaming Service and Java Consumer Service Since GG hashes based on transaction ID, it is guaranteed that all operations in a transaction will always be in the same partition. The Java service needs a consumer group used by all pods. At bootstrap, all pods register their client ID with the Kafka broker and reserve a partition. This mechanism makes sure a single pod always listens to one partition until a rebalance happens due to a pod going down or other reasons. This setup ensures the Kafka consumer in the Java service can poll and collect transactions while maintaining operation boundaries and sequence for downstream application logic within the same service. Separating polling logic from processing logic in the Java service helps avoid various timeout scenarios. We can dive deep into this mechanism in a follow-up article. Summary With this setup, we can guarantee that transactions at the source DB are accessed by a Java service in order and can run within a single transaction at a time. This workflow is useful for industries that need an eventing mechanism, such as fraud detection and financial applications, telecommunications and real-time billing, Retail and real-time inventory management, and AI and semantic search pipelines.
Agile
Career Development
Methodologies
Team Management
A/B Testing Under Shared Constraints: Supply, Inventory, and User Attention
November 16, 2026 by Arun Thomas
Building an AI-Powered Code Review Assistant in Java
November 16, 2026 by Akhil Sabu
November 13, 2026
by Stefan Wolpers
CORE
AI/ML
Big Data
Databases
IoT
Running an OpenStreetMap Tile Server for the Whole Planet on Modest Hardware
November 16, 2026 by Dmitrii Starikov
Your Agents Think Data Federation Means One Truth. It Means a Collage.
November 16, 2026 by Sushree Mishra
Building an AI-Powered Code Review Assistant in Java
November 16, 2026 by Akhil Sabu
Cloud Architecture
Integration
Microservices
Performance
Spring Boot's Hidden Secret for Configuration: EnvironmentPostProcessor
November 16, 2026 by Mikhail Polivakha
Modernizing a Legacy Java Application With OpenRewrite: What We Automated and What We Didn't
November 16, 2026 by Kirankumar Phirangi
November 13, 2026 by Himanshu Mandhyan
Frameworks
Java
JavaScript
Languages
Tools
7 Failure Modes Every Production AI Agent Should Be Tested Against
November 16, 2026 by Mihir Bhatt
Spring Boot's Hidden Secret for Configuration: EnvironmentPostProcessor
November 16, 2026 by Mikhail Polivakha
Building an AI-Powered Code Review Assistant in Java
November 16, 2026 by Akhil Sabu
Deployment
DevOps and CI/CD
Maintenance
Monitoring and Observability
7 Failure Modes Every Production AI Agent Should Be Tested Against
November 16, 2026 by Mihir Bhatt
A/B Testing Under Shared Constraints: Supply, Inventory, and User Attention
November 16, 2026 by Arun Thomas
Building an AI-Powered Code Review Assistant in Java
November 16, 2026 by Akhil Sabu
AI/ML
Java
JavaScript
Open Source
Spring Boot's Hidden Secret for Configuration: EnvironmentPostProcessor
November 16, 2026 by Mikhail Polivakha
Your Agents Think Data Federation Means One Truth. It Means a Collage.
November 16, 2026 by Sushree Mishra
Building an AI-Powered Code Review Assistant in Java
November 16, 2026 by Akhil Sabu