DZone Spotlight

Tuesday, August 25 View All Articles »

LLM Judgment for Document Pipelines: Bounded Pools and Typed Verdicts

By Deepak Gupta

This project creates a daily digest for sellers in an enterprise system. Each seller handles accounts at a set of companies and needs to know when something happens at one of them: a product launch, leadership change, new contract, or funding round. That news is often an opening for the next conversation. The pipeline reads the day's incoming news and sends each seller a short email with the handful of articles worth their time about the companies they cover. Every item in that email carries a thumbs-up and a thumbs-down button. At launch, we set a simple quality target. From historical user behavior on other surfaces, we knew that about 9% of served items would receive a user vote. Among those votes, we wanted 70% to be thumbs up. We expected that approval rate to show whether the pipeline was improving. A month later, the dashboard could report vote counts but not explain them. Every downvote triggered the same manual investigation: retrieve the document, retrieve the target profile, read both, and infer the cause. The possible causes belonged to different parts of the pipeline. A user may already have seen the story earlier that week. An article may concern the right parent company but the wrong business unit, such as iPad coverage sent to a seller responsible for iPhone accounts. A technically relevant article from a low-quality publisher may contain no useful signal. A stock-ticker recap may mention the company accurately and still offer nothing useful to the recipient. The feedback system reduced all of these outcomes to the same negative signal. The cost of this ambiguity became clear at production volume. The ingestion layer processed 700,000 to 1 million articles per day across more than 20,000 target entities. Candidate generation reduced that corpus to a bounded pool before any model call. To fix this, we changed the output contract between the judging model and the downstream pipeline. In a previous article, I described the system's three stages: cost-efficient triage, target-aware retrieval, and bounded LLM reasoning. Candidate generation controls most of the pipeline’s cost because it determines how many documents reach the model. This article focuses on Stage 3: the judgments the model should make, the structured verdict it returns, and how those verdicts make user feedback auditable. Judging vs Selecting The single most important design decision in Stage 3 is what job the LLM is not allowed to do: selection. For each target, Stage 2 supplies at most 100 candidates. The model evaluates each document independently and returns a verdict: whether it is relevant to the target, whether the event is new to that user's digest, and whether it should be sent in the email. A pool of 100 candidates usually produces 10 to 40 digest-worthy items. Keeping these jobs separate puts a hard limit on model work. Every increase to the Stage 2 pool cap increases the number of documents sent to the model, including documents that will later be rejected. Candidate-pool size is therefore a cost-control parameter owned upstream, where retrieval and ranking are cheaper and easier to inspect. The separation also changes the model’s task. Ranking requires the model to compare every candidate against every other candidate in the pool. Pointwise judgment asks a narrower question: is this document relevant and novel for this target? Stage 2 has already removed most of the distractors, so the model can evaluate each remaining document independently. Each verdict can then be stored and inspected later. Listwise LLM ranking also introduces position bias: a model can favor documents partly because of where they occur in the supplied list. An independent pointwise verdict avoids that failure mode because candidate order does not affect the task definition. A survey of LLM-as-a-judge biases documents this issue. Microsoft's UMBRELA evaluation work also found that LLM-based pointwise relevance assessments correlated strongly with human-derived rankings across five years of TREC data. This boundary makes missing coverage diagnosable. When a document is missing from a digest, there are exactly two possibilities: it never entered the pool (a Stage 2 problem with a deterministic, inspectable cause) or it entered the pool and the model judged it out (a Stage 3 problem with a recorded verdict). You never have to ask "did the model even see it?" The Output Contract Stage 3 returns a structured verdict for every candidate. Any field used by downstream code has a closed set of values. Each verdict contains a short rationale, relevance and novelty judgments, a reason code, and a one-sentence summary. rationale and summary are free text. The rationale records why the model reached its conclusion, and the summary is written for the recipient. Downstream services never branch on either field. They consume the enum values. The order of those fields is deliberate. Asking for labels before reasoning can hurt judgment quality because the model commits to an answer and then produces a justification for it. The EMNLP paper Let Me Speak Freely? describes this cost of constrained output formats. We put rationale first so the model can work through the document before returning the typed verdict. We made two deliberate choices in the code below. The model-facing schema and the stored record are separate classes: the model only sees ModelVerdict, while the pipeline wraps its output in Verdict and adds the document ID and status. extra="forbid turns a hallucinated field into a validation failure rather than silently accepting it: Python class VerdictStatus(str, Enum): VALID = "valid" CONTRACT_VIOLATION = "contract_violation" class Relevance(str, Enum): RELEVANT = "relevant" TANGENTIAL = "tangential" # mentions target, no actionable signal IRRELEVANT = "irrelevant" class Novelty(str, Enum): NEW = "new" UPDATE = "update" # known event, new material detail ALREADY_COVERED = "already_covered" class ReasonCode(str, Enum): NEW_CONTRACT_WIN = "new_contract_win" LEADERSHIP_CHANGE = "leadership_change" REGULATORY_ACTION = "regulatory_action" PRODUCT_LAUNCH = "product_launch" FINANCIAL_RESULTS = "financial_results" MARKET_MOVEMENT = "market_movement" OTHER = "other" # Watch this rate for vocabulary gaps class ModelVerdict(BaseModel): model_config = ConfigDict(extra="forbid") rationale: str # Free text, deliberately before enum fields relevance: Relevance novelty: Novelty reason_code: ReasonCode summary: str # Free text for the digest class Verdict(ModelVerdict): doc_id: str # Stamped by the pipeline, never by the model. This field is excluded # from the schema the model sees, so a verdict can't declare itself valid. status: VerdictStatus = VerdictStatus.VALID def parse_verdict(raw: str, doc_id: str) -> Verdict: try: model_verdict = ModelVerdict.model_validate_json(raw) return Verdict(doc_id=doc_id, **model_verdict.model_dump()) except ValidationError as first_error: repaired = repair_call(raw, error=str(first_error)) try: model_verdict = ModelVerdict.model_validate_json(repaired) return Verdict(doc_id=doc_id, **model_verdict.model_dump()) except ValidationError: metrics.increment("stage3.contract_violation") return Verdict( doc_id=doc_id, status=VerdictStatus.CONTRACT_VIOLATION, rationale="[verdict failed validation]", relevance=Relevance.IRRELEVANT, novelty=Novelty.ALREADY_COVERED, reason_code=ReasonCode.OTHER, summary="[verdict failed validation]", ) def belongs_in_digest(verdict: Verdict) -> bool: return ( verdict.status == VerdictStatus.VALID and verdict.relevance == Relevance.RELEVANT and verdict.novelty != Novelty.ALREADY_COVERED ) The contract earns its keep in three places: Ingestion is straightforward: Digest assembly, notification routing, and the frontend receive typed fields instead of free text that each service must interpret independently. Serving a document is the belongs_in_digest filter above: the verdict must be VALID, RELEVANT, and not ALREADY_COVERED. Checking status first ensures that a failed parse cannot masquerade as a model judgment.Debugging becomes a query, instead of an investigation: Every candidate has a stored verdict, with enum fields for filtering and aggregation plus the rationale for human review. Months later, an engineer can identify why a document was included or excluded without reconstructing the original model call.Monitoring is split into two signals: Calculate relevance, novelty, and reason-code distributions from VALID verdicts only. Track CONTRACT_VIOLATION separately. Mixing the two means a parser regression can look like a sudden change in model quality. The valid-verdict distributions are the first line of quality monitoring. A growing share of IRRELEVANT results for one vertical, an unusual increase in ALREADY_COVERED for one company, or an increase in OTHER each gives the team a place to start looking. There is a maintenance cost. New event types and business concepts eventually exceed the initial reason-code list. OTHER gives that gap a measurable home: if its rate rises among valid verdicts, the taxonomy needs review. This is a vocabulary-maintenance signal and not evidence that the model failed to return a valid response. Enums Close the Feedback Loop A thumbs-down on its own is almost useless. It says that a digest item disappointed the user, but not whether the problem was relevance, novelty, summary quality, source quality, or something earlier in the pipeline. The verdict schema lets the feedback UI ask a more specific question. Instead of a free-text "tell us more" box, the prompt can offer a small, closed set of answers that can be joined to the judge's structured verdict: Not relevant to this companyI already knew thisThe summary is inaccurateThe source was not useful The user is not asked to understand the pipeline or diagnose the model. They only identify what went wrong from their perspective. The response is stored beside the model's original verdict in a form that can be queried and aggregated. That makes feedback reconcilable. A user selecting "I already knew this" for an item the model labeled NEW is a novelty disagreement. Aggregate enough of those disagreements and the pattern starts to localize the fault: One company produces repeated staleness feedback: its recent-coverage window may be too short.One publisher produces repeated staleness feedback: the provider may be delivering articles days after the underlying event.Staleness rises across the whole system: the ingestion or digest schedule may be too slow. In the last two cases, the model may have judged the item correctly against the context it received. The defect is that the context did not contain enough recent coverage, or that the document arrived too late to be useful. One class of staleness should never reach the model. When a user has already received coverage of an event, follow-on articles about the same event should be removed from that user's candidate pool by a deterministic lookup against serve history. That belongs in Stage 2. It is cheaper and more reliable than asking Stage 3 to rediscover a fact the system already knows. We learned this the hard way. Our first version used near-duplicate cluster IDs for deduplication, but it did not retain user-level serve history. A story that remained in the news for several days kept resurfacing in the digest through different articles. It became one of the steadiest sources of “already knew this” feedback. The novelty verdict is for the cases that a lookup cannot resolve: a document covers an already-served event, but may contain a material update. A contract win reported on Monday and revisited on Thursday with a disclosed dollar amount is not a duplicate, even though the event is familiar. UPDATE gives the model and digest assembler a distinct outcome for that case. Relevance feedback points elsewhere. If a user marks an item “not relevant” when the model returned RELEVANT with MARKET_MOVEMENT, the document may be a stock-ticker recap that slipped past initial triage. This points to a Stage 1 triage gap; the Stage 3 relevance prompt is working as intended. The enums make those distinctions visible. Free-text feedback would leave a collection of dissatisfied users and an expensive investigation. A shared vocabulary turns recurring complaints into evidence about the pipeline stage that needs work. Develop on the Large Model, Serve on the Small One Model choice in Stage 3 is a tuning decision rather than an architectural one. Candidate generation bounds the number of calls, and the output contract bounds the work inside each call. That lets you change models without changing the rest of the pipeline. We developed the prompt and ran early production on a large-tier frontier model. At that point, the output contract, reason-code vocabulary, and feedback flow were still changing. We wanted one variable we did not have to question: model capability. Debugging a prompt and a model at the same time is miserable. When a verdict is wrong, the cause could be an ambiguous instruction, insufficient target context, a missing reason code, weak novelty context, or a model that cannot reliably follow the task. Starting with the stronger model removes one of those possibilities. Once the contract was settled, we built a gold set of human-labeled candidates and measured the large model against it. We then ran the cost-optimized small model over the same set. The switch was a measured decision: the small model had to preserve the verdict quality required for the product before it received production traffic. The bounded pool made that switch viable. Judging 50 to 100 pre-vetted candidates against a fixed enum contract is narrower than asking a large model to absorb retrieval, ranking, and summarization in an unstructured pipeline. Bound the task before reducing model cost. A cascade was the obvious alternative. Systems such as FrugalGPT send requests to a lower-cost model first and escalate uncertain cases to a stronger one. That pattern can preserve quality while reducing spend when requests vary widely in difficulty. We considered it and skipped it. After Stage 2 bounded the pool and Stage 3 reduced output to a small enum vocabulary, verdict difficulty was relatively uniform. A cascade would have added a confidence estimator, escalation policy, and second production path without much remaining cost to remove. The more immediate savings came from the workload shape: Prompt caching: The instructions, enum definitions, target profile, and recent-coverage context are shared across a target's candidate pool. A stable shared prefix makes prompt caching effective.Batch processing: Daily digest generation is not latency-sensitive. Discounted batch endpoints fit the workload better than synchronous calls, as long as the batch completes before the digest send window.Per-document records: Each candidate produces an independent verdict. Retries, failures, model comparisons, and later reprocessing can happen at the document level instead of rerunning an entire target pool. The order matters. First bound the pool and then define and stabilize the contract. Measure a stronger model against human labels. Only then test a smaller model on the same set. Cost optimization is much easier when the task, failure modes, and acceptance criteria are already known. Auditing Beats Labeling at Scale You cannot label your way to confidence at this scale, but you can audit. The pipeline processes too many candidate documents to build a comprehensive human-labeled corpus or to review every model verdict. A smaller gold set, maintained over time, is enough to calibrate the judge and catch meaningful regressions. The gold set should contain enough candidates to cover the major verdict classes, common edge cases, and the document types that matter most to the product. Each example receives the same fields the model produces: relevance, novelty, and reason code. Measure agreement per field; Cohen's kappa is useful when class imbalance makes raw accuracy look better than the system really is. Calibration decays, so you have to keep redoing it. Re-run the gold set whenever you change the model, the prompt, the reason-code vocabulary, the target-profile format, or the recent-coverage context. The tier-switch evaluation in the previous section is one example: it turned a model-cost decision into a measured comparison against a fixed baseline. Human labels are expensive, so we use the cheaper signals first. The verdict distributions described earlier often reveal a change before anyone reads an individual item: A rising OTHER rate can mean the reason-code vocabulary no longer fits the documents entering the system.A rising IRRELEVANT rate for one vertical can mean the Stage 2 retrieval query or entity aliases are pulling the wrong material.A sharp change in NEW, UPDATE, or ALREADY_COVERED for one company can point to a bad serve-history window, an ingestion delay, or a change in news volume.A rising CONTRACT_VIOLATION rate indicates a schema, prompting, or provider problem. It is not a model-quality signal and should remain separate from verdict distributions. Distribution drift does not prove what broke. It tells you which targets, document types, or pipeline stages deserve investigation. That is enough to direct limited human review where it has the highest value. Thumbs-down data provides another signal, but only for content that was served. That leaves a more dangerous failure mode: a target with little or no coverage. On a typical day, only about 6,000 of our 15,000 users received a digest. For the other 9,000, the system decided that nothing was worth sending. Usually that is correct. When it is wrong, no recipient has an item to downvote. We therefore audit low-coverage targets as well as high-complaint targets. A target that normally generates thirty useful documents a week but receives three may have a broken alias, a failed source feed, an overly strict retrieval threshold, or an upstream classifier rejecting valid material. Those failures are invisible in served-item feedback. The audit loop is deliberately small: Use verdict and feedback distributions to select suspicious targets, sources, and document types.Sample candidates from those pockets, including documents that entered the Stage 3 pool and documents Stage 2 excluded.Have human reviewers apply the same relevance, novelty, and reason-code contract.Compare their labels with model verdicts and upstream exclusion reasons.Fix the stage that owns the failure, then rerun the gold set before changing production behavior. LLM judges do not eliminate human assessment. They make it selective: the contract supplies the categories, distributions identify samples, and human reviewers determine whether the system is still making the judgments the product needs. Production Notes A few practices made this stage workable in production: Make every field that downstream code branches on an enum. Keep free text for the rationale and human-facing summary. Put the rationale before the enum fields so the model can reason before committing to a label.Keep contract failures visible. A response that fails validation should become a stored CONTRACT_VIOLATION record, never a silently dropped candidate or a fake IRRELEVANT judgment. Monitor that rate separately from model-quality metrics.Turn feedback into a debugging signal that points at the pipeline stage responsible. Ask recipients why an item was unhelpful in a small, structured vocabulary. Join that response to the original verdict and look for recurring disagreements by company, source, reason code, and pipeline stage.Stabilize the task before optimizing model cost. Develop the prompt and contract on a capable model, evaluate against a fixed gold set, then test a smaller model against the same set. Otherwise, prompt defects and model capability gaps look identical.Track distributions before reading individual documents. Relevance, novelty, and reason-code shifts identify where human review is most valuable. Keep OTHER under observation, a rising share among valid verdicts means the taxonomy is falling behind the domain.Audit low-coverage targets in addition to the ones with negative user feedback. Thumbs-down feedback exists only for items that were served. A target receiving suspiciously little coverage may have a retrieval, entity-resolution, source, or triage problem that no user can report. Stage 3 must be observable as well as accurate. A model verdict that cannot be stored, queried, compared, and challenged has limited value in a production decision pipeline. Conclusion The architecture in the previous article put the expensive model behind a bounded candidate pool. This article adds the other half: define the verdict before you tune the model. A pointwise judge over a fixed pool is easier to control than a model asked to retrieve, rank, and explain everything at once. An enum-based verdict gives downstream code stable inputs, gives operators something to monitor, and gives user feedback a route back to the stage that owns the problem. The LLM's job is to judge each candidate. The pipeline's job is to make that judgment inspectable: bounded by the pool, typed by the contract, and useful to the systems and people downstream. More

Multi-Account AWS Architecture: Isolating PHI Workloads Without Slowing Down Engineering Teams

By Garik H

Most engineering teams working on healthtech applications reach a point where someone asks a question that sounds simple but isn't: How do we make sure a developer testing a new feature can't accidentally access production patient data? The answer determines whether the architecture that follows will be auditable or not. Teams that answer it with process — "we have policies about that" — spend the next 18 months patching access-control gaps that reopen every time a new engineer joins or a new service gets wired in. Teams that answer it architecturally spend a week setting up AWS Organizations correctly and then largely stop thinking about it. This article covers the multi-account architecture pattern for HIPAA-compliant infrastructure — specifically, the account structure decisions that either enforce PHI workload isolation or make it a permanent source of audit findings. Why Single-Account PHI Isolation Fails at the Seams A single AWS account running production, staging, and development workloads creates a specific problem that IAM policies alone cannot fully solve. The issue is not that IAM is insufficient as a technology. IAM policies enforced within an account are only as reliable as the discipline of the people who manage them. A policy that restricts a developer's access to production RDS today can be modified tomorrow by anyone with sufficient IAM permissions. Nothing in the account structure itself prevents the boundary from being crossed. In practice, the gaps show up in predictable ways. A pipeline service role gets broad permissions during a sprint because scoping them properly would have taken an extra hour. An engineer copies an IAM role from staging to production because it was faster than creating a new one. A debugging session in production happens under an account that was supposed to be read-only. None of these are malicious decisions. They are the natural result of putting access control boundaries inside an environment where the people who need to cross them also have the permissions to do so. The access control problem that surfaces during security reviews is almost always this one — not a missing encryption setting or an unpatched vulnerability, but access boundaries that exist on paper and drift in practice. The Multi-Account Model: Enforcement at the Boundary AWS Organizations with a properly structured multi-account hierarchy solves this problem by moving the enforcement point outside the accounts being protected. The boundary is no longer an IAM policy that someone with IAM permissions can modify. It is an account boundary that the engineers inside those accounts cannot cross, enforced by Service Control Policies applied at the organizational unit level. The recommended structure has four organizational units under the root: a Security OU containing a Log Archive account and a Security Tooling account, a Production OU containing only the Production account where PHI workloads run, a Non-Production OU containing Staging and Development accounts, and a Shared Services OU containing the account used for CI/CD pipelines, DNS, and shared tooling. The Production OU sits under its own organizational unit with SCPs that restrict what can happen inside it, regardless of what IAM policies exist within the production account itself. An engineer whose IAM role in the development account grants broad permissions has those permissions scoped to the development account. Crossing into production requires a separate role, in a separate account, with a separate set of credentials. The architectural boundary is the enforcement mechanism, not the IAM policy. The Log Archive account under the Security OU serves a specific purpose: it is the only account to which CloudTrail logs from all other accounts are delivered, and it is an account to which production engineers have no write access. This means the evidence trail for PHI access events cannot be modified by the accounts generating those events - which is exactly what auditors verify when they ask about log integrity. Service Control Policies: What to Enforce at the OU Level SCPs applied to the Production OU are where the architectural enforcement becomes concrete. The first policy prevents anyone inside the production account from disabling CloudTrail, including account administrators: JSON { "Effect": "Deny", "Action": [ "cloudtrail:StopLogging", "cloudtrail:DeleteTrail", "cloudtrail:UpdateTrail" ], "Resource": "*" } CloudTrail continuity across the full audit period is not something that should depend on engineering discipline. It should be architecturally enforced. An account that can leave the organization can escape every SCP applied to it. This policy closes that path: JSON { "Effect": "Deny", "Action": "organizations:LeaveOrganization", "Resource": "*" } PHI that moves outside defined regions may fall outside data residency commitments. This policy locks the production account to specific regions: JSON { "Effect": "Deny", "Action": "*", "Resource": "*", "Condition": { "StringNotEquals": { "aws:RequestedRegion": ["us-east-1", "eu-west-1"] } }, "NotAction": [ "iam:*", "organizations:*", "route53:*", "budgets:*", "waf:*", "cloudfront:*", "globalaccelerator:*", "importexport:*", "support:*", "trustedadvisor:*" ] } EBS encryption is not enforced by default in all account configurations. This policy makes an unencrypted volume impossible to create in the production account: JSON { "Effect": "Deny", "Action": "ec2:RunInstances", "Resource": "arn:aws:ec2:*:*:volume/*", "Condition": { "Bool": { "ec2:Encrypted": "false" } } } Cross-Account Access: The Pattern That Doesn't Create New Gaps Multi-account architecture introduces a problem engineers feel immediately: how does anything talk to anything else? A CI/CD pipeline in the Shared Services account needs to deploy to production. A developer needs read access to production logs during an incident. A monitoring service needs metrics from all accounts. The answer is cross-account IAM roles with tightly scoped trust policies. A role created in the production account with minimum required permissions defines a trust policy that allows only specific principals from specific accounts to assume it, and only under specific conditions like MFA or an external ID: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::SHARED-SERVICES-ACCOUNT-ID:role/DeploymentRole" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "deployment-pipeline-prod" } } } ] } The deployment role in the Shared Services account can assume the deployment role in production - but only that role, only from that account, and only with the correct external ID. A developer's personal IAM credentials cannot assume it. An engineer who compromises the development account cannot use that foothold to pivot into production. This pattern creates cross-account access without creating a backdoor through the account boundary. The boundary holds because the trust relationship is explicit, narrow, and auditable through CloudTrail - every role assumption generates a log entry in both accounts. What This Architecture Makes Provable The operational argument for multi-account PHI isolation often focuses on security. The architectural argument that matters more for engineering teams dealing with audits and enterprise security reviews is about provability. In a single-account setup, proving that a developer did not touch production PHI during a given period requires auditing IAM policies, CloudTrail logs, and access history, and then arguing that the policies were correctly configured and consistently enforced throughout the period. There is always a gap between what the policy said and what actually happened, and that gap is what auditors probe. In a multi-account setup, the same question has a simpler answer. The developer's credentials are scoped to the development account. The development account has no access to the production account's resources. Access to production PHI requires a separate role assumption that is logged, requires separate credentials, and would appear immediately in CloudTrail. You are not arguing that the configuration was correct. You are pointing to an architectural boundary that makes the question moot. This shift from arguable to verifiable is what separates teams that sail through security reviews from teams that spend three weeks responding to follow-up questions. The Operational Overhead Is Smaller Than It Looks The most common objection to multi-account architecture from engineering teams is overhead. More accounts means more IAM configuration, more billing to reconcile, more consoles to log into. In practice, this friction is front-loaded and largely disappears once the structure is in place. AWS Control Tower reduces the account provisioning overhead significantly - new accounts inherit the correct SCP structure, logging configuration, and security baseline automatically. Account Vending Machine patterns built on top of Service Catalog or Terraform can provision a correctly configured new account in minutes. After the initial setup, adding a new account is not significantly more work than adding a new VPC. The billing concern is resolved through AWS Organizations consolidated billing, where all accounts roll up to a single payment method with unified cost visibility. The console switching concern is resolved through IAM Identity Center, which provides a single sign-on entry point across all accounts in the organization. The overhead that remains is real but small. The alternative - treating IAM policies inside a single account as the primary PHI protection mechanism - creates ongoing operational overhead that grows with the team and never fully goes away. Final Thoughts PHI workload isolation is an architectural problem, not a policy problem. IAM policies enforced inside an account are only as reliable as the operational discipline of the team maintaining them. Account boundaries enforced by SCPs at the organizational level are reliable by construction — they hold regardless of what happens inside the accounts they protect. The multi-account structure described here is not a compliance checkbox. It is the architecture that makes the access control claims in a security review actually true rather than approximately true with caveats. When an auditor asks how you prevent developer access to production PHI, the strongest answer available on AWS is an account boundary that the developer's credentials cannot cross. Building that boundary is a week of work. Not building it is a permanent source of audit findings. More

Text Analysis Without a Backend: Replacing an LLM Call With Intl.Segmenter and 60 Lines of JavaScript

By Kevin Brown

Refcard #267

Getting Started With DevSecOps

By Akanksha Pathak

CORE

Refcard #291

Code Review Core Practices

By Vidyasagar (Sarath Chandra) Machupalli FBCS

CORE

Ampere System Profiler: A Guide to System-Level Profiling

Executive Summary The Ampere® System Profiler (ASP) is a Python command-line interface utility that uses a set of Linux profiling tools to gather system-level performance metrics while running applications of interest. The system-level collectors run in parallel and provide detailed reporting on network, disk, CPU utilization, and top functions via perf during the sample period. This is helpful to determine system-level bottlenecks. One of the foundational features of the ASP is its easy-to-read HTML reports that provide a simple view of the collectors’ outputs. Additionally, all the raw data to generate reports are saved in logs should an end user need to explore system profiles in greater detail. Running this tool is simple and provides an easy-to-run command line with minimal overhead to accurately profile any benchmark. This tool is part of the Ampere Performance Toolkit (APT) and can be used by a performance engineer for a top-down approach to root-causing performance problems. What Is the Ampere System Profiler? The Ampere System Profiler (ASP) is comprised of multiple collectors that collect: numastatsocket powerCPU utilizationnetwork utilizationperf functions They run concurrently in the background while the user collects profiles of applications or benchmarks on Ampere systems. The ASP project can be found on Ampere’s GitHub page. The core of the ASP’s utility comes from open-source Linux profilers. Why Do We Need the Ampere System Profiler? Ampere System Profiler exists as part of the larger Ampere Performance Toolkit and is used as an application or benchmark performance analysis tool, and is particularly useful in identifying system-level bottlenecks to help identify sources of performance issues. It can be used to help understand: What system resources are undersaturated or are experiencing bottlenecksOS-level metrics like IRQ affinity and context switch rateThe amount of user and kernel time being spent on the CPUApplication-level functions that consume CPU cycles during the sample period When Do We Use the Ampere System Profiler? Understanding the APEX Framework Performance tuning is a process of systematic investigation, moving from a broad, system-wide view down to the specific interactions between code and hardware.  The Adaptive Profiling and Execution (APEX) Benchmarking and Optimization Funnel Performance optimization is as much art as it is science. The APEX framework uses tools and methodologies to add structure and rigor to the process and can bridge the gap between creative intuition and empirical fact. We propose applying the APEX (Adaptive Profiling and Execution) methodology to enable root cause analysis for solving performance problems. Follow the funnel above from top to bottom to effectively use the procedure. The methodology recommends starting with assessing platform health as a first step to ensure that the platform used for performance analysis is set up well. An unhealthy platform may mislead the performance analysis. Consider capturing initial performance metrics before tuning any system or application settings. This establishes a clear understanding of the current workload and identifies key scalability knobs. We recommend using Ampere’s PerfKit Benchmarker (APB), which supports many open-source applications, to create a reliable baseline for further analysis and tuning. Next is to assess system performance and any hardware or system bottlenecks while the code is running — this is where the Ampere System Profiler (ASP) is useful to eliminate any system or resource bottlenecks. The ASP can also be used to right-size the instance shape and ensure that the compute resources are efficiently consumed by the workload. One method is to use the APB’s automated benchmarking framework to start and stop ASP’s collectors during the run phase of a given APB benchmark. This ensures that the profile is collected while critical code paths are executed and a clear profile report is generated. Once system and resource bottlenecks are eliminated, if the performance issue persists and points to CPU cycles not being used efficiently, we propose going to the next step in the pyramid and using the Ampere PMU Profiler to root-cause the issue further. Finally, system benchmarking should be done after all bottlenecks are resolved or analyzed to effectively measure the system’s performance for the workload. Following this systematic APEX methodology ensures that we eliminate possible issues as part of a structured process to efficiently conduct root cause analysis. System-Level Analysis Goal: Understand the overall system health and identify the primary resource bottleneck. Is the application limited by CPU, Memory, Disk I/O, or Network?  Key Questions: Is the overall CPU utilization high?  Is it predominantly user time or system time (application or kernel)?Is the system swapping or under memory pressure? Is the application spending a lot of time waiting for I/O (iowait)?Are there system limitations? Is the network oversaturated?Is the CPU load evenly distributed?Are the top functions mostly spent in kernel? Common Tools: sarmpstatnumastatiostatsensorsperf The Ampere System Profiler utilizes all these collectors within a single command-line interface. Example Usage and Output: Plain Text “asp -n 20 -i 2 -N eboot0 –F 99 –o example” Let’s break down this command: “asp” is the CLI utility for invoking the tool. Passing “–n” is the number of samples a user wants to collect, and “-I" is the frequency in seconds to collect each sample. This is required to capture a network interface “-N”, which is capitalized, and tells the network profiler which interface to profile. Finally, “-F” indicates the frequency rate in Hz to collect its profile. The perf frequency rate will significantly impact file size; a lower rate reduces the overall file size, which is useful for longer-running profiles. The user can then pass “-o” to designate where they want data outputted. The above command collects: 20 samplesSets interval of 2 secondsRuns for a total of (samples x interval) - 40 secondsCollects network information (-N) on NiC labeled eboot0Uses perf record collection frequency of 99 Hz as the default sampling rateWrites logs to an output directory titled “example” Metrics reported by the Ampere-System-Profiler: metric namedescription CPU Utilization Percentage of CPU Utilization over Time. Includes percent of system time and percent of user time (application) Average Per Core Utilization Average User/System time per core during sample period CPU Frequency Average per-core frequency during sample period  Socket Power Shows CPU Socket Power over Time for CPU+IO  Numastat  Shows per-node memory statistics Disk I/O Outputs Disk Bandwidth over time during sample period Network I/O Shows network bandwidth during sample period  Perf Top Functions  Shows top perf record functions as percentage of cycles during sample period. Includes application code and kernel code Case Study: Redis Performance Regression Problem statement: A 55% performance regression was observed when running Redis in a virtual machine (VM). The ASP was used to help identify and mitigate two separate issues. First, the CPU profile indicated a large proportion of %soft IRQs being handled due to network saturation generated by the memtier traffic generation utility. Unbound IRQs accounted for up to 80% core utilization for %soft IRQs compared to 45% on a competitive platform. This finding led the team to choose tcp_stream as a simple reproducer to simulate the behavior of running Redis over the network to try and investigate the issue further. Pinning the IRQs to core 1 enabled the team to isolate the perf report generated by the ASP to compare hot functions running during the benchmark with the simple reproducer. The results concluded that a large proportion of the system time is being spent copying data from kernel space to user space during the critical period of the benchmark. This enabled the team to develop mitigations for reducing CPU time spent on this hot function. The second finding occurred while running tcp_stream in a lab environment, where performance did not align with what was observed, and performance observed in the cloud environment was not reproducible on bare-metal instances. However, a new problem was uncovered. After some configuration alignment, a system profile was performed again, showing an additional hot function where the host instance spends a significant time in spin locks. This provided clues to collect lock stat reports showing much higher wait times with the malformed NIC coalescing settings. This resulted in code fixes being made to kernel code and upstreamed to larger open-source communities. Example Report – Redis Network Bottleneck and High System Time Fig 1: CPU utilization over time The red line on the time series on the left indicates that a majority of CPU utilization is occurring because of high system time. A healthy application will spend the majority of its time in user space, where CPU time does the majority of the work in application code. This high amount of system time indicates that a lot of CPU time is spent outside of critical path code. Notice as well that the green line, which indicates %IOWait, is nearly 0%, indicating little to no IO operations. Fig 2: Network utilization over time The generated Network Utilization chart shows that on this system, the NIC is fully saturated and cannot handle any more network bandwidth being sent by the memtier load generator. Fig 3: Top CPU hotspots during sample period The perf report that generates the Top CPU Hotspots output shows that the redis-server is spending the majority of cycles servicing network-related mlx5e functions to process incoming network packets in kernel space. Conclusion The Ampere System Profiler (ASP) provides an efficient, system-level view of performance bottlenecks while an application or benchmark runs. By collecting CPU (user vs kernel), NUMA, disk and network utilization, socket power, and perf-based hotspot functions in parallel, ASP helps performance engineers quickly determine whether a workload is constrained by system resources or by inefficient CPU cycles in specific call paths. Following the APEX methodology, ASP is used first to eliminate platform and resource bottlenecks; if the issue persists, you can then proceed to deeper CPU root-cause analysis with PMU-based profiling and targeted instrumentation. The resulting HTML reports and raw logs enable both fast triage and deeper investigation when needed. We invite you to download and try the Ampere Performance Toolkit from the Ampere Performance Toolkit Repository. To learn more about our developer efforts and find best practices, visit Ampere’s Developer Center and join the conversation in the Ampere Developer Community. Check out the full Ampere article collection here.

By Tito Reinhart

The New Technical Debt: Working Code No One Can Explain

For the past several years, technical debt was something that was easy to identify. It came in the form of outdated frameworks, missing documentation, messy databases, etc. It was something companies racked up by moving too fast, skipping the best course of action, and patching up old systems instead of improving them. But now, a new kind of technical debt is fast emerging. At first glance, it may not look broken. However, it may even work perfectly fine initially. The app loads just fine, the feature responds, the workflow runs, and the AI-generated module seems to pass the first round of testing. On the surface, everything looks good. But the problem starts later on, when someone asks just one simple but very crucial question: how did this actually work? That is where many modern software teams begin to really feel the weight that comes with this new reality. They get easy and prompt access to a lot of code that functions, but cannot be easily explained or trusted. This is the new technical debt. Working Code Is No Longer Enough In the past, code that worked was a major milestone for any team. If the product worked, that meant that the team could finally move ahead. If the feature could be deployed, that meant the sprint was successful. If the app did not crash, that was a cause for celebration. That mindset was not the best one to have. Modern software does not live or work in isolation. There are several programs that it’s connected to, ranging from CRMs to cloud platforms, third-party APIs, AI models, and more. Just a single feature on the software may involve five or ten systems behind the scenes. So, if the team making the product does not understand how the code works, they cannot confidently tell what went wrong if the product runs into a wall or something changes, A feature that is running perfectly today may become a liability tomorrow if no one understands how it really works. That is why explainability is becoming just as important as functionality. AI Has Made This Problem Bigger With the assistance of AI in software development, the speed at which software is being created has increased monumentally. Developers can now do their tasks, including generating functions, writing test cases, building interfaces, and troubleshooting issues, at a fraction of the time that it took before. But speed alone does not automatically improve software delivery. According to Google’s 2025 DORA report, AI can be an amplifier of an organization's existing strengths and weaknesses, which means it can improve disciplined teams while exposing weaker engineering processes. This is a major advantage when used responsibly. But it also creates a new risk. AI can help come up with code that looks clean, runs correctly, and solves the immediate problems at hand. But is your development team able to follow along with the logic behind what the AI is doing? Are they accepting the AI-generated solution just because it passes a test, or do they understand every trade-off that it makes? AI tools can help a non-technical founder build a working prototype without knowing how secure or scalable it is. Just like that, your team, though moving faster in the short term, is quietly creating systems that they cannot confidently own in the long term. This does not mean AI-generated code is bad. The issue is not the tool. It is blind adoption. This concern is already showing up in the industry. GitLab’s 2026 research found that 73% of respondents are concerned about the maintainability of AI-generated code, while 82% believe it risks creating a new form of technical debt that their organization is not prepared to manage. When you use AI to accelerate development without reviewing it or passing human judgment on it, you are not just building software faster. You are building uncertainty faster. Explainable Software Requires Better Discipline The solution that I see to this is not to slow down innovation, but to build with more clarity. The basis of explainable software comes with strong requirements. Software development teams need to know what a certain feature is supposed to do, what it’s supposed to not do, what systems it connects with, and what happens when something goes wrong. Without this context, despite how well written the code is, it can become dangerous. The next thing to keep in mind is that the architecture behind the software can and should be understood by more than one person. A strong software system should not depend on one developer’s memory. It needs to have a clear structure, documentation, test coverage, and decision record that help future teams easily understand the system. Code reviews should also evolve. You shouldn’t just be reviewing your code for syntax or performance. Teams need to start asking deeper questions like why was this specific approach chosen? What assumptions does it make? What happens if the API fails? Can another developer understand this system six months from now? These questions may seem simple, but they are what separate working software from dependable software. The New Standard: Code That Works and Can Be Explained The old standard was simple: does the code work? The new standard needs to be stronger. Can it be explained? Can it be maintained? Can it be tested? Can it be scaled? Can a new engineer understand it without reverse engineering the entire system? If the answer is no, then your company may unfortunately be a victim of technical debt. It has only managed to hide it behind a working interface. That hidden debt will eventually come due. With AI tools and automated development workflows becoming common, more and more businesses are capitalizing on the opportunities to build software quickly. I get that it’s exciting, but speed without understanding is not progress. The next generation of technical debt will not always look like bad code. It will look like code that works, until the day someone needs to scale it or explain it. And by then, the real cost will become clear.

By Asim Rais Siddiqui

Ground Truth for AI-Written Code: Why Context Matters More Than Prompts

Ground Truth for AI-Written Code Session capture, per-line attribution, and selection-bias-free agent benchmarks, on top of the Git host you already use. A technical overview for engineers and engineering leaders evaluating how much of their codebase is now written by AI agents - and who is accountable for it. 1. The Problem: Git Blame No Longer Tells the Truth On most teams, AI agents now write a large share of new code. But the tools that record who wrote what were built for humans. When an agent edits files in your working tree and you commit them, git blame attributes every one of those lines to you. The prompt that produced them, the model that ran, the cost, the number of turns, and whether the code survived the next sprint — none of it is recorded anywhere. That gap has real consequences: Provenance – no answer to “which agent, from which prompt, wrote this line?” during review or an incident.Cost and efficiency – no ground truth on what a feature cost in tokens and dollars, or which agent got therein fewer turns.Quality – no measure of whether agent-written code survives, or gets reworked and reverted days later.Comparison – “which agent is better for us?” answered by vibes, because every naive comparison is poisoned by selection bias (the hard tasks go to the agent you already trust). Origin closes that gap. It captures the full agent session — prompt, diff, tokens, cost, tools, duration — attributes every surviving line back to an agent and a prompt using Git as the source of truth, and turns that data into honest, selection-bias-free comparisons between agents. It runs on top of GitHub or GitLab; there is nothing to migrate. 2. How Origin Captures an Agent Session Capture is deliberately boring and durable. A one-time origin enable registers the machine, auto-detects installed agents (Claude Code, Codex, Cursor, GitHub Copilot, Gemini, Aider, Devin, Antigravity, and more), and installs two kinds of listeners: Agent hooks – Origin hooks fire on the agent’s lifecycle events (session start, each user prompt, eachtool/file edit, and stop/end). They record the prompt text, the per-turn file diff, token and cost counters, tool calls, and the model.Transcript watchers – for agents that keep a durable on-disk transcript (e.g., Codex’s rollout logs, Devin’s local session DB), Origin reads that record directly instead of depending on hooks. The principle: if there is an authoritative transcript, read it; hooks are for context and policy. Capture is resilient by design. It writes locally first, retries on a durable queue when the network is down, resolves session end from heartbeat liveness rather than a fragile inactivity timer, and is aware of Git work trees so parallel sessions don’t collide. Sessions that never produced real work are swept so counts reflect reality. Figure 1. Every AI coding session Origin captured — agent, model, cost, tokens, branch, and review status. This is the raw material everything else is built on. 3. Attribution: First-Author Wins, With Git as the Source of Truth Recording a session is easy; attributing lines correctly is the hard part, and it is where Origin is opinionated. The model is first-author-wins: a line is credited to whoever introduced it, and later edits never reclaim it. For pushed commits, Git is the ground truth — Origin reconciles its capture against the committed diff rather than trusting a possibly lossy hook stream. A suite of invariants guards the accounting so numbers never drift: InvariantWhat it guaranteesFirst-author-winsA line counts once, for its original author - no double-credit when it’s later touched.Git-truth reconciliationPushed-commit line counts come from the real diff, not the (lossy) live hookstream.Hunk-aware countingAdd/remove tallies parse diff hunks correctly; content lines aren’t miscounted.Writes never claim linesA write/format/no-op operation cannot claim authorship it didn’t earn.Missing-commit self-healA commit-and-exit race is reconstructed at read time from the transcript-attested SHA. Figure 2. One session, decomposed: each prompt and its diff (committed vs uncommitted), the linked commit, and a 100%-AI verdict - the ground truth per-line blame is built from. The AI Blame tab drills to the line level. 4. Prompt-Level Time Travel Because Origin records the state before every prompt, each prompt becomes a restore point. You can undo an agent’s changes — the files revert — without rewriting or losing your commits. 5. Honest Benchmarking: The Agent Scorecard Once sessions are captured and attributed, Origin computes a per-agent scorecard — efficiency, outcome, and survival — for your real work. The point of difference is honesty: the scorecard refuses to draw conclusions the data can’t support. MetricDefinitionCost/taskMean cost per completed session for the agent.Tokens/produced lineToken spend normalized to lines that actually shipped.Median turnsHow many prompts it took to finish - lower is tighter.First-pass approvalShare of reviewed sessions approved without changes.Cost/merged PRDollars per PR that actually merged (outcome, not activity).Code survival @ 7/30dFraction of authored lines still present a week/month later.Rework rateThe inverse - how much of the agent’s output got reverted or rewritten. The guardrails matter as much as the metrics: Minimum sample size – agents aren’t ranked on a handful of sessions; below a threshold, a metric is shown as “not enough data,” not a misleading average.Confidence intervals – ratio metrics (e.g., tokens-per-line) carry a CI, so a noisy small sample can’t masquerade as a clear winner.Estimated tokens excluded – sessions whose token counts were estimated rather than reported are flagged and kept out of the money math.Line-weighted authorship – the AI-vs-human percentage is weighted by lines, not session count, so one giant human commit doesn’t get outvoted by many tiny agent ones. Figure 3. The agent scorecard — cost, tokens-per-line, median turns, approval, and survival per agent, with sample-size and confidence guardrails. 6. Bake-Offs: The Selection-Bias-Free Comparison The scorecard measures agents on the work you happened to give them — and you give the hard tasks to the agent you trust, which skews every comparison. A bake-off removes that bias by construction: it runs the same prompt through N agents, each in its own isolated Git work tree, and lets you compare the results side by side. Every arm gets identical work. Architecture: The Server Schedules, Your Machine Executes Coding agents run on your machine, with your keys — Origin’s cloud can never run them. So a bake-off is split cleanly in two: the server owns the queue and the schedule; a local runner daemon owns execution. Each arm branches from HEAD into bakeoff/<id>/<agent>, the agent works autonomously and commits, and Origin correlates the result back to the branch via normal session capture — nothing extra to wire up. The list nests each arm’s session inline (cost, tokens, lines, status), filters by status/repo/agent, pages ten at a time, and rolls up a head-to-head agent comparison across every bake-off you’ve run. Deliberately, Origin does not auto-declare a winner. It tints the cheapest and fewest-turns arms to help you scan, but “cheapest” and “best” are not the same thing - only a human reading the diff can decide. You pick the winner.Figure 4. Composing a bake-off — the same prompt, two or more agents, each running autonomously in its own git work tree. Results stream back as sessions and roll up into a head-to-head comparison. 7. For Teams: Governance Without a Second Source of Truth Everything above is per-developer value that also aggregates for a team. On top of it, Origin adds an org layer: typed policies enforced across review, PR checks, and CI; AI auto-review of agent sessions; secret and PII scanning on captured diffs; budgets and cost controls with per-agent visibility; role-based access; and an organization dashboard that shows what share of the codebase is AI-authored, by whom, at what cost - line-weighted, not guessed. Because attribution is per line and travels with the repo (prompts are carried in Git notes, and a dedicated sessions branch makes context portable across clones), the governance view is derived from the same ground truth developers see — not a parallel system that drifts. 8. Architecture, Privacy, and Getting Started Local-first capture – session data is recorded on your machine first. A fully standalone mode keeps everything in the repo with no account at all.Sits on your host – GitHub and GitLab, multiple connected accounts, native Windows/macOS/Linux CLI.No repo migration.Portable provenance – prompts live in Git notes; the origin-sessions branch is a zero-tooling vehicle, so a fresh clone still has the history.CLI-native – the CLI is a single Node binary distributed via signed GitHub releases; the platform API runs on a small, boring stack (Express + Prisma). Origin turns the invisible half of your codebase — the half an agent wrote — into something you can read, attribute, price, and compare. Solo, it’s your provenance and undo button. For a team, it’s the ground truth under every AI-code decision.

By Troian Serhii

Commissioning at Scale Is a Sequencing Problem, Not a Testing Problem

The first site I ever failed to place in service passed every acceptance test I wrote for it. Cameras streamed. Switches held their uplinks under a simulated fiber cut. Audio was intelligible at every measurement point. The paperwork was clean. It still sat dark for three weeks, because the room where one of the redundant paths terminated belonged to a different crew on a different contract with a different completion date, and nobody had drawn that edge on any schedule. My validation was fine. My ordering was wrong. I have spent about a decade deploying networked systems across large numbers of physical sites in a public transit environment, running in parallel, under live operating conditions, with no maintenance window that lets you take the whole thing down. Surveillance endpoints, station-level LANs, public address, intercom, two-way emergency communications. The engineering content of any single site is not especially exotic. What breaks at scale is not the depth of your testing. It is the order in which work becomes possible. Software teams hit the identical wall the first time they go from one deployment target to sixty. So I want to lay out the methodology that actually held up, in terms that map onto a release pipeline, because that is the shape of the problem. Sites Are Environments, Not Projects The instinct on a multi-site program is to treat each site as a self-contained project: design it, build it, test it, close it, move to the next. It feels rigorous. It is also the single most expensive decision available to you, because it forces every skill in the program to be present at every site, sequentially, and your throughput collapses to the speed of your scarcest crew. The reframe that fixed it for me: a site is an environment, not a project. Environments are provisioned from a shared definition. They differ by a small set of declared parameters and nothing else. If two sites differ in a way that is not captured in that parameter set, that difference is a defect in my design, not a fact about the world. Once I held that line, everything downstream got cheaper. Configuration became generation. Testing became parameterization. And the schedule became a dependency graph instead of a list. Addressing Is a Schema, Not a Spreadsheet For the first phase, I inherited what most programs have, which is a spreadsheet of IP assignments maintained by whoever touched it last. It works for a dozen devices. At a few hundred, it starts producing collisions and orphaned addresses, and at a thousand it produces the worst failure mode there is: a device that answers on the network but is not the device you think it is. So the addressing plan became a schema with a deterministic derivation, and the spreadsheet became a generated artifact rather than a source of truth. YAML # sites/site-042.yaml site_id: 042 tier: modernization mgmt_supernet: 10.42.0.0/16 subnets: management: { offset: 0, size: 24 } surveillance: { offset: 16, size: 22 } audio: { offset: 32, size: 24 } intercom: { offset: 40, size: 24 } endpoints: surveillance: - { tag: CAM-P-01, zone: platform_north, switch: SW-A, port: 1, poe: true } - { tag: CAM-P-02, zone: platform_south, switch: SW-A, port: 2, poe: true } intercom: - { tag: INT-EL-01, zone: elevator_lobby, switch: SW-B, port: 7, critical: true } Addresses are derived from site_id, the subnet offset, and the endpoint's index within its class. Nobody assigns an address by hand. A rendering step turns this into switch configuration, into the label schedule the field crew prints, and into the test suite. One input, three outputs that cannot drift from each other. Python def render_switch_config(site, switch_id): lines = [f"hostname {site['site_id']}-{switch_id}"] for cls, eps in site["endpoints"].items(): vlan = VLAN_MAP[cls] for i, ep in enumerate(e for e in eps if e["switch"] == switch_id): lines += [ f"interface Gi1/0/{ep['port']}", f" description {ep['tag']} {ep['zone']}", f" switchport access vlan {vlan}", " spanning-tree portfast" if not ep.get("critical") else "", f" power inline {'auto' if ep.get('poe') else 'never'}", ] return "\n".join(l for l in lines if l) The value here is not elegance. It is that a configuration error is now a class of error rather than an instance of one. When I found a wrong VLAN on one intercom port, I knew immediately whether it was a typo at one site or a bug that had shipped to forty. That distinction is the difference between an afternoon and a month. Batch by Equipment Type, Not By Site Here is the sequencing change that bought back the most schedule. Crews are specialized. The people who terminate and splice fiber are not the people who mount and aim cameras, who are not the people who tune audio for intelligibility, who are not the people who witness a formal acceptance test with an inspector present. If you sequence site by site, each of those crews shows up, works for a day or two, and leaves, and you pay the mobilization cost every time. Worse, the sequence is serial per site, so the whole program moves at the pace of one site's critical path multiplied by the number of sites. Group by equipment class across sites instead. The fiber crew runs its scope across a cluster of sites in one pass. The endpoint installers follow a fixed number of sites behind. The commissioning engineer follows them. It looks exactly like a staged pipeline, and it behaves like one: the throughput is set by the slowest stage, and work in progress between stages is inventory you are carrying. The thing that makes this legal, rather than reckless, is that batching only works when the stage boundary is a real gate with a machine-checkable entry condition. Otherwise, you are just moving unfinished work forward and discovering it later, at the most expensive possible moment, which is with an inspector standing next to you. Python GATES = { "physical_ready": lambda s: s.fiber_certified and s.power_energized, "network_ready": lambda s: s.config_pushed and s.uplinks_redundant, "endpoint_ready": lambda s: s.all_tags_resolve and s.poe_budget_ok, } def promotable(site, stage): return all(check(site) for name, check in GATES.items() if STAGE_ORDER.index(name) <= STAGE_ORDER.index(stage)) A site that fails physical_ready does not get an endpoint crew scheduled. Not "gets one and we will sort it out." Does not get one. Every exception I ever granted to that rule cost me more than holding it would have. Acceptance Tests Written Once, Parameterized Forever Because sites are environments, the acceptance suite is written against the manifest rather than against a site. Python @pytest.mark.parametrize("ep", endpoints_of_class("surveillance")) def test_stream_survives_uplink_failure(ep, site): with degrade_uplink(site, "SW-A"): assert stream_continuous(ep.address, seconds=120) assert resolved_tag(ep.address) == ep.tag That second assertion is the one I care about most. It checks that the device answering at an address is the device the design says should be there. Physical-world deployments generate transposition errors constantly; two ports swapped during termination, and a suite that only tests function will pass happily on a swapped pair. You will find out during an incident, when someone pulls up the wrong view. Phase One Writes the Template Whether You Intend It To or Not The procedures we developed during the initial rollout became the template for every later phase of the same multi-year program. That was mostly not deliberate. It happened because those procedures were the only written record of why a given check existed, and later teams adopted them rather than rediscover the reasoning. Which is worth saying plainly: your first phase is authoring the standard for everything that follows, and the artifacts that survive are the executable ones. Narrative test procedures rot. Nobody reads the PDF. A parameterized suite and a manifest schema get run, and when someone changes them, the change is visible. The rework we avoided in later phases did not come from testing harder. It came from the fact that the definition of "done" for a site was identical in phase three and phase one, and a new engineer could read it in an afternoon. Where It Still Breaks I do not want to oversell this. Two things reliably escape the model. The first is anything genuinely site-specific: a structure with an unusual pathway, an interface to an older system that predates the standard. Those exist. The discipline is to name them as exceptions with their own schedule, not to loosen the standard so they fit inside it. One exception absorbed into the template contaminates every site that follows. The second is that batching increases the blast radius of a design defect. When configuration is generated, a bad rule ships everywhere at once. That is the trade you accept, and the mitigation is the same one teams use: a canary. The first site through each stage gets scrutiny nobody else gets, and nothing promotes behind it until it clears. Coverage was never my constraint. Order was.

By Savni Sandbhor

The Real Skill Stack Behind Production-Ready AI Engineers

I've spent the better part of two years watching teams ship agentic AI systems, and a pattern keeps repeating. Two engineers read the same LangChain docs, attend the same conference talks, and build systems that look identical in a demo. Six months later, one system is handling thousands of requests a day with predictable behavior. The other gets quietly replaced by a simpler rules engine after it embarrassed someone in front of a customer. The gap between those two outcomes has almost nothing to do with model choice or framework familiarity. It comes down to a small set of skills that don't show up on most job postings for AI engineers, and that most online courses skip entirely. What Makes an Agentic AI System Different From a Chatbot Wrapper A chatbot wrapper takes input, sends it to a model, and returns the output. An agentic system makes decisions across multiple steps, calls tools, holds state, and sometimes calls itself. That difference sounds small written down. In practice, it changes everything about how the system fails. A wrapper that gives a bad answer wastes one turn. An agent that makes a bad decision at step two can compound that mistake across steps three through fifteen, calling the wrong API, writing bad data to a database, or looping on a task it can't complete. The failure modes are different in kind, not just in severity, and engineers who haven't built agentic systems before tend to debug them like they would debug a single bad response. They look at the final output instead of the decision trail that produced it. Skill One: Building Evaluation Before Building Features Most teams build the agent first and figure out how to test it later. The engineers who ship reliable systems do the reverse. Before writing the orchestration logic, they write a set of test cases the agent has to pass, with clear pass and fail criteria, and they run those cases against every change to the prompt, the tool definitions, or the model version. This sounds obvious stated plainly. It's rare in practice because agentic systems resist the testing patterns engineers already know. A unit test checks one function against one expected output. An agent's output depends on the conversation history, the tools available at that moment, and the specific phrasing of the user's request, so a single test case doesn't generalize the way a unit test does. Engineers who handle this well build small evaluation harnesses early, often before the agent does anything useful. They run twenty or thirty scenarios that represent the range of things the agent will see in production, including edge cases that look like they shouldn't happen. Then they track pass rate as a number they watch the same way they'd watch latency or error rate. When someone tweaks a system prompt to fix one issue, the harness catches the three other things that broke as a side effect. I've watched a team skip this step on a customer support agent, ship it, and discover three weeks later that a prompt change meant to improve tone had quietly disabled the agent's ability to escalate billing disputes to a human. Nobody caught it because nobody was running scenarios that exercised that path. A harness with even ten well-chosen test cases would have flagged it the same day. Skill Two: Treating Tool Definitions as an API Design Problem The tools an agent calls function as its only way of acting on the world, and most engineers write tool definitions the way they'd write internal function signatures: quick names, minimal descriptions, parameters that make sense to the person who wrote the code. That approach breaks down because the agent reads the tool description the same way it reads everything else, as natural language it has to interpret. A tool called search with the description "searches things" gives the model almost nothing to work with when it's deciding whether to call that tool or a different one, or what to pass as the query. Engineers who get this right write tool descriptions the way a technical writer would write public API documentation. They specify exactly when the tool should be used, what it returns, and what it doesn't do. They name parameters so the intent is obvious without a comment. A tool called search_customer_orders_by_email with a description stating it returns orders from the last 90 days and requires a verified email address gives the model far less room to misuse it than a generic search function does. This matters more as the number of available tools grows. An agent choosing between three tools can often guess right even with weak descriptions. An agent choosing between twenty tools, several of which sound similar, needs descriptions precise enough to disambiguate. Teams that scale past a handful of tools without revisiting this usually see a spike in wrong-tool-selected errors, and the fix is rarely a smarter model. It's better documentation. Skill Three: Designing for Partial Failure Traditional software either works or throws an exception. Agentic systems fail in a third way: the call succeeds, the response looks reasonable, and the content is wrong or incomplete. A tool call to fetch inventory data might return successfully while returning stale numbers. The model might decide a task is complete when it's only handled part of it. Engineers who've shipped production agents build explicit checkpoints into the flow where the system verifies its own progress against the actual goal, not just against whether the last API call returned a 200 status code. This might mean a verification step after a multi-stage task, where a separate prompt checks the agent's claimed output against the original request. It might mean structured outputs at each step that a deterministic function can validate, rather than trusting free text all the way through. The instinct to add more error handling here is correct, but the specific shape matters. Wrapping every tool call in a try-except block catches crashes. It doesn't catch an agent that confidently reports success on a task it didn't finish. That requires building verification logic that understands the task, not just the mechanics of the call. Skill Four: Knowing When Agentic Architecture Is the Wrong Choice The most consistent marker I've found for engineers who build agentic AI systems well is a willingness to argue against using one. 2026 has pushed agentic AI into the default answer for almost any automation problem, and that default is wrong often enough to matter. A task with a fixed sequence of steps and no real decision points doesn't need an agent reasoning through it each time. A deterministic pipeline runs faster, costs less, and fails in predictable ways that are easier to debug at 2 a.m. The engineers I'd trust with a production system are the ones who can look at a proposed agentic workflow and say plainly that a simpler architecture handles 90% of the cases just as well, reserving the agent for the genuine judgment calls. This isn't a popular position to take in planning meetings right now, with enterprise adoption of agentic AI accelerating across every sector and budget approval often tied to whether a project sounds sufficiently advanced. But the systems that hold up under real traffic tend to be the ones where someone pushed back on scope early, kept the agentic part narrow, and let boring code handle everything that didn't need a model making decisions. What This Looks Like Six Months In None of these four skills show up in a typical technical interview. They show up in incident reviews, in the difference between a system that degrades gracefully and one that fails in ways nobody anticipated, and in whether an engineer can explain why their agent made a specific decision three steps into a failed task. The teams shipping agentic systems that survive contact with real users aren't the ones with the most sophisticated prompts or the newest framework. They're the ones who treated evaluation as infrastructure, wrote tool descriptions like public documentation, built verification into the architecture instead of bolting it on after an incident, and stayed honest about when an agent was the wrong tool for the job. That combination is harder to hire for than "experience with LangChain" or "familiarity with RAG pipelines." It's also the actual difference between a demo and a system someone can depend on.

By Joshua Shelton

Cutting AI Token Costs With MgntUtils Stack Trace Filtering

A live production integration case study. Introduction and Purpose of This Article This article is written for mid- and high-level managerial and technical decision-makers. I am the author of the open-source Java library MgntUtils. The article presents an analysis of a real integration of the stack trace-filtering feature from that library into a live commercial production environment. A few important clarifications up front: This is not a side-project pilot and not a lab demo. The feature was integrated into a production service of a company that serves a high volume of real customers. Due to legal constraints, I am not at liberty to name the company.This is not a how-to article for implementers. If you came looking for code samples or logging-framework wiring, please see the dedicated articles listed in the Disclaimer below.MgntUtils can be used in Java projects and in other JVM-based languages such as Kotlin. Before diving into the production numbers, it is worth stating briefly what the feature does and why those numbers matter. Server-side stack traces are usually full of framework and infrastructure noise — proxies, filter chains, containers, thread pools, and similar boilerplate — while the few lines that actually explain the failure are easy to lose in the pile. The MgntUtils filtering utility keeps the application frames and the exception / Caused by chain, and collapses that noise. The result is a much shorter stack trace without losing the information you actually need. When those stack traces are later consumed — sent to an LLM for analysis, or opened by an engineer — that reduction can mean: Substantial AI token savingsTypically more accurate AI root-cause answers, because the model has less framework noise to latch onto and hallucinate aboutA meaningful productivity boost for human triage The rest of this article focuses on what was observed after integrating this feature in production: the measured benefits, how to interpret them, and the integration experience itself — including gotchas that only surfaced in a real live environment, as opposed to a pilot project. Disclaimer This article deliberately does not discuss the technical design of stack trace filtering or the technical details of the integration. Each of those topics has its own dedicated article: Filtering Java Stack Traces With MgntUtils Library DZone: https://dzone.com/articles/filter-java-stacktrace-mgntutilsDEV Community: https://dev.to/mgantman/java-stacktrace-filtering-utility-1c1i Zero-Code-Change Stack Trace Filtering for Spring Boot: An Infrastructure-Level Integration DEV Community: https://dev.to/mgantman/zero-code-change-stacktrace-filtering-for-spring-boot-an-infrastructure-level-integration-3fk5 Production Results and Benefits Below are the observations and conclusions from monitoring the live production system after the feature integration. The feature had been running for about a month, and filtering was also temporarily turned off for comparison. What the Production Environment Looked Like Anonymized sketch of the deployment (enough to judge fit, without identifying the company): High-traffic JVM/Spring Boot service in a commercial production estateStructured JSON logging to a major observability platformObservability billing dominated by per-event (not per-byte) pricingIn a typical production day, that service emitted on the order of ~70,000+ log events carrying a stack trace That is a large stream of stack trace payloads — expensive if fed to an LLM, and tiring if engineers open them by hand. Stack Trace Volume Reduction Range in Production Filtering was measured across production stack traces with filtering on vs off. Observed size/token reductions typically fell in roughly the ~75%–95% range: Toward the high end (~90–95%): framework-heavy request-handling traces (long security/container/proxy tails)Toward the lower end (~75%+): more application-dense traces, where a larger share of frames is your own code The average reduction on a typical trace in this environment was about ~91%. The table below is a real before/after example — shown so you can see what that looks like in practice: MetricUnfilteredFilteredReductionLines19518~91%Bytes~22,200~1,900~91%Input tokens (approx.)~6,300~540~91%Application framesall (buried in noise)all (kept)no signal lost Every application frame in the business call path was retained; what disappeared was framework and infrastructure noise (proxies, filter chains, container/thread-pool frames, and similar boilerplate). Stack traces tokenize poorly for LLMs — package separators, generated class names, and (File:line) markers all split into extra tokens — so the token reduction tracks the size reduction closely. Root-cause readability was unchanged. In both versions, the failure was identifiable from the application frames and the exception message. Filtering did not remove diagnostic signal; it removed the large majority of the payload that never helped. What Improved AI analysis: cheaper and more accurate (when exceptions are analyzed). For every exception sent to an LLM, the stack trace input payload shrank by roughly ~75–95% depending on the trace shape (~5,800 tokens saved on a typical ~91% trace). That saving repeats for every analyzed event. In an environment where tens of thousands of stack traces are emitted per day, any AI triage, clustering, or “explain this error” pipeline pays that tax over and over unless the noise is stripped first. Cost is only half of the AI benefit. Filtering also improves answer quality. The removed frames are framework and infrastructure boilerplate — identical across many errors and unrelated to the application failure. When those frames remain in the prompt, models often latch onto them and hallucinate a root cause in the noise. With them collapsed, the model is steered toward the application frames and exception message that actually explain the failure — so analysis is not only cheaper, but typically more accurate. Sensitivity calculator (illustrative — not this company’s AI spend). If your org analyzes exceptions with an LLM, you can size token cost roughly as: Plain Text annual token saving ≈ (exceptions analyzed per year) × (tokens saved per exception) × (model input price per token) Using ~5,800 tokens saved per exception (average on a typical ~91% trace) and an illustrative model input price of $3 per 1 million input tokens: Analyzed exceptions / dayApprox. tokens saved / dayApprox. saving / year5,000~29M~$32K50,000~290M~$318K250,000~1.45B~$1.6M Plug in your own analysis volume, your place in the ~75–95% reduction range, and your model pricing. The production measurement that is firm is the observed per-exception reduction range, with application frames preserved. Secondary AI upside: More errors per context window. Because a typical filtered stack trace is so much smaller (~540 tokens vs ~6,300 in the example above), many more distinct exceptions fit into a single model call. That is a capability change, not just a cost saving: cross-error analysis — clustering failures, or asking “what went wrong in the last N hours?” — becomes practical instead of blowing the context window on framework noise. It is secondary to the per-exception token and accuracy benefits, but it matters for any AI workflow that looks at more than one error at a time. Human triage productivity. Engineers reading a filtered typical trace see the full application call path at the top (~18 lines in the example above) instead of scrolling through ~195 lines to confirm there is no hidden nested cause and to piece the business path together. For on-call and incident review, that is a direct readability win. What Changed in Log Volume — and What Did Not It helps to separate event count from bytes per event. Event count did not change. A stack trace is still one log event whether it is 195 lines or 18. If your observability vendor bills per event (or per indexed log line item), filtering does not reduce that charge. In this production environment, that was the dominant billing model — so there were no savings on a per-event bill. Bytes per stack trace event did change. Each filtered stack trace was roughly ~75–95% smaller than its unfiltered counterpart (commonly ~90% for framework-heavy traces). There is a real reduction in stack trace payload size. How much that shows up in total log volume is not deterministic. Overall space / ingested-byte savings depend on what share of all logs are stack traces: Plain Text overall byte reduction ≈ (stacktrace share of total log volume) × (~75–95% reduction on those stacktraces) In this company’s environment, stack traces were only about ~1% of total log volume — which is unusually low (an anomaly for many systems, but what we observed here). Cutting ~90% of that 1% yields only a fraction of a percent of total logs, which is easy to lose inside normal day-to-day traffic variance. That is why aggregate ingested-byte charts did not show a clear step when filtering was toggled. In another organization where stack traces are a much larger share of log volume, the same per-trace cut would produce a more visible space saving. Those savings are real in principle, but variable by workload and not the main point of this case study. The main point here is consumption cost. The firm, repeatable benefit we are highlighting is what happens when a stack trace is analyzed by an LLM or read by an engineer: large payload reduction, same diagnostic signal. Treat log-space savings as a possible secondary effect, sized by your own stack trace-to-total-logs ratio — not as the success criterion for this feature. How to Read These Results as a Decision Maker QuestionAnswer from this production caseDid filtering remove useful diagnostic information?No — application frames and exception chain structure remained.How large is the per-exception reduction?Roughly ~75–95% across production traces (often ~90%+ on framework-heavy request traces).Does that reduce per-event log billing?No — event count is unchanged.Is there space / byte saving?Yes per stack trace (~75–95%); overall only if stack traces are a meaningful share of total logs (here ~1%, so barely visible).Where is the upside for AI analysis?Far fewer tokens and less hallucination on framework noise — cheaper and typically more accurate.AI context-window upside?More exceptions fit in a single context window — useful for clustering or “what failed in the last N hours?” analysis.Other upside?Time saved when humans read errors.Who should adopt it?Teams that already (or soon will) send production exceptions to LLMs at volume, and/or teams whose engineers routinely open noisy stack traces. The production evidence supports a clear, bounded claim: when stack traces are consumed, filtering delivers a large, repeatable reduction in payload size with no loss of application signal. Per-event log bills do not drop. Overall log-space savings may exist but depend on stack traces’ share of total volume — and are not the primary reason to adopt the feature. Integration Experience I started from an implementation I already had in the MgntUtilsUsage side-project repository — a runnable Spring Boot demo of MgntUtils features, meant to emulate real-life apps as closely as possible. It was a very good starting point. Still, as I worked through the live commercial integration, a few gotchas surfaced that a single-JVM demo simply does not force you to confront. Gotchas That Showed Up in a Real Production Environment 1. Feature Toggle Storage Across Multiple Containers My demo app runs in a single JVM. A real production service typically runs on several containers that scale in and out. In the demo, the on/off flag for stack trace filtering lived in memory — which is fine for one process, and useless once you have more than one. In a multi-container environment, you need an external, shared flag holder that every instance can read. Redis (or an equivalent shared store available to all containers) is a good candidate. 2. JSON Logging Adapters, Not Only the Classic Logback Pattern When I first modified the Logback configuration, my demo mainly used conventional Logback pattern-based adapters. A real production app will most likely also use a JSON encoder for external logging systems such as Datadog (and similar platforms). That special adapter has its own throwable-handling path, so wiring the filter there is a must — otherwise you can end up with filtered console output locally and unfiltered stack traces in the system that actually matters. 3. Hardening the Fail-Safe Path A fall-back option already existed for the case where anything goes wrong inside the filtering path. For production, that fail-safe had to be hardened a bit further to make it as bullet-proof as possible: if filtering ever fails, the system must still emit a full standard stack trace and must never drop the log event. 4. Logback Is Not the Only Popular Logging Framework This company uses Logback, so that is what the production integration targeted. But Logback is not the only widely used option — my own favorite, for example, is Log4J. For the dedicated integration article (linked in the Disclaimer), I also had to provide Log4J instructions, even though Log4J was not used in this particular environment. Anyone planning an org-wide rollout should assume more than one logging stack may need to be covered. Effort, Timeline, and Outcome All in all, the integration was smooth, and the side-project was close enough to the final result in the real app. About 4–5 hours to get an integrated version up and running in the staging environmentAbout one day of observing staging to make sure there were no unexpected behaviorsThen deployment to production, with about another day of close monitoring before declaring the feature live So roughly half a day of integration work, and about 1.5 working days of testing / staging observation / production monitoring. Not a single bug was found. There are two contributing factors for that: The stack trace-filtering feature itself is mature and battle-tested — I am tempted to say it has no bugs, but let’s just say it is highly stable and reliable.The integration itself is simple enough. The next integration should be even faster, since this one is now well documented (including the dedicated Spring Boot integration article linked in the Disclaimer). If you are interested in integrating this feature into your project, the detailed integration instructions are in the article Zero-Code-Change Stack Trace Filtering for Spring Boot: An Infrastructure-Level Integration. If you are interested in support for the integration, feel free to contact me at or through my LinkedIn profile. Conclusion This case study supports a simple decision: Adopt stack trace filtering if your organization already analyzes production exceptions with LLMs at a meaningful volume, or if engineers routinely open noisy stack traces during triage and on-call. In those cases, the live evidence is clear: typically about ~75–95% less stack trace payload (around ~91% on a typical trace), with application frames preserved — cheaper AI analysis, typically more accurate answers, and easier human reading. Do not adopt it expecting your per-event observability bill to drop, or expecting a large automatic cut in total log volume. Event count does not change. Overall byte savings depend on how large a share stack traces are of all logs — and that varies by organization. Consumption cost is the main point; log-space savings are secondary and workload-dependent. On effort and risk: in this live commercial integration, getting to staging took about half a day of work, followed by roughly a day and a half of staging observation and production monitoring. No bugs were found. The feature is mature, the integration is simple, and the demo-to-production gaps (shared toggle, JSON logging adapters, fail-safe hardening, and covering more than one logging framework) are now documented. If that profile matches your environment — high exception volume that is actually consumed by AI or by people — this is one of the cheaper, lower-risk improvements available. If exceptions are mostly logged and rarely looked at, the benefit will be thin, and that is an honest reason to pass.

By Michael Gantman

Building Meeting Audio RAG on Microsoft Foundry With Fast Transcription and Foundry IQ

Every recorded meeting your organization has ever held is already a knowledge base. It just happens to be stored in the least queryable format imaginable, which is a wall of MP4 files sitting in a storage account that nobody opens twice. The good news is that the gap between that wall of files and a working question-answering agent is now much shorter than it used to be, because Microsoft Foundry ships the two halves you need in one place. Fast transcription turns the audio into diarized text in seconds rather than in real time, and Foundry IQ turns that text into a permission-aware knowledge base that any agent can query through a single endpoint. This walkthrough builds the whole thing end to end. By the end you will have a pipeline that watches a blob container for new recordings, transcribes them with speaker labels, chunks them into speaker turns with enough metadata to make citations useful, indexes them as a Foundry IQ knowledge source, and exposes a Foundry agent that answers questions like "what did we decide about the pricing migration in Q2 and who pushed back" with real references back to the moment in the recording. A quick naming note before we start, because the ground has moved. At Ignite 2025, Microsoft renamed Azure AI Foundry to Microsoft Foundry, and the rename was formalized in the January 2026 Product Terms. The platform is the same platform, but there are now two portal experiences and two generations of SDK. The 2.x preview of azure-ai-projects targets the new Foundry portal and API, and the 1.x GA line targets what the docs call Foundry classic. Everything in this article uses the 2.x line and the Responses-based agent surface. What We Are Building, and the Shape of the Data Flow The pipeline has two independent halves that meet at a blob container of curated transcripts. The ingestion half is batch and event-driven. It cares about throughput and about not losing files. The retrieval half is synchronous and user-facing. It cares about latency and about grounding quality. Keeping them decoupled through storage means you can reindex, re-chunk, or swap the retrieval strategy without touching a byte of audio again. The flow is worth reading left to right once. A recording lands in raw-recordings. Event Grid picks up the Blob Created event and drops a message on a queue, which gives you retry semantics and a dead letter path for free. A queue-triggered Function pulls the message, POSTs the audio to the Foundry Speech fast transcription endpoint, and gets back a synchronous response containing diarized phrases. A second stage groups those phrases into speaker turns, attaches timestamps and meeting metadata, and writes JSONL into curated-transcripts. Foundry IQ indexes that container on a schedule. Why a queue between Event Grid and the Function rather than a direct trigger? Because fast transcription is synchronous and the audio files are large. A direct blob trigger gives you very little control over concurrency, and the first time somebody bulk-uploads six months of archived recordings, you will saturate your Speech resource and start collecting 429s. The queue lets you cap batchSize in host.json and shape the load. Standing up the Foundry Project and the Speech Resource Create a Foundry project first. In the portal, make sure the New Foundry toggle is on, then create or select a project. The thing you need out of the portal is the project endpoint, which has the form https://<resource-name>.services.ai.azure.com/api/projects/<project-name>. Install the preview packages. Shell pip install "azure-ai-projects>=2.4.0" azure-identity openai azure-storage-blob requests az login Entra ID is the only authentication method the project client supports, so there is no key-based escape hatch here. Give yourself the Azure AI User role on the project resource for development work. For the pipeline itself, use a user-assigned managed identity and grant it Azure AI User plus Storage Blob Data Contributor. Two environment variables carry the rest of the article. Shell export FOUNDRY_PROJECT_ENDPOINT="https://your-account.services.ai.azure.com/api/projects/meetings" export SPEECH_RESOURCE_NAME="your-speech-resource" Confirm the project client talks to the service before you build anything on top of it. Python import os from azure.ai.projects import AIProjectClient from azure.identity import DefaultAzureCredential with ( DefaultAzureCredential() as credential, AIProjectClient( endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"], credential=credential, ) as project, ): openai = project.get_openai_client() r = openai.responses.create( model="gpt-5-mini", input="Reply with the single word ready.", ) print(r.output_text) get_openai_client() returns an authenticated client from the openai package configured to run Responses operations against your Foundry project endpoint. That is the pattern to internalize. You use the project client for setup, configuration, agents, and evaluations, and the OpenAI-compatible client for the actual model calls. Turning an Hour of Audio Into Diarized Speaker Turns Fast transcription is the right tool for recorded meetings. It returns results synchronously and much faster than real time, which is exactly the tradeoff you want for a file that already exists. Batch transcription is the alternative, and it wins on very long archives and on advanced customization, but for a one-hour standard-format recording, fast transcription gets you a result in a small number of seconds with predictable latency. The endpoint is /speechtotext/transcriptions:transcribe and the current generally available API version is 2025-10-15. It takes multipart/form-data with the audio in one part and a JSON definition in another. Diarization is configured with a diarization object carrying maxSpeakers, and the service can separate up to 35 distinct speakers in a single channel before it errors out. Here is the worker in full, with the retry behavior that you will absolutely need. Python import json import os import time import requests from azure.identity import DefaultAzureCredential SPEECH_ENDPOINT = ( f"https://{os.environ['SPEECH_RESOURCE_NAME']}" ".cognitiveservices.azure.com/speechtotext/transcriptions:transcribe" "?api-version=2025-10-15" ) SCOPE = "https://cognitiveservices.azure.com/.default" RETRYABLE = {408, 429, 500, 502, 503, 504} def transcribe(audio_path, locales=("en-US",), max_speakers=8, max_attempts=5): """Fast transcription with diarization and bounded exponential backoff.""" credential = DefaultAzureCredential() definition = { "locales": list(locales), "diarization": {"enabled": True, "maxSpeakers": max_speakers}, "profanityFilterMode": "None", } for attempt in range(max_attempts): token = credential.get_token(SCOPE).token with open(audio_path, "rb") as fh: response = requests.post( SPEECH_ENDPOINT, headers={"Authorization": f"Bearer {token}"}, files={"audio": (os.path.basename(audio_path), fh)}, data={"definition": json.dumps(definition)}, timeout=600, ) if response.status_code == 200: return response.json() if response.status_code not in RETRYABLE: raise RuntimeError( f"Fast transcription failed {response.status_code} {response.text[:400]}" ) wait = float(response.headers.get("Retry-After", 2 ** attempt)) time.sleep(min(wait, 60)) raise RuntimeError(f"Giving up on {audio_path} after {max_attempts} attempts") A few things in there earn their place. The Retry-After header is honored when the service sends one, which matters a lot under throttling because blind exponential backoff on a shared Speech resource just means every worker retries in lockstep. Profanity filtering is set to None because the default is Masked and masked words in a transcript quietly damage retrieval, since the asterisks become tokens that match nothing. The 600-second timeout is generous on purpose, because a large file uploading over a constrained egress path can spend a long while before the service even starts work. The response contains a phrases array where each entry carries speaker, offsetMilliseconds, durationMilliseconds, and text. Phrases are the wrong chunk size for retrieval. They are usually a sentence or two, which means an embedding of a phrase carries almost no context, and a citation to a phrase drops the reader into the middle of a thought. Group them into speaker turns instead. Python from dataclasses import dataclass, asdict @dataclass class Turn: meeting_id: str meeting_title: str meeting_date: str speaker: str start_ms: int end_ms: int text: str @property def chunk_id(self): return f"{self.meeting_id}-{self.start_ms:09d}" def to_turns(result, meta, max_chars=2400, gap_ms=4000): """Collapse diarized phrases into speaker turns, splitting very long ones.""" turns, current = [], None for p in result.get("phrases", []): speaker = f"Speaker {p.get('speaker', 'unknown')}" start = p["offsetMilliseconds"] end = start + p["durationMilliseconds"] same_speaker = current and current.speaker == speaker contiguous = current and (start - current.end_ms) < gap_ms room = current and (len(current.text) + len(p["text"])) < max_chars if same_speaker and contiguous and room: current.text += " " + p["text"] current.end_ms = end continue if current: turns.append(current) current = Turn( meeting_id=meta["meeting_id"], meeting_title=meta["title"], meeting_date=meta["date"], speaker=speaker, start_ms=start, end_ms=end, text=p["text"], ) if current: turns.append(current) return turns The gap_ms guard is the part people leave out. Without it, a speaker who talks at minute three and again at minute forty gets merged into one chunk if nobody else spoke in between, which is rare but produces a chunk whose timestamp range is meaningless. Four seconds of silence is a reasonable turn boundary for meeting audio. Making Chunks That Are Worth Citing Retrieval quality on meeting transcripts lives or dies on what surrounds the raw text. A bare speaker turn like "yeah I think that's fine, let's go with option two" is nearly unretrievable, because it contains no nouns. The fix is to write a small amount of generated context into each record and let the hybrid search match on that. Python def contextualize(openai, turn, neighbors): """Prepend a one-line situating summary so short turns stay retrievable.""" window = "\n".join(f"{n.speaker}: {n.text}" for n in neighbors) r = openai.responses.create( model="gpt-4.1-mini", input=( "Write one sentence, under 25 words, situating the final utterance " "inside this meeting excerpt. Name the topic and any decision. " "Do not editorialize.\n\n" f"Meeting: {turn.meeting_title} ({turn.meeting_date})\n\n" f"{window}\n\nFinal utterance: {turn.speaker}: {turn.text}" ), ) return r.output_text.strip() def to_records(openai, turns): for i, turn in enumerate(turns): neighbors = turns[max(0, i - 3): i + 1] context = contextualize(openai, turn, neighbors) yield { **asdict(turn), "chunk_id": turn.chunk_id, "context": context, "content": f"{context}\n\n{turn.speaker}: {turn.text}", "timecode": f"{turn.start_ms // 60000:02d}:{(turn.start_ms // 1000) % 60:02d}", } This costs one small model call per turn, which, in a one-hour meeting, is a few hundred calls of a couple hundred tokens each. Run it concurrently with a semaphore rather than serially. The timecode field is what makes citations feel like a product feature rather than a footnote, because you can render it as a deep link into your video player. Write the records as JSONL to curated-transcripts, one file per meeting, and you are done with audio forever. Wiring the Transcripts Into a Foundry IQ Knowledge Base Foundry IQ is the knowledge and retrieval layer built on Azure AI Search. The mental model is two nested objects. A knowledge source points at searchable content, and a knowledge base wraps one or more knowledge sources behind a single endpoint that agents query. For indexed sources, Foundry IQ manages the whole indexing pipeline, so content gets ingested, chunked, vectorized, and prepared for hybrid retrieval without you standing up a skillset by hand. Agentic retrieval features are generally available in the 2026-04-01 REST API. The 2026-05-01-preview version exposes the fuller feature set, including preview knowledge source kinds and the ability to attach an LLM to non-web sources. Blob Storage is a generally available indexed source kind, which is exactly what we need. Point a knowledge source at the curated container. Python from azure.search.documents.indexes import SearchIndexClient from azure.search.documents.indexes.models import ( KnowledgeBase, KnowledgeSourceReference, AzureBlobKnowledgeSource, AzureBlobKnowledgeSourceParameters, ) from azure.identity import DefaultAzureCredential index_client = SearchIndexClient( endpoint=os.environ["SEARCH_ENDPOINT"], credential=DefaultAzureCredential(), ) source = AzureBlobKnowledgeSource( name="meeting-transcripts", description=( "Diarized speaker turns from recorded internal meetings, 2024 onward. " "Each chunk carries meeting title, date, speaker label, and timecode." ), azure_blob_parameters=AzureBlobKnowledgeSourceParameters( connection_string=os.environ["BLOB_CONNECTION"], container_name="curated-transcripts", embedding_model=..., # your deployed text embedding model chat_completion_model=..., # optional, enables verbalization ), ) index_client.create_or_update_knowledge_source(source) That description field is not decoration. When a knowledge base holds several sources, the retrieval engine plans which sources to query, and the description is the primary signal it uses to route. Write it like you are briefing a colleague who has never seen your data. Now the knowledge base. Python kb = KnowledgeBase( name="meetings-kb", knowledge_sources=[ KnowledgeSourceReference(name="meeting-transcripts", always_query_source=False), ], retrieval_instructions=( "Meeting transcripts. When the user asks who said or decided something, " "return the speaker turns that contain the statement plus the surrounding turns. " "Prefer recent meetings when the question is about current state." ), ) index_client.create_or_update_knowledge_base(kb) The retrieval engine plans which sources to query and performs iterative search when the first pass does not clear its relevance bar. Iterative search depends on setting a medium retrieval reasoning effort, either on the knowledge base or per request. That single knob is also the biggest lever on both latency and spend, so treat it as a tuning parameter rather than a set-and-forget value. Reasoning effortWhat the engine doesGood fit forMinimalSingle pass, extractive results, no query planningLookup-style questions where the user names the meetingLowLight query decomposition across sourcesMost interactive chat trafficMediumIterative search plus richer planning over sourcesAnalytical questions spanning many meetings Giving the Agent a Knowledge Base and a Personality With the knowledge base in place, the agent is short. Agent operations in the 2.x SDK are built on the Responses protocol, and agents are versioned objects created with create_version. Python from azure.ai.projects.models import PromptAgentDefinition INSTRUCTIONS = """You answer questions about internal meetings using only the meeting transcript knowledge base. Rules you follow without exception. 1. Every factual claim carries a citation naming the meeting title, date, and timecode. 2. When you cannot find support in the transcripts, say so plainly and stop. 3. Attribute statements to the speaker label exactly as it appears. Never guess a real name. 4. When speakers disagreed, surface the disagreement rather than flattening it into consensus. 5. Distinguish a decision from a suggestion. Quote the language that makes it one or the other. """ agent = project.agents.create_version( agent_name="meeting-analyst", definition=PromptAgentDefinition( model="gpt-5-mini", instructions=INSTRUCTIONS, tools=[{"type": "knowledge_base", "knowledge_base": {"name": "meetings-kb"}], ), ) print(agent.id, agent.version) Rule three is doing real work. Diarization gives you stable speaker identifiers within a recording, not identities, so you get generic labels rather than names. If the instructions do not forbid it, a capable model will cheerfully infer that Speaker 2 is the person whose name appears in the meeting title, and it will be wrong roughly as often as it is right. If you need real names, map them yourself in the chunking stage from calendar metadata or from multichannel capture, and write the resolved name into the record. Calling the agent looks like any Responses call. Python def ask(openai, agent_name, question, previous_response_id=None): return openai.responses.create( extra_body={"agent": {"name": agent_name, "type": "agent_reference"}, input=question, previous_response_id=previous_response_id, ) first = ask(openai, "meeting-analyst", "What did we decide about the pricing migration, and did anyone object?") print(first.output_text) follow_up = ask(openai, "meeting-analyst", "Which of those objections were ever resolved?", previous_response_id=first.id) print(follow_up.output_text) Threading through previous_response_id keeps the conversation server-side, which means you are not shipping a growing transcript of the chat on every turn and you are not writing your own history store. Failing Well When Retrieval or the Model Does Not Cooperate Two failure classes matter in production, and they want different handling. Transient service errors want retries. Empty or weak retrieval wants a different answer, not a retry, because running the same query again against the same index returns the same nothing. Python import random from openai import APIStatusError, APITimeoutError TRANSIENT = {408, 409, 429, 500, 502, 503, 504} def ask_resilient(openai, agent_name, question, attempts=4, **kwargs): last = None for i in range(attempts): try: return ask(openai, agent_name, question, **kwargs) except APITimeoutError as exc: last = exc except APIStatusError as exc: if exc.status_code not in TRANSIENT: raise retry_after = exc.response.headers.get("retry-after") last = exc if retry_after: time.sleep(min(float(retry_after), 30)) continue time.sleep(min(2 ** i + random.random(), 30)) raise last Full jitter on the backoff is not optional at any real concurrency. Without it, your retries synchronize into a thundering herd, and you turn a brief throttle into a sustained one. For the retrieval side, the answer is to make the agent's failure visible rather than silent. Instruction two above tells the model to say it found nothing, and you should assert on that in your evaluation set. A grounded system that admits ignorance is far more valuable than one that produces confident prose from three irrelevant chunks, and the second failure mode is much harder to notice in production because the output looks fine. Measuring Whether the Thing Actually Works Two separate quality questions live in this pipeline, and they need separate measurement. The transcription layer has an accuracy problem measured in word error rate. The retrieval and generation layer has a groundedness problem measured by a judge model. A regression in either one looks identical from the outside, which is a good argument for measuring them apart. Build a golden set first. A hundred or so questions written against meetings you have actually listened to is worth more than a thousand synthetic ones, because the value is in the expected answers and only a human who sat through the meeting can write those. Cover the awkward shapes deliberately. Include questions whose answer is genuinely absent so you can measure refusal behavior. Include questions that span two meetings. Include questions where two people disagreed. JSON {"question": "Who owned the migration rollback plan after the March review?", "expected": "Speaker 3 accepted ownership at 41:12 in Platform Review 2026-03-04.", "must_cite": "Platform Review 2026-03-04", "kind": "attribution"} {"question": "What was the agreed SLA for the batch job?", "expected": "Not discussed in any recorded meeting.", "must_cite": null, "kind": "refusal"} The evaluation operations live on the project client in the 2.x SDK, under properties such as evaluators, evaluation_rules, and schedules. For groundedness and relevance, you use built-in judge evaluators. For word error rate, you register a custom evaluator, because that one is arithmetic rather than judgment. Python import jiwer def transcript_wer(reference_text, hypothesis_text): transform = jiwer.Compose([ jiwer.ToLowerCase(), jiwer.RemovePunctuation(), jiwer.RemoveMultipleSpaces(), jiwer.Strip(), jiwer.ReduceToListOfListOfWords(), ]) return jiwer.wer(reference_text, hypothesis_text, truth_transform=transform, hypothesis_transform=transform) Hand-correct twenty minutes of audio across three or four recordings and keep it as your reference. Twenty minutes sounds thin, and it is, but it catches the failures that matter, which are domain vocabulary and acronyms coming back as phonetic mush. If your WER on product names is bad, the fix is a phrase list rather than a better model. Phrase lists let you hand the recognizer a set of words likely to appear, and they move the needle hard on proper nouns and internal jargon. The metrics worth gating a deploy on are these four. MetricWhat it catchesWhere it comes fromWord error rate on domain termsVocabulary drift, new product names, bad audioCustom evaluator against hand-corrected referenceGroundednessAnswers not supported by retrieved chunksBuilt-in judge evaluatorCitation validityFabricated meeting titles, timecodes outside the recordingDeterministic check against chunk metadataRefusal rate on absent answersConfident invention when nothing was retrievedGolden set questions with no supporting content Citation validity is the cheap one everyone skips. You already have the chunk metadata, so parsing the citations out of the answer and asserting that each meeting title exists and each timecode falls inside that recording's duration is maybe thirty lines of code. It catches a specific and embarrassing failure that judge models are surprisingly forgiving of. Getting This to Production Without Regrets Reindex on a schedule and expect churn. Foundry IQ triggers indexing and data synchronization automatically for indexed sources, but your curated container is the contract. If you change chunking strategy, you are rewriting every record, and a full reindex of a large corpus is not instant. Version your chunking logic and write the version into each record so you can tell mixed-generation content apart during a migration. Decide the permission model before you index anything. Meeting recordings are among the most sensitive content an organization has. Retrieval in Foundry IQ respects user permissions for supported knowledge source types, and for the remote SharePoint source, Purview sensitivity labels and data classifications flow through the indexing and retrieval pipeline. Blob-backed sources do not give you that for free. If access control per meeting matters, either enforce it with security filters at query time using a field on each chunk, or keep recordings in SharePoint and use the remote source, where content never leaves SharePoint, and SharePoint enforces permissions. Retrofitting this later means reindexing everything and auditing every conversation that already happened. Instrument with tracing from day one. The projects SDK ships GenAI tracing instrumentation, currently an experimental preview where spans and attributes may change between versions. Turn it on anyway. When a user says the agent gave a bad answer, you want the retrieved chunk IDs and the query plan from that exact response, and reconstructing them after the fact from logs you did not write is miserable. Watch the two meters. Retrieval bills token usage for subquery execution and semantic reranking, and the model you attach for query planning and answer synthesis bills separately on the model side. Reasoning effort, source count, and how much content you route into synthesis are the levers, in that order. Plan the migration if you are on the old pattern. If you are still using Azure OpenAI On Your Data, the "Add your data" flow in the classic chat playground, it is deprecated and retires on October 14, 2026. The official migration target is exactly the stack in this article, which is Foundry Agent Service plus Foundry IQ. How This Compares to Rolling the Pipeline Yourself The obvious alternative is a hand-built stack. Whisper for transcription behind your own GPU or an inference endpoint, pyannote for diarization, your own chunker, a vector database, and LangChain or a custom orchestrator on top. That stack is genuinely good, and it is genuinely more work. The honest comparison looks like this. ConcernFoundry with fast transcription and Foundry IQSelf-hosted Whisper plus pyannote plus a vector DBAmazon Transcribe plus Bedrock Knowledge BasesGoogle Speech-to-Text plus Vertex AI SearchDiarizationBuilt into the same call, up to 35 speakersSeparate model, separate tuning, best-in-class quality achievableBuilt into the transcription jobBuilt into the recognizerTime to first working answerHoursDays to weeksHoursHoursRetrieval planningAgentic, multi-query, iterative at higher effortWhatever you writeManaged retrieval, less query planningManaged retrieval with good semantic rankingPermission-aware retrievalNative for supported sources, Purview labels honored for remote SharePointYou build itIAM-scoped, coarser at the chunk levelIAM-scopedWhere the audio goesYour Azure regionWherever you run it, including fully on-premisesYour AWS regionYour GCP regionEscape hatchKnowledge bases callable from any app through the Search APIsTotal controlBedrock APIsVertex APIs The self-hosted path wins on two things, and they are not small. One is cost at very high volume, because at some point per-minute transcription pricing loses to a GPU you already own. The other is data residency in the strict sense, meaning audio that legally cannot leave your premises. If neither applies to you, the managed path buys back weeks of work you would otherwise spend on chunking heuristics and retry logic. Within Azure, there is also a smaller decision, which is fast transcription against batch transcription. Fast wins on latency and simplicity for files under the size limit. Batch wins when you need to process very large archives asynchronously, when you want webhook notifications on completion, or when you want to bring your own storage account for the outputs. Where to Take It Next The pipeline above is the spine. The interesting extensions hang off the chunking stage, because that is where you decide what the retrieval layer is even capable of answering. Extracting action items into a structured field lets you answer "what did I commit to last month" without any retrieval creativity. Writing a sentiment or disagreement flag onto each turn lets the agent find contested moments directly. Adding a second knowledge source pointed at your specs and design docs turns "what did we decide" into "what did we decide and does the shipped code match", and because a knowledge base fronts multiple sources behind one endpoint, that is a configuration change rather than an architecture change. The part worth protecting as you extend is the evaluation loop. Meeting corpora grow continuously and unevenly, and a retrieval strategy tuned on six months of transcripts behaves differently on three years. The golden set is what tells you when that has happened. References Use the fast transcription APISpeech-to-text REST API referenceWhat is Foundry IQCreate a knowledge base in Azure AI SearchConnect agents to Foundry IQ knowledge basesQuickstart: Get started with the Microsoft Foundry SDKAzure AI Projects client library for Python

By Jubin Soni, FBCS

CORE

How to Secure Fintech REST APIs Against BOLA Vulnerabilities

Broken Object Level Authorization (BOLA) occurs when a REST API exposes an object identifier—such as an account, transaction, or loan ID — without verifying whether the authenticated user is authorized to access that specific resource. To protect fintech REST APIs, implement server-side authorization checks for every object request, validate permissions using the user's authenticated context and resource ownership, and avoid relying on client-supplied IDs alone. Using unpredictable identifiers such as UUID v4 or ULIDs can reduce object enumeration, but they should be treated as an additional security layer—not a replacement for authorization. Expert Insight: In retail banking systems, BOLA can expose sensitive customer and financial data when attackers manipulate object IDs, such as changing /api/v1/accounts/1001 to /api/v1/accounts/1002. Randomized identifiers make enumeration harder, but the core defense is object-level authorization on every API request. Policy-based controls, including tools such as Open Policy Agent (OPA), can help enforce consistent ownership and access rules across services. 1. Understanding BOLA in Fintech Ecosystems Fintech APIs handle highly sensitive operations, including transaction retrieval, account information, payment processing, and ledger-related activities. Broken object-level authorization (BOLA) occurs when an API uses a client-supplied object identifier to retrieve or modify a database record without verifying that the authenticated user has permission to access that specific resource. Authentication vs. authorization: Authentication confirms who the user is, such as validating a JWT. Authorization determines what that authenticated user is permitted to access or modify. A valid login does not automatically grant access to every financial object.The scale of risk: Modern open-banking ecosystems connect banks, fintech platforms, payment providers, and third-party applications. A missing object-level authorization check can therefore expose sensitive account, transaction, or payment data beyond the intended user or organization. Example: If a customer can access /api/v1/accounts/1001 and simply change the ID to /api/v1/accounts/1002 to retrieve another customer's account, the endpoint has a potential BOLA vulnerability. 2. The Anatomy of a Banking BOLA Attack Consider a poorly secured REST API endpoint used to fetch a customer's monthly credit card statement: HTTP GET /api/v1/statements?account_id=89234 An attacker first logs in with their own valid account and accesses their statement using account_id=89234. They then use an interception proxy such as Burp Suite to change the account ID in the outgoing HTTPS request: HTTP GET /api/v1/statements?account_id=89235 If the backend directly uses 89235 to query the database without checking whether this account belongs to the authenticated user, the API may return the victim's private banking information. This is a classic BOLA vulnerability. The main issue is that the API checks whether the user is logged in, but fails to check whether that user is actually allowed to access the requested account. 3. Top 5 Architectural Practices to Mitigate BOLA a. Avoid Sequential Integer IDs in Public APIs Avoid exposing simple auto-increment database IDs such as 1, 2, or 3 through public API endpoints. Use unpredictable identifiers such as UUIDv4 or ULID (Universally Unique Lexicographically Sortable Identifier) instead. This makes automated ID guessing and enumeration much harder. However, random identifiers should be treated as an extra security layer, not as a replacement for proper authorization checks. b. Do Not Depend on Client-Supplied Parameters for Authorization The client should never decide the access boundary simply by sending an account or resource ID in the URL. Instead, the backend should get the authenticated user's identity from a securely verified session or validated JWT claims and then check whether that user has permission to access the requested resource. c. Use Fine-Grained Access Control (FGAC) Use authorization models such as attribute-based access control (ABAC) or relationship-based access control (ReBAC) when the application needs more detailed permission rules. For example, the system can maintain clear relationships between users, accounts, transactions, loans, and other financial resources. The API can then check whether the requested object is actually linked to the current user's permitted scope. d. Centralize Common API Security Policies In a microservices environment, repeating authorization logic separately in every service can create gaps and inconsistent rules. API gateways such as Kong, Apigee, or AWS API Gateway can help enforce common authentication, token validation, routing, and security policies at the edge. However, sensitive object-level authorization should still be enforced by the service that owns the resource. e. Shift Security Testing Left Include API authorization testing throughout the CI/CD pipeline instead of waiting until production. Automated security tests can change resource identifiers, use different user identities, and verify that unauthorized requests are rejected. For example, a test can confirm that User A cannot access User B's account and that the API returns an appropriate 403 Forbidden or 404 Not Found response according to the application's security design. Securing fintech APIs (such as AutoPay By NPCI) against BOLA is critical for safeguarding sensitive user data [OWASP]. Teams can utilize architecture resources and deployment calculators to audit system compliance costs, optimize processing infrastructure, and seamlessly bridge secure development workflows with enterprise-grade financial technology standards. 4. Implementing Contextual Code-Level Checks At the code level, a secure Java/Spring Boot controller should perform an object-level authorization check before passing the request to the service or repository layer. Java @GetMapping("/api/v1/accounts/{accountId}") public ResponseEntity<AccountDetails> getAccount(@PathVariable String accountId, @AuthenticationPrincipal JwtPrincipal principal) { // Check if the authenticated user UUID matches the requested resource ownership if (!authorizationService.isOwner(principal.getUserId(), accountId)) { throw new AccessDeniedException("Unauthorized resource access attempt."); } return ResponseEntity.ok(accountService.findById(accountId)); } 5. The Verdict: How to Audit Your System Step 1: Review all public REST API endpoints that accept user IDs, account IDs, transaction IDs, or other object identifiers through URL paths, query parameters, or JSON request bodies.Step 2: Make sure your QA and security tests include cross-user and cross-tenant access checks. For example, authenticate as User A and try to access User B's statement. The request should be rejected.Step 3: Use centralized authorization controls, middleware, or framework-level security components to apply identity and permission checks consistently across API endpoints. This helps reduce the chance of one controller accidentally missing an important authorization check. A proper BOLA audit should verify not only whether users are authenticated, but also whether they can access only the financial objects they are actually authorized to use.

By Nanne Parmar

From Bottlenecks to Reliability: A Practical Guide to Scaling Temporal in Production

Temporal is designed to preserve Workflow state through process crashes and infrastructure failures, but durable state does not remove ordinary capacity limits. In production, the control plane can remain healthy while throughput collapses because Worker slots are saturated, Task Queues mix incompatible workloads, or a failover activates a region without enough Worker capacity. Temporal Workers run outside the Temporal Service and execute Workflow and Activity code, so production scalability depends as much on Worker and routing design as on the service itself. The Worker Fleet Is Usually the First Capacity Boundary Schedule-to-Start latency is best treated as queueing delay rather than application execution time. It measures the interval between a Task being enqueued and a Worker starting it. Rising Schedule-to-Start latency, growing approximate backlog, and exhausted Worker task slots indicate that Tasks are arriving faster than the fleet can consume them. Temporal Cloud exposes temporal_cloud_v1_approximate_backlog_count, while SDK metrics expose Workflow and Activity Schedule-to-Start latency and available task slots. Temporal guidance recommends watching these signals together because backlog depth alone does not identify whether the limit is Worker count, Worker configuration, or polling behavior. Worker scaling has two layers. Horizontal scaling adds Worker processes, while concurrency tuning changes how many Tasks each process can execute simultaneously. For well-benchmarked workloads, fixed slot limits place a predictable ceiling on local resource consumption. The Java SDK exposes separate concurrency controls for Workflow Tasks and Activities, and a server-side Activity rate limit can cap dispatch across all Workers polling the same Task Queue. Java WorkerOptions options = WorkerOptions.newBuilder() .setMaxConcurrentWorkflowTaskExecutionSize(120) .setMaxConcurrentActivityExecutionSize(80) .setMaxTaskQueueActivitiesPerSecond(250) .build(); The values in this example are capacity-test outputs, not universal defaults. A CPU-heavy Activity fleet may need a lower Activity slot count than an I/O-heavy fleet. Newer Worker tuners can allocate slots dynamically from CPU and memory signals, while fixed-size suppliers remain more predictable when task resource cost is well understood. Temporal also recommends poller autoscaling for most workloads because too few pollers constrain ingestion and too many waste connections and reduce efficiency. Task Queue Topology Determines Isolation and Backpressure Adding replicas cannot repair a Task Queue topology that couples unrelated bottlenecks. A shared Task Queue is reasonable when Workflows and Activities have similar latency and resource characteristics, but it becomes risky when fast orchestration work shares capacity with slow database calls, GPU jobs, tenant bursts, or Activities constrained by a downstream API. Temporal supports specialized routing through separate Task Queues, and Activity-level server-side throttling applies to the entire queue. A throttled Activity therefore should not share a queue with work that must remain unrestricted. A Workflow can route a costly Activity to a dedicated fleet without changing the Workflow’s own Task Queue. The separation creates an independent scaling and backpressure boundary. Java ActivityOptions options = ActivityOptions.newBuilder() .setTaskQueue("payments-io") .setStartToCloseTimeout(Duration.ofSeconds(20)) .build(); PaymentActivities payments = Workflow.newActivityStub(PaymentActivities.class, options); With payments-io isolated, replicas, concurrency, credentials, network placement, and queue-wide rate limits can be tuned for payment traffic without changing the Worker pool that advances Workflow Tasks. The same principle applies to multi-tenant systems. Temporal documents per-tenant Task Queues as a strong isolation pattern and also supports fairness keys when many tenants share one queue. Priority and fairness operate within Task Queue partitions, so they manage contention inside a queue rather than replacing isolation when resource requirements differ fundamentally. Task Queue partitioning should also be distinguished from application-level queue proliferation. Temporal Task Queues are lightweight and scale internally through partitions; current documentation states that Task Queues use four partitions by default. Multiple partitions increase throughput but relax strict FIFO behavior because Tasks are distributed among partitions. Separate named queues should therefore be created for routing, isolation, or rate-control reasons, not merely to manufacture throughput that Temporal’s matching layer can already scale internally. Autoscaling Should Follow Queue Pressure, Not CPU Alone CPU-based autoscaling is insufficient for many Temporal workloads. An I/O-bound Activity can leave CPU utilization low while all Activity slots are occupied and backlog grows. Conversely, high CPU with near-zero Schedule-to-Start latency may mean that the fleet is efficiently utilized. A stronger autoscaling policy combines queue delay, backlog trend, slot availability, and host resource saturation. Temporal’s Worker health guidance treats Schedule-to-Start latency as a primary symptom of insufficient processing capacity and recommends correlating it with sync-match behavior and available slots before changing fleet size. On Kubernetes, Temporal’s Worker Controller can attach HPA or KEDA resources to versioned Worker deployments and scale from CPU, memory, Task Queue backlog, slot utilization, or custom metrics. Current guidance recommends HPA with a Prometheus adapter as the general default, while KEDA is positioned for scale-to-zero, long idle periods, or faster event-driven reactions. This matters because old and new Worker versions can coexist during safe rollout, so autoscaling should follow each active Worker Deployment Version rather than treating the fleet as a single anonymous pool. Scale-down deserves the same attention as scale-up. Backlog can reach zero while Activities are still running, and terminating aggressively can create retries or latency spikes. Worker shutdown should therefore be graceful, minimum replica counts should reflect availability requirements, and cooldowns should account for Activity duration and startup time. Pre-production tests should include Worker termination, burst recovery, and partial failure because Temporal durability preserves state but does not guarantee that an undersized replacement fleet will meet latency objectives. Regional Failover Has to Include Workers and Dependencies Regional failover is often mis-scoped as a Temporal Service feature. Temporal Cloud High Availability replicates a Namespace to a secondary region and can automatically promote the replica during an outage, but application Workers remain separately operated compute. Temporal documents a 20-minute RTO and sub-one-minute RPO for its HA service, yet application recovery can still be slower when the secondary region lacks ready Worker capacity, network access to the active Namespace, or available downstream systems. For latency-sensitive systems, Active/Hot-Passive is the most deterministic failover model: a full Worker fleet runs in both regions, the secondary fleet stays warm, and only the fleet local to the active replica processes Tasks. On failover, the warm fleet begins processing without a Worker cold start. Active/Passive costs less but requires starting or scaling Workers after failover, while Active/Active runs Workers in multiple regions even though the HA Namespace still has one active replica underneath. Connectivity must be tested as part of the failover path. For HA Namespaces, the Namespace Endpoint follows the active region through DNS; Temporal documents a 15-second TTL and roughly 30 seconds for clients to converge when resolvers honor that TTL. Private connectivity requires routes and DNS design that allow Workers to reach the promoted region. A test that switches only the Namespace but omits Worker connectivity, database promotion, queue access, secrets, codec servers, or proxies validates only part of the production path. Self-hosted multi-cluster deployments require explicit planning as well. Temporal’s Global Namespace model uses asynchronous cross-cluster replication and eventual conflict resolution, and successful failover requires Worker Processes to poll the Namespace in clusters that may become active. Replication versions determine which cluster can mutate Workflow history after failover, but they do not provision Worker compute or external dependencies. Conclusion Temporal becomes a production bottleneck when durable orchestration is treated as a substitute for capacity engineering. Stable performance comes from measuring queue delay and slot saturation, scaling Worker fleets from demand signals rather than CPU alone, separating Task Queues where workloads need independent isolation or rate control, and designing regional failover around ready Workers and reachable dependencies. With those boundaries in place, Temporal remains the durable coordination layer rather than the slowest component in the execution path.

By Akhil Madineni

CORE

Open Source as a Leadership Lab for Software Engineers

Leadership is challenging to develop in isolation. While you can practice programming, architecture, or databases independently, leadership relies on skills such as communication, influence, negotiation, feedback, conflict resolution, and decision-making, all of which require interaction with others. As leadership becomes more important for software engineers advancing in their careers, a key question arises: where can engineers practice these skills before becoming managers? Open source offers an ideal environment to develop both technical and leadership skills. Engineers tackle real technical challenges — such as coding, API design, architecture, testing, and documentation—while collaborating with individuals from diverse backgrounds, priorities, and perspectives. Although contributions often start with a pull request, advancing in the community requires explaining ideas, accepting feedback, building consensus, mentoring, and influencing technical direction. Open source is therefore more than a platform for technical growth; it serves as a practical setting for developing technical leadership. 1. Open Source as a Hard-Skill Accelerator For many software engineers, developing hard skills is a natural starting point. We are often eager to learn new languages, understand frameworks, enhance design skills, or explore different architectures. Open source offers a rich environment for this growth by exposing you to real software, real constraints, and ongoing evolution. Rather than working on isolated exercises, you can study and contribute to systems that have endured years or even decades of change. A key lesson is learning to manage software over the long term. Projects like Java, which have evolved for decades, reflect decisions about backward compatibility, modernization, deprecation, migration, performance, security, and ecosystem stability. This contrasts with greenfield applications, where ideas can be replaced freely. Mature open-source projects show that good engineering often means safely evolving an imperfect but widely used system, rather than aiming for perfect design. Open source provides practical experience with legacy modernization. You can observe how maintainers introduce new APIs without disrupting existing users, gradually remove obsolete abstractions, use tests to protect behavior during refactoring, and break down architectural changes into manageable steps. These challenges are common in enterprise environments but are difficult to replicate in personal projects. Another important area is documentation. In open source, documentation is not secondary to the code. API documentation, design discussions, migration guides, issue descriptions, proposals, release notes, and contribution guidelines are part of the engineering work itself. Writing clearly forces you to explain not only what the code does, but also why a decision exists and what trade-offs were considered. That ability becomes increasingly important as you move toward Staff Engineer or Architect responsibilities. Open source also offers opportunities to improve your coding and software design skills. You can study code written by engineers from diverse companies, countries, and technical backgrounds. This exposure is valuable because there is no single universal style of good software design. Projects optimize for different constraints, such as performance, compatibility, simplicity, extensibility, security, developer experience, or operational stability. Comparing these decisions helps you develop sound judgment rather than simply memorizing patterns. This is especially relevant in software architecture, where decisions are rarely clear-cut. Most architectural choices are shaped by context, constraints, history, and trade-offs. Open source allows you to observe these decisions openly, including API discussions, rejected proposals, compatibility concerns, implementation limitations, and competing approaches. You can see both the final architecture and the reasoning behind it. Open source offers a unique learning advantage: you can learn directly from the creators of the technologies you use. Instead of relying solely on tutorials or books, you can read their code, follow design discussions, review pull requests, and sometimes ask questions directly. Over time, you may even become one of the contributors shaping the project. Finally, understanding the internals of a framework, library, language, or specification can set you apart. Many engineers know how to use a technology, but few understand why it behaves as it does, its limitations, or its internal workings. Open source provides access to this deeper knowledge. For experienced software engineers, this understanding can make a significant difference when debugging complex issues, evaluating trade-offs, or making architectural decisions. 2. Open Source as a Soft-Skill Laboratory Many software engineers focused on technical expertise may overlook soft skills, assuming communication, persuasion, networking, and public speaking are primarily for managers. However, advancing in a technical career requires these abilities. Software is built collaboratively, key decisions are made through discussion, and achieving greater impact depends on others understanding, trusting, and supporting your ideas. Open source offers a practical environment to develop these skills, as the outcomes are tangible. You propose changes, defend technical decisions, receive feedback, collaborate with unfamiliar colleagues, and work to make your ideas clear and accepted by others. Learn to Communicate Through Writing A significant amount of software engineering leadership happens in writing. Issues, pull requests, design proposals, mailing lists, documentation, specifications, and code reviews all require you to organize your thoughts before requesting action. Open source provides frequent opportunities to practice this skill. This skill extends beyond open source. For example, the value of an Architecture Decision Record relies on your ability to describe context, explain alternatives, clarify trade-offs, and ensure the decision is understandable to future readers. Writing is not just documentation; it transforms technical reasoning into content that can be shared, challenged, and reused. Learn to Explain and Sell Technical Ideas Technical leadership also requires speaking. You may need to defend architectural decisions, explain preferred designs, challenge existing approaches, or persuade multiple teams to adopt new directions. Having an idea is only the first step; you must also make it understandable to those without your context. Open-source communities offer many opportunities to practice this: community calls, meetups, user groups, podcasts, workshops, and conferences. Preparing a presentation requires you to organize complex information, remove unnecessary details, build a clear narrative, and explain your reasoning so others can follow. That ability is crucial for any senior software engineer. Communicate Across Languages and Cultures Open source is global. If English is not your first language, as it is not mine, participating in international communities offers ongoing opportunities to improve. You regularly write issues, join discussions, review proposals, attend meetings, and present ideas in English. But the learning goes beyond vocabulary or grammar. You also learn how people from different cultures communicate, disagree, provide feedback, and make decisions. What seems normal in one culture may appear aggressive or ambiguous in another. For engineers in global organizations, effective cross-cultural communication can be as important as learning a new technical framework. Build Relationships and Reputation Open source can expand your network organically. You do not meet people simply to “network.” Instead, others recognize you through your consistent work, contributions, reviews, and participation in discussions. Over time, people learn your expertise and know what they can rely on you for. This is valuable because reputation extends beyond organizational boundaries. By sharing knowledge through technical decisions, pull requests, articles, documentation, or presentations, you can help people outside your company see your approach. External credibility can also strengthen your reputation within your organization. However, building reputation is a long-term investment. A few pull requests or a single conference talk will not transform your career. Reputation develops over months and years through consistent contributions. Learn to Manage Your Time and Context Open source also helps develop an underrated leadership skill: managing your attention. Most engineers contribute to open source while managing full-time jobs and other responsibilities. This requires deciding what deserves your time, breaking large initiatives into smaller tasks, prioritizing contributions, and switching contexts efficiently. These skills become increasingly important as your career progresses. Staff Engineers or Architects rarely focus on a single task. They often move between architecture discussions, code reviews, mentoring, incidents, multiple teams, and long-term initiatives within the same week. Maintaining focus while working across multiple contexts becomes essential. Build Discipline Through Consistency Open source also fosters discipline. While large contributions are visible, sustainable open-source involvement is built through smaller actions such as reviewing issues, improving documentation, answering questions, writing tests, fixing bugs, or joining design discussions. Success rarely comes from a single heroic contribution. It is consistency. Consistently doing small, meaningful work leads to long-term growth. You gain a deeper understanding of the project, earn recognition, take on more responsibility, and may eventually help shape the technology’s direction. The same principle applies to leadership. Leadership develops through repeated opportunities to communicate, influence, help others, make decisions, and earn trust, not simply by receiving a title. Open source simply gives you many more opportunities to practice. Conclusion A strong software engineer must develop both technical expertise and leadership skills to become a well-rounded professional. Excelling at coding, system design, or architecture is not enough if you cannot navigate challenging discussions, communicate with stakeholders, build trust, and clearly explain your ideas. Good technical ideas often fail when they are not understood, trusted, or convincingly presented. The reverse is equally risky. Strong communication and influence, without sufficient technical foundation, can lead teams astray. Leadership without technical judgment may result in persuasive presentations built on weak decisions. Conversely, technical depth without leadership can keep valuable ideas from being realized. High-impact engineering demands both skill sets. This balance is essential for those pursuing roles such as Software Architect, Staff Engineer, Principal Engineer, or technology executive. Complete knowledge is not expected. The key skill is the ability to shift between strategic discussions with C-level leaders and technical conversations with engineers to understand implementation details and design trade-offs. Open source offers valuable opportunities to develop both technical and leadership abilities, helping engineers grow as technologists and leaders.

By Otavio Santana

CORE