DZone Spotlight

AI agents become useful when they can do more than generate text. The moment an agent can update a CRM, approve a refund, create a purchase order, change a price, or send a customer response, the architecture must answer a harder question: Which actions should the agent execute automatically, and which should require human approval? That decision sits at the center of production-ready enterprise AI agent architecture. Too little oversight creates operational and compliance risk. Too much oversight turns the system into another approval queue. A well-designed human-in-the-loop system does not place a person behind every action. It uses risk-based approval gates, role-based permissions, auditability, and reversible execution to give AI agents useful autonomy without giving them uncontrolled authority. Full Autonomy Should Not Be the Default Many AI workflow automation projects begin with a simple assumption: if the agent can complete the task, it should be allowed to execute it. That assumption works poorly in enterprise environments. An agent may correctly understand a request but still act on incomplete data, use outdated policy, select the wrong customer record, or apply a technically valid action in the wrong business context. The risk is not limited to hallucination. Production systems also fail because of: Incorrect source dataAmbiguous instructionsPermission errorsDuplicate eventsStale workflow stateIntegration timeoutsDownstream system failures The right goal is therefore not maximum autonomy. It is bounded autonomy: the agent can act independently within predefined limits and escalate when those limits are crossed. Classify Actions by Risk Before designing an AI agent approval workflow, classify the actions the agent may perform. A practical model uses three levels. Low-Risk Actions These are easy to verify and easy to reverse. Examples include: Drafting an emailSummarizing a support ticketCategorizing a documentPreparing a CRM updateGenerating a reportSuggesting the next workflow step These actions can often run automatically, especially when the output remains internal or requires a later human action. Medium-Risk Actions These affect business records or external communication but remain recoverable. Examples include: Updating a CRM fieldScheduling a meetingSending a standard follow-upCreating a draft invoiceAssigning a support ticketUpdating an order status These actions may be automated when confidence is high, and policy conditions are satisfied. Otherwise, they should enter a review queue. High-Risk Actions These create financial, legal, compliance, security, or customer-impacting consequences. Examples include: Issuing a refundApproving a paymentChanging contract termsModifying production accessDeleting recordsChanging pricingSending regulated communications These should require explicit approval unless the organization has defined narrow, well-tested exceptions. The important point is that risk should be assigned to the action, not the model. A highly capable model should not automatically receive broader permissions. Put Approval Gates Before Irreversible Actions An approval gate should sit immediately before the step that creates external or irreversible impact. A common mistake is placing review too early. For example, asking a human to approve the agent’s plan before it has gathered data, validated records, or prepared the final action creates unnecessary work. A better sequence is: Receive the request.Gather relevant data.Validate identity, permissions, and workflow state.Generate the proposed action.Evaluate policy and risk.Request approval when required.Execute.Verify the result.Write to the audit log. This allows the agent to complete the preparation work while reserving human attention for the final decision. The approval screen should show more than a yes-or-no prompt. It should include: The proposed actionThe reason for the actionThe source data usedThe expected impactThe agent’s confidenceRelevant policy checksAvailable alternatives A reviewer should not need to reconstruct the agent’s reasoning from several systems. Use Policy-Based Approval, Not Confidence Alone Confidence scores can be useful, but they should not control approval decisions by themselves. A more reliable approval policy combines several signals: Action typeTransaction valueCustomer or account sensitivityConfidence thresholdData completenessPolicy exceptionsUnusual activityModel or tool failure history For example: Python def requires_approval(action): if action.type in HIGH_RISK_ACTIONS: return True if action.amount > action.auto_approval_limit: return True if action.confidence < 0.90: return True if not action.policy_checks_passed: return True if action.has_unusual_context: return True return False This is intentionally simple. In a production system, the policy engine should remain separate from the language model so that approval rules are deterministic, testable, and version-controlled. The model may recommend an action. The policy layer decides whether the system is allowed to perform it. Apply Role-Based Access Control An AI agent should not have one universal identity with access to every system. Secure AI workflow automation requires least-privilege access. Each agent or workflow should receive only the permissions required for its task. A finance agent may be allowed to prepare invoices but not release payments. A support agent may update ticket status but not alter customer contracts. A procurement agent may create a purchase request but not approve it. Human reviewers also need role-based permissions. An approval is meaningful only when the reviewer has authority over the action. Every approval event should record: Who approved or rejected itThe role usedThe action reviewedThe original proposalAny modificationsThe execution resultThe timestampThe policy version This creates AI agent audit logs that are useful for debugging, compliance reviews, and process improvement. Make Actions Reversible Approval gates reduce risk, but they do not eliminate errors. Where possible, design agent actions as reversible operations. Instead of immediately deleting a record, move it into a recoverable state. Instead of overwriting a value, preserve the previous version. Instead of sending a message without review, allow a delay window for cancellation. Useful patterns include: Soft deletionVersioned recordsCompensating transactionsDelayed executionIdempotency keysStaged updatesRollback workflows Reversibility is one of the most practical AI agent guardrails because it limits the damage from both model errors and system failures. Avoid Creating an Approval Bottleneck A badly designed human-in-the-loop system can be safe but unusable. If every action requires approval, reviewers become overloaded, response times increase, and users begin approving requests without proper inspection. The system should learn operationally, even if the model itself is not retrained. Track: Approval rate by action typeRejection reasonsAverage review timeCommon reviewer editsRepeated low-risk approvalsFalse escalationsIncidents after automatic execution If a category of actions is repeatedly approved without modification, it may be suitable for controlled automation. If a supposedly low-risk action is frequently corrected, its approval policy should become stricter. The goal is to move from broad manual oversight to targeted oversight based on evidence. A Practical Reference Architecture A production-ready design usually includes these components: Agent runtime: Interprets the request and prepares the actionTool layer: Connects the agent to enterprise systemsPolicy engine: Evaluates permissions, risk, and approval rulesApproval service: Presents the proposed action to an authorized reviewerExecution service: Performs approved actions using controlled credentialsAudit store: Records proposals, approvals, tool calls, and resultsMonitoring layer: Detects failures, unusual activity, and policy violations Separating these responsibilities prevents the language model from becoming the policy engine, identity provider, executor, and audit system at the same time. Final Takeaway Human-in-the-loop AI agents should not be designed as autonomous systems with an approval button added later. Approval, permissions, auditability, and reversibility must be part of the architecture from the beginning. The strongest enterprise systems do not ask humans to supervise every step. They automate low-risk work, escalate uncertain or sensitive actions, and preserve clear accountability for every decision. That is what makes an AI agent operationally useful: not unlimited autonomy, but the ability to act safely within well-defined boundaries.

A detector I built was scoring 0.067 recall on temporal errors, meaning it caught about one in fifteen of the wrong dates it was supposed to find. Wrong dates are supposed to be the easy category: extract the years from the claim, extract the years from the source, compare. There is no semantics to get wrong. I assumed the extraction was broken and went looking for the bug. The extraction was fine. The benchmark was the problem, and not in a way that showed up anywhere in the code. The contexts had been written in the wrong voice. That's the part worth passing on, and it has nothing to do with hallucination detection. It applies to anyone who builds a synthetic evaluation set, which by now is most of us. A Detector That Failed Because of Prose Style The setup: a claim, a source context, and a question about whether the claim is supported. The detector is part of HallucinoType, an open-source package I maintain, and its benchmark was built the way most synthetic evaluation sets are built. Take a faithful claim, inject an error of a known type, keep the label. Two hundred fifty pairs, stratified across eight failure categories, thirty-five of them faithful so I could measure false positives. When I wrote the contexts for the date items, I wrote them the way a person naturally writes when they know the claim is wrong. Something like the treaty was signed in 1928, not 1938. Read that as a human, and it is unambiguous. Read it as a program that treats the context as a reference document, and the string 1938 is sitting right there in the source. The detector extracted it, matched it against the claim, found agreement, and passed the item. Every one of the thirty temporal items had this property. Seven of thirty numerical items did too. Rewriting the contexts as ordinary reference prose, the kind a retrieval system would hand you, moved temporal recall from 0.067 to 1.000. I changed no code. The numerical items did not move. They stayed at 0.600, which told me their misses had a different cause and saved me from congratulating myself on a fix that only worked once. The generalization is short enough to put on a sticky note. A context that argues with the claim is not the context a production pipeline supplies. I had unconsciously written my source documents in a fact-checking register, because I was thinking like an annotator rather than like a retrieval index. The register leaked the answer key into the input, and the system read it, exactly as it was built to. What makes this uncomfortable is that nothing about the benchmark looked wrong. The labels were correct, and the errors were real errors; any reviewer would have signed off on it. The defect lived in a stylistic property of the prose that no one thinks to specify, and it moved a headline number by a factor of fifteen. The Same Bug Wearing a Different Hat The same mistake showed up a second time, and I did not recognize it at first. A second detector in the same system checks whether a claim names the wrong person, company, or place. It works by extracting entities from the claim and looking for them in the context. If the entity appears in the context, the detector skips the claim, on the theory that the source confirms it. I upgraded the entity recognizer, the component that reads a sentence and tags which words are people, places, or organizations, from a regular-expression fallback to a proper statistical model. Recall fell from 0.200 to 0.067. A better component made the system worse, which is the kind of result that stops you mid-sprint. The recognizer was not at fault. The skip rule was. A source document can mention a person in a role that has nothing to do with the claim under evaluation, and still mention them truthfully. In one item, the claim misattributed who was second to walk on the Moon, and the context named the substituted astronaut correctly in a different sentence, doing a different thing. The weaker recognizer missed that mention and flagged the error. The stronger one found it, read it as confirmation, and waved the error through. Of twenty-seven items the detector wrongly skipped, twenty-six followed this pattern. The heuristic was never checking the right relationship. It asked whether the entity appears in the document when it needed to ask what the entity is doing in the sentence. Improving the model's ability to answer the wrong question just made it answer the wrong question more reliably. This is a hazard anywhere a rule sits on top of a learned component. Ablating downward, swapping in a deliberately weaker component to confirm the strong one is earning its cost, is something I do routinely. Ablating upward is rarer, and it tells you more: a rule that degrades when its inputs improve is a rule whose logic was wrong all along, and no amount of model quality saves it. A third instance, smaller but the same shape: a pattern for matching units of measurement was absorbing a trailing word, which let bare four-digit years slip past a filter meant to exclude them from numeric comparison. Fixing one regular expression moved numerical precision from 0.857 to 0.947. A lot of apparently semantic behavior turns out to be lexical. What the Headline Number Was Hiding None of these three defects were visible in the metric I would have reported at a demo. On the binary question of whether a claim is unsupported, the full system reached precision 0.991 and recall 0.986: almost nothing it flagged was fine, and almost nothing that was wrong got past it. Those are the numbers that go in an abstract. Averaged across the eight failure categories the system is supposed to distinguish, precision was 0.578 and recall 0.723. One category sat at 0.067 recall. Another fired on nearly everything, reaching 0.960 recall at 0.198 precision, meaning it claimed credit for errors that more specific detectors had already identified correctly. The binary number was not wrong. It was answering a question so coarse that every interesting failure averaged out of it. A system can be excellent at deciding that something is broken and close to useless at saying what broke. If the only number on your dashboard is the first one, you won't find out until the fine-grained output reaches someone who depends on it. None of this is new. It's the same argument as reporting per-class results instead of overall accuracy on an imbalanced dataset. Everyone agrees with it in principle and skips it anyway, because the aggregate is the number that makes the case for the work. Not Getting Fooled by Your Own Corpus Four practices came out of this, all cheap, none of them clever. Write your evaluation inputs in the register your production system receives. If your system reads retrieved documents, your test contexts should read like documents, not like annotations about documents. Voice is a feature your model can see, and the voice of someone who already knows the answer is a particularly dangerous one to hand it.Ablate upward, not just downward. Replace a component with a better one and check that every metric moves in the direction you expect. When something moves the wrong way, the rule sitting on top of that component is making an assumption you have not written down.Report per-stratum results next to the aggregate, always in the same table. Not in an appendix, not on request. If a category is at 0.067, that fact should be as easy to see as the number you are proud of.Hold out items you did not write. A corpus built by the same people who defined the categories will flatter the categories. Mine did. That is the single largest caveat on everything above, and no amount of internal rigor substitutes for a test set authored by someone else. The first one I would not have thought of before it cost me a day, and it's the one I now suspect is quietly wrong in a lot of synthetic eval sets. Injected-error benchmarks are easy to build, and their labels are correct by construction, which makes them feel safer than they are. The label being right does not mean the input is representative. The Register Your System Actually Speaks The failures worth writing up are rarely the ones where the model underperforms. They are the ones where the measuring apparatus was quietly reporting on something other than what you thought. A detector that scores 0.067 because the test data argues with itself is not a model problem. Neither is a rule that gets worse as its inputs get better, or an aggregate that averages away the only result that mattered. A bigger model fixes none of it. What fixes them is reviewing the evaluation harness as carefully as the thing it measures, defects and all. That's unglamorous work, and where most of my debugging time went. Probably where most of yours goes too.

Oracle Database 23ai introduced the powerful DBMS_DEVELOPER package, giving developers and database administrators a streamlined way to access database object metadata in JSON format. This feature represents a significant advancement in how we interact with database schemas, offering a more structured and programmatic way to extract and analyze metadata compared to traditional dictionary views or the older DBMS_METADATA package. In this article, we'll explore the capabilities of DBMS_DEVELOPER, focusing on its GET_METADATA function through detailed examples and practical implementation scenarios. Understanding DBMS_DEVELOPER The DBMS_DEVELOPER package was designed specifically for modern application development patterns, where JSON has become a universal data exchange format. Rather than returning metadata as DDL statements (like DBMS_METADATA), this package returns structured JSON documents that can be easily parsed, processed, and integrated into applications or DevOps workflows. Key Benefits Structured data format: Returns metadata as JSON objects that can be easily parsed Programmatic access: Perfect for integration with applications and automation scripts Versioning capabilities: Built-in ETag mechanism for tracking object changesConfigurable detail levels: Ability to retrieve basic, typical, or comprehensive metadata Setting Up Our Environment Let's set up a sample schema to demonstrate the package functionality: SQL CREATE TABLE customers ( customer_id NUMBER(10) CONSTRAINT pk_customers PRIMARY KEY, first_name VARCHAR2(50) NOT NULL, last_name VARCHAR2(50) NOT NULL, email VARCHAR2(100) CONSTRAINT uk_customer_email UNIQUE, join_date DATE DEFAULT SYSDATE, status VARCHAR2(10) DEFAULT 'ACTIVE' ); CREATE INDEX idx_customer_name ON customers(last_name, first_name); CREATE OR REPLACE VIEW active_customers AS SELECT customer_id, first_name, last_name, email FROM customers WHERE status = 'ACTIVE'; GET_METADATA Basics The core function of the DBMS_DEVELOPER package is GET_METADATA, which returns metadata about database objects in JSON format. Let's start with a basic example: SQL -- Using JSON_SERIALIZE for formatted output SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA(name => 'CUSTOMERS') PRETTY) AS metadata; The result is a structured JSON document containing comprehensive information about the table, including: Table name and schema Column definitions with data types and constraints Primary key, unique key, and foreign key information Index definitions An etag value representing the current state of the object This structured format makes it significantly easier to extract specific information programmatically compared to parsing DDL statements. NAME and SCHEMA Parameters The NAME and SCHEMA parameters work together to identify the specific database object. These parameters are case-sensitive and must match the object definition in the data dictionary. SQL -- Explicitly specifying schema SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'CUSTOMERS', schema => 'FINANCE') PRETTY) AS metadata; -- Using current schema (implicit) SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA(name => 'CUSTOMERS') PRETTY) AS metadata; When the SCHEMA parameter is omitted, the function uses the current schema. This behavior provides flexibility when working with objects across different schemas in your database environment. OBJECT_TYPE Parameter The OBJECT_TYPE parameter allows you to explicitly specify the type of object you're retrieving metadata for. While often optional (as the database can infer the object type from the name), it becomes necessary in cases where name resolution alone is insufficient. Currently, `DBMS_DEVELOPER` supports three object types: TABLEINDEXVIEW Let's examine metadata for our index and view: SQL -- Retrieving index metadata SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'IDX_CUSTOMER_NAME', object_type => 'INDEX') PRETTY) AS metadata; -- Retrieving view metadata SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'ACTIVE_CUSTOMERS', object_type => 'VIEW') PRETTY) AS metadata; The OBJECT_TYPE parameter becomes particularly important when dealing with objects that share the same name but have different types, such as packages and package bodies. LEVEL Parameter The LEVEL parameter controls the amount of detail included in the JSON output. Oracle provides three levels: BASIC: Minimal informationTYPICAL: Standard level of detail (default)ALL: Comprehensive metadata This flexibility lets you balance concise output with detailed information based on your needs. SQL -- Basic level metadata SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'IDX_CUSTOMER_NAME', level => 'BASIC') PRETTY) AS metadata; -- All details SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'IDX_CUSTOMER_NAME', level => 'ALL') PRETTY) AS metadata; The output at the ALL level includes additional attributes such as segment information, compression settings, and physical storage details that aren't present at the BASIC level. ETAG Parameter One of the most powerful features of DBMS_DEVELOPER is the etag mechanism, which provides version tracking for database objects. The etag value changes whenever the object definition changes, making it invaluable for change detection. SQL -- Store the current etag value DECLARE v_metadata CLOB; v_etag VARCHAR2(100); BEGIN v_metadata := DBMS_DEVELOPER.GET_METADATA(name => 'ACTIVE_CUSTOMERS'); SELECT JSON_VALUE(v_metadata, '$.etag') INTO v_etag FROM dual; DBMS_OUTPUT.PUT_LINE('Current etag: ' || v_etag); END; / -- Modify the view CREATE OR REPLACE VIEW active_customers AS SELECT customer_id, first_name, last_name, email, join_date FROM customers WHERE status = 'ACTIVE'; -- Check if the object has changed using the stored etag SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'ACTIVE_CUSTOMERS', etag => 'A1B2C3D4E5F6G7H8I9J0') -- Previous etag value PRETTY) AS metadata; When you pass an ETag value that matches the current state of the object, the function returns an empty JSON document {}. If the object has changed, it returns the complete metadata with a new ETag value. Practical Scenario: Database Migration and Documentation Let's consider a practical scenario where DBMS_DEVELOPER proves invaluable: a large-scale database migration project with continuous schema changes. The Challenge You're leading a project to migrate a critical application database from on-premises to Oracle Cloud. The development team continues to make schema changes during the migration process, and you need to: Document the current state of all database objectsTrack changes between migration wavesValidate that objects were created correctly in the target environmentGenerate comprehensive documentation for compliance requirements The Solution Using DBMS_DEVELOPER, you can create a robust metadata management system: SQL CREATE TABLE schema_versions ( object_name VARCHAR2(128), object_type VARCHAR2(30), object_schema VARCHAR2(128), capture_date TIMESTAMP, etag VARCHAR2(100), metadata CLOB ); -- Procedure to capture all tables in a schema CREATE OR REPLACE PROCEDURE capture_schema_metadata(p_schema VARCHAR2) AS v_metadata CLOB; v_etag VARCHAR2(100); CURSOR c_objects IS SELECT object_name, object_type FROM all_objects WHERE owner = p_schema AND object_type IN ('TABLE', 'INDEX', 'VIEW'); BEGIN FOR obj IN c_objects LOOP BEGIN v_metadata := DBMS_DEVELOPER.GET_METADATA( name => obj.object_name, schema => p_schema, object_type => obj.object_type ); SELECT JSON_VALUE(v_metadata, '$.etag') INTO v_etag FROM dual; INSERT INTO schema_versions (object_name, object_type, object_schema, capture_date, etag, metadata) VALUES (obj.object_name, obj.object_type, p_schema, SYSTIMESTAMP, v_etag, v_metadata); COMMIT; DBMS_OUTPUT.PUT_LINE('Captured metadata for ' || obj.object_type || ' ' || p_schema || '.' || obj.object_name); EXCEPTION WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE('Error capturing ' || obj.object_type || ' ' || p_schema || '.' || obj.object_name || ': ' || SQLERRM); END; END LOOP; END; / This solution provides several key benefits: Efficient change tracking: Using etags to identify exactly which objects have changedStructured documentation: Storing metadata in JSON format for easy extraction of specific attributesHistorical record: Maintaining snapshots of schema evolution over timeValidation capabilities: Comparing source and target schemas during migration During migration, you can extend this system to compare environments: -- Procedure to compare object between environments CREATE OR REPLACE PROCEDURE compare_object( p_name VARCHAR2, p_type VARCHAR2, p_source_schema VARCHAR2, p_target_schema VARCHAR2, p_target_db VARCHAR2 ) AS v_source_metadata CLOB; v_target_metadata CLOB; v_source_etag VARCHAR2(100); v_target_etag VARCHAR2(100); BEGIN -- Get source metadata v_source_metadata := DBMS_DEVELOPER.GET_METADATA( name => p_name, schema => p_source_schema, object_type => p_type ); -- Get target metadata via database link EXECUTE IMMEDIATE 'SELECT DBMS_DEVELOPER.GET_METADATA( name => :1, schema => :2, object_type => :3 ) FROM dual@' || p_target_db INTO v_target_metadata USING p_name, p_target_schema, p_type; -- Extract etag values SELECT JSON_VALUE(v_source_metadata, '$.etag') INTO v_source_etag FROM dual; SELECT JSON_VALUE(v_target_metadata, '$.etag') INTO v_target_etag FROM dual; -- Compare and report IF v_source_etag = v_target_etag THEN DBMS_OUTPUT.PUT_LINE('Objects match exactly'); ELSE DBMS_OUTPUT.PUT_LINE('Objects differ - detailed comparison needed'); -- Further JSON comparison logic could be implemented here END; END; / Conclusion The DBMS_DEVELOPER package represents a significant advancement in Oracle's metadata management capabilities. By providing metadata in JSON format, Oracle has created a more developer-friendly interface that aligns with modern application architecture patterns. Key takeaways include: JSON-based metadata is more programmatically accessible than traditional DDL statements The etag mechanism provides a reliable way to track object changes Multiple detail levels allow you to retrieve just the information you need The package is particularly valuable for documentation, migration, and change tracking While currently limited to tables, indexes, and views, the DBMS_DEVELOPER package has tremendous potential for expansion in future Oracle releases. Database architects and developers should consider integrating this powerful tool into their workflows, particularly for projects involving schema documentation, migration, or programmatic metadata access. As databases continue to evolve toward more autonomous and programmable systems, tools like DBMS_DEVELOPER will become increasingly central to efficient database management practices.

By arvind toorpu CORE

Most production agent projects do not fail because the model is weak. They fail because one agent was asked to hold too much at once: routing, planning, tool use, memory, and error recovery all inside a single growing prompt. By 2026, this failure mode shows up in nearly every engineering retro, and the fix is usually the same. Split the work across several coordinated agents. The numbers back this up. Gartner reports that roughly 80% of enterprise applications shipped or updated in early 2026 embed at least one AI agent, up from about a third in 2024. Yet a figure cited across IDC and Forrester research puts pilot-to-production failure near 88%, and the root causes cluster on orchestration, data access, and evaluation gaps, not model quality. Architecture, not model choice, is where most of these systems are won or lost. This piece walks through the multi-agent patterns worth knowing, with notes on when each one fits and where it tends to break. What Is a Multi-Agent System? A multi-agent system is a set of specialized agents that split a task, coordinate through shared state or messages, and combine their outputs into one result. Each agent owns a narrow job: a planner decides steps, a researcher gathers context, a writer drafts, a critic reviews. This keeps prompts short, makes behavior easier to test, and lets you retry or swap one part without rerunning the whole chain. Why Single-Agent Designs Hit a Ceiling A single agent works well until the task branches. Add several tools, conditional logic, and long context, and the model starts to lose the thread. Instructions compete, the context window fills with irrelevant history, and one bad tool call derails everything downstream. Splitting responsibilities gives each agent a smaller decision space, which is easier to reason about and cheaper to debug. Core Architecture Patterns for Multi-Agent Systems 1. Orchestrator (Supervisor) Pattern A central agent receives the request, decides which worker should handle it, and routes accordingly. Workers do not talk to each other; they report back to the supervisor, which picks the next move. Python def supervisor(task, state): route = router_model(task, state) # pick the next worker if route == "research": return research_agent(task) if route == "code": return code_agent(task) if route == "done": return finalize(state) This is the most common starting point. Centralized control makes logging and human review straightforward. The tradeoff: the supervisor becomes a bottleneck and a single point of failure. 2. Sequential (Pipeline) Pattern Agents run in a fixed order, each consuming the previous output: extraction, then validation, then summary. Use it when steps are stable and order matters. It is simple to trace, but rigid. A change in requirements often means rewriting the chain. 3. Hierarchical Agent Teams Supervisors manage sub-supervisors, which manage workers. A top planner splits a goal into subgoals, hands each to a team lead, and each lead coordinates its own workers. This scales to larger problems and mirrors how organizations already divide labor, at the cost of more coordination overhead and latency. Anthropic's Claude Agent SDK added hierarchical subagent spawning in 2026 for exactly this shape of problem. 4. Network (Peer-to-Peer) Pattern Agents hand control directly to one another based on the task, with no fixed hub. The handoff model in the OpenAI Agents SDK works this way: a triage agent passes a conversation to a billing or support agent, which can pass it on again. It fits open-ended, conversational AI agents where the next step is not known in advance. The risk is loops and unclear ownership, so you need turn limits and explicit exit conditions. 5. Blackboard (Shared State) Pattern Agents read from and write to one shared store instead of messaging each other directly. Each agent watches the board, contributes when it can help, and stops when the goal is met. This decouples agents cleanly but makes state management the hard part. Concurrent writes and stale reads cause most of the bugs. State and Communication: The Real Design Decision Patterns are the visible layer. Beneath them sits the question that decides how hard your system is to operate: how do agents share information? Two options dominate. Shared state keeps one structured object that every agent updates, which is easy to inspect and checkpoint; LangGraph builds on this with checkpointing and time-travel debugging. Message passing sends discrete messages between agents, which maps well to conversational and event-driven designs such as AutoGen and its successor AG2. Shared state is easier to audit. Message passing is easier to distribute. Pick based on which one your team can debug at 2 a.m. Choosing the Right Pattern If you need... Reach for Central control and easy logging Orchestrator Fixed, ordered steps Sequential pipeline Large tasks split across teams Hierarchical Open-ended, conversational flow Network/handoffs Loose coupling, many contributors Blackboard A few rules hold across all of them. Start with the simplest pattern that could work, usually an orchestrator, and add structure only when a real limit appears. Give every agent a narrow role and a clear stop condition. And treat evaluation as part of the architecture, not an afterthought. Why This Matters in 2026 Teams that cross from pilot to production share one habit: they instrument everything. Failure analyses in 2026 point to observability and evaluation coverage as the largest single blocker, ahead of tool access and data quality. In practice, that means logging every agent decision, running automated evals on each step, and putting human review gates where a wrong action is expensive. Generative AI agents are only as trustworthy as the traces they leave behind. Multi-agent architecture is moving from research demos to standard practice, and the frameworks now converge on the same primitives: state, handoffs, checkpoints, subagents. That convergence means the durable skill is not in any single library. It is knowing which pattern fits the problem in front of you and being able to explain why.

Why This Combination Matters Most RAG tutorials stop at the same point: embed some documents, stuff them into a vector store, retrieve the top-k chunks, and paste them into a prompt. That gets you a demo. It does not get you a system another team can call, monitor, version, and trust. Three pieces close that gap: RAG – the retrieval-augmented generation pattern itself: chunk, embed, retrieve, ground the model's answer in real data.A vector database – the durable, queryable index that makes retrieval fast and scalable instead of a linear scan through embeddings in memory.MCP (Model Context Protocol) – the standard that lets any MCP-compatible host (Claude Desktop, Claude Code, your own agent runtime) call that retrieval capability as a tool, instead of every team hand-rolling its own glue code between the model and the data. Put together, the pattern looks like this: Plain Text Host (Claude / Claude Code / your agent) │ MCP protocol (JSON-RPC over stdio or HTTP+SSE) ▼ MCP Server ("docs-search") │ calls ▼ RAG Retrieval Layer → Vector DB (Chroma / pgvector / Qdrant) │ Embedding Model The host never talks to your vector database directly. It talks to a tool. That one architectural decision is what turns a notebook prototype into something you can put behind an SLA. Part 1: RAG, Built for Production, Not for a Demo The two places demo-quality RAG breaks in production are chunking and retrieval quality. Fix those first. Chunking With Overlap and Metadata Python from dataclasses import dataclass from typing import List @dataclass class Chunk: text: str source: str chunk_id: str page: int | None = None def chunk_document(text: str, source: str, chunk_size: int = 800, overlap: int = 120) -> List[Chunk]: """Sliding-window chunking with overlap to avoid cutting context across boundaries — the single highest-leverage fix for weak retrieval.""" chunks = [] start = 0 idx = 0 while start < len(text): end = min(start + chunk_size, len(text)) piece = text[start:end] chunks.append( Chunk(text=piece, source=source, chunk_id=f"{source}-{idx}") ) start += chunk_size - overlap idx += 1 return chunks Two things matter here that most tutorials skip: overlap (so an answer that straddles a chunk boundary doesn't get orphaned) and metadata on every chunk (source, page, chunk_id) so the model — and your logs — can cite where an answer came from. Embedding With Batching and Retry Python import time from openai import OpenAI client = OpenAI() def embed_batch(texts: list[str], model: str = "text-embedding-3-large", max_retries: int = 3) -> list[list[float]]: for attempt in range(max_retries): try: resp = client.embeddings.create(model=model, input=texts) return [d.embedding for d in resp.data] except Exception as e: if attempt == max_retries - 1: raise time.sleep(2 ** attempt) Batch embedding calls (not one request per chunk) and add exponential backoff — at index-build time you're often pushing tens of thousands of chunks through the embedding API, and that's where rate limits bite. Part 2: The Vector Database Layer A vector database earns its place the moment your corpus is too large to hold in memory, or the moment you need filtered retrieval (by tenant, document type, date range) alongside similarity search. Here's a production-shaped setup using Chroma, with the pattern identical if you swap in pgvector or Qdrant. Python import chromadb from chromadb.config import Settings client = chromadb.PersistentClient(path="./vector_store") collection = client.get_or_create_collection( name="product_docs", metadata={"hnsw:space": "cosine"} # cosine similarity, HNSW index ) def index_chunks(chunks: list[Chunk]): embeddings = embed_batch([c.text for c in chunks]) collection.upsert( ids=[c.chunk_id for c in chunks], embeddings=embeddings, documents=[c.text for c in chunks], metadatas=[{"source": c.source, "page": c.page or 0} for c in chunks], ) def retrieve(query: str, top_k: int = 5, source_filter: str | None = None): q_embedding = embed_batch([query])[0] where = {"source": source_filter} if source_filter else None results = collection.query( query_embeddings=[q_embedding], n_results=top_k, where=where, ) return list(zip(results["documents"][0], results["metadatas"][0])) Notice upsert, not insert — production indexes get re-crawled and re-embedded constantly, and re-indexing should be idempotent by chunk_id. Notice also the where filter — real retrieval almost always needs a metadata constraint alongside the similarity search, or you'll surface the right kind of chunk from the wrong tenant's documents. A layer worth adding before this goes live: a semantic cache in front of the vector query. If the same or a near-duplicate question comes in repeatedly (which it will, in any real user base), you don't want to re-embed and re-query every time. A thin cache keyed on embedding similarity — check for a cached answer within a cosine-distance threshold before hitting the vector DB — cuts both latency and embedding-API cost substantially in high-traffic RAG deployments. Part 3: Exposing Retrieval as an MCP Tool This is the piece that makes the difference between "a RAG pipeline I run in a notebook" and "a capability any Claude-based host can use." Instead of embedding your retrieval logic into every application that needs it, you expose it once, as an MCP server, and any compliant host — Claude Desktop, Claude Code, a custom agent — can call it the same way. Python # mcp_server.py from mcp.server import Server from mcp.server.stdio import stdio_server from mcp.types import Tool, TextContent import asyncio app = Server("docs-search") @app.list_tools() async def list_tools() -> list[Tool]: return [ Tool( name="search_docs", description=( "Search the product documentation vector index and " "return the most relevant passages with their sources." ), inputSchema={ "type": "object", "properties": { "query": {"type": "string", "description": "The search query"}, "top_k": {"type": "integer", "default": 5}, "source_filter": {"type": "string", "description": "Optional source doc to restrict to"}, }, "required": ["query"], }, ) ] @app.call_tool() async def call_tool(name: str, arguments: dict) -> list[TextContent]: if name != "search_docs": raise ValueError(f"Unknown tool: {name}") results = retrieve( query=arguments["query"], top_k=arguments.get("top_k", 5), source_filter=arguments.get("source_filter"), ) formatted = "\n\n".join( f"[Source: {meta['source']}, page {meta['page']}]\n{doc}" for doc, meta in results ) return [TextContent(type="text", text=formatted or "No matching passages found.")] async def main(): async with stdio_server() as (read_stream, write_stream): await app.run(read_stream, write_stream, app.create_initialization_options()) if __name__ == "__main__": asyncio.run(main()) Register it with Claude Code or Claude Desktop with a short config entry: JSON { "mcpServers": { "docs-search": { "command": "python", "args": ["mcp_server.py"] } } } From that point on, when a developer working in Claude Code asks a question that needs grounding in your documentation, the host discovers the search_docs tool, calls it with the right arguments, gets back cited passages, and folds them into its answer — with no custom integration code per application. That is the actual point of MCP: one retrieval service, called the same way by every host that speaks the protocol, instead of a bespoke RAG wrapper duplicated inside every app, chatbot, and IDE plugin your organization builds. Production Considerations Before You Ship This Observability – log every tool call: query text, top_k, latency, which chunks were returned, and — if you can capture it — whether the final answer used them. Without this, you're debugging RAG quality blind.Freshness – decide explicitly how re-indexing happens (scheduled crawl, webhook on document change, or both) and make upsert idempotent so partial re-index failures don't corrupt the collection.Access control at the MCP boundary – the MCP server, not the LLM, is the right place to enforce which documents a given caller is allowed to search. Filter by tenant/user in the call_tool handler before the query ever reaches the vector database.Timeouts and fallbacks – a vector DB query that hangs should not hang the whole conversation. Set a hard timeout on retrieve() and have the tool return a clear "search unavailable" message rather than blocking.Evaluation – keep a small, versioned set of query/expected-passage pairs and re-run it whenever you change the chunking strategy, the embedding model, or the index. Chunking changes are the single most common silent cause of retrieval regressions. Closing RAG gives you the pattern, the vector database gives you the scale, and MCP gives you the interface that lets any host reuse the pipeline without re-implementing it. None of the three pieces is complicated on its own — the production value comes from wiring them together deliberately: idempotent indexing, filtered retrieval, a caching layer in front of the vector store, and access control enforced at the tool boundary rather than left to the model's judgment.

By Balaji Venkatasubramaniyar CORE

After a decade of building and debugging large-scale data pipelines across financial services, payments processing, and analytics platforms, I can tell you that almost every slow Spark job I've investigated had the same root cause — and it wasn't the one the team thought it was. The default response when a Spark job is slow is to add more executor memory, increase the number of executors, or bump spark.sql.shuffle.partitions. Sometimes that helps. Usually it doesn't. What I've found, consistently, is that the real problems are structural — a join strategy mismatch that silently multiplies your intermediate dataset by ten times, a single slow task on a degraded node that holds an entire stage hostage, or a decrypt chain that re-reads source data six times when it only needed to read it once. This article is organized around five patterns I keep seeing across teams. Each one looks different on the surface but traces back to a misunderstanding of how Spark actually executes your code. For each pattern, I'll describe what it looks like, when it bites you, the failure mode, and how to fix it. Pattern 1: The OR Join That Quietly Multiplies Your Data What It Looks Like A join condition with an OR clause. Usually introduced when a business requirement adds a secondary matching rule — match on primary card number, or if the transaction is a virtual card transaction, match on the underlying physical PAN. The SQL looks reasonable. The engineer tests it on a sample, and it returns the right rows. When It Bites You At scale. With 100 million transaction rows and 50 million account rows, this query starts running for hours. The output size is also wrong — much larger than expected before DISTINCT trims it down. The Failure Mode Spark cannot use a hash join or sort-merge join when the join condition contains OR. It falls back to BroadcastNestedLoopJoin — for every row in the left table, scan every row in the right table. That's O(n x m). On real datasets, this produces an intermediate result in the hundreds of GB before any downstream filter runs. I've watched a pipeline that should produce 8 GB of output generate 400 GB of intermediate data because of exactly this pattern, taking a 20-minute job to 4 hours. You can verify this in 30 seconds: run df.explain(formatted) and look for BroadcastNestedLoopJoin in the physical plan. If you see it on a join involving any table over a few million rows, it's almost certainly unintentional. The Fix Split the join into two equi-join legs and UNION ALL the results: SQL -- Leg 1: primary match (equi-join — uses SortMergeJoin or BroadcastHashJoin) SELECT txn.*, acct.* FROM transactions txn JOIN accounts acct ON txn.card_number = acct.card_number UNION ALL -- Leg 2: fallback match, filtered scope only SELECT txn.*, acct.* FROM transactions txn JOIN accounts acct ON txn.fpan = acct.physical_pan WHERE txn.transaction_type = 'VIRTUAL' Each leg is a proper equi-join. Apply DISTINCT at the end to deduplicate rows that matched both. The performance difference is routinely an order of magnitude. Pattern 2: The Straggler Task That Nobody Notices Until It's Too Late What It Looks Like A stage that should take 10 minutes takes 3 hours. The Spark UI shows nearly all tasks completed quickly. One or two tasks are still running with a disproportionately long duration. When It Bites You Jobs running on shared YARN or cloud infrastructure where any node can have a bad disk, a noisy neighbor, or degraded network throughput. Also common in stages that call external services per partition — one slow API response can cause a single partition's tasks to take 100x longer than the others. The Failure Mode A stage doesn't complete until the last task completes. Not the median. Not p95. The absolute last one. If 2,200 tasks finish in under 2 minutes and one takes 3 hours and 7 minutes, the stage takes 3 hours and 7 minutes. The other 2,199 executors sit idle. This is the straggler problem, and it's distinct from data skew. The diagnostic: in the Stage detail view, check the task duration distribution. If MAX is dramatically higher than p99, that's a straggler (hardware or external service issue). If p75 is already much higher than p50, that's skew (data distribution issue). They require different fixes, and many teams treat them identically. The Fix For stragglers caused by degraded infrastructure, enable Spark speculation: Properties files spark.speculation=true spark.speculation.multiplier=3 # task must be 3x slower than median spark.speculation.quantile=0.9 # wait for 90% completion before speculating Speculation re-launches slow tasks on a different executor and uses whichever copy finishes first. The caveat: don't use this on stages that write to non-idempotent sinks. For read-heavy or compute-heavy stages — including external decryption calls — it's often the single most impactful config change you can make. Pattern 3: The df.rdd Decrypt Chain That Recomputes Everything Six Times What It Looks Like A pipeline that calls an external encryption or decryption service per record, implemented as a series of df.rdd.mapPartitions() calls, one per column that needs to be processed. When It Bites You When you have multiple columns to decrypt. Each .rdd call creates a new computation starting from the original DataFrame — Spark re-reads from source, re-executes all upstream joins and filters, and then runs the decryption for that column. With six columns to decrypt, you're doing that six times. The Failure Mode Two distinct sub-problems compound each other. First, going to RDD bypasses Catalyst entirely — no predicate pushdown, no column pruning, no Tungsten execution. Second, without a persist checkpoint before the chain, every decrypt call lineages all the way back to the source. I've seen this double the runtime of a job compared to the same pipeline with a single persist() before the decrypt chain. On top of that, the external call latency per partition is dominated by the number of HTTP round trips, not the payload size. Cutting your batch size in half doubles your request count and roughly doubles your wall-clock time for that stage. Most teams set an initial batch size and never revisit it. The Fix Two changes, applied together: Persist the input DataFrame before starting the decrypt chain. This means the join and filter logic runs once, and each decrypt call reads from the cached result.Increase the batch size for external calls. Test at several sizes — going from 20,000 to 40,000 records per batch often cuts stage time by 30-50% with no change to correctness. Scala val base = rawDf.filter(...).join(key1, ...).persist(StorageLevel.MEMORY_AND_DISK) val step1 = decryptColumn(base, secret1) // reads from cache val step2 = decryptColumn(step1, secret2) // reads from cache val step3 = decryptColumn(step2, secret3) // reads from cache Without persist, step2 re-executes everything step1 did from source. With persist, each step reads from the in-memory result of the previous. Pattern 4: The shuffle.partitions Setting That Nobody Updates What It Looks Like A job that works fine in staging — where data volumes are 10% of production — but runs slowly, spills to disk, or produces thousands of tiny output files in production. When It Bites You When the default spark.sql.shuffle.partitions=200 is left unchanged. 200 partitions made sense as a default for medium datasets but is almost always wrong at production scale — either too few (huge partitions, memory pressure) or too many (tiny partitions, scheduling overhead, small files problem). The Failure Mode Too few partitions means each executor handles a disproportionately large chunk of data. With 200 partitions on a 1 TB shuffle, each partition is 5 GB. That will spill to disk. Too many partitions means thousands of 1 MB tasks — the scheduling overhead becomes significant, and your output has thousands of tiny files that hurt downstream readers. With Adaptive Query Execution (AQE) enabled in Spark 3.2+, this problem largely manages itself. AQE merges small post-shuffle partitions automatically and can handle modest skew. But AQE can't help if it's disabled, and it can't fix the upstream causes of extreme skew. The Fix Enable AQE if you're on Spark 3.2+: Properties files spark.sql.adaptive.enabled=true spark.sql.adaptive.coalescePartitions.enabled=true spark.sql.adaptive.skewJoin.enabled=true If you need to set shuffle.partitions manually, target roughly 128-256 MB per partition post-shuffle. For a 500 GB shuffle, that means 2,000-4,000 partitions. Set it high and let AQE coalesce down — that's cheaper than setting it low and getting OOM errors. Pattern 5: The Incremental Job That Degrades Silently Over Time What It Looks Like A job that runs in 15 minutes when first deployed and runs in 4 hours six months later. No code changes. No obvious data quality issues. The team attributes it to data growth. When It Bites You When the job fails a few times in a row, and the recovery accumulates multiple windows' worth of data. Or when the watermark logic was designed for small windows but nobody anticipated that the underlying join tables would grow significantly. The Failure Mode Two separate causes, often confused. First, if the watermark is a single timestamp and the job has been failing, recovery runs can accumulate large backlogs. A job that normally processes 2 hours of data may need to process 48 hours on first successful recovery, with no change to the resource configuration. Second, growth in reference data (like an accounts table or lookup table used in a join) increases the size of every run regardless of whether the incremental input grew. I've seen a 30-minute job become a 3-hour job purely because the accounts table grew from 10 million rows to 80 million rows over 18 months, while the OR join condition (see Pattern 1) meant that growth was amplified into the intermediate result. The Fix Two design principles that pay off over the lifetime of the pipeline: Track processed partitions explicitly rather than using a single timestamp watermark. This makes recovery granular — you can replay specific missing partitions without re-processing everything after them.Add a fast-path no-op check before initializing the full Spark session. Check whether any new partitions exist first. A 5-second check that exits early is much better than a 2-minute executor startup that discovers there's nothing to process. For the reference table growth problem: if your lookup table grows significantly, revisit whether it can be broadcast (small enough to fit in executor memory) or whether the join itself needs to be redesigned. Quick Diagnostic Reference Use this table to map what you observe in the Spark UI to the likely pattern and first action to take: WHat you observeLikely patternconfirm withfirst action MAX task duration >> p99 Straggler (Pattern 2) Task timeline in Stage UI Enable spark.speculation p75 >> p50 task duration Data skew Input bytes per task Repartition on join key; AQE skewJoin BroadcastNestedLoopJoin in explain() OR join (Pattern 1) df.explain( formatted) Rewrite as UNION of equi-joins Stage runtime grows week on week; no code change Incremental accumulation or reference table growth (Pattern 5) Input bytes trend in History Server Audit watermark logic; check reference table size OOM errors or heavy disk spill Too few shuffle partitions (Pattern 4) Spill metrics in Stage UI Enable AQE or increase shuffle.partitions The Common Thread Every pattern here traces back to the same underlying issue: Spark is executing something different from what the engineer intended. The OR join was intended as a flexible matching rule; Spark turned it into a nested loop. The decrypt chain was intended as six independent transformations; Spark turned it into six full re-reads of source data. The incremental job was intended to process one window of data; without proper watermark design, it occasionally processes twelve. The Spark UI has everything you need to see this — task distribution, input and output sizes, physical plans, spill metrics. Most teams open it when something breaks and close it once they find the obvious error. Opening it proactively, forming a hypothesis, and then confirming or refuting it in the metrics is the practice that separates engineers who consistently improve pipeline performance from those who add executor memory and hope for the best. The mistake isn't choosing the wrong config. It's not understanding what Spark is actually doing with your code.

What Is Accessibility Testing? Imagine trying to use a website... With your eyes closed.Using only your keyboard, no mouse.With your hands busy, so you have to use voice commands.If you couldn't distinguish the color green from red. Accessibility testing (often called "a11y" testing) is the process of ensuring that your website or app can be used by everyone, including people with disabilities. It's not about political correctness; it's about building a web that works for all humans. It's also the law in many countries. The Main Testing Points to Consider Here are the most common and critical areas to test, framed as simple questions. 1. Keyboard Navigation (Operable) Can I use the website with just the Tab key? This is the #1 test. Try tabbing through all interactive elements.Is there a visible focus indicator? As you tab, can you always see where you are on the page? (A faint dotted line is not enough!)Can I trigger all actions with the Enter or Space key? Buttons, menus, etc. 2. Screen Reader Compatibility (Perceivable and Robust) Does every image have descriptive alt text? A screen reader can't describe a picture. alt="Company Logo" is good. alt="" is ok for decorative images. alt="image123.jpg" is terrible.Is the page structure logical? Use proper HTML tags (<h1>, <h2>, <nav>, <button>) so a screen reader user can understand the page layout.Do form fields have clear labels? A screen reader user needs to know what to type into each box. Use the <label> tag. 3. Color and Contrast (Perceivable) Is there enough contrast between text and its background? Light gray text on a white background is impossible for many to read. Use online tools to check contrast ratios.Is color alone used to convey information? For example, "The required fields are in red." This fails for colorblind users. There must be another indicator, like an asterisk (*). 4. Text Clarity (Understandable) Can the text be resized without breaking the layout? Try zooming the browser to 200%. Does the page become a mess, or does it reflow properly?Is the language simple and clear? Avoid complex jargon. 5. Multimedia (Perceivable) Do videos have captions? For users who are deaf or hard of hearing.Do audio clips have transcripts? For the same reason. 6. Predictable Navigation (Understandable) Is navigation consistent across the site? Menus shouldn't move around randomly.Do links clearly describe where they go? "Click here" is bad. "Download the syllabus (PDF)" is good. Those six areas are what you're checking for, whether you're doing it by hand or automating it. The rest of this tutorial is about the second part: how much of that you can actually catch with code, and how to wire it into a Playwright suite. In this article, we'll cover: What automated accessibility testing actually checks, and what it doesn'tHow to wire up Playwright with Axe-Core, using a demo page with deliberately planted bugs so the results are consistent every time you run it What Automated Accessibility Testing Actually Checks Automated accessibility testing runs a rule engine against your rendered DOM and flags violations of standards like WCAG 2.1/2.2. Axe-Core, built by Deque Systems, is the engine most Playwright and Cypress teams reach for, and for good reason — it has close to zero false positives, which is unusually rare for this kind of tooling. The honest caveat, worth repeating every time this topic comes up: automated scans catch roughly 20-30% of accessibility issues. Missing alt text, poor contrast, missing form labels, invalid ARIA attributes — that's exactly what a rule engine is good at. Whether your focus order makes sense to someone tabbing through with a keyboard, or whether a screen reader user can actually get through your custom dropdown — that still needs a human. Axe is a very thorough linter, not a replacement for real usability testing. Specifically, axe won't catch: Keyboard traps, like a modal or custom select you can tab into but not out ofWhether focus is correctly moved when a modal opens or closesWhether alt text is meaningful (alt="image" passes the rule and is still useless)Whether video captions actually match the audio That's the gap manual testing and keyboard-only walkthroughs are there to cover — axe is one layer, not the whole strategy. Instead of scanning a live third-party site (whose markup can change under you, making your screenshots and results go stale), we'll use a small self-contained HTML page built specifically for this: it has a known, fixed set of accessibility problems baked in, on purpose, so every run — yours or mine — turns up the same violations. Setting Up the Demo Page Save the following as accessibility-demo.html. It's a small page with a header, a features section, a contact form, and a modal — with a handful of accessibility bugs planted throughout: a button with no accessible name, a low-contrast paragraph, an image missing alt, an out-of-order heading, an iframe with no title, and a form field with no associated label. HTML <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8" /> <title>A11y Demo – Playwright</title> <meta name="viewport" content="width=device-width, initial-scale=1" /> <style> body { font-family: system-ui, sans-serif; line-height: 1.5; } .skip-link { position: absolute; left: -999px; top: -999px; } .skip-link:focus { left: 8px; top: 8px; padding: 8px; background: #eee; } /* BAD: low contrast */ .low-contrast { color: #9a9a9a; background: #fff; } .custom-select { border: 1px solid #ccc; padding: 8px; width: 240px; margin: 12px 0; } .custom-select [role="option"][aria-selected="true"] { outline: 2px solid; } #carousel { margin: 12px 0; height: 60px; overflow: hidden; border: 1px dashed #999; } .slide { display: none; padding: 8px; } .slide[aria-hidden="false"] { display: block; } #modal[hidden] { display: none; } .modal-content { background: white; padding: 16px; border: 2px solid; max-width: 360px; } .sr-only { position: absolute !important; height: 1px; width: 1px; overflow: hidden; clip: rect(1px,1px,1px,1px); white-space: nowrap; } </style> </head> <body> <!-- Skip link (good) --> <a class="skip-link" href="#main">Skip to main content</a> <!-- Page landmarks --> <header role="banner"> <h1 id="site-title">A11y Demo App</h1> <nav aria-label="Main navigation"> <ul> <li><a href="#main">Home</a></li> <li><a href="#features">Features (bad contrast)</a></li> <li><a href="#contact">Contact form (labels?)</a></li> <!-- Link opens new tab without rel (bad) --> <li><a href="https://example.com" target="_blank">External (no rel)</a></li> </ul> </nav> </header> <!-- Decorative + non-decorative images --> <section aria-labelledby="hero-heading"> <h2 id="hero-heading">Hero</h2> <!-- good decorative --> <img src="https://via.placeholder.com/600x100" alt="" aria-hidden="true" /> <!-- missing alt (bad) --> <img src="https://via.placeholder.com/120x60" /> </section> <!-- Headings out of order (bad) --> <h4>Out-of-order heading</h4> <main id="main" role="main" tabindex="-1"> <section id="features" aria-labelledby="features-h2"> <h2 id="features-h2">Features</h2> <!-- Low contrast text --> <p class="low-contrast">This paragraph has poor color contrast.</p> <!-- Accordion (good) --> <div class="accordion"> <button aria-expanded="false" aria-controls="acc-panel-1" id="acc-btn-1"> What is accessibility? </button> <div id="acc-panel-1" role="region" aria-labelledby="acc-btn-1" hidden> Accessibility means inclusive experiences for all users. </div> </div> <!-- Custom select --> <div class="custom-select" role="listbox" aria-labelledby="fruit-label" tabindex="0"> <span id="fruit-label">Favorite fruit (custom)</span> <div role="option" aria-selected="true">Apple</div> <div role="option">Banana</div> <div role="option">Mango</div> </div> <!-- Carousel --> <div id="carousel" aria-roledescription="carousel" aria-label="Rotating promos"> <div class="slide" aria-hidden="false">Slide 1</div> <div class="slide" aria-hidden="true">Slide 2</div> <div class="slide" aria-hidden="true">Slide 3</div> </div> <!-- Table missing scope --> <table id="price-table" border="1"> <caption>Pricing</caption> <tr><th>Plan</th><th>Price</th></tr> <tr><td>Basic</td><td>$10</td></tr> <tr><td>Pro</td><td>$20</td></tr> </table> <!-- Iframe without title (bad) --> <iframe src="https://example.com" width="300" height="100"></iframe> <!-- Video without captions (bad) --> <video id="promo-video" controls width="320"> <source src="" type="video/mp4" /> Sorry, your browser doesn’t support embedded videos. </video> <!-- Button without accessible name (bad) --> <button id="icon-only"><span class="icon-star" aria-hidden="true">★</span></button> <!-- Duplicate IDs --> <div id="dup">First duplicate id</div> <div id="dup">Second duplicate id</div> <!-- Live region --> <div aria-live="polite" id="live-region" class="sr-only"></div> <!-- Modal --> <button id="open-modal">Open Modal</button> <div id="modal" role="dialog" aria-modal="true" aria-labelledby="modal-title" hidden> <div class="modal-content" tabindex="-1"> <h2 id="modal-title">Subscribe</h2> <label for="email">Email</label> <input id="email" type="email" /> <button id="subscribe">Subscribe</button> <button id="close-modal">Close</button> </div> </div> <!-- Contact form --> <section id="contact" aria-labelledby="contact-h2"> <h2 id="contact-h2">Contact</h2> <form> <div> <label for="name">Name</label> <input id="name" type="text" /> </div> <div> <!-- missing label --> <input id="phone" type="tel" placeholder="Phone (no label)" /> </div> <div> <label for="msg">Message</label> <textarea id="msg"></textarea> </div> <button type="submit">Send</button> </form> </section> </section> </main> <footer role="contentinfo"> <p>© Demo</p> </footer> </body> </html> Also available on GitHub. Here's what the app should look like: Serve it locally with any static server — VS Code's Live Server extension, or: TypeScript npx http-server . -p 5501 Setting Up Playwright With Axe-Core Step 1: Install Playwright and the Axe integration. TypeScript npm init playwright@latest npm install -D @axe-core/playwright Step 2: Build a reusable Axe fixture. Rather than repeating the same AxeBuilder configuration in every spec file, it's worth wrapping it once as a Playwright fixture. Save this as axe-test-fixture.ts: TypeScript import { test as base } from '@playwright/test'; import AxeBuilder from '@axe-core/playwright'; type AxeFixture = { makeAxeBuilder: () => AxeBuilder; }; export const test = base.extend<AxeFixture>({ makeAxeBuilder: async ({ page }, use) => { const makeAxeBuilder = () => new AxeBuilder({ page }) .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa']); await use(makeAxeBuilder); }, }); export { expect } from '@playwright/test'; Every test that imports from this file instead of @playwright/test directly gets a makeAxeBuilder() fixture that's already scoped to WCAG 2.0/2.1 A and AA rules. If you ever need to add an exclusion for a known, already-ticketed issue, you change it once, here, instead of hunting through every spec file that runs a scan. Step 3: Write the test. TypeScript import { test, expect } from './axe-test-fixture'; test('demo page should have no critical or serious accessibility violations', async ({ page, makeAxeBuilder, }) => { await page.goto('http://127.0.0.1:5501/accessibility-demo.html'); const results = await makeAxeBuilder().analyze(); const blockers = results.violations.filter( (v) => v.impact === 'critical' || v.impact === 'serious' ); if (blockers.length > 0) { blockers.forEach((violation) => { console.log(`\n[${violation.impact?.toUpperCase()}] ${violation.id}`); console.log(`Help: ${violation.helpUrl}`); violation.nodes.forEach((node) => { console.log(` Element: ${node.html}`); console.log(` Fix: ${node.failureSummary}`); }); }); } expect(blockers).toEqual([]); }); Run it with: TypeScript npx playwright test accessibility.spec.ts --reporter=list A quick walkthrough of what's happening, since a chain of methods can look denser on the page than it is in practice: page.goto() loads the demo page in a real browser context. makeAxeBuilder().analyze() runs the scan and returns an object with violations, passes, incomplete, and inapplicable arrays. The filter() call is where the real decision gets made — it separates "bad enough to fail the build" from minor/moderate issues that most teams track separately rather than gate CI on. Because this page has deliberate bugs, the test is expected to fail. That's the point — it proves the scan actually works, before you point it at a real page where you don't already know the answer. What You'll See When It Fails Running this against the demo page surfaces violations like these, since they're planted on purpose: Button without an accessible name – <button id="icon-only"><span aria-hidden="true">★</span></button> needs either visible text or an aria-label.Insufficient color contrast – the low-contrast paragraph fails the 4.5:1 ratio required for normal text.Iframe missing a title – <iframe src="https://example.com"> has no title attribute, so a screen reader user has no idea what it contains.Heading order jump – the page goes from <h1> straight to <h4>, which breaks the document outline screen readers rely on.Image missing alt text – the placeholder image has no alt attribute at all.Form field with no label – the phone input relies on a placeholder instead of a real <label>, which disappears the moment the user starts typing. Since the test asserts expect(blockers).toEqual([]) and the page has several planted serious/critical issues, the test fails — and the console logging added in Step 3 prints each rule ID, a link to Deque's fix guidance, and the exact HTML node that triggered it, so a developer can go straight to the fix instead of parsing a JSON dump. The screenshot below shows a sample of the accessibility issues flagged during an actual test run: If you want to go deeper on any specific rule, Deque's rule descriptions explain the reasoning behind each one and how to resolve it. Scoping a Scan to One Section Scanning an entire page isn't always what you want — especially with a third-party embed or a section someone else owns. AxeBuilder supports .include() and .exclude() for exactly this: TypeScript test('contact section only', async ({ page, makeAxeBuilder }) => { await page.goto('http://127.0.0.1:5501/accessibility-demo.html'); const results = await makeAxeBuilder().include('#contact').analyze(); expect(results.violations).toEqual([]); }); This scopes the scan to just the contact form and ignores everything else on the page — useful when you want a fast, targeted check on the one section you're actively fixing. Wiring It Into CI None of this is worth much if it only runs on your laptop. Since it's a normal Playwright test, it drops into whatever CI you're already using without a separate accessibility dashboard to maintain: JavaScript name: Run accessibility tests run: npx playwright test accessibility.spec.ts --reporter=list Summary Automated accessibility testing with Playwright and Axe-Core won't catch everything a real user with a screen reader or a keyboard-only workflow would notice — that's still on manual testing. What it will do is catch the well-defined, common issues (contrast, labels, alt text, ARIA attributes, heading order) reliably, on every build, without anyone remembering to run a manual check first. A reusable fixture keeps your Axe configuration in one place instead of scattered across spec files. Filtering by impact level keeps your CI gate focused on what actually matters. And testing against a page with known, planted issues — before pointing the same setup at a real page — is a good way to prove the scan works at all. Treat this as a baseline, not a finish line. Pair it with periodic manual testing, and it holds up. Happy testing!

Here's the demo that always works: you point a notebook at a vector index, ask it a question, and it answers perfectly. Everyone claps. Three weeks later, the same system tells a customer that your refund window is 90 days when it's 30, cites a document that doesn't exist, and occasionally surfaces another tenant's invoice in the context. Nobody clapped for that part. RAG is deceptively easy to stand up and genuinely hard to keep honest. The retrieval step looks like a solved problem — embed the query, find the nearest neighbors, stuff them into a prompt — so teams treat it like plumbing and move on. Then quality quietly erodes, and because there's no eval harness, nobody can say when it broke or why. I've watched more RAG projects die from unmeasured drift than from any modeling problem. This article is a tour of the failure modes you will actually hit on Databricks Vector Search, each with the symptom, the root cause, and the specific fix. It's opinionated on purpose. Retrieval quality is not vibes. Pitfall 1: Chunking Like You’re Slicing Bread Chunking is the most ignored, highest-leverage knob in the whole pipeline. The lazy default is a fixed 1000-character window with zero overlap, applied to everything from API reference pages to legal contracts. It feels reasonable. It is not. Two failure shapes show up. Chunks too big: You embed a 4,000-token wall of text, the embedding becomes an average of six unrelated topics, and cosine similarity goes mushy — every query is sort of close to everything. Chunks too small: You split mid-sentence, the retriever returns ...the maximum is, and the model confidently invents the rest. The worst version is splitting on raw character count straight through a table or a code block, so the header lands in chunk 7 and the values land in chunk 8, and neither is useful alone. Fix it by chunking on structure first and size second. Split on headings and paragraph boundaries, keep tables and code blocks intact as their own chunks, and add a modest overlap so a thought that straddles a boundary survives in at least one chunk. Match the chunk size to your embedding model's real context window — databricks-gte-large-en handles 8,192 tokens, databricks-bge-large-en only 512, so a 1,000-token chunk silently truncates under bge and you embed half a paragraph. Python from langchain_text_splitters import RecursiveCharacterTextSplitter # Split on structure first (headings, paragraphs, lines), size second. # Tokens, not characters — match the embedding model's real window. splitter = RecursiveCharacterTextSplitter( separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " "], chunk_size=800, # tokens, comfortably under gte-large-en's 8192 chunk_overlap=120, # ~15% overlap so straddling thoughts survive length_function=lambda t: len(tokenizer.encode(t)), ) # Keep atomic blocks whole: don't let a table header and its rows split. def chunk_doc(doc): chunks = [] for block in split_into_blocks(doc): # your structure-aware pass if block.kind in ("table", "code"): chunks.append(block.text) # never split these else: chunks.extend(splitter.split_text(block.text)) return chunks Tip: There is no universal chunk size, but there is a universal debugging move: when an answer is wrong, read the retrieved chunks first. Half the time the model did its job, and the chunk was garbage. You can't fix that in the prompt. Pitfall 2: Embedding the Query and the Index With Different Brains This one is subtle because nothing errors. You built the index six months ago with databricks-bge-large-en. Last sprint, someone wrote a new query path and reached for databricks-gte-large-en because it was top of mind. Both output 1024-dimensional vectors, so the dimensions match, the similarity_search call succeeds, and the results are quietly nonsense — you're comparing coordinates from two different vector spaces. Same dimension count, completely different geometry. The cousin of this bug: you re-embed your corpus with a better model but only rebuild half the index, or you bump the embedding endpoint to a new version and forget the query side. Now your live queries are embedded with v2 and three-quarters of your index is still v1. Recall craters, and there's no exception to point at. The cure is to stop letting the embedding model be an implicit choice scattered across the codebase. Pin it once, centrally, and — the cleanest option on Databricks — use a Delta Sync index with managed embeddings so Vector Search owns embedding generation for both the index and query_text lookups. You physically cannot mismatch them, because you never embed the query yourself. Python from databricks.sdk import WorkspaceClient w = WorkspaceClient() # Managed-embedding Delta Sync index: Vector Search embeds BOTH the # source column AND query_text with the SAME endpoint. Mismatch impossible. EMBEDDING_ENDPOINT = "databricks-gte-large-en" # pin once, here, only here w.vector_search_indexes.create_index( name="prod.rag.kb_index", endpoint_name="rag-endpoint", primary_key="chunk_id", index_type="DELTA_SYNC", delta_sync_index_spec={ "source_table": "prod.rag.kb_chunks", "embedding_source_columns": [ {"name": "content", "embedding_model_endpoint_name": EMBEDDING_ENDPOINT} ], "pipeline_type": "TRIGGERED", "columns_to_sync": ["chunk_id", "content", "doc_id", "tenant_id", "updated_at"], }, ) # At query time you pass TEXT, never a vector. Same model embeds it server-side. res = w.vector_search_indexes.query_index( index_name="prod.rag.kb_index", columns=["chunk_id", "content", "doc_id"], query_text="What is the refund window?", num_results=5, ) Watch Out: If you must use self-managed embeddings (embedding_vector_columns), treat the model name and version as part of your schema. Write it into a column on the source table, assert it at query time, and re-embed the whole corpus on any change — not the half you remembered. Pitfall 3: The Index That Never Updates (You Forgot Change Data Feed) Symptom: You edit a document, the source Delta table clearly has the new text, you trigger a sync, the sync reports success — and the retriever still serves the old answer. People burn a full afternoon on this one. The endpoint is healthy, the index says ONLINE, nothing is red. It's just stale. Root Cause: A Delta Sync index syncs incrementally off the source table's Change Data Feed. If CDF was never enabled on the table, there's no change stream for the sync pipeline to read, so it has nothing to apply. Depending on how the table was created, you either get a hard error at index-create time or, worse, a sync that completes against an empty changelog and updates nothing. Either way, stale results. SQL -- The fix is one table property. Enable it on the SOURCE table before -- (or right after) you create the delta-sync index. ALTER TABLE prod.rag.kb_chunks SET TBLPROPERTIES (delta.enableChangeDataFeed = true); -- New tables: bake it in at creation so this never bites you. CREATE TABLE prod.rag.kb_chunks ( chunk_id STRING, doc_id STRING, tenant_id STRING, content STRING, updated_at TIMESTAMP ) TBLPROPERTIES (delta.enableChangeDataFeed = true); After enabling CDF, trigger the sync explicitly if you're on a TRIGGERED pipeline — it does not auto-run on source writes. This is the other half of the staleness story. People assume TRIGGERED means "sync when the table changes." It means "sync when you call sync." If you need the index to track writes automatically, that's CONTINUOUS. Python # TRIGGERED pipelines do NOT auto-sync. You call it, then poll until ready. w.vector_search_indexes.sync_index(index_name="prod.rag.kb_index") idx = w.vector_search_indexes.get_index(index_name="prod.rag.kb_index") print(idx.status.ready, idx.status.indexed_row_count) # If indexed_row_count never moves after edits -> check CDF on the source table. Pipeline type Sync Behavior Cost Use When TRIGGERED Syncs only when you call sync_index() Lower — compute runs on demand Batch refreshes, nightly doc loads, cost-sensitive CONTINUOUS Auto-syncs as the source table changes Higher — pipeline always on Live freshness, docs that change through the day Note: CONTINUOUS indexes cannot be manually synced — calling sync_index() on one raises an error. If your debugging instinct is "just hit sync again," check which pipeline type you actually have first. Pitfall 4: Plausible-But-Wrong Context, No Filtering, and Cross-Tenant Leakage Vector search always returns something. Ask about a product you don't sell, and you'll still get five neighbors back, ranked by similarity, looking authoritative. The model then dutifully grounds its answer in those five irrelevant chunks and produces a fluent, specific, completely wrong reply. This is the hallucination people blame on the LLM when the real culprit is retrieval handing it bad context with a straight face. The dangerous version is multi-tenant. If your index holds documents for many customers and you query without a tenant filter, nearest-neighbor search does not care about ownership — it'll happily return tenant B's contract to tenant A because it's semantically close. That's not a quality bug, but a data-leak incident. I have seen this ship to production because the filter was "on the backlog." Fix it on two fronts. First, always filter by the metadata that scopes the request — tenant, document type, recency — so the candidate set is correct before similarity even runs. Note the syntax differs by endpoint type: Standard endpoints take dict-style filters_json; Storage-optimized endpoints take SQL-like string filters via the databricks-vectorsearch client. Second, set a similarity floor: if the best match is below a threshold, treat it as "no relevant context found" and have the model say so instead of grounding on noise. Python from databricks.vector_search.client import VectorSearchClient vsc = VectorSearchClient() index = vsc.get_index(endpoint_name="rag-endpoint", index_name="prod.rag.kb_index") # Storage-Optimized endpoint: SQL-like string filters. # tenant_id is NON-NEGOTIABLE — it scopes the candidate set before ANN runs. resp = index.similarity_search( query_text=user_query, columns=["chunk_id", "content", "doc_id"], num_results=8, filters=f"tenant_id = '{tenant_id}' AND doc_type IN ('policy','faq')", ) rows = resp["result"]["data_array"] # last column of each row is the score # Similarity floor: refuse to ground on weak matches instead of hallucinating. SIM_FLOOR = 0.72 grounded = [r for r in rows if r[-1] >= SIM_FLOOR] if not grounded: answer = "I don't have a document that answers that." # honest > fluent else: answer = generate(user_query, context=grounded) Watch Out: Never interpolate tenant scope into a filter string from raw user input — derive tenant_id from the authenticated session, server-side. A tenant filter the user can override is not a tenant filter. Better still, enforce isolation in Unity Catalog with row-level policies on the source table so the index can only ever sync rows the caller may see. Pitfall 5: Stuffing the Whole Corpus Into the Context Window Bigger context windows tempted everyone into a bad habit: "retrieval is fuzzy, so just send top-20 and let the model sort it out." Two things go wrong. You pay for — and wait on — thousands of tokens of mostly irrelevant text on every call. And you walk straight into lost-in-the-middle: models reliably attend to the start and end of a long context and skim the middle, so the one chunk that actually answered the question — sitting at position 11 of 20 — gets ignored. The right answer was in the prompt. The model never read it. More retrieved chunks is not more knowledge; past a point it's more noise and worse recall of what matters. Retrieve a wider candidate set if you like, but then rerank and trim to a tight, high-precision few, and order them so the strongest land where the model actually looks. Python # Retrieve wide, then rerank and KEEP FEW. Quality over volume. candidates = index.similarity_search( query_text=user_query, columns=["chunk_id", "content"], num_results=20, filters=f"tenant_id = '{tenant_id}'", )["result"]["data_array"] reranked = reranker.rank(user_query, [c[1] for c in candidates]) # cross-encoder top = reranked[:5] # trim hard # Lost-in-the-middle hedge: put the strongest chunk LAST (nearest the question). ordered = sorted(top, key=lambda c: c.score) # ascending -> best at end context = "\n\n---\n\n".join(c.text for c in ordered) prompt = f"Use only the context below.\n\n{context}\n\nQuestion: {user_query}" Tempting move What it actually does Do this instead Send top-20 chunks Lost-in-the-middle; high token cost; recall drops Retrieve wide, rerank, keep top 3–5 No reranking ANN order ≠ relevance order Cross-encoder rerank the candidate set Random chunk order Best chunk buried in the middle Put strongest chunk at the edges Raw chunk dump Model can't tell sources apart Delimit chunks; cite doc_id in the answer Pitfall 6: “We Never Measured It” This is the one that actually kills projects. Every pitfall above is survivable if you can see it. The fatal mistake is shipping RAG with no evaluation harness, so quality becomes a matter of opinion, and the loudest anecdote wins. Someone says "it feels worse since the re-embed," someone else says "works for me," and there's no number to settle it. You can't improve what you refuse to measure. On Databricks, the harness is mlflow.genai.evaluate() with built-in LLM-judge scorers. The two that matter most for RAG live exactly at the failure modes above: RetrievalGroundedness checks whether the answer is actually supported by the retrieved chunks (catches Pitfall 4's confident fiction), and RelevanceToQuery checks whether the answer addresses the question at all. Add Correctness when you have ground-truth expected_facts. These are real judges, not heuristics — they read the trace and reason about it. Python import mlflow from mlflow.entities import SpanType from mlflow.genai.scorers import RetrievalGroundedness, RelevanceToQuery, Correctness mlflow.set_tracking_uri("databricks") mlflow.set_experiment("/Shared/rag-eval") # RetrievalGroundedness needs a RETRIEVER span in the trace — so trace retrieval. @mlflow.trace(span_type=SpanType.RETRIEVER) def retrieve(query, tenant_id): rows = index.similarity_search( query_text=query, columns=["chunk_id", "content"], num_results=5, filters=f"tenant_id = '{tenant_id}'", )["result"]["data_array"] return [{"page_content": r[1], "metadata": {"chunk_id": r[0]} for r in rows] @mlflow.trace def rag_app(query, tenant_id): docs = retrieve(query, tenant_id) return {"response": generate(query, docs)} # A small, curated eval set with ground truth beats a big unlabeled one. eval_data = [ {"inputs": {"query": "What is the refund window?", "tenant_id": "acme"}, "expectations": {"expected_facts": ["Refunds are accepted within 30 days"]}, {"inputs": {"query": "Do you support SSO?", "tenant_id": "acme"}, "expectations": {"expected_facts": ["SAML and OIDC single sign-on are supported"]}, ] results = mlflow.genai.evaluate( data=eval_data, predict_fn=rag_app, scorers=[RetrievalGroundedness(), RelevanceToQuery(), Correctness()], ) print(results.metrics) # now "feels worse" becomes a number that moved Run this on every change — new chunking strategy, new embedding model, new reranker — as a regression gate, not a one-time blessing. When a metric drops, MLflow Tracing tells you where: open the failing trace, look at the RETRIEVER span, and read what actually came back. The debugging loop is tight: bad answer → inspect retrieved chunks in the trace → was the right chunk even retrieved? If no, it's a retrieval problem (chunking, embedding, filter, staleness). If yes but the answer ignored it, it's a generation problem (context order, prompt, lost-in-the-middle). Metric What it tells you How to get it Retrieval groundedness Is the answer supported by retrieved chunks? RetrievalGroundedness() scorer (needs RETRIEVER span) Relevance to query Does the answer address the question? RelevanceToQuery() scorer Correctness Does it match known facts? Correctness() scorer + expected_facts Context recall Did retrieval find the chunk that holds the answer? Compare retrieved chunk_ids vs labeled relevant ids Context precision What fraction of retrieved chunks are relevant? Custom @scorer over the RETRIEVER span Retrieval latency Is the retrieve step the bottleneck? Span duration in the trace Pitfall 7: Treating Metadata and Governance as Someone Else’s Job The last pitfall is architectural. Teams flatten everything into (chunk_id, content) and throw away the metadata — doc_id, tenant_id, doc_type, updated_at, source URL. Then they can't filter (Pitfall 4), can't cite sources, can't expire stale docs, and can't answer the auditor who asks "why did the model say that?" because there's no path back from an answer to the document it came from. Carry metadata through the whole pipeline and put governance underneath it. Keep the source table in Unity Catalog's three-level namespace (catalog.schema.table), include the columns you need to filter and cite in columns_to_sync, stamp updated_at, and govern access on the source — the index inherits what the table exposes. The payoff compounds: the same tenant_id that prevents leakage also powers citations, recency filters, and lineage. Metadata is not overhead; it's the thing that makes retrieval auditable. Pitfall Symptom Root Cause Fix Bad chunking Mushy similarity or truncated answers Fixed-size splits ignore structure/model window Structure-aware splitter, overlap, size to model Embedding mismatch Nonsense results, no error Query and index embedded by different models/versions Managed-embedding delta-sync; pin model centrally Index staleness Edits don't show up after sync CDF off; or TRIGGERED never synced enableChangeDataFeed=true; sync_index() or CONTINUOUS Plausible-but-wrong/leakage Confident wrong answers; other tenant's data No metadata filter; no similarity floor Server-side tenant filter; similarity threshold Context overstuffing Slow, costly, ignores the right chunk Top-20 dump; lost-in-the-middle Rerank, trim to 3–5, order by edge position No evaluation "Feels worse" debates, silent drift Shipped with no eval harness mlflow.genai.evaluate as a regression gate Ignored metadata Can't filter, cite, or audit Flattened to id+text; no governance Carry metadata; govern source in Unity Catalog The Takeaway None of these failure modes are exotic. They're the default outcome of treating RAG as plumbing — chunk however, embed whatever, sync if you remember, send a pile of context, and hope. The fix in every case is the same posture: make retrieval explicit and measurable. Chunk on structure. Pin one embedding model and let managed delta-sync enforce it. Enable change data feed before you wonder why nothing updates. Filter by tenant server-side and refuse weak matches. Rerank and trim instead of dumping. And above all, wire up mlflow.genai.evaluate() with RetrievalGroundedness and RelevanceToQuery so "it got worse" becomes a number, and use MLflow Tracing to find out exactly which span betrayed you. If you can't open a trace and read the chunks your model was handed, you're not debugging RAG — you're guessing. Start small: stand up a Delta Sync index with managed embeddings, put twenty labeled questions behind mlflow.genai.evaluate(), and make that eval a gate on every change. The Databricks Vector Search and MLflow GenAI evaluation docs walk through both end-to-end. Build the harness before you build the features — your future self, staring at a confidently wrong answer at 4 p.m., will thank you.

A federated gateway provides secure, policy-aware access to tool servers. The thing that made me stop and rethink our whole approach to agentic tooling was a text file. An engineer on one of our platform teams had wired an AI coding assistant up to our internal source control. To do it, they had pasted a personal access token into a local MCP server config in their home directory. It worked. That also meant a long-lived credential with broad repository scope sat in plaintext in a file the agent could read, on a laptop, with no audit trail and no expiry. Multiply that by every engineer who wants their assistant to see internal code, artifacts, docs, and warehouse tables, and you have hundreds of copies of your crown-jewel credentials distributed across endpoints you do not control. That is the real problem with Model Context Protocol adoption in an enterprise. MCP itself is a good protocol. The failure mode is topological: the default deployment story puts the server, the credentials, and the client on the same machine, which is exactly where you least want them in a network-isolated environment. What we built instead was a federated control plane. One gateway, many backend tool servers, and a thin local connector that holds no secrets at all. The Three-Hop Topology The pattern is simple to state, and most of the engineering effort goes into the seams: Plain Text Connector -> Gateway -> Server The connector runs locally next to the IDE or agent. It speaks stdio to the client, because that is what most assistants expect, and streamable HTTP outbound to the gateway. It is deliberately dumb. It knows one URL and how to complete a browser-based login. It stores no client secret, no API key, no PAT. The gateway is the control plane. It terminates authentication, brokers OAuth on the user's behalf, resolves which backend server should handle a given request, enforces policy, and emits telemetry. It is the only component that ever touches a credential. The backend servers are the actual MCP implementations: source control, artifact repository, documentation search, static analysis, browser automation, warehouse metadata. Each is a separate deployment with its own least-privilege identity. They live in-cluster, on the internal network, with no default egress to the public internet. The property that matters is that the trust boundary sits at the gateway, not at the laptop. A compromised developer machine yields a session, not a credential. The Gateway as an OAuth Broker This is the part people underestimate. The gateway does not proxy the user's token; it exchanges an authenticated session for a narrowly scoped downstream credential, per backend, per request. Concretely, when a request arrives, the gateway resolves the caller's identity from the session, looks up the target server, and mints or fetches a downstream token with only the scopes that server is registered to need: Python async def broker(request: MCPRequest, session: Session) -> MCPResponse: server = registry.resolve(request.server_id) if server is None: raise PolicyError("unregistered_server") if not policy.allows(session.principal, server, request.method): audit.deny(session.principal, server.id, request.method) raise PolicyError("not_permitted") # Client secrets are held by the gateway only; never sent downstream # to the connector and never written to a client-side config. token = await broker_pool.token_for( principal=session.principal, provider=server.auth_provider, # e.g. saml_scm, google scopes=server.least_privilege_scopes, # e.g. ["repo:read"] ttl_seconds=900, ) return await transport.forward(server, request, bearer=token) Two design choices are worth calling out. First, least_privilege_scopes is a property of the registered server, not of the user's login. A developer authenticating once through the gateway does not thereby grant every backend the union of their permissions. A documentation server gets read scope on docs and nothing else, even if the same human has admin rights elsewhere. Second, we deliberately started with a static client registration model backed by the platform's own secret store, with a migration path to Dynamic Client Registration. DCR is where this should end up, but shipping a working broker with rotating short-lived tokens beat waiting for the spec ecosystem to settle. Secrets are created by CI/CD from a managed secret store; no human hands a production secret to a running workload. Guardrails Against Tool Poisoning Once agents can call tools, tool descriptions become an attack surface. A malicious or compromised server can return a tool definition whose description instructs the model to exfiltrate context, or can silently mutate a description after initial approval. Rate limiting alone does not help here. We enforce validation at the gateway in both directions of the exchange: Python POISON_PATTERNS = [ r"ignore (all )?(previous|prior) instructions", r"do not (tell|inform|mention to) the user", r"<\s*(system|assistant)\s*>", ] def validate_tool_manifest(server_id: str, manifest: dict) -> None: for tool in manifest["tools"]: blob = f"{tool['name']} {tool.get('description', '')}" for pattern in POISON_PATTERNS: if re.search(pattern, blob, re.IGNORECASE): quarantine(server_id, tool["name"], reason=pattern) raise PolicyError("suspect_tool_description") # Descriptions are pinned at review time. Drift requires re-approval. if sha256(blob) != registry.approved_digest(server_id, tool["name"]): raise PolicyError("manifest_drift") The digest pinning is the load-bearing control. Pattern matching catches the naive cases; pinning catches the case where an approved server changes its behavior after review. Any drift takes the tool out of rotation until a human re-approves it. On top of that: per-principal and per-server rate limits, an explicit allow/block list of methods, and argument validation before forwarding. We mapped these controls to published guidance for AI system risks so the security review had something concrete to assess rather than a narrative. Observability Is Not Optional Here When something goes wrong in an agentic workflow, the user's report is usually "the assistant got confused." That is not debuggable. Centralizing traffic through one gateway means you get, for free, the telemetry that makes it debuggable: latency percentiles per server and per method, error rates by status code, MCP method distribution, transport breakdown between stdio and streamable HTTP, and per-principal activity. Two things surfaced from that data that we would never have found otherwise. One backend was returning successful responses with empty payloads for a large share of calls, which looked healthy on an error-rate dashboard and terrible to users. And tool usage was heavily concentrated: a small number of servers and a small number of engineers accounted for most traffic, which told us where to spend reliability effort instead of guessing. Making It Self-Service, or It Dies A control plane that requires a platform engineer in the loop becomes the bottleneck it was meant to remove. The onboarding path we settled on is a scaffolded repository from an internal portal, image build and promotion through CI, infrastructure-as-code deployment via pull request, automated vulnerability scanning, and auto-registration into the gateway registry on merge. New server idea to registered production service is one pull request and two approvals. The lesson I would pass on: solve the credential topology first, then the ergonomics. Teams that start with developer convenience end up retrofitting security onto a distributed pile of local configs, and that retrofit is far more expensive than getting the trust boundary right on day one.

For decades, API design rested on a reassuring assumption: given valid input and a stable dependency, software should return a predictable result. Large language models break that assumption without breaking the API. A request can receive HTTP 200, perfectly valid JSON, and a confidently wrong answer. That distinction matters. The network contract may still be deterministic, but the semantic contract is now probabilistic. A model endpoint does not promise one correct output; it samples a likely output from a distribution shaped by the prompt, context, model version, retrieval results, and decoding process. Machine-learning engineer Chip Huyen puts it plainly: “LLMs are stochastic; there’s no guarantee that an LLM will give you the same output for the same input every time.” Reliable AI systems begin when developers stop treating that behavior as an exception. Redefine What the Contract Guarantees The traditional contract defines fields, types, status codes, and errors. The AI contract, however, has to specify the acceptable behavior: what kind of evidence the model can accept, what failure categories are allowed, when to abstain, what latency and costs budget are in place, and how to proceed in case there is not enough confidence. Confidence has to be measured using evidence coverage, validation outcomes, or classifier calibration, but not the self-assessment of the model. Thus, the goal changes from "function returns correct value" to something measurable: for a certain slice of traffic, the system passes some quality criteria at an acceptable frequency. Different tasks require different criteria. For example, summarizing movies does allow for some awkward sentences. Changing a customer's credit limit does not allow any inventions or ambiguities. Put a Deterministic Envelope Around the Model The model should be one component inside ordinary software, not the authority at the center of it. The surrounding application should normalize inputs, constrain outputs, validate results, and choose whether to accept, retry, fall back, or escalate. Structured generation is the first layer. Use a JSON Schema, enums, required fields, and explicit null states instead of asking for “JSON” in a prompt. OpenAI, for example, reported 100% schema adherence for one model in its complex JSON Schema evaluation. That solves a parsing problem, not a truth problem. A fabricated invoice number can still be a perfectly valid string. Semantic validation must follow structural validation. Check identifiers against source systems, dates against business rules, citations against retrieved passages, and calculated values with deterministic code. Treat every generated field as untrusted input. The control flow should be explicit: Plain Text generate -> validate schema -> verify evidence -> apply policy -> accept | bounded retry | fallback | human review The important output is not merely the model’s answer. It is a typed system decision such as accepted, rejected, or needs_review, accompanied by evidence and a machine-readable failure reason. Keep Side Effects Behind a Transaction Boundary Probabilistic text becomes dangerous when it can directly create a refund, delete a record, or send a message. Separate proposing an action from committing it. Let the model select only from allow-listed tools and produce typed arguments. Then let deterministic code authenticate the user, authorize the operation, verify current state, and enforce limits. Add idempotency keys so a retry cannot repeat a payment or ticket creation. For high-impact actions, show a preview or require human approval. This architecture also limits prompt injection. Untrusted content may influence a proposal, but it should never grant the model new permissions. Make Retries a Policy, Not a Reflex Retries can repair malformed output or a transient timeout. They can also multiply cost, latency, and side effects while reproducing the same semantic error. Set a small attempt budget and retry only failures that may be recoverable. Feed validation errors back in a structured form, use exponential backoff for provider faults, and stop when the remaining time or token budget is insufficient. If the evidence is missing, another generation is not a remedy; retrieval, clarification, or abstention is. Fallbacks should match the risk. A smaller model, cached result, or rules engine may preserve availability. A safe refusal or human queue may be the correct degraded mode when correctness matters more than speed. Test Distributions, Not Favorite Prompts A handful of convincing demos proves little. Build an evaluation set from real tasks, known edge cases, adversarial inputs, and failures observed in production. Run each important case multiple times when sampling variability matters, and report pass rates with confidence intervals rather than one aggregate score. Evaluation practitioners Hamel Husain and Shreya Shankar offer excellent advice: “Start with error analysis, not infrastructure.” Review traces with domain experts, classify concrete failures, then automate the checks that matter. Prefer deterministic assertions for schema, policy, and executable code; reserve model-based judges for qualities that rules cannot capture, and calibrate those judges against human labels. Version the entire behavior-producing system: model identifier, prompt, schema, retrieval corpus, tool definitions and safety rules. Run regression suites and canary traffic before changing any of them. In production, monitor validator failures, abstentions, retries, latency, cost, and user corrections. Store redacted traces where privacy permits, because averages alone rarely explain why a system failed. Reliability Moves Outward The model does not need to become deterministic for the product to become dependable. Databases still fail, networks still partition, and users still submit hostile input; engineering makes those systems useful by containing uncertainty. Generative AI demands the same discipline, applied at the semantic boundary. Define measurable behavior, constrain the output, verify claims, isolate side effects, test continuously, and fail safely. Google’s site reliability literature opens with a durable warning: “Hope is not a strategy.” With probabilistic APIs, it is not a contract either.