DZone Spotlight

Recap: Where This Fits Part 1 of this series covered exact-match caching: hashing the full request and returning a stored response when a new request matches an old one exactly. That layer is fast and safe, but it only helps when the input matches exactly, which is common for templated traffic and rare in open conversation. This article covers the layer that closes that gap: semantic caching. Part 3 covers what happens once a request actually reaches the model: self-hosted KV cache reuse and provider prefix caching, plus the production details: watching your metrics, working out your costs, and a full checklist. 2. Semantic Caching When a user asks the same thing in different words ("I am going from Bangalore to Mysore" vs. "I'm traveling Bangalore to Mysore and need hotels"), exact match fails. Semantic caching embeds the query and finds the nearest earlier response, and returns it if the two are similar enough. How It Works Embed the new query with a fast embedding model.Search a vector store, such as Redis Stack with RedisVL, pgvector, FAISS, or Milvus, for the closest matches.Use cosine similarity, the most common measure. Typical thresholds in production sit between 0.90 and 0.95.On a hit, return the stored response. On a miss, call the model and store both the embedding and the response. Trade-Offs Loss of detail. Embeddings compress meaning into numbers. "Laptop under $1000" and "laptop under $3000" can land close together in that number space, even though the right answer for each is different.Latency. A vector search, even a fast approximate one (HNSW), is slower than a simple hash lookup.Cost. Embedding a query is cheap and quick next to a full model call, but it is not free, and at high volume the embedding step becomes its own cost to plan for.Wrong answers. False positives are the main risk here. Reduce this risk with a stricter threshold, an embedding model tuned to your domain, a check after retrieval, or a mix of exact and semantic keys. Picking an Embedding Model The model you embed with has a direct effect on how many false positives you get, and just picking whatever is quickest to install is a common early mistake: General-purpose open models (sentence-transformers/all-MiniLM-L6-v2, all-mpnet-base-v2) are free, run on your own machines, and are fine for a first prototype. They struggle with domain-specific terms, product names, and short queries where one small word changes the meaning.Domain-tuned or retrieval-specific models (Redis's own redis/langcache-embed-v2, or a model tuned on your own past queries) usually tell near-miss pairs apart better, at the cost of an extra dependency, and, for hosted embedding APIs, a small per-call cost.Hosted embedding APIs (OpenAI, Cohere, Voyage) trade a small per-call cost for strong general performance, with no local GPU or model to manage. Most teams move to this once they are past the prototype stage. Whichever you pick, test it against your own real traffic instead of trusting a leaderboard. General embedding quality and "tells apart this exact near-miss pair in my domain" are not the same test. Python from redisvl.extensions.cache.llm import SemanticCache from redisvl.utils.vectorize import HFTextVectorizer # or OpenAIEmbeddings, etc. vectorizer = HFTextVectorizer(model="sentence-transformers/all-MiniLM-L6-v2") # or a stronger, domain-tuned model cache = SemanticCache( name="llm_semantic", redis_url="redis://localhost:6379", distance_threshold=0.1, # lower = stricter, this is cosine distance, not similarity vectorizer=vectorizer, ttl=3600 ) def cached_llm_call(prompt: str, llm_callable): hits = cache.check(prompt=prompt) if hits: return hits[0]["response"] # consider a cheap check here before trusting this blindly response = llm_callable(prompt) cache.store(prompt=prompt, response=response) return response Check the RedisVL docs for the current parameter names before you ship. Libraries like this change fast, and a name can shift between minor versions. One easy trap in the code above: distance_threshold is cosine distance, not cosine similarity, and the two run in opposite directions. As a rule of thumb, cosine distance is roughly 1 minus cosine similarity, so a similarity threshold of 0.92 is close to a distance threshold of 0.08. Reading distance_threshold=0.1 as "90% similar" instead of doing that conversion is a common way a threshold ends up looser or stricter than the team meant it to be. GPTCache follows the same embed, search, check, store pattern, and can sit in front of any OpenAI-compatible client if you would rather not build this loop yourself. Shipping Semantic Caching Safely: Threshold Plus Verification A raw similarity threshold, on its own, means "we tried semantic caching," not "we shipped it safely." The gap between those two is usually one cheap check placed between the vector hit and the return. Three patterns cover most production systems: High threshold, plus a cheap second check. Set the similarity bar high (0.94 to 0.97, or the matching distance value), then run the pair (new query, cached query) through a small, fast model or a rule-based check that answers one question: "do these two queries really want the same answer?" This costs a small fraction of a full model call and catches near-miss cases a threshold alone would let through.A verified semantic cache. Store a short note next to the cached answer when you write it, not just the answer itself; for example, a one-line summary of when the cached answer applies ("valid for laptop budgets between $900 and $1,100"). When there is a candidate hit, check the new query against that stored note before returning it, instead of trusting similarity alone.Fall back to a full call when confidence is low. Treat similarity as a sliding scale, not a strict yes or no, as the diagram above shows. Above a high threshold, return right away. In a middle band, run the check. Below that, treat it as a miss and call the model. This avoids the common mistake of picking one threshold and treating it as a perfect line, when it never really is. Python def cached_llm_call_verified(prompt: str, llm_callable, verifier_callable): hits = cache.check(prompt=prompt) if hits: candidate = hits[0] # Cheap check before trusting a vector-space match. if verifier_callable(prompt, candidate["prompt"], candidate["response"]): return candidate["response"] # Similarity said "close enough," verification disagreed. Treat as a miss. response = llm_callable(prompt) cache.store(prompt=prompt, response=response) return response verifier_callable can be as simple as a small classification model, a check against a second, differently-tuned embedding model, or a rule that checks specific details pulled out of both queries (price ranges, dates, entity names). The point is not to be fancy; it is to have a second, independent signal before a similarity score alone decides what a user sees. Semantic Cache Invalidation Is Harder Than It Looks TTL and LRU work well for exact matches, because a stale exact match is still at least a match for the same question. A stale semantic match is riskier: the cached answer can still look "close enough" to a new, slightly different question forever, because nothing about vector distance fades with time on its own. Practical fixes: Keep semantic TTLs shorter than exact-match TTLs by default, especially for anything tied to price, stock, or policy.Version the cache namespace whenever the underlying source of truth changes (a new product catalog, an updated policy page), instead of relying on TTL alone to catch it.Store a content hash or a last-updated time next to the cached response, and check it when reading, not just when writing. Multi-Turn Conversations Everything above treats a prompt as one standalone unit. Most production chat systems do not work that way. A conversation builds up history, and both the hashing and the embedding steps need to account for that: Exact match on multi-turn: Hash the full message history plus the new turn, not just the latest message. This means the exact-match hit rate on turn N of a conversation is close to zero unless the whole earlier conversation is replayed exactly, which rarely happens. Exact match is far more useful on the first turn of a session, or on isolated tool calls inside an agent loop, than on a deep conversation.Semantic match on multi-turn: Embedding only the latest user message loses the context that gives it meaning ("what about the second one" means nothing without the earlier turn). Common fixes are to embed a short summary of the running conversation state, or to embed the latest turn together with a summary of the relevant earlier turns. Neither one fully removes the risk described above, and that risk grows as the conversation gets longer.In practice, most teams apply exact and semantic caching to single tool calls and retrieval steps inside an agent loop, rather than to the full multi-turn exchange, and rely on model-side prefix reuse, covered in Part 3, for the conversation history itself. Streaming Responses If the model call streams tokens back to the client, the caching step cannot just wrap a normal, blocking call. Two common patterns: Buffer, then cache. Stream to the client as normal, build up the full response on the server, and write to the cache only once the stream is done. Simple, but a request that gets canceled mid-stream never gets cached, and a very long response delays the cache write.Cache and replay. On a cache hit, play back the stored response as a fake stream (in reasonable chunks) so the client's streaming UI still behaves as expected, instead of getting one big block of text. A UI built around token-by-token display looks broken if a cache hit suddenly returns the whole answer at once with no streaming pace. Some setups go a step further and cache partial answers: for a very long response, write the first N tokens to the cache as they come in, instead of waiting for the full stream, then return those N tokens right away on a hit while the rest streams live. This extra work is worth it mainly for long content, like reports or long code files, where the wait for the first token matters more to users than the total time to finish. Security Considerations Semantic cache poisoning. Because semantic caching returns a response based on similarity, not an exact match, an attacker who understands the embedding space can craft an input that lands close enough to a cached response that leaks sensitive information, gets around a guardrail, or returns a wrong but confident-sounding answer. High thresholds and a check after retrieval are the main defenses.One tenant seeing another tenant's data. In a multi-tenant app, a semantic cache without strict per-tenant filters can show one customer's cached response to a different customer's similar query. Filtering by tenant ID has to happen at the vector search step itself, not just in application code after the fact, or a bug in the app becomes a data leak instead of just a wrong answer. Part 3 covers the same risk at the model-serving layer.Personal data in cached responses. Any cached response that includes data tied to a specific person needs a retention and access policy at least as strict as the original response, since the cache is now a second copy of that data with its own TTL and its own access path. Testing Semantic Thresholds Before You Ship "Measure the false-positive rate" is easy to say and easy to skip. A simple process that works: Pull a sample of real (or realistic) query pairs from your own domain, marked by a person as "should share a cached answer" or "should not," including some deliberately tricky near-misses (same topic, different detail, like the laptop price example above).Run the sample through candidate embedding models at a few different thresholds, and check precision on the "should share" label. Optimize for precision here, not recall. A missed cache hit only costs latency. A wrong cache hit gives a wrong answer with full confidence.Re-run this test whenever the embedding model changes, and check it now and then against live traffic, since how real users phrase things shifts over time in ways a fixed test set does not catch. What's Next Exact and semantic caching skip the model call entirely on a hit, which is why they sit in front of everything else. But most requests still reach the model, and what happens there — self-hosted KV cache reuse on engines like vLLM and SGLang, or provider-side prefix caching on Anthropic, OpenAI, and Bedrock — is often the single biggest cost saver on repetitive workloads. Part 3 covers both, along with a provider pricing comparison, a worked cost example, the metrics that actually catch problems, and a full production checklist. Further Reading RedisVL documentation, semantic cache extension and vectorizersGPTCache project documentation

By Ram Ghadiyaram CORE

These are failure modes that actually page you — and the boring fixes that stop them. A Kafka consumer is a stateful client built on a membership protocol. It joins a group, accepts a partition assignment, processes records in order, and proves it is still alive on a fixed cadence of heartbeats and poll calls. Every guarantee the group offers — ordered processing, shared load, exactly one owner per partition — rests on that membership staying stable while the work flows through it. Kubernetes starts from the opposite assumption. A pod is disposable. The platform restarts pods, throttles CPU, rolls deployments, and reclaims nodes on a schedule the consumer group never agreed to, and each of those routine events reaches the group protocol as a member leaving, arriving, or going silent. Most consumer incidents on EKS are not Kafka bugs. They are the collision between a stateful, membership-based client and an orchestrator doing exactly what it was designed to do. In this post, I start with the baseline configuration that prevents half of these problems outright, then walk through the eight failure modes that account for nearly everything else — symptom, mechanism, and fix, which is usually a config line or a lifecycle hook rather than a redesign. I close with the design principle that ties the eight together and the debug routine that names a failure mode in minutes. The Core Tension A standalone consumer fails when its code fails or when its broker fails. A consumer on EKS carries a third failure source that never sleeps: the platform itself. Rescheduling a pod is normal platform behavior. Enforcing a CPU limit is normal platform behavior. Terminating a container during a rollout is normal platform behavior. The group protocol holds no category for these events except membership change, so it reacts to all of them with the tools it has — rebalances, session expirations, revoked partitions — whether the event was an accident or a planned deploy. That framing changes how the whole problem gets solved. A restart stops being an exception to recover from and becomes a permanent condition to design around: stable identities, generous timing, and a shutdown path the group can see coming. The eight modes below are what happens when a design skips that framing and assumes the laptop's stability on the platform's terms. Start With the Baseline Half of these issues never happen with a sane baseline: Properties files session.timeout.ms=10000 heartbeat.interval.ms=3000 max.poll.interval.ms=300000 max.poll.records=500 partition.assignment.strategy=CooperativeStickyAssignor enable.auto.commit=false auto.offset.reset=earliest Three decisions in that block do most of the work. Manual commits put offset progress under the application's control, so a restart replays records instead of silently skipping or duplicating them. Cooperative-sticky assignment keeps partitions with their current owners through a rebalance wherever it can, instead of revoking everything and starting the assignment from zero. A poll interval with real headroom gives slow processing room to finish before the group concludes the member has died. Everything below is what still goes wrong on top of that baseline — and what to do when it does. The Eight Failure Modes 1. Pod Restart and CrashLoop Symptoms: The pod keeps restarting; the consumer leaves and rejoins the group constantly. Cause: An application error or an OOMKill; a long startup; uncaught exceptions inside the poll loop. The poll loop is the consumer's heartbeat in the broad sense — the thread that polls is the thread that processes, so an exception that escapes it takes the whole member down, and the group sees a departure to rebalance around. Fix: Fix resource limits, handle exceptions in the loop, and commit sync or async with idempotent processing, so a restart replays safely instead of corrupting progress. Idempotency is the load-bearing half of that fix: a consumer that can replay a batch without harm treats a restart as a routine event, while a consumer that cannot turns every CrashLoop into a data problem on top of an availability problem. 2. Session Timeout — Heartbeats Missed Symptoms: Logs report "member has failed"; frequent rebalances; consumption stalls. Cause: GC pauses or CPU throttling starve the background heartbeat thread. The consumer is alive — it just cannot prove it in time. The heartbeat runs on its own thread precisely so that processing cannot block it. That separation holds only while the scheduler gives the thread its share of CPU; a throttled container breaks it quietly, and the first sign is the group expelling a member that was never down. Fix: Raise session.timeout.ms, fix CPU and memory limits so the heartbeat thread gets scheduled, and keep poll() being called regularly. Raising the timeout buys tolerance; fixing the limits removes the cause. The timeout increase without the limits change leaves a consumer that fails more slowly instead of one that stops failing. 3. max.poll.interval.ms Exceeded Symptoms: "Consumer is not polling" — the member is kicked out mid-processing. Cause: Processing per batch takes longer than the poll interval allows, usually because of slow database or API calls inside the loop. The poll interval is the group's second liveness check: it measures not whether the member exists but whether the member is still making progress through its assignment, and slow downstream calls stretch a single batch past the limit without anything being wrong with the consumer itself. Fix: Raise max.poll.interval.ms, shrink batches with max.poll.records, or hand work to worker threads with pause/resume so the poll loop keeps moving. The three options trade against each other — a longer interval tolerates slower batches, smaller batches bound the work per cycle, and worker threads decouple processing from polling at the cost of more moving parts. 4. Rebalance Storms Symptoms: The group spends more time reassigning partitions than processing records. Cause: Pods scaling up and down, rolling restarts, short timeouts, and an eager assignor that revokes everything on every change. A rebalance is not free: assignments pause while partitions move, in-flight work gets discarded or replayed, and a group that rebalances in a loop spends its capacity on coordination instead of consumption while lag climbs behind it. Fix: Use cooperative-sticky assignment, tune the HPA carefully, and stabilize the pod lifecycle — no random restarts. Cooperative assignment revokes only the partitions that actually need to move, and sticky ownership keeps the rest where they are, so a single pod event stops being a group-wide interruption. Autoscaling against a consumer group needs that softness underneath it, because every scale event is a membership event. 5. Offset Commit Failures Symptoms: CommitFailedException in the logs; records reprocessed after a restart. Cause: A stale generation after a rebalance, or the coordinator moved or is unreachable. A commit is a claim about progress made under a specific generation of the group, and the coordinator rejects claims from a generation that no longer exists — the consumer processed the records, and the group has already moved on without recording that fact. Fix: Commit only inside the poll loop, handle CommitFailedException and retry, and use commitAsync with callbacks carefully. Committing inside the loop keeps the commit attached to the generation that did the work. The reprocessing that follows a rejected commit is the group working as designed; the goal is making replays safe and rare. 6. Resource Limits — CPU and Memory Symptoms: High CPU throttling, long GC pauses, OOMKilled pods, missed heartbeats. Cause: Limits set too low for the batch sizes in use; JVM heap not matched to container memory. This mode sits underneath several of the others — throttling starves the heartbeat thread of mode 2, stretches the batch processing of mode 3, and turns the memory pressure of mode 1 into an OOMKill. A consumer's batch size is also a memory decision, since every record in the batch lives in the heap until the batch finishes. Fix: Set requests and limits thoughtfully, tune the JVM (-Xmx against container memory, not host memory), right-size message batches, and watch it all with Prometheus and Grafana. The host-memory mistake is the subtle one: a JVM sized for the node it lands on will be killed by a container limit it never knew existed, and the pod restart that follows presents as a consumer problem with a platform cause. 7. DNS and Bootstrap Servers Symptoms: UnknownHostException; the consumer cannot connect at startup; intermittent failures after that. Cause: Wrong bootstrap addresses, CoreDNS issues, or private DNS without the VPC setup to back it. This is the one mode on the list that fails before the group protocol is even involved — the consumer cannot reach a broker to join anything — and the intermittent version is worse than the hard failure, because it presents as flakiness instead of a name-resolution problem with a fixed address. Fix: Use the correct bootstrap DNS names, verify CoreDNS in the EKS cluster, and check VPC DNS settings and private hosted zones. Every layer of that chain is platform surface, and all of them have to agree before the consumer's own configuration gets a chance to matter. 8. Ungraceful Shutdown Symptoms: Every deploy triggers a full rebalance; partitions get reassigned the slow way. Cause: Kubernetes kills the pod with no chance to leave the group or commit offsets. From the group's side, a terminated member and a crashed member look identical — both simply stop responding — so the group waits out the session timeout and then reassigns everything, on every single deploy, even though the shutdown was planned in advance. Fix: Add a PreStop hook that calls close(), commit offsets before shutdown, and set terminationGracePeriodSeconds generously enough for the close to finish. A clean close() is the consumer announcing its departure: the group rebalances immediately and narrowly, offsets land before the pod dies, and a deploy stops looking like an outage. The grace period is the budget that makes the announcement possible, and it has to be long enough for the close to actually complete. Designing for the Platform None of these eight modes is exotic, and that is the point. Read together, they reduce to a short list of design commitments. Timing gets budgeted as a first-class resource, sized for a platform that throttles and pauses things rather than a quiet laptop. Shutdown becomes a feature with its own code path instead of an abrupt disappearance the group has to time out. Identity stays as stable as the platform allows, so sticky assignments have something to stick to. And processing is written to survive replay, because replays are the group's recovery mechanism and no configuration makes them fully disappear. The tradeoffs stay honest. Generous timeouts tolerate platform noise and also delay detection of a genuinely dead member. Cooperative-sticky assignment calms rebalances and still cannot hold a partition whose owner is gone. Bigger resource limits end throttling and cost money on every pod, every hour. The design point is choosing these tensions deliberately, not eliminating them. Debug Fast Shell kubectl get pods -n kafka-consumers kubectl logs <pod> kubectl get events kafka-consumer-groups.sh --describe --group <group> Lag plus member state tells you which issue you are in. Name the issue first; the fix follows from the name. The pod list and events stream show the platform's half of the story — restarts, kills, throttling — and the group description shows the protocol's half: who owns what, how far behind each partition sits, and whether membership is stable or churning. What Changes in Practice With the baseline in place and the eight modes handled, the operational picture changes in specific ways. Deploys stop triggering full rebalances, because the shutdown path announces the departure and the group reassigns narrowly. Restarts replay records safely instead of corrupting progress, because commits are manual and processing is idempotent. Timing stops being the hidden cause behind pages that present as broker problems. And diagnosis gets faster: lag and member state from kafka-consumer-groups.sh --describe identify which of the eight modes is active in minutes, and the fix follows from the name. Takeaways On EKS, the platform is part of the consumer's failure model: restarts, throttling, and shutdowns are normal events, not exceptions.Most consumer pages trace back to timing — heartbeats, poll intervals, and shutdown grace — not to broker problems.Cooperative-sticky assignment and manual commits remove the two loudest sources of churn before tuning even starts.A graceful shutdown path is a feature: PreStop hook, a final commit, and a grace period long enough to use them.Debug by symptom against the eight modes above; lag and member state from kafka-consumer-groups.sh --describe identify the mode in minutes. What's Next The same failure-model thinking extends to every stateful client the platform hosts. Any process that joins a group, holds an assignment, and proves liveness on a cadence will meet its own version of these eight modes, and the fixes generalize with it: budget the timing, announce the shutdown, keep the identity stable, and make the work replayable. The consumer is simply where the collision shows up first, because the group protocol writes its assumptions down in configuration — and configuration can be reviewed line by line. The platform is part of the failure model. Design for it there, in the open, and the pages stop.

Feature stores solve a problem that every machine learning (ML) team eventually hits: the features you compute during training are different from the ones you retrieve at serving time. Feast is an open-source feature store that bridges that gap, handling the registry, materialization, and online retrieval so a model sees the same data in production as it did during training. Often, Feast tutorials pair it with a relational database or a cloud data warehouse. In this article, we'll do something different. We'll use Neo4j Desktop as the feature computation engine. Five features are ordinary aggregations that Cypher handles as easily as SQL. The sixth, a customer community_id, comes from graph algorithms: customers with similar purchase mixes are found with Graph Data Science (GDS) node similarity and then grouped with Louvain community detection. We'll hand all six features to Feast, which materializes them into a SQLite online store, serves them at inference time, and builds point-in-time correct training data. We'll also test, on held-out orders, whether the graph feature tells a model anything that a plain favorite_category column doesn't. The full source code is available on GitHub. Architecture Figure 1. Architecture As shown in Figure 1, the architecture has three layers. Neo4j Desktop handles feature computation: standard aggregations via Cypher and a graph-native community feature via the Graph Data Science (GDS) library. The results, computed at several points in time, land in a Parquet file, which acts as Feast's offline store. Feast then materializes the latest values from Parquet into SQLite for low-latency online serving. At inference time, a single get_online_features() call retrieves the full feature vector, regardless of whether a feature came from a Cypher query or a GDS algorithm. For training data, get_historical_features() reads the same offline store. The key insight is that Feast doesn't know or care how the Parquet file was produced. What We're Building We'll work with a synthetic e-commerce dataset of 100 customers, 3,000 orders, and 5 product categories (Electronics, Clothing, Home, Books, and Sports) spread over the last 90 days. Each customer belongs to a hidden segment defined by a primary and a secondary category. For 55% of their orders the category is the segment's primary one, for 30% it is the secondary one, and for the remaining 15% it is drawn uniformly from all five categories. There are five segments: Electronics (primary) and Home (secondary)Electronics and SportsClothing and BooksClothing and HomeBooks and Sports The design is deliberate. Two segments share Electronics as their primary category and two share Clothing, so a customer's single most-bought category can't tell those segments apart. Only the combination of categories can. To make the churn example meaningful, 18 customers are also made to go quiet: 8 have not ordered for more than 60 days and 10 for more than 30 days. The segment is stored on each Customer node, but only so we can check our results later. It is never used to compute any feature. We'll compute six features per customer, as of a given date: total_orders – Number of orders placed up to the as-of datetotal_spent – Total order valueavg_order_value – Mean order valuedays_since_last_order – Recency, measured from the as-of datefavorite_category – The category with the most orderscommunity_id – Graph-native feature from node similarity and Louvain community detection We compute them at three as-of dates (60 days ago, 30 days ago, and today), so that Feast can later reconstruct what was known at any point in time. Setup A virtual environment is highly recommended. For example: Shell python3 -m venv ~/feast-env source ~/feast-env/bin/activate The notebook was run against Neo4j 2026.07.0 (Enterprise) and GDS 2026.07.0. We need Neo4j Desktop running with the Graph Data Science (GDS) plugin installed. In Neo4j Desktop, go to Plugins and install the Graph Data Science Library. Credentials go in environment variables before running the Jupyter notebook: Shell export NEO4J_URI="bolt://localhost:7687" export NEO4J_USERNAME="neo4j" export NEO4J_PASSWORD="your-password" export NEO4J_DATABASE="your-database" Warning: The notebook removes all data, indexes, and constraints from the database named in NEO4J_DATABASE before loading its sample data. Use a database you are happy to wipe. We'll connect to Neo4j with the standard driver: Python driver = GraphDatabase.driver( NEO4J_URI, auth = (NEO4J_USERNAME, NEO4J_PASSWORD) ) Feature Definitions Feast needs two files: a project config and a feature view declaration. The config points to SQLite for the online store: YAML project: ecommerce_features registry: feature_repo/registry.db provider: local online_store: type: sqlite path: feature_repo/online_store.db The feature definitions declare the customer_id entity and a feature view with all six features, including community_id as an Int64: Python from feast import Entity, FeatureView, Field, FileSource from feast.types import Float32, Int64, String, ValueType from datetime import timedelta customer = Entity( name = "customer_id", description = "Customer identifier", value_type = ValueType.STRING ) customer_features = FeatureView( name = "customer_features", entities = [customer], ttl = timedelta(days = 90), schema = [ Field(name = "total_orders", dtype = Int64), Field(name = "total_spent", dtype = Float32), Field(name = "avg_order_value", dtype = Float32), Field(name = "days_since_last_order", dtype = Int64), Field(name = "favorite_category", dtype = String), Field(name = "community_id", dtype = Int64), ], online = True, source = FileSource( path = "data/customer_features.parquet", timestamp_field = "event_timestamp" ), ) From Feast's perspective, community_id is just another integer column. It doesn't know it took graph algorithms to produce it. Loading Data into Neo4j Rather than rows in a table, we'll model the data as a graph. Figure 2 shows the structure. Figure 2. Graph Data Model Customer and Order nodes connect via PLACED relationships. Each Order connects to a Category node via IN_CATEGORY. We don't store a direct customer-to-category relationship. Instead, the GDS projection derives one at query time, so that it can be limited to orders placed before a given date. All timestamps are timezone-aware (UTC). Cypher UNWIND $orders AS o MERGE (c:Customer {id: o.customer_id}) ON CREATE SET c.segment = o.segment MERGE (cat:Category {name: o.product_category}) CREATE (ord:Order { id: o.order_id, value: o.order_value, timestamp: datetime(o.order_timestamp), quantity: o.quantity }) MERGE (c)-[:PLACED]->(ord) MERGE (ord)-[:IN_CATEGORY]->(cat) Standard Features via Cypher The first five features come from two Cypher queries, both limited to orders placed on or before the as-of date $as_of. The main aggregation computes counts, totals, averages, and recency in a single pass: Cypher MATCH (c:Customer)-[:PLACED]->(o:Order) WHERE o.timestamp <= datetime($as_of) RETURN c.id AS customer_id, count(o) AS total_orders, sum(o.value) AS total_spent, avg(o.value) AS avg_order_value, duration.inDays(max(o.timestamp), datetime($as_of)).days AS days_since_last_order Favorite category requires a separate query, since we need to find the top category per customer and collect() the first result after ordering. Ties are broken alphabetically so that the result is deterministic: Cypher MATCH (c:Customer)-[:PLACED]->(o:Order)-[:IN_CATEGORY]->(cat:Category) WHERE o.timestamp <= datetime($as_of) WITH c.id AS customer_id, cat.name AS category, count(*) AS cnt ORDER BY customer_id, cnt DESC, category WITH customer_id, collect(category)[0] AS favorite_category RETURN customer_id, favorite_category The event_timestamp that Feast uses is the as-of date itself. In other words, the moment the features describe, rather than the time of the customer's last order. Graph-Native Feature: Community ID The sixth feature groups customers whose purchase mixes are similar. It takes four steps, all run inside Neo4j. Step 1: Project customers and categories. We use the current Cypher projection, which builds an in-memory graph with the gds.graph.project aggregation function and replaces the deprecated gds.graph.project.cypher procedure. Each customer-category relationship is weighted by the number of orders placed up to the as-of date: Cypher MATCH (c:Customer)-[:PLACED]->(o:Order)-[:IN_CATEGORY]->(cat:Category) WHERE o.timestamp <= datetime($as_of) WITH c, cat, count(o) AS order_count WITH gds.graph.project( $name, c, cat, { sourceNodeLabels: labels(c), targetNodeLabels: labels(cat), relationshipType: 'PURCHASED_IN', relationshipProperties: {order_count: order_count} } ) AS g RETURN g.graphName AS graph, g.nodeCount AS nodes, g.relationshipCount AS relationships Step 2: Find similar customers. GDS node similarity with the cosine metric, weighted by order count, scores two customers close to 1 when their category mixes point in the same direction. We keep each customer's 15 most similar customers above a cutoff of 0.8: Cypher CALL gds.nodeSimilarity.stream($name, $config) YIELD node1, node2, similarity RETURN gds.util.asNode(node1).id AS src, gds.util.asNode(node2).id AS dst, similarity Python config = { "similarityMetric": "COSINE", "relationshipWeightProperty": "order_count", "topK": SIMILARITY_TOP_K, "similarityCutoff": SIMILARITY_CUTOFF, } Step 3: Project a customer-to-customer graph from those pairs, weighted by similarity: Cypher UNWIND $pairs AS p MATCH (a:Customer {id: p.src}), (b:Customer {id: p.dst}) WITH gds.graph.project( $name, a, b, { sourceNodeLabels: labels(a), targetNodeLabels: labels(b), relationshipType: 'SIMILAR', relationshipProperties: {similarity: p.similarity} }, {undirectedRelationshipTypes: ['*']} ) AS g RETURN g.graphName AS graph, g.nodeCount AS nodes, g.relationshipCount AS relationships Step 4: Run Louvain on that graph, using the similarity scores as relationship weights. consecutiveIds gives us small, tidy community IDs: Cypher CALL gds.louvain.stream($name, $config) YIELD nodeId, communityId WITH gds.util.asNode(nodeId) AS n, communityId WHERE n:Customer RETURN n.id AS customer_id, communityId AS community_id A customer with no sufficiently similar neighbor isn't in the similarity graph, so we give each such customer a community of their own. The two settings, SIMILARITY_TOP_K (15) and SIMILARITY_CUTOFF (0.8), are the only knobs, and we didn't tune them. Why not run Louvain directly on the customer-category graph and skip the similarity step? We tried, and the next section shows why we didn't keep it. Which Grouping Recovers the Hidden Segments? We compared four ways of grouping the same customers against the hidden segments, using the Adjusted Rand Index (ARI), where 1.0 means identical groupings and values near 0 mean no better than chance: Plain Text Grouping Groups ARI vs hidden segments ---------------------------------------------------- -------- ------------------------ 1. favorite_category 5 0.52 2. Louvain, customer-category graph, unweighted 2 0.08 3. Louvain, customer-category graph, weighted 3 0.41 4. Louvain, customer-similarity graph (community_id) 5 0.98 A plain favorite_category scores 0.52. It can't separate segments that share a primary category and the cross-tabulation shows exactly that: Electronics lumps together 21 customers from one segment and 22 from another and Clothing does the same with 20 and 15. Plain Text favorite_category vs hidden segment (latest snapshot): segment Books + Sports Clothing + Books Clothing + Home Electronics + Home Electronics + Sports favorite_category Books 14 2 0 0 0 Clothing 0 20 15 0 0 Electronics 0 0 0 21 22 Home 0 0 0 2 0 Sports 2 0 0 0 2 Louvain run directly on the customer-category graph does worse than that, with an ARI of 0.41 when weighted by order count and 0.08 when unweighted. We didn't investigate why, but the likely cause is that two segments sharing a primary category both attach to the same category node, which pulls them together. Without weights, every customer is linked to every category equally, and there is little left to separate them. The similarity-graph method scores 0.98 and finds five communities that line up with the five segments: Plain Text community_id vs hidden segment (latest snapshot): segment Books + Sports Clothing + Books Clothing + Home Electronics + Home Electronics + Sports community_id 0 0 0 0 23 0 1 0 1 15 0 0 2 0 21 0 0 0 3 0 0 0 0 24 4 16 0 0 0 0 Figure 3 shows the five communities, each labeled with the segment that makes up most of it. Figure 3. Communities Found by Node Similarity and Louvain. Only one customer lands in the wrong community. The algorithm was never given the segments, so it recovered them from purchase behavior alone. Does the Community Feature Add Predictive Signal? Recovering hidden segments is reassuring, but a feature earns its place by helping predictions. We test that without training a model, using only the features as they were 30 days ago. For each group of customers, we estimate a mix of product categories from orders up to that date, then score how well that mix predicts the category of every order placed afterward. The score is log-loss, the average of the negative log of the probability given to the category actually bought, so lower is better. There is no leakage: both favorite_category and community_id were computed from orders placed before the cutoff. Python def heldout_losses(orders, snapshot, cutoff_ts, group_col, alpha = 1.0): """How well does a grouping of customers predict the category of FUTURE orders? For each group, estimate the category mix from orders up to `cutoff_ts` (with add-`alpha` smoothing). Then score each later order by the negative log of the probability that mix gave its category. Lower is better. Returns one loss per held-out order.""" groups = snapshot[["customer_id", group_col]].rename(columns = {group_col: "group"}) train = orders[orders["ts"] <= cutoff_ts].merge(groups, on = "customer_id") test = orders[orders["ts"] > cutoff_ts].merge(groups, on = "customer_id") counts = (train.groupby(["group", "category"]).size().unstack(fill_value = 0) .reindex(columns = CATEGORIES, fill_value = 0) + alpha) probs = counts.div(counts.sum(axis = 1), axis = 0) idx = pd.MultiIndex.from_arrays([test["group"], test["category"]]) p = probs.stack().reindex(idx).fillna(1 / len(CATEGORIES)).values return -np.log(p) We compare no grouping at all, favorite_category, community_id and the hidden segment as an upper bound that no real feature could beat. A paired bootstrap gives a confidence interval for the improvement of community_id over favorite_category: Python diff = losses["favorite_category"] - losses["community_id"] rng = np.random.default_rng(SEED) boot = np.array([rng.choice(diff, size = len(diff), replace = True).mean() for _ in range(2000)]) low, high = np.percentile(boot, [2.5, 97.5]) Plain Text Held-out orders (placed after the 30-day snapshot): 827 Grouping Groups Held-out log-loss (lower is better) ------------------------------------- -------- ------------------------------------- No grouping (everyone shares one mix) 1 1.565 favorite_category 5 1.171 community_id 5 1.025 Hidden segment (oracle upper bound) 5 1.009 Improvement of community_id over favorite_category: 0.146 (95% CI 0.110 to 0.180) Grouping customers by favorite_category cuts the loss from 1.565 to 1.171. Grouping by community_id cuts it to 1.025, close to the 1.009 of the hidden segments. The improvement over favorite_category is 0.146, with a 95% confidence interval of 0.110 to 0.180. The reason is intuitive: knowing a customer's secondary category tells you what they are likely to buy after their favorite, and the community captures that while a single favorite category can't. Some caveats apply. The data are synthetic and designed so the combination of categories matters, so the size of the gain reflects that design, not what you would see in real orders. The predictor is a simple count-based one rather than a trained model. The bootstrap resamples individual orders and ignores that orders from one customer are related, so the interval is somewhat optimistic. And this is one dataset from one random seed. It is also worth saying what happened before we changed the data. In an earlier version of this example, every customer had a single preferred category. There, community_id and favorite_category agreed exactly (ARI 1.00), so the graph feature added nothing. The gain here appears when group membership depends on a combination of categories, which is the situation where a single aggregate column falls short. Snapshots and Stable Community IDs To support point-in-time retrieval, we compute all six features at three as-of dates: 60 days ago, 30 days ago, and today, and stack the results. Louvain runs independently for each date, and that creates a problem. Community IDs are arbitrary labels, and a label such as 3 in one run need not mean the same group in the next. A model trained on one snapshot would then be served IDs with a different meaning. We solve this by relabeling each snapshot to match the previous one, assigning each new community the old ID it overlaps with most: Python def align_communities(current, reference): """Relabel `current` community IDs to match `reference` by greatest member overlap.""" merged = current[["customer_id", "community_id"]].merge( reference[["customer_id", "community_id"]], on = "customer_id", suffixes = ("_cur", "_ref") ) overlap = (merged.groupby(["community_id_cur", "community_id_ref"]) .size().reset_index(name = "n") .sort_values("n", ascending = False)) mapping, used = {}, set() for _, r in overlap.iterrows(): cur, ref = int(r["community_id_cur"]), int(r["community_id_ref"]) if cur not in mapping and ref not in used: mapping[cur] = ref used.add(ref) # Any community with no counterpart in the reference gets a fresh ID next_id = int(max(reference["community_id"].max(), max(mapping.values(), default = -1))) + 1 for cur in current["community_id"].unique(): if int(cur) not in mapping: mapping[int(cur)] = next_id next_id += 1 aligned = current.copy() aligned["community_id"] = aligned["community_id"].map(lambda x: mapping[int(x)]).astype("int64") return aligned Python for days in SNAPSHOT_OFFSETS_DAYS: as_of = now - timedelta(days = days) df, stats = compute_features_as_of(as_of) if reference is not None: df = align_communities(df, reference) reference = df[["customer_id", "community_id"]] frames.append(df) features_df = pd.concat(frames, ignore_index = True) The notebook prints how well each snapshot recovers the segments and how much the labels would have disagreed without alignment: Plain Text As of 2026-08-27 (60 days ago): 100 customers, 844 similar pairs, 5 communities, ARI vs hidden segments 0.95 As of 2026-09-26 (30 days ago): 100 customers, 847 similar pairs, 5 communities, ARI vs hidden segments 0.95 label agreement with previous snapshot: raw 65% -> aligned 96% As of 2026-10-26 ( 0 days ago): 100 customers, 856 similar pairs, 5 communities, ARI vs hidden segments 0.98 label agreement with previous snapshot: raw 63% -> aligned 99% Each snapshot finds five communities and recovers the hidden segments well (ARI 0.95, 0.95 and 0.98), even the earliest one, which has far fewer orders per customer. Without alignment, only 65% and 63% of customers kept their raw label from one run to the next. After alignment, 96% and 99% did. The remainder are customers whose community changed as more orders arrived. Because community_id is an arbitrary label, a model should treat it as a categorical value (one-hot or embedding), never as an ordered number. Merging Features and Writing to Parquet The three snapshots are written to a single Parquet file, one row per customer per as-of date: Python features_df.to_parquet("feature_repo/data/customer_features.parquet", index = False) Plain Text Wrote 300 rows to Parquet. event_timestamp 2026-08-27 18:02:57.920521+00:00 100 2026-09-26 18:02:57.920521+00:00 100 2026-10-26 18:02:57.920521+00:00 100 From here, Feast takes over. It doesn't know that community_id came from graph algorithms and it's just another column in the Parquet file. Feast Apply and Materialize We'll apply the feature definitions and materialize into the SQLite online store: Python result = subprocess.run( ["feast", "-c", "feature_repo", "apply"], capture_output = True, text = True ) end_date = datetime.now(timezone.utc) start_date = end_date - timedelta(days = 90) result = subprocess.run( ["feast", "-c", "feature_repo", "materialize", start_date.isoformat(), end_date.isoformat()], capture_output = True, text = True ) feast apply registers the entity and feature view. feast materialize reads the Parquet file and loads the most recent value for each customer into SQLite, ready for low-latency retrieval. SQLite and Parquet are convenient for a local demo; in production you would swap in stores built for scale and concurrency. Feast keeps that largely a configuration change rather than a change to the feature code. Online Feature Retrieval We'll reinitialize the feature store and retrieve features for three customers, one active, one cooling, and one churned: Python fs = FeatureStore(repo_path = "feature_repo") feature_vector = fs.get_online_features( features = [ "customer_features:total_orders", "customer_features:total_spent", "customer_features:avg_order_value", "customer_features:days_since_last_order", "customer_features:favorite_category", "customer_features:community_id", ], entity_rows = [{"customer_id": cid} for cid in test_customers] ) The retrieved features look like this: Plain Text customer_id favorite_category total_spent total_orders days_since_last_order community_id avg_order_value ------------- ------------------- ------------- -------------- ----------------------- -------------- ----------------- CUST_0081 Books 7765.10 28 5 4 277.33 CUST_0007 Electronics 7000.73 26 35 0 269.26 CUST_0032 Clothing 6580.14 26 66 2 253.08 community_id appears alongside the standard aggregations, served identically through the same Feast interface. Using Features for Prediction A simple rule-based churn risk function shows how the features feed into inference. Recency drives the risk tier, and community_id is returned with the result so that downstream systems can segment or report by community: Python def predict_customer_churn(customer_id): """Use features for real-time churn prediction.""" features = fs.get_online_features( features = [ "customer_features:total_orders", "customer_features:total_spent", "customer_features:days_since_last_order", "customer_features:community_id", ], entity_rows = [{"customer_id": customer_id}] ).to_df() days_since = features["days_since_last_order"].iloc[0] community = features["community_id"].iloc[0] if days_since > 60: risk, action = "HIGH", "Send 20% discount" elif days_since > 30: risk, action = "MEDIUM", "Send personalized email" else: risk, action = "LOW", "Continue engagement" return { "customer_id": customer_id, "days_since_last_order": int(days_since), "churn_risk": risk, "action": action, "community_id": int(community), } predictions = [predict_customer_churn(c) for c in test_customers] print(tabulate(predictions, headers = "keys", tablefmt = "simple")) Plain Text customer_id days_since_last_order churn_risk action community_id ------------- ----------------------- ------------ ----------------------- -------------- CUST_0081 5 LOW Continue engagement 4 CUST_0007 35 MEDIUM Send personalized email 0 CUST_0032 66 HIGH Send 20% discount 2 The rule itself doesn't use community_id; the point is that serving is identical whichever way a feature was computed. One caveat applies to days_since_last_order: it was calculated when we materialized, so served a week later without re-materializing it would be a week out of date. In production, re-run materialization on a schedule, or store the last-order timestamp and compute recency at request time, for example with a Feast on-demand feature view. Point-in-Time Historical Features The other half of a feature store's job is building training data in which each row sees only what was known at that row's timestamp. We ask Feast for one customer's features at three timestamps: Python entity_df = pd.DataFrame({ "customer_id": [target] * 3, "event_timestamp": [ pd.Timestamp(now - timedelta(days = 45)), pd.Timestamp(now - timedelta(days = 20)), pd.Timestamp(datetime.now(timezone.utc)), ], }) historical_df = fs.get_historical_features( entity_df = entity_df, features = [ "customer_features:total_orders", "customer_features:total_spent", "customer_features:avg_order_value", "customer_features:days_since_last_order", "customer_features:favorite_category", "customer_features:community_id", ], ).to_df() Plain Text customer_id event_timestamp total_orders total_spent avg_order_value days_since_last_order favorite_category community_id ------------- -------------------------------- -------------- ------------- ----------------- ----------------------- ------------------- -------------- CUST_0081 2026-09-11 18:02:57.920521+00:00 12 3609.80 300.82 2 Books 4 CUST_0081 2026-10-06 18:02:57.920521+00:00 20 6086.00 304.30 6 Books 4 CUST_0081 2026-10-26 18:03:04.843967+00:00 28 7765.10 277.33 5 Books 4 Feast joins each timestamp to the most recent snapshot at or before it, so the customer's order count grows from 12 to 20 to 28 as time moves forward. Every snapshot, including its community_id, was computed using only orders placed up to its own date, so none of these rows contain information from the future. The event_timestamp shown is the one we asked about, not the snapshot's own date. Gotchas and Lessons Learned A few things worth knowing before you run the notebook. Dependency conflicts. To avoid incompatibilities between Feast and the graphdatascience Python client, the solution is to drop graphdatascience entirely and call GDS via raw Cypher through the Neo4j driver. Every GDS procedure used here (gds.graph.project, gds.nodeSimilarity.stream, gds.louvain.stream, gds.graph.drop) is callable this way and the result is a cleaner install. GDS notifications. GDS warns that the schema field returned by gds.graph.drop is deprecated. The warning is harmless, but it repeats every time the procedure is called. Rather than silencing all server notifications on the driver, we yield only the field we need: CALL gds.graph.drop($name, false) YIELD graphName RETURN graphName. With that change no deprecation notifications appeared in our run. Choose the graph that matches the question. Louvain on the customer-category graph groups customers by which categories they are attached to, and struggles when groups share a category. Comparing whole category mixes with node similarity first, then clustering that customer-to-customer graph, is what separated the overlapping segments. Community IDs are labels, not values. Independent Louvain runs relabel the same groups, as the 65% and 63% agreement above shows. Align IDs across snapshots, and treat them as categorical in your model. Check your environment variables. The notebook reads its connection details from NEO4J_* environment variables. Stale variables left over from another project can point the notebook at a different server, which can show up as confusing errors that have nothing to do with your code. Pandas version. Feast 0.66.0 requires pandas<3 and >=1.4.3. Summary Neo4j works well as a feature computation engine alongside Feast. The Cypher queries for standard aggregations are comparable in complexity to their SQL equivalents, and GDS lets you add graph-algorithm output to the same feature set with a few projections and procedure calls. In our example, node similarity followed by Louvain recovered the hidden customer segments with an ARI of 0.98, where a plain favorite-category column managed 0.52, and Feast served the result like any other column. On held-out orders, grouping customers by community_id predicted category mix better than grouping by favorite_category, by 0.146 log-loss (95% interval 0.110 to 0.180). Those numbers come from synthetic data built so that combinations of categories matter, and from a simple count-based predictor, so treat them as a demonstration of the pipeline and the method rather than a benchmark. When we first used single-category preferences, the graph feature was redundant with favorite_category, which is a useful reminder to test whether a graph feature adds something before building it into a pipeline. The pattern generalizes beyond community detection. PageRank scores and node embeddings (for example from FastRP, node2vec or GraphSAGE) can be written back to node properties and flow through the same pipeline, with embeddings stored as array-valued features. Two cautions apply. Some graph outputs, such as shortest-path distances, are defined between pairs of nodes, so they need a fixed reference node before they can become a per-entity feature. And graph features are computed in batch, so a brand-new customer has none until the next run. The offline store (Parquet) and the serving layer (Feast) stay the same regardless of how the features were computed. The full source code is available on GitHub.

By Akmal Chaudhri CORE

Introduction: From Voice AI Prototype to Production Phone Agent A voice AI demo is a model, a microphone, and a speaker. A phone agent is a distributed system. Audio arrives from a carrier over a SIP trunk, callers interrupt on bad lines, backends time out, model providers have bad minutes, and callers want a person the moment something goes wrong. This article shows you how to build an AI receptionist for a dental clinic. It answers calls to the clinic’s number, looks up patients by caller ID, looks up and books appointments through a backend API, and transfers calls to the front desk when a person is needed. It runs on LiveKit Agents, the open-source framework for Python and Node.js. The logs highlight which features require LiveKit Cloud. Every snippet comes from one working project, and every output shows what that project printed. The architecture has six parts: A telephony provider that owns the number.SIP connectivity that turns a call into a session participant.Real-time audio infrastructure between the caller and the agent.The AI models: speech-to-text, a language model and text-to-speech.Application logic: conversation state, tools, and escalation.Monitoring: latency, errors, and cost for every call. Step 1: Choose Your Real-Time Voice Infrastructure A phone call is not a request. Audio flows both ways at once, callers talk over the agent, and a pause more than a second feels broken. You need a transport that moves audio frames in real time and an agent that reacts as they arrive: SIP is the phone network's signaling protocol. It sets up, transfers, and ends calls, while the audio travels over RTP.WebRTC carries low-latency audio between participants and absorbs packet loss and jitter.A media server bridges the two and turns the call into a participant in a WebRTC room. In LiveKit, the telephony media layer consists of the LiveKit server plus the LiveKit SIP service, both open source. A call becomes a SIP participant in a room, and LiveKit dispatches a job to an agent server. Each job runs in its own process — in this setup, one per call. The agent server connects out over a WebSocket for job dispatch and uses WebRTC for media, so it needs no public inbound ports. Run it all yourself or on LiveKit Cloud, or connect self-hosted agents to Cloud. LiveKit Inference, Agent insights, LiveKit Phone Numbers, and the participant move behind this warm-transfer workflow are Cloud-only. Cloud also provides enhanced noise cancellation, while basic WebRTC cancellation and separately licensed alternatives are available outside Cloud. There are other routes. Twilio (the carrier) can also stream audio calls over WebSockets (Media Streams), which LiveKit Cloud accepts through its Twilio Connector. At the fully managed end, Voiceflow runs the whole stack: you design the agent, import a Twilio or Vonage number, and assign it. Calls, recordings, and transcripts just work. The SDK route trades that convenience for control over every model, timeout, and transfer rule. Step 2: Select and Configure Your AI Models You need a cascaded pipeline that chains speech-to-text (STT), a large language model (LLM) that decides what to say and which tools to call, and text-to-speech (TTS). A real-time speech-to-speech model — such as OpenAI Realtime or Gemini Live — does all three in one. Cascaded STT, LLM, TTSRealtime speech-to-speechLatencyModerate; streaming overlaps the stagesLowest; one model, no hand-offsChanging modelsAny stage, independentlyOne provider for the whole pathTool callingMature, text-basedLess matureDebugging and auditText at every stageTranscripts arrive lateExact phrasing with say()YesNot the same wayCostThree meters, each tunableOne meter A receptionist must read appointment data exactly and keep an auditable transcript, so the agent uses the cascaded pipeline — LiveKit's recommended production default. Keep the models in configuration, since prices, voices, and availability change. In LiveKit Agents, the pipeline is an AgentSession with one parameter per stage. In LiveKit Cloud, the inference module provides hosted models with server-side fallbacks and no provider keys: Python from livekit.agents import AgentSession, TurnHandlingOptions, inference session = AgentSession( stt=inference.STT(model="deepgram/nova-3", language="en", fallback=[{"model": "assemblyai/universal-3-6-pro"}]), llm=inference.LLM(model="google/gemma-4-31b-it"), tts=inference.TTS(model="cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc", fallback=[{"model": "fishaudio/s2.1-pro", "voice": "fa4c9eb3dccc4806b382b40d61c6b10a"}]), turn_handling=TurnHandlingOptions( turn_detection=inference.TurnDetector(), endpointing={"min_delay": 0.5, "max_delay": 3.0}, interruption={"min_duration": 0.5}, ), ) When self-hosted, the same session takes provider plugins and your own keys. Nothing else changes. The receptionist picks one session with a MODEL_PROVIDER variable, and the calls in this guide run on the plugin version: Python from livekit.plugins import openai, silero session = AgentSession( stt=openai.STT(model="gpt-4o-mini-transcribe", use_realtime=True), # streaming transcripts llm=openai.LLM(model="gpt-4.1-mini"), tts=openai.TTS(model="gpt-4o-mini-tts", voice="coral"), vad=silero.VAD.load(), turn_handling=TurnHandlingOptions(turn_detection="vad", endpointing={"min_delay": 0.5, "max_delay": 3.0}), ) Step 3: Connect Your AI Agent to the Telephone Network The PSTN is the public phone network. The SIP trunk connects this to your system: the provider owns the number and forwards each call as a SIP INVITE. LiveKit mirrors the trunk with an inbound trunk (which claims the number) and a dispatch rule (which picks the room and the agent). Get a number. LiveKit Cloud sells US numbers through LiveKit Phone Numbers, which need only a dispatch rule. With Twilio, create an Elastic SIP trunk, point its origination URI at your LiveKit SIP endpoint (your project ID without p_, followed by .sip.livekit.cloud), and attach the number: Shell twilio api trunking v1 trunks create --friendly-name "Harbor Dental" --domain-name "harbor-dental.pstn.twilio.com" twilio api trunking v1 trunks origination-urls create --trunk-sid <TRUNK_SID> --friendly-name "LiveKit SIP" \ --sip-url "sip:<subdomain>.sip.livekit.cloud;transport=tcp" --weight 1 --priority 1 --enabled twilio api trunking v1 trunks phone-numbers create --trunk-sid <TRUNK_SID> --phone-number-sid <NUMBER_SID> Create the inbound trunk and the dispatch rule. The rule puts each caller in a fresh room and dispatches the agent named receptionist: JSON { "trunk": { "name": "Harbor Dental inbound", "numbers": ["+15105550100"] } } JSON { "dispatch_rule": { "name": "Harbor Dental receptionist", "trunk_ids": ["ST_TgirZ2Wz75YG"], "rule": { "dispatch_rule_individual": { "room_prefix": "call-" } }, "room_config": { "agents": [{ "agent_name": "receptionist", "metadata": "{\"clinic\": \"harbor-dental\"}" }] } } } Shell lk sip inbound create inbound-trunk.json lk sip dispatch create dispatch-rule.json lk sip dispatch list Plain Text ┌───────────────────┬────────────────────────────┬─────────────────┬─────────────────────┬─────────────────────────┬──────────────┐ │ SipDispatchRuleID │ Name │ SipTrunks │ Type │ RoomName │ Agents │ ├───────────────────┼────────────────────────────┼─────────────────┼─────────────────────┼─────────────────────────┼──────────────┤ │ SDR_w4oMyPKcc2qT │ Harbor Dental receptionist │ ST_TgirZ2Wz75YG │ Individual (Caller) │ call-_<caller>_<random> │ receptionist │ └───────────────────┴────────────────────────────┴─────────────────┴─────────────────────┴─────────────────────────┴──────────────┘ Create both once and reuse them for every call. Pin the rule to your trunk with trunk_ids, since a rule without it matches every trunk. The metadata reaches the agent as ctx.job.metadata, so one agent can serve several clinics. Answer the call. LiveKit SIP matches the trunk and the rule, creates the room with the caller in it, and dispatches a job to an agent server registered under the same agent_name. The caller's number arrives as a participant attribute, so the receptionist agent finds the patient before saying a word: Python server = AgentServer() @server.rtc_session(agent_name="receptionist", on_session_end=on_session_end) async def receptionist(ctx: JobContext) -> None: await ctx.connect() participant = await ctx.wait_for_participant() customer = None if participant.kind == rtc.ParticipantKind.PARTICIPANT_KIND_SIP: phone = participant.attributes.get("sip.phoneNumber", "") logger.info("inbound call from %s to %s (call %s)", phone, participant.attributes.get("sip.trunkPhoneNumber"), participant.attributes.get("sip.callID")) try: customer = await clinic_api("GET", "/customers", params={"phone": phone}) except ToolError: customer = None # greet anonymously; the agent can identify the caller later session = build_session() observe(session) await session.start(room=ctx.room, agent=Receptionist(customer), room_options=room_options()) # A scripted greeting goes straight to TTS: no LLM round trip before the caller hears a voice. who = f"Hello {customer['name'].split()[0]}, " if customer else "Hello, " await session.say(f"{who}this is Ava from Harbor Dental Clinic, how can I help you today?") Check it with a call. A number that rings without an answer may indicate that no agent is registered under the configured name. A call that connects to silence may indicate the agent is not publishing audio. A healthy call looks like this in the agent log: Plain Text INFO livekit.agents - registered worker {"agent_name": "receptionist", "id": "AW_MMW9r2DMkYYP", "url": "ws://localhost:7880"} INFO livekit.agents - received job request {"job_id": "AJ_8pRxKpmvNjmf", "room": "call-_+15550002_BFK5wQ3zB25J", "agent_name": "receptionist"} INFO receptionist - inbound call from +15550002 to +15105550100 (call SCL_LeFbnPRqGmDK) lk room list shows the call- room while the call is up, and lk room participants get prints the SIP attributes, including sip.twilio.callSid, for finding the call in the Twilio console. Outbound calls and existing phone systems. Outbound calls and the warm transfer in Step 5 need an outbound trunk configured with the provider’s termination address, credentials if required, and the caller ID to present. Your code dials with CreateSIPParticipant. A SIP-capable PBX or contact center connects the same way, using an inbound trunk, an outbound trunk, or both, depending on the call direction. JSON { "trunk": { "name": "Harbor Dental outbound", "address": "harbor-dental.pstn.twilio.com", "numbers": ["+15105550100"], "auth_username": "<user>", "auth_password": "<password>" } } Step 4: Implement Conversations, Interruptions, and Backend Integrations Conversation state. AgentSession keeps the conversation history, including turns, tool calls, and results. The active agent’s chat context is sent to the LLM each turn. The receptionist stores the identified patient on the agent and includes that information in its instructions. Without that explicit context, the model may ask known callers for their number again: Python class Receptionist(Agent): def __init__(self, customer: dict | None = None) -> None: end_call = EndCallTool( extra_description="Only end the call after the caller confirms they need nothing else.", end_instructions="Thank the caller for calling Harbor Dental Clinic and say goodbye.", ) instructions = INSTRUCTIONS if customer: # matched by caller ID before the session started instructions += (f"\nThe caller ID matches the patient {customer['name']} (id {customer['id']}), " "so they are already identified: do not ask for their phone number.") super().__init__(instructions=instructions, tools=end_call.tools) self.customer = customer Turns and interruptions. Voice activity detection (VAD) finds speech, while turn detection estimates when the caller has finished talking. LiveKit’s default audio turn detector weighs both what the caller says and how they say it. With interruptions enabled, LiveKit stops the agent’s speech and trims the conversation history to reflect what was played before the interruption. This supports corrections such as “actually, make it Thursday” without custom interruption-handling code, though updating the appointment still requires application logic. The sessions wait 0.5 to 3 seconds before closing a turn, and LiveKit Cloud ignores interruptions under half a second, which filters coughs. LiveKit Cloud's adaptive interruption model also tells "mm-hm" from a real interruption; without Cloud credentials, the agent falls back to VAD: Plain Text INFO livekit.agents - adaptive interruption disabled due to unrecoverable error, falling back to VAD-based interruption Tools. A method decorated with @function_tool becomes a tool, and its docstring and type hints define the tool description and parameter schema the LLM sees. The receptionist calls a clinic API through the job’s shared HTTP session with a four-second timeout and turns handled failures into messages the model can act on: Python async def clinic_api(method: str, path: str, **kwargs: Any) -> Any: http = utils.http_context.http_session() try: async with http.request(method, f"{BACKEND_URL}{path}", timeout=aiohttp.ClientTimeout(total=4), **kwargs) as resp: if resp.status == 404: return None data = await resp.json() if resp.status >= 400: raise ToolError(data.get("error", f"clinic system error {resp.status}")) return data except (aiohttp.ClientError, asyncio.TimeoutError) as exc: raise ToolError("The clinic system is not responding. Apologize, and offer a transfer or a callback.") from exc class Receptionist(Agent): @function_tool() async def get_appointments(self, ctx: RunContext) -> list[dict]: """List the identified caller's upcoming appointments.""" if self.customer is None: raise ToolError("The caller is not identified yet. Ask for the phone number on file.") return await clinic_api("GET", f"/customers/{self.customer['id']}/appointments") @function_tool() async def book_appointment(self, ctx: RunContext, day: str, time: str, service: str) -> dict: """Book an appointment for the identified caller. Call it only after they said yes. Args: day: The date in YYYY-MM-DD format. time: The start time in 24 hour HH:MM format. service: The service, for example "dental cleaning". """ ctx.disallow_interruptions() # a write must finish even if the caller keeps talking if self.customer is None: raise ToolError("Identify the caller before booking.") return await clinic_api("POST", "/appointments", json={"customer_id": self.customer["id"], "service": service, "start": f"{day}T{time}"}) Three rules help keep tools safe on a phone line. Lookups may be interrupted, but completed lookup results stay in history. Writes call ctx.disallow_interruptions() first, because interrupting a booking does not automatically roll it back. A ToolError returns an actionable error to the LLM so the agent can explain the failure to the caller. The agent should confirm an appointment only after the backend reports success. The receptionist can book but not cancel, and booking requires spoken confirmation from the caller. Step 5: Add Human Call Transfers and Failure Handling An agent with no way out of the conversation is a dead end. The receptionist hands the call over when the caller asks for a person—for billing, insurance, complaints, or medical questions—or after two failed attempts to complete the caller’s request. A cold transfer forwards the call using SIP REFER: the carrier connects the caller to the new number, and the agent drops out once the transfer succeeds. A warm transfer puts the caller on hold, calls the human into a private room, briefs them, and merges the calls, reducing the need for the caller to explain the problem twice. The receptionist uses warm transfer for billing, insurance, and complaints. It uses cold transfer for other requests to speak with a person. Cold transfer. TransferSIPParticipant sends the REFER. The caller's participant identity is not the phone number, so sip_caller() finds the caller by participant type: Python @function_tool() async def transfer_to_human(self, ctx: RunContext, reason: str) -> str | None: """Cold transfer to the front desk. Call it only after the caller agreed to it.""" ctx.disallow_interruptions() line = sip_caller() # (job, SIP participant), or None outside a phone call if line is None: return "There is no phone line to transfer. Offer a callback instead." job, caller = line await ctx.session.say("Transferring you to the front desk now.", allow_interruptions=False) try: await job.api.sip.transfer_sip_participant(api.TransferSIPParticipantRequest( room_name=job.room.name, participant_identity=caller.identity, transfer_to=FRONT_DESK_URI, # "tel:+1..." on Twilio, "sip:+1...@host" on most PBXs play_dialtone=True, )) except api.SipCallError as exc: # the front desk rejected the call or did not answer logger.warning("transfer failed: SIP %s %s", exc.sip_status_code, exc.sip_status) return "The front desk did not answer. Apologize and offer to take a message or arrange a callback." except api.ServerError: return "The transfer could not be started. Apologize and offer a callback." return None # the caller has left the room The trunk must allow transfers. On Twilio, run twilio api trunking v1 trunks update --sid <SID> --transfer-mode enable-all. The target format depends on the carrier. Twilio takes tel:+15105550199, while a PBX such as Asterisk answers a tel: target with 400 Bad Request and needs sip:+15105550199@host. A front desk that rejects the call or doesn't answer within ringing_timeout (30 seconds by default) raises a SipCallError, and the caller is still on the line to be offered a callback. Warm transfer. The prebuilt WarmTransferTask (Python’s beta.workflows module) plays hold music, dials the front desk through the outbound trunk, and uses a second agent to brief the front desk based on the chat history. Once the front desk accepts the transfer, the task moves their SIP participant into the caller’s room: Python @function_tool() async def transfer_with_handoff(self, ctx: RunContext, summary: str) -> str | None: """Warm transfer: put the caller on hold, brief the front desk, then connect them.""" ctx.disallow_interruptions() if sip_caller() is None: return "There is no phone line to transfer. Offer a callback instead." if not ON_LIVEKIT_CLOUD: # the merge uses MoveParticipant, a LiveKit Cloud API return "Warm transfers are not available here. Use transfer_to_human; the caller already agreed." await ctx.session.say("Please hold while I brief the front desk.", allow_interruptions=False) try: result = await WarmTransferTask( sip_call_to=FRONT_DESK_NUMBER, sip_trunk_id=os.environ["LIVEKIT_SIP_OUTBOUND_TRUNK"], chat_ctx=self.chat_ctx, extra_instructions=f"Context for the front desk: {summary}", ringing_timeout=30.0, ) except ToolError: # no answer, voicemail, or the front desk declined return "The front desk could not take the call. Apologize, then offer a cold transfer or a callback." await ctx.session.say("You're now connected with the front desk. Goodbye!", allow_interruptions=False) ctx.session.shutdown() # the agent leaves; the caller and the front desk stay in the room return None Three details matter. First, merging the calls requires MoveParticipant, which is available only in LiveKit Cloud. On a self-hosted deployment, the transfer workflow can dial and brief the front desk, but the merge request returns not implemented. Second, the caller’s room must remain open after the agent session ends. Setting delete_room_on_close=False prevents session shutdown from deleting it. Third, a ringing timeout, detected voicemail, or an explicit decline ends the transfer task with a ToolError, allowing the agent to resume the caller conversation. Any fallback to a cold transfer must be handled separately by the application. The following log from the agent and LiveKit SIP service shows a billing-call transfer on a self-hosted deployment: Plain Text # agent (local time) 13:17:50 DEBUG livekit.agents - executing tool {"function": "transfer_with_handoff", ...} 13:17:51 DEBUG livekit.agents - executing tool {"function": "transfer_to_human", ...} 13:17:51 INFO receptionist - transferring sip_+15550002 to sip:+15105550199@asterisk (The patient Maria Lopez called reporting being charged twice on their last bill and needs assistance with billing.) # LiveKit SIP service (UTC) 12:17:55.918Z INFO sip transferring SIP call {"callID": "SCL_Gfe6warmSi7b", "transferTo": "<sip:+15105550199@asterisk>"} 12:17:56.938Z INFO sip inbound call transferred {"callID": "SCL_Gfe6warmSi7b"} Other failures. Catch backend timeouts and raise a ToolError with a useful recovery message, then instruct the agent to explain the problem to the caller. Model outages switch providers through fallbacklists on LiveKit Cloud. You can also configure stt.FallbackAdapter, llm.FallbackAdapter, and tts.FallbackAdapter in the agent process, whether Cloud-hosted or self-hosted. The session’s error event exposes ev.error.recoverable to guide error handling. A caller hang-up closes the session with the reason participant_disconnected. Adding the prebuilt EndCallTool lets the agent end the call cleanly when the conversation is over. Step 6: Deploy, Monitor, and Scale Your AI Phone Agent In production, the agent is a container running python agent.py start, which registers with LiveKit and waits for jobs. LiveKit's Dockerfile template builds in two stages and runs as a non-root user: Dockerfile ARG PYTHON_VERSION=3.12 FROM python:${PYTHON_VERSION}-slim AS base ENV PYTHONUNBUFFERED=1 ENV PIP_DISABLE_PIP_VERSION_CHECK=1 FROM base AS build RUN apt-get update && apt-get install -y gcc g++ python3-dev && rm -rf /var/lib/apt/lists/* WORKDIR /app COPY requirements.txt ./ RUN python -m venv .venv ENV PATH="/app/.venv/bin:$PATH" RUN pip install --no-cache-dir -r requirements.txt COPY . . FROM base ARG UID=10001 RUN adduser --disabled-password --gecos "" --home "/app" --shell "/sbin/nologin" --uid "${UID}" appuser WORKDIR /app COPY --from=build --chown=appuser:appuser /app /app ENV PATH="/app/.venv/bin:$PATH" USER appuser CMD ["python", "agent.py", "start"] The image is 1.01 GB. In start mode, the agent answers a health check on port 8081 and writes JSON logs, ready for a log drain. On LiveKit Cloud, lk agent create builds and deploys it, lk agent deploy ships new versions, lk agent logs tails them, and --region keeps the agent in the same area as your callers. Capacity. Each call is a job in its own process. LiveKit recommends using 4 cores and 8 GB per agent server to start, for 10 to 25 concurrent calls — depending on the models you run locally. A server stops taking jobs above load_threshold (CPU, 0.7 by default), so scale out earlier, around 0.5. Give servers a grace period of ten minutes or more, so calls finish before shutdown. Latency. Every message in the history logs carries a metrics report. Log it to take note of the time spent for every turn: Python @session.on("conversation_item_added") def _on_item(ev: ConversationItemAddedEvent) -> None: if not isinstance(ev.item, ChatMessage): return m = ev.item.metrics if ev.item.role == "user" and m.get("end_of_turn_delay") is not None: logger.info("user turn: end_of_turn=%.2fs transcription=%.2fs", m["end_of_turn_delay"], m.get("transcription_delay") or 0.0) elif ev.item.role == "assistant" and m.get("e2e_latency") is not None: logger.info("agent turn: e2e=%.2fs llm_ttft=%.2fs tts_ttfb=%.2fs", m["e2e_latency"], m.get("llm_node_ttft") or 0.0, m.get("tts_node_ttfb") or 0.0) Plain Text 13:18:26,334 INFO receptionist - user turn: end_of_turn=1.21s transcription=1.21s 13:18:26,877 DEBUG livekit.agents - executing tool {"function": "get_appointments", "lk.pii.arguments": "{}"} 13:18:38,520 INFO receptionist - agent turn: e2e=3.45s llm_ttft=0.50s tts_ttfb=1.00s Maria's appointment question took 3.45 seconds from her last word to the agent's first, and tamesu (an open-source testing framework) measured 3.2 seconds at the end of the line (Step 7) — over three times the one-second allotment. The log shows where it went: 1.21 seconds to close the turn while the final transcript arrived, half a second for the LLM to call get_appointments, another half second to start the answer, and a full second to the first TTS audio. Each number points to a fix: an STT with faster finals or its own endpointing, the turn detector model instead of plain VAD, a short filler before slow tools, and models served near your agent servers. What to watch. The following metrics will tell you whether the agent is healthy. The SDK has the data for all of them: MetricWhere it comes fromCall connection and completionSIP participant attributes and the session close_reasonResponse latencye2e_latency per turn, as p50 and p95STT, LLM and TTS errorsthe error event and the report's errors countersTool failuresToolError raises in your logsTransfer successthe outcome each transfer tool logsCost per callsession.usage per model, plus the carrier's per-minute rate After a test call, the session report from ctx.make_session_report() answers most questions in one file. The following is an excerpt from an appointment call: JSON { "room": "call-_+15550002_BFK5wQ3zB25J", "duration_ms": 47621, "user_turns": 2, "close_reason": "CR_USER_INITIATED", "errors": { "llm": 0, "stt": 0, "tts": 0, "realtime": 0, "interruption": 0 }, "usage": [ { "model": "gpt-4.1-mini", "input_tokens": 3193, "output_tokens": 73 }, { "model": "gpt-4o-mini-tts", "input_tokens": 69, "output_tokens": 561, "audio_duration": 21.77 }, { "model": "gpt-4o-mini-transcribe", "input_tokens": 90, "output_tokens": 30, "audio_duration": 8.94 } ] } On LiveKit Cloud, session data appears in Agent Insights, with recordings and traces when agent observability is enabled. For Cloud-deployed agents, log drains forward runtime logs to Datadog, CloudWatch, Sentry, or New Relic. For fully self-hosted deployments, store the session report yourself, and export traces with OpenTelemetry. Step 7: Test the Complete Calling Workflow Before Launch Local audio tests confirm that the pipeline works under local test conditions. They don’t confirm that calls to the number reach the agent, that caller ID arrives, that telephone audio is clear enough for STT, or that a transfer reaches the front desk. Test in three layers. Unit tests on every commit. session.run() runs one conversation turn using text input and returns the results. You get assertions for messages and tool calls, an LLM judge to check whether a response matches the expected intent, and mock_tools to simulate tool failures: Python def called(result, name: str) -> bool: try: result.expect.contains_function_call(name=name) return True except AssertionError: return False @pytest.mark.asyncio async def test_billing_gets_a_warm_transfer(clinic_backend): # fixture: the mock clinic API async with openai.LLM(model="gpt-4.1-mini") as judge, AgentSession(llm=judge) as session: await session.start(Receptionist(customer=MARIA)) first = await session.run(user_input="I was charged twice on my last bill. Can I talk to someone?") second = await session.run(user_input="Yes please, transfer me.") assert any(called(r, "transfer_with_handoff") for r in (first, second)), "no warm transfer" await second.expect[-1].is_message(role="assistant").judge( judge, intent="Explains the transfer is not possible right now and offers a callback or to take a message.") Plain Text $ pytest -q 7 passed in 22.13s The seven tests cover the greeting, the lookup, confirmation before booking, both transfers, and a backend outage. Write judge intents without ambiguity: “offers a transfer or a callback” may be interpreted as requiring both options. If either is acceptable, write “offers at least one of these options: a transfer or a callback; either option alone is sufficient.” Simulated callers on every pull request. Agent Simulations, a LiveKit Cloud feature, runs an LLM-driven caller with a goal against your real agent and grades the whole conversation. Run each scenario several times, since the caller and the judge are both models. Real phone calls before launch. tamesu is an open-source testing framework whose siptarget calls your agent through a LiveKit SIP service and a trunk you provide, like a customer would. A simulated caller with a persona and a goal speaks through TTS and listens through VAD and STT. It can interrupt or add background noise, while deterministic checks and LLM judges evaluate the call, regardless of the framework used to build the agent. SIP testing requires the full build from the releases page; the lite build does not include SIP support. YAML suite: receptionist-phone targets: clinic: type: sip direction: outbound to: "+15105550100" # the clinic from: "+15550002" # Maria's number, so the agent should recognize her ring_timeout: 30s max_call_duration: 3m tests: - name: a patient asks for her next appointment target: clinic persona: { name: Maria, style: "polite", language: en, voice: en-US-female } scenario: script: - wait_for: "Harbor Dental Clinic" # a real caller waits for the greeting goal: "You are Maria Lopez. Find out when your next appointment is, then say goodbye." max_turns: 4 stop_when: "the agent has said the date and time of the next appointment" assert: - judge: "The agent greets Maria by name and says her next appointment is a dental cleaning on November 12 at 10 in the morning" - metric: { name: greeting_ms, lt: 3000 } # call answered to the agent's first word - name: a billing question reaches the front desk target: clinic persona: { name: Maria, style: "firm", language: en, voice: en-US-female } scenario: script: - wait_for: "Harbor Dental Clinic" goal: "You were charged twice on your last bill and want to talk to a person. Say yes to a transfer. When someone from the front desk answers, say thanks and goodbye." max_turns: 5 stop_when: "someone from the front desk has answered the call" assert: - judge: "The agent offers to transfer the billing question, and then a person from the front desk answers the call" - call_ended_by: [agent, caller] Without the wait_for step, the simulated caller starts talking the moment the call connects, right over the greeting. tamesu sip setup creates the outbound trunk on your LiveKit SIP service, then tamesu doctor checks the secrets, the models, and the SIP service. Meanwhile, tamesu run places the calls and keeps a transcript, a stereo recording, and the call timeline of each run: Plain Text $ tamesu run tamesu run: 2 tests, output in .tamesu/runs/2026-11-09T14-16-40 PASS receptionist-phone / a billing question reaches the front desk 47.6s PASS receptionist-phone / a patient asks for her next appointment 48.2s 2 passed, 0 failed, 0 errors in 1m36s · 14 model calls, 3892 tokens, $0.0087 · llm cache: 0 hits, 6 misses Pre-Launch Checklist The number rings the agent from mobiles, landlines, and withheld caller IDs.Known patients are identified by caller ID, and unknown callers still get help.Every tool has a timeout and a ToolError path, and the agent never logs data it did not receive.Writes need a spoken "Yes" and cannot be interrupted, and the agent has no tool it should not have.Transfers work on a real call, and a failed transfer brings the caller back to the agent.Hang-ups, silence, and network drops leave no rooms open or jobs left behind.Latency, errors, transfer outcomes, and usage are logged and charted per call.Simulated callers at peak load plus a margin stay under load_threshold.The hard cases pass: speakerphone in a car, accents, interruptions, long silences, requests the agent has no tool for, a backend or model provider down, and a front desk that doesn't answer.Caller ID alone never unlocks sensitive actions. Callers hear a recording notice where consent is required, recordings and transcripts expire, and keys live in a secret store.Every model or prompt change reruns all of the above. Conclusion: Building Reliable AI Phone Agents An AI phone agent is a complete application, not a language model wired to a microphone. What makes it production-ready is invisible in a demo: a trunk and a dispatch rule that reliably reach the agent, swappable models, turn handling tuned for phone lines, tools with timeouts and limits, an escalation path that works when nobody picks up, and metrics for every call. Keep those parts modular, with narrow interfaces between them. Then you can test each one alone, replace a model when a better one ships, and scale agent servers without touching the trunk. Resources and Documentation LiveKit Agents, the voice AI quickstart and pipeline typesTurns and interruptions and function toolsTelephony: inbound trunks, dispatch rules, Twilio and testing your setupCall transfers, WarmTransferTask, and the warm transfer exampleDeployment, self-hosted deployments, and metrics and data hooksTesting: unit tests and Agent Simulationstamesu: the SIP target and SIP setupVoiceflow phone numbers and the LiveKit Twilio Connector Versions used: LiveKit Agents 1.8.10 on Python 3.12, LiveKit CLI 2.19.3, a self-hosted livekit-server 1.13.11, and LiveKit SIP service 1.10 with Asterisk as the carrier, tamesu 1.20.1, and GPT-4.1 mini for the agent and the judges.

By Xavier Portilla Edo CORE

Why Prompt Caching Matters for LLM Applications? Modern LLM apps often get the same, or almost the same, prompt more than once. This happens with form submissions, common FAQ questions, agent tool calls, or a user asking the same thing in different words. Without caching, each of these repeat requests pays full token cost, full latency, and adds unwanted pressure on your rate limits. There are four caching methods used in production today. This article, Part 1 of a 3-part series, covers the first and most basic one: exact-match caching. Part 2 covers semantic caching, for requests that mean the same thing but are worded differently. Part 3 covers what happens at the model-serving layer itself: self-hosted KV cache reuse and provider prefix caching, plus the production details every team ends up needing: watching your metrics, working out your costs, and a full checklist. Gateway or application exact-match caching. A hash lookup on a byte-identical or normalized request. Zero cost on a hit. Covered here.Semantic caching. Embedding plus a vector similarity search. Zero model cost on a hit for reworded queries. Covered in Part 2.Inference-engine KV cache reuse, for teams running their own models (vLLM, SGLang). Covered in Part 3.Provider prompt or prefix caching (Anthropic, OpenAI, Bedrock). Covered in Part 3. These layers stack. Check exact match first, then check semantic, then let the request reach the model, where either your own inference engine or the API provider reuses whatever part of the prompt stayed the same. The Full Picture Exact match sits right at the front of the stack. It is the cheapest check to run, so it should catch as many requests as it can before anything more costly runs. Layers 1 and 2 sit in front of the model and can skip the model call entirely. Layers 3 and 4, covered in Part 3, sit inside the model call itself and never skip it. They just make it much cheaper when a part of the prompt repeats. 1. Exact Query Caching (Baseline) Treat the prompt, or a cleaned-up version of it, as a key in a fast key-value store: Redis, Memcached, or an in-process LRU cache. Key Design Hash the full context that matters: system prompt, user message(s), tool definitions, and any settings (like temperature) that change the output.Clean up whitespace, and sort tool lists if their order does not change how the model behaves.Store the full response, or structured output, plus details like timestamp, model version, and TTL.For agents, the key needs more than just the user message. Include the full tool schema (name, description, parameters), any tool-call results you fold back into the context if they are always the same, and leave out anything you don't want changing the key, such as a request ID or trace ID that changes every call but doesn't affect the output. Two common ways a cache silently breaks here: tool definitions listed in a different order between calls, and JSON keys written in a different order by different parts of the same codebase. Sort both before hashing. Don't count on the order matching by chance. Where Should the Cache Live? The store you pick changes how it can fail, more than most teams expect at first: In-process LRU (a local dict, or a library like cachetools) is the fastest choice, under a millisecond, with no network call. It only sees traffic hitting that one process, so on a deployment with many instances, each instance builds its own cache on its own, and the hit rate stays low until every instance has warmed up. This is fine for small deployments, or for caching things that are cheap to redo per instance anyway.A shared store (Redis, Memcached) gives every instance the same view, so a hit on instance A also counts as a hit for a request landing on instance B. This is the right default once you go beyond a single instance, at the cost of a network round trip, usually a few milliseconds on the same network.A two-level setup (a small in-process LRU in front of shared Redis) gives you the best of both: the busiest keys never leave the process, and everything else falls through to the shared store. Worth the extra work once a small number of keys account for most of your traffic, which is common on FAQ-style workloads. Handling Stale Data A fixed TTL, for example, 60 seconds, for time-sensitive data such as travel prices or stock levels.LRU eviction once the cache fills up.Clear the cache on purpose when something changes: a price update, an outage, a policy change. Python example (Redis): Python import hashlib import json import redis from typing import Optional r = redis.Redis(host="localhost", port=6379, db=0) def make_key(prompt: str, model: str, temperature: float = 0.0) -> str: payload = json.dumps({"prompt": prompt, "model": model, "temp": temperature}, sort_keys=True) return "llm:exact:" + hashlib.sha256(payload.encode()).hexdigest() def get_exact(prompt: str, model: str, temperature: float = 0.0) -> Optional[str]: key = make_key(prompt, model, temperature) try: return r.get(key) except redis.RedisError as exc: # Cache being down should never break the request. # Log it and fall through to the model. logger.warning("exact cache read failed: %s", exc) return None def set_exact(prompt: str, response: str, model: str, temperature: float = 0.0, ttl: int = 300): key = make_key(prompt, model, temperature) try: r.setex(key, ttl, response) except redis.RedisError as exc: logger.warning("exact cache write failed: %s", exc) Treat the cache as a nice-to-have, not something the app depends on. If the cache goes down, fall back to full model calls; don't fail the request. Should You Cache Errors? Most production systems say no, or only in some cases, and it is easy to skip thinking about this until something goes wrong. A few good defaults: Never cache a 429 (rate-limited) or 5xx error from the model provider as if it were a real answer. Retry with backoff instead of remembering the failure.If your own guardrail or content filter rejects a response, that is a judgment call. Some teams cache the rejection briefly (a few seconds to a minute) just to stop a broken or hostile client from triggering the same expensive rejection path again and again. Others never cache rejections at all, so they don't end up serving a stale "rejected" result after the input has been fixed.If you do cache rejections, use a clearly different TTL and a different key prefix (llm:reject: instead of llm:exact:) so you can clear them, or leave them out of your hit-rate numbers, separately from real answers. Warming the Cache A cache that just went live is empty, and the first few minutes after any deploy that clears the cache see a drop in hit rate. If your logs keep old prompts and responses, a simple warm-up job that loads the most common prompts from the last few days into the new cache, before it takes real traffic, avoids that drop. This matters most for FAQ-style traffic where a small number of questions make up most of the volume. Request Coalescing: Solving the Thundering Herd Exact-match caching has one failure mode that only shows up under real load: if fifty identical requests arrive at almost the same time, before the first one has filled the cache, all fifty can miss, and all fifty can call the model on their own. This is called a cache stampede, and it is not rare at real traffic levels. A viral post, a batch job starting up, or a retry storm from another system can all set it off. How to Fix It Request coalescing, also called single-flight. The first request for a key takes a short lock (or sets an in-flight marker), and the other requests wait on that same result instead of calling the model on their own. Redis's SET NX pattern, or a library like singleflight in Go, handles this cleanly.Cache the "in progress" state too, not just finished responses, so other requests can see "someone is already working on this" instead of racing to do it themselves.Put a fair timeout on the wait, with a fallback to a normal model call if the in-flight request does not finish in time, so one stuck request does not hold up every other request waiting on it. Without this fix, exact-match caching can actually make traffic spikes worse, not better. Every process learns about a busy key at almost the same moment and hits the model all at once, right when the model can least handle it. When It Shines Templated queries, form-driven flows, test suites, batch jobs — anywhere the same string keeps coming back. Hit rates can be very high in narrow domains and close to zero in open, free-form chat, which is exactly the gap semantic caching, covered in Part 2, is built to close. What's Next Exact-match caching only helps when the input matches exactly, which is common for templated and form-driven traffic but rare in open conversation. Part 2 covers semantic caching: matching reworded questions with embeddings, doing it safely with a verification step, and the invalidation, multi-turn, and security issues that come with trusting a similarity score instead of an exact match. Further Reading Redis documentation, SET command options (including NX for locking patterns)Redis documentation, key eviction policies (LRU, LFU, TTL-based)

By Ram Ghadiyaram CORE

Software teams are adopting AI agents to write, run, and repair tests faster than ever. The breakthrough is usually not the model alone. ChatGPT and Cursor can share the same underlying intelligence, yet one feels like a copy-and-paste assistant, while the other feels like a teammate that can edit files, run commands, and iterate until the job is done. The difference is the agent harness: the orchestration layer that wraps the model with instructions, tools, context, and guardrails so raw reasoning becomes reliable action. In test automation, that gap matters even more. A failing scenario can produce thousands of lines of Playwright output, span page objects, step definitions, test data, and multiple environments. Dump all of that into a single long chat, and the model slows down, costs rise, and decisions get noisier. A well-designed harness treats the model as a planner, not a dump truck. It indexes the codebase so the agent can ask structural questions instead of reading entire files. It delegates test execution and file generation to smaller, isolated workers that return compact summaries. It validates credentials, environment profiles, and reference fixtures before a run starts, so failures are caught early instead of mislabeled as “broken locators.” And when something does break, it routes the failure through a governed path: diagnose, fix, audit the patch, re-run only what changed, report results in a form humans can act on. This article walks through that architecture in practice: what an agent harness is, how memory and code indexing reduce token waste, why tiered model routing balances cost and quality, and how governance gates keep AI-assisted automation predictable rather than magical. Whether you are building on Cursor, Claude Code, or your own SDK integration, the pattern is the same. The model is the engine. The harness is what makes it drivable at scale. Agent Harness An agent harness is the software infrastructure that wraps around an AI model (like Claude, GPT-4) to turn it into an autonomous working agent. Harness acts more like an operating system, providing the model with memory, tools, execution environments, and guardrails to complete multi-step tasks in the real world What the Harness Does Harness handles three things: Instructions – system prompts, rules (like the CLAUDE.md), skills, and subagent definitions that shape behavior.Tools – file read/write, terminal, search, MCP servers, browser automation, etc.The loop – the model proposes an action → the harness executes it → results go back to the model → repeat until the task is done Beyond that, a good harness also manages: Context – which files, docs, and tool outputs the model sees (and when to trim them)Orchestration – routing tasks to specialized subagents (planner vs. runner vs. reviewer)Safety boundaries – sandboxing, permissions, hooksTermination – knowing when the job is finished vs. stuck What can a typical agent harness look like: Agent harness How Agent Harness Can Be Achieved Agent harness cannot be achieved by a single library — it is a layered orchestration system built from Claude Code config files under .claude/, and CLAUDE.md as the control plane. The codebase needs to add rules for who does what, when to stop, and how information flows back. Control pane – CLAUDE.md + .claude/settings.jsonSkills – Orchestrated WorkflowSubagents – constrained workersInfrastructure – MCP tools + hooks Control Pane Considering we are using Anthropic's ecosystem, the main session is always Opus (planning, judgment, MCP). Heavy but repetitive work goes to Haiku subagents in isolated contexts. CLAUDE.md defines the routing table and hard rules — e.g., never run Playwright directly in Opus; always delegate to test-runner: Markdown ## Tiered model routing (Opus parent + Haiku runner) This project pins the main Claude Code session to **Opus** (`.claude/settings.json` → `model: claude-opus-4-7`) and delegates **test execution only** to a Haiku subagent defined in `.claude/agents/test-runner.md`. ... 1. When the user says "run", "execute", "verify", or "re-run" a scenario id / tag / feature file → invoke the `test-runner` subagent. Do not run `npx playwright test` directly in the main session. ... 4. If `failure_category` is `auth`, `config`, `network`, or `other` → do **not** self-heal. Surface the JSON report and stop. Skills: Orchestrated Workflows Skills are markdown playbooks the Opus session follows. They encode multi-step pipelines with explicit gates. /selfheal — failure → fix → verify: Markdown # Self-Heal Test 0. **Preflight.** Invoke `/preflight` ... 0.5. **Fixture check** ... 1. Run the failing scenario(s) via the `test-runner` subagent. 2. Inspect failures via MCP Playwright browser snapshot ... ... 5. **Invoke the `heal-edit-reviewer` subagent** ... 7. Identify **all affected scenarios** via `get_impact_radius_tool` ... 8. **Re-run in BOTH env1 and env2 — in parallel.** /codegen — CSV → BDD with planner/renderer split: Markdown 0. **Preflight.** ... 2. **Plan reuse via graph.** ... `semantic_search_nodes_tool` ... 6. **Invoke `codegen-writer`** with the BuildSpec. 7. **Audit the `.feature` file** via `feature-reviewer` ... 8. **Audit the emitted `.ts` files** via `heal-edit-reviewer` ... 10. **Smoke-run one scenario.** ... via `test-runner` ... Skills are the harness’s state machine — they define order, stop conditions, and which actor handles each step. Subagents: Constrained Haiku Workers Each subagent in .claude/agents/*.md is a specialized worker with a narrow tool set, fixed output schema, and hard prohibitions. Example — test-runner: YAML --- name: test-runner description: ... Does NOT self-heal, edit code, or modify any files. model: claude-haiku-4-5 tools: Bash, Read, Glob --- It must return strict JSON (~200 bytes) instead of flooding Opus with Playwright stdout: JSON { "status": "pass" | "fail", "exit_code": <number>, ... "failure_category": "locator-timeout" | ... | null, "stdout_tail": "<last ~30 lines of stdout/stderr>" } So the different subagents that the codebase harness are: Subagent harness MCP servers extend what the harness can touch: Playwright browser – live UI inspection for locator discovery and debuggingcode-review-graph – structural + semantic code index (avoid reading full files) Hooks in .claude/settings.json run automatically outside the agent loop: Hooks used in the Claude ecosystem Memory and Indexing: Two Complementary Systems “Memory indexing” here is not one thing. The project uses structural code indexing (knowledge graph) and persistent session memory (conventions/feedback). They solve different problems. Code Knowledge Graph (Structural Indexing) Plugin: code-review-graph v1.8.x Update cadence: Incremental after every Edit/Write via PostToolUse hooks (~2s typical)What gets indexed (Tree-sitter AST parsing): AST parsing output Each node stores: qualified name, file path, line range, parameters, return type, SHA-256 hash (for change detection). Semantic layer: After embed_graph_tool, all ~611 code nodes get 384-dim vectors (all-MiniLM-L6-v2). This powers meaning-based search — e.g., CSV step "Validate that the username field is present" matches Username.verifyUsernameFieldVisible() even when names differ. Primary MCP tools: Graph-first rule (from CLAUDE.md): Query the graph before reading files. Read narrowly with offset/limit only when you need exact insertion points. Session Memory (Governance & Conventions) Location: .claude/memory/ (initialized from .claude/memory-template/) Purpose: Persist decisions and feedback across sessions so the agent doesn’t re-learn conventions every time. Documented memory categories include: Session memory is always loaded alongside CLAUDE.md. It governs how the agent behaves; the knowledge graph governs what exists in the codebase. How Indexing Is Used in Practice During/codegen (CSV → BDD): Opus parses CSV → scenario titles and stepsFor each step: semantic_search_nodes_tool("<step text>") → reuse existing step or mark reuse: nullGraph checks existing page object methods, fixtures, scenarios before emitting anythingHaiku codegen-writer renders files from BuildSpec only — no judgment, no MCPReviewers gate output before lint/smoke run During/selfheal (locator failure): test-runner returns compact JSON (not raw stdout in Opus context)Opus scopes via graph: callers_of(brokenMethod) → read only affected stepsMCP browser diagnoses live UIAfter fix: get_impact_radius_tool → re-run only affected scenariosParallel test-runner on redama + fedramp During reviews: /code-review-graph:review-delta — changes since last commit, target <800 tokens context/code-review-graph:review-pr — full PR with blast-radius and test-coverage gaps Tiered Model Routing (Cost + Quality Governance) The harness explicitly separates judgment from execution: Tiered routing architecture Haiku costs ~1/15th of Opus per token — so even when total system tokens rise slightly, the dollar bill drops. Clean Governance of AI Usage Preflight Gates (Fail Fast, Don't Self-Heal the Unhealable) Governance here means predictable behavior, auditable outputs, and hard stops on bad patterns — not just “use AI less.” Separation of Concerns (No Guessing) If spec is ambiguous → error JSON, not a guess Hygiene Rules Enforced By Reviewers Context Budget Rules (9 Rules in CLAUDE.md) Graph-first queriesNarrow reads (offset/limit)Narrow MCP snapshotsDelegate bulk I/O to subagentsTrim subagent payloads (no stdout on pass)Reuse over rewrite (semantic search before authoring)Impact-radius re-runs (not whole suite)No echo-back of raw artifactsNo cat/head/tail on source Automatic Enforcement via Hooks From .claude/settings.json: Hooks run silently — governance without Opus token spend. Output Contracts (Structured, Not Prose) Every subagent returns strict JSON. The parent renders tables — never pastes raw JSON or full traces to the user. This is both UX governance and context governance. End-to-end: One failure, fully governed Conclusion The value of AI in test automation is not unlocked by choosing a larger model. It is unlocked by building a harness that knows what to delegate, what to validate, and what to keep out of the planner’s context. At a high level, four principles hold across any stack: Separate judgment from execution. One session plans, inspects, and decides. Specialized workers run tests, emit files, and audit changes — each with a narrow mandate and a structured response format. That boundary is what prevents a single conversation from drowning in logs and generated code. Validate before you execute. Credential sessions, environment profiles, and reference fixtures should be checked at the door. Failures that originate in setup or data drift must not enter a self-heal loop designed for code and locator problems. Fail fast, report clearly, and reserve automated repair for repairable categories. Index before you read. A maintained structural and semantic index turns broad codebase questions into small, precise queries. That reduces cost, speeds up triage, and keeps the planner focused on the lines that actually need to change. Govern every exit path. Delegation rules, audit gates, bounded heal attempts, parallel multi-environment re-verification, and human confirmation on data substitution are not bureaucracy — they are the difference between a demo and a production workflow. Taken together, these standards describe an agent harness that teams can adopt incrementally: start with an execution worker and structured JSON results, add preflight gates, introduce a code index, then layer in render and review workers as codegen volume grows. The model remains the reasoning engine throughout. The harness is what makes that reasoning trustworthy enough to run unattended, cheap enough to run often, and clear enough that engineers still know what happened when something goes wrong.

A Claude Code skill is a Markdown document that the assistant loads when a task matches its description. A collection of skills, agents, and hooks can be packaged as a plugin and distributed through a marketplace. Since the summer of 2026, my development environment has relied on such a plugin for Java work, consisting of a set of Spring Boot, Quarkus, Apache Camel, and Jakarta EE agents and skills. This collection started as a small set of artifacts but, little by little, I extended it such that, after several months, it contained 43 files and 17,338 lines. This raised a precise question: were the stack conventions provided by this plugin really contributing anything measurable to the final outcome, or could the model, by its intrinsic general knowledge, infer the same result with much fewer tokens? This article describes how the Anthropic evaluation tool answered the question, and what the measurements revealed about the skills and the evaluation method itself. A Criterion for Useful Skill Content A skill contributes when it changes what the model produces. Four kinds of content satisfy that criterion: Personal or team conventions, which the model has no means to infer.Facts more recent than the model's training data, such as new versions and renamed artifacts.Non-obvious pitfalls and the decisions that avoid them.Corrections to guidance that is wrong elsewhere, including in upstream plugins. Textbook material, such as the definition of an Enterprise Integration Pattern or the meaning of an HTTP status code, belongs to what the model already knows. Trying to provide this same knowledge in skills consumes between 5,000 and 8,000 tokens each time a skill loads. Additionally, the given skill might be outdated. For example, my files still referenced Spring Boot 3.2.1 months after the move to Spring Boot 4. Restructuring Before Measuring The first step was to collect all the loose folders in ~/.claude, defining agents, skills, and hooks, to remove the duplicates and to consolidate them in stand-alone plugins. The result is a marketplace named simplex-software that lists five plugins: PluginActivationspring-boot-templateper projectquarkus-templateper projectjava-extrasuser scope, every sessioncamelper projectjakarta-eeper project The stack plugins take precedence on every conflict. java-extras holds personal rules, cross-stack decisions, operational notes, and a UserPromptSubmit hook that reminds the model, on every prompt, which skills apply to the detected stack. Activation per project keeps stack material out of sessions that don't use it. The Measurement Method Claude Code provides claude plugin eval, an open-source evaluation tool allowing you to test the pertinence of your skills, agents, and plugins. For example, you might want to evaluate the efficiency of a given Claude Code plugin. To do that, you need to define "test cases," execution scenarios that specify an input prompt, the required context or tools, and the logic or criteria for determining success. A case is a directory with a prompt and a set of graders. The harness runs each case with and without the plugin, several times per arm, and reports the score difference. A case whose score difference stays near zero identifies a plugin that contributes little to the content, since the model can produce the same result without it. Such a plugin must be ablated because its skills add only tokens. Instead of creating new files in the project, each case asked Claude to provide the requested code and configuration directly in its answer. Therefore, they don't require a prepared sample project, write permission, etc. The graders examined the answer text. Each case combined three kinds of graders: Regular expressions for the specific facts the skill claims. These cost nothing.A judge model (Claude Haiku by default) with three votes for overall correctness.A tool_used grader that checks whether Claude invoked the skill through the Skill tool. It shows whether the plugin arm actually loaded the skill. It is excluded from the score because the arm without the plugin has no skill to invoke and would fail it in every run, which would inflate the difference. A case for the Camel plugin illustrates the format. The prompt file: Markdown --- name: quarkus-route-test tags: [camel, quarkus] max_turns: 8 allowed_tools: [Skill, Read, Glob, Grep] --- Quarkus 3 project using Camel Quarkus. A RouteBuilder defines route id `process-route` from `direct:process` to `kafka:processed-orders`. Write a test that verifies one message reaches the Kafka endpoint without a real Kafka broker being used for that endpoint. Include the Maven test dependencies. Its regular expression grader, graders/test-artifact.md: Markdown --- type: regex pattern: "camel-quarkus-junit(?!5)" match: contains --- Uses camel-quarkus-junit, the successor of the deprecated camel-quarkus-junit5. Its judge grader, graders/judge.md, whose body is the criterion the judge model applies to the answer: Markdown --- type: llm --- Pass only if ALL hold: - test dependency is org.apache.camel.quarkus:camel-quarkus-junit (not camel-quarkus-junit5) - test extends CamelQuarkusTestSupport, overrides isUseAdviceWith() to return true, advises route "process-route" (mockEndpoints / mockEndpointsAndSkip) and then starts the context - does not use @MockEndpoints (not available in Camel Quarkus) - no Thread.sleep Its skill indicator, graders/skill-fired.md, which passes when Claude loads a skill whose name contains camel: Markdown --- type: tool_used tool: Skill input_match: "camel" --- Indicator only: the plugin skill was loaded. The command runs every case of a plugin, three runs per arm, with a cost ceiling: Shell claude plugin eval . -j 3 --no-publish --max-cost-usd 5 First Results Ten cases covered the three own plugins. The table reports the mean score with the plugin, without it, and the difference. Two scores are corrected for grader defects described in the next section. PluginCaseWithWithoutDifferencecamelCamel Quarkus route test1.000.33+0.67camelSpring Boot route unit test1.000.33+0.67camelSpring Boot Kafka integration test1.000.92+0.08jakarta-eeintegration test on WildFly0.890.33+0.56jakarta-eepom setupequalequal0jakarta-eeresource, service and injectionequalequal0java-extrasCircleCI with Testcontainers1.000.33+0.67java-extrasgraceful shutdown on Kubernetes1.000.50+0.50java-extrasSpring Boot to Quarkus migration1.000.89+0.11java-extrasReactor to Mutiny translation0.560.67about 0 The cases with a large difference show that the plugin had a consistent contribution to the final result. In the case of Camel, for example, without the plugin the model estimated the Maven artifact version as 4.18.0, whereas the skill states 4.23.0, built on Spring Boot 4.1.1. It also selected the test artifacts camel-test-spring-junit6 instead of camel-quarkus-junit with the skill. For CircleCI it chose a plain Docker executor, which offers no Docker daemon to Testcontainers. For graceful shutdown, it omitted the arithmetic that relates the preStop delay, the application shutdown timeout and terminationGracePeriodSeconds. The cases with a difference near zero show the opposite pattern. The Reactor to Mutiny cheat sheet, the Jakarta EE pom and the CDI constructor injection rules describe knowledge the model already applies correctly. Reading the answers confirmed it: both arms produced the same code. Trimming, and the Regression the Evaluation Caught The trimming pass removed content with near-zero differences and reduced the rest to facts, decisions, and pitfalls. The Reactor to Mutiny section disappeared. The Spring Boot to Quarkus table kept only the rows where a mistake is likely; the Jakarta EE setup became a short list of facts and The Camel Kafka test skeleton became one sentence. The second run, after trimming, confirmed that every case with a large difference scored 1.00 with the plugin. Only the migration from Spring Boot to Quarkus case declined from 1.00 to 0.78, but part of the decline came from judge noise. However, a part was genuine: the shortened table row stated that the Quarkus @Transactional annotation has no readOnly attribute, which led the model to mistakenly remove the read-only behavior of the original method. Without the skill, the model preserved it through a read-only Hibernate session. The corrected rule prescribes that the intended behavior should be achieved via a read-only Session with FlushMode.MANUAL, or a org.hibernate.readOnly query hint. A third run of that case, judged by Claude Sonnet with a criterion that checks the read-only behavior, passed six runs out of six. The episode is the strongest argument for the evaluation: a detail that appeared harmless changed the generated code, and only the measurement detected it. Final State and Cost MeasureBeforeAfterAgent and skill files436Lines of agent and skill content17,338638Tokens added to every sessionabout 2,100about 520, plus about 150 for each optional plugin enabled in the projectEvaluation casesnone6, each with a measured difference The complete exercise, including the exploratory run, both full runs and the verification run, cost about 20 US dollars in model usage. A full run of the remaining suite costs about 6 dollars and takes about five minutes with three parallel runs. Lessons The useful content of a skill is narrow: versions, renamed artifacts, corrections and non-obvious conventions. Explanations of well-known frameworks contribute nothing measurable.An evaluation with and without the plugin is inexpensive and decisive. It replaces a debate about relevance with a number per case.Graders require the same scrutiny as the skills, and a small judge model is inconsistent on long answers.A reduction of content is a change like any other and deserves a regression test. The readOnly episode demonstrates it. Limitations Each arm ran three times, which indicates a direction rather than a statistically established effect. The results describe one model at one date. Here, our Camel plugin pinned version 4.23, which a later model may know without assistance. This is precisely why the suite should run again after each model change. The cases reflect one developer's projects, and the personal rules in `house-rules` were deliberately left out of the evaluation, since the model has no means of guessing them.

By Nicolas Duminil CORE

Every GitOps setup meets the same moment: something is on fire, someone runs kubectl scale on prod, and the fix works. What happens next depends on two settings that rarely get checked before an incident: self-healing on the GitRepo, and pause on the cluster. I ran all three combinations in a lab on Rancher 2.15.1 with Fleet (biased, I work at SUSE). Three EKS clusters: dev and staging on Kubernetes 1.36, prod on 1.35. Cases 1 and 2 used the guestbook app from the public rancher/fleet-examples repo, with the frontend Deployment at 3 replicas. Case 3 used the podinfo Helm chart from my own repo. Every manual change went through the scale button in the Rancher UI, which sends a PATCH to the Deployment's scale subresource, the same request kubectl scale makes. The Two Settings Self-healing is spec.correctDrift.enabled on the GitRepo. It is off by default. In the Rancher UI it sits in step 4 of the GitRepo wizard, Advanced, Resource Handling, "Enable self-healing". To change it on an existing GitRepo: Continuous Delivery, App Bundles, the GitRepo, the three-dot button top right, Edit Config, then Next until step 4. Pause lives on the cluster: Continuous Delivery, Clusters, three-dot menu, Pause. Fleet stops deploying to that cluster until you unpause it. 1. Self-Healing On Turning self-healing on for a GitRepo that already had drift fixed the drift in about 11 seconds. Then I scaled the frontend from 3 to 1 again. Fleet had it back at 3, with 3 pods ready, in under 7 seconds. The GitRepo flickered NotReady for about 3 seconds and went back to Active. Anyone watching the UI would likely never see the drift. That is what you want from GitOps on a normal day. During an incident it works against you: your fix is gone before you can tell whether it helped. 2. Self-Healing On, Cluster Paused I paused the dev cluster and scaled the frontend to 5. It held, 5 of 5 ready, for the twelve minutes the cluster stayed paused. While the cluster was paused, the GitRepo read Ready, 2/2 clusters, with no Modified state anywhere. The app looked healthy in Continuous Delivery while one cluster ran something Git did not say. If you review drift by looking at the GitRepo, a paused cluster drops out of the review. When I unpaused, the frontend went from 5 back to 3 in about 9 seconds. The GitRepo showed Modified for a moment and then Active. The fix lasted exactly as long as the pause. 3. Self-Healing Off With self-healing off, the manual scale stays, and Fleet reports it precisely. I scaled prod from 2 to 4 and the GitRepo went to Modified, 2/3, and stayed there. The Ready condition read: Modified(1) [Cluster fleet-default/c-kqgtk]; deployment.apps shop/shop-podinfo modified {"spec":{"replicas":2} That is the best drift report in the product. Two details to know when you read one: it names the cluster by its ID rather than its display name, and the number it shows is the one Git wants. The value currently on the cluster is missing. The surprise came with the next commit. The repo deploys one app in two lanes: two cluster groups, non-prod and prod, and one fleet.yaml with prod pinned a chart version behind. Promotion is a one-line commit that bumps the prod target. I promoted podinfo 6.14.0 to 6.15.0 while the manual scale was still in place. The GitRepo sat in WaitApplied, 2/3, and prod stayed on the old version with 4 replicas. The Helm release on the prod cluster recorded the reason: Upgrade "shop" failed: conflict occurred while applying object shop/shop-podinfo apps/v1, Kind=Deployment: Apply failed with 1 conflict: conflict with "Mozilla" with subresource "scale" using apps/v1: .spec.replicas This is server-side apply doing its job. The manual scale made the browser's client the field manager of spec.replicas. "Mozilla" is the user agent the browser sent, so managedFields records the client that made the call. The person who clicked appears nowhere in the object. From a terminal the manager would be kubectl. When Fleet's upgrade tried to set spec.replicas back to 2, the API server refused to take the field from another manager without being forced. None of that showed in Continuous Delivery. The GitRepo showed WaitApplied and the drift message. The conflict text existed only in the failed Helm release on the downstream cluster. The fix needed no force. I scaled prod back to 2, the value Git asks for, and the promotion landed about 40 seconds later: new chart version, 2 of 2 ready. The Kubernetes docs explain why that works. When two managers set a field to the same value, they share ownership and there is no conflict. What to Check on Your Own Clusters Read the setting before the incident. Check spec.correctDrift.enabled on each GitRepo that targets prod, so you know which of the three cases you are in.If you pause a cluster, keep a list of what you paused. Continuous Delivery will not show drift on it until you unpause.If a promotion sits in WaitApplied after an incident, look on the downstream cluster. Run helm history on the release and read the description of the failed revision.Inspect ownership with kubectl get deployment <name> -n <namespace> -o yaml --show-managed-fields. A manager like "Mozilla" or kubectl next to a field that Git also sets is a manual change waiting to block a deploy. Limits of This Test One run, on Rancher 2.15.1 with Fleet and EKS 1.35 and 1.36. Every timing is a single measurement, and the GitRepos polled Git every 60 seconds. I scaled from the UI, not from kubectl in a terminal. I did not test correctDrift.force, which the Fleet docs say can recreate resources, or Helm's force option for this case. After the fix, the browser client may still co-own spec.replicas. Per the server-side apply docs, a later commit that changes replicas could conflict again. I did not test that. What I'd Take From It Pause is a working break-glass. Use it, and keep in mind that while a cluster is paused, Continuous Delivery tells you nothing about it.Put the fix in Git before you unpause, or you lose it.With self-healing off, an emergency change that never makes it back to Git blocks your next deploy. If a promotion sits in WaitApplied after an incident, look for a field someone changed by hand. This is a lab finding, not official guidance.

TL; DR: The Product Backlog Refinement Agent and the Risk When a Suggestion Becomes a Decision Preparing a refinement session is tedious, and AI is good at the tedious part: reading every Product Backlog item, every repeated comment column, and the team’s own records on the afternoon before. The risk sits elsewhere. A refinement agent asked to “help with refinement” can take the conversation away from the team without writing a single acceptance criterion: “suggesting” to the Product Owner or Product Manager which items to close, merge, split, or create is enough, and the questions the team was supposed to work out arrive in the brief already answered. The remedy is a boundary decided before the first run: the agent marks evidence and asks questions, the Product Owner checks the brief and sets the agenda, and the team refines. Writing that boundary into an A3 Handoff Canvas, together with what the agent reads and who checks its output, turns a good intention into a rule the agent follows and the Product Owner can verify. Thesis: This article explains how Product Owners and Product Managers can use AI to prepare Product Backlog refinement without pre-empting the team’s conversation: four evidence checks, a four-part test separating suggestions from decisions, written boundaries, and human verification of the agent’s brief. Disclaimer: I read Charniak/McDermott’s book on “Artificial Intelligence” decades ago; of course, I use AI for research, translations, proofreading, challenging story arcs and article structures, creating artifacts such as graphics, and summarization. It is a production tool, not a substitute for thinking. The Danger of a Preparation Turning Into a Decision The familiar worry about AI in Product Backlog refinement is that a model writes the Product Backlog items, the acceptance criteria, and the estimates, and a tired team nods them through on a Tuesday afternoon. The worry is justified; however, it describes the easy case, because a drafted acceptance criterion is visible, and a team would usually notice that someone else did its homework. The more challenging case looks like help. Consider a hypothetical brief by a refinement agent that contains no descriptions, no acceptance criteria, and no new order, yet recommends closing one item as a duplicate, splitting another, creating a third, and calling four items “ready candidates” for the next Sprint. Every line reads as friendly advice to the Product Owner, and every line settles a question before the team has discussed it. A recommendation decides nothing until someone accepts it and acts on it. Once it sits on the agenda, however, the team is likely to discuss the recommendation instead of the question behind it, and that is how preparation quietly turns into a decision. It sneaks in through the back door, feeding on the human desire to avoid unnecessary work when organizational pressure is already high. What Refinement Is For, and What Preparation Should Leave Alone The Scrum Guide describes refinement as “an ongoing activity to add details, such as a description, order, and size,” which captures the mechanics and leaves out the purpose. I see refinement as the final part of product discovery, once hypothesis validation has been established: the team asks whether an item is really the best use of its time or whether something else would create more value for customers. A Product Backlog item is a token for that discussion, an invitation to align, and a good session ends with shared understanding sufficient to make the next investment decision, including the decision not to build something. Many teams hold a regular session for this, and often those sessions turn into a hunt for problems someone could have spotted the day before: an item a stakeholder insists on moving to the top, a backlog item without acceptance criteria that everyone expected someone else to write, an item nobody has touched for 6 months. Preparation exists to spare the team that kind of administrative work, and reading a Jira export and the team’s notes on the afternoon before is exactly the kind of thorough, tedious work an AI agent does well. The question is where preparation ends. In September, I described ten anti-patterns that get worse when teams add AI to a dysfunctional Product Backlog process, “Refinement Without the Team” among them, and an assistant that hands the Product Owner ready-made resolutions is a polite version of that anti-pattern: the team still meets, yet the decisions it was supposed to reach already arrive pre-made. In my view, the most critical consequence of dysfunctional refinement is a team that loses its ability to have productive discourse. Typically, building the wrong thing follows; think of lack of return on investment and opportunity costs. So my rule for such a refinement agent is short: mark, suggest, never solve. The agent marks what the team needs to talk about and may suggest what to discuss, and solving stays the team’s job. Four Things a Product Backlog Refinement Agent Can Check, and What It Cannot Know Of the 27 Product Backlog and refinement anti-patterns I have described, many are behavioral and leave no trace in a Jira export. Four do leave traces, provided the agent reads the team’s own records next to the export: the Product Goal, the product roadmap with its change log, the anti-product backlog, the notes from previous Sprint Reviews, and the actions from the last few Retrospectives. Readiness The agent looks at items early in the current order and reports what the records contain: whether an item states a problem or an intended outcome, whether it lists open questions, and whether a Product Backlog item waits for an open spike for technical clarification. Whether that is enough for a useful conversation remains the Product Owner’s judgment. A missing acceptance criterion at the start of refinement is no defect; it is an agenda question, because the Product Owner will have ideas about how an item should meet the desired outcome, and the team negotiates the criteria together by the end of refinement. Being early in the order does not preselect an item for the next Sprint, either: the order reflects the team’s best current judgment of how to use its time, and in Sprint Planning, the team first agrees on the Sprint Goal, and then the Developers select Product Backlog items they deem necessary to accomplish the goal. Goal Fit and Prioritization by Proxy The agent compares the Product Backlog with the Product Goal, the roadmap, and the anti-product backlog, which lists the ideas and requests the team decided not to build, either not now or not in this product, with the reason for each decision. It marks items that serve no part of the Product Goal and items that revive an idea the team already rejected. It also marks prioritization by proxy, which an export reveals only indirectly, because no outsider has write access to the Product Backlog: a stakeholder can neither create an item nor change its position. What the export does show is a member of the Scrum Team, usually the Product Owner, creating or moving an item on a stakeholder’s behalf: a comment such as “per the sales director’s request,” a label such as “mgmt-request,” a creation date right after an escalation, and no discovery decision and no link to the Product Goal behind the item. An item that skipped discovery because someone with political weight asked for it was never put to the validation test. However, the agent cannot know what the conversation behind the item looked like, or whether a rejected idea deserves a second look; that takes the people involved. Outdated Items Items nobody has touched for months, or that still name a release that shipped long ago, deserve a look, because a Product Backlog that turns into a storage for ideas hides the work the Product Goal needs. Age is a signal for inspection, nothing more: an untouched item can be the very work the Product Goal depends on, and a refinement agent that simply sorts by the “Updated” column will happily declare it dead without inspection. Missing Work The Scrum Guide calls the Product Backlog “the single source of work undertaken by the Scrum Team.” The agent can hold that claim against the team’s records: open items from the Sprint Review, Retrospective actions, and even work done outside the Sprint Backlog. A discrepancy is a finding. Whether it leads to a new item is for the Product Owner and the team to decide, because the work may sit in another team’s Product Backlog, stay outside it on purpose, or already be resolved. This check is also why the agent needs more than the export, since an export cannot show what is missing from it. For any of the four checks, “no supported finding” and “insufficient evidence” are valid results. However, the teams that would gain most from this preparation often keep the fewest records: no anti-product backlog, no product roadmap change log, Sprint Review notes that say little more than “stakeholders loved what we built,” and the reasons behind past decisions stored in the heads of the Product Owner and a few senior engineers, or buried in Slack threads. Such a team will get a brief full of “insufficient evidence.” That brief is still a finding because it shows where the team has no memory of its own decisions. Also, it helps only when people read it as a statement about their records rather than a failed run. Otherwise, the team has added AI on top of a dysfunctional system. A small start is enough to make the checks useful: the Product Goal, the last Sprint Review notes, and the last Retrospective actions all cover readiness and missing work, and a short list of recent “no” decisions, one line of reason each, gives the goal-fit check something to work with. In the agent’s instruction, name only the records the team actually keeps, so the brief does not report the same missing roadmap change log on every run. Where a Suggestion Turns Into a Decision A hypothetical example shows where this red line runs; the item keys, people, and dates are invented. A Jira export taken on Monday shows EXP-187, “Export for enterprise accounts,” created last Tuesday by the Product Owner, in the second position of the order, with the Priority field set to “Highest,” a one-line description, and the Product Owner’s own comment “Per the sales director’s request, moved to the top; a big customer is waiting.” The roadmap’s change log keeps the export feature in “Next,” no discovery decision mentions the request, and EXP-112, a CSV export item from June, may cover part of the same need. A brief that does its job marks this as prioritization by proxy and cites its sources: the comment, the creation date, the position in the order, the roadmap entry, and the missing discovery decision. It adds what it cannot know (which customer, and what problem that customer has), and it may suggest that the Product Owner revisit the request with the sales director before refinement. A brief that crosses the line writes “Close EXP-187 as a duplicate of EXP-112,” or “Move EXP-187 back down the order,” or “Ready candidates for the next Sprint: EXP-187.” Each of these settles a decision that belongs to the Product Owner, however helpful it sounds. What follows is the Product Owner’s work, and most of it happens outside refinement. The sales director’s request is exactly what my discovery article describes as a passive source: “Product requirements from internal stakeholders, which arrive unrequested and usually with political weight attached.” It goes through the funnel I described in AI-Assisted Product Discovery, where only what serves the general product vision and the current product goal passes the neck and earns a place in the work: it may become a hypothesis to validate with an experiment, an entry in the anti-product backlog with its reason, or an item the Product Owner places where it belongs. Until then, EXP-187 has no claim on the second position, until the Product Owner reconsiders the order of the Product Backlog. A sales director who can push an item to the top through the Product Owner points to an impediment for the Product Owner and the Scrum Master. Also, it is not an agenda item for the Developers, as refinement time stays reserved for work that has legitimately entered the Product Backlog. Questions that belong in refinement need the same discipline. Take a story that waits for a spike that is still open: “What do we need to learn from the spike before we can negotiate the story’s acceptance criteria?” leaves the conversation to the team. “Should we split the story by permission level?” ends with a question mark, yet it carries a solution, and it leaves the team to say yes or no to an answer the agent chose. A valid refinement suggestion by an AI agent has to pass four tests: It rests on the supplied records,It makes the uncertainty visible,It leaves materially different answers open, andIt does not assume the disputed resolution. Why be this strict about a polite suggestion? A ready-made answer shapes the judgment of the person who reads it. In two preregistered experiments with 2,582 participants, Sterling Williams-Ceci, Mor Naaman, and colleagues found that people writing about societal issues with a biased AI writing assistant shifted their attitudes toward its position, most of them unaware of the bias and its influence, and warnings before or after the task did not mitigate the shift (Biased AI Writing Assistants Shift Users’ Attitudes on Societal Issues, Science Advances, 2026). The study tested autocomplete suggestions during writing, which influenced people more than similar arguments shown as a static list. An AI-assisted refinement brief is a static list, so the study is a reason for caution and does not prove that a brief anchors a Product Owner. Writing the Line Down Before the First Run Writing the red line down before a refinement agent runs for the first time turns it from an intention into a rule. The A3 Handoff Canvas is the place for it: it records what the agent reads, what its brief looks like, who checks it, what the agent must never do, and the last part goes into the agent’s instruction word for word. For an AI-assisted refinement agent, the stop rules read: Never write acceptance criteria,Never estimate,Never split, reorder, or remove an item, and never move one to the anti-product backlog,Never advise closing, merging, splitting, creating, or editing an item; ask the team a question insteadNever propose what goes into the next Sprint or what the Sprint Goal should beNever invent a source, a reason, or a figure. (If a record needed for a check was not supplied, cannot be located in the supplied material, or cannot be read, state that limitation under “What I could not check.” Do not infer that the record does not exist. Do not treat an experiment, proposal, or unverified note as a completed decision or shipped release; state the exact status recorded in the source.) Three of these rules deserve a word. Acceptance criteria stay out because the team negotiates them by the end of refinement, and an agent’s draft would preempt that negotiation; a different workflow can still help with criteria after the team has talked. Estimates stay out because their value lies in the Developers’ discussion that reveals diverging ideas of the effort before work starts, as I argued in Estimates Are Useful, Just Ditch the Numbers; the number was never the point. And the last rule addresses a quieter error: a model that reads a planned experiment as a commitment, or a hypothesis as a result, turns the team’s open question into a fact nobody decided. In terms of the A3 Framework, this preparation is Assist: “AI expands options. You own the outcome.” The refinement agent works from a read-only export, so it cannot change the Product Backlog even if a rule fails. And if you already run an agent for product discovery, it needs its own Handoff Canvas, because it reads different inputs, delivers different output, and stops at a different line; I consider that AI transparency at the operational level. The Product Owner Routes Every Finding A brief that sounds thorough invites a dangerous shortcut: forwarding it to the team unread. Polished AI artifacts quickly fool people into believing that the AI is doing the real job. So the Product Owner opens the item or record behind every finding that could change the agenda and then looks at a few things the brief did not flag, for example, the items nearest the top that it never mentions, because a brief cannot report its own omissions. Then the Product Owner decides where each finding goes, and only some of them belong in refinement: Settled by the Product Owner: Housekeeping, such as an obvious duplicate created by mistake or a release field nobody updated, and restoring the order of the Product Backlog.Sent into discovery: Requests that skipped the funnel, such as the sales director’s export, and rejected ideas whose revisit condition may have occurred.On the refinement agenda: Questions that need the team’s knowledge, such as what a Product Backlog item needs before the team can negotiate its acceptance criteria.Raised as an impediment: Systemic causes, such as stakeholder pressure that bypasses discovery or records the team does not keep, for the Product Owner and the Scrum Master, or for the next Retrospective. Sharing the agenda with the Developers and the Scrum Master before the session gives them the chance to add the questions the brief missed. If a brief proposes resolutions anyway, correct the instruction and run it again; if it keeps failing, prepare the agenda by hand, the way the Product Owner hopefully did before the agent existed. The remaining failure mode relates to the Product Owner: Treating the brief as sufficient grounds to settle a question that needs the team’s knowledge brings back the very problem the stop rules were meant to prevent, only with a human signature. The brief must not become a substitute for a discussion that has not happened. Conclusion: Try It Before Three Refinements, Then Decide Treat the first runs as a local experiment. Note how long your usual preparation takes, then run the refinement agent on a sanitized export, in an AI account your company has approved, before three or four refinements. Record the total time, including checking the findings that proved useful, the findings the records did not support, and the omissions you noticed, knowing you won’t catch all of them. If checking and correcting the brief keeps taking longer than preparing the agenda by hand, stop, because the agent then costs more than it saves. Otherwise, ask the team whether refinement gained time for the critical discussion, and compare the results with your usual preparation; a few sessions give you the basis to decide whether to continue, adjust, or stop. If you would rather practice first, the AI4Agile Online Course v4, which launches on November 27, 2026, includes an exercise in which you build this refinement preparation agent: you fill its A3 Handoff Canvas, run it on a 40-item Product Backlog export and the team records of MegaBrain.io, the course’s fictional startup, check its brief, and set the Product Owner’s agenda before comparing your decisions with my worked solution. When the next brief arrives on your desk the afternoon before refinement, which of its suggestions already contains an answer your team has not given yet? Key Questions This Article on the Product Backlog Refinement Agent Answers How Can AI Help a Product Owner Prepare Product Backlog Refinement? A refinement agent can read the Product Backlog export and the team’s records on the afternoon before refinement and mark Product Backlog anti-patterns: items not ready for a conversation, items that serve no part of the Product Goal or entered at a stakeholder’s request without a discovery decision, outdated items, and work the Product Backlog does not show. The Product Owner checks the brief and routes each finding; the team refines. What Can an AI Agent Check in a Product Backlog Before Refinement? Four checks leave traces in an export and a team’s records: readiness, goal fit, outdated items, and missing work. Prioritization by proxy shows only indirectly, because no outsider can write to the Product Backlog: an item the Product Owner created or moved up at a stakeholder’s request, visible in comments, labels, and a missing discovery decision. None of the checks can establish what the team understands or still wants; the people involved need to do that. How Do You Tell an Open Question From a Disguised Recommendation? A question passes four tests when it rests on the supplied records, makes uncertainty visible, leaves materially different answers open, and does not assume the disputed resolution. “Should we split the story by permission level?” fails despite its question mark, because it carries a solution. “What do we need to learn from the spike before we can negotiate the acceptance criteria?” leaves the answer to the team. Should a Refinement Agent Write Acceptance Criteria or Estimates Before the Team Refinement? Not in a preparation brief. The Product Owner brings ideas about how an item should meet the desired outcome, and the team negotiates the acceptance criteria together by the end of refinement; a refinement agent’s draft would preempt that negotiation. The value of an estimate lies in the Developers’ discussion of diverging ideas of the effort; the Scrum Guide assigns sizing to “the Developers who will be doing the work.”

By Stefan Wolpers CORE

Finding an eval() call in a Python repository is easy. Understanding where its argument comes from takes more work. The same problem appears with shell commands. A scanner can point to subprocess.run(), but a reviewer still needs to trace the command through assignments, string formatting, and branches. Then comes the harder question: can someone outside the intended trust boundary control that input? I built rce-path to make that investigation more structured. It scans a local Python checkout, identifies modeled execution sinks, and records the input flow it can establish from the source. Each finding includes locations, ordered evidence, and the assumptions that still need review. The package also supports optional OpenAI and Ollama annotations. Static scanning works independently, so you can inspect the evidence before deciding whether to involve a model. The Problem I Wanted to Address A suspicious function call is a starting point for security review. It leaves several questions unanswered: Where did the argument originate?Which assignments or expressions carried it to the execution call?Does the path depend on a branch or an unknown value?Is the source request data, a function parameter, or local input?Which parts of the flow remain unresolved? These details matter because a function parameter is not necessarily remotely controlled. A request-derived value may pass through authentication or validation outside the scanner's view. A shell invocation using fixed literals needs a different investigation from one assembled from request data. My goal with rce-path is to give the reviewer a traceable path and make the remaining uncertainty visible. The Core Idea The current release, version 0.1.0, uses Python's abstract syntax tree and per-scope control-flow analysis to track modeled inputs toward execution sinks. A source is an input origin the analyzer recognizes. A sink is an operation that can execute Python code or a shell command. Between them, the analyzer follows supported assignments and expressions, including concatenation and f-strings. It also handles branches, early returns, and bounded loop analysis. The report preserves the steps connecting these operations. If a call's return value cannot be resolved, the scanner records that uncertainty rather than filling the gap with an assumed data flow. Targets are read as source data. A normal scan does not import the target package, install its dependencies, invoke its build hooks, or execute its code. What the First Release Covers The initial sink rules cover: eval() and exec(), including supported built-in aliases.os.system() and os.popen().Shell execution through subprocess.run(), call(), check_call(), check_output(), and Popen().subprocess.getoutput() and getstatusoutput(), which imply shell execution. Modeled sources include function parameters and Flask request inputs such as request.args, request.form, request.data, and request.get_data(). Environment variables, command-line input, stdin, and file reads are categorized as local inputs. The subprocess rules account for shell behavior. An unknown shell value introduces an explicit condition. Cases using shell=False have coverage limitations; they should not be interpreted as a general safety guarantee. This is a focused first release. It lets researchers and Python developers investigate supported execution paths without requiring an AI backend. A Small Example Consider this deliberately unsafe example. Save it as demo.py, but do not run it as an application: Python from flask import request import subprocess def lookup(): host = request.args.get("host", "") command = f"nslookup {host}" subprocess.run(command, shell=True) The security question is whether request data reaches a shell command. Here, the analyzer can model the request source, its assignment to host, its inclusion in the f-string, and the assignment to command before the shell invocation. There is no route registration in this snippet. It illustrates a data-flow pattern; it does not establish that the function is deployed or reachable over a network. To try rce-path, use Python 3.11 or newer and install the package from PyPI: PowerShell pip install rce-path Save the example as demo.py, then generate a Markdown report: PowerShell rce-path scan demo.py --format markdown --output report.md For this example, the report identifies the subprocess.shell rule at line 7, with static_path evidence and a network_request source. Its reachability field is network_source_modeled. The evidence follows the request read on line 5, assignment to host, f-string construction on line 6, assignment to command, and the execution sink on line 7. Those fields need careful interpretation. network_source_modeled means the analyzer recognized a network-input pattern. Authentication, deployment exposure, and actual exploit feasibility remain questions for human review. Using the Python API The command is named rce-path; the Python import is rce_path. The same scan is available through the public API: Python from rce_path import ScanConfig, scan if __name__ == "__main__": report = scan( "demo.py", config=ScanConfig(ai_enabled=False), ) print(report.summary) for finding in report.findings: print( finding.rule_id, finding.location.file, finding.location.line, ) for step in finding.steps: print( step.kind, step.location.line, step.description, ) The report and findings are typed Pydantic records. You can export them with report.to_json() or report.to_markdown() and use the structured fields in your own review workflow. Each finding carries a fingerprint, source category, evidence tier, source locations, and review state. Conditions, assumptions, and unresolved calls are available alongside the evidence. The report also records analyzed and skipped files, source content hashes, and coverage diagnostics. Fingerprints exclude the absolute checkout root and AI annotations, which helps compare the same finding across equivalent checkouts. They are location-based, so moving code can change a fingerprint. Where AI Review Fits An LLM can explain a finding, identify follow-up questions, or assess whether the available evidence supports an interpretation. In rce-path, that review is an optional annotation layer. Both provider implementations use a common evidence-and-review contract. Responses are validated, including references to evidence IDs. AI annotations do not rewrite the static finding or change its fingerprint. For a local Ollama review, install the extra: PowerShell pip install "rce-path[ollama]" Start your Ollama server and choose a model you already have installed: PowerShell rce-path scan demo.py --ai ollama --model YOUR_MODEL The default endpoint is http://127.0.0.1:11434. The package does not automatically download a model or fall back to a cloud provider. For OpenAI, install the corresponding extra and set OPENAI_API_KEY in your environment: PowerShell pip install "rce-path[openai]" rce-path scan demo.py --ai openai --model YOUR_MODEL --allow-remote-code YOUR_MODEL is a placeholder for an operator-selected model that supports the provider's required structured output. OpenAI and non-loopback Ollama endpoints require explicit permission to transmit code. The package sends bounded evidence slices and applies best-effort redaction of known secret patterns. Redaction can miss sensitive content, so review what you permit it to transmit. OpenAI requests use store=False, which is not a promise of zero retention. Review failures stay visible. A refused, unavailable, invalid, or budget-exceeded review is reported as unsuccessful rather than being presented as a completed assessment. Running It in a Review Pipeline For machine-readable output, scan a local checkout and save JSON: PowerShell rce-path scan ./checkout --format json --output findings.json You can also choose a findings policy: PowerShell rce-path scan ./checkout --fail-on static-path This fails the selected policy when the report contains a modeled static path. It does not mean the package has confirmed an exploitable vulnerability. The CLI uses three exit codes: the scan completed and the selected policy passed.the findings policy failed.an operational error, incomplete scan, or failed requested AI review occurred. The default policy is --fail-on none. Completion status takes precedence over the findings policy, so an incomplete scan cannot be treated as a successful clean scan. A repository-level rce-path.toml can configure rules, exclusions, and analysis budgets. Repository configuration cannot enable AI or grant permission to transmit code; those decisions require runtime opt-in. Current Limits rce-path uses bounded static analysis. Cross-function and cross-file flow, dynamic dispatch, closures, complex heap mutation, comprehensions, and precise exception handling are outside its current coverage. Branches are joined conservatively. A reported path can include conditions that are not jointly feasible at runtime. An unknown function is not automatically treated as a sanitizer, and a guard does not by itself prove an argument safe. For that reason, static_path means the analyzer established a path within its models. A sink_only finding identifies a sink whose input flow remains unresolved. Neither establishes remote code execution, and the package assigns no automatic CVSS score or exploitability probability. Read the coverage diagnostics alongside the findings. An empty completed report means "No modeled findings," not that the repository has been proved secure. What Comes Next The roadmap includes bounded local call summaries, additional framework and deserialization models, SARIF output, and separate human-approved validation workflows. These capabilities are planned for later phases and are not implemented in the current release. For now, I want rce-path to make a specific part of vulnerability review easier to inspect: how an input reaches an execution operation, what evidence supports that path, and what remains unknown. Start with a small file whose behavior you understand. Read the evidence steps, check the assumptions, and then apply the same workflow to a local repository. That is the most useful way to judge whether the package fits your research or code-review process. Project Links PyPIGitHubUsage and limitations