DZone Spotlight

An AI inference service makes a model available to an application through an API. When you ask an inference API a question, the application sends your prompt to the service. The service processes it and generates a response, often streaming the answer back in batches. Producing a correct answer is only part of the job. The service also needs to respond quickly, handle many requests, and use GPU compute resources efficiently. Measuring each incoming request helps services understand what users experience and investigate problems when performance bottlenecks occur. AI Inference Metrics Models process input in units called tokens. Hence, common metrics include time to first token (TTFT), time between tokens (TBT), and time per output token (TPOT). Other measurements could be end-to-end latency, input and output token counts, request rate, error rate, etc. Also, TTFT and TBT describe different analyses. A response might start almost immediately but stream slowly, or take several seconds to start and then finish. Measuring only total response time hides that distinction. Averages can also hide problems. Services often examine P95 latency. This helps reveal slower experiences that an average can mask. To have a deeper understanding of such service behavior, a service can emit a simplified log record like the one below when a request finishes. One record explains one request. Millions of such records may be used to analyze the service and answer questions such as “Why did service-side latency increase during a particular time window?” “Did service errors increase with incoming traffic?” etc. This article explores using Parquet, Vortex, and DuckDB for such analytical queries. JSON { "event_time": "2026-09-01T12:30:00", "model": "llama", "input_tokens": 100, "output_tokens": 200, "ttft_ms": 100 } Columnar Formats Most telemetry queries need only a handful of columns; for example, a high-latency investigation may need only one hour of records. Columnar storage lets query engines access selected columns, and data layouts can help them skip irrelevant data. One such columnar format, Parquet, organizes data into row groups, column chunks, and pages. Another format, Vortex, takes a compressed array approach that supports some operations without fully decoding the data. Neither design guarantees faster queries; the outcome depends on the data, query implementation, and work being performed. This article uses DuckDB as the query engine because it can run SQL queries against both Parquet directly and Vortex through its extensions without setting up a database server. Scenario: Investigating High Latency When an AI service feels slow, a common starting point is to examine requests from the affected time window. For example, “How many requests were handled?” “What was the P95 TTFT?” etc. This experiment explores how file format and row ordering affect the time it takes to query one hour of inference latency data. For this purpose, one million inference records are synthetically generated using the structure shown earlier. Each record contains an event timestamp, model name, input and output token counts, and time to first token (TTFT) in milliseconds. All timestamps are interpreted as UTC. A given record is written in two formats, i.e., Parquet with Zstandard compression and Vortex with default writer settings. For each format, time-sorted and shuffled versions are created. This produces the four files below. Plain Text sorted.parquet - Time sorted records, Parquet with Zstandard compression. shuffled.parquet - Shuffled records, Parquet with Zstandard compression. sorted.vortex - Time sorted records, Vortex with default writer settings. shuffled.vortex - Shuffled records, Vortex with default writer settings. The following query calculates request count, average TTFT, and P95 TTFT for a populated one-hour window. The query is then repeated against the shuffled files. Request counts and latency statistics are checked across all four configurations before timing results are compared. SQL SELECT COUNT(*) AS request_count, AVG(ttft_ms) AS avg_ttft_ms, QUANTILE_CONT(ttft_ms, 0.95) AS p95_ttft_ms FROM read_parquet('data/sorted.parquet') WHERE event_time >= TIMESTAMP '2026-09-01 12:00:00' AND event_time < TIMESTAMP '2026-09-01 13:00:00'; For Vortex, only the file reader and path are changed: SQL FROM read_vortex('data/sorted.vortex') Step 1: Generate the Data Files Python from pathlib import Path import duckdb # Refuse to overwrite an existing dataset. Path("data").mkdir(exist_ok=False) con = duckdb.connect() con.execute("SET threads = 1") con.execute("SET preserve_insertion_order = true") con.execute("INSTALL vortex") con.execute("LOAD vortex") print("Writer DuckDB:", duckdb.__version__) con.execute(""" CREATE TABLE logs AS SELECT -- One request per second; timestamps are interpreted as UTC. TIMESTAMP '2026-09-01' + i * INTERVAL '1 second' AS event_time, -- Synthetic model mix: 70%, 20%, and 10%. CASE WHEN i % 10 < 7 THEN 'llama' WHEN i % 10 < 9 THEN 'gemma' ELSE 'qwen' END AS model, -- Different hash constants give each field a distinct sequence. -- Input tokens: 100–8099; output tokens: 100–1099. CAST(100 + hash(i, 11) % 8000 AS INTEGER) AS input_tokens, CAST(100 + hash(i, 22) % 1000 AS INTEGER) AS output_tokens, -- Baseline TTFT: 100–499 ms; add 800 ms during the slow hour. CAST(100 + hash(i, 33) % 400 AS INTEGER) + CASE WHEN event_time >= TIMESTAMP '2026-09-06 12:00:00' AND event_time < TIMESTAMP '2026-09-06 13:00:00' THEN 800 ELSE 0 END AS ttft_ms FROM range(1000000) AS t(i) """) formats = { "parquet": "FORMAT PARQUET, COMPRESSION ZSTD", "vortex": "FORMAT VORTEX", } layouts = { "sorted": "event_time", # Timestamp breaks any hash ties, giving a deterministic order. "shuffled": "hash(event_time), event_time", } for layout, order in layouts.items(): for fmt, options in formats.items(): path = f"data/{layout}.{fmt}" con.execute(f""" COPY ( SELECT * FROM logs ORDER BY {order} ) TO '{path}' ({options}) """) print("Created", path) con.close() Shell Writer DuckDB: 1.5.5 Created data/sorted.parquet Created data/sorted.vortex Created data/shuffled.parquet Created data/shuffled.vortex Step 2: Run the Experiment Each configuration receives one warmup, followed by five measured executions. Execution order is randomized across configurations, and DuckDB is configured to use one query worker. File generation, sorting, and extension loading are excluded from query timings. Median query latency is reported alongside the observed minimum and maximum. File size is recorded separately. Two different kinds of latency are involved: TTFT describes the synthetic AI requests, while query latency measures how long DuckDB takes to analyze those records. The experiment measures the latter, including filtering, aggregation, and fetching the small result. Python import csv import math import random import statistics import time from pathlib import Path import duckdb con = duckdb.connect() con.execute("SET threads = 1") con.execute("LOAD vortex") print("Reader DuckDB:", duckdb.__version__) print("Vortex:", con.execute(""" SELECT extension_version FROM duckdb_extensions() WHERE extension_name = 'vortex' """).fetchone()[0]) query = """ SELECT count(*), avg(ttft_ms), quantile_cont(ttft_ms, 0.95) FROM {source} WHERE event_time >= TIMESTAMP '2026-09-06 12:00:00' AND event_time < TIMESTAMP '2026-09-06 13:00:00' """ jobs = [ (layout, fmt) for layout in ("sorted", "shuffled") for fmt in ("parquet", "vortex") ] times = {job: [] for job in jobs} rng = random.Random(42) reference = None # Round 0 is warmup; rounds 1–5 are measured. for repeat in range(6): rng.shuffle(jobs) for layout, fmt in jobs: sql = query.format( source=f"read_{fmt}('data/{layout}.{fmt}')" ) start = time.perf_counter() result = con.execute(sql).fetchone() elapsed = (time.perf_counter() - start) * 1000 if reference is None: reference = result assert result[0] == 3600 assert all( math.isclose(a, b, rel_tol=1e-9, abs_tol=1e-6) for a, b in zip(result[1:], reference[1:]) ), "Results differ across configurations" if repeat: times[(layout, fmt)].append(elapsed) rows = [] for (layout, fmt), values in sorted(times.items()): rows.append({ "format": fmt, "layout": layout, "size_mib": Path(f"data/{layout}.{fmt}").stat().st_size / 2**20, "median_ms": statistics.median(values), "min_ms": min(values), "max_ms": max(values), }) with open("results.csv", "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=list(rows[0])) writer.writeheader() writer.writerows(rows) print("\nRequests, average TTFT, p95 TTFT:", reference) print("\nFormat Layout MiB Median ms Min–max ms") for r in rows: print( f"{r['format']:9} {r['layout']:9} {r['size_mib']:7.2f}" f" {r['median_ms']:12.2f}" f" {r['min_ms']:.2f}–{r['max_ms']:.2f}" ) con.close() Shell Reader DuckDB: 1.5.5 Vortex: 9a8eb67 Requests, average TTFT, p95 TTFT: (3600, 1100.8336111111112, 1281.0) Format Layout MiB Median ms Min–max ms parquet shuffled 9.40 11.91 11.32–12.49 vortex shuffled 6.60 7.14 6.99–9.14 parquet sorted 6.79 2.75 2.30–2.86 vortex sorted 4.10 1.68 1.38–4.12 Takeaway: File Format vs. Row Ordering The selected hour contained 3600 requests, with an average TTFT of 1100.83 ms and a P95 TTFT of 1281 ms. These values describe the synthetic inference records, and the query times measure how long DuckDB took to calculate that summary. Time-sorted Vortex had the lowest median query time and smallest file size. However, ordering improved query performance substantially for both formats. Note that these findings apply to this synthetic dataset and hour window query after warmup. Timings varied across runs, and the results do not establish a universal format ranking or directly measure how much data was skipped. QuestionRESULTSTakeawayDid ordering matter more than format?Sorting improved median query time by 4.33x for Parquet and 4.25x for Vortex. Switching from Parquet to Vortex improved it by 1.64–1.67x within the same layout.Yes, for this workload. Time ordering had the larger effect.Did the smaller file produce the faster query?Shuffled Vortex was slightly smaller than sorted Parquet (6.60 vs. 6.79 MiB), but its median query time was about 2.6x longer.Not necessarily. File size alone did not predict query speed.Which configuration performed best?Sorted Vortex had the smallest file (4.10 MiB) and lowest median query time (1.68 ms). Sorted Vortex used the least storage and had the fastest median query time in this experiment. Conclusion AI inference services generate telemetry that helps explain token usage, identify latency changes, and investigate high latency issues. Efficient storage and querying can make that data less expensive to retain and faster to analyze. In this synthetic DuckDB experiment, time ordering improved median query performance. Sorted Vortex produced the smallest file and fastest median query, but sorted Parquet outperformed shuffled Vortex, highlighting the importance of organizing telemetry around its access patterns. These local warm-up-based results do not establish a universal format winner or demonstrate faster inference. Instead, they illustrate how format choice and data layout can support more efficient AI telemetry analysis, making it easier to investigate performance problems before deciding how to address them. References Build a DIY AI Model Hosting Platform With vLLMUnderstanding Inference Time ComputeFile Format, Parquet Vortex DocumentationDuckDB Documentation, VortexReading and Writing Parquet Files, DuckDB Docs

By Bhala Ranganathan CORE

Kafka is a distributed event streaming platform that enables real-time data ingestion, storage, and processing of continuous event streams. To do that, Kafka has a complex internal structure. In this part, I will try to cover an overview of the full internal structure. Overview Kafka uses a publish-subscribe model where producers produce events and consumers consume messages. Kafka producers publish events to the broker, and consumers consume the events from the broker. In the meantime, the broker has a different layer of architecture that allows Kafka to be an event streaming platform. The producer, broker, and consumer-related metadata are managed by the Kafka controller, which is elected by ZooKeeper or KRaft. Key Terms Topic: A topic is a logical name or category where Kafka stores a stream of related events/messages. Think of a topic as a database table for events or a channel where producers publish messages and consumers subscribe to read them. Partition: A partition is a physical subdivision of a Kafka topic that stores messages in an ordered, immutable sequence. It is the basic unit of parallelism and scalability in Kafka. Segment: A segment is a physical file inside a Kafka partition where messages are stored on disk. A partition is not stored as one huge file. Kafka divides it into multiple smaller segment files. Offset: An offset is a unique sequential number assigned to each message within a Kafka partition. It represents the position of a message in that partition. Metadata: Metadata is information about the Kafka cluster, not the actual message data. It describes the structure and state of the Kafka cluster. Controller: The Kafka Controller is a special broker responsible for managing cluster-level operations. In a Kafka cluster, one broker is elected as the controller. ZooKeeper (OLD): ZooKeeper is an external distributed coordination system used by older Kafka versions to manage cluster metadata and controller election. Kraft (Modern): KRaft (Kafka Raft metadata mode) is Kafka's new architecture that removes ZooKeeper and manages metadata internally using the Raft consensus algorithm. Leader: A leader is the broker that owns the primary copy of a partition and handles all read and write operations for that partition. Only one broker can be the leader for a partition at a time. Replica: A replica is a copy of a partition stored on another broker to provide fault tolerance. Internal Details With the Kafka Library At first, download Kafka from this link. After extracting, we will get the following directory structure: Here, each folder has its own functionality. The bin folder contains all the executable scripts. These scripts can be used from the command line to manage Kafka components and perform various tasks, including those listed below: Start serversStop serversTopic managementCluster managementAct as a publisher/consumer Another important folder is the config folder, where we can configure the broker and controller nodes. Initially, this folder contains the default configuration files for the Kafka components. Here we find property files named controller.properties, server.properties, broker.properties, consumer.properties, producer.properties, and some other configuration files. In a Kafka installation, these .properties files are configuration files used by different Kafka components and client applications. However, not all of them are used by the Kafka server itself. We can think of the configuration files in three major categories: Servers - server.properties - broker.properties - controller.propertiesClients - producer.properties - consumer.propertiesTools - other properties file 1. server.properties This is the main configuration file for a Kafka broker. It controls how the Kafka broker runs, including: Broker ID/node identityListener and network configurationLog directoriesNumber of network and I/O threadsDefault number of partitionsReplication settingsRetention policiesMessage size limitsController configuration in KRaft modeCluster-related settings For example: Properties files node.id=1 process.roles=broker,controller listeners=PLAINTEXT://:9092,CONTROLLER://:9093 controller.quorum.voters=1@localhost:9093 log.dirs=/tmp/kafka-logs num.partitions=1 log.retention.hours=168 In modern Kafka using KRaft, server.properties can configure both the broker and controller when a combined node is used. 2. broker.properties This is generally used as a separate broker configuration file when you want to maintain configuration for a broker independently. For example, you might create: Plain Text config/ ├── broker.properties ├── controller.properties and start Kafka with a particular configuration: Properties files kafka-server-start.sh config/broker.properties The exact files supplied with Kafka can vary depending on the Kafka version/distribution. broker.properties is not a universal mandatory filename in Kafka; Kafka mainly cares about the properties file you pass to the startup command. 3. controller.properties This is used for configuring a KRaft controller. Kafka's modern architecture doesn't require ZooKeeper. Instead, Kafka uses the KRaft consensus mechanism. A controller is responsible for managing cluster metadata, such as: Which brokers are part of the clusterTopic metadataPartition informationLeader assignmentsCluster configurationController elections A dedicated controller configuration could contain things such as: Properties files process.roles=controller node.id=1 listeners=CONTROLLER://:9093 controller.quorum.voters=1@localhost:9093 controller.listener.names=CONTROLLER A production Kafka cluster may use dedicated controller nodes rather than running broker and controller roles together. 4. producer.properties This is not a broker configuration. It contains configuration for a Kafka Producer client — the application that sends messages to Kafka. For example: Properties files bootstrap.servers=localhost:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer Other producer settings can control: acksretriesbatch.sizelinger.mscompression.typebuffer.memoryenable.idempotencehghg For example: Properties files acks=all retries=10 compression.type=snappy A producer application reads these settings when connecting to Kafka. 5. consumer.properties This is also a client configuration, not a broker configuration. It controls how a Kafka Consumer reads messages. Example: Properties files bootstrap.servers=localhost:9092 group.id=my-consumer-group key.deserializer=org.apache.kafka.common.serialization.StringDeserializer value.deserializer=org.apache.kafka.common.serialization.StringDeserializer auto.offset.reset=earliest Important consumer settings include: bootstrap.serversKafka broker addressesgroup.idConsumer group identitykey.deserializerConverts message key bytesvalue.deserializerConverts message value bytesauto.offset.resetWhat to do when no valid offset existsenable.auto.commitControls automatic offset commitsmax.poll.recordsMaximum records returned per pollfetch.min.bytesMinimum data fetched from broker 6. Other Configuration Files Depending on your Kafka version, you may also find files such as: Plain Text config/ ├── connect-distributed.properties ├── connect-standalone.properties ├── connect-log4j.properties ├── tools-log4j.properties └── ... Some are for Kafka's additional components. For example: Kafka Connect Plain Text connect-distributed.properties connect-standalone.properties These configure Kafka Connect workers. Logging Plain Text connect-log4j.properties tools-log4j.properties These configure logging behavior for particular Kafka tools/components. So, server.properties/broker.properties/controller.properties configure Kafka infrastructure, while producer.properties and consumer.properties configure applications that communicate with Kafka. Step by Step: Test Producer and Consumer From CLI We will discuss the Kafka cluster later, but for now we will check how to run the Kafka controller, broker, and an example producer and consumer from the CLI. 1. Initialize Kafka Storage For a new Kafka installation running in KRaft mode, first generate a cluster ID. If we move to the Kafka bin directory, we can find a file called kafka-storage.sh Shell ./kafka-storage.sh random-uuid Copy the generated UUID. Then format the Kafka storage: Shell ./kafka-storage.sh format \ -t <CLUSTER_ID> \ -c ../config/server.properties For example: Shell ./kafka-storage.sh format \ -t abc123xyz \ -c ../config/server.properties You normally perform this initialization only once for a fresh Kafka storage directory. Don't repeatedly format an existing cluster unless you intentionally want to initialize/reset it. 2. Start the Kafka Server Because we are now inside the bin, specify the configuration file relative to the parent directory: Shell ./kafka-server-start.sh ../config/server.properties Or we can run it in the background: Shell ./kafka-server-start.sh -daemon ../config/server.properties The important relationship is: Plain Text bin/ └── kafka-server-start.sh config/ └── server.properties So the command is Shell ./kafka-server-start.sh ../config/server.properties 3. Verify That Kafka Is Running We can also check the Kafka port: Plain Text lsof -i :9092 4. Create a Topic Still inside the bin directory: Shell ./kafka-topics.sh \ --create \ --topic test-topic \ --bootstrap-server localhost:9092 \ --partitions 1 \ --replication-factor 1 List topics: Shell ./kafka-topics.sh \ --list \ --bootstrap-server localhost:9092 5. Start a Consumer Open another terminal and go to Kafka's bin directory: Shell ./kafka-console-consumer.sh \ --topic test-topic \ --bootstrap-server localhost:9092 The consumer will wait for messages. 6. Start a Producer Open another terminal: Shell ./kafka-console-producer.sh \ --topic test-topic \ --bootstrap-server localhost:9092 Now type: Plain Text Hello Kafka The message should appear in the consumer terminal. This is the introduction to this series. We will discuss more details in the upcoming articles.

This is Part 3 of a series on running sentiment analysis inside Neo4j. Part 1, Running Sentiment Analysis Inside Neo4j With a Java Plugin, used the VADER lexicon via a pure Java UDF – rule-based, fully local, high throughput.Part 2, Wasm Inside Neo4j: Building the Example That Didn't Exist, embedded a Wasm runtime inside a Java UDF using wasmtime-java — sandboxed, still local, lower throughput. Part 3 takes a different approach. Instead of bundling a scoring model inside the plugin JAR, we call TypeSafe AI's Jev from inside a Java UDF. Jev is a System One model – it doesn't generate text. It takes a state (the text to evaluate) and a set of typed questions and returns structured decisions with explicit probability and confidence information. No prompt engineering, no JSON schema coercion, no parsing. The result is callable directly from Cypher: Cypher RETURN com.example.jev.sentiment('The movie was great!') AS result; The full source code is available on GitHub. What Jev Gives You That VADER Doesn't VADER gives you a compound score. For "The movie had some good moments but ultimately disappointed," it returns something around -0.3. That's a number, but it doesn't tell you how much to trust it. Jev separates the decision from the uncertainty. For the same sentence, it returns: JSON { "choice": "negative", "confidence": 1.0, "score": 1.55, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } The winning sentiment option has a confidence value of 1.0. But the score — which is the result of the separate confidence question we asked Jev, rated on a scale from 0 to 2 — is 1.55, landing between "Somewhat confident" and "Very confident." The separate confidence score provides an additional signal about the classification and that information can be actionable in a way that a raw sentiment score isn't. For unambiguous text, Jev is equally unambiguous: JSON { "choice": "positive", "confidence": 1.0, "score": 2.0, ... } This structured uncertainty output is the key differentiator of Part 3. If you need to act differently on low-confidence classifications — routing them to human review, flagging them for follow-up, or simply surfacing the uncertainty in a UI — VADER's scores don't provide a model-derived estimate of uncertainty about the classification. Jev returns an explicit confidence assessment alongside the decision. Confidence information should not be confused with accuracy: a model can be highly confident and still be wrong. What We're Building A Neo4j Java UDF that calls the Jev API over HTTPS, passing the input text as the evaluation state and asking two typed questions: a sentiment classification (positive, negative, or neutral) and a confidence score. For non-blank input, the UDF returns a Map<String, Object> with six keys: choice, confidence, score, positive, negative and neutral. For blank input, it returns only choice and confidence as a guard response without calling the API. No Wasm runtime, no bundled lexicon, no native libraries. The plugin has no Wasm runtime, native library, or bundled sentiment lexicon. Its only runtime dependency beyond Neo4j itself is Jackson, which is relocated into a private package namespace to avoid classpath conflicts with Neo4j's own Jackson dependencies. Prerequisites Java 21. OpenJDK 21 or later. Confirm with java -version.Maven 3.9.6. Install manually rather than via Homebrew on Apple Silicon. Confirm with mvn -version.Neo4j Desktop. Download and install Neo4j for Desktop. We're using version 2026.07.0. Update the neo4j.version property in pom.xml to match your installation.A Jev API key. Sign up for access at TypeSafe AI. The key is passed to Neo4j via a JVM property in neo4j.conf — more on this in the deployment section. Getting the Code The full source is on GitHub. Clone the repo and navigate to the neo4j-jev-udf directory: Shell cd ~ git clone --filter=blob:none --sparse https://github.com/VeryFatBoy/neo4j.git cd neo4j git sparse-checkout set jev-udf mv jev-udf ../jev-udf cd ../jev-udf/neo4j-jev-udf The Maven Project The pom.xml has two things worth noting. First, org.neo4j:neo4j is declared as provided scope — Neo4j is already present in the database JVM at runtime. Second, we use maven-shade-plugin to relocate Jackson into a private package namespace. Neo4j ships its own version of Jackson internally, and without relocation, the two can conflict at runtime in ways that produce silent failures. XML <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>jev-neo4j-poc</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <properties> <maven.compiler.source>21</maven.compiler.source> <maven.compiler.target>21</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <neo4j.version>2026.07.0</neo4j.version> </properties> <dependencies> <dependency> <groupId>org.neo4j</groupId> <artifactId>neo4j</artifactId> <version>${neo4j.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.17.2</version> </dependency> </dependencies> <build> <plugins> <plugin> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>21</source> <target>21</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.5.1</version> <executions> <execution> <phase>package</phase> <goals><goal>shade</goal></goals> <configuration> <artifactSet> <excludes> <exclude>org.neo4j:*</exclude> </excludes> </artifactSet> <relocations> <relocation> <pattern>com.fasterxml.jackson</pattern> <shadedPattern>com.example.jev.shaded.jackson</shadedPattern> </relocation> </relocations> <shadedArtifactAttached>false</shadedArtifactAttached> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> Update the neo4j.version property to match your own Neo4j Desktop installation. The Java UDF The UDF uses Java's built-in HttpClient — no third-party HTTP library needed. The Jev API key is read first from the environment variable TYPESAFE_API_KEY, then from a JVM system property of the same name as a fallback. The fallback matters because Neo4j Desktop runs as a separate process and doesn't inherit your shell's environment variables; we'll set the JVM property in neo4j.conf in the deployment section. Java package com.example; import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.node.ArrayNode; import com.fasterxml.jackson.databind.node.ObjectNode; import org.neo4j.procedure.Description; import org.neo4j.procedure.Name; import org.neo4j.procedure.UserFunction; import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; import java.util.HashMap; import java.util.Map; public class JevUDF { private static final String API_URL = "https://api.typesafe.ai/v1/systemone"; private static final ObjectMapper mapper = new ObjectMapper(); private static final HttpClient client = HttpClient.newHttpClient(); private static String getApiKey() { String key = System.getenv("TYPESAFE_API_KEY"); if (key == null || key.isBlank()) { key = System.getProperty("TYPESAFE_API_KEY"); } if (key == null || key.isBlank()) { throw new RuntimeException("TYPESAFE_API_KEY not set as environment variable or JVM property"); } return key; } @UserFunction("com.example.jev.sentiment") @Description("Classifies text sentiment using Jev (TypeSafe AI). Returns choice, probabilities and confidence.") public Map<String, Object> sentiment(@Name("text") String text) throws Exception { if (text == null || text.isBlank()) { Map<String, Object> empty = new HashMap<>(); empty.put("choice", "neutral"); empty.put("confidence", 0.0); return empty; } String apiKey = getApiKey(); ObjectNode body = mapper.createObjectNode(); body.put("model", "jev-latest"); body.put("state", text); ObjectNode questions = mapper.createObjectNode(); ObjectNode sentiment = mapper.createObjectNode(); sentiment.put("type", "choice"); sentiment.put("instructions", "What is the overall sentiment of this text?"); ObjectNode criteria = mapper.createObjectNode(); criteria.put("positive", "Text expresses positive sentiment"); criteria.put("negative", "Text expresses negative sentiment"); criteria.put("neutral", "Text is neutral or factual"); sentiment.set("criteria", criteria); questions.set("sentiment", sentiment); ObjectNode confidence = mapper.createObjectNode(); confidence.put("type", "score"); confidence.put("instructions", "How confident are you in the sentiment classification?"); ArrayNode levels = mapper.createArrayNode(); levels.add("Not confident"); levels.add("Somewhat confident"); levels.add("Very confident"); confidence.set("criteria", levels); questions.set("confidence", confidence); body.set("questions", questions); HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(API_URL)) .header("Content-Type", "application/json") .header("Authorization", "Bearer " + apiKey) .POST(HttpRequest.BodyPublishers.ofString(mapper.writeValueAsString(body))) .build(); HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString()); if (response.statusCode() != 200) { throw new RuntimeException("Jev API error " + response.statusCode() + ": " + response.body()); } JsonNode result = mapper.readTree(response.body()); JsonNode answers = result.get("answers"); JsonNode sentimentAnswer = answers.get("sentiment"); JsonNode confidenceAnswer = answers.get("confidence"); Map<String, Object> output = new HashMap<>(); output.put("choice", sentimentAnswer.get("choice").asText()); output.put("confidence", sentimentAnswer.get("confidence").asDouble()); output.put("score", confidenceAnswer.get("score").asDouble()); output.put("positive", sentimentAnswer.get("probabilities").get("positive").asDouble()); output.put("negative", sentimentAnswer.get("probabilities").get("negative").asDouble()); output.put("neutral", sentimentAnswer.get("probabilities").get("neutral").asDouble()); return output; } } A few things worth noting in the Jev request body. The choice question uses a criteria object — a map of option names to descriptions. The score question uses a criteria array — an ordered list of level descriptions. These are different shapes, and mixing them up produces a 422 error from the API. Both questions are answered in a single API call; Jev evaluates them in parallel against the same state. Deploying to Neo4j Build the JAR: Shell mvn -q clean package Stop Neo4j in Desktop, then copy the JAR to your plugins folder: Shell cp target/jev-neo4j-poc-1.0-SNAPSHOT.jar \ ~/Library/Application\ Support/neo4j-desktop/Application/Data/dbmss/<your-dbms-id>/plugins/ On Linux, the plugins folder is typically under $NEO4J_HOME/plugins/. On Windows, the plugins folder is typically under %NEO4J_HOME%\plugins\. Edit neo4j.conf in the conf/ folder of the same dbms directory and add three lines: Plain Text dbms.security.procedures.unrestricted=com.example.jev.* dbms.security.procedures.allowlist=com.example.jev.* server.jvm.additional=-DTYPESAFE_API_KEY=<your-api-key> Two lines are shown in the security configuration. allowlist controls which extension functions are loaded and makes the function callable from Cypher. unrestricted gives matching extensions full database access, bypassing Neo4j's security restrictions and should be used only where necessary. For this example, both settings are configured for the com.example.jev.* namespace; use the narrowest configuration that your Neo4j version and deployment require. The server.jvm.additional line passes the API key to Neo4j's JVM as a system property. For this local Neo4j Desktop example, that's a practical way to make the key available without modifying the UDF code. The Neo4j Desktop process does not automatically receive environment variables exported in your terminal session, so the JVM property is the reliable alternative. Do not commit this value to source control. For production, use your deployment environment's secret-management mechanism and inject the value into the Neo4j process without storing the secret in a tracked configuration file. Restart Neo4j in Desktop, then select Query from the left-hand navigation pane, connect to your instance, and confirm the function is registered: Cypher SHOW FUNCTIONS YIELD name WHERE name STARTS WITH 'com.example' RETURN name; Expected: "com.example.jev.sentiment" Results Positive Sentence Cypher RETURN com.example.jev.sentiment('The movie was great!') AS result; Expected: JSON { "choice": "positive", "confidence": 1.0, "score": 2.0, "positive": 1.0, "negative": 0.0, "neutral": 0.0 } Negative Sentence Cypher RETURN com.example.jev.sentiment('The movie was absolutely terrible.') AS result; Expected: JSON { "choice": "negative", "confidence": 1.0, "score": 2.0, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } Mixed Sentiment Cypher RETURN com.example.jev.sentiment('The movie had some good moments but ultimately disappointed.') AS result; Expected: JSON { "choice": "negative", "confidence": 1.0, "score": 1.55, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } The winning sentiment option has a confidence value of 1.0. But the score drops to 1.55 out of 2 – somewhere between "Somewhat confident" and "Very confident." This is structured uncertainty output, and it's what distinguishes Jev from a rule-based scorer. There are two different quantities here. confidence belongs to the sentiment choice itself: it describes the probability concentration of the classification. score comes from a separate question that explicitly asks Jev how confident it is in that classification. They answer different questions and should not be interpreted as two versions of the same confidence value. Blank-Input Guard Cypher RETURN com.example.jev.sentiment('') AS result; Expected: JSON { "choice": "neutral", "confidence": 0.0 } The blank-input guard returns immediately without making an API call. Here neutral is a guard value rather than a Jev classification: no sentiment analysis is performed for blank or whitespace-only input. Scalability and Limitations Every call to com.example.jev.sentiment() makes an outbound HTTPS request to the Jev API. That shapes where this approach fits and where it doesn't. Latency. Jev responds in 70-500ms per call. For an occasional interactive request — a customer submitting a review, a user looking up a specific node — that latency may be acceptable. For a batch pipeline, if 1,000 calls are made sequentially at a 200ms average, that's over three minutes of API time; concurrent execution changes the wall-clock time but introduces its own rate-limit and resource considerations. Cost. At the pricing available when this article was written, input tokens cost approximately $0.042 per million and output tokens were free. The test requests in this article averaged approximately 381 input tokens per call, giving a per-call cost of around $0.000016 (381 ÷ 1,000,000 × $0.042). Pricing is subject to change; check TypeSafe AI's current pricing before estimating production cost. At this workload size, cost is unlikely to be the primary constraint; latency, throughput, network dependency, and data residency are more significant considerations. At that rate, 1,000 calls would cost roughly $0.016, while one million calls would cost roughly $16, assuming the same average input size. Network dependency. Parts 1 and 2 of this article series are fully local – no network, no external dependency, no data leaving the database. Part 3 introduces all three. The UDF itself runs inside Neo4j, but the inference does not: the text crosses the network boundary to TypeSafe AI, and only the structured result returns to Neo4j. If the Jev API is unreachable, the UDF throws. In a production deployment, that's an operational risk worth planning for: the HttpClient in this example has no explicit timeout configured; in production, set a connect timeout and a per-request timeout to prevent a slow or unresponsive API from blocking Neo4j worker threads indefinitely. A fallback return value and monitoring on the call path are also worth considering. Note also that the UDF makes a synchronous blocking HTTP call inside database query execution; concurrent Cypher invocations increase pressure on Neo4j worker threads, the API's rate limits, and the network. Data residency. Text passed to com.example.jev.sentiment() leaves Neo4j and is sent to TypeSafe AI's API. If your data residency requirements prohibit that, this approach isn't suitable. Review TypeSafe AI's data handling policies before deploying in a regulated environment. Caching as a mitigation. For workloads where the same text is scored repeatedly, results can be stored as node properties on first call and read directly on subsequent queries — skipping the API entirely. That's a natural pattern in Neo4j and largely eliminates the latency and cost concerns for stable datasets. The right use case. This approach is best suited to low-frequency, interactive workloads where explicit uncertainty information matters more than throughput and where data leaving the database is acceptable. For high-volume batch ingestion, Part 1 (the Java VADER UDF) remains a strong fit — local, fast, no network dependency. Part 3 trades local throughput and offline operation for a richer decision output and explicit uncertainty information. Comparing the Three Approaches Part 1: Java VADERPart 2: Wasm/VADERPart 3: JevOutputPolarity scores (float)Polarity scores (float)Typed decision + explicit uncertainty informationLocationFully localFully localNetwork (TypeSafe AI API)Latency< 1ms50-200ms (init)70-500msCostFreeFree~$0.000016/callOfflineYesYesNoExecution boundaryJava UDFWasm runtimeExternal APIUncertainty outputNoNoYesTypical fitHigh-volume batchSandboxed local inferenceInteractive, uncertainty-aware decisions Summary We added a third option to the series: calling TypeSafe AI's Jev from inside a Neo4j Java UDF, returning a typed sentiment decision with explicit uncertainty information directly from Cypher. The implementation has the smallest inference footprint of the three – no Wasm runtime, no bundled lexicon, no native libraries. The JAR contains only the UDF class, a relocated Jackson, and Java's built-in HttpClient. The main configuration consideration is passing the API key to Neo4j's JVM via server.jvm.additional in neo4j.conf, since the Neo4j Desktop process does not automatically receive environment variables exported in your terminal session. The key contribution of Part 3 isn't the mechanics — it's the explicit uncertainty output. For clear-cut sentences, Jev is unambiguous. For genuinely mixed sentiment, it returns both the decision and a structured confidence measure that Cypher can act on directly. That's something neither VADER nor the Wasm implementation provides. A deeper discussion of putting Jev inside Neo4j can be found in the free online book Lessons Learned Putting Jev Inside Neo4j. The full source code is available on GitHub.

By Akmal Chaudhri CORE

What separates a good engineer from a great one when AI can generate a solution in seconds? For Raghava Dittakavi, it comes down to judgment, curiosity, and a willingness to learn from what breaks. This week, I sat down with Raghava to discuss the experiences that have shaped his approach to technology, his perspective on AI-assisted development, and the insights he hopes the next generation of engineers will carry forward. 1. What first got you interested in technology? I have always been curious about how different parts of a system work together and how to use technology to solve real problems or create measurable value. Early in my career, that curiosity was mostly about understanding how systems/software worked. Over time, it expanded into how software, infrastructure, cloud platforms, automation, and engineering teams can work together effectively at scale. What continues to keep me interested is how quickly technology evolves. Today, I am particularly interested in how AI is changing the way we build, operate, and improve software systems. I see AI as another important engineering capability, not a replacement for strong fundamentals, but something that can significantly amplify what good engineering teams are able to accomplish. 2. What’s a technical challenge or failure that taught you something you still apply to your work today? One technical lesson that has stayed with me is that an issue found in staging cannot be dismissed simply because production is a different environment. The question I ask is: “What evidence supports our conclusion about whether this issue could occur in production?” That means understanding the failure conditions and examining the relevant configuration, data patterns, load, and dependencies. If a difference between environments prevents the issue, we should be able to explain and demonstrate why. The principle I carry forward is simple: uncertainty is not evidence of safety. Until we understand the cause and assess whether those conditions could exist in production, the risk remains unresolved. 3. As AI becomes a bigger part of how engineers work, what does good engineering judgment look like in an AI-assisted development environment? For me, good engineering judgment means being able to defend a decision without saying, “The AI suggested it.” As an experienced engineer, I want to understand the impact of every change I introduce: what behavior it alters, what else depends on it, and what new risks it creates. Even a small code change can have consequences beyond the immediate problem it solves. AI can produce a convincing solution quickly, but I still need to question its assumptions and validate how it behaves when things go wrong. Passing tests is useful evidence, but I also need to know whether those tests cover the risks that matter. My standard is straightforward: if I cannot explain the change, its potential impact, and the evidence supporting its safety, I am not ready to accept it. AI can help me move faster; accountability for the decision remains mine. 4. If you could only keep three tools in your developer toolkit, what would they be and why? I would choose Git, CI/CD automation, and AI-assisted engineering tools. Git because understanding what changed, why it changed, and being able to collaborate safely around code is fundamental to software engineering. CI/CD automation because engineering does not stop when code is committed. Building, testing, validating, deploying, and recovering software should be repeatable and automated. Good automation improves release velocity, reliability, and confidence. AI-assisted engineering tools because they are becoming an important part of how engineers work. I find the biggest value is not just generating code. It is helping engineers understand large systems faster, troubleshoot issues, connect information across different sources, automate repetitive work, and move from an idea to a working solution more efficiently. 5. What role do you think experienced engineers should play in sharing their knowledge with the broader tech community? I believe experienced engineers should share how they think, as well as what they know. A tutorial can explain the steps to build something. Experience adds another layer: why you would choose that approach, what trade-offs you accept, and when you should reconsider it. Sharing that reasoning helps another engineer decide whether the same approach fits their own problem. That is what I aim to bring to writing, mentoring, and technical discussions. I want someone to leave with a better way to evaluate a problem, rather than just another solution to copy. We also need to be comfortable explaining where our understanding ends and what has changed our minds. Sharing experience responsibly means giving others enough context to question our conclusions and make their own informed decisions. 6. What advice would you give someone just getting started in your field? Build strong fundamentals and stay curious. Learn how software actually works end-to-end: code, operating systems, networking, source control, infrastructure, cloud, testing, deployment, and monitoring. Individual tools will change, but understanding the underlying concepts will remain valuable. More importantly, build things. Deploy them, most importantly, break them, troubleshoot them, and automate them. A lot of practical engineering knowledge comes from understanding why something failed and figuring out how to make it more reliable the next time. I would also encourage engineers starting today to learn how to work effectively with AI. Use it to accelerate your learning and engineering work, but continue to understand and validate what is happening beneath the surface. AI is much more useful when it is combined with strong engineering judgment. 7. What’s your ideal way to spend a weekend? My ideal weekend is a balance of family, community, learning, and creativity. I enjoy spending quality time with my family and being involved in community and spiritual activities. I also like setting aside some quiet time to listen, explore new technology, experiment with ideas, or write about lessons and perspectives. Writing is something I enjoy because it helps me organize my own thinking while also sharing something useful with the broader technology community. For me, a great weekend is one where I can slow down, spend time with the people who matter, and still have some space to learn, think, and create. Explore Raghava's content here.

As AI-generated code expands across production systems, provenance becomes essential for tracing where each change originated, which model or prompt influenced it, and what evidence supports its review, ownership, and trustworthiness. AI-assisted development changes the economics of software creation. Source code can now be produced faster than many engineering organizations can review, understand, test, secure, and maintain it. The resulting risk is not simply "bad AI code." Recent research is more nuanced, as one 2026 study of real repositories found AI-generated code to be more verbose and to exhibit different duplication patterns, while another maintenance study found lower subsequent maintenance activity for sampled AI-generated files than for comparable human-generated files. A 2025 controlled study likewise found fewer defects in some LLM-generated solutions but also structural problems on harder tasks. The operational conclusion is therefore not that generated code is inherently inferior, but that code volume can grow faster than governance capacity. The language and stack are unspecified, so the implementation patterns below use Git, Java, Python, TypeScript, and GitHub-style CI as representative examples. The central strategy is to treat code generation as an input to an engineering control system so we can record provenance, establish ownership, prevent duplicate abstractions, enforce complexity budgets, map tests to changed behavior, automate small refactorings, constrain dependencies, and make security and repository policy executable. Code Abundance Requires Traceability and Ownership Traditional repositories mostly answer who committed a change and when. AI-assisted repositories also need to answer how a change was produced, which tool or agent contributed, which task authorized the change, and which human or team accepted responsibility. Git trailers are a practical mechanism because Git supports structured key-value metadata at the end of commit messages and can parse that metadata with git interpret-trailers. Shell git commit -m "Add invoice reconciliation" \ --trailer "AI-Origin: assisted" \ --trailer "AI-Tool: coding-agent" \ --trailer "Task: PAY-431" This metadata should describe provenance rather than transfer accountability to a model. Ownership remains organizational. A CODEOWNERS file can map sensitive or high-change paths to responsible teams, and GitHub rules can require approval from those owners before merge. GitHub rulesets can also require status checks, dismiss stale approvals after new pushes, and require approval of the latest reviewable push. Plain Text /payments/ @acme/payments /auth/ @acme/security /platform/ @acme/platform The important repository metric is therefore not generated lines per week. It is generated surface area without durable ownership. A repository can absorb high code generation throughput when every merged path has an accountable maintainer, searchable provenance, and enforceable review boundaries. Quality Budgets Must Stop Net-New Entropy Code abundance makes absolute cleanup programs impractical. Quality gates should instead prevent each change from making the repository worse. Duplication is a useful example. PMD’s Copy/Paste Detector supports multiple languages, can fail CI when duplicates are detected, and can normalize identifiers for structurally similar Java or C++ fragments. Its documentation explicitly recommends removing duplication rather than attempting to keep copied blocks synchronized. Shell pmd cpd --minimum-tokens 100 \ --dir src/main/java \ --ignore-identifiers This check becomes more useful when applied differentially, as existing duplication can be baselined, while newly introduced clones fail the pull request. The same principle applies to complexity. ESLint exposes a cyclomatic-complexity rule with a configurable maximum, allowing TypeScript or JavaScript repositories to turn complexity from a review opinion into a merge condition. JavaScript export default [{ rules: { complexity: ["error", { max: 12 }] } }]; A complexity budget should be paired with automated refactoring rather than only rejection. OpenRewrite operates on semantic trees and provides recipes for framework migrations, security fixes, and consistent source transformations while preserving formatting. That model is particularly suitable for code-abundant repositories because repetitive cleanup can be expressed once and applied consistently. Recent industrial research from Bloomberg also suggests that automated maintenance is easier to review when it arrives as very small pull requests, as its experimental system targeted roughly ten line changes and merged most generated patches in a one-month deployment, although the sample was small and should not be generalized broadly. Tests and CI Need Change-Level Intelligence A large test suite does not automatically create confidence. Maintenance becomes expensive when a changed function cannot be connected to the tests that exercise it. Coverage.py supports dynamic contexts that record which test function executed each covered line, making a code-to-test map available for Python repositories. Plain Text [run] dynamic_context = test_function That mapping enables risk-based CI. Changed files can select directly related tests first, while shared libraries, authentication code, persistence layers, or other high-blast-radius modules can trigger broader suites. Coverage remains a gate rather than a vanity percentage. For Java, JaCoCo can halt a build when configured line, branch, instruction, class, or complexity thresholds are not met. A compact CI stage can combine these controls without treating AI-originated code as a separate species: YAML quality: steps: - run: npm run lint - run: pmd cpd --minimum-tokens 100 --dir src/main/java - run: pytest --cov=src --cov-context=test - run: ./mvnw verify The stronger policy is "no unexplained quality regression," not "all generated code receives maximal testing." Provenance can increase scrutiny for high-risk generated changes, but changed behavior, dependency reach, security sensitivity, and ownership should determine the actual gate. Dependency and Security Hygiene Must Scale With Generation Code-generation workflows capable of editing manifests can expand a repository’s dependency surface as quickly as application code. Dependency review therefore belongs at pull request time. GitHub’s dependency review reports direct and indirect changes from manifests and lockfiles, includes vulnerability information, and can fail a required check when vulnerable packages are introduced. This supports a repository policy in which new dependencies require an explicit purpose, lockfile changes receive review, and unused packages are removed during routine maintenance. Security analysis should operate on every merge path rather than rely on prompt instructions. GitHub code scanning can run CodeQL and other analyzers on pull requests and the default branch, and CodeQL data-flow analysis can identify unsafe flows such as dangerous arguments or sensitive data leakage. Required code-scanning checks can block merging until findings are resolved. Runtime-facing checks provide a complementary boundary. OWASP ZAP’s baseline scan spiders a deployed target, performs passive scanning, supports configurable fail conditions, and is explicitly designed for CI/CD environments. Static analysis can therefore guard source changes while dynamic scanning checks observable application behavior. The important distinction is between generation-time guidance and repository-time enforcement, as model instructions are advisory, while CI gates are reproducible controls. Repository governance ties the system together. Rulesets should require code-owner approval for protected areas, passing quality and security checks, review of dependency changes, and fresh approval after material updates. Such controls keep throughput high without allowing generated volume to bypass the same engineering standards applied to manually written code. Conclusion The maintenance problem created by AI is fundamentally a throughput mismatch. Code production can accelerate immediately, while ownership, review capacity, testing, security analysis, and architectural coherence remain finite. Current research does not justify treating AI-generated code as uniformly worse than human-written code, but it does show meaningful differences in verbosity, structure, duplication, defects, and maintenance behavior that make passive governance increasingly risky. Sustainable code abundance therefore depends on converting maintenance expectations into repository mechanics where provenance is attached to commits, ownership is attached to paths, quality budgets are enforced on new changes, tests are mapped to behavior, refactoring is delivered in small reviewable patches, dependencies are examined before merge, and security checks are made mandatory. The durable advantage will not come from generating the most code. It will come from operating a codebase that can absorb generated code without allowing maintainability to collapse under its volume.

By Uthej Mopathi CORE

Building a RAG application is relatively easy. Scaling it for millions of users and millions of documents is a different engineering problem. At production scale, the architecture matters. Ingestion, search, indexing, permissions, latency, and updates all need to work independently without slowing down user requests. The first design decision is to separate document ingestion from user queries. Q1. Why Should Ingestion and Querying Be Separate? They have different requirements. Ingestion runs in the background. It processes documents, cleans them, removes duplicates, creates chunks, generates embeddings, adds metadata, and updates indexes. Querying runs for every user request. It needs to return relevant results quickly and consistently. Think of it like a library. Ingestion is the librarian organizing new books. Querying is a visitor looking for a book. The visitor should not have to wait while the librarian processes a new shipment. This separation allows the ingestion pipeline to optimize for throughput, while the query pipeline can optimize for low latency. Q2. Why Does Chunking Matter, and Is "Semantic Chunking" Always Right? A chunk is a small slice of a document that gets stored and searched. Where you cut that slice matters a lot. Cut it at a random character count, like every 500 characters, and you might slice a sentence in half, or separate a question from its answer. When that happens, the AI model only sees half the picture and gives a worse answer. Semantic chunking cuts the text at natural breaks, like the end of a paragraph or a change in topic, instead of a fixed length. This keeps ideas whole. One honest caveat, since accuracy matters here: semantic chunking is not free. Because chunks come in different sizes, it is harder to batch-process them efficiently, and search scoring can get uneven when some chunks are tiny and others are huge. Many real systems use a middle ground, a target size that still respects natural sentence and paragraph breaks, instead of pure semantic chunking. Every chunk also needs extra information: which customer it belongs to, who can see it, when it was created, and where it came from. Skip this, and you cannot control access, cannot handle updates properly, and cannot debug problems later. Q3. Why Use "Approximate" Search Instead of Exact Search? When the AI looks for the most relevant chunks, it is comparing the meaning of your question to the meaning of millions of stored chunks. Doing this comparison perfectly, checking every single one, is called exact search. It gives the truest answer, but it is painfully slow once you have millions of chunks. Approximate Nearest Neighbor search, called ANN, is a shortcut. It finds chunks that are very likely the best matches, without checking absolutely everything. It is dramatically faster and uses much less memory. The honest tradeoff: ANN occasionally misses the single best match and returns the second or third best instead. In practice, this loss is small, and the speed gain is large, but it is a real tradeoff, not a magic trick with no downside. Q4. What Does "Hybrid Retrieval" Mean and Why Bother? There are two ways to search: by keyword and by meaning. Keyword search, using a method called BM25, is great at finding exact terms: product codes, error messages, specific names. It struggles when your question uses different words than the document, even if they mean the same thing. Meaning-based search, using vector embeddings, understands intent. It can match "how do I get my money back" to a document titled "refund policy" even though no words overlap. It struggles with exact codes and rare technical terms. Hybrid retrieval runs both searches, combines the results, and then uses a stronger, more careful model to re-rank just the top candidates from that combined list. You get the best of both without paying the cost of running the expensive re-ranker on everything. The standard way to combine two different ranked lists like this is called Reciprocal Rank Fusion. It blends the rankings from both searches fairly, without needing the two scoring systems to be on the same scale, which they usually are not. Q5. The Chatbot Gives a Confident, Wrong Answer. How Do You Actually Debug That? Just calling it "a hallucination" and shrugging does not fix anything. You need to find out exactly where the process broke down. There are only two possible places: retrieval or generation. Step 1: Check retrieval first. Look at what chunks were actually pulled for that question. Is the correct information even in there? If not, the problem happened before the model ever saw the question. Something upstream failed: maybe the chunking cut the answer in half, maybe the embedding did not capture the right meaning, maybe the search weighting favored the wrong chunks. Step 2: Check generation. If the right information was retrieved but the model still got it wrong, the problem is how the model used what it was given. Common causes: the prompt did not clearly tell the model to stick to the provided context, the important information was buried in the middle of a long prompt (models pay more attention to the start and end), or a distracting, irrelevant chunk got pulled in alongside the correct one and confused the model. Build this check into an ongoing habit, not a one-time fix. Keep a running set of real user questions and grade the system against them regularly, so you catch new problems the moment they appear instead of after users complain. Q6. Why Does an AI Agent Call the Wrong Tool, and How Do You Fix It? Usually this is not the model being "dumb." It is the tools being poorly described. If two tools sound similar or overlap in what they do, the model genuinely cannot tell them apart, no matter how capable it is. Fixes that actually work: Give every tool exactly one clear job, with no overlap with other tools.Use strict rules for inputs: required fields, fixed lists of allowed values where possible, and check the input before running anything.Show the model a few examples of correct tool choices, right in the instructions.For anything risky, like sending money or deleting data, add a confirmation step before it actually happens. The tradeoff is real. More rules and more checks mean more time per request, more cost, and more code to maintain. It is worth it for anything with real consequences if it goes wrong. Q7. Why Can AI Training Run in Parallel, But Generating a Response Cannot? A language model builds its answer one word (technically one "token") at a time, and each new word depends on every word it already wrote. To say "New Delhi," it must generate "New" first, because it needs that word in front of it before it can decide what comes next. This forces the model to work in a strict, one step at a time sequence when generating a response. Training is different because the model already knows the correct full sentence in advance; it is learning from real examples. So during training, it can check its predictions for every position in the sentence all at once, in one parallel pass, using a trick called teacher forcing. That is why training scales beautifully across many GPUs while generating a single response does not. One real technique worth knowing: speculative decoding. A small, fast model guesses several words ahead, and the big model checks and confirms them in one pass instead of one word at a time. It does not remove the one-word-at-a-time nature of the model, but it does meaningfully speed up how fast text appears in practice. Q8. What Problem Does an "LLM Gateway" Solve? Once more than one team in a company is calling AI models directly, things get messy fast. No shared view of who is spending what. No consistent security checks. Every team is reinventing the same plumbing. An LLM Gateway is a single front door that every team's requests go through. It handles login and per-team usage limits, picks which model or provider to send a request to, and automatically switches if one goes down, logs everything for cost tracking, and checks requests against safety and policy rules before they reach the model. The biggest win: if the company wants to switch AI providers or upgrade to a newer model, they change it once, behind the gateway. No product team has to touch their own code. Q9. Why Not Just Use One Big, Powerful Model for Everything? Cost and speed. Big models are expensive per word generated and slower to respond. A lot of the work inside a production system does not need that much intelligence: figuring out what a user is asking for, pulling out a name or date, deciding which of a few tools to call. Give that simple work to a small, fast, cheap model. Save the big, expensive model for the parts that genuinely need deep reasoning or high-quality writing, usually the final answer the user actually reads. This only works well if you are checking quality regularly. If the small model starts quietly making mistakes on its part of the job, you will not notice unless you are testing for it. Q10. Latency Is Not Just One Number. What Should You Actually Measure? Two separate things, because users experience them differently and they are caused by different parts of the system. Time to First Token (TTFT). How long the user stares at a blank screen before anything shows up. Mostly caused by how long your prompt is and how much retrieval work has to happen before generation even starts. Tokens Per Second (TPS). Once text starts appearing, how fast it keeps coming. Caused by the model's size and how busy the GPUs currently are. TTFT matters more for how the experience feels. If words start appearing right away, the whole thing feels fast, even if the total time to finish is exactly the same as a version that made you wait first and then streamed quickly. This matches general research on perceived speed in interfaces: how fast something feels and how fast it actually is are not the same thing, and feel usually wins for user satisfaction. Q11. What Changes When You Scale From Ten Million Documents to One Hundred Million? At that size, one machine cannot hold the entire search index anymore, period. This stops being a "make the search better" problem and becomes a distributed systems problem. Every question now has to be sent to several machines at once, called shards, and the results merged back into one final ranked list. Two more things stop being optional. First, incremental indexing: when a document changes, you update just that piece of the index; you do not rebuild everything from scratch every time; that would be far too slow. Second, caching: repeated or popular questions get served from a fast cache instead of running the full search and generation process every single time. Q12. How Should an AI Agent's "Memory" Actually Be Organized? Four separate layers, each solving a different problem. Mixing them together makes the system confusing to build and hard to debug. Short-term memory. The working notes for the task happening right now. Lives inside the model's current context window and disappears when the task ends.Long-term memory. Facts that should stick around, like a user's preferences. Stored outside the model and pulled back in only when relevant.Shared memory. When several agents are working together, this is the common notepad they all read from and write to.Episodic memory. A running, timestamped log of what happened before, so the system can look back and learn from past interactions. This is a useful, commonly used way to think about agent memory, though it is a design pattern, not a fixed rulebook everyone follows the same way. The real cost is honest: better memory makes the agent smarter, but it also means more moving parts to keep consistent and in sync. Q13. What Does Security Look Like Specifically for a RAG System? The core rule to remember: treat every retrieved document like it is untrusted input from a stranger, the same caution you would use for a raw form submission on a website. Never let content pulled from your document store trigger an action on its own without a server-side check first. Why this matters: a bad or compromised document sitting in your corpus could contain hidden instructions aimed at tricking the model, a real attack pattern known as prompt injection through retrieved content. For systems that serve multiple customers, called multi-tenant systems, isolation needs to be enforced at three separate levels, not just checked once in the application code: Data isolation. One customer's documents and embeddings must never be retrievable by another customer. This needs to be enforced at the index itself, not just trusted to application logic that could have a bug.Compute isolation. One very active customer should never be able to hog all the processing power and slow everyone else down.Cache isolation. A cached answer generated for one customer must never accidentally be served to a different customer. These three checks are especially important in regulated industries like banking and finance, where keeping customer data separated is often a legal requirement, not just good practice. Q14. Where Is the Real Value Shifting for Engineers Working in AI? Building a basic AI demo is turning into a baseline skill, something most engineers can put together in an afternoon with existing tools. What actually sets someone apart now is knowing how to run this in production: handling real scale, enforcing real security and tenant separation, and keeping agentic workflows reliable when thousands of real users are hitting the system at once. Closing The question engineers get asked is changing. It used to be "can you build this?" Now it is closer to "can you make this work correctly, safely, and affordably for millions of people, every single day, without it falling over."

By Ram Ghadiyaram CORE

A few years back, I sat through a production incident review where the root cause turned out to be a single stored procedure nobody on the team had touched in four years. The original author had left the company. Nobody knew why a particular join filtered out cancelled orders, only that removing it broke three downstream reports. That afternoon summed up, for me, everything wrong with how a lot of organizations still build their data pipelines. Legacy ETL was never really about "extract, transform, load" as much as it was about hiding logic somewhere convenient at the time and hoping the convenience lasted. It usually didn't. Why ETL Started Creaking Traditional ETL tools like Informatica or SSIS made sense when compute was expensive and storage was the constraint you designed around. You transformed data before it landed in the warehouse because you couldn't afford to store raw, messy versions of everything. That constraint shaped an entire generation of architecture decisions, and honestly, it wasn't a bad approach for its time. The trouble is that constraint mostly disappeared. Cloud storage is cheap now. Compute is elastic. Yet a lot of teams are still running the same transform-before-load pattern, just on newer infrastructure, which is a bit like buying a Tesla and still driving it like it takes ten minutes to warm up. Here's roughly what that legacy shape looks like in practice: Notice where the pain actually concentrates. It's not really the source systems or the warehouse; it's that middle stretch where an ETL server holds all the transformation logic, usually in a proprietary format, usually without meaningful version control, and usually without anyone testing whether a schema change upstream will quietly break a job three hops downstream. What ELT Changes, and What It Doesn't Moving to ELT isn't a magic fix, and I'd push back a little on anyone who frames it that way. What it does is separate two things that legacy ETL bundled together: moving data and transforming data. You load raw data first, cheaply, and you transform it later using the compute and tooling that's actually good at transformation, which for most teams now means dbt on top of a lakehouse like Databricks. This matters more than it sounds like on paper. When transformation logic lives in dbt models rather than a GUI-based ETL tool, it becomes something you can review in a pull request, test automatically, and trace through version history. A junior engineer six months from now can open the repo and actually see why cancelled orders get filtered, because it's sitting there in a documented model, not buried in a stored procedure that requires archaeology to understand. A Governed Architecture, Not Just a Faster One The part people skip, though, is governance. Plenty of "modernization" projects swap Informatica for dbt and call it done, then wonder why data quality issues and access sprawl show up six months later, just in a different shape. Speed without governance is still going to bite you; it just takes longer. Here's the architecture I'd actually recommend, built around the medallion pattern with governance woven through it rather than added on afterward: A few things worth calling out here, because the diagram alone won't tell you everything: Bronze is genuinely raw. No transformation, no filtering, just an append-only landing zone. This is your safety net. If a transformation bug corrupts your Silver layer, you can always rebuild from Bronze because nothing was thrown away. Silver is where dbt earns its keep. Cleaning, deduplication, conforming dimensions, basic business rules. This layer should be boring and predictable, which is exactly what you want from something everything else depends on. Gold is opinionated on purpose. These are business-facing marts, built for specific consumption patterns, whether that's a finance dashboard or a feature store feeding a fraud model. Unity Catalog isn't decoration. Row and column level access control, PII tagging, and lineage tracking across the whole Bronze to Gold journey happen here. When an auditor asks who touched customer PII and when, you actually have an answer instead of a guess. dbt tests gate the pipeline, not the postmortem. Schema tests, not-null checks, referential integrity, and increasingly, formal data contracts between teams run before code merges rather than after something breaks in production. I'll admit the CI and testing layer is the part teams most often underinvest in, mostly because it's less visible than a shiny new lakehouse. But it's the difference between catching a schema drift issue in a pull request versus catching it because a VP asks why the revenue dashboard looks wrong on a Monday morning. Where This Still Falls Short I don't want to oversell this. Governance frameworks like Unity Catalog add real operational overhead, and smaller teams sometimes find the setup cost disproportionate to their scale, at least initially. dbt also assumes a certain SQL-first maturity in your team; if most of your transformations genuinely need Python or Spark-native processing, you're looking at a hybrid setup rather than a pure dbt one. And migration itself is rarely a clean cutover. Most organizations run legacy ETL and modern ELT in parallel for a stretch, which is its own kind of operational headache. Still, for teams dealing with genuine scale, audit requirements, or just the accumulated technical debt of a decade of ad hoc SQL scripts, this pattern tends to hold up. Not because it's trendy, but because it separates concerns that never should have been tangled together in the first place, and it gives you a paper trail for decisions that used to live only in someone's memory. That's really the underlying shift here. Legacy ETL asked you to trust the person who wrote the job. Governed ELT asks the system to show its work.

An AI agent can understand a request, develop a plan, and still fail for a mundane reason: It cannot reach the system where the information — or the ability to act — lives. Enterprise context is scattered across customer relationship management (CRM) platforms, ticketing systems, document repositories, analytics tools, and internal applications. Connecting them has traditionally required organizations or software vendors to build and maintain separate integrations for every system. That approach has left many companies with significant integration debt. Every point-to-point connector becomes another piece of code to secure, monitor, update, and repair. AI agents make that debt more visible because a single missing connection can interrupt an otherwise automated workflow. Model Context Protocol, or MCP, offers a much better architectural pattern. It is one of the most important developments in the emerging agent ecosystem. But standardized connectivity is only half of what enterprises need to operate agents successfully. What MCP Changes MCP gives AI applications a consistent way to connect to external data sources and tools. Instead of requiring a separate integration for every pairing of applications, an MCP server can advertise the capabilities it makes available. A compatible AI application can then discover and use those capabilities through a common interface. The USB-C analogy is useful. Before common connection standards, every device required its own proprietary cable. Standardization let many devices use the same interface. MCP applies that principle to AI applications and the enterprise systems they need to reach. That is why MCP is generating so much justified excitement. It reduces integration friction, expands what agents can do, and makes it easier to connect new systems as requirements evolve. Integration work does not disappear entirely, however. It moves. The provider of the underlying application — or an internal team supporting a proprietary system — must still expose useful capabilities through an MCP server. Their quality also depends on the underlying application and API. Putting an MCP wrapper around a poor search service does not produce better search. It simply gives the agent standardized access to the same limitations. But MCP only standardizes the connection. It does not provide the complete environment required to operate enterprise agents. Connectivity Does Not Replace Governance When users connect agents to enterprise applications through MCP, the agent may operate using their identity and existing permissions. A person connecting to Jira, for example, authenticates as themselves, and the agent subsequently calls Jira tools using that person’s delegated access. That creates an important distinction between what a user is technically permitted to do and what an agent acting on the user’s behalf should be allowed to do. A developer may have access to tools that can write or delete code. It does not follow that every agent should receive those capabilities. Enterprise governance therefore needs several layers. Administrators should establish which MCP servers users and agents may access and which tools from those servers are available. Within those boundaries, an agentic AI platform can automatically discover and select the tools relevant to a particular task. A GitHub MCP server might expose dozens of actions, while a code-review agent receives only read-only tools. A specialized service agent might receive only the Jira tools needed to review or create tickets, without access to unrelated systems or more destructive capabilities. These controls should be enforced through the operating environment rather than left to a prompt. Telling an agent not to use a destructive tool is guidance. Preventing that tool from being available is a guardrail. Standardized Access Creates a Tool Problem The easier it becomes to connect enterprise systems, the more tools an agent may theoretically have at its disposal. A single MCP server can expose dozens of actions. Connect an agentic AI platform to 10 or 20 enterprise applications and the available toolset can quickly grow into the hundreds. Giving every tool definition to an agent at once consumes context, increases token usage, and introduces irrelevant choices. If several connected systems expose functions called “get user,” “search,” or “update record,” the agent must determine which one applies. The answer is progressive disclosure: Provide tools only when they become relevant. A specialized agent should begin with a focused set of capabilities rather than unrestricted access to the enterprise. A Jira service agent does not need tools for HubSpot, GitHub, or Microsoft 365. Limiting the available toolset improves security, but it also helps the agent work more efficiently and reduces the likelihood that it selects the wrong tool. Agents also need instructions for using tools effectively. A tool definition may tell an agent that a search capability exists. It may not explain when to use keyword search rather than semantic or hybrid search, how to structure the query, or what to do if the results are incomplete. Skills provide that operating guidance. A skill functions like a handbook that explains how to complete a particular type of task and which tools should be used. Once the skill becomes relevant, the platform can disclose the associated capabilities. No organization would onboard an employee by providing access to every corporate system without explaining the job. Agents should not be treated differently. From Connectivity to Enterprise Readiness MCP is one of the most consequential advances in the emerging agent ecosystem. It gives organizations a standardized way to connect agents to the systems where enterprise work actually happens and can eliminate substantial repetitive integration development. But the connection is the starting point, not the complete architecture. To turn access into dependable business execution, enterprises still need an agentic AI platform that can govern tools, provide agents with the right instructions, manage context and memory, orchestrate complex work, and maintain visibility into what agents are doing. The opportunity is not to choose between MCP and an agentic AI platform. It is to combine them: MCP provides a universal way for agents to reach enterprise systems, while the platform provides the structure agents need to use those connections safely and effectively.

The Confusion Starts With the Word Memory Imagine giving an AI assistant a large software repository, several months of incident reports, and a complete set of operating procedures. All of it fits inside a million-token context window. It is tempting to say that the model now remembers the repository. That description hides three different mechanisms. The model may already contain general knowledge in its weights. The current request may temporarily expose it to your documents. A retrieval system may also search an external corpus and supply only the most relevant passages. These mechanisms can work together, but they differ in where information lives, how long it lasts, how it is updated, and how reliably it can be traced. A useful mental model is a knowledge worker at a desk. The context window is the desk surface. Model memory is the worker's learned expertise. RAG is a librarian who brings the most relevant pages to the desk. Application memory is the notebook or case file that the surrounding system saves between sessions. 1. Context Window: Temporary Working Space A context window is the maximum amount of tokenized information a model can reference while producing a response. Depending on the model and API, that budget can include system instructions, the current question, recent conversation turns, uploaded documents, retrieved passages, tool results, and generated output. The training corpus is not part of this window; it influenced the weights earlier. Figure 1: A context window is a finite, temporary token budget for one inference request. A large window increases capacity, but capacity is not the same as effective use. The model still has to locate relevant details, resolve contradictions, connect evidence across distant sections, and follow instructions while ignoring distractors. Research on long-context models has repeatedly separated advertised context length from usable task performance. The Lost in the Middle study found that answer quality could depend on where relevant evidence appeared in a long input. RULER expanded evaluation beyond simple retrieval and showed that performance can fall as sequence length and reasoning complexity increase. These results do not mean long context is ineffective. They mean that every model and workload must be tested at the lengths, evidence positions, and reasoning patterns expected in production. A model that retrieves one literal sentence from a long prompt may still struggle to reconcile ten scattered facts. Example: A team places 700,000 tokens of design documents, incident reports, and chat exports into one request. The correct timeout setting appears once in the middle of an old incident timeline, while a conflicting value appears near the end of a newer runbook. Both fit. The model must still identify which source is authoritative and current. A bigger window made the evidence available; it did not make the precedence decision automatic. What a Context Window Does Not Provide Persistence: unless the application saves and re-sends information, a later request does not automatically contain it.Guaranteed recall: a detail can be present yet overlooked or combined incorrectly.Freshness: an old document remains old even when the model can read all of it.Authority: the model needs signals such as version, date, owner, and precedence to resolve conflicting sources. Context caching is not memory: Caching can avoid recomputing an unchanged prefix and reduce cost or latency. It does not teach the model new facts or guarantee that the cached content will be used correctly. 2. Model Memory: Knowledge Encoded in Weights During training, optimization adjusts billions of numerical parameters so that the model becomes better at predicting and generating language. The resulting weights can encode language patterns, concepts, procedures, and factual associations. Researchers often call this parametric knowledge or parametric memory. Figure 2: Parametric memory is learned during training; application memory is stored and reintroduced by the surrounding system. Parametric memory is durable because the same weights are reused across requests. It is also compressed and probabilistic. The model does not store a clean row for every fact, expose a reliable last-updated timestamp, or provide a source record for each answer. It may recall common knowledge well while producing an incomplete or outdated answer for a rare fact. A normal inference request changes temporary activations, not the underlying weights. Telling a model that a service endpoint changed from /v1 to /v2 does not normally rewrite the model. The assistant can use that fact in the current context, but a fresh request will not know it unless the platform stores the fact, the caller sends it again, RAG retrieves it, or the model is updated. Example: A model's weights may encode what HTTP 503 means, how circuit breakers work, and how to analyze a stack trace. They will not automatically contain your private runbook published yesterday or remember that a particular customer case was escalated this morning. General expertise belongs in the model; current operational state belongs outside it. Application Memory Is a Separate Layer Chat products often appear to remember users because the application stores conversation turns, summaries, preferences, or workflow state. Before the next model call, the application selects some of that data and places it back into the prompt. This creates a stateful user experience around a model that does not permanently modify itself after each turn. Google Cloud describes the same production pattern as externalizing conversational memory into a backing service. The KV cache is another commonly confused term. It stores intermediate attention keys and values so token generation can proceed efficiently. It is runtime state for an active sequence, not durable semantic memory and not a replacement for a conversation or knowledge store. 3. RAG: Retrieve Before Generating Retrieval-augmented generation connects a model to knowledge stored outside its weights. The original RAG work described generation that combines parametric memory in a pretrained model with explicit non-parametric memory in an external corpus. In a production architecture, retrieval typically has an ingestion path and a query-time path. Figure 3: A RAG system indexes external knowledge, retrieves relevant evidence, and inserts selected passages into the model's context. During ingestion, the system parses content, divides it into useful units, attaches metadata and permissions, and builds one or more search structures. At query time, it authorizes the request, searches for candidate evidence, reranks or filters the candidates, and adds the selected passages to the prompt. Google Cloud's RAG documentation describes the same sequence of ingestion, transformation, chunking, embeddings, and retrieval [6]. RAG is not synonymous with a vector database. Retrieval can use keyword search, vector similarity, hybrid search, metadata filters, SQL queries, a knowledge graph, or a live API. The defining idea is that external information is selected at request time and used to ground generation. Example: An employee asks, 'How many days can I work from another country?' The model's training data cannot know the company's current policy. A RAG system can retrieve the latest approved policy, filter by the employee's region and role, place the relevant paragraphs in context, and return an answer with the policy version and source link. When the policy changes, the organization updates the source and index instead of retraining the model. RAG Is a Pipeline, Not an Accuracy Switch RAG improves grounding only when the pipeline returns good evidence, and the model uses it correctly. Failure can occur because a document was not indexed, chunk boundaries removed essential context, a query did not match the right terminology, permissions were applied too late, stale content outranked the current version, or too many weak passages diluted the answer. Retrieval quality and generation quality should therefore be evaluated separately. Key distinction: RAG does not make the model remember a corpus, and it does not expand the model's maximum context. It chooses a small, relevant subset of external knowledge to occupy that context. Context Window vs. Model Memory vs. RAG Dimension Context window Model memory RAG Primary role Temporary workspace for the current generation Learned capabilities and parametric knowledge Fetch external evidence for the current question Where information lives Tokens supplied to the current request Parameters or weights Documents, indexes, databases, graphs, or APIs Lifetime Until the request or managed session is cleared Until weights are changed As long as the external source is retained and indexed How it is updated Send different tokens on the next request Retraining, fine-tuning, or model editing Update the source and refresh the index or query path Best for Current documents, instructions, and recent conversation Language, broad patterns, stable skills, and behavior Private, changing, large, or traceable knowledge Typical failure Relevant detail is truncated, diluted, or poorly used Knowledge is stale, approximate, or inconsistently recalled Retriever returns missing, stale, irrelevant, or unauthorized evidence Traceability Only if supplied content has clear provenance Usually weak: an answer cannot point to a specific training record Strong when passages, versions, and source links are preserved Table 1: The three mechanisms differ in location, lifetime, update path, and traceability. How the Three Layers Work Together The strongest production design is usually hybrid. Model weights supply language ability, general knowledge, and reasoning patterns. The context window holds the current task. RAG supplies fresh or private evidence. Application memory preserves the small amount of user or workflow state that should survive between turns. Figure 4: A production application selects persistent external information and assembles a temporary context for each model call. End-to-end example: A support assistant receives a question about repeated 503 errors. The model's weights provide general networking and troubleshooting knowledge. Recent conversation and the current error message occupy the context window. RAG retrieves the latest service runbook and an active incident notice. Application memory supplies the case ID, environment, and steps already attempted. The model combines those inputs into a response, while the application records the next approved action for the following turn. Does a Million-Token Context Replace RAG? Not generally. Long context and RAG solve different problems. Long context increases how much information can be considered in one request. RAG reduces a much larger or changing information space to the evidence most likely to matter. Use Long Context When The complete input is known, bounded, and important to the current task.Relationships across distant sections matter and aggressive retrieval could omit a small but essential detail.You are analyzing a repository snapshot, contract set, transcript, or research packet as one coherent artifact.The model has been evaluated at the required length, and the latency and cost are acceptable. Use RAG When The corpus is larger than the usable context or changes frequently.Answers must reflect private data, current policies, live systems, or user-specific permissions.You need source links, versions, provenance, or an auditable evidence trail.Most questions require only a small portion of the available knowledge. Rely on Model Memory When The task depends on general language, broad domain patterns, coding ability, or stable learned skills.The information does not require exact provenance or frequent updates.You are choosing a model or fine-tuning behavior, style, classification boundaries, or output structure rather than maintaining a factual document store. Practical Design Rules Name the layer precisely. Say context, parametric knowledge, application memory, retrieval index, or runtime cache instead of calling everything memory.Budget the context. Reserve room for instructions, the user request, selected evidence, tool results, and the expected output. A maximum is a ceiling, not a target.Retrieve less, but better. Prefer a small set of authoritative, diverse passages over a large block of loosely related text.Authorize before retrieval. The model should never receive content the user was not permitted to access, even if the final answer is later filtered.Preserve provenance. Carry source, version, timestamp, owner, and document status into the retrieved context and the generated citation.Evaluate retrieval and generation separately. Measure whether the correct evidence was found before judging how well the model used it.Write important state explicitly. Save approved facts, decisions, and workflow progress in application storage instead of hoping the model will infer them from a growing transcript.Test realistic long inputs. Move evidence between the beginning, middle, and end; introduce conflicting versions; vary the number of relevant passages; and include questions with no supported answer. Conclusion A large context window is valuable, but it is not a permanent or perfectly searchable memory. It is a larger desk. Model weights are learned expertise, not a factual database. RAG is a retrieval mechanism that brings selected evidence to the desk. Application memory is the system that decides what to record and supply again later. The production question is therefore not, 'How much can the model remember?' It is, 'Which information belongs in weights, which belongs in the current context, which should be retrieved, and which state must the application persist?' Once those responsibilities are separated, accuracy, freshness, security, cost, and observability become much easier to engineer.

A mobile app crash or force-quit can kill the in-memory trail of user actions and network events leading up to the fault. Standard crash reports (like Apple's or Google's store consoles) often only show a stack trace without the user context or sequence of events. In a backend server, one might examine recent log lines or distributed traces on a smartphone. Those live events may never have been sent. For example, imagine a purchase flow where the app sends a payment request that succeeds on the server, then a network hiccup occurs, and the app finally crashes or times out. The payment was made, but the user never saw confirmation. If all the user's taps and network errors were only in volatile memory, the crash report will be blind to them. This gap is the "evidence that matters" that is lost. Mobile SDKs must handle such conditions gracefully by retaining information locally without exhausting limited device resources. Record Your Breadcrumbs A key step is to record breadcrumbs, which are the timestamped events representing user or system actions, into a local, bounded buffer, then persist that buffer to disk. For example, one might implement a function like this in Swift: Swift func recordBreadcrumb(_ event: String, metadata: [String: String]) { let breadcrumb = Breadcrumb(timestamp: Date(), event: event, metadata: metadata) buffer.append(breadcrumb) if buffer.count > 50 { buffer.removeFirst(buffer.count - 50) } persist(buffer) } Here, Breadcrumb is a simple struct (timestamp, event name, optional metadata). Each invocation appends a new entry and then removes the oldest if more than 50 entries are held, ensuring the buffer size stays bounded. The persist(_:) call writes the current buffer to local storage (for instance, a file or database). In practice, production libraries often enforce limits in this range like for instance, Bugsnag's iOS SDK raised its default breadcrumb capacity to 25 and capped it at 100. After this change, Bugsnag explicitly noted that it "persist[ed] breadcrumbs on disk to allow reading upon next boot in the event of an uncatchable app termination". This confirms the approach that durable storage of the last seconds of events survives a crash that the cloud can never directly observe. Of course, simply saving raw logs locally is not enough, as those breadcrumbs must be meaningful in context. One crucial practice is correlating mobile actions with backend telemetry. A request from the app should carry a trace or session identifier so that if the server logs complete or fail, they tag the same trace ID. In Swift, a request builder could inject custom headers: Swift func makeRequest(_ url: URL, traceId: String) -> URLRequest { var request = URLRequest(url: url) request.setValue(traceId, forHTTPHeaderField: "X-Trace-ID") request.setValue(sessionId, forHTTPHeaderField: "X-Session-ID") return request } Each API call includes, say, "X-Trace-ID" and "X-Session-ID", linking this mobile action to any backend spans. This technique is common in distributed tracing. For example, OpenTelemetry's iOS SDK will "inject W3C trace context headers into outgoing requests" so that the mobile span is connected to the backend trace. On the server side, log statements might then look like: Java String traceId = request.getHeader("X-Trace-ID"); log.info("Payment request completed traceId={} status={}", traceId, response.getStatus()); With these headers, if the app later crashes, developers can look up the same trace ID in backend logs to see what happened on the server. This cross-correlation turns the breadcrumbs into a coherent story as the mobile side shows "User tapped Pay at 10:05:12", the server log shows "Payment succeeded for trace 12345 at 10:05:13", and even if the mobile UI crashed at 10:05:30, the combined log tells a complete timeline. Actions After a Crash The final piece is recovery: uploading the persisted breadcrumbs after a crash or termination. On app launch or resume, code should check whether the last session ended unexpectedly (many SDKs provide such hooks). If so, load the saved breadcrumbs and send them to the observability backend before clearing them. For instance: Swift func recoverPendingTelemetry() async { let breadcrumbs = loadPersistedBreadcrumbs() guard !breadcrumbs.isEmpty else { return } do { try await telemetryClient.upload(breadcrumbs) clearPersistedBreadcrumbs() } catch { scheduleRetry() } } This asynchronous function reads the stored array of recent events. If there is any history (indicating a crash or interruption), it attempts to upload it via telemetryClient. On success, it clears the buffer, and on failure, it schedules a retry later (perhaps on network reconnection). Many analytics SDKs follow this pattern. For example, on iOS, Sentry will persist crash reports to disk, then try to send them on the next launch since synchronous network calls at crash time are unreliable. As one crash reporting doc notes, "Crash reports are sent on the next app launch, not at the time of the crash" due to system restrictions. The same principle applies to breadcrumbs, as they get queued up and transmitted later, filling in the gap left by the abrupt termination. All of these strategies must respect mobile constraints. The on-device journal should only cover a short time window and limited count. One can further trim the buffer by timestamp, for example: Swift func persist(_ events: [Breadcrumb]) { let recent = events .filter { Date().timeIntervalSince($0.timestamp) < 120 } .suffix(50) storage.save(Array(recent)) } This pseudo-code keeps only events less than two minutes old and the latest 50 entries, then saves them. Such aging ensures the app does not hoard days of data. It also helps enforce user privacy as only the immediate context is preserved (older actions automatically drop off), and storage of sensitive information can be avoided or encrypted. In practice, you would redact or omit any personal data from these breadcrumbs. Remember that writing and reading from disk consumes battery and I/O, so the buffer should be flushed efficiently (batch writes, compressed if possible). The upload itself should be done asynchronously and possibly deferred until the user's next action (for example, when they launch the app again and network is available). If many crashes occur without relaunch, periodic background sessions (using background fetch or push notifications) could flush small payloads, though iOS often limits what can run in the background. In exchange for these costs, the benefit is that the observability gap closes dramatically. A sequence like "Cart view loaded → Item selected → Checkout started → Payment API error → Crash" is preserved. Once uploaded, these breadcrumbs appear alongside crash stack traces in monitoring dashboards. The unique trace ID ties the client log to what the server saw, enabling full cross-service tracing. If a network error turned the app offline, those error events will be in the journal and reveal that, for example, "payment request timed out" instead of nothing. Developers can then implement safe behaviors. For example, if a payment did succeed on the server but the app didn't know, the client might accidentally retry. Knowing this pattern suggests using idempotency keys or a reconciliation API on the server side. If the same operation is replayed on relaunch, a proper backend check can detect the duplicate. Thus, capturing that last-chance context not only aids debugging but also guides the design of reliable end-to-end workflows. A Final Word In summary, mobile observability must assume that data in flight can vanish. The key is to treat the app's memory as a fragile buffer whenever a significant event occurs, snapshot a record of it durably (up to sensible limits), and sync to observability when possible. This prevents the situation where a crash or network flap leaves only the message "App Closed Unexpectedly" in the logs. By maintaining a bounded on-device breadcrumb log, propagating trace IDs, and recovering after crashes, one ensures that "the crash 30 seconds ago" isn't a mystery. The application's own records provide the missing scene, enabling engineers to reconstruct failures and keep mobile reliability high.

By Uthej Mopathi CORE