DZone Spotlight

In Part 1 of this series, we built a Quarkus-based MCP tool server and connected it to the Goose AI agent over Streamable HTTP. The tools worked, the demo was clean, and everything ran on localhost. But the moment you imagine 50 developers running Goose on their laptops, all hitting the same set of backend MCP servers, the architecture starts to crack. Who authenticated that tool call? Which role authorized the getAuditTrail invocation? What stops a poisoned tool name from injecting payloads into your backend? This article answers those questions by placing agentgateway — the Linux Foundation's open-source proxy for agentic AI traffic — between Goose clients and the Quarkus MCP microservices we built in Part 1. The Problem: Direct Agent-to-Backend Connections Don't Scale When Goose (or any MCP client) connects directly to a backend MCP server, every tool call is a point-to-point trust relationship: This works for demos. It breaks in production for three reasons: No authentication. The MCP Streamable HTTP endpoint accepts any JSON-RPC call. There is no token verification, no session binding, and no identity propagation.No authorization. Every caller can invoke every tool. An intern running Goose has the same access as an SRE — getAuditTrail, getOrderStatus, everything.No guardrails. A compromised or misconfigured agent can send tool names containing prototype-pollution payloads (__proto__), path-traversal sequences (../), or CRLF-injected headers. The backend has to defend itself alone. The Solution: agentgateway as a Unified Control Plane agentgateway is a Rust-based proxy purpose-built for AI agent traffic. It understands the MCP protocol natively — it doesn't just forward HTTP; it parses JSON-RPC envelopes, manages MCP sessions, and applies policies at the tool-call level. Here is the architecture we're building: Goose connects to agentgateway on port 3000. agentgateway validates the JWT, checks the caller's roles against tool-level RBAC rules, passes the call through an ExtMCP guardrail server that sanitizes headers and blocks poisoning attempts, and only then forwards the clean request to the Quarkus backend on port 8080. Prerequisites You'll need everything from Part 1, plus: agentgateway binary (v1.4+): Shell curl -sL https://agentgateway.dev/install | bash Verify your Part 1 Quarkus MCP server still works: Shell cd part1-quarkus-mcp mvn quarkus:dev Then confirm the MCP endpoint responds: Shell curl -s http://localhost:8080/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"curl","version":"1.0"}}' | jq . Step 1: Deploy agentgateway Alongside the Quarkus MCP Server Create the agentgateway configuration at part2-agentgateway/agentgateway/config-dev.yaml. This development config proxies MCP traffic without requiring JWT, so you can validate the plumbing first: YAML # yaml-language-server: $schema=https://agentgateway.dev/schema/config mcp: port: 3000 policies: cors: allowOrigins: - "*" allowHeaders: - mcp-protocol-version - content-type - mcp-session-id exposeHeaders: - Mcp-Session-Id targets: - name: customer-tools mcp: host: http://localhost:8080/mcp Start agentgateway: YAML agentgateway -f part2-agentgateway/agentgateway/config-dev.yaml Now test the proxied MCP endpoint. Note that agentgateway returns SSE format (event: message\ndata: {...}), so we extract the JSON from the data: line: Shell curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"curl","version":"1.0"}}' \ | grep '^data: ' | sed 's/^data: //' | jq . You should see the same customer-tools server info as Part 1, but the traffic now flows through agentgateway. Open http://localhost:15000/ui to see the agentgateway admin UI with your MCP target listed. Step 2: Add JWT Authentication With the proxy working, let's lock it down. The mcpAuthentication policy implements the MCP Authorization specification — it validates JWT bearer tokens on every MCP request and supports OAuth 2.1 with PKCE for browser-based flows. Update the config to part2-agentgateway/agentgateway/config.yaml: YAML # yaml-language-server: $schema=https://agentgateway.dev/schema/config mcp: port: 3000 policies: cors: allowOrigins: - "*" allowHeaders: - mcp-protocol-version - content-type - mcp-session-id - authorization exposeHeaders: - Mcp-Session-Id mcpAuthentication: issuer: http://localhost:9000 audiences: - "http://localhost:3000/mcp" jwks: url: http://localhost:9000/.well-known/jwks.json resourceMetadata: resource: http://localhost:3000/mcp scopesSupported: - "mcp:tools:read" - "mcp:tools:execute" bearerMethodsSupported: - header targets: - name: customer-tools mcp: host: http://localhost:8080/mcp How It Works When a Goose client (or any MCP client) connects to http://localhost:3000/mcp: Discovery. The client fetches /.well-known/oauth-protected-resource from agentgateway and discovers it needs a bearer token with the mcp:tools:execute scope.Token acquisition. The client runs the OAuth 2.1 Authorization Code flow with PKCE against the issuer (http://localhost:9000), obtains an access token, and includes it as Authorization: Bearer <token> on subsequent MCP requests.Validation. agentgateway downloads the JWKS from the issuer, verifies the token signature, checks exp, iss, and aud claims, and extracts the sub and role claims for downstream authorization.Forwarding. Only after validation does agentgateway forward the JSON-RPC call to the Quarkus backend. Connecting to a Real OIDC Provider For production, replace the issuer and JWKS URL with your OIDC provider. Here is an example using Keycloak: YAML mcpAuthentication: issuer: https://keycloak.example.com/realms/mcp audiences: - "https://gateway.example.com/mcp" jwks: url: https://keycloak.example.com/realms/mcp/protocol/openid-connect/certs provider: keycloak: {} agentgateway has built-in support for Keycloak, Auth0, Okta, Microsoft Entra ID, and other OIDC providers. Step 3: Configure Tool-Level RBAC With CEL Expressions JWT authentication tells you who is calling. MCP authorization tells you what they're allowed to do. agentgateway uses CEL (Common Expression Language) to define fine-grained, tool-level RBAC rules. Add the mcpAuthorization policy to your config: YAML mcpAuthorization: rules: # Operators can call any tool - 'has(jwt.roles) && "operator" in jwt.roles' # Viewers can only read status and health - > has(jwt.roles) && "viewer" in jwt.roles && mcp.tool.name in ["getCustomerStatus", "getZoneHealthLogs", "getSLACompliance"] # Auditors can access audit trail and SLA compliance - > has(jwt.roles) && "auditor" in jwt.roles && mcp.tool.name in ["getAuditTrail", "getSLACompliance"] How the Rules Work Each rule is a CEL expression that evaluates to true (allow) or false (deny). agentgateway evaluates them in order — the first match wins. RoleAllowed ToolsDenied ToolsoperatorAll five toolsNoneviewergetCustomerStatus, getZoneHealthLogs, getSLACompliancegetOrderStatus, getAuditTrailauditorgetAuditTrail, getSLACompliancegetCustomerStatus, getZoneHealthLogs, getOrderStatusNo roleNoneAll These aren't abstract labels — at Acme FinServ they map to real people and a real segregation-of-duties story: PersonaRoleWhy this scopeSofia — SRE, on-call for the platformoperatorNeeds to drive operational tools during incidents; full access is justified and logged.Acme Status Dashboard — an internal read-only serviceviewerShows customers and health at a glance; must never read getOrderStatus or getAuditTrail (PII/financial).Priya — external SOC 2 auditorauditorReviews the audit trail and SLA posture only. Giving her getCustomerStatus would violate least privilege — an auditor reading live customer data is itself a finding. The auditor scope is the one a SOC 2 assessor will scrutinize: it proves the audit function is separated from the operational function, and that access is granted by need, not convenience. agentgateway also auto-filters tools/list responses — if a viewer calls tools/list, they only see the three tools they're authorized to invoke. The agent never even learns that getAuditTrail exists. Available CEL Variables VariableDescriptionmcp.tool.nameThe tool being invoked (e.g., getCustomerStatus)mcp.tool.targetThe backend target name (e.g., customer-tools)jwt.subThe subject claim from the JWTjwt.rolesRole claims extracted from the JWThas(jwt.<claim>)Check whether a JWT claim exists Step 4: Prevent Tool Poisoning With ExtMCP Guardrails JWT and RBAC protect the identity layer. Guardrails protect the content layer. A valid, authenticated operator can still send a tool call with a poisoned name like getCustomerStatus/../../../etc/passwd or arguments containing <script> tags. The Quarkus backend's @Pattern annotations from Part 1 catch some of this, but defense in depth means filtering at the proxy too. agentgateway's ExtMCP guardrails intercept MCP method calls before they reach the backend, passing them through an external gRPC policy server that can inspect, mutate, or deny each call. Building the Guardrail Server With Quarkus gRPC Instead of relying on a third-party Docker image, we'll build our own ExtMCP guardrail server using Quarkus gRPC — keeping the entire stack in Java. The guardrail server lives in part2-agentgateway/extmcp-guardrail/ and implements the agentgateway ExtMCP protocol. First, the protobuf service definition (src/main/proto/extmcp.proto): ProtoBuf syntax = "proto3"; package agentgateway.dev.ext_mcp; option java_package = "com.example.guardrail.grpc"; import "google/protobuf/struct.proto"; service ExtMcp { rpc CheckRequest (McpRequest) returns (McpRequestResult); rpc CheckResponse (McpResponse) returns (McpResponseResult); } message McpRequest { repeated string service_names = 1; string method = 2; google.protobuf.Struct metadata_context = 3; optional bytes mcp_request = 4; repeated McpHeader headers = 5; } message McpRequestResult { oneof result { Pass pass = 1; bytes mutated = 2; AuthorizationError error = 3; } HeaderMutation header_mutation = 4; } message AuthorizationError { enum Code { UNKNOWN = 0; PERMISSION_DENIED = 1; RESOURCE_EXHAUSTED = 2; INVALID = 3; } Code code = 1; string reason = 2; optional bytes mcp_error = 3; } The Quarkus service implementation performs header sanitization and tool-poisoning detection: Java @GrpcService public class ExtMcpGuardrailService implements ExtMcp { private static final Pattern DANGEROUS_HEADER = Pattern.compile( "(?i)^(x-mcp-|x-forwarded-|x-real-ip)"); private static final List<String> BLOCKED_PATTERNS = List.of( "__proto__", "constructor", "../", "eval(", "exec(", "<script"); @Override public Uni<McpRequestResult> checkRequest(McpRequest request) { if (!"tools/call".equals(request.getMethod())) { return passRequest(); } // 1. Sanitize x-mcp-* headers for CRLF injection String headerError = sanitizeHeaders(request.getHeadersList()); if (headerError != null) { return denyRequest("header sanitization failed: " + headerError); } // 2. Check tool name and arguments for poisoning patterns if (request.hasMcpRequest()) { String poisonError = checkToolPoisoning( request.getMcpRequest().toStringUtf8()); if (poisonError != null) { return denyRequest("tool poisoning detected: " + poisonError); } } return passRequest(); } @Override public Uni<McpResponseResult> checkResponse(McpResponse response) { if (!"tools/list".equals(response.getMethod())) { return passResponse(); } // Append [guardrail-verified] marker to every tool description String original = response.getMcpResponse().toStringUtf8(); String mutated = original.replace("\"description\":\"", "\"description\":\"[guardrail-verified] "); return Uni.createFrom().item(McpResponseResult.newBuilder() .setMutated(ByteString.copyFrom(mutated, StandardCharsets.UTF_8)) .build()); } } Start the guardrail server on port 9001: Shell cd part2-agentgateway/extmcp-guardrail mvn quarkus:dev Configuring the Guardrail Policy Add the mcpGuardrails section to the agentgateway config: YAML mcpGuardrails: processors: - kind: remote host: "localhost:9001" failureMode: failClosed methods: tools/call: request tools/list: response The key settings: SettingValueWhyfailureModefailClosedIf the guardrail server is down, deny all tool calls rather than allowing unfiltered traffictools/call: requestPre-forwardInspect and sanitize before the call reaches the Quarkus backendtools/list: responsePost-forwardAnnotate or filter the tool list after the backend responds How Tool Poisoning Prevention Works When a tools/call request arrives, the guardrail flow is: Plain Text Goose → agentgateway → [JWT verified] → [RBAC checked] → → ExtMCP CheckRequest() → guardrail server inspects: 1. Scan x-mcp-* headers for CRLF injection 2. Validate header value lengths (≤ 256 bytes) 3. Check tool name for blocked patterns (__proto__, ../, eval()...) 4. Check tool arguments for injection payloads → Pass / Mutate / Deny → [if passed] → Quarkus MCP backend Sanitizing x-mcp-header Values The x-mcp-* headers carry protocol metadata between MCP clients and servers. A malicious client can inject CRLF sequences (\r\n) into these headers to smuggle additional HTTP headers or split responses. The guardrail server strips these by: Matching any header whose name starts with x-mcp-, x-forwarded-, or x-real-ipRejecting values that contain \r or \n charactersEnforcing a 256-byte maximum length on these header values Building a Custom Guardrail Server For production, implement the ExtMCP gRPC protocol with two methods: CheckRequest – Called before the tool call reaches the backend. Inspect the tool name, arguments, and headers. Return Pass, Mutate (rewrite params), or Deny with an AuthorizationError.CheckResponse – Called after the backend responds. Inspect the result. Return Pass, Mutate (redact sensitive data), or Deny. The part2-agentgateway/extmcp-guardrail/ directory contains the complete Quarkus gRPC implementation with the proto definition, the guardrail service, and the Maven build. Verifying the Guardrail With all three services running, test that the guardrail is active: Shell # Initialize session export MCP_SESSION_ID=$(curl -s -D - http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"curl","version":"1.0"}}' \ | grep -i "mcp-session-id:" | sed 's/.*: //' | tr -d '\r') # Complete handshake curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -H "mcp-session-id: $MCP_SESSION_ID" \ -d '{"jsonrpc":"2.0","method":"notifications/initialized"}' # List tools — descriptions should show the guardrail marker curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -H "mcp-session-id: $MCP_SESSION_ID" \ -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}' \ | grep '^data: ' | sed 's/^data: //' | jq '.result.tools[].description' Each tool description should start with the [guardrail-verified] marker, confirming that every tools/list response passes through the ExtMCP guardrail before reaching the client. Step 5: Configure Goose to Use agentgateway The final step is the simplest. In Part 1, Goose connected directly to the Quarkus backend: YAML # Part 1 — direct connection extensions: customer-tools: enabled: true type: http uri: http://localhost:8080/mcp headers: Content-Type: "application/json" For Part 2, change the URI to point to agentgateway: YAML # Part 2 — through agentgateway extensions: customer-tools: enabled: true type: http uri: http://localhost:3000/mcp headers: Content-Type: "application/json" Copy the updated config: YAML cp part2-agentgateway/goose-extension-config.yaml ~/.config/goose/config.yaml Now launch Goose and test the same prompts from Part 1: Plain Text Check customer status for CUST-4091 and verify health logs for their region The response is identical to Part 1, but the traffic now flows through agentgateway with JWT validation, RBAC enforcement, and guardrail inspection. You can verify this by checking the agentgateway UI at http://localhost:15000/ui — every tool call appears in the request log with its authentication status and policy decisions. The Complete Configuration Here is the full config.yaml combining all four security layers: YAML # yaml-language-server: $schema=https://agentgateway.dev/schema/config config: tracing: endpoint: http://localhost:4317 protocol: grpc sampling: parent: true default: 1.0 mcp: port: 3000 policies: cors: allowOrigins: - "*" allowHeaders: - mcp-protocol-version - content-type - mcp-session-id - authorization exposeHeaders: - Mcp-Session-Id mcpAuthentication: issuer: http://localhost:9000 audiences: - "http://localhost:3000/mcp" jwks: url: http://localhost:9000/.well-known/jwks.json resourceMetadata: resource: http://localhost:3000/mcp scopesSupported: - "mcp:tools:read" - "mcp:tools:execute" bearerMethodsSupported: - header mcpAuthorization: rules: - 'has(jwt.roles) && "operator" in jwt.roles' - > has(jwt.roles) && "viewer" in jwt.roles && mcp.tool.name in ["getCustomerStatus", "getZoneHealthLogs", "getSLACompliance"] - > has(jwt.roles) && "auditor" in jwt.roles && mcp.tool.name in ["getAuditTrail", "getSLACompliance"] mcpGuardrails: processors: - kind: remote host: "localhost:9001" failureMode: failClosed methods: tools/call: request tools/list: response targets: - name: customer-tools mcp: host: http://localhost:8080/mcp Bonus: Interactive Security Console The demo includes a browser-based SPA (index.html) that lets you visualize the entire security flow without touching the command line. The start-all.sh script serves it automatically on port 8888. Open http://localhost:8888/index.html and you'll see an enterprise-style console with: Config selector – switch between three tiered configs to see how each maps to a real deployment stage:Live stat tiles – session status, request count, tools discovered, and security checks passed/deniedAnimated architecture diagram – watch MCP requests flow from Goose through agentgateway's security layers to the Quarkus backend in real timeConfig-aware security layers – JWT, RBAC, and ExtMCP layers animate as "checking → passed" when enabled in the selected config, or appear as "skipped" with a badge when not configured ConfigUse CaseSecurity Layersconfig-dev.yamlLocal development — pure proxy pass-through for rapid iteration without security overheadNoneconfig-guardrails.yamlStaging / shared environments — blocks tool poisoning and header injection before requests reach the backendExtMCPconfig.yamlProduction deployment — full security stack with JWT identity verification, role-based tool access via CEL, and input sanitizationJWT + RBAC + ExtMCP The four demo steps — Initialize, List Tools, Call Tool, and Poison Test — make real MCP requests through agentgateway and display the JSON-RPC responses. Switching configs lets you demonstrate the difference: with config-dev.yaml, the poison test passes through unblocked; with config-guardrails.yaml, the ExtMCP guardrail catches and denies it; with config.yaml, every request also passes through JWT authentication and RBAC authorization before reaching the guardrail layer. What We Achieved Starting from the unprotected Quarkus MCP server in Part 1, we added four security layers without changing a single line of the backend Java code: LayerWhat It Doesagentgateway FeatureAuthenticationVerifies caller identity via JWT/OAuth 2.1 with PKCEmcpAuthenticationAuthorizationEnforces tool-level RBAC per rolemcpAuthorization with CELInput sanitizationBlocks tool poisoning and header injectionmcpGuardrails (ExtMCP)ObservabilityTraces every tool call through the proxyOpenTelemetry integration The Quarkus MCP backend remains a clean, focused tool server. All governance concerns live in the agentgateway configuration and the Quarkus gRPC guardrail service — keeping the entire stack in Java, exactly where platform engineers expect to find them. What's Next: Part 3 In Part 3: End-to-End Tracing and Observability Across Goose, agentgateway, and Quarkus, we'll wire up distributed tracing across the full agent traffic path. You'll see how a single Goose prompt generates a trace that spans the agent, the gateway, and the Quarkus backend — with tool-call latency, RBAC decisions, and guardrail verdicts all visible in a single Jaeger or Grafana Tempo timeline. We'll configure OpenTelemetry exporters in all three components and build a Grafana dashboard that gives platform teams real-time visibility into their agentic infrastructure. Stay tuned.

By Daniel Oh CORE

Fifteen years in, and the conversation I have most often with security leads still starts the same way: how's your perimeter, how's your endpoint coverage, how's your SOC staffed? Almost nobody opens with "how's your pipeline." That's the gap I want to talk about, because 2025 was the year the gap turned into a crater. Here's the number that should reorder every security roadmap for 2026: major DevOps platforms — GitHub, GitLab, Azure DevOps, and Atlassian's Jira and Bitbucket — patched 236 vulnerabilities in 2025, according to GitProtect's DevOps Threats Unwrapped report. Of those, 59% were rated high or critical: 14 critical, 126 high, 75 medium, 21 low. The trend line is worse than the total. Critical flaws jumped from 4 in the first half of the year to 10 in the second. High-severity findings climbed 55%, from 39 to 87 over the same stretch. November 2025 alone produced 36 patched vulnerabilities — 15% of the entire year's total in one month. These aren't obscure internal tools. GitHub alone hosts more than 180 million developers across 630 million repositories. When the platforms holding that much code accelerate their vulnerability disclosures quarter over quarter, that's not noise. That's a trend with a direction (DevOps.com, SecurityBrief). "You Hack the Runner, and You're in the Whole Building With a Master Key" I want to be careful here and use a real source, because this argument gets thrown around a lot without anyone actually backing it up. Paweł Budzan, a technology consultant and AI and cybersecurity architect at Xopero, put the stakes in terms I haven't heard bettered: a compromised CI/CD pipeline hands an attacker the repo, the cloud, the production secrets, and the deployment path all at once. Hack the runner, he said, and you're not in one room — you're in the whole building with a master key to every door. His point about "shift-left" culture stung a little because I've made the same mistake myself: plenty of teams scan the code, call it a day, and never apply the same scrutiny to the infrastructure that actually moves that code into production (GitProtect/Xopero). Budzan's list of the ten most commonly overlooked CI/CD vulnerabilities reads like a checklist written by someone who's cleaned up after every one of them: secrets echoed into build logs and never scrubbed; runners granted full cluster-admin rights because restricting them "might break the build"; a single shared runner handling both untrusted external pull requests and production deployments; blind trust in third-party GitHub Actions with a few stars and no real vetting; long-lived service tokens nobody rotates because rotation is scary; unprotected workflow YAML files that get far less code-review scrutiny than application logic. None of these are exotic. That's exactly the problem. When AI Turns a Script Kiddie Into a Supply-Chain Threat The part of Budzan's analysis that actually changed how I think about this: the barrier to entry for a serious pipeline attack has, in his words, dropped to the level of writing prompts in English. Malicious large language models — he named WormGPT and FraudGPT specifically, tools sold on dark-web forums and Telegram channels for a monthly fee — are trained specifically for offensive use, with none of the guardrails a mainstream model would apply. An attacker doesn't need deep AWS or Git expertise anymore. They can feed a workflow YAML file into one of these tools and ask it to locate secrets or draft a plausible-looking pull request. What comes back is a clean, credible "fix" that sails through code review, and when the pipeline runs, the token leaks straight out. Budzan's own estimate of the human-versus-sophistication split surprised me: from his practice, it's roughly 80% human error and 20% advanced attack — a developer under sprint pressure who leaves a token in a config file, tells themselves they'll fix it after the weekend, and never does (GitProtect/Xopero). The real-world version of that pattern already happened. In March 2025, attackers compromised the popular tj-actions/changed-files GitHub Action — retroactively rewriting version tags to point at a malicious commit — and the poisoned action ended up exposed in more than 23,000 repositories before it was caught and patched. It didn't require breaking any cryptography. It required trust in a dependency nobody was individually vetting (GitHub Advisory Database). The CI/CD Tools Themselves Are Now the Target, Not Just the Delivery Mechanism Then there's the newer wrinkle: the AI tooling that's increasingly wired directly into these pipelines is itself shipping critical flaws. In April 2026, researchers at Novee Security disclosed a maximum-severity, CVSS 10.0 remote code execution vulnerability in Google's Gemini CLI and its companion run-gemini-cli GitHub Action — a flaw that let an unprivileged external attacker force their own malicious content to load as the tool's configuration, effectively turning an AI coding assistant embedded in a CI/CD workflow into a remote-execution foothold. Google assigned it the highest score the scale allows. This is the pattern Budzan's framework predicts almost exactly: the pipeline as master key, an AI tool as the unguarded runner, and a single crafted input as the way in (Novee Security). The Uncomfortable Final Word: No Defense Is 100%, So Plan For the Day It Fails What I respect about Budzan's take is that he doesn't oversell prevention. Backup and disaster recovery, he argues, are the actual last line of defense — and anyone who claims a tool stops 100% of attacks is selling you something, full stop. His specific standard for what counts as a real backup is worth repeating exactly because it's so unglamorous: isolated, offline, in a separate cloud tenant and a separate account — not a separate folder in the same S3 bucket, and definitely not sitting in the same AWS account as production. If the same compromised credentials that let an attacker into your pipeline can also reach your backups, you don't have a backup. You have a second copy of the crime scene (GitProtect/Xopero). Where This Leaves Security Leaders Every thread here points the same direction: CI/CD pipelines have quietly become as valuable a target as production itself, and in most organizations they're guarded with a fraction of the rigor. The 236 patched vulnerabilities, the accelerating severity curve through the back half of 2025, the tj-actions compromise, the Gemini CLI RCE, and Budzan's own field experience all describe the same failure mode from different angles: trust extended to a pipeline, a runner, a third-party action, or an AI assistant, with nobody watching that trust closely enough. I don't think the fix is more scanning tools bolted onto the same workflow. It's treating the pipeline itself — the runners, the tokens, the workflow files, the AI assistants wired into it — with the same access control, isolation, and adversarial testing you'd apply to a production database. And it's accepting, the way Budzan does, that prevention will eventually fail, so the backup sitting behind it needs to be somewhere the attacker who got in through the front door can't also reach. Most teams I talk to still don't have that. The threat landscape isn't going to wait for them to build it. Sources are linked inline throughout. Reporting and analysis are current as of July 2026.

By Igboanugo David Ugochukwu CORE

Deployments become harder when one release changes both the machinery governing a Kubernetes platform and the workloads that depend on it. A new CustomResourceDefinition, admission webhook, controller, or routing API can change what the cluster accepts and how it behaves; a new application image can simultaneously depend on those changes. Argo CD and Argo Rollouts solve different parts of this problem. Argo CD is suited to establishing declarative prerequisites in a deterministic order, while Argo Rollouts limits production exposure as a workload version moves toward stable. Safe delivery comes from composing those responsibilities rather than treating either controller as a universal deployment engine. Argo Rollouts documentation explicitly discourages using Rollouts for infrastructure components such as cert-manager, CoreDNS, and NGINX. Compatibility Must Exist Before Progression Starts The core rule is an expand-and-contract sequence. Control-plane changes first add capabilities without removing behavior required by the running data plane. Only after old and new workloads can both operate against the expanded control plane should production traffic move to the new workload. Destructive cleanup follows after promotion. This separation preserves the ability to run multiple workload versions during progressive delivery while avoiding an immediate dependency on an irreversible platform change. This matters most for Kubernetes APIs. A CRD can serve multiple versions, with exactly one storage version, and conversion webhooks can translate between representations. Kubernetes documentation recommends removing an old API version only after stored objects have migrated away from it, and conversion support is no longer required. That lifecycle maps naturally to staged GitOps delivery: add the new version and compatible controller behavior, roll out consumers, migrate stored state if necessary, then remove the legacy version later. The same principle applies to admission, routing, and controller behavior. During the transition, the new policy must not reject objects still produced by the stable workload, and the new routing configuration must preserve the stable path. This compatibility window keeps rollback viable while both data-plane versions may coexist, which is also a prerequisite called out by Argo Rollouts for applications using progressive delivery. Let Argo CD Establish the Prerequisites Argo CD sync phases and waves provide deterministic ordering inside an Application. Resources are ordered by phase, numeric wave, kind, and name. Argo CD applies the first wave containing an out-of-sync or unhealthy resource and continues only as earlier work becomes synchronized and healthy. Negative waves place infrastructure prerequisites ahead of workloads. A CRD and controller can therefore precede the Rollout resource without external pipeline orchestration: YAML metadata: annotations: argocd.argoproj.io/sync-wave: "-20" # CRD YAML metadata: annotations: argocd.argoproj.io/sync-wave: "-10" # controller YAML metadata: annotations: argocd.argoproj.io/sync-wave: "0" # Rollout The exact values are less important than the health boundary between them. A controller Deployment that never becomes healthy prevents the dependent wave from advancing. Waves are sequencing gates rather than transactions, so they cannot make an incompatible schema change atomic; backward compatibility must still survive the interval between applied resources. For custom resources with nonstandard status, Argo CD supports custom Lua health checks, allowing readiness to reflect controller reconciliation rather than mere object creation. Hooks add active validation. A failing PreSync hook stops synchronization before normal resources are applied, while PostSync runs after synchronized resources are healthy and can perform smoke tests. Safety-critical flows should avoid selective sync because hooks do not run during selective synchronization and that operation is not recorded in history. Pruning also warrants conservative treatment during control-plane evolution. Argo CD supports Prune=false and Prune=confirm, allowing critical objects to be protected from automatic deletion. When a CRD is introduced in the same sync as its custom resources, Argo CD automatically skips dry run for those new custom-resource types, avoiding failure before the API exists. Let Argo Rollouts Control Production Exposure After the control plane is compatible and healthy, the data-plane update can progress independently. A canary rollout exposes changes in steps rather than replacing all replicas at once. With a traffic-routing provider, Argo Rollouts manages stable and canary Services and adjusts routing according to Rollout state; the stable ReplicaSet remains available while traffic shifts. For Istio host-level traffic splitting, the Rollout controller also updates the referenced VirtualService weights to match the current canary step. A concise policy can combine small initial exposure, automated analysis, and progressively larger weights: YAML strategy: canary: stableService: checkout-stable canaryService: checkout-canary trafficRouting: istio: virtualService: name: checkout routes: [primary] steps: - setWeight: 5 - pause: {duration: 2m} - analysis: templates: - templateName: checkout-slo - setWeight: 25 - pause: {duration: 5m} - setWeight: 50 - analysis: templates: - templateName: checkout-slo The stable and canary Service references give Rollouts explicit targets for traffic control, while the named Istio route identifies the route whose weights may change during progression. This pattern follows the host-level traffic-splitting model documented for the Istio integration. An AnalysisTemplate defines metrics, measurement intervals, and success or failure conditions, so failed analysis can stop progression automatically. Blue-green deployments provide related gating through pre-promotion analysis before the active Service switch and post-promotion analysis that can abort and restore traffic to the previous stable ReplicaSet. The analysis signal should reflect behavior that distinguishes stable from canary operation. Error rate, latency, saturation, or a domain success ratio provides evidence beyond pod readiness. Timed pauses create an observation window, but time alone is not a success criterion; automated analysis turns that window into an explicit promotion gate. Readiness establishes that a process can receive traffic, while progressive analysis determines whether receiving production traffic remains acceptable. Prevent Git Reconciliation From Fighting Traffic Management Progressive delivery creates an ownership boundary. Argo CD owns desired configuration in Git, while Argo Rollouts intentionally mutates live routing fields as progression advances. Without explicit diff rules, Git reconciliation can reapply static weights while Rollouts is setting dynamic weights, creating brief traffic-weight flapping. Argo Rollouts documents this conflict for Istio VirtualServices and recommends ignoring differences together with applying only out-of-sync resources. The Argo CD Application can exclude only controller-owned weight fields while keeping the rest of the route declarative: YAML spec: ignoreDifferences: - group: networking.istio.io kind: VirtualService jqPathExpressions: - .spec.http[].route[].weight syncPolicy: syncOptions: - ApplyOutOfSyncOnly=true - RespectIgnoreDifferences=true Argo Rollouts specifically documents ignoring VirtualService HTTP route weights as a way to prevent dynamic Rollouts changes from making the Argo CD Application appear out of sync. ApplyOutOfSyncOnly=true then prevents already synchronized resources from being reapplied unnecessarily. RespectIgnoreDifferences=true makes the ignore policy apply during synchronization rather than only during diff calculation. The boundary should remain narrow. Ignoring only route weights preserves Git ownership of hosts, matches, destinations, and other static policy while granting Rollouts authority over the values that legitimately vary during progressive delivery. Broad exclusions weaken drift detection and obscure changes unrelated to the rollout itself. Argo CD supports path-level and field-manager-based difference customization specifically to constrain such exceptions. Failure Must Converge Back to Git An aborted rollout is not a complete GitOps rollback. Argo Rollouts can restore the stable ReplicaSet, but Git may still request the failed image. The Rollout then remains degraded because the live stable state differs from the desired state. Argo Rollouts documentation states that reapplying the previous stable manifest restores health and is recognized as a rollback, allowing the controller to fast-track the stable ReplicaSet instead of replaying normal analysis steps. That distinction is critical for combined changes. A data-plane failure should normally revert the workload commit while leaving a backward-compatible control-plane expansion in place. Removing newly added API or routing capability during the same emergency action can make recovery less predictable. Control-plane contraction belongs in a later, independently verified change. Sync windows can further restrict high-risk synchronization to approved periods because Argo CD supports allow and deny windows scoped by application, namespace, or cluster. Safe delivery with Argo CD and Argo Rollouts is therefore a matter of ownership, compatibility, and sequencing. Argo CD should establish and verify backward-compatible platform prerequisites, while Argo Rollouts should control how much production traffic reaches the new data plane and stop promotion when evidence turns negative. Controller-owned routing fields must be excluded narrowly from Git reconciliation, and an aborted rollout must be followed by a Git change that restores the stable desired state. When expansion precedes exposure and contraction follows verified promotion, control-plane evolution and application delivery remain independently reversible, turning progressive deployment into a reliable safety mechanism rather than merely a rollout technique.

By Akhil Madineni CORE

A pod can look healthy in Kubernetes while the model behind it is still not ready to answer a request. That is the gap I wanted to measure. Kubernetes Ready means the pod passed the readiness condition you configured. It does not automatically mean the model is loaded, resident in memory, or able to complete inference. For a normal web service, an HTTP check is often good enough. With an LLM serving pod, it can be too shallow. The process may be running. The API may respond. The model file may even be on disk. The first real request can still spend several seconds loading the model before it completes. I ran controlled Ollama recovery experiments on Kubernetes to see how big that window was. Ready Is Only One Point in the Recovery Path I measured five timestamps: Plain Text T0 - pod replacement requested T1 - Kubernetes reports Ready T2 - inference runtime responds to HTTP T3 - first post-recovery inference request begins T4 - inference request completes successfully Figure 1: Kubernetes Ready vs. Functional Recovery That gave me four useful timings: Plain Text Kubernetes recovery = T1 - T0 Runtime recovery = T2 - T0 Functional recovery = T4 - T0 Ready -> inference gap = T4 - T1 The last one is where the problem becomes visible. The Results I ran 10 pod-replacement tests for each configuration: local Minikube on Mac, CPU-onlyAzure Standard_D16s_v5 Linux VM running Minikube, CPU-onlyOllamallama3.2:1bllama3.2:3bsame 2 CPU / 4 GiB container limit for the 1B and 3B comparison Figure 2: LLM Recovery Experiment Architecture Mean results: MetricLocal 1BLocal 3BAzure 1BAzure 3BKubernetes Ready1.66 s1.96 s1.61 s1.69 sRuntime reachable2.43 s2.44 s2.19 s2.17 sFunctional recovery11.11 s16.27 s5.43 s7.73 sReady -> inference9.45 s14.31 s3.83 s6.05 sModel load5.51 s8.60 s2.16 s3.96 s Kubernetes reported the pod Ready in about two seconds or less in all four configurations. Successful inference came later. The mean Ready-to-inference gap ranged from about 3.8 seconds to 14.3 seconds. The Azure environment was faster than the local environment for the inference-dependent part of recovery, but the gap was still there. I did not try to explain the cross-platform difference with one cause. CPU, storage, virtualization, architecture, and cache behavior can all affect the result. The point was simpler: Kubernetes recovery and inference recovery were not the same event. There Is More Than One Kind of "Ready" The experiments also exposed a few other states that are easy to mix together. The Runtime Can Be Up While the Model Is Gone One early version used emptyDir for Ollama model storage. After pod replacement, Ollama started normally. But: Shell ollama list returned no model. The runtime had recovered. The model artifact had not. Moving the model data to a PVC fixed the persistence problem. The Model Can Be on Disk Without Being Loaded A larger llama3.1:8b test made this very clear. Before inference, ollama list showed the model artifact, but ollama ps showed nothing resident. Cgroup memory usage was only around 14 MiB. After the first request, the model became resident and memory rose to roughly 5.27 GiB. So "model exists" and "model is ready to serve" are different checks. A Warm Node Can Make Recovery Look Better I also ran 10 warm-cache and 10 cold-cache tests for the 3B model on the same Azure node. For the cold condition: Shell sync echo 3 > /proc/sys/vm/drop_caches This clears the Linux page cache, dentries, and inode caches. It is a host filesystem/page-cache test, not an Ollama-specific model cache. metricwarmcoldFunctional recovery7.58 s8.09 sReady -> inference5.70 s6.26 sModel load3.95 s4.61 sRequest wall time5.11 s5.70 s Model load increased by about 16.6% under the cold condition. Kubernetes recovery barely moved. That is a useful warning for repeated recovery tests on the same node: the host may be helping more than you realize. Model Residency Can Overlap During memory testing, loading the 3B model under a 4 GiB limit once failed with: Shell signal: killed It looked like the 3B model did not fit. That was not the actual problem. A 1B model from an earlier request was still resident. When I tested the 3B model alone under the same limit, it worked, and the cgroup showed no OOM kill. The failure came from overlapping residency, not the 3B model by itself. A simple runtime health check would not have told me that. So What Should Readiness Check? A normal readiness probe usually asks something like: Plain Text Is the HTTP endpoint responding? That proves the runtime is reachable. For an LLM workload, I care about a stronger question: Plain Text Can this pod actually complete inference with the model it is supposed to serve? One way to test that is with a minimal inference request: YAML readinessProbe: exec: command: - sh - -c - | curl -sf -X POST http://localhost:11434/api/generate \ -H 'Content-Type: application/json' \ -d '{"model":"llama3.2:1b","prompt":"ping","stream":false}' \ | grep -q '"done":true' periodSeconds: 2 failureThreshold: 1 The exact command will depend on the serving image. The point is not curl. The point is that readiness now checks the model-serving path, not just the process. What Happened During Rollouts? For the 3B readiness test, I sampled Kubernetes EndpointSlice state at roughly 0.5-second intervals during 10 local rollouts and 10 Azure rollouts. metriclocal 3Bazure 3BMean new-endpoint non-serving duration47.6 s11.0 sSampled intervals with zero ready + serving endpoints00Rollouts observed1010 Across those 20 rollouts, I did not observe a sampled interval with zero ready-and-serving endpoints. That is not the same as proving packet-level availability between every sample. What it does show is that the replacement endpoint stayed out of Service eligibility until the inference-aware readiness condition succeeded. That is much closer to what I wanted Ready to mean. Readiness Is a Contract This was the main lesson for me. Readiness is not a universal definition of application health. It is a contract between the workload and Kubernetes. For a normal API, the contract might be: Plain Text My process is initialized and can accept requests. For an LLM workload, it may need to be closer to: Plain Text The runtime is running. The model exists. The model can be loaded. Inference can complete. If the probe only checks the first line but the team reads Ready as all four, the problem is not Kubernetes. The signal is just weaker than the expectation. What This Does Not Prove These tests were CPU-only. They used Ollama. They measured same-node pod replacement. And they used 10 repetitions per condition. So the numbers here should not be treated as universal timings or production SLAs. Cold-node relocation is also a separate problem. Moving an LLM workload to another node brings node-local cache state and possibly image or model acquisition into the recovery path. I am measuring that separately rather than mixing it into these same-node results. Takeaway In these experiments, Kubernetes readiness came back quickly. Inference recovery followed a different timeline. The mean Ready-to-inference gap ranged from about 3.8 seconds to 14.3 seconds, depending on the model and environment. The fix is not to distrust Kubernetes. It is to make the readiness condition represent the state you actually care about. A pod can be healthy. The runtime can answer HTTP. The model can exist on disk. And inference can still not be ready. Those are different states. For the full experiment setup, raw results, methodology, environment captures, and ongoing cold-node work, see the project write-up and repository: https://github.com/opscart/k8s-llm-recovery-lab. For the full experiment setup, methodology, raw results, environment captures, and ongoing cold-node work, see the complete OpsCart write-up and project repository.

By Shamsher Khan CORE

In a previous article, Working with Spreadsheets in Java: A Practical Overview, we walked through the common scenarios where Java applications need to interact with spreadsheets and the categories of tools available for the job. One of the factors mentioned there was support for modern Excel formulas — a topic that deserves more space than a single bullet point. Java applications interact with Excel more often than most teams plan for: file uploads from finance, calculation logic authored in a workbook, reporting exports back to business users. The files these users produce today are not the same as the files they produced five years ago. Excel 365 and Excel 2021 introduced a new formula model, and workbooks authored in those versions routinely use it. Depending on which library you use, those formulas may evaluate correctly, fail silently with stale cached values, or throw exceptions at recalculation time. This article goes deeper on that topic: what dynamic arrays and spill behavior are, what the new function set looks like, why supporting them is technically difficult, and what Java developers should look for when evaluating whether a library handles them correctly. Why You're Seeing Them in Real-World Workbooks Dynamic arrays spread rapidly because they eliminate many of the helper columns, copied formulas, and Ctrl+Shift+Enter array formulas that older Excel workbooks depended on. Workbooks become shorter, easier to audit, and easier to maintain. As organizations migrate to Microsoft 365, these newer formulas increasingly appear in spreadsheets exchanged with Java applications, even when the application itself hasn't changed. What Changed: One Formula, Many Values Before dynamic arrays, formulas that returned multiple values generally required a pre-sized array range and legacy array-formula syntax. Dynamic arrays changed this by allowing a single formula to return a variable-sized array and automatically spill into neighboring cells. For example: =UNIQUE(A1:A6) Entered in one cell, this returns the full list of distinct values from A1:A6. The result fills as many cells as there are distinct values. If the source data changes and the number of unique values changes, the spill range automatically grows or shrinks. This is not just a new function. It is a change in the evaluation model itself. The Mechanics of Spill A spilled formula produces a region of cells with specific roles: The anchor cell is the one cell that contains the formula. It "owns" the result.The spilled cells are the neighboring cells that display the additional values. They do not contain formulas of their own; they mirror slices of the anchor's result. You can reference the entire spilled range from another formula using the # operator. The # reference is not a fixed cell range such as A1:A6; it refers to whatever range the anchor currently spills into. If A1 contains =UNIQUE(...) and the result spills into A1:A6, then =COUNTA(A1#) counts the values in the entire spilled range. If the spill range grows or shrinks, the # reference adjusts automatically. #SPILL! errors. If the range a formula needs to spill into is blocked by an existing value, a merged region, or an Excel Table, the formula cannot spill, and the anchor cell shows #SPILL! instead of a result. Clearing the obstruction allows the formula to complete. Implicit intersection with @. Older Excel silently reduced arrays to single values in many contexts. Modern Excel returns the full array unless the formula uses the @ prefix. For example, =A1:A10 entered in a cell in modern Excel spills the values from A1:A10, while =@A1:A10 applies implicit intersection and returns the value corresponding to the formula's row. Files migrated from older Excel versions often contain automatically inserted @ prefixes to preserve their original behavior. The New Function Family The modern functions commonly associated with Excel's dynamic-array model can be grouped into three broad categories. It is worth understanding the grouping because the groups behave differently. Group 1: Language Features These are not really functions in the traditional sense. They add expression-level constructs to Excel's formula language. LET binds names to intermediate values inside a formula, so you can write =LET(total, SUM(B2:B100), tax, total*0.1, total+tax) instead of repeating SUM(B2:B100) three times.LAMBDA defines a reusable function inside a workbook. Combined with named ranges, LAMBDA effectively adds user-defined functions without VBA.ISOMITTED is used inside LAMBDA to detect whether an optional argument was supplied. These functions do not inherently produce a spilled array. LET returns the result of its calculation, which may itself be an array. Group 2: Dynamic Array Functions These are the functions people usually mean when they talk about "the new Excel functions." These functions are designed to return arrays, and when their results contain multiple values, Excel can spill those results into neighboring cells. UNIQUE returns distinct values from a range.SORT and SORTBY return sorted arrays.FILTER returns rows that match a condition.SEQUENCE generates a sequence of numbers.RANDARRAY generates an array of random numbers. Array-shaping functions form a subset of this group. They take arrays as input and return reshaped arrays: CHOOSECOLS, CHOOSEROWS, DROP, EXPAND, HSTACK, VSTACK, TAKE, TOCOL, TOROW, WRAPCOLS, WRAPROWS. TEXTSPLIT also fits here — it splits a string into an array. Other functions, including BYROW, BYCOL, MAP, REDUCE, and SCAN, build on the same dynamic array model. Group 3: Scalar Functions Added in the Same Era Dynamic arrays are primarily an evaluation model; modern functions are a collection of functions that take advantage of, or coexist with, that model. Group 3 functions were introduced as part of the broader set of modern Excel functions, but they are not themselves primarily array-producing functions. XLOOKUP and XMATCH are modern replacements for VLOOKUP and MATCH. They normally return a single value, though they can return an array when passed an array of lookup values.TEXTAFTER and TEXTBEFORE return substrings.VALUETOTEXT and ARRAYTOTEXT convert values to text (ARRAYTOTEXT takes an array as input but returns a single string). These are often lumped in with dynamic array functions because they arrived together, but their evaluation model is closer to VLOOKUP than to UNIQUE. Why This Is Hard for a Formula Engine Supporting these features is not just a matter of adding new function names to a list. The dynamic array model requires substantial changes to the evaluation engine itself. A traditional one-cell-at-a-time formula model is not sufficient to implement dynamic arrays. An engine must be able to represent a formula whose result has a variable shape and propagate that result across multiple cells. A modern engine has to handle four additional concerns: Array-shaped results. A formula's return value may be a 2D array whose dimensions depend on the input data. =UNIQUE(A1:A100) returns a different number of rows depending on how many unique values the range contains. The engine must determine the result shape at evaluation time, not at parse time. Spill range tracking. The engine must reserve the cells the formula spills into and prevent other content from occupying them. When something occupies a spill target, the anchor must return #SPILL! rather than overwrite the obstruction. The reserved region must also update when the shape of the result changes. Downstream references. Expressions like A1# refer to the entire spilled range. When the shape of the anchor formula changes, every downstream reference must be re-evaluated with the new dimensions. This makes the dependency graph more dynamic than in a one-value-per-cell model. Implicit intersection compatibility. Older Excel silently collapsed arrays to single values in many contexts. Modern Excel returns the whole array. When files authored in older Excel are opened in modern Excel, @ prefixes are inserted automatically to preserve original behavior. An engine that reads modern .xlsx files needs to honor the @ operator, or the imported formulas will produce different results. Adding these behaviors to an engine designed around the one-formula-one-value model is a substantial rewrite, not an incremental feature addition. This is part of why support across the Java ecosystem has been uneven. What Java Developers Should Check Support for these capabilities varies significantly across Java spreadsheet libraries. Some engines were originally designed around traditional one-cell-one-result evaluation and only implement subsets of the modern Excel model. Others have extended or redesigned their evaluators to support dynamic arrays. Rather than relying on feature lists, it is worth validating behavior against workbooks representative of your own application. If your application needs to evaluate modern Excel formulas, the following checks are worth running before committing to a library. Test with a file containing a spilled formula. Create a small .xlsx with =UNIQUE(A1:A100) or =SORT(A1:A100) in a cell. Load it in your candidate library and try to recalculate the anchor cell. A library that supports dynamic arrays will return the array; one that does not will typically throw an exception or return only the first value. Check for the # spill operator. In the same file, add another cell containing =COUNTA(A1#) where A1 is the anchor. This tests whether the library understands spilled range references, which is a separate capability from evaluating the anchor formula itself. Test the @ operator. Add =@A1:A10 in a cell and check whether the library correctly returns the value at the current row rather than the full array. Files migrated from older Excel routinely contain @ prefixes; a library that doesn't handle them will produce different results than Excel. Test with LET and LAMBDA. Write a formula like =LET(total, SUM(A1:A100), total * 1.1) and check both evaluation and .xlsx round-trip. Test LET and LAMBDA independently. Parsing, preserving, and evaluating these functions are separate capabilities, so a library that can read or write the formula text may not necessarily be able to evaluate it correctly. Test round-trip. Save the workbook, reopen it in Excel, and check that the formulas still produce correct results. Some engines strip modern constructs on save. Check what happens on failure. When a library encounters a function it does not implement, does it raise an exception, return an error value, or silently fall back to the cached value from the file? Silent fallback is the most dangerous behavior because it masks the problem during development and only fails in production when the data changes. Conclusion Excel's formula language has changed more in the last few years than in the two decades before it. Dynamic arrays, spill behavior, and the new function set are not experimental — they are standard in Excel 365 and Excel 2021, and they show up in workbooks that Java applications routinely have to process. For Java developers, the practical implication is that "Excel formula support" is no longer a single property that a library either has or doesn't have. There are several distinct capabilities involved, and libraries vary widely on each. As covered in the previous article, the Java spreadsheet landscape spans open source libraries such as Apache POI, commercial headless engines, and embedded spreadsheet components like Keikai. Whichever category fits your use case, the checks above are a reasonable way to verify that a candidate library handles modern Excel behavior against the workbooks your real users produce.

By Hawk Chen CORE

Almost every conversation about observability budgets I have been in ultimately arrives at the same conclusion: “we need to reduce our telemetry volume.” That sentence is usually followed by a number. Thirty percent. Half. Whatever the finance spreadsheet needs it to be. Then someone says the thing that makes everyone in the room relax. "Good news: most of it’s noise anyway. We can cut the volume and improve the signal at the same time." It is a comforting idea, because it turns an unpleasant budget cut into an engineering improvement. But it only gets you so far. It is true that some of your telemetry is noise, but it’s much less of it than "most." But it doesn’t follow that you can then simply cut volume and automatically improve signal. There is real noise in your telemetry, and I will get to where it lives. But "reduce volume by thirty percent" is not an instruction to remove noise. It is an instruction to remove bytes, and your noise and your signal are made of the same bytes. The target doesn’t differentiate, so what you end up removing is dictated by whatever is easiest to find. What is easy to find is a category. All INFO logs. All user agent strings. Everything below WARN. Categories are easy because your pipeline already knows them, and that is the whole of their appeal. Whether a category happens to be useful or not is a coincidence. So your telemetry is full of junk, but the problem isn't that there is too much of it. It is that by adopting a volume reduction target, you are not looking at whether the telemetry data you cut has any value. Once you hit the byte target, the exercise is seen as a success. Two Axes, Loosely Coupled When you change your telemetry pipeline, two things move. The first is easy: bytes through the pipeline, or active series if it is metrics, or whichever unit your contract happens to price. One number, on a chart, updated hourly. This is what we call volume. The second is what those bytes enable you to find out. Whether, six weeks from now, you can still answer the question in front of you. This is what we commonly call signal, and everything else is noise. It is measurable, but it is not measured in bytes, and it is probably not on any chart you are currently looking at. The two are related, obviously. Delete everything, and both go to zero. But across the range you actually operate in, they are only loosely coupled, because the bytes in your telemetry are not distributed anything like the value. The smallest fields often do the most work. A tenant identifier is a few dozen bytes, and it tells you whether something is impacting everyone or just one customer. A trace ID is thirty-two hex characters, but without it you are correlating your signals by hand, across three browser tabs. If the resource attributes naming the deployment are missing, good luck telling a bad release from a bad node. On the flipside, fields that take the most space frequently do the least. Meanwhile, the ten-thousandth identical stack trace in an hour is several kilobytes and tells you the same thing the first one did. So a lever that operates on bytes will spend most of its effect in the wrong place, and no exchange rate exists that would let you convert one axis into the other. Drawing them as two axes is a crude picture for that reason. But it is still worth doing, because it separates four moves that a byte count reports as only two. Let me walk through each one. Q1: The Free Lunch, Real But Limited This is the noise I promised at the top, and finding it feels great. Every tutorial on making your observability pipeline better has these prominent examples: Kubernetes liveness and readiness probes logging every few seconds, per pod, forever. A debug logger somebody enabled during an incident last quarter, and nobody turned off. The same records shipped twice because a node agent and an application-level exporter both picked them up. Most of this can go. But be careful even here, because a health check is not the same thing as a worthless record. Probe failures and probe latency are how you find a sick node before your users do. What you want to drop is the successful ones, the ninety-nine percent that only ever confirm that nothing is happening. The filter processor will do it: YAML processors: filter/healthchecks: log_conditions: - 'IsMatch(log.attributes["http.route"], "^/(healthz|readyz)$") and log.attributes["http.response.status_code"] == 200' This assumes http.route has been promoted onto the log record; it is a span attribute by default, so on the trace side the equivalent lives under trace_conditions, with a span. prefix instead of log.. That status code check is the difference between Q1 and Q2. Without it, you have removed probe observability rather than probe noise, and you will find that out the next time readiness starts flapping and nothing in the logs can tell you when it began. With it, volume goes down, and signal is untouched, or arguably goes up, because you are no longer scrolling past successful probe traffic to find a real request. Sounds like a good deal, right? This is the quadrant everybody is imagining when they say "most of it is noise anyway." The same trade is available on the retry storm that repeats one stack trace ten thousand times in an hour. The logdedup processor collapses each ten-second window into one record carrying the count, so the storm stops drowning the query you are running, and you can still see how big it was. Finding the rest of this kind of waste means clustering records by shape and looking at what dominates, which is a different class of tool than a filter, and it is the part most volume-reduction programs skip. The challenge is that this quadrant is finite. In my experience, it is somewhere in the range of 10-20%, depending on how neglected the pipeline has been. If your mandate was 30%, you exhaust Q1 in the first week, and then you keep going, because the mandate does not stop when the free lunch does. Q2: Paying With Data Instead of Money So the free lunch got you 15%, the middle of that range, and the mandate was 30%, so the next 15% has to come out of data that somebody might actually need. Which is a good moment to read the mandate again, because almost nobody means it literally. "We need to reduce our telemetry volume by thirty percent" is very rarely a statement about telemetry. It is a statement about an invoice. Does anybody in that meeting actually want fewer log lines? They want a smaller number at the bottom of a bill. Volume is simply the variable their contract happens to be calculated on. The distinction matters because volume reduction and reducing your bill have different solution spaces. Reducing volume by 30% has one family of answers, and every one of them involves deleting something. Reducing observability spend by 30%, has a different set of options, several, and deleting your data is the one with the worst terms. A logging config goes from INFO to WARN and ships with the next release. Retention drops from thirty days to seven. Traces get sampled at 5%: YAML processors: probabilistic_sampler: sampling_percentage: 5 None of these options is free. Each one of them is defensible in isolation, and what makes them defensible is that they have a big impact. INFO is most of your log volume, seven days covers most incidents, and 5% is a perfectly good sample if all you want is a latency distribution. You end up paying the bill twice, but only one of the payments shows up on the invoice. You are also settling the bill in a second currency: answers you will not have, because you didn’t store the data needed for them. Nobody counts that. Nothing fails and nothing alerts, because a trace that was never recorded does not raise anything. When a customer sends an order ID on Thursday, and the trace behind it was one of the ninety-five per cent, the investigation stalls; somebody says we do not have that, and nobody goes back to look at the config change from earlier in the year that caused it to be dropped. My position is that most of this work should not exist. The engineering is fine! The sampler is correct, the retention change is correct, and both do exactly what they say on the tin. It is just that the whole exercise is effort spent making a bad unit price easier to swallow. It's like an old fridge: defrost it, keep the door shut, put less in it, and yes, your bill really does go down every month. Somebody should still go and look at what a new fridge costs. Q3: The Enrichment Nobody Gets To There is a second way to improve signal-to-noise: instead of removing noise, you add signal. You make the data you are already paying for be more useful. Attaching Kubernetes and cloud metadata with the k8sattributes processor, so a log line knows which namespace, deployment, node, and pod produced it. Parsing an unstructured message body into named, queryable fields with OTTL. Making sure trace context actually propagates across the boundary where it currently drops, so your logs and traces can be correlated instead of merely coexisting. Carrying code.file.path and code.line.number on the records that warrant it, so a log line points at the statement that emitted it instead of leaving you to grep the repository for the format string. YAML processors: k8sattributes: extract: metadata: - k8s.namespace.name - k8s.deployment.name - k8s.pod.name - k8s.node.name transform/parse_access_log: log_statements: - context: log statements: - merge_maps(attributes, ExtractPatterns(body, "^(?P<method>\\w+) (?P<path>\\S+) (?P<status>\\d{3}) (?P<duration_ms>\\d+)$"), "insert") These changes make your telemetry substantially more valuable, but they also increase volume. But most of that is cheaper than you would guess. The Kubernetes metadata are resource attributes, written once per batch in OTLP and shared by every record from the same pod, so at the collector's egress they cost a fraction of a byte per record. The parsing is the real exception: you keep the original body alongside the extracted fields, so the record roughly doubles, and no amount of batching recovers that. A bytes-per-day chart shows you none of that. The enrichment that costs almost nothing and the one that doubles every record show up the same way: the budget line went up. So the work never really gets argued about. Nobody is blocking k8sattributes – it ships enabled in half the Helm charts you might install – and most teams already intend to do all of the above. They just do not do it now, because a volume program has a number in it, and programs with numbers in them end when the number is hit. Q1 gets you fifteen percent, Q2 grinds out the rest, somebody screenshots the graph for the quarterly review, and the work is closed. There is no step after "we reduced it by 30%," because reducing it by 30% was the entire brief. Whether your observability spend is value for money is unanswerable while the telemetry is unusable. You can't defend a bill for data nobody can query, and you can't really attack it either, so the argument settles on price – the only number anybody in the room actually has. Enriched telemetry gets used, and usage is evidence. Most of what produces it is unglamorous work: consistent structure, correlation IDs that survive a hop, log levels that mean the same thing across services. But a team that can name the investigations that resolved faster this quarter, and the correlation that did it, walks into the budget meeting with something to say. Q4: The Change You Were Sure About The framework logs the request. Then the middleware logs it, because the framework's version does not carry the tenant. Then the application logs it a third time with slightly different wording, because by that point nobody trusts the other two. Three records, one event, and no reliable way to say which is authoritative. Every one of those lines was added by somebody trying to improve matters, and each has a different team behind it. That is what Q4 actually is, and why I think of it as the backfire. It is not really the stuff that piles up while nobody is looking; that was the double-shipping back in Q1, where either copy is safe to delete because they are identical. These three records differ from one another, and none of them goes without a conversation. Logging whole request and response bodies for completeness is the same story: you add a great deal of data, and the four fields anybody queries end up inside a blob that nothing has parsed. The same thing happens with a processor from the previous section. Take the k8sattributes block from Q3, change nothing about it, and point it at a different pipeline: YAML service: pipelines: metrics: processors: [k8sattributes] On logs, that was enrichment. On metrics, as soon as the backend treats resource identity as series identity, it is a separate series for every pod – and a fresh set of them after every deploy, because pod names churn. That is how a well-meaning label addition takes out a Prometheus. The config did not change, and neither did the intention behind it. Underneath all three is an assumption that more data is the same thing as more signal, and that if the answer is not in there yet then adding should get you closer. It is the same mistake the volume mandate makes, pointed the other way, and I have watched one team make both inside about two years. The awkward thing is that Q3 and Q4 are not separable at the time, and not only on the chart. From the inside, they are the same act: somebody adds something to a pipeline because they are fairly confident it will help. The engineer putting a pod name on a metric is doing what the engineer putting it on a log did. One of them is right. Review will not catch it either, because the reviewer is working from the same information and the same instinct. You need something that checks whether a question actually got easier to answer. What to Govern Instead Put the four quadrants back together, and the problem shows up in one line. Q1 and Q2 both report as a reduction in volume, so dropping probe traffic and dropping the log lines that explain a failure show up in the quarterly review as the same green arrow. Q3 and Q4 both report as volume up, so the enrichment that made an incident tractable and the label that took out your metrics backend are reported as the same red arrow. A bytes-per-day number cannot separate any of that, but it is the number the entire program is steered by. None of which is an argument against governing telemetry. It grows without limit if nobody is watching, somebody has to own the bill, and a team that has never questioned its telemetry costs is not being principled, is just not looking. The argument is about which variable should be on the dashboard. The goal is to try and measure signal, and it is less work than it sounds. Take the ten questions your team actually asks during an incident. Can I segment this failure by tenant? Can I get from this alert to the trace that caused it? Can I tell which deployment introduced it? Write each one as a literal query, in a file, checked into the repository that holds your collector config, and run them in CI against a replay of real telemetry, once with the proposed change and once without. If any answer moves, the build fails. Not just if it comes back empty: sampling does not empty a result; it quietly changes it. That is the difference between Q3 and Q4 made mechanical. The engineer adding pod name to a metric finds out in the pull request instead of during the next incident. It works in reverse too, which is the part that matters for Q3: adding a question and watching it fail is how you justify an enrichment to somebody whose only other number is bytes per day. And if you would rather start with something off the shelf, the Instrumentation Score is an open specification for grading OTLP against semantic conventions and instrumentation best practice, which is a different cut at the same question. Either way: your observability pipeline is probably the only production system you own with no tests on it, and there is no particular reason for that. Changing the Constraints I want to end somewhere slightly uncomfortable, because I do not think this is really a discipline problem or an education problem. Which quadrants you can operate in is dictated by your observability platform's cost model, not by your engineers. If ingest cost scales linearly with bytes, and retention is tiered so that older data becomes slow or expensive or both, then the economics have already made your architectural decisions. Q3 is priced out of existence. Q2 becomes not just permitted but mandatory, because it is the only lever that moves the number anybody is measured on. Your telemetry strategy is a downstream consequence of a pricing page. Teams under that constraint are not making bad choices. They are making the only choices available, and then rationalizing them as noise reduction, because "we improved our signal-to-noise ratio" is a much better sentence than "we deleted data we may need." The interesting question is what changes when volume stops being the binding constraint. When enriching a log record does not require a budget conversation, the matrix opens up. You can attack Q4 aggressively and invest in Q3, which is the combination that actually improves the ratio. Until then, at minimum, name the quadrant. When somebody proposes a pipeline change, ask which of the four it is. It is a five-second question, and I have not yet seen it fail to change the conversation.

Snowflake has become a go-to platform for storing and querying operational data at scale. SQL is excellent at filtering rows, joining tables, and aggregating numbers. But there's a class of questions where SQL starts to struggle: questions about connections. Which machines in a production line depend on this one? If this component fails, what else goes down with it? Which assets play equivalent structural roles across parallel workflows? These are fundamentally questions about relationships, and answering them in SQL requires increasingly complex recursive queries as the number of hops grows. Graph analytics is a natural complement here. Rather than replacing SQL, it adds a new lens on data you already own. In this article, we'll see how to use the Neo4j Graph Analytics Native App, available from the Snowflake Marketplace, to run graph algorithms directly on Snowflake tables — no data movement, no separate infrastructure, no new data store to maintain. The full source code is available on GitHub. The Scenario We'll work with a manufacturing plant dataset: 20 machines (Cutters, Welders, Presses, Assemblers, and Painters) connected by directed material flow relationships. Each machine has a risk level (low, medium, or high), and each relationship carries a throughput rate. This is representative of the kind of operational data that already exists in Snowflake for real systems — asset registers, process flows, supply chain graphs. The questions we ask of it apply equally to those domains. Setup The Neo4j Graph Analytics app is installed from the Snowflake Marketplace — just search for "Neo4j." Once installed, we'll create a database, load the demo data, and configure the permissions the app needs to read from and write to our tables. The data lives in two tables: nodes (one row per machine) and rels (one row per material flow connection). Graph algorithms need a simplified view of these — just node IDs and source/target pairs — so we create two projection-ready tables: Python # Node view - just the IDs, which is what graph projections need session.sql(""" CREATE OR REPLACE TABLE ga_demo.public.nodes_vw AS SELECT machine_id AS nodeId FROM ga_demo.public.nodes """).collect() # Relationship view - aggregate to ensure one weight per pair session.sql(""" CREATE OR REPLACE TABLE ga_demo.public.rels_vw AS SELECT src_machine_id AS sourceNodeId, dst_machine_id AS targetNodeId, CAST(SUM(throughput_rate) AS FLOAT) AS total_amount FROM ga_demo.public.rels GROUP BY src_machine_id, dst_machine_id """).collect() Thinking in Graphs Before running algorithms, it's worth establishing a shared vocabulary. A graph is made of nodes (entities) and relationships (connections between them). Both can carry properties. In our plant, each machine is a node — its machine_type and risk_level are properties on that node. Each material flow connection is a relationship — its throughput_rate is a property on that relationship. The data are already in Snowflake. A graph is not a separate thing you import data into. It's a lens on data you already own. Every algorithm call in Neo4j Graph Analytics includes a project block that tells the app which Snowflake tables to use as nodes and which to use as relationships. The app reads those tables, builds a temporary in-memory graph structure, runs the algorithm, writes results back to a Snowflake table we specify, and then discards the in-memory structure. Our data never leaves Snowflake. We can visualize the plant graph before running any algorithms to get a sense of its structure. Figure 1. Manufacturing Plant Graph A few things are immediately visible: one node appears to receive connections from many others, and one node seems to sit between otherwise separate sections of the plant. The algorithms that follow will confirm these observations numerically. Connectivity Analysis: Weakly Connected Components Our first question is foundational: is this plant one integrated system, or does it split into isolated subsystems? Weakly Connected Components (WCC) treat the graph as undirected — it ignores the direction of material flow and asks simply: can every machine reach every other machine through some path? The output assigns each machine a component ID. Multiple component IDs would indicate isolated sub-plants. Python session.sql(""" CALL neo4j_graph_analytics.graph.wcc('CPU_X64_XS', { 'project': { 'defaultTablePrefix': 'ga_demo.public', 'nodeTables': ['nodes_vw'], 'relationshipTables': { 'rels_vw': { 'sourceTable': 'nodes_vw', 'targetTable': 'nodes_vw' } } }, 'compute': {}, 'write': [{ 'nodeLabel': 'nodes_vw', 'outputTable': 'ga_demo.public.nodes_wcc' }] }) """).collect() The results show a single component containing all 20 machines — the plant operates as one integrated network. This is a useful baseline: it tells us there are no isolated subsystems that might be invisible to centralized monitoring. Criticality Analysis: PageRank and Betweenness Centrality Knowing the plant is connected, we can ask: which machines are most critical? We use two algorithms that measure criticality in different ways. A machine can be critical for one reason but not the other, and the distinction has real operational implications. PageRank: Flow Importance PageRank asks which machines receive material from many well-connected upstream machines. A high PageRank score means a machine is a destination for flow from important sources. If it slows down, the backlog ripples upstream. Python session.sql(""" CALL neo4j_graph_analytics.graph.page_rank('CPU_X64_XS', { 'project': { 'defaultTablePrefix': 'ga_demo.public', 'nodeTables': ['nodes_vw'], 'relationshipTables': { 'rels_vw': { 'sourceTable': 'nodes_vw', 'targetTable': 'nodes_vw' } } }, 'compute': { 'mutateProperty': 'score' }, 'write': [{ 'nodeLabel': 'nodes_vw', 'outputTable': 'ga_demo.public.nodes_pagerank', 'nodeProperty': 'score' }] }) """).collect() Machine 20 comes out on top — it sits at the confluence of multiple upstream chains, the assembly hub where material from across the plant converges. Figure 2. PageRank Visualization Betweenness Centrality: Structural Importance Betweenness asks a different question: which machines appear most often on the shortest path between other machines? A high Betweenness score means a machine is a structural bridge. It may not handle the most flow, but its position connects otherwise separate parts of the plant. If it goes offline, it disconnects or lengthens paths across the network. Python session.sql(""" CALL neo4j_graph_analytics.graph.betweenness('CPU_X64_XS', { 'project': { 'defaultTablePrefix': 'ga_demo.public', 'nodeTables': ['nodes_vw'], 'relationshipTables': { 'rels_vw': { 'sourceTable': 'nodes_vw', 'targetTable': 'nodes_vw' } } }, 'compute': { 'mutateProperty': 'score' }, 'write': [{ 'nodeLabel': 'nodes_vw', 'outputTable': 'ga_demo.public.nodes_betweenness', 'nodeProperty': 'score' }] }) """).collect() Machine 3 has the highest Betweenness score — despite having a much lower PageRank than Machine 20. It's not the busiest machine; it's the one whose failure would do the most structural damage. Figure 3. Betweenness Centrality Heatmap This is the key insight from running both algorithms: PageRank and Betweenness reveal different kinds of importance. A maintenance plan that uses only one of them is missing half the picture. Structural Similarity: FastRP and KNN So far we've identified individual critical machines. This section asks a different question: which machines play the same structural role in the workflow, even if they're different types? Machines with structurally equivalent positions can share maintenance windows, act as backups for each other, or be treated as a unit for risk modeling — even if they look different on paper. We use two algorithms in sequence. Fast Random Projection (FastRP) FastRP generates a compact embedding vector for each machine by sampling the graph structure around it. Two machines with similar upstream and downstream neighbors will end up with similar embedding vectors, regardless of their type or risk level. We use 16 dimensions — a good balance for a 20-node graph. Python session.sql(""" CALL neo4j_graph_analytics.graph.fast_rp('CPU_X64_XS', { 'project': { 'defaultTablePrefix': 'ga_demo.public', 'nodeTables': ['nodes_vw'], 'relationshipTables': { 'rels_vw': { 'sourceTable': 'nodes_vw', 'targetTable': 'nodes_vw' } } }, 'compute': { 'mutateProperty': 'embedding', 'embeddingDimension': 16 }, 'write': [{ 'nodeLabel': 'nodes_vw', 'outputTable': 'ga_demo.public.nodes_fastrp', 'nodeProperty': 'embedding' }] }) """).collect() K-Nearest Neighbor (KNN) KNN takes the embeddings and finds, for each machine, its most structurally similar peer. Similarity is measured using cosine similarity of the embedding vectors — a score of 1.0 means identical structural position, 0.0 means completely different. Note that KNN operates on node properties rather than graph edges, so its projection block contains no relationship table — the one exception to the pattern seen in the other algorithm calls. Figure 4. KNN Structural Similarity Matrix The results show high-similarity pairs between machines of different types. This is expected: FastRP captures structural position in the graph, not machine attributes. Two machines with similar upstream and downstream neighbors will have similar embeddings regardless of their type, risk level, or throughput rate. Failure Simulation Static risk analysis tells us which machines are currently important. We can turn that into a dynamic tool by asking: what actually happens to the rest of the plant when Machine 3 goes offline? We simulate the failure by creating filtered views that exclude Machine 3 and all its connections, then re-run PageRank and Betweenness on the degraded graph. Normalization matters here: raw scores shrink after failure because the graph is smaller. We divide each score by the sum of all scores in that run so we're comparing relative importance within each graph, not absolute values. Python session.sql(f""" CREATE OR REPLACE VIEW ga_demo.public.nodes_failure_vw AS SELECT machine_id AS nodeId FROM ga_demo.public.nodes WHERE machine_id != {EXCLUDED} """).collect() session.sql(f""" CREATE OR REPLACE VIEW ga_demo.public.rels_failure_vw AS SELECT src_machine_id AS sourceNodeId, dst_machine_id AS targetNodeId, CAST(SUM(throughput_rate) AS FLOAT) AS total_amount FROM ga_demo.public.rels WHERE src_machine_id != {EXCLUDED} AND dst_machine_id != {EXCLUDED} GROUP BY src_machine_id, dst_machine_id """).collect() Figure 5. Betweenness Delta Bar Chart The key finding: machines that were not flagged as high risk in the baseline analysis gain significant Betweenness importance after Machine 3's failure. The network reroutes through alternative paths, promoting machines that were structurally insignificant in the baseline into critical bridge positions. Static risk labels don't capture this — graph analysis does. The notebook is designed to support experimentation: change the EXCLUDED variable to any machine ID and re-run the section to see how the network responds to a different failure. Community Detection: Louvain The previous sections analyzed individual machines. Louvain community detection asks: does the plant naturally organize itself into clusters? Louvain finds groups of machines that are more densely connected to each other than to the rest of the network. These communities often correspond to real operational sub-units — parallel production lines, shared workflow stages, or tightly coupled machine groups. Python session.sql(""" CALL neo4j_graph_analytics.graph.louvain('CPU_X64_XS', { 'project': { 'defaultTablePrefix': 'ga_demo.public', 'nodeTables': ['nodes_vw'], 'relationshipTables': { 'rels_vw': { 'sourceTable': 'nodes_vw', 'targetTable': 'nodes_vw' } } }, 'compute': { 'mutateProperty': 'community' }, 'write': [{ 'nodeLabel': 'nodes_vw', 'outputTable': 'ga_demo.public.nodes_louvain', 'nodeProperty': 'community' }] }) """).collect() Figure 6. Louvain Community Detection Joining the community results back to the risk levels reveals that the smaller community has a disproportionate concentration of high-risk machines relative to its size. This also explains the failure simulation results: Machine 3 sits in this community and acts as its main bridge to the rest of the plant. Community detection connects the structural analysis back to operational risk in a way that neither algorithm produces on its own. Bringing It All Together The final step joins all four algorithm outputs into a single risk summary table: Python risk_summary = session.sql(""" SELECT n.machine_id, n.machine_type, n.risk_level, ROUND(p.score, 4) AS pagerank_score, ROUND(b.score, 4) AS betweenness_score, l.community FROM ga_demo.public.nodes n JOIN ga_demo.public.nodes_pagerank p ON n.machine_id = p.nodeid JOIN ga_demo.public.nodes_betweenness b ON n.machine_id = b.nodeid JOIN ga_demo.public.nodes_louvain l ON n.machine_id = l.nodeid ORDER BY pagerank_score DESC """).to_pandas() This table combines flow importance, structural importance, and community membership into a single view — one that would be difficult to produce from SQL alone and impossible without running the underlying graph algorithms. Summary SQL and graph analytics aren't competing approaches — they're complementary ones. Snowflake handles what it does well: storing, filtering, and aggregating operational data at scale. Neo4j Graph Analytics, running as a Native App inside Snowflake, adds a layer of analysis that SQL alone can't easily provide: understanding how entities relate to each other, which ones are structurally critical, and how the network behaves under failure conditions. The full source code is available on GitHub.

By Akmal Chaudhri CORE

Modern database environments rarely run a single type of workload. Most production systems handle both transactional operations and analytical queries simultaneously. These mixed workloads, often referred to as hybrid workloads, place significant pressure on traditional database indexing and storage strategies. In such environments, disk-based indexes can become a performance bottleneck. When transactional and analytical queries compete for disk I/O, it often results in increased latency, reduced throughput, and inconsistent query performance. To address these challenges, SQL Server leverages memory-optimized tables and indexes as part of its In-Memory OLTP capabilities. These features reduce reliance on disk I/O by enabling data and index access directly from memory, while still maintaining durability through logging and checkpoint mechanisms. This article explores how memory-optimized indexing works and demonstrates how it can significantly improve performance in real-world hybrid workload scenarios. Core Characteristics Mandatory inclusion: Every memory-optimized table must have at least one index, as they serve as the "entry points" for row access.Purely in-memory: Indexes are rebuilt entirely from scratch during database recovery based on their definitions and the data loaded into memory.Non-persistent: Unlike traditional indexes, changes to these indexes are not written to the transaction log, reducing I/O overhead.Fragmentation-free: These structures do not suffer from traditional page fragmentation, eliminating the need for regular REORGANIZE or REBUILD operations. Index TypeBest Use CaseBehaviorHash IndexEquality SearchesUses an array of buckets; highly efficient for point lookups (e.g., WHERE ID = 5).Nonclustered IndexRange QueriesUses a lock-free B-tree structure (Bw-tree); ideal for range scans and sorted results (e.g., WHERE Price > 100). The Challenge With Traditional Indexing Traditionally, database indexes are stored on disk to ensure durability. While this design protects data, it introduces a major limitation: disk I/O latency. In environments with heavy workloads, disk access becomes a bottleneck. This is particularly noticeable when: Large analytical queries scan index rangesTransactional queries require fast point lookupsMany concurrent users access the system When both workloads run together, index operations often compete for disk resources, resulting in slower queries and higher latency. Introducing Memory-First Indexes Memory-First Indexes in SQL Server 2025 take a different approach. Instead of relying primarily on disk-based indexes, the system prioritizes in-memory index access for frequently used data while maintaining a synchronized copy on disk for durability. The key idea is simple: Hot data (frequently accessed index ranges) is kept in memory.Cold data remains on disk.Changes made in memory are synchronized with disk replicas in the background. This approach allows SQL Server to serve many queries directly from memory while still maintaining persistence. The feature also includes monitoring mechanisms that track query patterns. When the system detects frequently accessed index partitions, it moves them into memory automatically. Less frequently accessed portions are pushed back to disk to conserve memory resources. The result is faster query execution without requiring manual tuning from database administrators. Real-World Example: Retail E-Commerce Database To understand the benefits, consider a retail company running an e-commerce platform. The company stores millions of products in a table with the following structure: ProductID – unique identifierProductCategory – category of the productPrice – product priceStockQuantity – available inventory The application runs two types of queries. Transactional Query This query checks stock availability for a specific product. SQL SELECT StockQuantity FROM Products WHERE ProductID = 102345; Analytical Query This query calculates aggregated metrics by product category. SQL SELECT ProductCategory, AVG(Price) AS AvgPrice, SUM(StockQuantity) AS TotalStock FROM Products WHERE Price > 500 GROUP BY ProductCategory; In a traditional setup, both queries rely on disk-based indexes. When concurrency increases, disk access becomes saturated, and query performance suffers. With Memory-First Indexes, the most frequently used index ranges, such as ProductID and ProductCategory, are loaded into memory, allowing much faster lookups. Testing the Feature To evaluate the impact of Memory-First Indexes, we can simulate a large dataset and compare query performance before and after enabling the feature. Step 1: Create the Table SQL CREATE TABLE Products ( ProductID INT PRIMARY KEY, ProductCategory NVARCHAR(50), Price DECIMAL(10,2), StockQuantity INT ); Step 2: Populate Test Data The following script generates a large dataset for testing. SQL INSERT INTO Products (ProductID, ProductCategory, Price, StockQuantity) SELECT TOP 50000000 ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS ProductID, CASE WHEN ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) % 5 = 1 THEN 'Electronics' WHEN ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) % 5 = 2 THEN 'Clothing' ELSE 'Home Appliances' END AS ProductCategory, ABS(CHECKSUM(NEWID()) % 1000) + 1.00 AS Price, ABS(CHECKSUM(NEWID()) % 5000) + 1 AS StockQuantity FROM sys.all_objects a CROSS JOIN sys.all_objects b; Step 3: Create Traditional Indexes SQL CREATE INDEX IX_Products_ProductID ON Products (ProductID); CREATE INDEX IX_Products_Category ON Products (ProductCategory); At this stage, run the transactional and analytical queries and capture baseline metrics using Query Store or dynamic management views. Step 4: Enable Memory-First Indexes Next, recreate the indexes with Memory-First enabled. SQL DROP INDEX IX_Products_ProductID ON Products; CREATE INDEX IX_Products_ProductID ON Products (ProductID) WITH (MEMORY_FIRST = ON); DROP INDEX IX_Products_Category ON Products; CREATE INDEX IX_Products_Category ON Products (ProductCategory) WITH (MEMORY_FIRST = ON); Step 5: Execute Test Queries SQL SELECT StockQuantity FROM Products WHERE ProductID = 102345; MS SQL SELECT ProductCategory, AVG(Price) AS AvgPrice, SUM(StockQuantity) AS TotalStock FROM Products WHERE Price > 500 GROUP BY ProductCategory; Record execution time, CPU usage, and disk activity again. Observed Performance Improvements The results typically show noticeable performance gains. For example: Transactional queries Before: ~50 msAfter: ~15 ms Analytical queries Execution time reduced by about 50% System metrics also reveal additional improvements: Disk I/O reduced by more than 70%Memory usage increased only moderatelyCPU utilization became more stable during peak workloads These improvements occur because queries are able to retrieve indexed data directly from memory rather than waiting for disk operations. Why This Matters for Modern Workloads Hybrid workloads are becoming the norm across many industries, including retail, finance, and IoT platforms. Systems must support both real-time transactions and large analytical queries without sacrificing performance. Memory-First Indexes help address this challenge by: Reducing disk I/O bottlenecksImproving response time for critical queriesAutomatically adapting to changing workload patternsMaintaining durability with synchronized disk replicas Final Thoughts Memory-First Indexes represent an important improvement in SQL Server 2025’s indexing architecture. By prioritizing in-memory access for frequently used data, SQL Server can deliver significantly faster query performance while still preserving data durability. For organizations running mixed transactional and analytical workloads, this feature can reduce latency, improve system stability, and make better use of available hardware resources. As hybrid workloads continue to grow, features like Memory-First Indexing will play a key role in helping database platforms keep up with modern application demands.

By arvind toorpu CORE

Hadera, Israel, September 8th, 2026, TechnologyWire This article was provided by TechnologyWire and does not represent the editorial content of DZone. RavenDB, a NoSQL document database used by more than 12,000 customers, announced today the launch of its new product, Quill, a context layer for SQL databases that makes them ready for production AI agents without migrating the system of record or architecting a custom AI stack. With AI becoming a board-level mandate, CTOs and VPs of engineering are under pressure to ship AI capabilities fast. But for organizations whose mission-critical data sits in legacy SQL systems, built years before embeddings or agents existed, AI can’t access their data. Modernizing or replacing the systems is expensive, risky, and time-consuming. By the time the system is updated, nobody remembers what the project was supposed to achieve or how it measured ROI. Recently, a Gartner survey of infrastructure and operations leaders found that one in five AI initiatives fail, and only 28% report a positive ROI, which is linked to how well the technology is integrated, governed, and aligned with operational needs, not to the sophistication of the model. As AI becomes the industry standard, organizations have been left without a clear path to deliver until now. "Anyone can stand up an AI demo in an afternoon, but getting that demo into production with data pipelines, semantic search, security, governance, all the plumbing a small proof of concept doesn't need until it has to run at scale, is the hard part," said Oren Eini, founder and CEO of RavenDB. "Quill exists because we'd rather hand teams that plumbing already assembled than watch them rebuild the same project after project. You get access to the live data you need, decide the scope on day one, and change it as you go, instead of building everything from scratch." Quill connects directly to an organization's existing SQL database and adds a context layer on top, making it possible to launch production-ready agents in weeks rather than the 18 to 24 months of a typical in-house build. The source system stays exactly where it is and remains authoritative, and the full AI stack- search, retrieval, and agents that can answer questions- is included. Agents built on Quill support web chat, WhatsApp, Telegram, Slack, and Discord out of the box. "With Quill, the plumbing was already there, so we spent our time building the actual feature," said Hagay Albo, CEO at Albos Technologies and Holdings, an early adopter of Quill. By default, Quill is governed, sitting between the AI and the source system, and it is built on the assumption that the model itself cannot be trusted with unrestricted access, so organizations decide exactly what an agent can and cannot see, independent of the source database's own permissions. In a healthcare setting, for example, an agent can answer a patient's question about an upcoming appointment, while prescription data is never part of the dataset it can query. What is usually a custom security project becomes a configuration choice. Quill is also model-agnostic, so teams can use any AI model, switch providers, or run entirely on their own hardware. Quill is now available for organizations running PostgreSQL, SQL Server, or MySQL, with more databases to be supported in the future, and can be deployed in the cloud or on-premises to meet data-residency or regulatory requirements. To start using Quill today, visit: https://ravendb.net/quill About RavenDB: RavenDB is a hybrid NoSQL document database built for modern application development. Used by more than 12,000 customers across 50 industries, RavenDB helps teams move faster with seamless data management across cloud, on-prem, and edge environments. With full-text search, automatic indexes, and an easy-to-use studio for monitoring and administration, RavenDB is the database developers love and enterprises trust. Learn more at www.ravendb.net

Apple is turning a former engineer’s old MacBook into the latest flashpoint in its increasingly bitter fight with OpenAI. Apple told a federal court Monday that evidence recovered from former engineer Chang Liu’s Apple-issued MacBook strengthens its claims that confidential company technology was accessed and used after he joined OpenAI. In a filing supporting its request for expedited discovery, Apple said the defendants produced the laptop only after weeks of delay. An initial forensic review allegedly found that Liu and others at OpenAI “were well aware” of his continued unauthorized access to Apple’s third-party cloud storage providers, according to Bloomberg. Apple also alleges that Liu downloaded a confidential circuit schematic and later used it in his work at OpenAI. The company further claims he used a tool at OpenAI with the same name as an internal Apple engineering application. More seriously, Apple alleges that Liu instructed OpenAI colleague Yu-Ting Peng to destroy evidence after learning he was under investigation and that Peng agreed to comply. Apple’s lawyers wrote that the laptop “shows Apple is not conducting ‘fishing expeditions’ but that its trade secrets are being used and evidence is being destroyed,” according to Bloomberg. The newly cited material is largely sealed or redacted, so the public record does not yet provide a complete picture of what Apple found. OpenAI blames Apple’s security practices OpenAI disputes Apple’s interpretation of Liu’s actions. The company says Liu accessed Apple files to help former colleagues with Apple-related work and argues that the documents he accessed were irrelevant to his responsibilities at OpenAI. OpenAI has also criticized Apple’s handling of departing employees, arguing that poor access controls allowed Liu to retain access after leaving the company. It has published excerpts of Liu’s messages as part of its defense. OpenAI has called the dispute “a mess of Apple’s own making” and warned that a broad injunction could “chill employee mobility,” according to Bloomberg. The competing accounts leave several questions for the court, including whether Liu accessed protected material without authorization, whether it qualified as a trade secret and whether OpenAI acquired or benefited from it. The bigger fight is about hardware The stakes extend beyond one engineer. Apple alleges that more than 400 former Apple employees now work at OpenAI. That employee movement has become more sensitive as OpenAI expands into consumer hardware, including products being developed with former Apple design chief Jony Ive. For Apple, proving that confidential engineering material crossed into OpenAI’s hardware work could strengthen its case for court restrictions. For OpenAI, a broad order could complicate hiring and product development while the company builds its hardware business. For IT and security leaders, the dispute also highlights the importance of immediately revoking departing employees’ access, recovering company devices and preserving records when staff move to competitors. What happens next Apple is seeking damages, expedited discovery, and a preliminary injunction to prevent OpenAI from using the disputed technology while the lawsuit proceeds. It also wants proprietary material allegedly obtained by OpenAI destroyed. A federal judge is scheduled to hear arguments on Apple’s request for a preliminary injunction on Oct. 1. The hearing could determine whether OpenAI faces immediate restrictions while the lawsuit proceeds, although the court has not ruled that any trade-secret misappropriation or evidence destruction occurred. Editor’s note: This article originally appeared on our sister publication, TechRepublic.