Cloud Architecture Resources

DZone's Featured Cloud Architecture Resources

Calling GCP From AWS Without Static Keys Using Open-Source MultiCloudJ

By Sandeep Pal

In Part 1, we solved one direction of the multi-cloud connectivity problem: a workload running in Google Cloud interacting with an AWS cloud resource. A GKE pod read a Google-issued OIDC token from the metadata server, handed it to AWS STS via AssumeRoleWithWebIdentity, and received short-lived AWS credentials, with no static access keys stored anywhere. MultiCloudJ wrapped the token dance behind a portable client so the application code never touched a provider SDK directly. This article covers the return trip: a workload running in AWS calling into Google Cloud — specifically, an Amazon EKS pod reading and writing a Google Cloud Storage (GCS) bucket — again with zero long-lived credentials. The zero-trust principle is identical. The mechanism is a bit different. And that asymmetry is the single most important thing to understand before you build it. Authentication Flow from AWS to GCP 1. Build a SigV4-signed GetCallerIdentity request (signed with STS credentials): It's assumed that the EKS pod already holds temporary AWS credentials. 2. Call sts.googleapis.com for token exchange: The pod sends that signed request to Google Cloud as the input to an OAuth 2.0 token exchange. It is asking Google, "Here is proof of who I am on AWS - please give me a Google token to access cloud resources." 3. Replay the GetCallerIdentity signed request: Google does not trust the request blindly. It runs the signed request against AWS STS on the caller's behalf. 4. Response with ARN: AWS checks the signature and replies with the caller's ARN (the AWS role identity) as part of the GetCallerIdentity response. Now Google knows exactly which AWS identity is asking - proven by the signature, with no shared secret. 5. Validate the ARN with the pool: Google checks that ARN against the Workload Identity Pool rules - which AWS account and which role are allowed in, and how the ARN maps to a Google identity. 6. Access token: Once the ARN passes, Google returns a short-lived access token to the EKS pod. 7. Access the resource with the access token: The pod uses that token to read and write Cloud Storage. When the token expires (usually within an hour), the flow repeats. Nothing long-lived is ever stored. Summary: AWS proves the pod's identity by answering Google's replayed request, and Google issues a short-lived token based on that proof. No access keys, no service-account key files - just a signed request and a temporary token crossing the trust boundary. Please note that this authentication flow can be used for any cloud service and is not specifically for cloud storage. Direct Pool Access vs. Service Account Impersonation Once Google has verified the caller's AWS identity through the signed request, it still has to map that AWS identity to something that actually holds permissions on the bucket. There are two ways to do this mapping, and you should pick one before you grant any IAM role. Option 1: Direct Pool Access You grant the Cloud Storage role straight to the federated identity. In IAM, the member looks like this: principalSet://iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/aws-pool/* The permission sits on this pool principal, not on the AWS role. The AWS role never holds any GCP permission. Its only job is to prove identity: it answers Google's replayed GetCallerIdentity request so Google knows which AWS identity is asking. Google then checks that identity against the pool rules and, if it is allowed in, treats the caller as this pool principal. The bucket role, such as roles/storage.objectAdmin, is bound to that principal, so that is where the actual access comes from. No service account sits in the middle. In your code, the value you pass is the pool provider resource name (the audience), and Google issues a token that represents the pool identity directly. Option 2: Service Account Impersonation You create a GCP service account, grant that service account the bucket role, and then let the federated identity impersonate it. The federated identity needs roles/iam.serviceAccountTokenCreator on that service account, and the exchange gets a second hop: first a pool token, then an impersonated service-account token. In your code, the value you pass is the service account email. Which to Choose For a straight AWS EKS to GCS case like this one, direct pool access is the better default: Fewer moving parts. No service account to create, no token-creator grant to manage, and no second token hop.Tighter blast radius. The bucket permission is tied to identities coming through this specific pool, not to a service account that other workloads might also be able to impersonate. You can narrow it further to a single AWS role with an attribute condition on the principal.Less to audit. One IAM binding on the bucket tells the whole story. Reach for impersonation only when you actually need what a service account gives you: You must reuse an existing service account that already carries permissions across many GCP resources.A downstream Google API or tool only understands service-account identities and cannot evaluate a principalSet:// member.Your organization standardizes on service accounts as the single unit of access, to stay consistent with other human and machine grants. In short, direct pool access is simpler and safer, so use it unless a concrete requirement forces impersonation. Set Up Workload Identity Pool on GCP Before any code runs, you configure the trust relationship on Google Cloud once. Three things: a pool, an AWS provider inside it, and an IAM grant on the bucket. Create the Workload Identity Pool: The pool is the identity container that your AWS workloads will be represented as.gcloud iam workload-identity-pools create aws-pool --location="global" --display-name="AWS workloads"Create the AWS provider inside the pool: The provider is the entry gate. It tells Google to trust GetCallerIdentity results from a specific AWS account, how to map the caller's ARN into a Google attribute, and which callers are allowed in.Two important parts here: The attribute mapping turns the caller's raw ARN into a stable attribute.aws_role value with the session name stripped, so grants survive session rotation.The attribute condition is the first gate: only callers from your AWS account are admitted, before any IAM binding is even checked. Shell gcloud iam workload-identity-pools providers create-aws aws-provider \ --location="global" \ --workload-identity-pool="aws-pool" \ --account-id="123456789012" \ --attribute-mapping="google.subject=assertion.arn,attribute.aws_role=assertion.arn.contains('assumed-role') ? assertion.arn.extract('{account_arn}assumed-role/') + 'assumed-role/' + assertion.arn.extract('assumed-role/{role_name}/') : assertion.arn,attribute.account=assertion.account" \ --attribute-condition="assertion.account == '123456789012'" Grant the bucket role to the pool principal: This is the direct pool access model. The permission binds to the AWS role (via the mapped attribute), not to a service account. Shell gcloud storage buckets add-iam-policy-binding gs://my-archive-bucket \ --role="roles/storage.objectAdmin" \ --member="principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/aws-pool/attribute.aws_role/arn:aws:sts::123456789012:assumed-role/my-eks-role" After this, the EKS pod's role can federate into the pool and read/write the bucket, and the application code in the next section never touches any of this setup again. Implementation With MultiCloudJ MultiCloudJ exposes the same BucketClient abstraction you saw in Part 1; you build it for the "gcp" provider and attach a CredentialsOverrider that carries the federated identity. The library handles the SigV4 signing, the STS token exchange, and (on the impersonation path) the generateAccessToken call internally; your code just does blob operations (full example). Java private static final String REGION = "us-west-2"; // The audience is the full Workload Identity Pool provider resource name. // We grant the bucket role directly to this pool principal (direct pool // access), so no service account sits in the middle. private static final String AUDIENCE = "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/aws-pool/providers/aws-provider"; // The supplier runs on every GCP token refresh. Each time it signs a fresh // GetCallerIdentity request with the pod's AWS role (IRSA, picked up from the // ambient AWS credential chain) and returns the subject token GCP expects. Supplier<String> webIdentityTokenSupplier = GcsFromAws::buildSubjectToken; CredentialsOverrider overrider = new CredentialsOverrider.Builder(CredentialsType.ASSUME_ROLE_WEB_IDENTITY) .withRole(AUDIENCE) .withWebIdentityTokenSupplier(webIdentityTokenSupplier) .build(); // Portable client: same API as the AWS side in Part 1, // only the provider string changes. BucketClient bucketClient = BucketClient.builder("gcp") .withBucket("my-archive-bucket") .withCredentialsOverrider(overrider) .build(); ListBlobsPageResponse page = bucketClient.listPage(ListBlobsPageRequest.builder().withMaxResults(10).build()); page.getBlobs().forEach(b -> System.out.println(b.getName())); // Signs a GetCallerIdentity request with the pod's AWS role, then shapes the // signed request into the URL-encoded JSON envelope that Google STS expects // as an AWS4 subject token. private static String buildSubjectToken() { // Google requires the audience to travel inside the signed headers, so it is // bound to the signature and the request cannot be replayed against any other // target. SignOptions options = SignOptions.builder() .withCustomHeader("x-goog-cloud-target-resource", AUDIENCE) .build(); StsUtilities stsUtil = StsUtilities.builder("aws").withRegion(REGION).build(); // Passing null means "just sign a GetCallerIdentity request, there is no // service payload to hash." The library fills in Action=GetCallerIdentity. SignedAuthRequest signed = stsUtil.newCloudNativeAuthSignedRequest(null, options); JsonObject envelope = .. // construct json object from signed request uri return URLEncoder.encode(envelope.toString(), StandardCharsets.UTF_8); } Conclusion Part 1 showed GCP calling AWS, and Part 2 completes the picture with AWS calling GCP. Both use the same idea: federation, no static keys, and only short-lived credentials. They differ only in how identity is proven. GCP to AWS presents a Google OAuth identity token, while AWS to GCP sends a signed request that GCP verifies with AWS. This is exactly where MultiCloudJ earns its place. All of these provider-specific differences, such as the bearer token here, the signed request and replay there, the STS token exchange, the service-account impersonation, and the token refresh, are abstracted away inside the library. You build one portable client, attach a credentials overrider, and call the API. Your application code never learns which cloud it is talking to or which way the call is going, so it stays clean, portable, and free of long-lived secrets in both directions. More

Building an Async Validation API With AWS Bedrock Agents and Serverless Architecture

By Rohit Nagpal

As a data engineer, I’ve noticed business teams submitting intake forms, compliance documents, and project proposals that a tech team then manually validates against a set of predefined business rules stored in a database that gets updated quarterly. The time it takes to validate a single form is typically in the hours, and by the time you’ve validated the form, the submitter has moved on to other work. When I needed to validate project intake forms against 60+ business rules of financial, compliance, and other types of business rules and guidelines (some of them to be used in a deterministic way and others to be used in a more nuanced manner), I knew that a simple if-else logic-based manual review process would not scale. This article walks through how I developed an async, AI-powered validation API with AWS Bedrock Agents and Serverless Architecture to process and validate intake forms within 60 seconds without blocking the user. The architecture also manages cross-account authentication to get access to the AI-powered engine and shows failure recovery gracefully. Why Async? The Problem With Synchronous AI APIs Integrating AI into an API synchronously means users send a request, the server processes it, and returns results in one HTTP response, but many systems that use AI-powered validation take more than 30 seconds. The AI agent I built was taking anywhere from 30 seconds to 1 minute to evaluate all of the form fields for all the applicable rules and conditions. But the hard limit for the API Gateway is 29 seconds (HTTP timeout). One approach to make this API request work is to transform the synchronous request and response into an async request with a subsequent background processing step and poll the results from a separate endpoint. This can be implemented as follows: Client submits the form via POST, receives a request_id immediately (under 2 seconds)Validation runs asynchronously in the background (30–60 seconds)Client polls a GET endpoint with the request_id until results are ready By making the form submission step separate from the AI validation of that form in the background, users can continue working on other tasks instead of being stuck staring at a page waiting 30 to 60 seconds for the form to be validated. Architecture Overview As a data engineer, I was required to tackle three main challenges to create a production AI validation API: 1) the frontend application is deployed in a different AWS account, 2) AI agent-based form validation is extremely computationally expensive to run, and 3) business rules for this type of validation are likely to change from time to time without API code deployment. The architecture consists of five components: API Gateway (REST API): With Cognito Authorizer for cross-account JWT authenticationAsync Handler Lambda: It’s an entry point for the API. An Async Handler Lambda function is invoked by a POST request. It will store the form payload on S3, then trigger the Validation Lambda function and store an initial "processing" status in S3. The function immediately returns a request_id to the frontend client within 2 seconds.Validation Lambda: This function loads up all the rules for a given request from S3. It then builds up all the prompts for the Bedrock Agent and runs the Agent. The results of the Agent are then saved off in S3 for the Polling API.Polling Lambda: Handles GET requests and checks S3 for completed resultsRules Sync Lambda: Separate independent process to read validation rules from the data warehouse using EventBridge scheduler and sync to S3 for validation with AI model. Implementation: The Async Handler The async handler is the entry point. Its task is quite straightforward. It accepts the payload, stores it, triggers the Validation Lambda function, stores an initial "processing" status in S3, and returns the “processing” status with the request ID to the client. The function does all of this within a couple of seconds. Here is the core implementation: Python import json, boto3, uuid from datetime import datetime s3 = boto3. client(' s3') Lambda_client = boto3. client('Lambda') S3_BUCKET = 'my-validation-bucket' VALIDATION_LAMBDA = 'ai-validation-function' def lambda_handler(event, context): payload = json. loads (event. get ('body', "{}')) request_id = str(uuid.uuid4()) # Store initial processing status s3.put_object( Bucket=S3_BUCKET, Key=f'validation-output/(request_id)/status.json', Body=json.dumps({ 'request_id': request_id, 'status': 'processing', 'submitted_at': datetime. ttenew() .isoformat() }) ) # Fire-and-forget: invoke validation async pay Load ['_request_id'] = request_id lambda_client.invoke( FunctionName=VALIDATION_LAMBDA, InvocationType='Event', # Async invocation Payload=json. dumps (payload) ) return { 'statusCode': 202, 'body': json. dumps ({ 'request_id': request_id, 'status': 'processing' }) } In the above code snippet, I specifically invoke the validation lambda from the async handler by setting the InvocationType='Event'. This allows the async handler to return immediately to the frontend with the request_id for the submitted request. The Validation Lambda will then complete asynchronously and store the results in S3. Implementation: The Polling Handler The Polling Handler Lambda function manages the GET endpoint; it polls S3 for the updated status file and returns the current status of Validation Lambda processing: completed or failed. Here is the core implementation: Python def lambda_handler(event, context): request_id = event['pathParameters']['request_id'] try: status_obj = s3.get_object( Bucket=S3_BUCKET, Key=f'validation-output/{request_id}/status.json' ) status = json.loads(status_obj['Body'].read()) if status['status'] == 'processing': return {'statusCode': 200, 'body': json.dumps(status)} # Completed - return full results results_obj = s3.get_object( Bucket=S3_BUCKET, Key=f'validation-output/{request_id}/results.json' ) results = json.loads(results_obj['Body'].read()) return {'statusCode': 200, 'body': json.dumps(results)} except s3.exceptions.NoSuchKey: return {'statusCode': 404, 'body': 'Request not found'} S3 Decoupling: Using S3 as an intermediary between the validation Lambda and the polling handler allows for natural decoupling. The validation Lambda writes the results of the validation to S3, and the polling handler reads from S3 to return the latest status to the frontend. There is no shared state between the validation handler and the polling handler; there are no database connections, and there are no race conditions. Integrating the Bedrock Agent for Intelligent Validation An intelligent validation function would need more than just a set of rules to check for requirements and best practices. There are a lot of judgment calls that a human would make based on examples of how a policy or guideline would be applied in real life. To achieve that, the more effective way is to integrate with an existing AI function that is designed to handle a wide variety of scenarios and functions The Bedrock Agent architecture solved this by combining: Knowledge base: Containing policy documents, guidelines, and past examples of work for the intelligent validation to reference during the evaluation process.Dynamic prompts: The prompts for the AI model are built dynamically from the current validation rules. These are loaded from S3 as a JSON file and then injected with the current values for the specific field being evaluated.Structured output: Parse the assessment’s pass/fail status, confidence in the assessment, and a set of detailed recommendations made by the agent. The prompt for the AI agent is generated at runtime by the validation function. The rules are loaded from S3 earlier in the function's execution. Here is an example prompt: “Evaluate field [Project Justification] with value [user input] against rule: The justification must clearly describe the business problem being solved and include quantified impact. Reference the knowledge base for examples of approved justifications.” The AI returns a structured assessment of whether or not the field has passed validation, the confidence that the AI has in the assessment, and recommendations. Dynamic Rules Management: Keeping Rules in Sync Without Code Deploys Rules typically change on a monthly or quarterly basis by the business teams. To keep up with the current policy, the rules must be separate from the rest of the application code. To achieve that, I used Rules Sync Lambda, triggered daily by EventBridge: EventBridge fires at 6 AM daily.The Rules Sync Lambda queries the Data Warehouse (Redshift) for the current validation rules for the application.It also takes a copy of the most current version of the rules in S3 for purposes of rollback.It transforms and then uploads the new rules file to S3 as a new copy of the Validation_Rules.json file.Upon failure to update the rules in S3, a CloudWatch Alarm is triggered, which in turn triggers an SNS notification to the appropriate engineering team. The rules are managed as a database of rules (as opposed to being stored within the application code), which allows business analysts to easily update the rules on a quarterly basis without requiring any code changes or deployments. Cross-Account Authentication With Cognito In this case, the frontend application and the AI backend were set up in two different AWS accounts. When deployed within different accounts (as within an enterprise), cross-account authentication is required. Since the frontend application was already authenticated against a company’s SSO (Single Sign On) using Cognito, it was only a matter of how to reuse these tokens within another account without involving the Frontend team for changes. The solution was to create a Cognito Authorizer and attach it to a REST API created in the API Gateway. This API can then be set up to trust the User Pool from the frontend account. Below is a simplified representation of this configuration: API Gateway REST API with a Cognito Authorizer pointing to the frontend account’s Cognito User Pool ARN.CORS (Cross-Origin Resource Sharing) configuration for only that frontend domain.The frontend application is already authenticated with CognitoThe backend application accepts the tokens that the frontend application is using for authenticationThe frontend application simply sends the existing Cognito tokens that the frontend application already has created in the authentication process From the frontend team’s perspective, this was a simple implementation that required them to send the existing Cognito token with the request and to implement a polling loop for the GET endpoint. Results and Lessons Learned After deploying to production: Validation time: reduced from 2 -3 hours (manual) to less than a minute (automated)API response time for form submission: less than 2 seconds for GET API using an async pattern, meaning the frontend never has to wait for the backend60+ validation rules: per form, including both deterministic and AI-judgement rules Zero code deploys: for changes to the rules, which are stored in the database, sync daily Key lessons as a developer building this: Design for async from the start: Retrofitting a synchronous API to be async is very hard. If your AI inference takes more than 5 seconds, which is generally the case, then design your API to be async from day one.Use S3 as your state machine: S3 is the simplest, cheapest, and most reliable way to pass results between decoupled Lambdas. No databases, no queues, no DynamoDB for this pattern.Separate dynamic rules from code: Separate process for managing rules which are dynamic and change often to avoid deployment bottleneck Bedrock Agents are good for making judgment calls. If you have a deterministic check (is a field empty), then you can code that. But for a judgment call (does a justification make sense), then use an AI agent to make the call. Conclusion There is an entirely new way to approach the request lifecycle for APIs in this AI-powered validation API development. The asynchronous API with polling for validation is better than simply trying to work around the timeout limits of APIs. Bedrock Agents, along with S3 to manage the state of the workflow and EventBridge to synchronize rules on a daily basis from a database created by business users via a simple UI created by frontend team, while backend team does not need to write any code for new rules, all integrated together to form complex data validation system powered by AI-powered judgment calls while maintaining simple to deploy and scalable system. As a data engineer, there’s nothing quite like watching hours of manual work by a reviewer get compressed down into 60 seconds or less of automated work while maintaining the high level of evaluation that a business stakeholder expects. More

Containerizing and Testing a Python Backtesting System With Docker and GitHub Actions

By Gillian Lu

Retrieval Augmented Generation With Spring AI 2.0, Claude, and PGvector

By Murat Balkan

CORE

Deploying a Spring Boot Microservice on AWS Fargate: Lessons From the Outage That Forced Me to Get It Right

By Vishal Rameshchandra Shah

I Built a RAG Agent on Azure AI Foundry in an Afternoon. Here's What Nobody Tells You.

Six months ago, building a RAG pipeline meant a full week of plumbing: an embedding job here, a vector store there, a retriever glued on with duct tape, and an orchestration layer that broke every time you touched it. I've built enough of these the hard way — hand-rolled vector search, custom chunking scripts, the works — to know exactly how much pain that "week" usually hides. Last week, I rebuilt the same thing on Azure AI Foundry. It took an afternoon. Not because the underlying problem got easier — grounding an LLM in your own data is still genuinely hard — but because Microsoft finally killed most of the integration tax that used to eat the first sprint of every RAG project. Here's what actually happened, warts included. The Old Way Was a Trap If you've built RAG before, you know the pattern: you don't fail at RAG, you fail at the seams between the pieces. Your chunking strategy doesn't match your embedding model's context window. Your retriever returns great results in a notebook and garbage in production because nobody wired up hybrid search. Your "agent" is really just a for-loop that stuffs retrieved text into a prompt and hopes. Foundry's whole pitch is that it owns those seams instead of leaving them to you. I was skeptical. I'm less skeptical now. What I Actually Did Step one: spin up a Foundry project. Not a hub-based one — those are legacy at this point, and if a tutorial has you creating one, skip it. The newer Foundry project type is the one to use. Step two: deploy two models. A chat model and an embedding model. Click, click, done. Both show up with their own endpoints. This part genuinely takes five minutes, and it's the first sign you're not building infrastructure anymore — you're configuring it. Step three: point Foundry at my documents. Blob storage in, Azure AI Search out. Foundry handles the chunking and embedding generation itself. I turned on hybrid search (keyword plus vector) because pure vector search on enterprise docs tends to miss exact terms people actually search for — product names, error codes, that sort of thing. If your content has a lot of that, don't skip this. Step four — and this is the part that's different from every tutorial I read two years ago. I didn't write a retrieval pipeline. I registered the search index as a tool on the agent and let the agent decide when to call it. Here's the whole thing: Python from azure.ai.projects import AIProjectClient from azure.identity import DefaultAzureCredential project = AIProjectClient.from_connection_string( credential=DefaultAzureCredential(), conn_str=os.environ["AIPROJECT_CONNECTION_STRING"], ) agent = project.agents.create_agent( model="gpt-4o-mini", name="docs-assistant", instructions=( "Answer only using retrieved context. " "Cite the source document for every claim. " "If the answer isn't in the retrieved content, say so." ), tools=[{ "type": "azure_ai_search", "index_connection_id": search_connection_id, "index_name": "example-index", }], ) thread = project.agents.create_thread() project.agents.create_message(thread.id, role="user", content="What's our refund policy for enterprise plans?") run = project.agents.create_and_process_run(thread.id, agent.id) No manual embedding calls at query time. No hand-written "retrieve top-k, stuff into prompt" logic. The agent framework does that internally, and it does it well enough that I stopped fighting it after the first try. Step five: For anything beyond simple lookups, I turned on agentic retrieval in Azure AI Search. Classic RAG fires one query per user turn, which quietly falls apart the moment someone asks a compound question — "compare our Q3 and Q4 policy and tell me what changed for renewals" is two questions wearing a trench coat. Agentic retrieval breaks that into sub-queries, runs them in parallel, and merges the results before generation. If your users ask messy, multi-part questions — and they do — turn this on from day one. Retrofitting it later is more annoying than it should be. Step six: Tested in the playground, then deployed the same agent behind a REST endpoint. Nothing about the agent changed between prototype and production. That alone would've saved me a full day on past projects. Now, the Part Everyone Skips I'm not going to pretend this is magic, because it isn't, and the tutorials that pretend otherwise are setting people up to get burned in a security review. Access control is on you. Foundry doesn't look at your documents and infer that HR files shouldn't be visible to the sales team. You configure document-level security filters in Azure AI Search yourself, and if you skip this, you've built a very articulate way to leak sensitive data. API keys are a prototype crutch, not a production plan. Move to Microsoft Entra ID before anything customer-facing goes live. This migration is a real afternoon of work, not a checkbox — budget for it. Retrieved documents are untrusted input. Prompt injection through a poisoned PDF is a real attack surface in every RAG system, Foundry included. Your system instructions need to assume the retrieved content might be trying to manipulate the model, because eventually it will. The costs stack. Embedding generation, index storage, and the extra tokens from stuffing retrieved passages into every call — none of this is free, and it compounds faster than people expect once you're past a demo and into real traffic. Model it before you commit to a chunking strategy at scale, not after. Was It Actually Worth It? Yes — but not for the reason most "look how easy this is" posts claim. The value isn't that RAG got simple. Grounding a model in the right data, with the right access controls, still takes real thought. The value is that Foundry took the boring week — the SDK wrangling, the manual retrieval loops, the glue code nobody wants to own — and turned it into an afternoon of configuration. That frees up the time you actually need for the parts that matter: is your data any good, is it chunked sensibly, and can you trust what comes back? If you've been putting off a RAG project because the infrastructure felt like too much, this is the moment to try again. Just don't skip the access control step to save time. That's the part that actually bites.

By Balaji Venkatasubramaniyar

Build Your Own Local AI QA Engineer With Docker, Ollama, LibreChat, and Playwright MCP

Artificial intelligence is rapidly transforming software testing by enabling QA engineers to generate test cases and test plans, automate browser interactions, analyze and debug failures, and execute complex testing workflows using simple natural-language prompts. While cloud-based AI assistants offer impressive capabilities, they often require subscriptions and sharing potentially sensitive application data with third-party services. Running an AI-powered testing assistant locally addresses these concerns by providing better privacy, lower operating costs, and complete control over the testing environment. In this tutorial, we’ll learn how to build our own local AI QA engineer using Docker, Ollama, Qwen3:8b, LibreChat, and Playwright MCP. It will allow us to perform browser automation and interact with web applications using natural language, all without relying on cloud-based AI services. Understanding the Architecture Every interaction begins with the user. For example, a user enters a prompt in LibreChat, such as “Open the Playwright website and click the ‘Get Started’ button.” LibreChat serves as the conversational interface through which users interact with the AI assistant. Rather than processing the request itself, it forwards the prompt to a locally hosted large language model, Qwen3:8b, running via Ollama. After receiving the prompt, Qwen3:8b interprets the user’s intent and generates a step-by-step execution plan. Instead of interacting with the browser directly, the model determines which tools are required and communicates those instructions using the Model Context Protocol (MCP). These MCP requests are handled by the Playwright MCP Server, which acts as the bridge between the language model and the browser. It translates the AI-generated instructions into executable Playwright commands. The Playwright MCP Server then launches a Chrome browser and performs the requested actions. Depending on the prompt, it can navigate to websites, click buttons, complete forms, extract text from web pages, capture screenshots, and execute a wide range of browser automation tasks. Once the browser completes the requested operations, the execution results are returned to Qwen3:8b. The language model analyzes the browser output and transforms the technical details into a clear, human-readable response. LibreChat then presents this response to the user. Instead of displaying raw Playwright logs, it provides a concise summary such as: “Navigation completed successfully. The Playwright website was opened, and the Get Started button was clicked successfully.” This architecture enables browser automation through natural language while ensuring that every component runs locally. As a result, we benefit from enhanced privacy, greater security, and complete control over the entire AI-powered automation workflow. Prerequisites Before getting started, ensure that the following software is installed on your machine: DockerNode.js 20 or higher versionGitOllama We’ll use Docker Desktop to run LibreChat, Node.js to install and run the Playwright MCP Server, Git to clone the required repositories, and Ollama to download and serve the local large language model. Having these tools installed beforehand will make the setup process smooth and straightforward. System Requirements Running a local AI-powered browser automation stack requires a reasonably capable machine. A system with 16 GB of RAM or more is recommended to run Docker containers and the language model efficiently. We’ll also need 20–25 GB of available disk space, preferably on an SSD, to accommodate Docker images and downloaded models. While a dedicated GPU can significantly improve model inference speed, it is entirely optional, and the setup works well on modern CPUs. For this tutorial, I’m using the following configuration: Operating system: macOS (M2 Pro)Memory: 16 GB RAM We can have the same setup on Windows and Linux, with only minor platform-specific differences in the installation steps. Setting Up the Environment for the Local AI QA Engineer Docker, Node.js, and Git are widely used development tools, and detailed installation guides for each are readily available online. Installing Ollama To install Ollama, either download the installer from the official website or use the installation command provided for your operating system. For macOS, it can also be installed using the following Homebrew command: Plain Text brew install ollama Once the installation is complete, it can be verified by running the following command in the terminal: Plain Text ollama --version Installing Qwen3:8b Qwen3:8b is chosen for this setup because it offers a strong balance of reasoning, code generation, and performance, making it ideal for Playwright TypeScript test generation, AI agents, MCP integration, and modern QA automation workflows while running efficiently on a local machine. However, other higher models can also be chosen if you know a better one. Another factor in choosing this model was the available system memory. Since my machine has 16 GB of RAM, some memory also needs to be reserved for other tools used in this setup, such as Docker, LibreChat, and Playwright. We need to start Ollama first by running the following command from the terminal. (It should be kept running in the background): Plain Text ollama serve Open a new terminal and run the following command to pull the Qwen3:8b model: Plain Text ollama pull qwen3:8b It should take some time to complete the pull, as the model is around 5.2GB. Once the download completes, we can check the model by running the command: Plain Text ollama list It should list the model downloaded. Next, we can quickly verify by running the model using the command: Plain Text ollama run qwen3:8b Once the model starts, it will prompt you to enter a query. To verify that everything is working correctly, try a simple prompt such as “What is 2 + 2?”. Observe how the model processes the request and generates its response. If the setup is successful, it should return the correct answer, 4, confirming that the model has been downloaded, installed, and is functioning properly. To stop the model, type “/bye” in the prompt, and it should exit. Qwen3:8b provides a good balance between performance and resource usage, making it a suitable choice for this hardware configuration. If more RAM is available, you can opt for larger LLMs that offer stronger reasoning and coding capabilities. Installing LibreChat With Docker LibreChat is an open-source AI platform that provides a unified and customizable interface for interacting with multiple AI models. It enables us to manage all our AI conversations from a single application while supporting features such as AI agents, Model Context Protocol (MCP) servers, custom tools, and integrations with both local and cloud-based LLMs. LibreChat acts as the front-end chat interface that communicates with the locally running Qwen3:8b model through Ollama. It allows us to execute AI-powered browser automation workflows entirely on our local machine. Follow the steps below to install LibreChat: Step 1: Clone the LibreChat GitHub Repository The repository can be cloned by running the following command: Plain Text git clone https://github.com/danny-avila/LibreChat After cloning the repository, navigate to the LibreChat folder, copy the .env.example file, and create a new .env file from it. Plain Text cd LibreChat cp .env.example .env Let's keep the .env file as it is, using the default values. Step 2: Connect Ollama to LibreChat Ollama can be connected to LibreChat by updating its configuration in the “librechat.yaml” file. The example file is already available in the cloned repo. Run the following command to copy librechat.example.yaml and create librechat.yaml. Plain Text cp librechat.example.yaml librechat.yaml Update the following configuration in the file to connect Ollama to LibreChat: YAML endpoints: custom: - name: "Ollama" apiKey: "ollama" baseURL: "http://host.docker.internal:11434/v1" models: default: - "qwen3:8b" fetch: true titleConvo: true titleModel: "current_model" summarize: false summaryModel: "current_model" modelDisplayLabel: "Ollama" Make sure that this configuration is added to the “custom” block, which falls under the “endpoints” block. This configuration adds Ollama as a custom AI endpoint in LibreChat. The baseURL tells LibreChat where to connect to the Ollama API, while the default model specifies that Qwen3:8b should be used by default. Since LibreChat is running inside a Docker container while Ollama is running directly on the host machine, we use http://host.docker.internal:11434/v1 instead of localhost. The special hostname host.docker.internal allows the Docker container to access services running on the host system, enabling LibreChat to connect to the locally running Qwen3:8b model through Ollama. Setting fetch: true allows LibreChat to automatically detect and display all models available in Ollama. The remaining options configure the user interface by generating conversation titles using the current model, disabling conversation summarization, and displaying the endpoint with the label Ollama in the LibreChat interface. Step 3: Mount the Configuration in the docker-compose-override.yml The docker-compose-override.yml can be copied and created in the same way as we did “librechat.example.yaml”. Plain Text cp docker-compose.override.yml.example docker-compose.override.yml The following block should be updated in the docker-compose.override.yml file. YAML services: api: volumes: - ./librechat.yaml:/app/librechat.yaml This file mounts the custom “librechat.yaml” configuration file into the LibreChat container. By mapping ./librechat.yaml to /app/librechat.yaml, Docker ensures that LibreChat uses the custom configuration each time the container starts. This approach allows us to modify settings, such as custom endpoints and AI models, without rebuilding the Docker image. Step 4: Start the LibreChat Application Using Docker Compose The LibreChat application can be started using the following command: Plain Text docker compose up -d It will take some time for the Docker images to download, and containers will start. Run the following command from the terminal to check the Container status: Plain Text docker ps -a This command displays the status of all Docker containers. If any container is unhealthy or encounters an issue, its status will be clearly indicated in the output. In case any container is unhealthy or encounters an issue, the following command can be run to check its logs: Plain Text docker logs <container name> Once all the containers are started successfully, open a new browser and navigate to http://localhost:3080 to start LibreChat. Since we are accessing LibreChat for the first time, we will be prompted to register and create a new user account. After completing the registration process, we can sign in and start using the application. Step 5: Selecting Ollama > Qwen3:8b Model By default, the gpt-5.5 model is selected. To select the Qwen3:8b model: Click on the gpt-5.5 modelSelect Ollama > Qwen3:8b Once the Qwen3:8b model is selected, we can verify if it is working by sending a simple prompt such as “What is 2+2?” Make sure the command “ollama serve” is already running in the terminal in the background, else the model Qwen3:8b won't work on LibreChat. Once we receive a successful response from the model, we can confirm that the Qwen3:8b model has been configured and integrated successfully with LibreChat. Install Playwright MCP Server Playwright MCP can be installed by running the following command in the terminal: Plain Text npx @playwright/mcp@latest \ --host 0.0.0.0 \ --allowed-hosts "*" \ --port 8931 \ By default, Playwright MCP listens only on localhost, which means applications running inside Docker (like LibreChat) cannot connect to it. Using --host 0.0.0.0 makes the server accessible from Docker containers, while --allowed-hosys "*" allows requests from host.docker.internal instead of restricting access to localhost. Once the Playwright MCP server is started, we can leave it running in the terminal. After the Playwright MCP server starts, it shows the following message at the bottom: “For legacy SSE transport support, you can use the /sse endpoint instead”. We will configure the Playwright MCP server using the SSE (Server-Sent Events) transport. Although Playwright MCP also supports the Streamable HTTP transport, LibreChat currently does not support connecting to it via the /mcp endpoint. Therefore, the SSE transport is used to establish a reliable connection between LibreChat and the Playwright MCP server. Configure Playwright MCP Server in LibreChat Playwright MCP server can be added to LibreChat by updating the following configuration in the “librechat.yaml” file. YAML mcpServers: playwright: type: sse url: http://host.docker.internal:8931/sse timeout: 120000 This configuration registers the Playwright MCP server with LibreChat. The type: sse setting specifies that the connection uses the Server-Sent Events (SSE) transport, while the url points to the Playwright MCP server running on the host machine. The hostname host.docker.internal allows the LibreChat Docker container to communicate with services running outside the container. The timeout: 120000 sets the request timeout to 120 seconds, giving the AI agent sufficient time to complete browser automation tasks before the connection expires. However, the timeout can be extended to 15–20 minutes or more, as there is no harm in doing that. YAML mcpSettings: allowedDomains: - 'host.docker.internal:8931' - 'localhost:8931' The mcpSettings configuration also needs to be added under the ‘actions’ block in the “librechat.yaml” file. The mcpSettings.allowedDomains section defines the list of trusted MCP server endpoints that LibreChat is allowed to connect to. By including both host.docker.internal:8931 and localhost:8931, LibreChat can establish a secure connection to the Playwright MCP server, whether it is accessed from within the Docker container (host.docker.internal) or directly from the host machine (localhost). Any MCP server not included in this list will be blocked, providing an additional layer of security. Restart the LibreChat app so it reads the newly configured Playwright MCP server: Plain Text docker compose restart That, or we can also shut down the already running LibreChat and start it again by using the commands below: 1. To shut down LibreChat: Plain Text docker compose down 2. To start it again: Plain Text docker compose up -d After restarting LibreChat, log in and navigate to the home page, and follow the steps below: Click on the MCP Settings menu on the left-hand menu panel.In the MCP Settings window, click on the “+” button to add MCP. Fill in the details for adding the Playwright MCP server; make sure to add the following settings: MCP server URL: http://host.docker.internal:8931/sseTransport: SSEAuthentication: NoneTick the “I trust this application” checkbox. Click on the “Create” button to save the details. Make sure that the Playwright MCP server is started and running on the terminal as discussed in the earlier section Click Connect for the newly created MCP server to establish the connection and begin using it. If everything is fine, a message should be displayed on successful connection. Understanding Model Context Protocol (MCP) By itself, a large language model (LLM) is limited to generating text. It can answer questions, explain concepts, write code, or summarize information, but it cannot directly interact with external systems or perform real-world actions. Model Context Protocol (MCP) changes this by enabling AI models to communicate with external tools and services through a standardized interface. Instead of simply providing suggestions, an AI model can execute tasks such as interacting with browsers, reading files, querying databases, or creating pull requests. Think of MCP as USB for AI A simple way to understand MCP is by comparing it to the USB standard. Before USB became the universal standard, every hardware manufacturer used its own proprietary connector. Printers, keyboards, cameras, and other peripherals all required different cables and custom software integrations. This made connecting devices unnecessarily complicated. USB solved this problem by introducing a common communication standard. Once both the computer and the device supported USB, they could communicate regardless of the device type. Whether you connected a keyboard, webcam, microphone, or external hard drive, the same protocol handled the communication. MCP brings the same level of standardization to AI systems. Without MCP, every AI application requires building and maintaining custom integrations for every external tool it wants to use. If we switch to a different AI application, those integrations often need to be recreated from scratch, resulting in duplicated effort and increased maintenance. A collection of awesome servers for the Model Context Protocol can be found at mcpservers.org. With MCP, tools expose a common interface that any MCP-compatible AI application can use. The AI model only needs to understand the MCP protocol, while the implementation details are handled by the individual MCP servers. Why MCP Matters for QA Automation For QA Automation Engineers, MCP unlocks the ability to automate complete testing workflows rather than isolated tasks. Consider the following request: “Read the Jira story, generate Playwright tests, execute them, analyze any failures, and create a GitHub pull request.” With MCP, the AI agent can coordinate multiple tools to complete the entire workflow. For example, it can: Read the user story from JiraAccess the application’s source code from GitHubGenerate Playwright TypeScript testsExecute the tests in a real browserCapture screenshots, logs, and execution reportsCommit the generated tests to GitHubUpdate the Jira ticket with the test results Each of these actions may be handled by a different MCP server, such as a Jira MCP server, GitHub MCP server, and Playwright MCP server. From the AI model’s perspective, however, every server is accessed using the same standardized MCP protocol. This standardization is what makes MCP so powerful. Rather than building custom integrations for every tool, AI systems communicate through a single, consistent protocol. As a result, MCP servers for Playwright, GitHub, databases, and many other services can be integrated and used in a uniform, scalable manner, significantly simplifying the development of AI-powered automation workflows. Creating an AI Agent With Playwright MCP Server in LibreChat for Automation Testing Let’s create a new AI Agent for browser automation testing with Playwright MCP using the steps below: Step 1: Click on the Agent Builder menu on the left-hand menu panel. Step 2: Enter the following mandatory details to create a new agent: Name: Provide a meaningful name to the agent.Category: Provide a category to the agent.Model: Select Qwen3:8bMCP Servers: Click on the Add MCP Server Tools button > Select the Playwright MCP Server that we created in the earlier section.Click on the Save button. Step 3: Update the model parameters. Clicking on the Model field, which has Qwen3:8b selected, should open the Model Parameters page. The following parameters can be set using this page: Provider: OllamaModel: Qwen3:8bTemperature: 0.2Top P: 0.85Frequency Penalty: 0.00Presence Penalty: 0.00Reasoning Effort: MediumReasoning Summary: Auto Click on the Save button to set the parameters. Step 4: Setting the instructions for the AI agent. The Following instructions can be pasted into the Instructions field in the Agent Builder window, or a “SKILL.MD” file can be created and uploaded using the Skills section of this agent. Markdown # Skills for the Local AI Agent for automation testing You are an expert QA Automation Engineer controlling a browser through Playwright MCP. Your goal is to execute browser actions safely and reliably. ## Tool Usage Rules - Do not run all MCP tools at the same time - Use only one Playwright MCP tool at a time. - Wait for the result of each tool before deciding the next action. - Never assume the page state. - Inspect the current page before interacting. - Do not start the next MCP tool unless the first one is complete ## Navigation Rules Treat the following actions as navigation-triggering actions: - Clicking Login, Submit, Continue, Save, Next, Checkout, etc. - Clicking any hyperlink. - Form submission. - Any action that changes the URL or reloads the page. - Wait until the page is fully loaded before making another tool call. After any navigation-triggering action: 1. Do not call any DOM inspection tool immediately. 2. Wait until the page has completely loaded. 3. Wait for the URL to stabilize if it changes. 5. Continue only after the new page is available. 6. Never inspect the previous page after navigation. ## Rules for locating web elements - Take a fresh snapshot to inspect the current page - Do not use XPath locator strategy - Use the same field name to locate elements, do not hallucinate and add prefix or suffix to field names - Use Semantic locator strategy: getByRole, getByText, getByLabel, getByPlaceHolder, getByAltText, getByTitle, getByTestId - Never use brittle CSS selectors such as .btn-primary, .container > div:nth-child(2), #content div span, or auto-generated classes. - Avoid nth() unless there is no unique locator. ## Interaction Rules - Verify and confirm that an element exists before interacting. ## Error Recovery If any Playwright tool fails: - Stop issuing new actions. - Inspect the current page. - Check Interaction Rules - Determine whether navigation has occurred. - Retry only if the page state confirms it is safe. - Do not repeat the same action more than once without confirming that the page state has not changed. Never repeat the same click more than once without checking the current page. ## Important If a click causes navigation, always assume the previous execution context has been destroyed. Do not read the DOM until the new page has fully loaded and a fresh snapshot has been obtained. Show a summary of test execution with the step count and pass or fail status - Run only the steps that are provided; do not hallucinate - Any deviation from these rules is not acceptable - Do not generate any additional steps - Always prioritize stability over speed. Providing instructions to an AI agent helps define its behavior, responsibilities, and the boundaries within which it should operate. These instructions act as persistent guidance, ensuring the agent follows consistent practices every time it performs a task instead of relying solely on the user’s prompt. For detailed setup instructions and troubleshooting guidance, refer to the GitHub repository. With these steps, the local AI agent is now ready to take commands. Running the AI Agent for Browser Automation To start using the AI Agent, click on New Chat.Click on the model name dropdown and select My Agents > The name of the agent that you created. Let’s use the following simple prompt and see how it works. Plain Text open http://playwright.dev verify the page title Once the prompt is submitted, we can observe the browser as the AI agent begins executing the task. The agent invokes the Playwright MCP server, which automatically launches a browser and performs the requested actions to navigate to the website and interact with the page. After the task is completed, Qwen3:8b analyzes the outcome and returns the results directly in the LibreChat conversation, demonstrating browser automation powered by Playwright MCP and Qwen3:8b. Let’s run another prompt for a login test scenario: Plain Text Navigate to https://parabank.parasoft.com/parabank/index.htm Locate "Username" field using "name=username" Enter "john" into the "Username" field. Locate "Password" field using "name=password" Enter "demo" into the "Password" field. Locator "Log In" button using "input[type="submit"] Click on the "Log In" button Verify that the "Accounts Overview" page is displayed This prompt also takes some time to understand the request before execution begins. It is important to note that the clearer and more specific the prompt, the more efficiently the AI agent can interpret and execute it. Well-structured prompts reduce ambiguity, minimize the chances of hallucinations, and typically result in faster execution and more accurate outcomes. As a best practice, break complex tasks into clear, sequential instructions whenever possible to improve the agent’s reliability and overall performance. As shown in the screenshot above, the AI agent invoked five tools from the Playwright MCP server to interact with the application and complete the requested workflow. It navigated to the website, located the username and password fields, entered the provided credentials, and submitted the login form. Finally, it verified that the login was successful by confirming that the “Accounts Overview” page was displayed. Since this setup runs entirely on a local machine, the AI agent takes approximately one minute to begin execution and around 4–5 minutes to complete a simple scenario. For more complex scenarios involving multiple steps, validations, or integrations, the AI agent is expected to take longer to analyze the request and complete the execution. But Execution time can be significantly reduced by running the setup on a machine with more powerful hardware, such as additional RAM, a faster CPU, or a dedicated GPU. Watch the step-by-step YouTube tutorial for Building your Local AI QA Engineer. Final Words Building a local AI QA engineer with Docker, Ollama, LibreChat, and Playwright MCP is an excellent way to explore the future of AI-powered software testing while keeping complete control over the data and infrastructure. By running everything locally, we eliminate recurring API costs, improve data privacy, and create a flexible environment for experimenting with AI-assisted browser automation using natural language. This setup is only the beginning of what’s possible. As we become more familiar with MCP and AI agents, the local QA assistant can be extended by integrating tools such as GitHub, Jira, databases, or custom MCP servers to automate even more of the testing workflow. Happy AI-powered testing!!

By Faisal Khatri

CORE

What Nobody Tells You About Running AI Models in Docker

It was 2:14 in the morning when the pager went off. Our recommendation model's inference service had started returning 503s under a traffic spike that, frankly, wasn't even that big. Maybe three times the normal load. By the time I'd opened my laptop, the container had been OOM-killed four times in ten minutes, and Kubernetes was cheerfully restarting it into the same wall every ninety seconds. The image was 14GB. Cold start took eighty seconds. Nobody on the team had looked closely at any of that until it started costing us actual money in lost requests. That night is the reason I now have strong opinions about Docker and AI infrastructure. Why This Keeps Happening Containers became the default way to ship machine learning models because they solve a real problem: a model trained with CUDA 11.8, PyTorch 2.1, and a very specific glibc version doesn't reliably run on a colleague's machine, let alone a fleet of GPU nodes spread across three cloud regions. “It works on my machine” isn't a joke in ML infra; it's a recurring incident report. Docker gives you a way to freeze that dependency tree and ship it as one artifact, and that part genuinely works. What doesn't get discussed enough is the many issues that Docker does not resolve, along with the subtle ways teams exacerbate problems by forcing AI workloads into a packaging model that was originally designed for stateless web services. What We Tried First (and Why It Blew Up) Our first version of the inference image was, in hindsight, a small crime. We started from nvidia/cuda:12.2.0-devel-ubuntu22.04 because someone had seen it in a tutorial, installed the full CUDA toolkit, pip-installed every dependency without pinning, and copied in not just the model weights but three checkpoint versions “just in case.” Fourteen gigabytes. Every deployment pulled the entire image onto a fresh node, and during autoscale events, we experienced over a minute of waiting just for the image to be pulled before the container started loading the model into GPU memory. The first fix everyone reaches for is a multi-stage build, and yes, it helps — but it's not the silver bullet people pitch it as. Splitting a devel build stage from a runtime stage cut us from 14GB to roughly 6GB: Dockerfile FROM nvidia/cuda:12.2.0-devel-ubuntu22.04 AS builder WORKDIR /build COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt --target=/deps FROM nvidia/cuda:12.2.0-runtime-ubuntu22.04 COPY --from=builder /deps /usr/local/lib/python3.10/site-packages COPY model/ /app/model/ COPY serve.py /app/ WORKDIR /app CMD ["python3", "serve.py"] That's better, but the real lie in that Dockerfile is the COPY model/ line. We baked multi-gigabyte weights into an image layer, causing a full re-push of the weights with every code change, even minor ones, since Docker re-hashes the entire build context. We moved weights to an external volume pulled from object storage at container start, with a checksum cache to skip redundant downloads. That alone cut most of our deployment time. The trade-off is a slightly more complex startup script and a new dependency on storage being reachable at boot, which is its failure mode. There's no free lunch here; you're just choosing which problem is going to page you at 2 am. The Detour: Skipping Containers Entirely We also tried, briefly, running everything on bare metal with conda environments and skipping containers altogether, mostly because one engineer was convinced Docker added overhead with no real benefit for GPU workloads. It's not a crazy position; Docker's GPU story is genuinely leaky. <nvidia-container-toolkit exposes the host driver directly to the container, leading to potential mismatches that application-level packaging cannot resolve, so there is no real isolation>. However, we reverted that experiment within a sprint because as soon as more than two people interact with the training pipeline, environment drift reoccurs immediately. “Works on my conda env” is just “works on my machine” wearing a hat. What Actually Held Up in Production The architecture that worked was less about clever Docker tricks and more about admitting that an inference container and a training container have almost nothing in common and shouldn't share a Dockerfile, a registry strategy, or a deployment pattern. For serving, we kept images lean, stateless weights externalized, and a health assessment that actually runs a tiny dummy inference instead of just pinging an HTTP port. A server can report itself as “up” while a model failed to load correctly, and that gap has burned us more than once. Locally, a docker-compose GPU reservation block mirrored how production scheduled GPUs, so dev environments stopped lying about resource contention: YAML services: inference: image: registry.internal/rec-model:latest deploy: resources: reservations: devices: - capabilities: [gpu] count: 1 healthcheck: test: ["CMD", "python3", "healthcheck.py"] interval: 15s timeout: 5s retries: 3 For training, we accepted slower builds because training jobs run for hours, and a ninety-second image pull is negligible compared to that, even though the images are larger. Spending engineering time shrinking training images was effort we'd burned for no real payoff. A contrarian point I'll happily defend: not every container needs to be small, only the ones sitting in your hot path. The other decision that mattered more than any Dockerfile tweak was plain layer-caching discipline, putting "pip install before COPY." It sounds obvious when written down, but I've reviewed more than one ML team's Dockerfile that copies the whole repo first “for simplicity” and invalidates every cached layer on a README change. Key Takeaways Externalize model weights from the image; baking them in wrecks your caching and your deploy speed.Multi-stage builds help, but they don't resolve a runtime image that's still hauling around a full CUDA devel toolkit.GPU isolation via containers is partial; complete driver and toolkit mismatches between host and container remain entirely your problem.Training and serving images deserve different optimization priorities; don't apply the same size obsession to both.A health check that only confirms the process is running, without verifying that the model has loaded correctly, will mislead you at the most critical moments. Closing Thought Docker didn't fail us that night; rather, it was our incorrect assumptions about its free services that let us down. It's easy to treat containers as a solved problem because the tooling is so mature for web services and then be surprised when AI workloads expose every shortcut you took. I still think GPU container isolation lacks a clean solution, and I’m curious if upcoming tools will address it or if we'll continue improving our health-check scripts. What's the worst 2 am lesson your infrastructure has taught you?

By Pruthvi Raj Seknametla

Coordinating AI Agents With AWS SQS: A Practical Queue-Based Architecture

Building a single AI agent is not usually the hard part. You send a prompt to a model, get a response back, and wire it into your app. Done. The hard part starts when that agent becomes one step in a larger system. A real AI workflow might need to ingest a file, extract text, chunk it, generate embeddings, call an LLM, write results to a database, sync to an external API, and notify a user. Those steps do not behave the same. Text extraction might finish in seconds. An LLM call might take minutes. A sync job might fail because some external API is having a bad day. That is where a lot of "agent" systems stop looking magical and start looking like regular distributed systems. I have seen this fail in boring ways: The same job gets processed twice.A worker writes to the database, then crashes before marking the job complete.A model call runs longer than expected and the message gets picked up again.A retried tool call creates duplicate external writes.Failed jobs sit in processing until someone manually checks the database. None of this is new. AI agents do not magically avoid old infrastructure problems. They still need queues, retries, idempotency, durable state, and monitoring. AWS SQS is a good fit for that middle layer. It is not a full workflow engine. I would not use it for every orchestration problem. But if you need a durable queue between independent agent stages, SQS is simple, reliable, and usually enough. The Coordination Problem A basic multi-stage AI workflow often looks like this: Plain Text Input source -> ingestion -> processing -> generation -> sync The first version is usually a database table with a status column. That works for a while. Then concurrency shows up. Two workers read the same pending row. A process crashes and leaves a job stuck in processing. Someone adds sleep(30) because the previous step "usually finishes by then." That last one is the kind of fix that works just long enough to become a production bug. A queue gives each stage a cleaner boundary. One stage publishes work. Another stage consumes it. If the next stage slows down, the queue absorbs the backlog instead of forcing the whole pipeline to wait. Plain Text Input Source -> ingest_queue -> Ingestion Worker -> chunk_queue -> Chunking Worker -> embedding_queue -> Embedding Worker -> summary_queue -> Summary Worker -> sync_queue -> Sync Worker Now ingestion can scale separately from summarization. If LLM generation is slow, messages pile up in summary_queue. That is not automatically a failure. That is what the queue is there for. A failed summary worker does not corrupt the whole workflow. The message can be retried. If it keeps failing, it moves to a dead letter queue. Standard Queues vs. FIFO Queues SQS gives you two main queue types: standard queues and FIFO queues. Standard Queues Standard queues give at-least-once delivery and best-effort ordering. A message can be delivered more than once. Messages may not arrive in the exact order sent. That sounds scary, but most background AI work should already handle this. Use standard queues for work like document processing, embedding generation, batch classification, independent user requests, and webhook processing. For these jobs, throughput matters more than strict ordering. FIFO Queues FIFO queues preserve ordering within a MessageGroupId and support deduplication. Use when sequence actually matters: conversation turns, per-user workflows, ordered state transitions. Python response = sqs.send_message( QueueUrl=queue_url, MessageBody=json.dumps(payload), MessageGroupId=payload["user_id"], MessageDeduplicationId=payload["task_id"] ) Be careful with the group ID. If every message uses the same MessageGroupId, you have serialized the whole queue by accident. Give each conversation, user, or workflow its own group ID so you preserve ordering per entity while allowing parallelism across different ones. My default rule: start with standard queues unless ordering is clearly required. Then make the handler idempotent. That matters more than the queue type. Ensuring Idempotency in Your Agent Flow Idempotency means the same task can run more than once without creating duplicate or incorrect side effects. This is the part I would not skip. SQS standard queues use at-least-once delivery, so duplicates are part of the contract. But this matters even more with AI workloads because model calls are expensive and outputs can be non-deterministic. Retrying the same prompt may cost money and return a different answer. Retrying the same tool call may send a duplicate email or write a second database row. The basic pseudo workflow: Plain Text receive message check if task already completed if completed, delete message and exit if not completed, process task store result delete message Simple version: Python def handle_message(message, store, sqs, queue_url): payload = json.loads(message["Body"]) task_id = payload["task_id"] if store.already_completed(task_id): sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=message["ReceiptHandle"]) return {"status": "skipped", "task_id": task_id} result = run_agent_logic(payload) store.mark_completed(task_id, result) sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=message["ReceiptHandle"]) return {"status": "completed", "task_id": task_id} The store can be Postgres, DynamoDB, Redis, or anything durable with atomic writes. For Postgres, a unique constraint saves you: SQL CREATE TABLE agent_task_results ( task_id TEXT PRIMARY KEY, status TEXT NOT NULL, result JSONB ); INSERT INTO agent_task_results (task_id, status) VALUES ($1, 'processing') ON CONFLICT (task_id) DO NOTHING; If the insert succeeds, this worker owns the task. If it does nothing, another worker already claimed or completed it. The Failure Case I Designed Around Plain Text summary_queue -> Summary Worker -> Postgres -> sync_queue The summary worker receives a message, calls an LLM, writes the summary to Postgres, then deletes the SQS message. Now suppose the worker writes to Postgres but crashes before deleting the SQS message. From SQS's point of view, the job never finished. After the visibility timeout expires, another worker receives the same message and runs the task again. Without idempotency, that retry may call the LLM again, generate a slightly different summary, and write a second result. A safer handler checks whether model output already exists before calling the model: Python def summary_handler(payload, store): task_id = payload["task_id"] existing = store.get(task_id) if existing and existing.get("model_output"): summary = existing["model_output"] else: text = load_text(payload["input"]["text_uri"]) summary = call_llm(text) store.save_model_output(task_id, summary) store.save_final_result(task_id, {"summary": summary}) return {"next_stage": "sync", "next_input": {"summary_task_id": task_id} That avoids repeating the expensive part if the first attempt already got that far. Visibility Timeout When a worker receives a message, SQS hides it from other workers for the visibility timeout. If the worker finishes, it deletes the message. If the worker crashes, the message becomes visible again after the timeout expires. Too short: another worker receives the same message while the first is still running. Duplicate execution. Too long: failed jobs take too long to retry. Plain Text visibility_timeout = 2x to 5x expected processing time Reference: Metadata validation: 30-60 secondsEmbedding generation: 1-5 minutesLLM-heavy summary: 5-15 minutesLong document analysis: 15+ minutes with heartbeat For long-running tasks, extend visibility: Python sqs.change_message_visibility( QueueUrl=queue_url, ReceiptHandle=receipt_handle, VisibilityTimeout=extension_seconds ) The message should describe the work, not carry the workload. Bad: JSON {"task_id": "123", "full_pdf_text": "... thousands of lines ..."} Better: JSON { "task_id": "123", "stage": "summarize", "input": {"document_uri": "s3://bucket/docs/input.pdf"}, "metadata": {"user_id": "789", "priority": "normal"} } Store large files in S3. Send references through SQS. Do not let the queue become your storage layer. Dead Letter Queues A DLQ captures messages that fail repeatedly. Without one, poison messages cycle forever. Python sqs.set_queue_attributes( QueueUrl=main_queue_url, Attributes={ "RedrivePolicy": json.dumps({ "deadLetterTargetArn": dlq_arn, "maxReceiveCount": 5 }) } ) Use 3-5 as a starting point. A DLQ is not a trash bin - it's an alert. AI-Agent-Specific Failure Modes Duplicate LLM calls: Bigger bill, possibly different answer. Use task_id as idempotency key.Non-deterministic outputs: Store first successful output.Tool-call side effects: Make idempotent.Long-running inference: Use visibility heartbeat. What to Monitor MetricWhyApproximateAgeOfOldestMessageUser-facing delayApproximateNumberOfMessagesVisibleBacklogDLQ message countRepeated failures Two alerts: Oldest message exceeds latency targetDLQ has messages When SQS Is Not the Right Tool RequirementBetter fitSimple async tasksSQSVisual multi-step workflowStep FunctionsComplex event routingEventBridgeHuman approvalsStep Functions I have seen teams burn hours building multi-agent systems with database polling and sleep timers. It works at demo scale. It usually does not survive production traffic. SQS gives you durable message delivery primitives. But the app still needs idempotent handlers, visibility timeout tuning, and DLQ monitoring. Default architecture: One queue between major stagesStandard queues unless ordering requiredEvery handler idempotentLarge payloads outside the queueVisibility timeouts based on real processing timeDead letter queues for failures The difference between an AI demo and a reliable AI system is rarely the prompt. It is the infrastructure around the prompt. Build that layer intentionally.

By Lucas Yoon

From Idle Infrastructure to Elastic Capacity: Rethinking Kubernetes Scaling

Sponsored By: NutanixThe following is sponsored content. It may not reflect the views of our editorial staff. Most platform engineers are kept awake at night with some form of the same common complaint: the infrastructure bill does not align with what the infrastructure is really doing. For example, a GPU node pool provisioned for a monthly batch job might sit idle, burning budget for 20+ days out of 30. Or perhaps a business builds a standby data center designed specifically to account for a potential major outage, but that sits idle doing nothing every other day. CI/CD runners wait listlessly for the next pipeline trigger: fully provisioned, fully billed, but mostly idle. The above scenarios are nothing new. In fact, they could be considered the oldest problem in infrastructure: provisioning for peak, yet paying for average — or below. In modern technology, however, these pain points are felt mostly at scale. While enterprises spread Kubernetes® across hybrid clouds (on-premises clusters, cloud regions, and edge sites), idle capacity spreads beyond one team’s budget line. This spread compounds across every cluster that cloned or copied the same pattern of “just keep it running,” until it inevitably becomes a major structural cost that belongs to no one. So what is the solution? The assumption might be to contrive a smarter way of bin-packing always-on nodes. The real, better solution is to remove the notion of “always-on” entirely; instead, platform engineers should focus on building infrastructure that sits at true zero and becomes real, schedulable capacity only when required. Why does hot standby become a liability? The concept of over-provisioning is common, and it is a reasonable instinct that most platform engineers pursue. Teams that run batch jobs or use AI and ML pipelines often deal with expensive, hard-to-find resources (GPUs in particular), and the dreaded fear of a cold start leading to a delay in a critical job is all too real. The usual approach is to keep any needed resources reserved, even if the job requires them only once or twice in a given month. That basic instinct, when applied at the level of a whole data center, is the cause that results in the classic “hot standby” disaster recovery pattern: a fully provisioned secondary site that mimics (or mirrors) production, yet does not do real work, as it is waiting for a failover event that might never occur. It’s an expensive insurance policy that organizations hope never to use, and the expense often largely sits idle. As enterprises lean into AI and automated workloads, as well as retail and edge deployments, we see this pattern replicate further. Monthly recurring jobs don’t need their own dedicated GPU pool sitting idle for the rest of the month. Likewise, CI/CD pipelines don’t need agents that run around the clock to handle an occasional pull request. Multiply that behavior across every team, region, and cluster operating with the same "play it safe" mindset, and idle capacity stops looking like a rounding error. It becomes a real budget item. What appears to be a dozen reasonable decisions is, in reality, one costly pattern. That makes the root cause much harder to identify and resolve. While hot standby remains common for disaster recovery, the more immediate opportunity for Kubernetes platform teams is eliminating permanently provisioned node pools for intermittent workloads. Scale-from-zero applies the same principle — capacity exists only when demand requires it — but at the infrastructure layer where operational costs accumulate most quickly. How does scale-from-zero actually work? Scale-from-zero is easy to understand in practice, but a little more difficult to engineer. In essence, a node pool with zero running nodes should still be visible to the Kubernetes scheduler as “available capacity.” If the schedule cannot see it, it cannot plan for it, and a pod requesting resources from an empty pool will sit in a pending state until someone notices and takes steps to intervene. This is where capacity annotation becomes important. Instead of relying on the old approach of provisioning tiers (i.e.; the “gold, silver, and bronze” classes used to define the offerings of a node pool), Nutanix Kubernetes Platform (NKP) attaches metadata directly to the node pool definition, letting the scheduler know exactly what capacity would exist if a node were running CPU, memory, and GPU resources at a finer grain than “one whole GPU.” What is the Nutanix Kubernetes Platform? Nutanix Kubernetes Platform (NKP) is an enterprise Kubernetes platform that simplifies deploying, managing, and scaling fleets of Kubernetes clusters across hybrid and multicloud environments while giving organizations the flexibility of an open, Kubernetes-native architecture. Dynamic Resource Allocation for GPU workloads is a solid example of where this trend is heading. Instead of allocating your entire GPU to a job that only requires 50 GB of memory out of a 100 GB card, a scheduler can consider the actual resource needs and assign workloads in a more precise manner, even before provisioning any hardware. In both theory and practice, this means the scheduler can make a placement decision on a node pool with zero nodes running, treating the annotated capacity as though it were live, queuing the pod against the pool, and making decisions to trigger the autoscaler to actually provision resources. This is the part of the architecture where the real work occurs, behind the scenes. Cluster API (an open-source Kubernetes project for declarative cluster lifecycle management), with the Nutanix-specific provider acting as the implementor, gives IT teams a definitive way to define how a cluster or node pool looks. Cluster API then acts as the engine that drives the infrastructure to that state. What is CAPX? CAPX (the Cluster API Provider for Nutanix) is the component that translates Kubernetes infrastructure intent into concrete operations on Nutanix AHV, Nutanix's enterprise hypervisor. When Cluster API determines that a MachineDeployment needs additional capacity, CAPX reconciles that desired state by provisioning virtual machines, applying the appropriate templates and networking configuration, bootstrapping Kubernetes components, and registering the new node with the cluster. From the platform engineer's perspective, scaling remains declarative: the desired node count changes, while CAPX handles the infrastructure orchestration required to make that state a reality. CAPX allows admins to stand up Kubernetes clusters rapidly across multiple locations, including retail and AI edge deployment, providing an autoscaler that expands or shrinks the infrastructure automatically, without relying on someone to manually provision a VM at an inconvenient time. From an operator's perspective, the scaling workflow follows a predictable sequence: A workload is created with CPU, memory, GPU, or other scheduling requirements.The scheduler determines no existing node satisfies those requirements, leaving the pod in a Pending state.Cluster Autoscaler, the Kubernetes component responsible for adjusting node capacity based on pending workloads, evaluates the unschedulable workload and identifies a compatible scale-from-zero node pool based on its capacity annotations.Cluster API updates the corresponding MachineDeployment to request additional infrastructure.CAPX provisions the required virtual machine in Nutanix AHV, attaches networking, and performs the bootstrap process.The new node joins the Kubernetes cluster and reports a Ready status.Kubernetes binds the pending workload to the newly available node.Once demand subsides and scale-down thresholds are reached, the autoscaler removes the node and the pool returns to zero capacity. Logs, events, and the full picture It should go without saying that if the platform engineer and team can’t see it happening, then none of the above is useful. Kubernetes does give you basic autoscaling visibility out of the box, but it lacks the observability level many enterprises really need. To validate a correctly running scaling event, you need two sources in particular: Source What it tells you Cluster Autoscaler logs These show you the actual scaling decision, why it chose to scale a pool, the capacity calculation that triggered it, and whether it considered alternative pools first. Kubernetes event traces These showcase what happened to the workload and the node, such as pod scheduling outcomes, node registrations, and readiness condition transitions. When viewed together, these two sources reconstruct the full state-machine transition from a pod against a zero-capacity pool to a pod running on live infrastructure. This is where many real-world misconfigurations emerge. For example, an application may be deployed without resource limits. If a workload consumes more memory than anticipated, the autoscaler can mistakenly provision additional nodes, even though the underlying issue is resource allocation rather than insufficient capacity. The fix is straightforward: set resource limits. Yet under deadline pressure, this step is easy to overlook, and the resulting costs may not become apparent until they appear on a cloud bill. NKP enhances enterprise cloud-native security beyond native open-source tooling through strategic ecosystem integrations. By partnering with RapidFort for vulnerability management and Canonical to deliver Ubuntu Pro as a trusted, built-in base OS option, NKP is designed to provide a secure, resilient, and compliant foundation for production workloads. Beyond securing the platform itself, production Kubernetes environments require operational consistency and visibility to run at scale. GitOps, specifically FluxCD, helps keep the desired state of managed clusters reconciled against a single Git source of truth. In addition, observability tools like Prometheus Alert Manager deal with the notification layer, routing scaling events to communication avenues like Slack, Microsoft Teams, or SMS messaging. This provides platform engineers with better visibility into scaling events, reducing the need to examine logs to determine whether a workload scaled when it shouldn't have. Automating Node Pool Lifecycle Scale-from-zero becomes far more valuable when node pool configuration is automated rather than managed manually. As environments grow, editing individual MachineDeployment manifests quickly becomes difficult to maintain, particularly when GPU pools, edge clusters, and development environments all require different scheduling policies. NKP provides the operational workflow for creating and managing node pools. Rather than manually editing manifests, platform teams can inject labels, taints, and capacity annotations into MachineDeployment definitions as part of a repeatable automation pipeline before committing those changes through GitOps. GPU node pool configuration checklist As an example, a GPU node pool might receive: Labels: identifying the workload typeTaints: preventing general-purpose schedulingCapacity annotations: describing available GPU, CPU, and memory resources Because these changes are generated consistently through automation instead of manual editing, platform teams reduce configuration drift across clusters. Combined with GitOps reconciliation through FluxCD, the desired configuration remains version-controlled, repeatable, and significantly easier to audit as infrastructure evolves. The next shift What ties these approaches together — active-active architectures instead of hot standby, capacity annotations instead of static machine classes, and scale-from-zero node pools instead of permanently reserved infrastructure — is a shift away from provisioning for the worst-case scenario and toward provisioning for actual demand. Infrastructure is no longer something you size once and live with. Instead, it becomes an elastic resource that expands and contracts in response to real-world signals, such as pending pods, queued pipeline jobs, or traffic spikes. This broader focus on infrastructure automation also extends to initiatives such as Nutanix’s bare-metal deployment capability NKP Metal, which aims to reduce cluster deployment times, reinforcing the same operational principle: infrastructure should be provisioned quickly and only when required. None of this can replace or eliminate the need for engineering judgment. Deciding which workloads belong on scale-from-zero pools versus always-on infrastructure still requires careful evaluation of latency requirements, cold-start risk, and workload characteristics. However, the infrastructure debt created by defaulting to a "just keep it running" approach is no longer an unavoidable consequence of operating Kubernetes at scale. Increasingly, capacity can be provisioned only when demand requires it, allowing infrastructure to align more closely with actual workload needs. Visit Nutanix to learn more about how Nutanix Kubernetes Platform enables deterministic scale-from-zero, infrastructure elasticity, and automated node pool lifecycle management across hybrid cloud environments.

By DZone Staff

Going Stateless: Scaling MCP Servers to Cloud-Native Java and HTTP

The Model Context Protocol (MCP) completely changed how we connect large language models to real-world data and tools. However, early versions of the protocol had a massive bottleneck for enterprise developers: they relied heavily on stateful, long-lived sessions. If you wanted to scale out your AI tools to handle thousands of concurrent agent workflows, you had to deal with sticky sessions, complex load balancing, and heavy memory overhead. The newest updates to the MCP specification solve this problem by introducing a completely stateless HTTP foundation. By removing the traditional initialization handshake and session IDs, MCP servers can now function as lightweight, independent microservices. When you combine this stateless evolution with cloud-native Java, you get the ultimate stack for cloud-native AI infrastructures. Why Stateless MCP Matters for Your Cloud Architecture In older stateful setups, an LLM host maintained an open connection to your server. If that specific server instance crashed or scaled down, the entire context of the conversation loop was lost. The latest specification shifts the paradigm. Every request sent from an AI agent or LLM host to an MCP server is now fully self-contained. The routing relies on two standard HTTP headers: Mcp-Method: Specifies the action (such as executing a tool or fetching a resource)Mcp-Name: Directs the request to the specific tool definition. Because the server no longer needs to remember who is calling it, you can place a standard load balancer in front of a cluster of MCP servers, distribute incoming requests evenly, and scale down to zero when traffic stops. The Cloud-Native Java Advantage: High-Density AI Tools While languages like Python and Node.js are popular in the AI space, they often struggle with heavy production workloads, multi-threading, and deep enterprise integration. Traditional Java solves these enterprise issues but comes with a high memory footprint and slower startup times—making it expensive to run as serverless microservices. This is exactly where cloud-native Java (e.g., Quarkus) shines. By utilizing ahead-of-time (AOT) compilation and GraalVM native images, Quarkus strips away the boilerplate runtime overhead. Plain Text ┌─────────────────────────────────────────────────────┐ │ Traditional Java MCP: ~150MB Ram | 2.5s Startup │ └─────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────┐ │ Cloud-Native Java MCP: ~18MB Ram | 0.015s Startup │ └─────────────────────────────────────────────────────┘ Instead of a single heavy backend trying to host dozens of different LLM tools, you can break your tools into highly specialized microservices. You can deploy a database-lookup tool, an internal API proxy, and a document parser as completely separate cloud-native Java applications. They will start instantly, use less than 20MB of RAM each, and scale up instantly when an AI agent triggers them. Building a Stateless MCP Resource With Cloud-Native Java Implementing a stateless tool in cloud-native Java with Quarkus is remarkably clean. By leveraging the reactive routing capabilities of Quarkus and standard Java objects, you can map the incoming JSON-RPC payloads directly to your business logic. Here is a conceptual example of how a stateless MCP tool controller looks in Quarkus using standard REST annotations: Java package com.example.mcp; import jakarta.ws.rs.POST; import jakarta.ws.rs.Path; import jakarta.ws.rs.HeaderParam; import jakarta.ws.rs.Produces; import jakarta.ws.rs.core.MediaType; import io.smallrye.mutiny.Uni; @Path("/mcp/v1") public class StatelessMcpResource { @POST @Path("/tools") @Produces(MediaType.APPLICATION_JSON) public Uni<McpResponse> handleToolExecution( @HeaderParam("Mcp-Method") String method, @HeaderParam("Mcp-Name") String toolName, McpRequestPayload payload) { // The request is entirely self-contained; no session lookup required. if ("tools/call".equals(method) && "fetch_customer_data".equals(toolName)) { return executeCustomerLookup(payload.getArguments()); } return Uni.createFrom().item(McpResponse.error("Tool or method not found")); } private Uni<McpResponse> executeCustomerLookup(JsonElement arguments) { // Business logic interacting with reactive databases or internal services return Uni.createFrom().item(new McpResponse("Customer data retrieved successfully.")); } } Summary The combination of a stateless protocol and a cloud-native Java framework removes the operational friction in building enterprise AI features. By deploying stateless MCP servers on cloud native Java - Quarkus, you gain the type of predictable scaling, rapid response times, and bulletproof reliability that modern production environments demand. Check out more from my series here.

By Daniel Oh

CORE

12 Factor Framework for Building Secure and Compliant Cloud Applications

It began with a late-night alert. A critical cloud application, serving thousands of users, had just been flagged for a security violation. No “hack” had occurred; nothing obviously was broken. What appeared to be a minor misconfiguration had quietly exposed sensitive data. The system was still running. The business was still operating. But compliance? Already compromised. The team scrambled. Was it an identity issue? A pipeline gap? A missing policy? Every layer seemed secure in isolation—but together, something had slipped through. That night revealed a hard truth: security and compliance aren’t features you add—they are properties you design into every layer of a cloud application. This is where a structured approach becomes essential—a way to think systematically about building applications that are not just scalable and observable but inherently secure and compliant by design. This blog explores a 12-factor security framework to do exactly that. What Does “Secure and Compliant by Design” Mean? “Secure and compliant by design” means that security and compliance are built into the foundation of a cloud application—not added later as patches, tools, or audit activities. Traditionally, teams would: Build the application firstTest functionalityAdd security checks before releasePrepare compliance evidence only during audits This approach creates gaps because security becomes reactive and compliance becomes periodic. "Secure and compliant by design" flips this model and introduces three key shifts: Shift left: Security and compliance should start early. Secure coding practicesDependency scanning in developmentPolicy checks in CI/CD pipelinesOutcome: Issues are prevented rather than fixed later.Continuous, not periodic: Compliance is no longer an annual or quarterly exercise. Policies are enforced automaticallySystems are continuously validatedDrift is detected in real timeOutcome: You're always audit-ready.Embedded across layers: Security and compliance are enforced at every layer of the system. Application layer – secure code, input validationInfrastructure layer – hardened configurationsIdentity layer – strict access controlsRuntime layer – monitoring and threat detectionOutcome: No single point of failure. The 12 Factors Overview Security and compliance are not a single layer—they are a system of interconnected controls surrounding and protecting the application at every stage. The proposed 12 factors are organized across five architectural pillars: Category Objective Associated Factors Application Foundations Establish secure, consistent, and portable application design principles Codebase, Dependencies, Configuration Identity, Trust, and Security Controls Protect identities, secrets, and trust boundaries across the application lifecycle Credentials & Secrets Management, Identity and Access Control Runtime and Delivery Architecture Govern application packaging, deployment, and runtime execution behavior Build–Release–Run, Processes, Port Binding Observability, Governance, and Compliance Enable monitoring, auditability, policy enforcement, and operational visibility Logs, Admin Processes Operational Resilience and Scalability Improve elasticity, fault tolerance, and operational continuity Concurrency, Disposability, Dev/Prod Parity The architecture diagram below shows the proposed structure of the 12 factors for secure and compliant cloud applications; the factors are grouped into five capability domains. Rather than functioning as isolated practices, these domains collectively establish a secure-by-design, resilient, scalable, and compliance-aware cloud-native architecture that supports both technical and business outcomes. Note: Operational resilience is not represented by a single control but emerges from the combined implementation of incident response, observability, workload protection, and robust infrastructure practices. Operationalizing the 12 Factors Modern cloud applications cannot use siloed security controls or compliance checks that come into play at later stages of the development process. Security and compliance should be built into the development lifecycle and applied consistently across architecture, deployment workflows, runtime environments, and operational processes. The 12-factor framework outlines a framework for organizing security and compliance practices that consists of five key, interlinked layers: Application Foundation, Identity and Trust, Runtime and Delivery, Operational Resilience, and Observability & Governance. Each layer addresses a specific objective, but they all help to form a secure-by-design, compliant-by-default architecture. Application Foundation This layer builds the baseline structure and security posture of the application. It focuses on ensuring that application configurations, dependencies, and code artifacts remain consistent, reproducible, and externally managed. Key considerations include: Externalizing configurations and secretsManaging dependencies through controlled mechanismsMaintaining immutable and version-controlled artifactsStandardizing application packaging and deployment patterns Having a good foundation reduces configuration drift, minimizes hidden dependencies, and creates predictable application behavior across environments. Identity and Trust Identity becomes the primary security boundary in cloud-native systems where applications, services, and workloads communicate dynamically. This layer focuses on: Strong workload and service identitiesSecure authentication and authorization mechanismsPrinciple of least privilege accessSecret lifecycle and credential management The objective is to establish trusted interactions between users, applications, services, and infrastructure resources. Runtime and Delivery Applications continuously evolve through deployment pipelines and operational updates. Secure runtime execution and delivery processes ensure that changes can be introduced without compromising reliability or compliance. Key areas include: Secure CI/CD pipelinesImmutable deployment patternsControlled rollout strategiesContainer and workload security enforcementPolicy-driven deployment validation This layer enables rapid delivery while preserving operational safety. Observability and Governance Visibility and governance provide continuous assurance that systems operate within expected security and compliance boundaries. This layer includes: Metrics, logs, and distributed tracingContinuous compliance monitoringPolicy-as-Code enforcementAudit evidence collectionSecurity posture assessment and reporting Effective observability transforms operational signals into actionable insights while supporting governance requirements. Operational Resilience Security and compliance also depend on maintaining application availability and handling failures gracefully. Important capabilities include: Self-healing mechanismsControlled failure handlingHigh availability strategiesBackup and recovery proceduresAutomated incident response Resilience mechanisms reduce operational risk and help maintain service continuity under adverse conditions. These five layers build a comprehensive defense architecture where security, compliance, operational reliability, and governance are not discrete activities but rather integrated functions of the application. The subsequent sections describe each of the twelve factors in detail and explain their practical implementation within cloud-native environments. Architectural Anti-Patterns in Cloud-Native Security and Compliance Although many organizations are investing in cloud security tools and compliance frameworks, most of the time failures cannot be attributed to technology but rather to recurring anti-patterns, habits, and decisions that unintentionally introduce risk. Understanding these pitfalls is key in developing systems that are truly secure and compliant by design. Below are some of the most common anti-patterns: Hard-coded secrets and configuration: Credentials, API keys, or environment-specific settings are embedded directly in the source code.Impact: Increased risk of credential exposure, security breaches, and configuration drift.Over-privileged access and shared identities: Users and services receive permissions beyond operational requirements.Impact: Expands the attack surface and increases the blast radius of compromised workloads.Security as a late-stage activity: Security validation occurs after development and deployment activities are completed.Impact: Delayed remediation, higher operational cost, and inconsistent policy enforcement.Mutable infrastructure and manual changes: Direct modifications are applied to running environments without controlled deployment processes.Impact: Creates configuration drift and reduces reproducibility.Limited observability and reactive monitoring: Insufficient metrics, logs, and traces limit operational visibility.Impact: Slower incident detection and longer recovery times.Siloed governance and compliance processes: Governance activities operate independently from engineering workflows.Impact: Compliance gaps, duplicated effort, and reduced delivery efficiency.Ignoring runtime security controls: Security controls focus only on build-time validation and neglect runtime monitoring.Impact: Undetected threats and reduced visibility into active workloads.Missing continuous feedback loops: Application metrics, security events, operational incidents, and compliance findings are not continuously integrated back into development and operational workflows.Impact: Repeated failures, delayed remediation, limited learning from incidents, and slower improvement of security and operational practices. Aligning With Industry Standards The framework aligns with global security and compliance standards. The framework embeds governance, access control, observability, and resilience practices directly into the software lifecycle by not treating compliance as a distinct validation exercise. The table below shows how the 12-factor framework aligns with common industry security and compliance standards. Standard / Framework Primary Focus How the 12-Factor Framework Supports It NIST Cybersecurity Framework Identify, Protect, Detect, Respond, Recover Supports policy enforcement, monitoring, identity controls, and resilience practices SOC 2 Security, availability, processing integrity Improves auditability, access management, and operational monitoring ISO 27001 Information security management Encourages risk-based controls, governance processes, and secure operational practices CIS Benchmarks Secure system and workload configuration Reinforces secure configurations and standardized deployment practices Zero Trust Architecture Continuous verification and least privilege Strengthens workload identity, authentication, and access controls HITRUST Security and compliance for regulated data Enhances governance, audit controls, and protection of sensitive information Getting Started: A Practical Roadmap Adopting a secure and compliant cloud application framework is not a one-time effort, and it is a progressive journey. This needs to be treated as a phased transformation with continuous improvements to be successful. Phase 1—Assess and Baseline: Before implementing controls, it is critical to understand your current posture. Focus areas: Inventory applications, services, and dependenciesEvaluate current security practices across the lifecycleIdentify gaps in identity, configuration, and observabilityMap existing controls to compliance requirements (e.g., SOC2, ISO 27001)Outcome: Clear visibility into risk exposure and compliance gapsA prioritized list of areas needing attentionPhase 2 - Establish Secure Foundations: Build the baseline capabilities that enforce security by default. Focus areas: Implement secure CI/CD pipelines with integrated scanning. Centralize secrets management and eliminate hardcoded credentials. Enforce least-privilege IAM policies Define secure configuration baselines (IaC templates, guardrails)Outcomes: Strong foundation layer aligned with Application Foundation and Identity pillars Reduced risk from common vulnerabilitiesPhase 3 - Automate Security and Compliance: Manual processes do not scale in cloud environments; automation is essential. Focus areas: Introduce policy-as-code (OPA, Kyverno)Enable continuous compliance monitoringAutomate security checks in pipelinesDetect and remediate configuration driftOutcome: Shift from reactive to proactive enforcementAlways-on compliance posturePhase 4 - Strengthen Runtime and Resilience: Once the foundation is secure, focus on protecting systems in production. Focus areas: Implement runtime threat detection and workload protectionEnable network segmentation and encryption (Zero Trust)Define incident response playbooksBuild resilience mechanisms (failover, DR, fault tolerance)Outcome: Systems that are not only secure, but also resilient to failure and attackPhase 5 - Enable Observability and Continuous Improvement: Security and compliance must evolve with the system. Focus areas: Centralize logs, metrics, and tracesCorrelate observability data for threat detectionEstablish feedback loops from operations to developmentContinuously refine policies and controlsOutcome: A closed-loop system where insights drive ongoing improvementFaster detection, response, and optimization Example Technology Enablers Layer Capability Example Tools Application Foundation Infrastructure as Code & Packaging Terraform, Helm Source Control & Artifact Management Git, Artifact Registry CI/CD & Pipeline Automation Jenkins, GitHub Actions, Tekton, ArgoCD Supply Chain & Security Scanning Snyk, Trivy, Dependabot Secrets Management HashiCorp Vault, Kubernetes Secrets, IBM Cloud Secrets Manager Identity & Trust Identity & Access Management (IAM) IAM platforms, Azure AD, IBM Cloud IAM Workload Identity & Zero Trust SPIFFE/SPIRE, Keycloak Authentication & Authorization OAuth/OIDC providers, Keycloak Runtime & Delivery Container & Workload Security Falco, Prisma Cloud, Aqua Deployment & Continuous Delivery Jenkins, ArgoCD, Tekton Network Security & Service Mesh Istio, Linkerd, Service Mesh Configuration & Posture Management CSPM tools (Wiz, Prisma, AWS Config) Observability & Governance Metrics, Logs & Tracing Prometheus, Grafana, OpenTelemetry, Instana Policy Enforcement (Policy-as-Code) OPA, Kyverno Security & Compliance Monitoring Splunk, ELK, Security & Compliance platforms Operational Resilience High Availability & Scaling Kubernetes HPA Disaster Recovery & Backup Velero, IBM Cloud Backup and Recovery Chaos Engineering & Testing Chaos Monkey, Litmus Incident Management PagerDuty, Opsgenie Conclusion Imagine two organizations adopting cloud-native technologies. One continuously responds to security vulnerabilities, operational problems, and compliance needs as they become apparent. The other incorporates security, resilience, and governance through architecture from inception. Over time, the difference becomes clear. One struggles to keep up with change, while the other moves with confidence as security and compliance are no longer separate but inherent capabilities. The proposed 12-factor framework is ultimately about enabling this shift, moving from reactive controls toward secure-by-design and compliant-by-default cloud applications.

By Josephine Eskaline Joyce

CORE

Cloud Cost Optimization Was Hard; AI Cost Optimization Will Be Worse.

For the last decade, cloud cost optimization has been one of the most painful disciplines in enterprise technology. Every CTO, CIO, Head of Engineering, platform leader, and FinOps team knows the story. The cloud made infrastructure faster, more flexible, and more scalable. But it also created a new problem: spending became too easy and unnoticed. An engineer could launch compute in minutes.A team could overprovision storage without realizing it.A forgotten environment could quietly burn money for months.A poorly tagged workload could make cost accountability almost impossible to identify. That was the first era of cloud financial discipline. We learned to manage it through rightsizing, tagging, reserved instances, savings plans, autoscaling, storage lifecycle policies, unit economics, chargeback, showback, and FinOps governance. It was difficult. But compared to AI, traditional cloud cost optimization may look simple. AI is introducing a new cost model that most enterprises are not ready for. And the companies that fail to understand this early will not just overspend. They will struggle to prove AI ROI. The Cloud Cost Problem Was Mostly Infrastructure Visibility Traditional cloud cost problems were usually tied to infrastructure waste. Oversized computeIdle resourcesUnused storageOver-retention of logsPoor environment hygieneLack of ownershipWeak forecastingNo accountability between engineering and finance These problems were hard, but they were measurable (and with the right discipline, they are solvable; I have seen the benefits personally). You could look at CPU utilization.You could identify unattached volumes.You could review storage growth.You could analyze I/O patterns.You could map spend to teams, products, environments, and customers. Cloud costs were complex, but at least the cost drivers were relatively visible. AI changes that. AI cost is not just infrastructure cost. It is the usage cost.It is the token cost.It is GPU cost.It is data cost.It is an experimentation cost.It is a model-selection cost.It is an agent-loop cost.It is an observable cost.It is a governance cost.It is the cost of mistakes made by systems that can now act, not just respond. That is a very different engineering-to-financial problem. The AI Cost Curve Will Surprise Many Enterprises The FinOps Foundation’s 2026 State of FinOps research shows how quickly this shift is happening: 98% of surveyed organizations now manage AI spend, up from 31% two years earlier, and AI cost management is now the number-one skill set FinOps teams need to develop. That is the beginning of a new operating discipline. Gartner has also forecast that worldwide AI spending will reach $2.5 trillion in 2026, with AI-optimized servers growing sharply as enterprises and technology providers build the foundation for AI adoption. McKinsey has estimated that the AI data center buildout alone could require $5.2 trillion in investment by 2030 to meet projected demand. These numbers matter because they point to a simple reality: AI is not just a software feature. AI is becoming an infrastructure economy, and every infrastructure economy eventually faces a cost discipline problem. Why AI Cost Optimization Is Harder Than Cloud Cost Optimization Cloud cost optimization was mostly about resource efficiency. AI cost optimization is about decision efficiency. That distinction matters. In traditional cloud, the question was: “Are we using the right amount of infrastructure for this workload?” In AI, the question becomes: “Are we using the right model, with the right context, for the right task, at the right level of reasoning, with the right data, at the right cost, for the right business outcome?” That is much harder. A simple AI feature can create hidden cost multipliers: A long prompt increases input tokens.A long answer increases output tokens.A large context window increases cost.A reasoning model may consume more compute.An agent may call multiple tools.A failed agent may retry repeatedly.A RAG workflow may increase vector database and storage costs.A poorly designed workflow may call a premium model when a smaller model would work.A high-volume internal assistant may become expensive before anyone connects usage to business value. This is where many organizations will get hurt; not because AI does not work, but because AI works just enough to spread quickly before the cost model is mature. The Real Risk Is Not AI Spend. It Is Unmeasured AI Spend. Spending money on AI is not the problem; unmeasured AI is. A company can justify a high AI bill if it clearly improves revenue, productivity, compliance, reliability, customer experience, or engineering velocity, but many organizations will not have that clarity. They will know the invoice. They will not know the value. That is dangerous. The next generation of AI governance cannot stop at model safety and data privacy. It must include economic governance. Every serious enterprise AI platform will need answers to questions like: Which team is consuming the most AI spend?Which product feature is driving the most token usage?Which customers are creating the highest AI cost-to-serve?Which prompts are inefficient?Which agents are looping?Which models are overpowered for the task?Which workflows should use caching?Which workloads need premium models, and which can use smaller models?Which AI use cases are producing measurable business value? Without this visibility, AI becomes another uncontrolled cloud bill — only faster, more abstract, and harder to explain. The New Discipline: AI FinOps Cloud FinOps brought engineering, finance, and business teams together to manage cloud value. AI FinOps will need to go further. It must connect four layers: Infrastructure economics. GPU usage, compute utilization, storage, networking, inference endpoints, vector databases, model hosting, and cloud-native scaling.Token economics. Input tokens, output tokens, context windows, prompt size, reasoning depth, retry behavior, and agentic tool calls.Application economics. Cost per workflow, cost per customer, cost per ticket, cost per deployment, cost per document processed, cost per support case, or cost per transaction.Business economics. Revenue impact, productivity gain, risk reduction, cycle-time reduction, customer experience improvement, and operational leverage. The companies that master AI FinOps will not be the ones that simply reduce AI spend. They will be the ones that understand which AI spend deserves to grow. That is the maturity shift. Cost optimization should not mean “spend less.” It should mean “spend intelligently.” The Mistake: Treating AI Cost Like a Vendor Invoice Problem Many companies will initially treat AI cost management as a procurement problem. They will negotiate model pricing. They will compare vendors. They will look for cheaper tokens. They will cap usage. They will ask finance to control the bill. That will help, but it will not be enough. The biggest AI cost decisions are not made in procurement, but in architecture. They are made when engineering teams decide: Which model to useHow much context to sendWhether to cache responsesHow agents should retryHow much history to includeHow retrieval should workHow evaluation should gate changesHow observability should track usageHow workflows should fail safely AWS’s Generative AI Lens also frames cost optimization as an architectural discipline, not just a billing exercise. This is the correct direction. AI cost optimization must move left. It has to be designed into the platform. The Next Executive Question For years, executives asked: “What is our cloud spend?” Then the better question became: “What is our cloud spend per product, customer, environment, and business outcome?” Now AI forces a new question: “What is our AI cost per decision, per workflow, per customer, and per unit of business value?” This question will separate mature AI organizations from experimental ones, because AI adoption without cost intelligence is not transformation. It is uncontrolled automation. What Leaders Should Do Now Enterprises do not need to slow down AI adoption, but they do need to stop pretending AI cost can be managed later. The right move is to build the financial control plane early. Start with five actions: Tag and attribute AI usage from day one. Every AI call should be connected to a team, product, environment, use case, and business owner.Measure unit economics. Do not only track total AI spend. Track cost per workflow, per user, per transaction, per ticket, and per successful outcome.Create model-routing standards. Not every task needs the most powerful model. A mature platform should route work across premium models, smaller models, open-source models, cached responses, and deterministic automation.Monitor agent behavior. Agentic systems need cost guardrails. Tool calls, retries, loops, memory usage, and context expansion must be observable.Connect AI spend to business value. If a use case cannot show measurable value, it should not receive unlimited scale. This is not about slowing innovation. It is about preventing AI from becoming the next uncontrolled infrastructure wave. The Future Belongs to Economically Intelligent AI Platforms The first era of cloud rewarded companies that could move fast. The second era rewarded companies that could move fast and control cost. The AI era will reward companies that can move fast, control cost, measure value, and govern autonomous systems. That is a much higher bar. The winners will not be the companies with the most AI pilots. They will be the companies with the strongest AI operating model. They will know what to automate.They will know what not to automate.They will know which models to use.They will know where the money is going.They will know where AI is creating value.They will know when AI is simply creating activity. Cloud cost optimization was hard because cloud made infrastructure consumption easy. AI cost optimization will be worse because AI makes decision consumption easy, and decisions, at enterprise scale, are far more expensive than servers. The next great discipline in technology leadership will be making AI economically sustainable. That is where AI transformation becomes real

By Raghava Dittakavi

CORE

AWS Glue ETL Design Principles for Production PySpark Pipelines

AWS Glue makes it easy to get a PySpark pipeline running quickly. It is significantly harder to build one that stays maintainable as logic grows, performs reliably at scale, and does not quietly accumulate operational debt over time. Most Glue pipelines start simple and become difficult to manage gradually — formulas get hardcoded, modules grow without boundaries, output files proliferate, and before long a single job is doing too many things in ways that are hard to test, hard to debug, and expensive to change. This article presents a set of design principles drawn from production Glue ETL pipelines processing billions of rows. Each principle is independent — you do not need to adopt all of them to benefit from any one. But together they form a coherent approach to building Glue pipelines that are modular, observable, cost-efficient, and built to last. Principle 1: Externalize Logic Into Config, Not Code The single most impactful structural decision in a Glue pipeline is where business logic lives. When formulas, dataset references, column selections, and filter conditions are hardcoded in PySpark, every change requires modifying job code, redeploying, and re-validating the full pipeline. A one-line formula change carries the same deployment risk as a structural refactor. Over time, this creates a strong disincentive to make changes, and the pipeline calcifies. The better pattern is to treat the Spark job as a generic executor and externalize all business-specific declarations into configuration. Formulas are declared as config entries with operands, rounding rules, and output names. Dataset loading behavior — which table, which columns, which filters, whether to cache — is declared per source rather than scripted per job. Schema shapes for complex types are declared explicitly rather than inlined. JSON { "source_table": "headcount_actuals", "database": "finance_db", "select_columns": ["site", "badge_type", "headcount", "fiscal_week"], "filters": [{"column": "is_active", "value": "Y"}], "rename": {"hc_count": "headcount"}, "cache": true } When a new dataset is needed, a new config entry is added — no Spark code changes. When a formula changes, the config entry is updated — no job redeployment required. The job itself becomes stable and generic; only config changes as business requirements evolve. This principle pays increasing dividends over time. Pipelines with externalized logic are faster to modify, safer to deploy, and easier to hand off because the business rules are readable independently of the execution engine. Principle 2: Design Modules With Explicit Boundaries A Glue job that does everything in one place is easy to write and hard to maintain. As pipelines grow, the instinct to add more logic to an existing job accelerates technical debt faster than almost any other decision. The more durable pattern is to decompose computation into modules with explicit input and output contracts. Each module receives one or more DataFrames, applies a focused set of transformations, and produces a named output DataFrame. Modules communicate exclusively through in-memory DataFrame references — there is no disk I/O between stages, no shared mutable state, and no implicit dependency on execution order beyond what the data flow itself requires. Utilities follow the same boundary principle, organized into two layers. Generic pipeline utilities handle cross-cutting concerns — file writing, dataset loading, filtering, deduplication, pivot operations — and are shared across all modules. Module-specific utilities implement transformation logic scoped to a single module and are never invoked outside it. This structure means adding a new module requires only writing its scoped utilities and wiring it into the pipeline. The generic layer is never touched. Existing modules are never at risk from new module development. The downstream benefit is testability. Each module with clean boundaries can be validated independently using mocked PySpark DataFrames with no Glue environment required. Engineers can run pytest locally against individual modules, iterate quickly, and deploy only after local validation passes. Principle 3: Choose Your Job Topology Deliberately A common default in complex pipelines is to split computation across multiple Glue jobs, using S3 as the handoff layer between stages. This is sometimes the right choice — but it should be a deliberate decision, not an instinct. Multi-job topologies make sense when stages have genuinely different compute profiles, when intermediate outputs need to be reused independently by other consumers, or when a stage failure should not force a full recompute from the beginning. In these cases, job separation gives you independent retry boundaries, independent DPU sizing, and the ability to schedule stages on different cadences. Single-job topologies — where the full pipeline runs within one Spark session — make sense when all computation is tightly coupled, modules share the same input datasets, and intermediate outputs have no standalone value. Running everything in one session eliminates cold start overhead for intermediate stages, avoids the cost of serializing data to S3 and deserializing it back between jobs, and keeps the execution model simple to reason about: one trigger, one job, one result. The question to ask is whether the stages truly need to be independent. If intermediate S3 persistence adds coordination complexity without adding value — no independent consumers, no differential retry requirements, no meaningful DPU difference between stages — then collapsing to a single job is usually faster, simpler, and cheaper. If stages have real independence requirements, splitting them is the right call and the operational overhead is justified. Neither topology is inherently superior. The mistake is defaulting to one without evaluating the trade-offs for the specific pipeline at hand. Principle 4: Overlap Writes With Computation When Latency Matters Overlapping writes with computation is a well-established technique in high-performance computing, deep learning training, and heavy database operations. The core idea is to hide the slow latency of I/O operations by running them in the background while the CPU or GPU continues processing data. Rather than waiting for a write to complete before starting the next computation, both proceed simultaneously — I/O latency is absorbed into computation time rather than added on top of it. In Glue ETL pipelines, the same principle applies directly. In a pipeline where multiple output DataFrames are produced, the naive write strategy — complete all computation, then write all outputs sequentially — has two compounding problems. First, it creates a peak memory spike: all computed results are held in memory simultaneously while writes proceed one by one. Second, it serializes work that does not need to be serial: every millisecond spent waiting for S3 acknowledgment is a millisecond the Spark executors are idle. This is worth addressing only when latency is a meaningful constraint. For low-frequency batch jobs running overnight with no user-facing SLA, sequential writes are perfectly adequate. But for pipelines where users or downstream systems are waiting on results — or where job duration directly affects infrastructure cost — overlapping writes with computation delivers measurable wall-clock reduction. The two-phase write strategy implements this directly. Outputs from early modules are written to S3 in background threads immediately after those modules complete, running in parallel with later computation stages. By the time all computation finishes, a significant portion of the output data has already landed in S3. Remaining outputs are then flushed concurrently in a second phase. The implementation leans on Python's concurrent.futures.ThreadPoolExecutor to manage background write threads while the main Spark session continues computation on the driver. A generic write orchestration utility can wrap this pattern so individual modules never need to manage thread lifecycle directly — they simply declare their output and the utility handles scheduling, thread management, and error propagation. Python from concurrent.futures import ThreadPoolExecutor, as_completed def write_phase_a(write_tasks): with ThreadPoolExecutor(max_workers=len(write_tasks)) as executor: futures = {executor.submit(task["fn"], task["df"], task["path"]): task["name"] for task in write_tasks} for future in as_completed(futures): name = futures[future] future.result() logger.info(f"[Phase A] Write complete: {name}") The practical effect is that peak memory pressure is distributed over the job's lifetime rather than concentrated at the end, and total wall-clock time is reduced by the overlap between I/O and CPU-bound computation. For pipelines with many output datasets and a latency SLA to meet, the savings compound significantly. Principle 5: Right-Size Output Files With a Reusable Writer Utility Right-sizing output files is the practice of tuning file sizes to balance disk I/O performance, network transfer speeds, and downstream processing efficiency. Too many small files and downstream readers spend more time on metadata operations and S3 API calls than on actual data reads. Too few large files and parallelism suffers — readers cannot split work efficiently across threads or nodes. The target is consolidated, evenly sized files that match the read patterns of downstream consumers. Spark's default output behavior writes one file per partition, and partition counts are typically tuned for computation throughput rather than output shape. A job optimized for shuffle performance might produce hundreds of partitions, each containing a few megabytes of output data — perfectly reasonable for Spark internals, but harmful for any reader that comes after. This small file problem compounds over time as output partitions accumulate in S3 and the Glue Catalog metadata grows with them. The fix is a reusable writer utility that decouples output file sizing from Spark's internal partition count. Rather than accepting the default, the utility estimates the DataFrame's actual size, calculates the appropriate number of output files for a target file size — typically 128MB to 256MB per file — and coalesces partitions before writing. Python def write_optimized(df, output_path, partition_cols, target_file_size_mb=128): estimated_size_mb = df.rdd.map(lambda row: len(str(row))).sum() / (1024 * 1024) optimal_partitions = max(1, int(estimated_size_mb / target_file_size_mb)) df.coalesce(optimal_partitions) \ .write \ .partitionBy(*partition_cols) \ .parquet(output_path, mode="overwrite") Making this a shared generic utility rather than inline logic in each module has two practical benefits. First, it enforces consistent file sizing behavior across all outputs in the pipeline — no module accidentally writes thousands of tiny files because an engineer forgot to coalesce. Second, it centralizes the tuning knob: when the target file size needs to change — because downstream query patterns shift or a new consumer has different read characteristics — it changes in one place and applies everywhere. Right-sized output files improve Athena scan performance, reduce per-query S3 API costs, keep Glue Catalog partition metadata manageable, and make the output data easier to consume for any downstream system reading from S3. This is a low-effort, high-payoff improvement that applies to virtually every Glue pipeline writing to S3. Principle 6: Use Complex Types to Defer Denormalization SQL-based pipelines are constrained to flat, fully denormalized row structures at every intermediate stage because SQL has no native complex type support. This forces denormalization to happen early, inflating data volume at every subsequent join and aggregation. PySpark has native support for structs, maps, and arrays. Using these types at intermediate stages allows related values to be grouped logically without inflating row counts. A row that would require five denormalized rows in SQL can be represented as a single row with a struct or array column in Spark. Denormalization is then deferred to the final output layer only — applied once, at write time, for consumers that require flat structures. Everything upstream of the final write benefits from reduced volume, fewer shuffles, and faster joins. This principle is particularly impactful in pipelines with multi-level aggregations or wide schemas where dozens of metrics attach to the same dimensional key. Keeping those metrics grouped in a struct until the final output stage reduces the effective row count and join complexity throughout the pipeline. Principle 7: Build Observability Into Every Stage Glue jobs that fail silently or surface errors as opaque stack traces at the end of a long execution are expensive to debug. The investment in step-level observability pays back quickly the first time something goes wrong in production. The minimum viable observability pattern is row count logging at every materialization point. After each module completes and after each write, log the output row count with a descriptive label. This gives a running picture of data volume through the pipeline and makes it immediately obvious when a transformation has dropped rows unexpectedly or produced more rows than expected. Python def log_step(df, step_name): count = df.count() logger.info(f"[{step_name}] Row count: {count:,}") return df Pair this with a try/except/finally pattern at the job level that ensures spark.catalog.clearCache() is always called on exit — whether the job succeeds or fails — to release cached DataFrames and avoid memory leaks across retries. Python try: run_pipeline() except Exception as e: logger.error(f"Pipeline failed: {e}") raise finally: spark.catalog.clearCache() CloudWatch captures all logs automatically. When a job fails, the row count trail shows exactly where in the pipeline the problem occurred, making triage faster and reducing the time between failure and fix. Principle 8: Isolate Executions for Concurrency Pipelines that share compute resources across simultaneous executions create contention that is difficult to predict and expensive to manage. The common response — queue-based serialization — adds operational complexity without solving the underlying resource constraint. AWS Glue's execution model eliminates this problem structurally. Each job execution gets its own isolated DPU allocation. There is no shared compute pool. Ten simultaneous executions consume ten independent DPU allocations and do not interfere with each other in any way. Designing for this means treating each execution as fully independent: no shared state, no cross-execution coordination, no assumption about what other executions are running. Combined with idempotent writes — using overwrite mode so a retry produces the same result as the original execution — the pipeline becomes safe to run concurrently at any scale without additional coordination logic. The cost model reinforces this. Glue bills per DPU-second of actual compute consumed. An execution that takes eight minutes on 240 DPUs costs the same whether it runs alone or alongside a hundred other executions. There is no premium for concurrency and no shared pool to provision for peak load. Putting It Together These eight principles are independent but complementary. A pipeline that applies all of them is modular enough to develop in parallel, observable enough to debug quickly, cost-efficient enough to run at scale, and stable enough to maintain over time without accumulating structural debt. The quickest wins for most existing pipelines are Principles 1, 5, and 7 — externalizing logic into config, right-sizing output files with a shared utility, and adding row count logging at every stage. Each can be applied incrementally without restructuring the full pipeline. The remaining principles become more valuable as pipeline complexity grows and concurrency requirements increase. The underlying thesis is simple: a well-designed Glue pipeline should be easy to change, easy to test, easy to debug, and cheap to run. None of those properties require exotic infrastructure. They require deliberate design decisions applied consistently from the start.

By Janani Annur Thiruvengadam

CORE

Machine Identity Debt: Why Human Identity Is No Longer Cloud Security's Primary Boundary

Cloud-native systems now create far more machine identities than human ones. Security strategies built around workforce identity are no longer sufficient. Here's what engineering leaders should build instead. The Breach That Didn't Need a Password On August 8, 2025, a threat actor now tracked by Google's Threat Intelligence Group as UNC6395 began quietly moving through the Salesforce instances of hundreds of companies. No phishing email landed in an inbox that day. No password was cracked. No multi-factor prompt was bypassed with a fatigue attack. The attacker simply had something better than a password: a valid OAuth token, stolen months earlier from Salesloft's GitHub account, that let it impersonate the Drift chatbot integration and act with all the trust that integration had been granted. Over the following ten days, the group ran automated Salesforce Object Query Language searches against more than 700 organizations — Cloudflare, Zscaler, Palo Alto Networks, and PagerDuty among them — harvesting account records, support case text, and, crucially, the AWS keys and Snowflake tokens that customers had pasted into support tickets months earlier. Google's investigation later found the same stolen tokens had reached into Google Workspace mailboxes too. Cory Michal, CSO at AppOmni, put his finger on what made the campaign notable: it wasn't a single lucky break but a methodical operation against hundreds of tenants using nothing but credentials the tenants themselves had issued to a vendor they trusted. That's the detail worth sitting with. Every access control that companies had built around human identity — MFA, conditional access, session monitoring, SSO — was irrelevant to this attack, because no human ever logged in. The identity that mattered was a machine's, and almost nobody was watching it the way they watch people. This wasn't an isolated case. In the same twelve months, a compromised API key issued to a DOE staffer gave a stranger standing access to more than 50 large language models at xAI — and stayed active for days after the exposure was discovered, according to reporting from KrebsOnSecurity. A supply-chain attack against the widely used tj-actions/changed-files GitHub Action, relied on by over 23,000 repositories, scraped AWS keys, GitHub tokens, npm credentials, and private RSA keys directly out of CI/CD workflow logs. GitGuardian's 2026 State of Secrets Sprawl report counted 28.65 million new hardcoded secrets pushed to public GitHub repositories in 2025 alone — a 34% jump year over year — and found that AI-assisted commits leak secrets at roughly twice the baseline rate of human-written ones. None of these incidents required a zero-day. They required an organization to have created a machine identity, granted it access, and then stopped paying attention to it. That is now the default failure mode of cloud security — and it's a failure mode that identity programs built for humans were never designed to catch. Section 1: Identity Has Already Changed Underneath Us For most of the last two decades, "identity and access management" meant managing people: employees, contractors, customers. A person logged in, proved who they were, and was granted access based on their role. The infrastructure existed to serve human judgment. That model quietly stopped matching reality. In a modern cloud environment, the majority of authentication events aren't between a person and a system — they're between systems. A pod in a Kubernetes cluster calls another pod. A CI/CD pipeline authenticates to a cloud provider to deploy an artifact. A SaaS integration holds an OAuth token that lets it act on a company's behalf indefinitely. Each of these is an identity in every meaningful sense — it can be granted permissions, it can be revoked, it can be stolen — but almost none of them are managed with the rigor applied to a human employee's badge. The mechanisms behind this shift are now familiar to anyone running production infrastructure: Kubernetes service accounts that authenticate workloads to the API server, workload identity federation that lets a pod assume a cloud IAM role without a stored credential, SPIFFE and SPIRE issuing cryptographically verifiable identities to workloads at runtime, OAuth client-credential grants powering service-to-service calls, and service meshes like Istio wrapping every internal request in mutual TLS. Layer on top of that the identities created by CI/CD systems, and it becomes clear that a mid-sized cloud environment can easily contain ten or twenty machine identities for every human one. Security researchers at IDMWorks, reviewing the identity breaches of the last three years for their 2026 NHI Reality Report, described the pattern bluntly: these attacks succeeded through poor governance, not sophisticated malware. There was no payload to detect — just valid credentials doing exactly what valid credentials are allowed to do. That's a much harder thing to catch than a virus, because there's nothing anomalous about the code path. The only thing that's wrong is which entity is walking it. Section 2: Why the Existing Security Model Fails Here Identity and access management built for people assumes a handful of things that simply don't hold for machines. It assumes credentials are issued to a known, accountable owner. It assumes a login event is rare enough to be worth alerting on. It assumes a compromised credential will eventually show up in unusual behavior — an impossible-travel alert, an after-hours login, a new device. None of that transfers cleanly to a service account. IDMWorks' research is direct about the resulting blind spot: a service account that authenticates ten thousand times a day isn't behaving anomalously — that's just Tuesday. Detecting misuse requires knowing what a credential is supposed to be doing well enough to notice a deviation, and almost no organization has that baseline built for its non-human identities the way it does for its people. The ownership problem compounds this. Aembit's running catalog of non-human identity breaches documents a 2025 flaw in a major identity provider that let anyone holding a valid API key enumerate every OIDC application in a tenant and pull its client secrets — a bug that, if exploited, would have let an attacker impersonate entire applications and move laterally across an organization's stack. It was responsibly disclosed and patched, but it illustrates how identity providers themselves can become breach multipliers the moment a machine credential leaks. Then there's lifespan. GitGuardian's research, cited in Snyk's 2026 analysis of the secrets sprawl problem, found that private repositories are six times more likely to contain hardcoded secrets than public ones — largely because private repos get cloned, forked, and handed to contractors without anyone revisiting what's inside them. And because git's data model is append-only, a secret committed and later deleted in a follow-up commit should still be treated as exposed; it lives on in history whether or not it's still visible in the latest diff. The legal exposure is no longer theoretical, either. In United States v. Sullivan, Uber's former Chief Security Officer was criminally convicted of obstruction of justice for concealing a 2016 breach that began with hardcoded AWS credentials sitting in a GitHub repository — credentials that let attackers pull data on 57 million riders and drivers. The Ninth Circuit's 2025 ruling upheld that conviction, establishing that executives can face personal criminal liability for how they respond to a credential-based breach, not just for the breach itself. That should recalibrate how seriously engineering leadership treats "just another leaked API key." An Honest Name for the Problem: Machine Identity Debt Engineering teams already have a vocabulary for the gap between "shipped quickly" and "built correctly" — they call it technical debt. There's no equivalent term for the identical pattern happening in identity, so let me propose one: machine identity debt. Technical debt accumulates in code: shortcuts taken under deadline pressure that someone eventually has to pay down. Identity debt accumulates in trust: every API key issued and never revisited, every OAuth grant approved by someone who's since left the company, every IAM role created with "just give it admin, we'll fix it later" and never fixed. None of it shows up in a sprint retro. None of it fails a build. It just sits there, compounding, until an attacker finds it and collects the interest all at once — which is close to a literal description of what happened to the 700-plus organizations caught in the Salesloft Drift breach, where OAuth grants approved months or years earlier turned out to still carry far more reach than anyone had tracked. A rough way to think about what's accumulating: Plain Text Machine Identity Debt ≈ long-lived credentials with no expiration policy + service accounts no longer tied to an active workload + OAuth grants no one has reviewed since approval + secrets discovered in tickets, chat, and docs rather than a vault + IAM roles scoped broader than the task requires + any machine identity with no accountable human owner This isn't a precise formula you can drop into a dashboard query today — treat it as a checklist for a conversation, not a KPI. But naming each line item is useful, because each one is independently measurable, and most organizations have never measured any of them. When enough of this debt accumulates that nobody can produce an accurate answer to "what machine identities exist, who owns them, and what can they reach" — that's not an IAM maturity gap anymore. It's identity bankruptcy: the point where inventory, ownership, and trust have diverged so far from reality that incremental cleanup stops being realistic and the organization needs a forced reconciliation, usually triggered by an incident rather than a planning cycle. The mechanism that gets organizations there is worth naming too. Every new SaaS integration, every GitHub Action, every Terraform module, every AI agent granted API access mints a new unit of trust — a new thing the organization implicitly promises to govern. Nobody budgets for governing it; the integration just gets approved because it unblocks a project. Multiply that across a growing stack and you get something like trust inflation: the total quantity of trust an organization has extended growing faster than its ability to actually track or revoke any single unit of it. Eventually a credential's nominal access — what the ticket said it was for — and its real access — everything it can actually still reach — drift far enough apart that the gap itself becomes the attack surface. None of these terms are industry standard — I'm proposing them here because the pattern needed a name and didn't have one. Judge them by whether they make the problem easier to talk about, not by whether you've heard them before. Section 3: A New Boundary — Adaptive Machine Trust Architecture If the perimeter used to be defined by "who logged in," it now has to be defined by a different question: can this specific workload be trusted, right now, to do the specific thing it's asking to do? That's a shift from identity as a static credential to identity as a continuously re-evaluated claim. A workable framework for this — call it Adaptive Machine Trust Architecture, or AMTA — rests on a small number of principles that reinforce each other: Continuous verification. A workload's identity is checked at the moment of each request, not once at startup. Trust isn't a badge you're handed at the door; it's re-earned per transaction. Cryptographic workload identity. Instead of a static API key sitting in an environment variable, a workload is issued a short-lived, cryptographically verifiable identity document — the SPIFFE Verifiable Identity Document (SVID) model is the clearest existing implementation of this idea — that ties the identity to what the workload is, not to a secret it happens to be holding. Just-in-time authorization. Access is granted for the duration of a task and expires automatically, rather than being provisioned once during a rushed deployment and left in place indefinitely, which is precisely the pattern IDMWorks identified as the root cause of most CI/CD credential compromises. Policy-driven trust decisions. Authorization decisions are externalized to a policy engine that can evaluate context — the requesting workload's identity, its recent behavior, the sensitivity of the resource — rather than being baked into application code as a hardcoded allow-list. Identity lifecycle management. Every machine identity has a documented owner, a defined purpose, and an expiration path. The absence of exactly this — what IDMWorks calls "no ownership model" — is the single most commonly cited root cause across the non-human identity breaches of the last three years. Continuous attestation. The system periodically re-proves that a workload is still what it claims to be — still running the expected code, in the expected environment — rather than trusting a credential indefinitely once it's issued. None of these principles is exotic on its own. What's new is treating them as a single coherent architecture for machine trust, instead of a scattered collection of best practices that get implemented inconsistently across teams. Section 4: What Implementation Actually Looks Like The tooling to build this exists today, and it's more mature than most security teams realize. SPIFFE and its reference implementation, SPIRE, provide the identity layer: workloads receive short-lived X.509 or JWT SVIDs based on attested properties of the environment they're running in — the specific pod, the specific node, the specific Kubernetes namespace — rather than a secret baked into a config file. A workload requesting an SVID doesn't present a password; it presents proof of what it is, and SPIRE's server verifies that against a registration policy before issuing anything. In a service mesh like Istio, this identity layer can be paired with mutual TLS enforced at the sidecar proxy, so every service-to-service call is authenticated and encrypted without the application code needing to know anything about certificates. Authorization decisions can be externalized to Open Policy Agent, letting teams write access policy as code — reviewable, versioned, testable — instead of scattering if user.role == 'admin' checks through a codebase. For software supply chain integrity — relevant given that the tj-actions/changed-files compromise spread through a CI/CD pipeline — Sigstore's Cosign and Fulcio provide a way to sign build artifacts and verify their provenance using short-lived certificates tied to an OIDC identity, rather than a long-lived signing key that itself becomes another secret to protect. None of this is a rip-and-replace project. Teams typically start by identifying their highest-value machine credentials — the ones with production database access, the ones with broad cloud IAM permissions — and migrating those first to short-lived, attested identities, while instrumenting logging so that every machine identity's access can actually be reviewed rather than assumed. What This Looks Like When Someone Actually Ships It The architecture described above isn't hypothetical. Pinterest has publicly documented using SPIFFE alongside its internal secrets-management system, Knox, specifically to solve identity in a multi-tenant environment where workloads from different teams share infrastructure and can't be trusted by network location alone. Square presented its adoption of SPIFFE and SPIRE at a SPIFFE Community Day, describing how it used the framework to secure communication across a hybrid infrastructure — cloud and on-premises systems that previously had no consistent way to authenticate to each other. Uber's security team gave a KubeCon talk walking through why it built an internal workload identity platform on these same principles, and ByteDance has separately documented replacing a homegrown certificate system with SPIRE to get PKI-based authentication working at the scale TikTok's infrastructure requires. The common thread across all four is the same one this piece has argued from the incident side: none of them adopted workload identity because a compliance checkbox required it. They adopted it because operating at their scale made network-location-based trust and long-lived shared secrets genuinely unworkable — the same pressure that's now reaching far smaller organizations as their own machine identity counts climb. The trade-off they all had to work through in public is worth naming honestly: SPIRE introduces real operational overhead — a server and agent fleet to run, node and workload attestation to configure correctly for each hosting environment, and a learning curve for teams used to thinking about secrets rather than attested identity. None of the public talks describe it as a drop-in replacement. They describe it as an infrastructure investment that pays off once the number of services and the rate of change outgrow what static credentials can manage safely. Section 5: The Metrics That Actually Indicate Progress Security leaders asking for budget need numbers, not architecture diagrams. The ones worth tracking: Mean credential lifetime – how long, on average, does a machine credential remain valid before rotation or expiration? GitGuardian's finding that some leaked keys remained live for months is really a mean-lifetime failure.Percentage of workloads using attested workload identity versus static, long-lived secrets – this is the single clearest proxy for how exposed an environment is to the failure pattern behind the xAI and tj-actions incidents.Secret rotation frequency, measured against an actual policy rather than an aspirational one.Unauthorized service-to-service request rate – a signal that requires the behavioral baselining IDMWorks flagged as largely absent today.Credential exposure rate in code, tickets, and chat – Snyk's research found leaks occurring in Slack messages, Jira tickets, and Confluence pages at meaningful rates, not just in source code, so this metric has to look beyond the repository.Policy compliance rate for third-party OAuth integrations – the exact control gap that let the Salesloft Drift tokens retain broad, long-lived access to Salesforce, Google Workspace, and AWS simultaneously. The Reports Keep Saying the Same Thing Independently It's worth pausing on how many separate organizations, using separate datasets, landed on the same conclusion in the same twelve-month window. Verizon's 2025 Data Breach Investigations Report — built from 22,052 incidents across 139 countries, the kind of dataset no single vendor could assemble on its own — found 441,780 exposed secrets sitting in public code repositories, with a median remediation time of 94 days once discovered. Nearly half of those were high-privilege Google Cloud API keys tied to automated infrastructure, not human logins. GitGuardian's own 2026 research, working from a different pipeline entirely, arrived at the same order of magnitude: 28.65 million new hardcoded secrets added to public GitHub in 2025 alone. IDMWorks, analyzing three years of non-human identity incidents rather than scanning code, described the same underlying failure in different language: no ownership model, no rotation cadence, detection tooling built for human login patterns that generates nothing but noise against machine behavior. Snyk's research adds the vector most of these reports don't emphasize enough — over a quarter of credential incidents originate entirely outside source code, in Slack messages, Jira tickets, and Confluence pages. Different data sources, different methodologies, different commercial incentives — and all of them converge on the same sentence: non-human identities are growing faster than the governance built to manage them. That's not a marketing claim from any one vendor. It's what independent datasets keep saying when you line them up next to each other. What the Trajectory Actually Implies I won't pretend to know the machine-to-human identity ratio a cloud-native enterprise will have in 2030 — nobody has the longitudinal data to state that number with confidence, and treating a guess as a fact would undercut everything else in this piece. What can be said with more confidence is the direction and the reason. Every driver behind today's machine identity growth — CI/CD automation, service mesh adoption, multi-cloud workload identity, and now AI agents authenticating to APIs on an organization's behalf — is accelerating, not leveling off. AI agents in particular are a new category of machine identity, not just more volume in an existing one: an agent can be granted a credential, use it in ways its creator never explicitly authorized, and, in the case of the Common Crawl training-data exposure, potentially reproduce a credential it was never supposed to have seen in the first place. If the ratio of machine to human identities is already in the double digits at a typical mid-sized cloud shop today, as the SPIFFE/SPIRE and workload-identity adoption patterns suggest, then adding an autonomous-agent layer on top doesn't nudge that ratio — it compounds it. The honest prediction isn't a specific number for 2030. It's that any organization treating machine identity governance as a 2026 problem to revisit later is already behind a curve that isn't slowing down. A Rough Map of How Organizations Get Here Most organizations don't leap from careful to reckless. They drift through recognizable stages, usually without anyone deciding to: Plain Text Centralized human IAM ↓ Cloud IAM roles multiply per service ↓ Service accounts proliferate, ownership blurs ↓ Workload identity adopted for some, not all, systems ↓ AI agents added as a new identity class ↓ Identity sprawl outpaces any team's ability to inventory it ↓ Machine Identity Debt crosses into Identity Bankruptcy ↓ Incident forces the reconciliation that governance should have Most organizations reading this are somewhere between stage two and stage five. Very few have consciously decided which stage they're in — which is itself the point: nobody plans to reach identity bankruptcy; they just never stop to check how much debt they've taken on since the last audit. Section 6: Where This Goes Next A few developments will make machine trust an even sharper problem over the next few years rather than a solved one. Confidential computing — running workloads inside hardware-enforced trusted execution environments — is moving from research curiosity to something cloud providers offer as a standard instance type, which will let attestation extend down to the hardware layer rather than stopping at the software identity. AI agents that authenticate to APIs and take autonomous action on an organization's behalf are a new and rapidly growing category of machine identity, and the data-poisoning risk is already visible: Truffle Security's scan of Common Crawl's December 2024 archive, covering roughly 400 terabytes of public web data, found close to 12,000 live, working credentials embedded in text that's now part of the training data feeding future models. An AI system trained on that data can, in principle, reproduce or act on a credential it was never supposed to have. Post-quantum cryptography considerations will eventually reach workload identity systems, since the SVIDs and certificates underpinning frameworks like SPIFFE rely on cryptographic assumptions that are being reassessed industry-wide. And identity graphs — mapping which machine identities can reach which resources, and through which chains of trust — are becoming the tool that lets a security team answer the question that mattered most in the Salesloft Drift breach: not "was this OAuth token valid," but "what could this token reach, and did anyone actually decide it should be able to?" The Question Worth Asking The organizations that got hit in 2025 weren't running unpatched software or ignoring known vulnerabilities. Cloudflare, Palo Alto Networks, and Zscaler — security vendors with mature programs — were among the hundreds caught in the Salesloft Drift breach. The tokens that got them were valid. The access was, technically, authorized. That's what makes machine identity the harder problem: it doesn't fail loudly. The practical shift for engineering leadership is to stop asking "who is the user?" as the primary security question and start asking "can this workload be trusted right now, for this specific action?" That means building ownership records for every service account before an incident forces the question, migrating high-value credentials to short-lived attested identities before a leaked key becomes a header on KrebsOnSecurity, and treating third-party OAuth grants with the same scrutiny given to a new employee's laptop. None of this is speculative. Every incident cited here happened in the past eighteen months, to organizations with real security budgets. The architecture to prevent the next one already exists. What's missing, in most companies, is the decision to build it before the postmortem forces the issue. The pattern underneath every breach in this piece is the same one: a credential nobody was actively watching, doing exactly what it was built to do, for whoever happened to be holding it. Passwords get the attention because a stolen password is a story people understand — a human made a mistake, or got tricked. A stolen service-account token is a harder story to tell, because the mistake happened months earlier, in a decision nobody remembers making, and the debt just sat there accruing until someone else cashed it in. Paying that debt down before it's due is a less dramatic project than responding to a breach. It's also the only version of this problem that ends with a postmortem you never have to write. Sources Google Cloud / Google Threat Intelligence Group, "Widespread Data Theft Targets Salesforce Instances via Salesloft Drift," August 26, 2025The Hacker News, "Salesloft OAuth Breach via Drift AI Chat Agent Exposes Salesforce Customer Data," August 28, 2025Anomali, "Reviewing the Salesforce–Salesloft Drift OAuth Supply Chain Breach," December 2025Guardz, "The Salesloft Drift Breach and the Impact on Google Workspace," September 2025Defakto, "xAI API Key Leak by DOGE Staffer Reveals Cracks in API Security," December 2025Snyk, "Why 28 million credentials leaked on GitHub in 2025, and what to do about it," March 2026Aembit, "Real-Life Examples of Non-Human Identity Security Breaches," updated regularlyIDMWorks, "When Service Accounts Attack: How Identities are Weaponized," May 2026PointGuard AI, "AI Training Data Secret Leak 2025 | 12,000 API Keys Exposed," January 2026CybelAngel, "API Threat Report 2025: Key Findings for Security Teams," March 2026United States v. Sullivan, 9th Cir. 2025 (referenced via Snyk's legal-consequences analysis, above)Verizon, "2025 Data Breach Investigations Report" (18th edition; 22,052 incidents, 139 countries)GitGuardian, "The Secrets Sprawl is Worse Than You Think: Key Takeaways from the 2025 Verizon DBIR," April 2025SPIFFE Project, "Case Studies" (Pinterest, Square, Uber, ByteDance talks)

By Igboanugo David Ugochukwu

CORE

Cloud Architecture

DZone's Featured Cloud Architecture Resources

Monthly Top Cloud Architecture Experts

The Latest Cloud Architecture Topics