Coding

Why Query Optimization Matters A Spark query written by a human and a Spark query executed by the engine are often very different things. The gap between them — the optimization — is what separates a job that runs in 3 minutes from one that runs in 3 hours on identical hardware. Databricks compounds Spark's native Catalyst optimizer with two additional layers: Adaptive Query Execution (AQE) – re-optimizes the query at runtime using actual statistics collected mid-jobPhoton – a C++ vectorized execution engine that replaces the JVM-based Spark executor for eligible operators Understanding all three lets you write queries that cooperate with the engine rather than fight it. The Catalyst Optimizer Pipeline Catalyst is Spark's rule-based and cost-based query optimizer. Every query — whether written in SQL, DataFrame API, or Dataset API — passes through the same four-stage pipeline before a single byte of data is read. Stage 1: Parsing — From SQL to Unresolved Logical Plan Python # ── Catalyst Stage 1: Parsing ───────────────────────────────────────────────── # Spark uses ANTLR4 to parse SQL into an Abstract Syntax Tree (AST). # At this point column names are NOT validated — the plan is "unresolved". from pyspark.sql import SparkSession spark = SparkSession.builder.appName("catalyst-demo").getOrCreate() # Both of these produce identical internal representations df_api = ( spark.table("prod.silver.events_clean") .filter("event_type = 'purchase'") .groupBy("platform") .agg({"revenue": "sum"}) ) sql_api = spark.sql(""" SELECT platform, SUM(revenue) AS total_revenue FROM prod.silver.events_clean WHERE event_type = 'purchase' GROUP BY platform """) # Inspect the unresolved logical plan (before analysis) df_api.explain(mode="formatted") # Output includes: # == Parsed Logical Plan == # 'Aggregate ['platform], ['platform, unresolvedAlias('sum('revenue), None)] # +- 'Filter ('event_type = 'purchase) # +- 'UnresolvedRelation [prod, silver, events_clean] The key insight here: UnresolvedRelation and unresolvedAlias mean Spark hasn't touched the catalog yet. Column names could be typos at this point and Catalyst doesn't know. Stage 2: Analysis — Binding to the Catalog The Analyzer walks the unresolved AST and looks up every relation and attribute against the Catalog (in Databricks, this is Unity Catalog). It resolves column names, infers data types, validates references, and binds functions. Python # ── Catalyst Stage 2: Analysis ──────────────────────────────────────────────── # After analysis, every column is resolved to a specific attribute with a type. # AnalysisException is thrown HERE if a column doesn't exist. from pyspark.sql import functions as F from pyspark.sql.utils import AnalysisException # Example of what Analysis catches: try: spark.table("prod.silver.events_clean") \ .select("nonexistent_column") \ .show() except AnalysisException as e: print(f"Analysis failed: {e}") # → AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] # A column or function parameter with name `nonexistent_column` cannot be resolved. # After successful analysis, inspect the resolved plan df = ( spark.table("prod.silver.events_clean") .filter(F.col("event_type") == "purchase") .select("platform", "revenue", "user_id") ) # The analyzed plan shows fully qualified attribute IDs like: # == Analyzed Logical Plan == # platform: string, revenue: double, user_id: string # Project [platform#42, revenue#67, user_id#31] # +- Filter (event_type#39 = purchase) # +- Relation prod.silver.events_clean[...] parquet print(df._jdf.queryExecution().analyzed()) Stage 3: Logical Optimization — Rule-Based Rewrites This is where Catalyst applies its ~100+ built-in rules to produce an equivalent but cheaper logical plan. Rules fire repeatedly in fixed-point iteration until the plan stabilises. Python # ── Catalyst Stage 3: Key Optimization Rules ────────────────────────────────── # RULE 1: Predicate Pushdown # Catalyst moves filters as close to the data source as possible, # so Spark reads fewer rows from Parquet. df_before = ( spark.table("prod.silver.events_clean") .join( spark.table("prod.silver.users_clean"), on="user_id" ) .filter(F.col("event_type") == "purchase") # ← filter AFTER join ) # Catalyst rewrites this internally as if you wrote: df_after_equivalent = ( spark.table("prod.silver.events_clean") .filter(F.col("event_type") == "purchase") # ← filter BEFORE join .join( spark.table("prod.silver.users_clean"), on="user_id" ) ) # Result: potentially millions fewer rows shuffled during the join # RULE 2: Column Pruning # Catalyst removes columns not needed by downstream operators. # Even if you select(*), Spark will only read the columns it needs. df_pruned = ( spark.table("prod.silver.events_clean") .select("*") .filter(F.col("event_type") == "purchase") .groupBy("platform") .agg(F.sum("revenue").alias("total_revenue")) ) # Internally, Catalyst prunes all columns except: event_type, platform, revenue # RULE 3: Constant Folding # Expressions with only literals are evaluated at plan time, not per-row. df_constants = spark.range(1000).select( F.lit(2 + 3 * 4).alias("always_14"), # folded to Literal(14) at plan time F.col("id") * F.lit(1).alias("same_id"), # simplified to just col("id") ) # RULE 4: Boolean Simplification # AND/OR chains with tautologies or contradictions are collapsed df_simplified = spark.range(100).filter( (F.col("id") > 10) & F.lit(True) # simplified to just (col("id") > 10) ) # See all optimizations applied: print(df_pruned._jdf.queryExecution().optimizedPlan()) Stage 4: Physical Planning — Strategies and Cost Models The physical planner maps each logical operator to one or more physical implementations and selects the best one using a cost model. The most impactful decision here is join strategy selection. Python # ── Catalyst Stage 4: Physical Planning & Join Strategies ──────────────────── # JOIN STRATEGY 1: Broadcast Hash Join (BHJ) # Best when one side is small enough to fit in executor memory. # No shuffle — the small table is broadcast to all workers. spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "10mb") # default large_df = spark.table("prod.silver.events_clean") # 500GB small_df = spark.table("prod.gold.product_catalog") # 8MB ← will be broadcast result_bhj = large_df.join(small_df, on="product_id") # BHJ auto-selected # Force BHJ with a broadcast hint (overrides threshold check): from pyspark.sql.functions import broadcast result_forced = large_df.join(broadcast(small_df), on="product_id") # JOIN STRATEGY 2: Sort Merge Join (SMJ) # Default for large-large joins. Both sides are sorted and merged. # Requires a full shuffle — expensive but handles any size. spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1") # disable BHJ large_df2 = spark.table("prod.silver.transactions_clean") # 200GB result_smj = large_df.join(large_df2, on="user_id") # SMJ selected # JOIN STRATEGY 3: Shuffle Hash Join (SHJ) # Hash-based, no sort. Chosen by AQE when one side is much smaller # than the other but still above the broadcast threshold. spark.conf.set("spark.sql.join.preferSortMergeJoin", "false") # WHOLE-STAGE CODEGEN: Spark fuses multiple operators into a single # Java function to avoid virtual dispatch overhead and intermediate objects. # Verify it's active in your plan: spark.conf.set("spark.sql.codegen.wholeStage", "true") # default result_bhj.explain(mode="formatted") # Look for: *(1) BroadcastHashJoin — the *(N) prefix = WholeStageCodegen stage N Adaptive Query Execution (AQE) AQE is Databricks' most impactful runtime optimization layer. It materializes shuffle map output statistics at shuffle boundaries and uses them to make three key decisions after data has been partially processed. Python # ── AQE Configuration ───────────────────────────────────────────────────────── # AQE is ON by default in Databricks Runtime 7.3+ spark.conf.set("spark.sql.adaptive.enabled", "true") # 1. Dynamic Partition Coalescing # Merges small post-shuffle partitions to avoid thousands of tiny tasks spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true") spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "128mb") spark.conf.set("spark.sql.adaptive.coalescePartitions.minPartitionNum", "1") # 2. Dynamic Join Strategy Switching # Allows AQE to downgrade SMJ → BHJ at runtime if a side turns out small spark.conf.set("spark.sql.adaptive.localShuffleReader.enabled", "true") # AQE broadcast threshold (can be higher than static threshold since # we now KNOW the actual size) spark.conf.set("spark.sql.adaptive.autoBroadcastJoinThreshold", "30mb") # 3. Skew Join Optimization # Splits oversized partitions and replicates the non-skewed side spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true") spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionFactor", "5") # 5x median spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes", "256mb") # Verify AQE decisions in the query plan: df = ( spark.table("prod.silver.events_clean") .join(spark.table("prod.silver.users_clean"), on="user_id") .groupBy("platform") .agg(F.sum("revenue").alias("total")) ) df.explain(mode="formatted") # Look for: AdaptiveSparkPlan isFinalPlan=true # and: == Final Physical Plan == (shows post-AQE decisions) The Photon Engine Photon is Databricks' native vectorized query engine written in C++. It replaces the JVM-based Spark executor for eligible operations, processing data in column-oriented batches (vectors) rather than row-by-row. Python # ── Photon Configuration & Verification ─────────────────────────────────────── # Photon is available on Databricks Runtime 9.1+ with Photon-enabled clusters. # Enable it at the cluster level (UI: Cluster > Configuration > Enable Photon) # or via config: spark.conf.set("spark.databricks.photon.enabled", "true") # Photon-accelerated operators (as of DBR 13.x): # ✅ Scan (Parquet, Delta) ✅ Filter / Project # ✅ Hash Aggregate ✅ Sort # ✅ Broadcast Hash Join ✅ Sort Merge Join # ✅ Window functions ✅ Union / Expand # ✅ String functions ✅ Math functions # ❌ UDFs (Python/Scala) ❌ Some complex types # ❌ Streaming (partial) ❌ RDD-based operations # Verify Photon is executing your query: df = spark.sql(""" SELECT platform, DATE_TRUNC('month', event_ts) AS month, SUM(revenue) AS total_revenue, COUNT(DISTINCT user_id) AS unique_buyers, AVG(revenue) AS avg_order_value FROM prod.silver.events_clean WHERE event_type = 'purchase' AND event_ts >= '2024-01-01' GROUP BY platform, DATE_TRUNC('month', event_ts) ORDER BY month DESC, total_revenue DESC """) df.explain(mode="formatted") # Look for operators prefixed with "Photon" in the physical plan: # == Physical Plan == # PhotonResultStage # +- PhotonSort [month DESC NULLS LAST, total_revenue DESC NULLS LAST] # +- PhotonShuffleExchangeSink hashpartitioning(platform, month) # +- PhotonGroupingAgg [platform, month], [sum(revenue), count(user_id), avg(revenue)] # +- PhotonFilter (event_type = purchase AND event_ts >= 2024-01-01) # +- PhotonScan parquet prod.silver.events_clean # Photon performance metrics appear in Spark UI under "Photon Metrics": # - Photon scan time # - Photon total compute time # - Rows processed by Photon vs fallback JVM Reading Explain Plans The explain(mode="formatted") output is your primary debugging tool. Here's how to read it efficiently: Python # ── Explain Plan Modes ──────────────────────────────────────────────────────── df = ( spark.table("prod.silver.events_clean") .filter(F.col("event_type") == "purchase") .join(broadcast(spark.table("prod.gold.product_catalog")), on="product_id") .groupBy("platform", "category") .agg( F.sum("revenue").alias("total_revenue"), F.count("*").alias("transaction_count") ) ) # Mode 1: simple (default) — compact tree df.explain() # Mode 2: extended — all 4 plan stages side by side df.explain(mode="extended") # Mode 3: formatted — human-readable with operator details (RECOMMENDED) df.explain(mode="formatted") # Mode 4: cost — includes estimated row counts and sizes (requires ANALYZE TABLE) df.explain(mode="cost") # Mode 5: codegen — shows generated Java code for WholeStageCodegen df.explain(mode="codegen") # ── Key Signals to Look For ─────────────────────────────────────────────────── # ✅ GOOD signs: # *(N) prefix → WholeStageCodegen active (operators fused) # BroadcastHashJoin → small table correctly broadcast, no shuffle # PhotonXxx → Photon accelerating this operator # AdaptiveSparkPlan → AQE is engaged # PartitionFilters → Delta/Parquet file skipping active # PushedFilters → filters pushed to Parquet reader # ❌ WARNING signs: # Exchange (shuffle) → unexpected shuffle (missing broadcast hint?) # SortMergeJoin → large-large join (may need Z-ORDER or AQE tuning) # HashAggregate x2 → partial + final agg = shuffle involved # CartesianProduct → missing join condition! Will OOM on large tables # ObjectHashAggregate → non-codegen path, JVM overhead # GenerateXxx → explode() or similar, can't be fused # ── ANALYZE TABLE: feed statistics to CBO ───────────────────────────────────── # Without stats, Catalyst uses default estimates (1M rows, 8 bytes/col). # Run ANALYZE to give the Cost-Based Optimizer real numbers. spark.sql("ANALYZE TABLE prod.silver.events_clean COMPUTE STATISTICS") spark.sql(""" ANALYZE TABLE prod.silver.events_clean COMPUTE STATISTICS FOR COLUMNS user_id, event_type, platform, revenue """) # Now explain(mode="cost") shows real row counts and sizes Tuning Reference Table A quick-reference guide for the most impactful Spark/Databricks configs, what they control, and when to change them: Config KeyDefaultWhat It ControlsWhen to Tunespark.sql.adaptive.enabledtrueMaster AQE switchKeep on; only disable for debuggingspark.sql.adaptive.advisoryPartitionSizeInBytes64mbTarget post-coalesce partition sizeIncrease to 128mb–256mb for large shufflesspark.sql.adaptive.skewJoin.enabledtrueAQE skew splitKeep on; tune skewedPartitionFactor if neededspark.sql.autoBroadcastJoinThreshold10mbStatic BHJ thresholdIncrease to 50mb–100mb if executor memory allowsspark.sql.adaptive.autoBroadcastJoinThreshold30mbAQE runtime BHJ thresholdIncrease if AQE isn't catching small tablesspark.sql.shuffle.partitions200Default shuffle partition countSet to 8 × num_cores for your clusterspark.sql.files.maxPartitionBytes128mbMax bytes per Parquet read partitionReduce for high-parallelism scansspark.databricks.photon.enabledtruePhoton vectorized engineKeep on; disable only for UDF-heavy jobsspark.sql.codegen.wholeStagetrueWhole-Stage CodeGen fusionKeep on; disable only for debuggingspark.sql.statistics.histogram.enabledfalseColumn histograms for CBOEnable after running ANALYZE TABLEspark.sql.cbo.enabledtrueCost-Based OptimizerKeep on; requires ANALYZE TABLE to be usefulspark.databricks.delta.optimizeWrite.enabledtrueAuto bin-pack write filesKeep on for all Delta writes Key Takeaways Catalyst has four stages: Parse → Analyze → Optimize → Plan. Each stage has a distinct job, and understanding them tells you exactly where to look when a query misbehaves.Predicate pushdown and column pruning are the two most impactful automatic optimizations — they reduce the data volume Spark has to move before any aggregation or join.AQE is not a set-and-forget feature: tune advisoryPartitionSizeInBytes to your actual data sizes, and verify its decisions with explain(mode="formatted") — look for AdaptiveSparkPlan isFinalPlan=true.Photon drops in transparently for most SQL and DataFrame operations. The exceptions are Python UDFs, RDD operations, and some complex types — refactor these away from hot paths.Run ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS on your most-joined tables. The CBO's join ordering and strategy decisions improve dramatically with real statistics vs. default estimates.explain(mode="formatted") is your most important debugging tool — learn to read it before reaching for cluster config changes. References Apache Spark — Catalyst Optimizer (Deep Dive Paper, Armbrust et al., SIGMOD 2015)Databricks — Adaptive Query ExecutionApache Spark Docs — Adaptive Query ExecutionDatabricks — Photon RuntimeDatabricks Blog — Photon: A Fast Query Engine for Lakehouse SystemsDatabricks — Cost-Based OptimizerApache Spark — Performance Tuning GuideDatabricks — Broadcast Join Hints"Photon: A Fast Query Engine for Lakehouse Systems" (Behm et al., SIGMOD 2022)Spark by Examples — Explain Plan Modes

By Jubin Abhishek Soni CORE

Every major agent framework now has a story for multi-agent systems. Most of them are incompatible with each other. An agent built in AutoGen cannot natively receive a task from a deepagents orchestrator. An OpenAI Agents SDK cannot talk to a LangGraph subgraph. A CrewAI crew cannot delegate to a Pydantic AI team without custom glue code. This is the inter-agent protocol problem: We have multi-agent frameworks, but no agreed-upon protocol for agents to communicate across frameworks. This post breaks down the four main approaches, compares them, and examines the need for standardization. Why Is Inter-Agent Communication Hard? Single-agent systems have a clean interface: you send a prompt, you get a response. Multi-agent systems need to express: Task delegation – what work is being handed off and to whomContext transfer – what is the background the receiving agent needsState propagation – what the delegating agent needs backError and cancellation – what happens when the receiving agent failsStreaming – how partial results flow back during long tasks Most frameworks solve some of these, but with different schemas, different transport assumptions, and no interoperability. The 4 Current Approaches 1. ACP (Agent Communication Protocol): deepagents Installation: Shell pip install deepagents-acp # Spec: https://github.com/langchain-ai/deepagents/tree/main/libs/acp # deepagents-acp version: 0.0.6 (requires agent-client-protocol>=0.8.0) ACP is an open protocol: a published JSON schema for agent-to-agent communication. Langchain's deepagents ships deepagents-acp, a Python client and server implementation, as a separate package, so any framework can implement it. ACP Message Flow A deepagents orchestrator routing a task to an ACP-compatible worker: Python from deepagents import create_deep_agent from deepagents.middleware import AsyncSubAgentMiddleware, AsyncSubAgent # Declare an ACP-compatible worker agent research_worker = AsyncSubAgent( name="research", description="Deep research agent — use for tasks requiring web search and synthesis", url="http://research-agent:8080", # ACP server endpoint ) # The orchestrator routes tasks to workers via the ACP protocol automatically orchestrator = create_deep_agent( model="anthropic:claude-sonnet-4-6", middleware=[ AsyncSubAgentMiddleware(subagents=[research_worker]), ], ) Serving a deepagents agent as an ACP endpoint: Python import asyncio from acp import run_agent as run_acp_agent from deepagents import create_deep_agent from deepagents_acp.server import AgentServerACP agent = create_deep_agent(model="anthropic:claude-haiku-4-5") # Wrap the compiled graph and expose it over ACP acp_agent = AgentServerACP(agent=agent) asyncio.run(run_acp_agent(acp_agent)) What makes ACP different: It's an open schema, not a framework-internal call. Any framework can implement an ACP server or client, which means a CrewAI crew could delegate to a deepagents worker over ACP without any shared code. Current limitation: Adoption is early. As of v0.9.0, the primary implementations are deepagents-native. Support for other frameworks requires each to implement the server interface independently. 2. Handoffs: OpenAI Agents SDK Installation: Shell pip install openai-agents # Docs: https://openai.github.io/openai-agents-python/handoffs/ OpenAI's SDK handles delegation through handoffs: an agent declares which other agents it can hand off to. At runtime, the orchestrator agent decides when to delegate and execution transfers. Python from agents import Agent, Runner, handoff research_agent = Agent( name="ResearchAgent", instructions="You are an expert at web research. Answer research questions thoroughly.", tools=[web_search_tool], ) code_agent = Agent( name="CodeAgent", instructions="You write and review Python code.", tools=[run_code_tool], ) orchestrator = Agent( name="Orchestrator", instructions=( "Route tasks to the right specialist. " "Use ResearchAgent for questions requiring web search. " "Use CodeAgent for programming tasks." ), handoffs=[ handoff(research_agent), handoff(code_agent, tool_name_override="delegate_to_coder"), ], ) result = await Runner.run(orchestrator, "Write a Python script to fetch weather data") You can also add context and filters to handoffs: Python from agents import handoff, RunContextWrapper def on_handoff_to_research(ctx: RunContextWrapper, input_data: str) -> None: print(f"Handing off to research agent with: {input_data}") research_handoff = handoff( research_agent, on_handoff=on_handoff_to_research, input_filter=lambda inp: inp, # transform input before handoff ) Limitation: Handoffs are framework-internal. A handoff target must be an Agent instance from the same SDK. There's no published schema, no HTTP transport, no cross-framework interoperability. If your orchestrator is a LangGraph graph and your specialist is an OpenAI agent, handoffs can't bridge them. 3. Agent Delegation via Tools: Pydantic AI Installation: Shell pip install pydantic-ai # Docs: https://pydantic.dev/docs/ai/guides/multi-agent-applications/ Pydantic AI does not have a dedicated inter-agent messaging primitive. The idiomatic multi-agent pattern is agent delegation via tools: a specialist agent is called from within a tool of the orchestrator agent, and the result is returned to the orchestrator like any other tool return value. Python from pydantic_ai import Agent, RunContext research_agent = Agent( "anthropic:claude-haiku-4-5", system_prompt="You are a research specialist. Answer questions with citations.", ) code_agent = Agent( "anthropic:claude-sonnet-4-6", system_prompt="You are a Python expert. Write clean, tested code.", ) orchestrator = Agent( "anthropic:claude-sonnet-4-6", system_prompt="Route tasks to the right specialist tool.", ) @orchestrator.tool async def research(ctx: RunContext[None], query: str) -> str: """Delegate a research question to the research specialist.""" result = await research_agent.run(query, usage=ctx.usage) return result.output @orchestrator.tool async def write_code(ctx: RunContext[None], task: str) -> str: """Delegate a coding task to the code specialist.""" result = await code_agent.run(task, usage=ctx.usage) return result.output result = await orchestrator.run( "Research quantum computing and write a Python simulation" ) Passing usage=ctx.usage propagates token accounting from the delegate run back to the parent, so result.usage() that the orchestrator covers all sub-agent calls. Limitation: Delegation is framework-internal: all agents must be Pydantic AI Agent instances. There is no pub-sub bus, no HTTP transport, and no cross-framework protocol. For true cross-framework delegation, a pattern like ACP is required. 4. AgentProtocol Backbone: Agno Installation: Shell pip install agno # Docs: https://docs.agno.com/introduction Agno takes the most ambitious approach: an AgentProtocol backbone that acts as a multi-framework adapter. The goal is to let agents from different frameworks (LangGraph, DSPy, Claude's SDK) plug into the same Agno team: Python from agno.agent import Agent from agno.team import Team from agno.models.anthropic import Claude # Native Agno agent research_agent = Agent( name="Researcher", model=Claude(id="claude-haiku-4-5"), tools=[web_search_tool], description="Specializes in web research and synthesis.", ) # Native Agno agent code_agent = Agent( name="Coder", model=Claude(id="claude-sonnet-4-6"), tools=[python_repl_tool], description="Specializes in Python development.", ) team = Team( name="FullStackTeam", mode="coordinate", members=[research_agent, code_agent], model=Claude(id="claude-sonnet-4-6"), db=agno_db_storage, # sessions + memory persistence enable_agentic_state=True, ) team.print_response( "Research quantum computing trends and prototype a simulation in Python" ) Limitation: The multi-framework adapter is still maturing. Most production Agno deployments use native Agno agents. The cross-framework vision is real, but the published adapter surface for LangGraph and DSPy is sparse at the time of writing. Side-by-Side Comparison DimensionACP (deepagents)Handoffs (OpenAI)Agent delegation (Pydantic AI)AgentProtocol (Agno)Published open schema✓✗✗PartialHTTP transport✓✗✗✓Streaming support✓ (SSE)✓ (run-level)✗✓Cross-framework workers✓ (any ACP server)✗✗PartialContext/metadata passing✓✓ (input_filter)✓ (usage propagation)✓Error/cancellation schema✓Partial✗✓Built-in state persistence✗ (LangGraph handles it)✗✗✓ (Agno DB)Production deploymentsEarlyGrowingMatureGrowing The Fragmentation Cost in Practice Consider this real scenario: you have a research pipeline where: The orchestrator is a deepagents agent (LangGraph-backed)The research worker is a CrewAI crew (good at parallel research tasks)The code worker is an OpenAI agent (good at code + sandboxed execution) Today, wiring this up requires: Python # Option 1: Wrap every non-deepagents agent as a plain function tool # Loses: streaming, cancellation, structured error handling @tool def run_crewai_research(query: str) -> str: crew = ResearchCrew() result = crew.kickoff(inputs={"query": query}) return str(result) # no streaming, no structured output # Option 2: Host each agent as an HTTP service and call it manually # Loses: shared context, standard error handling, progress tracking import httpx @tool async def call_openai_agent(task: str) -> str: async with httpx.AsyncClient() as client: response = await client.post( "http://openai-agent-service/run", json={"task": task}, ) return response.json()["result"] With ACP, the same cross-framework delegation looks like this: Python from deepagents.middleware import AsyncSubAgentMiddleware, AsyncSubAgent # Any ACP-compliant server — regardless of what framework runs inside orchestrator = create_deep_agent( model="anthropic:claude-sonnet-4-6", middleware=[ AsyncSubAgentMiddleware(subagents=[ AsyncSubAgent(name="research", url="http://crewai-acp-server:8080"), AsyncSubAgent(name="coder", url="http://openai-acp-server:8081"), ]), ], ) The wrapping frameworks are invisible. The protocol standardizes streaming, cancellation, and structured error handling. Where Is This Heading? The inter-agent space is actively consolidating around a few patterns: ACP is the most explicit attempt at standardization. It bets that an open schema survives framework churn. You can swap the worker implementation without changing the orchestrator.Handoffs are winning on simplicity within the OpenAI ecosystem. For teams already on the OpenAI SDK, they're ergonomic and production-proven. The cross-framework limitation only matters if you leave the ecosystem.The agent-delegation model (Pydantic AI via agent-as-tool, AutoGen's event-driven redesign) is a better fit for peer networks where no single orchestrator coordinates everything.Framework-native protocols will likely remain dominant for the near term. Cross-framework standardization requires enough pain from fragmentation to motivate all players -> we're getting there, but not quite yet. If you're building a multi-agent system today and expect to stay within one framework, pick that framework's native protocol. If you're building infrastructure that needs to orchestrate agents across frameworks or you expect your team to evaluate multiple frameworks, investing in ACP-compatible interfaces from the start gives you the most flexibility.

On December 2, 2024, a security vendor called BeyondTrust noticed something wrong inside its own AWS account. By the time the investigation closed, the story that emerged was almost absurdly simple for something with this much fallout: an attacker — later attributed to the Chinese state-sponsored group Silk Typhoon — had used a software flaw to reach into a BeyondTrust cloud account and pull out an API key. Not a password. Not a phishing victim's login. A string of characters that a piece of software used to talk to another piece of software. With that one key, the attacker walked straight into the U.S. Department of the Treasury, reset internal passwords, accessed workstations inside the Office of Foreign Assets Control, and read unclassified documents before anyone noticed. The Treasury disclosed it to Congress on December 30. The Department of Justice indicted the alleged operators in March 2025. If you've never worked in security, here's the plain-English version of what happened: somewhere inside the machinery that runs modern software, there's almost always a "key" — a credential one computer program shows another to prove it's allowed to be there. Humans log in with passwords and, increasingly, a second factor on their phone. Software mostly doesn't. It just holds a key, often for months or years at a time, and whoever holds that key gets treated as trustworthy, no questions asked. The Treasury breach happened because one of those keys ended up in the wrong hands and nothing else stood between that key and a federal agency's internal documents. Two months later, a different flavor of the same problem produced the largest theft of digital assets in history. $1.5 Billion, One Developer's Laptop In February 2025, the cryptocurrency exchange Bybit lost approximately $1.5 billion in Ethereum in a single operation. Palo Alto Networks' Unit 42 threat research team later tied the attack to Slow Pisces, a North Korean state-linked group also known as Lazarus or TraderTraitor, and traced the entry point back to a developer at a third-party vendor that managed Bybit's multi-signature wallet infrastructure. The attackers didn't break Ethereum's cryptography. They stole that developer's AWS session tokens — another form of machine credential — and used them to gain administrative access to cloud infrastructure that could authorize transactions, then quietly altered what a routine-looking transaction actually did before it executed. Unit 42 then found the same pattern at a second cryptocurrency exchange later in 2025, this time running through Kubernetes, the orchestration system that now runs much of the cloud-native world. The attackers phished a developer, used the access on the developer's machine to drop a malicious workload directly into the exchange's production Kubernetes cluster, and had that workload expose its own service account token — a credential Kubernetes automatically hands to every running pod so it can talk to the cluster's control plane. The stolen token happened to belong to a CI/CD management identity with sweeping permissions. From there, the intruders queried secrets across namespaces, planted a backdoor, and pivoted into the exchange's cloud-hosted backend, reaching the financial systems behind it. Unit 42's broader research found suspicious activity consistent with service-account-token theft in 22 percent of cloud environments analyzed in 2025, and recorded a 282 percent year-over-year jump in Kubernetes-directed attacks overall. Different industries, different attackers, same root cause: a non-human credential that was both long-lived and broader in scope than the task in front of it ever needed. Why This Keeps Happening Identity and access management, as a discipline, was built for people. People have managers, onboarding dates, performance reviews, and an HR system that flags them the day they leave. A workload has none of that. A microservice can spin up, do its job, and disappear thousands of times a day; a service account, by contrast, often gets created once and never revisited again. CyberArk's research has been blunt about the resulting imbalance: machine identities now outnumber human ones by more than 80 to 1 in the average enterprise, and the security architecture protecting most of them still assumes the old, human-shaped world — an org chart, not a fleet of ephemeral containers. That mismatch is exactly why static secrets sprawl the way they do. A developer hardcodes a key during a deadline crunch, intending to externalize it "later." A Terraform state file ends up holding plaintext cloud credentials because nobody flagged it in review. A default Kubernetes service account token, more permissive than anyone realized, gets mounted into a pod by default because turning that off requires deliberate configuration most teams never get around to. None of these are exotic mistakes. They're the ordinary residue of moving fast, and they accumulate the way unpaid debt does — quietly, until the day someone calls it in. The structural fix has a name by now, even if adoption is uneven: frameworks like SPIFFE and its production runtime SPIRE replace the static key with a short-lived, cryptographically attested identity — something closer to a backstage pass that's reissued before every single show rather than a master key cut once and handed out forever. A workload proves what it actually is — which Kubernetes service account launched it, which container image it's running — and receives an identity document valid for minutes, not months. Steal that, and an attacker is racing a clock that resets automatically rather than one that only resets when a human notices something is wrong. Cloud providers offer narrower versions of the same idea for their own platforms — AWS's IAM Roles for Service Accounts, Google's Workload Identity Federation — letting a workload trade a short-lived token for cloud access instead of carrying a standing key in the first place. But identity alone doesn't close the loop, and this is the part most "zero trust" conversations skip past. None of it matters if nothing in your pipeline actually enforces it. Security By Design Is a Promise. CI/CD Is Where You Find Out If It's Kept. Plenty of organizations will tell you, with complete sincerity, that they practice "security by design." Most of them mean it stopped at an architecture review months before the first line of code shipped. That's not a fix, it's a memory of one. Code that deploys daily — sometimes hourly — doesn't wait for an annual audit to catch a misconfigured token or an over-privileged service account, and by the time a quarterly review would have caught the BeyondTrust-style key or the Bybit-style session token, the damage in both real cases was already done. The only version of "security by design" that survives contact with a real production pipeline is the one written as code and enforced automatically, at every stage, by something that can actually say no. Picture the pipeline this way: Plain Text Developer commits code | v CI build triggers | +--> SAST (code flaws) + SCA (dependency CVEs) + secrets scan | | | fail? -----> build blocked, developer notified | | | pass v Generate SBOM + sign artifact (Cosign) + build provenance (SLSA) | v Policy-as-code gate (OPA / Kyverno) | +--> checks: image from approved registry? running as non-root? | signature valid? provenance matches expected builder? | service account scoped to least privilege? | | fail? -----> deployment rejected, logged, alert raised | pass v Deploy to production | v Runtime monitoring + short-lived workload identity (SPIFFE/SPIRE, IRSA) | v Continuous re-verification — nothing trusted indefinitely Every box in that chain is a place where the Treasury breach or the Bybit breach could have stopped instead of escalating. A policy-as-code rule using Open Policy Agent's Rego language, or Kyverno's Kubernetes-native YAML equivalent, can flatly refuse to schedule a pod requesting broader RBAC permissions than its declared task needs — which would have directly undercut the over-privileged CI/CD identity that the crypto-exchange attackers rode into the cluster. A signing and attestation step using Cosign, tied to SLSA provenance, means a deployed artifact has to prove which build system actually produced it before it runs at all — closing exactly the kind of trust gap that let a single compromised AWS asset cascade into a stolen infrastructure API key at BeyondTrust. None of this is theoretical tooling. Red Hat's own Enterprise Contract documentation describes signing as tying an image to a specific builder identity precisely so an attacker can't substitute a malicious binary without the signature itself breaking and announcing the tampering. The Uncomfortable Bottom Line I don't think either of this year's headline breaches happened because anyone involved was careless in some obvious, fireable way. They happened because the credential — not the firewall, not the encryption, not the cleverness of the malware — was the actual asset under attack the entire time, and almost nothing downstream of "the key worked" was built to ask a second question. Gartner named non-human identity management a top strategic security trend for exactly this reason in 2025, and OWASP followed with a dedicated Non-Human Identity Top 10 the same year, an overdue acknowledgment that the tooling built for human logins was never going to be enough. My honest prediction, watching this pattern repeat across a federal agency and two of the largest crypto exchanges on earth within twelve months of each other: the organizations that treat policy-as-code enforcement and short-lived machine identity as default infrastructure — not optional hardening bolted on after an incident — are the ones that won't end up writing the next version of this story. Everyone else is currently running on borrowed time, secured by a key that, statistically, is already older than it should be.

By Igboanugo David Ugochukwu CORE

Overview This article explores whether dependency injection (DI) can exist in Rust without sacrificing the language’s core philosophy of zero-cost abstractions. We will approach the question from three angles: Why dependency injection still matters in Rust, even for systems built with zero-sized types and compile-time guarantees.How DI evolved in other ecosystems, using Java as a reference point.A practical Rust-oriented approach to implementing DI with compile-time guarantees. We’ll also show how Rust traits enable DI patterns that scale across crates, preserving zero-cost guarantees. All Rust source code used in this article is available in this repository. Rust DI: The Problem Rust Hasn’t Solved Yet Rust has solved problems most languages haven’t even dared to touch: memory safety without a garbage collector, fearless concurrency, and powerful zero-cost abstractions. But there is a class of problems Rust hasn’t fully confronted yet. Not because Rust is incapable — but because these problems exist above the machine level. They are not about memory safety or performance. They are about composition, modularity, and architectural correctness in large systems. Managing dependencies between dozens or hundreds of components is fundamentally different from managing memory or threads. Rust gives us powerful primitives, but the question remains: How do we scale composition safely and maintainably? What “Enterprise” Really Means in Rust Terms When Rust developers hear enterprise, they often think slow, over-engineered, and bloated. But that perception is misleading. Enterprise systems are not bloated by accident. They are complex because composition eventually stops being trivial. The complexity comes from business requirements, not from the technology stack. Enterprise: The Burden We Can’t Avoid When a company reaches a certain scale, several things inevitably happen: Products serve thousands or millions of usersSystems integrate with vendors, partners, and third-party servicesTeams work independently on modules and featuresSoftware must evolve continuously without stopping the business These realities create architectural pressure. From a technical perspective, systems must support: Scalability: At multiple levels — both in terms of users and data, including hundreds, thousands, or millions, or even up to billions of concurrent users, as well as functional modules interacting across teams.Reliability: Systems run 24/7. Services must handle failures because dependencies on vendors, partners, or third-party services mean that failures are inevitable, and the system must continue operating despite them.Modularity: Independent teams need to work on isolated components without breaking other parts of the system.Flexibility: Infrastructure choices may change. Databases, messaging systems, or integrations might need to be swapped without rewriting the entire application.Observability: To detect and respond to performance bottlenecks, integration failures, or unexpected behaviors quickly.Extensibility: New products, markets, and regulations require systems to evolve incrementally rather than being rebuilt from scratch.Maintainable: Every business decision introduces new dependencies. And every dependency increases the complexity of the system’s composition. Ensuring that the system doesn’t become so convoluted that small changes introduce cascading errors. Even with Rust’s ownership model and strong type system, manually managing this dependency graph eventually becomes impractical. These pressures are not theoretical — they define the daily reality of enterprise software engineering. Every design decision must balance immediate business needs with long-term sustainability, especially under high concurrent load. Where Dependency Injection Becomes Relevant This is exactly where dependency injection becomes useful. DI allows systems to manage complexity by separating what components need from how those dependencies are created and connected. In practice, this means: Components declare their dependencies without constructing them directlyDependencies are provided externally, keeping components isolatedSystems evolve gradually without breaking existing modulesOptional features and plugins can be integrated without tightly coupling the system DI is not just a convenience. It is a structured approach to handling inevitable architectural complexity. Enterprise Isn’t Just Complexity — It’s Heterogeneity Large systems are rarely uniform. They typically contain: Independent components with their own dependency treesStateful infrastructure such as databases, caches, and message brokersOptional features and plugin-style modulesMultiple implementations of the same interface This heterogeneity appears naturally over time. Systems accumulate tools built years apart, libraries maintained by different teams, and components that survive long after their original authors have moved on. Enterprise systems grow gradually, and they rarely get the chance to start over. Rust does not eliminate these pressures. Any real system eventually faces them. Java’s Historical Perspective: DI Was Inevitable Java did not adopt dependency injection because it was fashionable. It adopted DI because large systems were becoming impossible to manage without it. Without DI, developers quickly ran into familiar problems: Tight coupling between componentsFragile initialization orderHard-coded dependencies scattered across the codebaseChanges in one module unexpectedly breaking another Dependency injection emerged as a discipline for managing complexity. Components declare what they depend on, and the system provides those dependencies when constructing the application. This separation allows systems to evolve without collapsing under their own architecture. DI in a Nutshell You can think of dependency injection as a kind of runtime composition system. If your application contains many services, modules, plugins, or optional components, something must assemble them and ensure they are wired correctly, and that role belongs to the DI system. DI is conceptually similar to package managers such as Cargo or Maven, but it operates at a different level: Package managers resolve dependencies between libraries at build time.Dependency injection resolves dependencies between components at runtime. Loading executable code into memory is easy — the operating system handles that. What is harder is creating objects, initializing them correctly, and ensuring that all components interact with the right dependencies. This becomes increasingly difficult as systems grow. Dependency injection addresses this problem directly. How Dependency Injection Is Typically Solved in Java Java provides one of the most mature ecosystems for dependency injection. Frameworks such as Spring or Guice automate object creation and dependency wiring almost entirely. Let’s revisit the same example from the previous section: a simple User Management API. We have two controllers: ReadController — retrieves users from a databaseWriteController — creates users and publishes events to a message broker Both controllers depend on infrastructure services that must be created and wired correctly. Without Dependency Injection In a traditional manual setup, object creation and wiring might look like this: public class Application { public static void main(String[] args) { Database database = new PostgresDatabase(); MessageBroker broker = new KafkaBroker(); ReadController readController = new ReadController(database); WriteController writeController = new WriteController(database, broker); // start application } } At first glance, this appears manageable. But as the application grows, the initialization code expands rapidly: Multiple infrastructure servicesOptional modulesConfiguration logicConditional wiring depending on the environment The main method eventually becomes responsible for constructing the entire dependency graph of the application. This approach becomes difficult to maintain and extremely fragile as the system evolves. Dependency Injection With Spring Dependency injection frameworks solve this by moving the responsibility of object creation and wiring to a container. Components simply declare what they need. @Service public class Database { } @Service public class KafkaBroker implements MessageBroker { } @RestController public class ReadController { private final Database database; @Autowired public ReadController(Database database) { this.database = database; } } Dependencies are declared in constructors, and the DI container automatically provides the correct instances. The application no longer manually constructs the object graph. Instead, the framework scans components and resolves dependencies automatically. Polymorphism in Java DI Java DI frameworks also support multiple implementations of the same interface. For example, an application may support several message brokers simultaneously: @Service public class KafkaBroker implements MessageBroker { } @Service public class RabbitBroker implements MessageBroker { } A controller can receive all implementations at once: @RestController public class WriteController { private final List<MessageBroker> brokers; @Autowired public WriteController(List<MessageBroker> brokers) { this.brokers = brokers; } } The DI container automatically collects all implementations of MessageBroker and injects them into the controller. This makes the system highly extensible: New brokers can be addedExisting ones can be removedThe controller remains unchanged The Cost of Traditional DI Java DI frameworks provide powerful capabilities, but they come with trade-offs: Dependency resolution happens at runtimeReflection is heavily usedErrors may only appear during application startupDependency graphs are not always fully visible to the compiler This runtime flexibility works well for the Java ecosystem, but it introduces overhead and reduces compile-time guarantees. Rust, on the other hand, encourages a different philosophy: If something can be verified at compile time, it should be. This raises an interesting question: Can Rust achieve the same flexibility of dependency injection while preserving compile-time guarantees and zero runtime cost? Journey into Rust Coding Let’s try to build a dependency injection approach in Rust gradually. We will follow the same conceptual example used in the Java section: A ReadControllerA WriteControllerMultiple implementations of a MessageBrokerAn abstraction for database connectivity Rust Without Dependency Injection In the first example, we will implement a small Rust application without dependency injection. However, we will introduce use-traits, which will later allow us to transition naturally to a dependency injection model. 1. Defining Database Interfaces First, let’s define the interface used to access the database. 1.1 DatabaseConnection Trait This trait represents an abstraction for database connectivity that can support multiple implementations (Postgres, MySQL, etc.). trait DatabaseConnection { fn read_query(&self, query: &str); fn write_query(&self, query: &str); } 1.2 UseDatabaseConnection Trait Next, we define a trait that allows components to request a database connection from a context. trait UseDatabaseConnection { type T: DatabaseConnection; fn database_connection(&self) -> &Self::T; } This trait will later be used as the foundation of dependency resolution. Instead of components knowing the entire application context, they simply declare that they require a DatabaseConnection. This keeps components decoupled from the full application structure. 2. Database Implementation Now we provide a concrete implementation of DatabaseConnection. #[derive(Default)] struct PostgresDatabaseConnection {} impl DatabaseConnection for PostgresDatabaseConnection { fn read_query(&self, query: &str) { println!("Reading from Postgres DB: {}", query) } fn write_query(&self, query: &str) { println!("Writing into Postgres DB: {}", query) } } For simplicity, this example only prints messages instead of connecting to a real database. In a real system, this could be implemented using any production database library. 3. Controllers Now we define the controllers responsible for performing application logic. 3.1 Controller Structs #[derive(Default)] struct ReadController {} #[derive(Default)] struct WriteController {} Rust allows structs with no fields. These zero-sized types have no runtime cost, but they still represent concrete types at compile time and can participate in abstractions. 3.2 Controller Use Traits Next, we define traits that expose controllers to other components. trait UseReadController { fn read_controller(&self) -> &ReadController; } trait UseWriteController { fn write_controller(&self) -> &WriteController; } These traits allow components to access controllers without knowing anything about the application context. 3.3 Controller Context Now we combine the previously defined traits into a context trait. trait ControllerContext: UseDatabaseConnection + UseReadController + UseWriteController {} This context describes the minimal environment required for controllers to function. Controllers will depend only on this trait instead of the full application context. 3.4 Controller Implementation Now we implement the controller logic. impl ReadController { fn do_something<C: ControllerContext>(&self, ctx: &C, argument: &str) { ctx.database_connection() .read_query(format!("SELECT * FROM table WHERE id = '{}'", argument).as_str()); } } impl WriteController { fn do_something<C: ControllerContext>(&self, ctx: &C, argument: &str) { ctx.database_connection().write_query( format!("UPDATE table SET value = 'new' WHERE id = '{}'", argument).as_str(), ); } } Notice something important here: The controllers do not know about the full application context. They only know about the traits they depend on. This means the controller and database code could already be extracted into separate crates, reusable by any application implementing the required use-traits. 4. Wiring the Application Now we wire all components together. 4.1 Application Context We define a struct that holds all application components. #[derive(Default)] struct ApplicationContext { read_controller: ReadController, write_controller: WriteController, postgres_database_connection: PostgresDatabaseConnection, } This struct acts as the composition root of the application. 4.2 Implement Use Traits Next we implement the previously defined traits. impl UseReadController for ApplicationContext { fn read_controller(&self) -> &ReadController { &self.read_controller } } impl UseWriteController for ApplicationContext { fn write_controller(&self) -> &WriteController { &self.write_controller } } impl UseDatabaseConnection for ApplicationContext { type T = PostgresDatabaseConnection; fn database_connection(&self) -> &Self::T { &self.postgres_database_connection } } By implementing these traits, ApplicationContext becomes capable of providing dependencies to components. 4.3 Controller Context Implementation impl ControllerContext for ApplicationContext {} Since ApplicationContext already implements the required traits, it automatically satisfies ControllerContext. 5. Running the Application Finally we run the application. pub fn run() { let ctx = ApplicationContext::default(); ctx.read_controller().do_something(&ctx, "argument"); ctx.write_controller().do_something(&ctx, "argument"); } Key characteristics of this approach: No dyn traitsNo Arc or RcNo runtime dependency container All wiring is resolved at compile time through generics and monomorphization. Multi-Threading An attentive reader may ask: Will this approach work in multi-threaded environments? In Rust, thread safety is typically ensured using the Send and Sync traits. These traits are automatically implemented by the compiler if all fields of a struct are also Send + Sync. We can verify thread safety with a compile-time assertion: const _: () = { const fn assert_send_sync<T: Send + Sync>() {} assert_send_sync::<ApplicationContext>(); }; If this compiles, the entire application context can safely be shared between threads. In real systems, some components (such as database connections) may not be inherently thread-safe. In such cases, a connection pool or synchronization mechanisms such as Mutex are required. This limitation is not related to the dependency injection approach itself, but rather to shared resource management in concurrent systems. What the Compiler Actually Generates If we inspect the compiled output with: cargo asm rust_di_example::main ... 26 │ lea rbx, [rsp, +, 32] 27 │ mov rdx, rbx 28 │ call qword, ptr, [rip, +, _ZN5alloc3fmt6format12format_inner17he42ed4cf3cdc276bE@GOTPCREL] 29 │ movups xmm0, xmmword, ptr, [rsp] 30 │ mov rax, qword, ptr, [rsp, +, 16] 31 │ movups xmmword, ptr, [rsp, +, 48], xmm0 32 │ mov qword, ptr, [rsp, +, 64], rax 33 │ mov qword, ptr, [rsp, +, 32], r14 34 │ mov qword, ptr, [rsp, +, 40], 18 35 │ mov qword, ptr, [rsp], rbx 36 │ mov qword, ptr, [rsp, +, 8], r13 37 │ lea rdi, [rip, +, .Lanon.63c02f0152e6743e61fdeaf76f1d4051.26] 38 │ mov rsi, rsp 39 │ call qword, ptr, [rip, +, _ZN3std2io5stdio6_print17hba8f5eda1e4e495eE@GOTPCREL] 40 │ lea rax, [rip, +, .Lanon.63c02f0152e6743e61fdeaf76f1d4051.27] 41 │ mov qword, ptr, [rsp, +, 32], rax 42 │ mov qword, ptr, [rsp, +, 40], 19 43 │ mov qword, ptr, [rsp], rbx 44 │ mov qword, ptr, [rsp, +, 8], r13 45 │ lea r14, [rsp, +, 48] 46 │ mov qword, ptr, [rsp, +, 16], r14 47 │ lea r15, [rip, +, _ZN60_$LT$alloc..string..String$u20$as$u20$core..fmt..Display$GT$3fmt17h9d11f1d81b352ac8E] 48 │ mov qword, ptr, [rsp, +, 24], r15 49 │ lea rdi, [rip, +, .Lanon.63c02f0152e6743e61fdeaf76f1d4051.7] 50 │ mov rsi, rsp 51 │ call qword, ptr, [rip, +, _ZN3std2io5stdio6_print17hba8f5eda1e4e495eE@GOTPCREL] 52 │ lea rax, [rip, +, .Lanon.63c02f0152e6743e61fdeaf76f1d4051.28] 53 │ mov qword, ptr, [rsp, +, 32], rax 54 │ mov qword, ptr, [rsp, +, 40], 21 55 │ mov qword, ptr, [rsp], rbx 56 │ mov qword, ptr, [rsp, +, 8], r13 57 │ mov qword, ptr, [rsp, +, 16], r14 58 │ mov qword, ptr, [rsp, +, 24], r15 59 │ lea rdi, [rip, +, .Lanon.63c02f0152e6743e61fdeaf76f1d4051.8] 60 │ mov rsi, rsp 61 │ call qword, ptr, [rip, +, _ZN3std2io5stdio6_print17hba8f5eda1e4e495eE@GOTPCREL] ... We see extremely flat assembly code with series of invocation to _ZN3std2io5stdio6_print17hba8f5eda1e4e495eE@GOTPCREL that is just printing subroutine in rust runtime. There are no runtime dependency resolution mechanisms, no dynamic dispatch, and no container logic. The generated code mostly contains calls to standard library functions such as printing. This demonstrates that the abstractions introduced here do not introduce runtime overhead. Why Use-Traits Matter At first glance, the use-trait might look like unnecessary indirection. Why not simply pass ApplicationContext directly to every component? The reason is crate-level decoupling. Enterprise applications often grow into multiple crates. Controllers, database access layers, messaging integrations, and domain logic are very often implemented as reusable libraries. For example, a Spring Boot actuator–style module may contain all layers inside the DB, provide REST API endpoints, and integrate with a monitoring aggregator service — it acts as a standalone sub-program. However, if a component directly depends on ApplicationContext, it becomes tied to the executable crate that defines it. That creates an architectural problem: Libraries would depend on the application crateThe application crate would depend on the libraries This circular dependency makes reuse impossible. Use-trait solve this by defining capability-based interfaces. Instead of depending on the application context, components depend only on the capabilities they require. Example: trait UseDatabaseConnection { type T: DatabaseConnection; fn database_connection(&self) -> &Self::T; } A controller does not know anything about the application structure. It simply requires that the context provides access to a database connection. impl ReadController { fn do_something<C: ControllerContext>(&self, ctx: &C, argument: &str) { ctx.database_connection() .read_query(format!("SELECT * FROM table WHERE id = '{}'", argument).as_str()); } } Because of this design: ReadController can live in its own crateThe crate only exports traits describing the capabilities it needsAny application can use the controller by implementing those traits The application context becomes an adapter, wiring together independent components. Application ├── implements UseDatabaseConnection ├── implements UseReadController └── implements UseWriteController This pattern enables a powerful architectural property: Components become fully reusable libraries, while the application remains responsible only for wiring them together. In other words, use-traits allow dependency injection to cross crate boundaries while preserving Rust’s compile-time guarantees. Without this indirection, the system collapses into a monolithic application context that cannot be decomposed into reusable modules. Limitations of This Approach Although this example demonstrates many useful properties, it is not yet a complete dependency injection system. The main limitation is that ApplicationContext still has too much knowledge about component internals. In real DI frameworks, modules often contain many components, initialization logic, and internal dependencies. For example, consider a Spring Boot module such as Spring Data. When you add the dependency to your project, it automatically provides: Database driver integrationConnection poolingRepository interfacesTransaction managementEntity scanningMetrics integrationHealth check integration All of this functionality is assembled automatically by the DI framework. From the application developer’s perspective, only minimal configuration is required. Real dependency injection modules therefore consist of entire subgraphs of components, not just individual services. In our example we intentionally introduced two controllers to demonstrate that even a simple module may contain multiple cooperating components. A complete dependency injection framework must also manage: Module compositionInitialization lifecycleDependency resolutionOptional componentsMultiple implementations This is where the real challenge begins. Rust With Dependency Injection To implement dependency injection in Rust, we will build iteratively. We start from the previous “no DI” approach and gradually close the gap toward a complete DI system. The good news is that we already have use-traits, and our components are decoupled. We can extract certain code into reusable modules. What’s missing for a true dependency injection system: ApplicationContext still has too much knowledge about the components it uses.Some wiring and initialization steps are still manual. Our goal is to move the wiring into DI modules, giving each component full control over how it is connected. Because we are still targeting compile-time injection, we cannot rely on runtime reflection (like Java DI frameworks do). Instead, we will push this logic into Rust macros, allowing compile-time wiring while preserving zero-cost abstractions. 1. Registering Components in ApplicationContext In traditional DI, the application knows which modules it depends on (like Spring Data). But modules themselves should control which components they export. In our previous example, ApplicationContext was a struct, and registering a component meant adding a field manually. This ties the application to module internals. We need a way to add fields to ApplicationContext automatically, without putting module-specific code into the executable. We can achieve this using the combine-structs crate, which provides macros to embed multiple structs into one. Each module defines an embeddable struct as a context extension. When imported, ApplicationContext automatically merges all fields from these extensions. 1.1 Context Extension for PostgreSQL #[allow(dead_code)] #[derive(Fields)] struct PostgresDatabaseContextExtension { postgres_database_connection: PostgresDatabaseConnection, } The Fields derive macro allows this struct to be merged into ApplicationContext. 1.2 Context Extension for Controllers #[allow(dead_code)] #[derive(Fields)] struct ControllerContextExtension { read_controller: ReadController, write_controller: WriteController, } The controller module exports two controllers. More components can be added without touching the main executable. 1.3 Embedding Context Extensions #[combine_fields(PostgresDatabaseContextExtension, ControllerContextExtension)] #[derive(Default)] struct ApplicationContext {} The combine_fields macro merges all fields from the context extensions. ApplicationContext now has all components automatically wired. 2. Providing Use-Traits Previously, wiring was done via use-traits. Now that ApplicationContext doesn’t know which components exist, modules must export use-trait implementations via macros. 2.1 Macro for Database Connectivity macro_rules! inject_postgres_impl { () => { impl UseDatabaseConnection for ApplicationContext { type T = PostgresDatabaseConnection; fn database_connection(&self) -> &Self::T { &self.postgres_database_connection } } }; } 2.2 Macro for Controllers macro_rules! inject_controller_impl { () => { impl UseReadController for ApplicationContext { fn read_controller(&self) -> &ReadController { &self.read_controller } } impl UseWriteController for ApplicationContext { fn write_controller(&self) -> &WriteController { &self.write_controller } } impl ControllerContext for ApplicationContext {} }; } 2.3 Injecting Components #[combine_fields(PostgresDatabaseContextExtension, ControllerContextExtension)] #[derive(Default)] struct ApplicationContext {} inject_postgres_impl!(); inject_controller_impl!(); The executable only calls these macros. Components remain isolated from the main application, and the wiring happens automatically. 3. Intermediate Conclusion At this stage: No component code has been changed.Modules can add or remove components freely.Components are decoupled from each other and from the container.Wiring happens automatically through macros and use-traits. This gives us a bare-minimum dependency injection system: application components are decoupled, wiring is automatic, and no single component needs full knowledge of the application. 4. Limitations Even though we now have a working DI mechanism, it isn’t fully production-ready: Initialization: Components may require setup before wiring.Lifecycle Management: Controlling initialization order, cleanup, or optional components can be challenging. Next, we will explore a Rust DI framework capable of automating component initialization and lifecycle management, moving closer to a complete solution. Dependency Injection and Initialization Cycle in Rust So far, we have built a dependency injection (DI) container where all components are stored as fields in ApplicationContext. The next challenge is initializing these components. Press enter or click to view image in full size The goal is to: Enumerate the fields of ApplicationContext.Identify which fields require initialization.Call an initialization method for each such component. Since we want everything to happen at compile time, we need a macro to generate a Rust method that calls init() on every tagged component without runtime loops or collections. I could not find an existing macro for this, so I implemented one myself. If you want the details, check the implementation here: di_macro/src/lib.rs. We will focus on how to use this macro, not how it works internally. Macro Example: Enumerating Tagged Fields Full example code: struct_enumerator.rs 1. Define a Struct with Tagged Fields #[allow(dead_code)] #[derive(Debug, FieldEnumerator, Default)] pub struct MyStruct { #[tag(init_listener)] field_1: i32, #[tag(init_listener)] #[tag(start_listener)] field_1_2: i32, field_2: i32, #[tag(start_listener)] field_3: i32, } FieldEnumerator is our custom derive macro.Fields can have one or more tags (init_listener, start_listener). 2. Define a Callback Macro macro_rules! my_callback { ($struct_name:ident, $field_name:ident, $listener_type:ident) => { println!( "struct = {}, field = {}, type = {}", stringify!($struct_name), stringify!($field_name), stringify!($listener_type), ) }; } For every tagged field, the callback macro is called at compile time.Arguments passed: struct_name, field_name, and listener_type. 3. Invoke the Field Enumerator pub fn run() { let my_struct = MyStruct::default(); println!("my_struct = {:?}", my_struct); enumerate_tags_MyStruct_init_listener!(my_callback); enumerate_tags_MyStruct_start_listener!(my_callback); } enumerate_tags_MyStruct_init_listener! and enumerate_tags_MyStruct_start_listener! are generated automatically by the FieldEnumerator macro.The macro expands into a flat sequence of println!() calls. Macro Example Output: // enumerate_tags_MyStruct_init_listener!(my_callback); // my_callback!(MyStruct, field_1, init_listener) println!("struct = {}, field = {}, type = {}", "MyStruct", "field_1", "init_listener") // my_callback!(MyStruct, field_1_2, init_listener) println!("struct = {}, field = {}, type = {}", "MyStruct", "field_1_2", "init_listener") //enumerate_tags_MyStruct_start_listener!(my_callback) // my_callback!(MyStruct, field_1_2, start_listener) println!("struct = {}, field = {}, type = {}", "MyStruct", "field_1_2", "start_listener") // my_callback!(MyStruct, field_3, start_listener) println!("struct = {}, field = {}, type = {}", "MyStruct", "field_3", "start_listener") Notice: No vectors, arrays, loops, or runtime collections — everything happens at compile time. Rust Dependency Injection with Initialization We can now use the same macro to enumerate all fields in ApplicationContext and initialize them. Code reference: di_init.rs We introduce a Configuration component to demonstrate how initialization can depend on runtime data. 1. Configuration Module #[derive(Default)] struct Configuration { run_arguments: &'static str, } #[allow(dead_code)] #[derive(Fields, Default)] struct ConfigurationContextExtension { configuration: Configuration, } trait UseConfiguration { fn configuration(&self) -> &Configuration; fn configuration_mut(&mut self) -> &mut Configuration; } macro_rules! inject_configuration_impl { () => { impl UseConfiguration for ApplicationContext { fn configuration(&self) -> &Configuration { &self.configuration } fn configuration_mut(&mut self) -> &mut Configuration { &mut self.configuration } } }; } Steps: Define the component struct (Configuration).Define a context extension for ApplicationContext.Define a use-trait (UseConfiguration) for wiring.Provide a macro to implement the trait on ApplicationContext. Note: Configuration is no longer zero-sized—it contains runtime data (run_arguments). 2. Database Connection Initialization 2.1 Update PostgresDatabaseConnection #[derive(Default)] struct PostgresDatabaseConnection { connection_string: String, } Now contains runtime data.Initialization depends on configuration. 2.2 Tag Component for Initialization #[allow(dead_code)] #[derive(Fields, ContextExtension)] struct PostgresDatabaseContextExtension { #[tag(init_listener)] postgres_database_connection: PostgresDatabaseConnection, } init_listener signals that the component requires initialization. 2.3 Define Initializable Trait trait Initializable<C> { fn init(ctx: &mut C); } Components implementing this trait can be initialized automatically. 2.4 Implement Initialization impl<C: UseConfiguration + UsePostgresDatabaseConnection> Initializable<C> for PostgresDatabaseConnection { fn init(ctx: &mut C) { println!("Init sequence = {}", ctx.configuration().run_arguments); ctx.postgres_database_connection_mut().connection_string = format!("Postgres DB on {}", ctx.configuration().run_arguments); } } Accesses ApplicationContext mutably for initialization of any of component. 2.5 Prepare ApplicationContext #[combine_fields( ConfigurationContextExtension, PostgresDatabaseContextExtension, ControllerContextExtension )] #[derive(Default, FieldEnumerator)] struct ApplicationContext {} inject_postgres_impl!(); inject_controller_impl!(); inject_configuration_impl!(); Added FieldEnumerator for tag enumeration.Configuration module bindings included. 2.6 Initialization Sequence impl ApplicationContext { fn init(&mut self) { fn call_init<T: Initializable<ApplicationContext>, F: Fn(ApplicationContext) -> T>( ctx: &mut ApplicationContext, _closure: F, ) { T::init(ctx); } macro_rules! init_callback { ($struct_name:ident, $field_name:ident, $listener_type:ident) => { call_init(self, |x| x.$field_name); }; } enumerate_tags_ApplicationContext_init_listener!(init_callback); } } How it works: call_initfunction This helper function takes a generic type T that implements Initializable<ApplicationContext>.It also takes a closure _closure of type Fn(ApplicationContext) -> T.The trick here: the Rust compiler monomorphizes the closure to the actual type of the field passed in, so T::init(ctx) is called with the concrete type.init_callback!macro The macro expands for each field tagged with init_listener.It calls call_init with the correct field from self, ensuring the proper Initializable implementation is invoked.enumerate_tags_ApplicationContext_init_listener!macro This macro iterates over all fields in ApplicationContext that are marked with #[init_listener].For each field, it invokes init_callback!, which triggers Initializable::init for that specific component. Key rust trick: By using the Fn trait and generics in call_init, the compiler resolves the actual type of the field at compile time. This avoids any runtime type checks and ensures zero-cost initialization while keeping strong type safety. 2.7 Running the Application pub fn run() { let mut ctx = ApplicationContext::default(); ctx.configuration_mut().run_arguments = "DB_URL=127.0.0.1:5555"; ctx.init(); ctx.read_controller().do_something(&ctx, "argument"); ctx.write_controller().do_something(&ctx, "argument"); } Sample Output: Init sequence = DB_URL=127.0.0.1:5555 Reading from Postgres DB on DB_URL=127.0.0.1:5555: SELECT * FROM table WHERE id = 'argument' Writing into Postgres DB on DB_URL=127.0.0.1:5555: UPDATE table SET value = 'new' WHERE id = 'argument' run_arguments successfully propagated into runtime data. Performance Considerations In this demo, some structs now hold runtime data — but this is intentional. It’s added to demonstrate initialization, just like in real applications where components manage runtime state. The wiring mechanism itself remains zero-cost: All bindings are resolved at compile time through monomorphization. Even with the initialization sequence broadcasting multiple init calls, the compiler generates a flat sequence of calls: no loops, no runtime collections, no dynamic dispatch — everything happens at compile time, efficiently. Limitations This approach is now mature and production-ready for wiring, decoupling, and initialization.Next steps can explore advanced topics, such as polymorphism and more complex runtime behaviors. Dependency Injection and Polymorphism This is the final example of the article and introduces what I would consider an advanced topic for the core engine of any dependency injection framework: polymorphism. Press enter or click to view image in full size Many DI frameworks handle basic dependency wiring well. For example, Java Spring Boot provides a very mature implementation. However, in many other DI implementations, one important capability is often missing — the ability to handle multiple implementations of the same abstraction in a flexible and compile-time-safe way. Let’s extend our example with a new requirement. New Requirement Our application should support multiple message brokers, for example: KafkaRabbitMQ After writing data to the database, the controller should publish a message to one or more brokers. However: The component does not know which brokers existThe container may contain multiple brokersThe DI framework must maintain this one-to-many relationship One component should be able to call many broker implementations without knowing which ones exist. To make things even more interesting, we introduce the concept of profiles. Each profile represents a different configuration of the application context. Example: Profile1 PostgreSQL databaseKafka brokerRabbitMQ broker Profile2 Oracle databaseRabbitMQ broker only See the complete example. Injection Macros and Profiles First, we slightly modify our injection macros so they accept the application context type as an argument. macro_rules! inject_configuration_impl { ($ctx:ident) => { impl UseConfiguration for $ctx { fn configuration(&self) -> &Configuration { &self.configuration } fn configuration_mut(&mut self) -> &mut Configuration { &mut self.configuration } } }; } This change is necessary because the DI module does not know which profile will be used. Each executable can choose a different application context profile, and the macros must work with whichever profile is selected. Oracle Database Component Now we introduce a new database implementation. #[allow(dead_code)] #[derive(Fields, ContextExtension)] struct OracleDatabaseContextExtension {} And the injection macro: macro_rules! inject_oracle_impl { ($ctx: ident) => { impl DatabaseConnection for $ctx { fn read_query(&self, query: &str) { println!("Reading from Oracle DB: {}", query) } fn write_query(&self, query: &str) { println!("Writing into Oracle DB {}", query) } } impl UseDatabaseConnection for $ctx { type T = $ctx; fn database_connection(&self) -> &Self::T { self } } }; } Here we apply a small trick. Instead of defining a separate struct for the database connection, we implement the trait directly on the application context. This approach avoids additional boilerplate and works well when we know there will only be one database implementation per profile. Defining Message Brokers Now we define the abstraction for message brokers. Broker Interface trait BrokerSender { fn send_to_broker(&self, value: &str); } RabbitMQ Broker #[allow(dead_code)] #[derive(Default, Fields, ContextExtension)] struct RabbitMqContextExtension { #[tag(broker)] rabbit_mq: RabbitMq, } #[derive(Default)] struct RabbitMq; impl BrokerSender for RabbitMq { fn send_to_broker(&self, value: &str) { println!("{} sent to RabbitMq", value); } } Notice the important detail: #[tag(broker)] This tag allows the DI framework to enumerate all brokers automatically using the same mechanism we previously used for initialization. Kafka Broker Kafka is implemented in exactly the same way. #[allow(dead_code)] #[derive(Default, Fields, ContextExtension)] struct KafkaContextExtension { #[tag(broker)] kafka: Kafka, } #[derive(Default)] struct Kafka; impl BrokerSender for Kafka { fn send_to_broker(&self, value: &str) { println!("{} sent to Kafka", value); } } Publisher — Compile-Time Polymorphism Now comes the most interesting part. We define a Publisher component that sends messages to all available brokers. trait Publisher { fn publish(&self, value: &str); } Injection macro: macro_rules! inject_publisher_impl { ($ctx:ident) => { impl Publisher for $ctx { fn publish(&self, value: &str) { macro_rules! broker_callback { ($struct_name:ident, $field_name:ident, $listener_type:ident) => { self.$field_name.send_to_broker(value); }; } enumerate_tags!($ctx, broker, broker_callback); } } impl UsePublisher for $ctx { type T = $ctx; fn publisher(&self) -> &Self::T { self } } }; } The key idea: the publisher does not know which brokers exist. Instead, the FieldEnumerator macro generates code that calls send_to_broker for each tagged broker. This gives us: One-to-many relationshipCompile-time wiringNo dynamic dispatchNo runtime overhead Helper Macro for Tag Enumeration macro_rules! enumerate_tags { ($ctx:ident, $tag:ident, $callback:ident) => { paste! { [<enumerate_tags_ $ctx _ $tag >]!($callback) } }; } This macro simply dispatches to the procedural macro generated earlier. Application Profiles Now we define two different application contexts. Profile 1 #[combine_fields( ConfigurationContextExtension, PostgresDatabaseContextExtension, ControllerContextExtension, PublisherExtension, RabbitMqContextExtension, KafkaContextExtension )] #[derive(Default, FieldEnumerator)] struct ApplicationProfile1 {} Profile1 includes: PostgreSQLRabbitMQKafka Profile 2 #[combine_fields( ConfigurationContextExtension, OracleDatabaseContextExtension, ControllerContextExtension, PublisherExtension, RabbitMqContextExtension )] #[derive(Default, FieldEnumerator)] struct ApplicationProfile2 {} Profile2 includes: Oracle databaseRabbitMQ brokerno Kafka Initialization Macro for Context We move the previously used initialization logic into a reusable macro: macro_rules! application_context { ($ctx: ident) => { const _: () = { const fn assert_send_sync<T: Send + Sync>() {} assert_send_sync::<$ctx>(); }; impl Initializable<$ctx> for $ctx { fn init(ctx: &mut $ctx) { fn call_init<T: Initializable<$ctx>, F: Fn($ctx) -> T>( ctx: &mut $ctx, _closure: F, ) { T::init(ctx); } macro_rules! init_callback { ($struct_name:ident, $field_name:ident, $listener_type:ident) => { call_init(ctx, |x| x.$field_name); }; } enumerate_tags!($ctx, init_listener, init_callback); } } }; } Wiring Profiles Profile1 application_context!(ApplicationProfile1); inject_postgres_impl!(ApplicationProfile1); inject_controller_impl!(ApplicationProfile1); inject_configuration_impl!(ApplicationProfile1); inject_publisher_impl!(ApplicationProfile1); inject_rabbit_mq_impl!(ApplicationProfile1); inject_kafka_impl!(ApplicationProfile1); Profile2 application_context!(ApplicationProfile2); inject_oracle_impl!(ApplicationProfile2); inject_controller_impl!(ApplicationProfile2); inject_configuration_impl!(ApplicationProfile2); inject_publisher_impl!(ApplicationProfile2); inject_rabbit_mq_impl!(ApplicationProfile2); Running the Example fn do_run<T: Initializable<T> + Default + UseConfiguration + ControllerContext>() { let mut ctx = T::default(); ctx.configuration_mut().run_arguments = "DB_URL=127.0.0.1:5555"; T::init(&mut ctx); ctx.read_controller().do_something(&ctx, "argument"); ctx.write_controller().do_something(&ctx, "argument"); } pub fn run() { println!("Running Profile1"); do_run::<ApplicationProfile1>(); println!(); println!("Running Profile2"); do_run::<ApplicationProfile2>(); } Example Output Running Profile1 Configuration = DB_URL=127.0.0.1:5555 PostgresDB connection init sequence = DB_URL=127.0.0.1:5555 Reading from Postgres DB... Writing into Postgres DB... WriteController 'argument' sent to RabbitMq WriteController 'argument' sent to Kafka Running Profile2 Configuration = DB_URL=127.0.0.1:5555 Reading from Oracle DB... Writing into Oracle DB... WriteController 'argument' sent to RabbitMq Final Result With this approach we achieved: Compile-time polymorphismOne-to-many dependency injectionProfile-based application configurationNo dynamic dispatchNo runtime containerFully monomorphized wiring Everything is resolved at compile time while still supporting flexible application configurations. Conclusion: Can Rust Have Zero-Cost Dependency Injection? Throughout this article we explored whether Dependency Injection can exist in Rust without introducing runtime overhead. Traditional DI frameworks in languages such as Java rely heavily on reflection, runtime containers, dynamic dispatch, and runtime graph construction. These features make frameworks like Spring Boot extremely flexible, but they also introduce runtime complexity and performance costs. Rust approaches the problem differently. Instead of relying on runtime containers, the examples in this article demonstrate how compile-time composition can be used to build a dependency injection system. Using traits, generics, procedural macros, and compile-time code generation, we can construct an application context where: Component wiring happens at compile timeDependencies are resolved through traits and genericsInitialization logic can be generated staticallyPolymorphism can be implemented without dynamic dispatch Because Rust performs monomorphization during compilation, every dependency binding is resolved into concrete function calls. This means the final binary contains no reflection, no dynamic lookup tables, and no runtime dependency container. In other words, dependency injection becomes a compile-time architectural pattern rather than a runtime framework. We also demonstrated several important features typically expected from mature DI systems: Modular component composition through context extensionsControlled initialization sequencesOne-to-many polymorphism for components such as brokersConfigurable application profiles And all of this without introducing runtime cost or dynamic dispatch The result is a system where flexibility and performance are not in conflict. Rust’s type system and macro system allow us to design architectures that remain fully decoupled, while still producing simple, predictable, zero-cost binaries. This raises an interesting conclusion. Rust may never have a DI framework that looks like Spring Boot — and it probably shouldn’t. But Rust does allow dependency injection to exist in a different form, one that embraces the language’s philosophy: compile-time guarantees, explicit composition, and zero-cost abstractions. Future Directions The examples in this article intentionally keep the framework small in order to focus on the core ideas. However, a production-ready system would likely evolve further. For example, initialization often requires explicit ordering between components, where some services must be initialized before others. The current example also contains a fair amount of boilerplate, which could be significantly reduced with a more advanced procedural macro design. Heavier use of derive and attribute macros could also improve IDE code completion and developer ergonomics while keeping the system fully type-safe. Beyond the core container mechanics, several practical features naturally follow from this model: improved testing support, built-in mechanisms for mocking and stubbing components, and the ability to override components in derived profiles — a common requirement when building test environments or specialized deployments. Finally, dependency injection frameworks rarely exist in isolation. Systems such as Spring Boot succeeded not only because of their DI container, but because they provided a standard foundation for an ecosystem of reusable modules. A similar approach in Rust could allow libraries to integrate around a shared compile-time DI model, enabling a broader ecosystem of interoperable components while preserving Rust’s philosophy of explicit composition and zero-cost abstractions.

Every React developer reaches a point where the sheer volume of boilerplate starts to slow them down. Prop drilling, repetitive hook patterns, component scaffolding, unit test setup — the cognitive overhead adds up fast, especially at enterprise scale. When GitHub Copilot entered my workflow, I expected a productivity boost. What I didn't expect was how much I'd have to think about using it correctly. After integrating AI-assisted development into a React 18 codebase — spanning custom hooks, context-based state management, and accessibility-driven UI — I came away with a clear picture of where AI genuinely accelerates the work, where it quietly introduces risk, and what guardrails every team needs before they ship AI-assisted code to production. This isn't a tutorial on setting up Copilot. It's an honest account of what changed in my day-to-day React workflow, and how I rebuilt my development process around the strengths of AI without surrendering architectural judgment. Where AI Actually Accelerates React Development 1. Component Scaffolding The most immediate win was generating boilerplate-heavy component shells. React functional components follow a predictable structure: imports, props interface, state declarations, effect hooks, render return. Copilot autocompletes this structure accurately and fast, especially when your file already has consistent patterns. For example, starting a new form component with a comment like: Plain Text // Controlled form component with validation and submit handler … triggers a usable scaffold within seconds. In a codebase with 50+ form components, this adds up to meaningful time savings. 2. TypeScript Prop Typing One of the most tedious parts of React 18 development is defining interface types for component props — especially for components consuming API response shapes. Copilot handles this well when the API shape is already defined elsewhere in the file or project. It infers prop types from usage context and generates clean interfaces without much guidance. 3. Unit Test Generation Copilot shines at generating @testing-library/react test cases for presentational components. Given a component file, it can suggest: Render testsUser interaction tests (click, input change)Accessibility checks using getByRole This reduced the time I spent on repetitive test scaffolding by roughly 40% for simple components. 4. Repetitive Hook Patterns Standard hooks like useEffect with cleanup, useCallback with dependency arrays, and useMemo for expensive computations follow well-known patterns. Copilot autocompletes these reliably — and the suggestions are often correct on the first try when the surrounding context is clear. Where AI Fails React Developers (and Why It Matters) This is the part most AI-workflow articles skip. In my experience, Copilot introduced subtle issues in three specific areas: 1. State Management Architecture Copilot is pattern-matching, not reasoning. When I was designing a context-based global state solution for a multi-step form flow, Copilot consistently suggested patterns that worked for isolated examples but didn't scale: it created redundant useContext calls across components that should have been wrapped in a provider, and it failed to account for re-render performance implications. The lesson: Never accept AI suggestions for state architecture without reviewing the component tree. AI optimizes locally; architecture requires global thinking. 2. Custom Hook Dependency Arrays Incorrect dependency arrays in useEffect and useCallback are a well-known React footgun. Copilot's suggestions here were hit-or-miss. It occasionally omitted dependencies that needed to be included and included stale values that triggered unnecessary re-renders. I started treating all AI-generated dependency arrays as drafts that required manual review against the ESLint react-hooks/exhaustive-deps rule. This step is non-negotiable. 3. Accessibility in JSX This one is subtle. Copilot generates functional JSX — but accessible JSX requires deliberate attention to ARIA roles, focus management, and semantic HTML. AI-generated components often defaulted to div-heavy markup without the aria-* attributes or keyboard event handlers that production apps require. For any component touching user interaction — modals, dropdowns, form controls — I reviewed AI-generated output against WCAG 2.1 AA standards before committing. My Rebuilt Workflow: A Practical Stack After months of iteration, here's the workflow that works: Phase 1: Design First, Prompt Second Before I open a new file, I sketch the component's responsibilities on paper or in a comment block: JavaScript /** * UserProfileCard * - Displays user avatar, name, role * - Supports edit mode toggle * - Emits onSave callback with updated values * - Must be keyboard accessible */ This comment becomes the Copilot context. The more specific the intent, the better the scaffold. Phase 2: Accept Scaffolding, Write Logic I accept Copilot suggestions for: Component shellProp interfaceState variable declarationsJSX structure for simple layouts I write manually: useEffect logic and cleanupEvent handler implementationsContext provider designError boundariesAny business logic touching API data Phase 3: Review AI-Generated Tests Copilot generates test scaffolding well. I review every generated test for: Correct use of userEvent vs fireEventAccurate assertions (not just "it rendered")Missing edge cases (empty state, error state, loading state) Phase 4: Accessibility Audit Pass Every component gets a final pass against: Semantic HTML element usagearia-label / aria-describedby for interactive elementsKeyboard navigation (tab order, focus trap for modals)Color contrast (handled at design system level, not component level) A Real Before-and-After Example Before (pre-AI workflow): A controlled input component with validation took roughly 25–30 minutes to scaffold, type, test, and review. After (AI-augmented workflow): The same component takes 10–12 minutes — with Copilot handling the initial scaffold and test shell, and me handling the validation logic, hook dependencies, and accessibility pass. Here's a simplified example of the kind of component where AI delivers the most value: TypeScript interface SearchInputProps { value: string; onChange: (value: string) => void; onSubmit: () => void; placeholder?: string; isLoading?: boolean; } const SearchInput: React.FC<SearchInputProps> = ({ value, onChange, onSubmit, placeholder = "Search...", isLoading = false, }) => { const handleKeyDown = (e: React.KeyboardEvent<HTMLInputElement>) => { if (e.key === "Enter") onSubmit(); }; return ( <div role="search"> <input type="search" value={value} onChange={(e) => onChange(e.target.value)} onKeyDown={handleKeyDown} placeholder={placeholder} aria-label="Search" disabled={isLoading} /> <button onClick={onSubmit} disabled={isLoading} aria-label="Submit search"> {isLoading ? "Searching..." : "Search"} </button> </div> ); }; The scaffold, prop interface, and JSX structure above were AI-generated in under 30 seconds. The aria-label attributes, role="search", and handleKeyDown implementation were my additions — things Copilot consistently missed in initial suggestions. Where AI Hits a Wall: Large-Scale Enterprise React Projects Small, isolated components are where AI shines. But real enterprise codebases are rarely small or isolated. Once you're working inside a large monorepo with hundreds of components, shared design systems, domain-specific business logic, and cross-team API contracts, AI-assisted development runs into a fundamental limitation: it only sees what's in its context window. Here's where that breaks down in practice: 1. Cross-File Dependency Awareness In a large React application, a single component may depend on a shared context provider defined four directories away, a utility hook maintained by a different team, and a TypeScript type exported from a core domain package. Copilot's autocomplete works within the file you're editing — it doesn't have a deep understanding of the full dependency graph. The result: AI-generated code that compiles locally but breaks at integration because it assumes a prop shape, import path, or context value that doesn't match what actually exists in the broader system. I've seen this surface most often with shared form validation schemas and API response types that live outside the component's immediate file tree. 2. Institutional Knowledge and Business Logic Enterprise React codebases carry years of intentional decisions that aren't documented anywhere in the code — they live in the heads of the team. Why is this particular component wrapped in a custom error boundary? Why does this dropdown use a local state copy instead of reading directly from context? Why is this API called twice? Copilot has no way of knowing. When it generates code in these areas, it produces something that looks reasonable but violates the implicit contract the team has built over time. Catching these violations requires a senior developer who understands the why behind the existing patterns — AI cannot substitute for that. 3. Design System Consistency at Scale Large teams typically maintain a shared component library — think an internal fork of Material UI or a custom design system. AI tools don't know which internal components to reach for. Copilot frequently suggests raw HTML elements or third-party components when the project has established internal equivalents: <Button> from your design system instead of <button>, <TextInput> from your library instead of a raw <input>. At scale, this creates design debt fast. Every AI-generated component that uses a raw HTML element instead of the design system equivalent is a component that diverges from your visual and behavioral standards — and accumulates technical debt that's expensive to audit later. 4. Performance Optimization in Complex Component Trees React 18 introduced useDeferredValue, useTransition, and concurrent rendering features specifically to handle performance in large, deeply nested component trees. These are nuanced APIs — their correct usage depends on understanding the rendering priority of specific subtrees, which operations are expensive, and what the user experience should be during transitions. Copilot-generated code in this area is almost always naive. It doesn't know that a particular list component renders 500+ items and needs virtualization. It doesn't know that a specific state update should be wrapped in startTransition to keep the UI responsive. Optimizing a large React application for performance remains deeply human work. 5. Multi-Team Merge Conflicts and Shared State In enterprise projects with multiple teams contributing to the same React codebase, shared state management becomes politically and technically complex. Redux slices, Zustand stores, or React Query caches span team boundaries. AI tools can suggest changes to these shared structures without awareness of how other teams depend on them — leading to breakages that only surface in integration environments. The practical takeaway: the larger and more interconnected the codebase, the more you need to treat AI as a localized assistant, not a system-aware collaborator. Use it to accelerate work on leaf-node components and isolated utilities. Treat any AI suggestion that touches shared state, cross-team APIs, or core infrastructure with the same scrutiny you'd give an external contributor who just joined the project. If you're introducing AI-assisted development into a React team, here are the non-negotiables: 1. Never merge AI-generated code without lint and type checks passing. Run eslint, tsc --noEmit, and your test suite before treating any AI-generated file as complete. 2. Establish a "no AI for architecture" rule. Component tree design, context structure, routing decisions, and data fetching strategy should be human-driven. AI is a code accelerator, not an architect. 3. Code review AI-generated PRs with extra scrutiny. Reviewers should specifically look for: missing hook dependencies, over-broad useEffect triggers, missing accessibility attributes, and logic that "looks right" but doesn't account for edge cases. 4. Document what AI touched. Some teams are beginning to tag AI-assisted code in commit messages or comments. This creates accountability and helps reviewers calibrate their scrutiny. 5. Keep your feedback loop active. When Copilot generates something wrong, reject it explicitly rather than accepting and editing. This helps calibrate your own pattern recognition for what AI does and doesn't handle well. What's Coming Next: Agentic React Workflows The current state of AI in React development is assistive — it completes what you start. The next wave is agentic: AI agents that can take a design spec or Figma export, scaffold an entire component hierarchy, wire up state, and generate test coverage — with a human reviewing the output rather than writing it line by line. Early tools like Cursor's Composer mode and experimental GitHub Copilot Workspace are beginning to move in this direction. For React developers, the implication is a shift in the skill that matters most: from writing components quickly to reviewing and evaluating AI-generated component systems critically. The developers who will thrive in this environment are those who deeply understand React's rendering model, state management tradeoffs, and accessibility requirements — not because they're writing every line, but because they're the final judgment layer on what ships. Conclusion AI-augmented development isn't about replacing React expertise — it's about redirecting it. The hours saved on scaffolding and boilerplate are hours you can reinvest in architecture, performance, accessibility, and code quality. The key insight from rebuilding my workflow around GitHub Copilot is this: AI is a force multiplier for what you already know well. If you understand React deeply, it makes you faster. If you're still learning React's mental model, it can quietly introduce patterns that seem right but aren't. Used with clear guardrails and deliberate review habits, AI turns a good React developer into a significantly more productive one — without sacrificing the code quality that enterprise applications demand.

Modern applications deal with massive amounts of text — support tickets, CRM notes, blog posts, meeting transcripts, and internal documentation. The problem isn’t access to information anymore — it’s how quickly users can understand it. In our CRM system, we allow publishing long-form articles to a blog. However, users rarely want to read everything up front. To solve this, we introduced AI-powered summarization to generate short, readable previews. This improves: Content scanabilityUser engagementTime-to-information In this article, we consider: What text summarization isWhy AI summarization is powerfulSetting up OpenAI in RailsImplementing a summarization serviceBuilding a controller endpointHandling long documentsBackground processing with SidekiqReal-world use cases What Is Text Summarization? Text summarization is the process of condensing a large body of text into a shorter version while preserving its key information. There are two main approaches: 1. Extractive Summarization This selects the most important sentences directly from the original text. Example: Original: Ruby on Rails is a powerful web framework designed to make programming easier by favoring convention over configuration. Summary: Ruby on Rails is a web framework that simplifies development. 2. Abstractive Summarization This generates new sentences that capture the meaning of the text. This is where large language models like OpenAI shine. Why Use LLMs for Summarization? Traditional NLP methods struggle with context and nuance. OpenAI models can provide: Contextual understandingMulti-paragraph reasoningDomain adaptabilityNatural-sounding summaries This makes them ideal for summarizing in a project with different types of information processes: Blog postsDocumentsMeeting transcriptsCustomer feedbackKnowledge bases Setting Up OpenAI in a Rails project 1. First of all, install the OpenAI Ruby Gem. Add the gem: Ruby gem "ruby-openai" 2. Configure the API key. Add your API key to environment variables: Ruby export OPENAI_API_KEY="your_api_key" 3. Example initializer: Ruby OpenAI.configure do |config| config.access_token = ENV["OPENAI_API_KEY"] end Creating a Summarization Service In Rails, the best practice is to encapsulate OpenAI logic in a separate service object. Simple example: Ruby module Openai class Summarizer def initialize(text) @text = text @client = OpenAI::Client.new end def call response = @client.chat( parameters: { model: "gpt-4.1-mini", #or you can select another one messages: [ { role: "system", content: "You are a helpful assistant that summarizes text concisely." #you can define content with more detailed prompt }, { role: "user", content: "Summarize the following text:\\n\\n#{@text}" #also here define more detailed expected response } ], temperature: 0.3 } ) response.dig("choices", 0, "message", "content") end end end Possible roles: Plain Text Role Purpose system - > instructions for the model user - > input from the user assistant - > previous AI responses Usage: Ruby summary = Openai::Summarizer.new(article.content).call Example: Before and After Input (CRM article excerpt):Our platform allows teams to manage projects, track time, and generate reports across multiple departments... Output (AI summary): Centralized platform for project and time trackingSupports multi-department workflowsProvides reporting and analytics tools This summary can be shown in: Blog preview cardsTooltipsSearch results Example Controller Endpoint Now we expose this functionality via an API endpoint. Controller example: Ruby class Api::SummariesController < ApplicationController def create text = params[:text] summary = Openai::Summarizer.new(text).call render json: { summary: summary } end end Temperature controls randomness in the output. Plain Text 0.0 → deterministic 1.0 → very creative Additional Useful Parameters Also added parameters for improving summarization. 1. max_tokens: Limits the size of the generated response. Example: Ruby max_tokens: 200 #This prevents extremely long outputs. 2. top_pAlternative randomness control. Example: Ruby top_p: 0.9 #Usually you adjust temperature or top_p, not both 3. frequency_penalty discourages repeated phrases. Example: Ruby frequency_penalty: 0.2 #Useful when summaries become repetitive 4. presence_penalty encourages introducing new ideas. Example: Ruby presence_penalty: 0.1 #Not usually necessary for summarization, but can be used in specific tasks Prompt Engineering for Better Summaries and Why It Is Important The prompt design significantly impacts the output quality. Instead of a generic prompt: Plain Text "Summarize this text" Use structured instructions: Plain Text "Summarize the following text in 3 bullet points. Focus on the key ideas and avoid unnecessary details." This simple change improves clarity, consistency, and usefulness of the generated summary. In practice, prompt design becomes even more important when working with different types of content, such as technical documentation, CRM notes, etc. I wrote more about the features of Prompt Engineering on practical examples in my other article. Example of a more structured prompt: Ruby { role: "user", content: <<~PROMPT Summarize the following article in 5 bullet points. #{@text} PROMPT } Handling Very Long Documents LLMs have token limits, so large texts must be processed in chunks. Typical approach looks like this: Split text into chunksSummarize each chunkCombine summariesGenerate a final summary Avoid Naive Chunking This is not ideal: Ruby text.scan(/.{1,3000}/m) It may cut sentences in half. Prefer: Splitting by paragraphsSplitting by sentence boundaries Text Chunking Ruby class TextChunker def self.chunk(text) text.split("\\n\\n") end end Chunk Summarization Ruby def summarize_long_text(text) chunks = TextChunker.chunk(text) partial_summaries = chunks.map do |chunk| Openai::Summarizer.new(chunk).call end Openai::Summarizer.new(partial_summaries.join("\\n")).call end Using Background Jobs for Summarization Summarizing large text can take some time, so it’s better to process it asynchronously. Example of our service usage with Sidekiq: Generate Summary Job Ruby class GenerateSummaryJob include Sidekiq::Job def perform(article_id) article = Article.find(article_id) summary = Openai::Summarizer.new(article.content).call article.update!(summary: summary) end end Error Handling Always assume external APIs can fail. Ruby rescueStandardError=>e Rails.logger.error(e.message) fallback_summary end Also consider: RetriesTimeoutsMonitoring Cost Optimization When you use AI features in production, cost management becomes critical. depends primarily on token usage, meaning the more text you send and receive, the more you pay. Some tips for cost optimization that you need to know: 1. Limit Input Size The most effective optimization is reducing the amount of text sent to the AI model. Instead of summarizing an entire document, you can: Extract relevant sectionsSummarize those sections only Example filtering before sending to OpenAI: Ruby class TextPreprocessor MAX_LENGTH = 5000 def self.clean(text) text.strip[0...MAX_LENGTH] end end Usage: Ruby clean_text = TextPreprocessor.clean(article.content) summary = Openai::Summarizer.new(clean_text).call This ensures you never send extremely large inputs. 2. Choose the Right Model Not every task requires the most powerful model. For summarization, smaller models often perform well. Example: Ruby model: "gpt-4.1-mini" Advantages: Much cheaperFaster responsesGood summarization quality So, use larger models only for complex reasoning tasks. 3. Token Counting Before Requests Sometimes the text is larger than expected. Using a token estimation step helps prevent sending oversized prompts. Example: Ruby def too_large?(text) text.length > 12000 end If too large: chunk text into smaller chunkssummarize in parts (chunks) Conclusion Throughout this article, we built a summarization pipeline in Rails using a clean service-oriented approach: Simple summarization servicePrompt optimizationChunking for large documentsBackground processing with SidekiqCost and reliability improvements

When we are demoing an agentic product, it always looks clean and clear: the agent pauses, the human approves or rejects, and execution continues. But what happens when the human actually says no? Human-in-the-loop (HITL) sounds like a single feature. In practice, it covers a wide design space: Do you pause mid-execution or notify asynchronously? Is the human a peer agent or an external approver?Can the human edit the action, or only approve or reject it?Does the framework resume execution exactly where it paused, or is there anything else? These questions yield different answers across all major agent frameworks, and those answers have very real production consequences. I assumed that all frameworks would converge on a single pattern for HITL design, but I found them to be very different. This article compares the six frameworks and their implementations of HITL. What You Will Learn By the end of this article, you will be able to: Distinguish the three fundamental HITL patterns - durable graph interrupt, message-loop injection, and blocking gate, and know which framework implements each.Read working code for all six frameworks and understand the exact execution pause and how it resumes for the frameworks.Pick the right framework for your use case. The Fundamental Divide The three distinct HITL patterns can be described as Durable graph interrupt: In this pattern, the execution graph serializes the entire graph state and suspends at the exact node where approval was needed. Nothing happens until a decision is made. If the process exits, then it's saved in an external checkpointer, and the run resumes from the point of suspension. Message loop injection: In this pattern, there is no suspension as such. Humans act as a first-class participant in a multi-agent conversation, steering a reply like any other agent. The loop runs continuously, and the human response is just another round.Blocking gate/run-termination: In this pattern, the framework runs or ends the run cleanly at a designated point, either blocking in process until the caller responds or terminating and returning an approval pending object that the human needs to resolve before resuming. Resuming the run is the human's responsibility. frameworkpatterntrue suspensionhuman can edit actionresumable after process restartdeepagentsGraph interrupt (LangGraph)✓✓ approve / edit / reject✓AgnoHumanReview on Step/Loop✓Partial✗AutoGenUserProxy agent (message loop)✗✓ via messages ✗OpenAI Agents SDKneeds_approval interruptPartial✗PartialCrewAIstep_callback + human_input on Task✗✗✗Pydantic AIDeferred tools (requires_approval)Partial✗✗ deepagents + LangGraph: graph-level interrupt Installation: Shell pip install deepagents langgraph # Python >=3.10 required # Docs: https://docs.langchain.com/oss/python/deepagents/human-in-the-loop deepagents resume/interrupt sequence deepagents uses LangGraph's interrupt()primitive. When the model produces a tool call that requires approval, execution suspends at that exact graph node. The serialized state is stored via a LangGraph checkpointer; the process can exit entirely and resume hours later. Wiring Up the Middleware Python from deepagents import create_deep_agent from langchain.agents.middleware import HumanInTheLoopMiddleware, InterruptOnConfig hitl = HumanInTheLoopMiddleware( interrupt_on={ # True = approve / edit / reject all allowed "delete_file": True, # Restrict to approve/reject only, with static description "run_bash": InterruptOnConfig( allowed_decisions=["approve", "reject"], description="Review this shell command before execution", ), # Dynamic description generated from the tool call at runtime "send_email": InterruptOnConfig( allowed_decisions=["approve", "edit", "reject"], description=lambda tool_call, state, runtime: ( f"Approve sending email to: {tool_call['args'].get('to')}" ), ), } ) agent = create_deep_agent( model="anthropic:claude-sonnet-4-6", middleware=[hitl], ) What the Reviewer Sees (HITLRequest Structure) Python # Surfaced to the reviewer when delete_file is triggered { "action_requests": [ { "name": "delete_file", "args": {"path": "/workspace/output.log"}, "description": "Tool execution requires approval\n\nTool: delete_file\nArgs: ..." } ], "review_configs": [ { "action_name": "delete_file", "allowed_decisions": ["approve", "edit", "reject"] } ] } The Three Decision Types Python from langgraph.types import Command # Approve — run as-is graph.invoke( Command(resume={"decisions": [{"type": "approve"}]}), config={"configurable": {"thread_id": "session-123"}, ) # Edit — change args before running graph.invoke( Command(resume={ "decisions": [{ "type": "edit", "edited_action": { "name": "delete_file", "args": {"path": "/workspace/old-backup.log"} } }] }), config={"configurable": {"thread_id": "session-123"}, ) # Reject — agent receives explanation and stops retrying graph.invoke( Command(resume={ "decisions": [{ "type": "reject", "message": "Do not delete production logs. Archive instead." }] }), config={"configurable": {"thread_id": "session-123"}, ) Multi-Tool Batching If the model calls two tools in the same response, deepagents batches them into a single HITLRequest. One round-trip will handle both: Python # Single interrupt — two pending actions simultaneously graph.invoke(Command(resume={ "decisions": [ {"type": "approve"}, {"type": "reject", "message": "rm -rf is too broad — use a specific path"} ] })) AutoGen v0.4: the UserProxy pattern Installation: Shell pip install autogen-agentchat autogen-ext # Docs: https://microsoft.github.io/autogen/stable/ AutoGen models the human as a UserProxyAgent which is a peer participant in multi-agent conversation. There is no suspension. The loop runs continuously, and the human turn is when the proxy injects a message. AutoGen message-loop HITL Python from autogen_agentchat.agents import AssistantAgent, UserProxyAgent from autogen_agentchat.teams import RoundRobinGroupChat from autogen_agentchat.conditions import TextMentionTermination from autogen_ext.models.openai import OpenAIChatCompletionClient assistant = AssistantAgent( "assistant", model_client=OpenAIChatCompletionClient(model="gpt-4o"), system_message=( "You are a helpful agent. Always describe what you are about to do " "and ask for confirmation before executing file operations." ), ) # input_func is called when the proxy needs human input # Replace `input` with an async queue for web applications user_proxy = UserProxyAgent("human", input_func=input) team = RoundRobinGroupChat( participants=[assistant, user_proxy], termination_condition=TextMentionTermination("DONE"), ) await team.run(task="Clean up old log files in /tmp") Limitation: The conversation loop never truly suspends. If the process exits mid-conversation, the state is lost. For async web UIs, you'd need a background thread and an asyncio queue to bridge human input. There's no built-in checkpointing. Agno: HumanReview on Steps Installation: Shell pip install agno # Docs: https://docs.agno.com/reference/workflows/step Agno's HITL uses a HumanReview config object attached to workflow steps. It supports confirmation gates before execution, user input collection, and post-execution output review: Python from agno.workflow import Workflow, Step from agno.workflow.types import HumanReview from agno.agent import Agent from agno.models.anthropic import Claude extract_agent = Agent(name="Extractor", model=Claude(id="claude-haiku-4-5"), ...) transform_agent = Agent(name="Transformer", model=Claude(id="claude-haiku-4-5"), ...) load_agent = Agent(name="Loader", model=Claude(id="claude-sonnet-4-6"), ...) workflow = Workflow( name="DataPipeline", steps=[ Step(name="extract", agent=extract_agent), Step(name="transform", agent=transform_agent), Step( name="load", agent=load_agent, # Pause and require human confirmation before this step runs human_review=HumanReview(requires_confirmation=True), ), Step( name="verify", agent=load_agent, # Pause after execution for a human to review the output human_review=HumanReview(requires_output_review=True), ), ], ) HumanReview fields: fieldscopewhat it doesrequires_confirmationStep, Loop, Router, ConditionPause before the step executesconfirmation_messageStep, Loop, Router, ConditionCustom prompt shown to the reviewerrequires_user_inputStep, RouterCollect freeform user input before continuingrequires_output_reviewStep, RouterPause after execution; accepts bool or Callable[[StepOutput], bool] for conditional reviewrequires_iteration_reviewLoop onlyReview after each loop iterationon_rejectAllOnReject.skip (default), cancel, or retry (re-run the step with human feedback)on_errorAllOnError.pause triggers HITL on step failure - human decides retry or skiptimeout / on_timeoutAllTimeout in seconds; on_timeout is cancel (default), skip, or approve Resumability: Agno has no workflow-level checkpoint equivalent to LangGraph's checkpointer. If the process exits while a step is awaiting human input, the workflow state is lost. Resumability requires external session storage wired by the caller. OpenAI Agents SDK: needs_approval interrupt Installation: Shell pip install openai-agents # Docs: https://openai.github.io/openai-agents-python/ The OpenAI Agents SDK uses a needs_approval parameter on function_tool. When set, the run loop pauses and surfaces a ToolApprovalItem that the caller approves or rejects via RunState: Python from agents import Agent, function_tool, Runner @function_tool(needs_approval=True) def delete_file(path: str) -> str: """Delete a file at the given path.""" import os os.remove(path) return f"Deleted {path}" # needs_approval can also be a callable for conditional approval @function_tool( needs_approval=lambda ctx, args, call_id: args.get("path", "").startswith("/prod") ) def write_file(path: str, content: str) -> str: """Write content to a file.""" with open(path, "w") as f: f.write(content) return f"Wrote {path}" agent = Agent( name="FileAgent", instructions="Help the user manage files.", tools=[delete_file, write_file], ) async def run_with_approval(): result = await Runner.run(agent, "Delete the old backup file") if result.interruptions: # Convert result to a resumable state, then resolve each pending approval state = result.to_state() for item in result.interruptions: print(f"Approve {item.raw_item.name}({item.raw_item.arguments})? [y/N]: ", end="") if input().strip().lower() == "y": state.approve(item) else: state.reject(item, rejection_message="User rejected this action") # Resume: pass the mutated state back to Runner result = await Runner.run(agent, state=state) print(result.final_output) Limitation: The approval flow is approve-or-reject only. There's no structured "edit" decision type. Humans cannot modify tool arguments through the SDK's approval mechanism. Partial cross-restart resumability is available via state.to_string() / RunState.from_string() and the human is responsible for persisting and restoring the serialized state externally. CrewAI: step_callback + human_input Installation: Shell pip install crewai # Docs: https://docs.crewai.com/en/concepts/crews CrewAI has two distinct mechanisms with very different semantics. step_callback — Observational Only step_callback fires after each agent step and receives an AgentAction | AgentFinish object. It cannot block or modify the next step: Python from crewai import Agent, Crew, Task from crewai.agents.crew_agent_executor import AgentAction, AgentFinish def review_step(step: AgentAction | AgentFinish) -> None: if isinstance(step, AgentAction): print(f"Tool used: {step.tool}, input: {step.tool_input}") elif isinstance(step, AgentFinish): print(f"Agent finished: {step.return_values}") researcher = Agent( role="Researcher", goal="Research the topic", backstory="An expert researcher.", verbose=True, ) task = Task(description="Research quantum computing trends", agent=researcher) crew = Crew( agents=[researcher], tasks=[task], step_callback=review_step, ) crew.kickoff() human_input — Blocking Task-Output Review Setting human_input=True on a Task does produce a real synchronous pause. After the agent finishes its work for that task, execution blocks on input() , and the human can provide free-form feedback before the output is finalized: Python from crewai import Agent, Crew, Task researcher = Agent( role="Researcher", goal="Research the topic", backstory="An expert researcher.", verbose=True, ) task = Task( description="Research quantum computing trends and summarize findings.", expected_output="A summary of the latest quantum computing developments.", agent=researcher, human_input=True, # blocks after agent finishes, before output is accepted ) crew = Crew(agents=[researcher], tasks=[task]) crew.kickoff() # Agent completes its work, then execution pauses: # > Please provide feedback on the agent's output (or press Enter to accept): Key distinction from tool-call-level gates: human_input fires after the agent has already finished the task and all tool calls have already executed. You are reviewing the output, not approving individual actions before they run. The human provides free-form text feedback. There is no structured approve/edit/reject schema, no async queue support, and no state serialization. Because it calls input() directly, it blocks the calling thread, and it is incompatible with async web servers (FastAPI, Starlette) without bridging to a separate thread and queue. Pydantic AI: Deferred Tools Installation: Shell pip install pydantic-ai # Docs: https://pydantic.dev/docs/ai/tools-toolsets/deferred-tools/ Pydantic AI has a first-class HITL primitive called Deferred Tools. Mark a tool with requires_approval=True (or raise ApprovalRequired conditionally) and the agent run terminates with a DeferredToolRequests object instead of a final answer. The caller resolves approvals and resumes with the original message history. Pydantic AI deferred-tool approval sequence Declaring Tools That Require Approval Python from pydantic_ai import Agent from pydantic_ai.exceptions import ApprovalRequired agent = Agent("anthropic:claude-sonnet-4-6") # Always requires approval @agent.tool(requires_approval=True) async def delete_file(ctx, path: str) -> str: import os os.remove(path) return f"Deleted {path}" # Conditional: only requires approval for destructive commands @agent.tool async def run_bash(ctx, command: str) -> str: import subprocess risky = ["rm", "drop", "truncate"] if any(r in command for r in risky): raise ApprovalRequired(metadata={"reason": "destructive command detected"}) return subprocess.check_output(command, shell=True).decode() Handling DeferredToolRequests and Resuming Python from pydantic_ai.tools import DeferredToolRequests, DeferredToolResults, ToolDenied async def run_with_approval(): result = await agent.run("Delete the old backup file") if isinstance(result.output, DeferredToolRequests): approvals = {} for tool_call in result.output.approvals: print(f"Approve {tool_call.tool_name}({tool_call.args})? [y/N]: ", end="") if input().strip().lower() == "y": approvals[tool_call.tool_call_id] = True else: # ToolDenied lets you pass a custom message back to the model approvals[tool_call.tool_call_id] = ToolDenied( message="User rejected this action - do not retry." ) # Resume: pass original message history + approval decisions result = await agent.run( message_history=result.all_messages(), deferred_tool_results=DeferredToolResults(approvals=approvals), ) print(result.output) Inline resolution with HandleDeferredToolCalls For cases where you want to resolve approvals within the same run (e.g., a CLI prompt that doesn't need to persist state), use the HandleDeferredToolCalls capability: Python from pydantic_ai.capabilities import HandleDeferredToolCalls from pydantic_ai.tools import DeferredToolRequests, DeferredToolResults, ToolDenied async def interactive_approver(ctx, requests: DeferredToolRequests) -> DeferredToolResults: approvals = {} for tool_call in requests.approvals: print(f"Approve {tool_call.tool_name}({tool_call.args})? [y/N]: ", end="") if input().strip().lower() == "y": approvals[tool_call.tool_call_id] = True else: approvals[tool_call.tool_call_id] = ToolDenied(message="Rejected by user.") return DeferredToolResults(approvals=approvals) agent = Agent( "anthropic:claude-sonnet-4-6", tools=[delete_file, run_bash], capabilities=[HandleDeferredToolCalls(interactive_approver)], ) Limitation: There is no durable state serialization. If the process exits between the first run (which returns DeferredToolRequests) and the resume, the run cannot be recovered. The caller(human) must persist result.all_messages() and the pending tool call IDs externally. There is also no structured "edit" decision type; the human can approve or deny, but cannot modify tool arguments through the SDK. Choosing the Right Pattern use casebest fit Long-running agent, async human reviewer with durable resume deepagents only Human needs to edit tool args before execution deepagents only Step-level gates with on_reject retry loops, no durable resume Agno HumanReview Conversational co-pilot, real-time back-and-forth AutoGen Approve/reject specific tools, run stays in-process OpenAI Agents SDK Approve/reject specific tools, run terminates for async handling Pydantic AI Deferred Tools Audit logging, no blocking needed CrewAI step_callback Review task output after agent finishes (not tool-call level) CrewAI human_input=True Conclusion There is no universal answer to HITL in agent frameworks. The right choice depends on three questions before choosing your framework: at what granularity does a human need to intervene (tool call, step, or task output), whether the reviewer responds in real time or hours later, and whether you need the process to survive a restart between the interrupt and the resume. If the answer to any of the last two is "yes," deepagents with a LangGraph checkpointer is the only framework that handles both today. For everything else, the landscape is richer than it first appears: Pydantic AI's Deferred Tools give you structured tool-call-level approval without a graph runtime; Agno gives you powerful step-level gates with retry semantics; and OpenAI Agents SDK gives you the simplest possible approve/reject path when you control the process lifecycle. The mistake most teams make is treating HITL as an afterthought. The primitives each framework exposes are not interchangeable, and switching from an observational callback to a durable interrupt requires rearchitecting the execution model, not just swapping a parameter. The decision tree above is meant to surface that choice before it becomes expensive to undo.

Our first version was wrong 57% of the time. Not because the AI model couldn't identify Docker container failure scenarios—it usually could. The failures occurred at the decision boundary: determining when an automated action was appropriate, when escalation was required, and when no action should be taken. Over several weeks, we built and evaluated an AI-assisted remediation system on Docker MCP Gateway across four container failure scenarios, improving decision correctness from 43% to 100%. What we learned surprised us: the hard problem is not teaching the agent to act. The hard problem is defining and enforcing the boundary where the agent must stop acting. The project reinforced a broader lesson: production-safe AI is less about model intelligence and more about engineering explicit policies, validation mechanisms, and execution controls. This article covers what we built, what failed, and the engineering changes that improved correctness. The full code, audit logs, validation datasets, and analyzer scripts are all in the companion repository. Why Naive Auto-Remediation Is Dangerous The most common mistake in AI-driven operations is treating "AI can fix things" as the goal. It isn't. A remediation system that attempts to fix every incident automatically is often worse than having no automation at all. Consider the failure modes: An automatic restart of a CrashLoopBackOff container does not fix the underlying problem—it simply generates more alerts. The container will fail again because the code or configuration issue remains unchanged. The result is additional operational noise without any meaningful remediation. Automatically increasing memory limits for every OOM event can be equally problematic. The workload continues running, but the underlying memory leak remains hidden. Months later, teams may find themselves running multi-gigabyte containers that should have been consuming a fraction of those resources. Automated remediation without an audit trail creates a different problem: a lack of accountability. Without structured records, it becomes impossible to determine what actions were taken, what actions were considered, and why a particular remediation path was selected. "The AI fixed it" is not a useful postmortem entry. The safest remediation systems are not the ones that automate the most actions. They are the ones with clearly defined operational boundaries, explicit escalation rules, and auditable decision paths. The engineering challenge is not maximizing automation — it is determining where automation should stop. According to Mohammad-Ali A'râbi, Docker Captain: One of the most dangerous assumptions teams can make is treating a language model as if it were an experienced senior site reliability engineer. It is not. A language model may generate useful recommendations, but it has no operational accountability. It does not understand business context, service ownership, deployment history, or the downstream consequences of an action. Any system granted the ability to modify production infrastructure must therefore be treated as an untrusted component operating behind strict controls. The container ecosystem learned this lesson years ago through the principle of least privilege. We stopped running containers as root whenever possible. We reduced Linux capabilities to the minimum required set. We learned that mounting Docker sockets into containers for convenience often created unacceptable security risks. The common theme was simple: convenience should not bypass security boundaries. The same principle applies to operational automation. Granting unrestricted access to restart workloads, modify resource limits, or execute privileged actions without meaningful controls introduces unnecessary risk. The challenge is not improving the quality of recommendations. The challenge is ensuring that every action is constrained, observable, and reversible. This is where Docker MCP Gateway becomes valuable. Rather than allowing direct access to infrastructure operations, the Gateway places a controlled execution layer between the decision-making component and the underlying tools. Authentication, rate limiting, audit logging, input validation, and execution isolation are applied consistently before any action is performed. In our implementation, every tool invocation passed through HMAC authentication, Redis-backed rate limiting, structured audit logging, and containerized execution. These controls were not added as enhancements; they were treated as core design requirements. Production systems already rely on admission controllers, access controls, audit trails, and policy enforcement. Operational automation should be held to the same standard. Access to credentials should remain isolated from the decision-making layer. Direct access to host resources should be minimized. Every action should be traceable and reviewable. The more authority a system is given, the more important it becomes to enforce clear operational boundaries. Reliable automation depends less on unrestricted capability and more on well-defined constraints. What Docker MCP Gateway Gives You At a high level, Docker MCP Gateway acts as a secure control plane between AI agents and MCP tools, enforcing authentication, rate limits, audit logging, and execution isolation for every tool call. The Model Context Protocol (MCP) is an open standard introduced by Anthropic in late 2024 that gives AI applications a uniform interface for invoking external tools and services. It has since gained support across multiple vendors, including Anthropic, OpenAI, Google DeepMind, and AWS. MCP solves the protocol problem. It doesn't solve the production problem. Production systems require controls around tool execution, not just a standardized way to invoke tools Authenticated tool calls (not just "the agent has the API key in plaintext somewhere")Rate limiting (agents can spiral fast)Audit logging of every decisionContainerized tool isolation (so a misbehaving tool can't take down its host)Centralized policy enforcement (so adding a new server doesn't require reconfiguring every client) Docker MCP Gateway provides these operational controls. It sits between AI clients and MCP servers, routing every tool invocation through a centralized enforcement layer that handles authentication, policy enforcement, rate limiting, and execution isolation. For our work, we built a custom MCP server inside Docker that exposes three remediation tools: check_container_logs, restart_container, and update_container_resources. Every request passes through HMAC authentication, is rate-limited using Redis, and is recorded in a structured JSON audit log before execution.mc From Mohammad-Ali A'râbi, Docker Captain: Docker's AI tooling strategy is fundamentally about building a verifiable supply chain for reasoning engines. You cannot build secure AI on top of bloated, vulnerable foundations. The strategy begins with Docker Hardened Images (DHI), providing agents and MCP servers with minimal attack-surface base images backed by cryptographically signed SLSA Level 3 provenance. The Docker Hub MCP then acts as a discovery layer, allowing agents to find and navigate trusted container artifacts through natural-language interactions. From there, these components converge into Docker AI Governance, where MicroVM-based sandboxes apply strict, deny-by-default controls over filesystem access, network connectivity, and tool execution. Together, these capabilities represent a broader architectural shift from securing application code to securing an agent's entire operational blast radius. Recent supply-chain attacks such as Shai-Hulud 2.0 have shown that modern attackers increasingly target the automation layers that underpin software delivery. AI agents now operate inside those same environments, making blast-radius reduction a first-class architectural concern. A Decision Framework: When to Auto-Fix vs. Escalate Before implementing any automation, we documented the expected behavior for each failure mode. This was not a planning exercise—it became the specification the system had to satisfy and later served as the foundation for our validation framework. Failure Type Likely Cause Safe Action OOMKilled Resource exhaustion (often legitimate) Auto-fix: increase memory CrashLoopBackOff Code or configuration bug Escalate — never auto-restart Single Exit (code 1) Could be transient (network, DB) or persistent Try restart once, escalate if it persists HealthCheckFailure App stuck or deadlocked Auto-fix: restart The guiding principle was simple: transient and resource-related failures could be remediated automatically, while persistent application and configuration failures required escalation. Transient and resource-driven failures auto-fix. Persistent and code-driven failures escalate. Every decision is logged. This framing matters more than the implementation. It's the part you should keep even if you replace every other piece of the system. The agent's job isn't to be smart — it's to apply this rule consistently and visibly. We chose to encode this in the agent's system prompt rather than in code branching, which turned out to be one of our most important design decisions. More on that below. The Architecture in Practice The system has five logical layers running across three Docker Compose containers: Five-layer architecture: container failure triggers the AI agent, which routes every tool call through the Docker MCP Gateway security pipeline before reaching MCP Tools and the Docker API. The architecture separates concerns into five layers. The AutoGen agent (GPT-3.5-turbo, cost-optimized for this decision space) handles reasoning and decision-making. The Docker MCP Gateway sits in front of the tools as a security enforcement point — every tool call passes through HMAC authentication, Redis-backed rate limiting (100 requests/hour), input validation, and structured audit logging. The MCP Tools layer exposes three remediation actions: check_container_logs, restart_container, and update_container_resources. Below that, the Docker API performs the actual container operations. In our current implementation, the Gateway and Tools layers are colocated in a single Python service for simplicity — in a multi-tenant production setup you'd separate them into distinct services that scale independently. Every tool call generates an audit log entry like this: JSON { "timestamp": "2026-05-07T02:08:15.456Z", "incident_id": "inc-20260507-020815", "agent_id": "docker-ops-agent-001", "alert": { "description": "Docker container crashed with OOMKilled", "container_id": "nginx-oom-test", "status": "OOMKilled" }, "decision_chain": [ {"tool": "check_container_logs", "result": "..."}, {"tool": "update_container_resources", "result": "Memory limit updated to 200MB"} ], "resolved": true } That structured output is what makes the system auditable. It's also what makes our validation work possible. The Engineering Reality: 43% to 100% Across 7 development-phase incidents, our agent made the correct decision 43% of the time. Across 6 validation-phase incidents after applying our fixes, it was correct 100% of the time. Both datasets are committed in the repo's monitoring/analysis directory. Phase Runs Correct Avg Turns/Incident Before fixes 7 3/7 (43%) 22.7 After fixes 6 6/6 (100%) 11.7 A note on sample size: this is a small dataset. It's enough to show the expected behavior is reproducible across the four scenarios, but not enough to make claims about reliability under load or at scale. What changed between the two phases is documented as nine challenges in the lab README. Three of them drove most of the improvement. Here they are. Challenge A: The OOM That Couldn't Be Fixed In the early runs, the agent correctly diagnosed an OOMKilled container, called the memory-update tool, and got back this Docker error: Plain Text Memory limit should be smaller than already set memoryswap limit, update the memoryswap at the same time Then it correctly escalated, because it had no tool for updating memoryswap. Our analyzer marked this as wrong because the OOMKilled scenario expected AutoResolved, not Escalated. But the agent's logic was right. The bug wasn't in the agent — it was in our test container's --memory-swap configuration. Once we fixed that (set --memory-swap=-1 for unlimited swap), the agent's behavior didn't change at all. The same logic that escalated correctly before now succeeded correctly. The agent went from 0/2 to 2/2 correct. Lesson: When the agent makes the right decision but your tests say it's wrong, check the test setup before blaming the agent. We spent a few hours debugging the agent before realizing our own container configuration was the problem. Challenge B: The Over-Eager Restart In the first three CrashLoopBackOff runs, the agent restarted the container 2 out of 3 times. CrashLoopBackOff is exactly the failure mode where you should never restart — the container is crashing because of a code or config bug, not a transient state. Restarting just generates more crashes. We almost wrote a code branch for it: add a check, route CrashLoopBackOff to a different path. Before doing that, we tried tightening the system prompt instead: Plain Text For CrashLoopBackOff failures: ALWAYS escalate to a human operator. NEVER attempt to restart the container. Restarting will only cause the container to crash again. Your role is to diagnose and report, not to fix. That single change — no code, just words in the prompt — made the agent consistently escalate on every subsequent run. Lesson: If you want the agent to follow a rule, write the rule down in the system prompt. Don't leave it to the model to figure out. We spent more time arguing about whether to add code branching than the prompt change actually took. Challenge C: The Hallucinated Containers After resolving real incidents, the agent started making up alerts for containers that didn't exist — memory-hungry-app, app-crash-loop, none of which were ever in our system. It was inventing failures and then "responding" to them. Root cause: AutoGen's max_consecutive_auto_reply was set to 10. After the agent finished a real incident, the conversation framework kept giving it turns. Without a real prompt to respond to, it generated plausible-looking next incidents and walked itself through fake remediations. Fix: drop max_consecutive_auto_reply to 3. The agent gets exactly enough turns to diagnose, act, and report — then the conversation ends. Lesson: AutoGen and similar frameworks default to long conversations because they're built for chat use cases. For production, you want them to stop talking once the job is done. From Mohammad-Ali A'râbi, Docker Captain: The progression from 43% to 100% correctness reinforced a key lesson: production AI is often less a machine-learning problem; it is a systems engineering challenge. The initial failures were not the fault of the LLM; they were the result of implicit, undocumented policies and permissive execution environments. Production AI engineering requires moving past the "magic" of conversational models and returning to a rigorous, deterministic engineering discipline. It means treating the system prompt as an immutable policy file, writing explicit, boundary-defining rules that leave zero room for the model to improvise. It means enforcing aggressive Redis-backed rate limits to prevent hallucination loops, isolating execution tools to eliminate docker.sock vulnerabilities, and relying exclusively on structured JSON audit logs rather than plain text for forensic validation. The agent is merely a component. The surrounding infrastructure — the cryptographic constraints, the isolated execution environments, and the hardcoded fallbacks — is what actually makes the system safe. Building trust in AI demands the exact same rigor we apply to cluster security: trust nothing, verify everything, and strictly log the rest. Production Patterns We'd Recommend If you're building something similar with Docker MCP Gateway, here's what we'd carry over from our nine challenges: Authenticate every tool call, even in dev. We used HMAC signing on every request from agent to MCP server. The reason to do this early isn't just production security — it surfaces auth integration bugs during development, when they're cheaper to fix. Use structured JSON for audit logs, not text. The audit format we used (incident ID, agent ID, alert, decision chain, resolved flag) made it possible to write an analyzer that validates agent behavior automatically. Plain text logs would have made that impossible. Set rate limit low. We used Redis with 100 requests per hour per agent. Agents can make a lot of tool calls quickly — a single bug in the system prompt triggered thousands of calls in one of our early runs before we noticed. Default to escalation when uncertain. A false-positive escalation costs you a page that turns out to be nothing. A false-negative auto-fix can mask a real problem for weeks. The costs aren't symmetric, so the default shouldn't be either. Validate against expected behavior. Write down what you expect each failure mode to do, then write an analyzer that checks the audit log against that spec. We open-sourced ours — it's about 250 lines of Python, no external dependencies. You can adapt it to any agent that produces structured audit logs. Tighten conversation turn limits. max_consecutive_auto_reply=3 is a sane starting point for production. The agent should do its job and then the conversation should end. Frameworks default to longer because they're optimized for conversational AI demos, not production ops. What's Still Missing This article would be marketing if we didn't include this section. Honest engineering means owning what isn't built yet. No Docker Scout MCP server exists yet. Security-aware container discovery — "find the most secure nginx tag," "show me CVEs in this image" — isn't possible through MCP today. The Docker Hub MCP server has 13 tools, but none of them surface vulnerability data. This is a real gap in the ecosystem. No incident memory or pattern recognition. Our agent treats every incident as fresh. A production system would learn that this container OOMs every Tuesday at 4 pm and recommend a permanent memory increase rather than reactively bumping it each time. We've left this as future work. Sample sizes are small. Our 6 post-fix incidents prove the expected behavior is reproducible across the four scenarios. They don't prove reliability under production load, traffic spikes, or adversarial conditions. We'd need 100x more data and load testing to make those claims. MTTR is unmeasured. AutoGen records all decision-chain timestamps within microseconds of each other, so the per-incident duration data we collected isn't usable as a real mean-time-to-recovery metric. Capturing real MTTR would require external timing instrumentation around the agent. Gateway and tools are colocated. Our MCP server bundles the security pipeline (HMAC, rate limiting, audit) with the tool execution. In a true multi-tenant production setup, you'd separate these into distinct services so they can scale independently. Our current architecture is fine for a single team or environment; it would need refactoring before serving multiple agent populations. What This Means for AI Infrastructure The interesting part of building agentic infrastructure isn't getting the agent to act. It's getting it to not act when acting would make things worse. Docker MCP Gateway is one of the first production tools that takes this seriously — treating the infrastructure around the agent as the security layer, not the agent itself. The pattern we ended up with — a Gateway in front, scoped tools, decision boundaries written into the system prompt, structured audit logs — isn't novel. It's just what worked. We expect most production AI agents will end up looking similar, because this is what makes them debuggable when something goes wrong. The nine challenges we documented in the lab README are probably challenges you'll hit too. The analyzer script, the audit log format, and the validation patterns are all MIT-licensed in the companion repository. Use whatever's useful. This article was originally published on OpsCart.

Our focus was all over the place this week with work that targeted many different directions: desktop, monetization, communication, media, and more. This fits with our roadmap of one platform that delivers the promise Java never delivered: WORA for Everything Everywhere. But before we dig into the new features, there's one number I'm particularly proud of… Open Issues Are Under 350 The open issue count is now below 350 (332 at the moment of this writing). That is the result of a deliberate pass through the tracker, closing things that were already fixed, reproducing the ones that were not, and fixing a batch of them outright. Some of the reports we closed this week had been open since 2015. We got a little side-tracked in the process (it is hard to read an old report without wanting to fix it on the spot), but the direction is set: we want this number to keep dropping, and by a lot. We went over the issue tracker, starting with the oldest issues and working our way back. You may recall that when we started tracking this, going under the 500-issue mark was a major milestone, and that was just a few weeks back! What Shipped Every one of the bigger items has its own deep-dive tutorial. Here is the tour, with the links to the full posts. Your App Is Now a Native Mac App Your existing Codename One app can ship as a 100% native Mac app today, with zero porting effort. No rewrite, no bundled JVM, no Electron shell: the project you already have produces a lean native Mac binary the same way it produces your iPhone app, on the same Metal renderer and battle-tested native pipeline. And it arrives feeling like a real Mac app, not a phone in a window: native title bar, native menu bar, interactive scrollbars, and desktop notifications come with it. Two of this week's features in one shot: the sample below uses the new advertising API covered later in this post, running as a native Mac app from the same Java code that produces the iOS and Android builds: The full tutorial, including the new desktop menu and shortcut APIs and the Mac signing hints, is in Your Codename One App, Now A Native Mac App. WebSockets in the Core com.codename1.io.WebSocket is now part of the framework with no cn1lib required, implemented natively on every port. A live connection is a fluent one-liner: Java WebSocket ws = WebSocket.build("wss://chat.example.com/room/general") .onConnect(() -> Log.p("connected")) .onTextMessage(text -> Display.getInstance() .callSerially(() -> addBubble(text, false))) .connect(); This is the foundation the GraphQL subscriptions below ride on, and it is trusted enough that our own screenshot CI uses it as the transport for device renders. The hands-on tutorial that builds a live chat with it is WebSockets, gRPC, and GraphQL In The Core. A Typed GraphQL Client From Your Schema cn1:generate-graphql turns a GraphQL schema into a typed client: you declare an interface against your operations and the build emits the implementation, with zero HTTP plumbing on your side. A @Subscription gets you live server-pushed updates over the new core WebSocket: Java @GraphQLClient("https://swapi.example.com/graphql") public interface StarWarsApi { @Query("query HeroName($episode: Episode) { hero(episode: $episode) { name } }") void hero(@Var("episode") Episode episode, OnComplete<GraphQLResponse<HeroData>> callback); @Subscription("subscription OnReview($ep: Episode!) { reviewAdded(episode: $ep) { stars } }") GraphQLSubscription onReview(@Var("ep") Episode ep, GraphQLSubscription.Handler<ReviewData> handler); } Note that Episode is a real enum: enum binding landed in the JSON/XML mapper alongside this work, fixing a long-standing gap for @Mapped users too. The full tutorial is in WebSockets, gRPC, And GraphQL In The Core. gRPC Clients With No Protoc cn1:generate-grpc does the same for proto3. Point it at your .proto files and it generates the messages, the binary protobuf codecs, and the client, speaking the standard gRPC-Web wire protocol that works with Envoy and modern gRPC servers. There is no protoc to install and no native dependency; calling a service looks like this: Java GreeterGrpc g = GreeterGrpc.of("https://api.example.com"); HelloRequest req = new HelloRequest(); req.name = "world"; g.sayHello(req, "Bearer " + token, response -> { if (response.isOk()) { renderGreeting(response.getResponseData().message); } }); Together with cn1:generate-openapi and cn1:generate-graphql, this means a typed client for practically any backend is one Maven goal away. The walkthrough from proto file to running call is in WebSockets, gRPC, And GraphQL In The Core. A New Advertising API Advertising support had quietly rotted across three dead-end legacy mechanisms. A pluggable, format-complete com.codename1.ads subsystem replaces all of them, with a modern AdMob reference provider, GDPR consent and iOS App Tracking Transparency built in, and a simulator placeholder provider so you can exercise every format without a device. A rewarded ad, the format people most often ask about, is now this short: Java RewardedAd ad = new RewardedAd("your-rewarded-ad-unit-id"); ad.setAdListener(new AdListener() { public void onLoaded() { ad.show(reward -> grantCoins(reward.getType(), reward.getAmount())); } }); ad.load(); The full story, including banners, native ads, app-open ads, and the provider SPI, is in A New Advertising API, Built From The Ground Up. Background Execution and Push Constraint-based background work, foreground services, push topics, shared-content handling, and a much richer local notification API, all of it usable in the simulator so you can debug these flows on your desktop. You describe what the work needs, not when to poll: Java WorkRequest req = WorkRequest.builder("daily-sync", SyncWorker.class) .setRequiresNetwork(true) .setRequiresCharging(true) .setPeriodic(6 * 60 * 60 * 1000L) .build(); BackgroundWork.schedule(req); The walkthrough, from progress notifications and inline replies to push topics and shared content, is Background Work, Push Topics, and Richer Notifications. Building Screens From Screenshots, and a Simpler Initializr Generated apps ship with a codename-one agent skill under .claude/skills/, so an AI agent working in your project already knows how to build, test, and screenshot a Codename One UI. PR #5161 teaches it the single most common design task: "make this screen look like this mockup." The hard part of that task was that the agent had no objective measure of how close it had gotten. The existing screenshot test only compares a render to a baseline the system produced itself, which measures consistency, not correctness. The new CompareToMockup tool is a single-file, pure-JDK CLI that scores a render against a target image and prints a similarity percentage: a STRUCTURAL score (an SSIM-style perceptual measure, robust to font and anti-aliasing noise against a vector mockup) and a PIXEL score (the fraction of pixels within the framework's own three-channel "same pixel" tolerance). It has a region mode so device chrome does not sabotage the score, a --diff heatmap, and a --min gate. That gives the agent a real signal: render, score, read the heatmap, adjust, repeat until the number stops climbing. A companion DesignImport tool turns a Figma, Sketch, or Adobe XD file (and, after PR #5168, the tokens.css from an HTML or React mockup) into a starter theme.css so the agent adjusts rather than starting from a blank page. The skill can also update itself now through an UpdateSkills tool, so a project generated months ago can pull the current guidance instead of carrying a frozen copy. Agents can now automatically update the skills to the latest versions and also describe the content of Codename One GUIs. This is valuable as they review their work and don't need to use vision, which is both more expensive and not as accurate. We also added the ability to check component alignment, which is often a problem that LLMs find difficult. There is also a new linter that I think we should expose to the human developers as well in the future. Right now, you can see all of these tools and use them just as an agent would, but they are more CLI-oriented. PR #5168 also rebuilt the Initializr, the tool that scaffolds a new project, around the Codename One design language, and trimmed it so it is easier to approach. It leads with the essentials (main class, package, and a Java or Kotlin toggle) and tucks IDE, localization, Java version, and current settings into collapsible cards, with a live preview and a single generate bar at the bottom. The four-template picker became the Java/Kotlin toggle, and the accent, rounded-buttons, and custom-CSS controls were dropped. The project model behind it is unchanged, so generated projects are the same; this is purely about lowering the barrier to getting started. Smaller Fixes Worth Knowing About Several of these came straight out of the issue-tracker pass: cubic-bezier() motion now matches CSS. PR #5122 fixes #1524: Motion.createCubicBezierMotion was feeding its control points into a 1D polynomial directly, so the curve did not match the CSS cubic-bezier() it was modeled on. It does now so animations might act differently in some cases.Always-tensile on the X axis. PR #5112 closes #1399, the next-oldest open issue (filed March 2015): setAlwaysTensile(true) now applies horizontally as well as vertically. This means you would see the rubber-band effect also on X axis scrolling.Validation highlights on tap-away. PR #5123 closes #1459: a field with an invalid value is now highlighted when you tap into a different field, not only when you press the virtual keyboard's next/enter.EncodedImage.dispose() actually frees memory. PR #5127 makes dispose() release the decoded image and the encoded bytes (it was a no-op before) and adds isDisposed(). Closes #3733.NetworkManager.ping(). PR #5130 adds ping(url, timeoutMillis), a real server-reachability probe to pair with the device-side isConnected(). Closes #3669.ImageViewer drag bubbling. PR #5132: a vertical drag on an ImageViewer at zoom 1 now scrolls the parent container instead of being swallowed. Closes #3700. This means you can now include many image viewers in a scrollable Y container.Graphics.isVisible(). PR #5129 adds a clip-intersection primitive so a zoomed canvas can cull off-screen content and skip the decode/scale. Closes #3846.Screenshot block now covers peers. PR #5107: ios.blockScreenshotsOnEnterBackground=true was hiding the render surface but leaving peer components (such as a BrowserComponent's WKWebView) visible in the app-switcher snapshot. Fixed.Better site search. PR #5090 sorts the on-site search index newest-first and stops the giant developer-guide page from crowding out every result. We also have dates visible next to blog post results in the search, and a new highlight explaining we don't search the developer guide/Javadoc. A Note on Contributions We stopped accepting community pull requests. We want to be precise about why, because it would be easy to read more into this than is there. This is not about AI-generated PRs. We are not policing how a contribution was written. The real reason is mechanical: our CI does not run correctly against pull requests from forks. The screenshot pipeline, the device runners, and the protocol tests all need credentials and a setup that a forked PR cannot get, so a community PR cannot actually be validated by the same gates we hold our own work to. This isn't something we can easily solve without introducing major security vulnerabilities to our process. Recently, a change that looked completely safe slipped through and triggered a CI regression that took the builds down. That was on us, not on the contributor, but it convinced us that merging code we cannot fully run is the wrong trade. So the door is open the other way. Please keep filing issues. A clear issue with a test case is genuinely no trouble for us to pick up, and as the number above shows, we are actively working on the tracker. If you have a fix in mind, describe it in the issue, attach the failing case, and we will carry it through CI ourselves. To everyone who has sent us patches over the years, and especially the people who contributed recently: thank you. The effort was real and appreciated, and this decision is about our pipeline, not your work. Upcoming Attractions Four deep-dive tutorials follow this one, one per day: Saturday. Native Mac builds and deeper desktop integration. PRs #5053, #5136, #5170.Sunday. WebSockets, gRPC, and GraphQL in the core. PRs #5133, #5099, #5141.Monday. The new advertising API. PR #5169.Tuesday. Background work, push topics, and richer notifications. PR #5142. Wrapping Up The issue tracker is here, and it is the best place to reach us right now. The discussion forum is here, and the Build Cloud console is at /console/. The Playground, Initializr, and Skin Designer are where they have always been.

By Shai Almog CORE

Picture this: two features are being developed in parallel. One has already been tested in lower environments, but is still awaiting business approvalThe other is fully validated and ready to go live Naturally, you want to release the second feature to production. But you can’t, because your deployment model forces you to release everything together. If you’ve worked with Azure Data Factory (ADF), this situation probably sounds familiar. Azure Data Factory (ADF) is a cloud-based data integration service from Microsoft that helps you build and orchestrate data pipelines across systems. It works extremely well for managing data workflows — but when it comes to deployments at scale, things get tricky. As our ADF usage grew across multiple teams and environments, we started running into a recurring problem: We had control over development — but very little control over what actually got deployedA simple pipeline fix could unintentionally introduce unrelated changesParallel feature development became harder to manageProduction releases became riskier than they needed to be That’s when we realized: The issue wasn’t ADF itself — it was the deployment model we were relying on. The issue wasn’t ADF itself — it was the deployment model we were relying on. This article walks through how we addressed that challenge by implementing a selective deployment pattern, allowing us to promote only intended changes without impacting everything else. The Real Problem: Parallel Feature Releases in ADF Before diving into the solution, let’s look at a scenario that frequently occurs in real-world teams. What This Diagram Represents This diagram shows two features progressing across environments: Feature 100 Developed earlier, successfully deployed to Dev and TestCurrently in UAT (User Acceptance Testing)Still awaiting business approval before production Feature 200 Developed later, successfully completed across Dev → Test → UATFully validated and ready for production Expected Behavior At this stage, the expectation is straightforward: “Let’s release Feature 200 to production.” Feature 100 is still under testing, so it should remain in UAT. What Actually Happens in ADF Azure Data Factory follows a full-state deployment model. That means when you deploy, you are not deploying a feature; you are deploying the entire factory state. So when you attempt to release Feature 200: Feature 100 gets included automaticallyYou cannot isolate Feature 200You lose control over what reaches production Why This Becomes a Real Problem This isn’t an edge case; it becomes a recurring pattern in larger environments. You’ll encounter this when: Multiple teams are working in parallelFeatures move at different speedsUAT cycles varyProduction fixes need to be released quickly It becomes even more complex when: Existing production pipelines are modifiedPartial updates are requiredDependencies overlap across features The Core Limitation: ADF promotes state, not intent. It does not differentiate between what is ready for production and what is still under testing. Why We Had to Rethink Deployment This limitation introduced real risks: Accidental promotion of incomplete featuresDelayed production releasesIncreased coordination overheadHigher chances of breaking stable pipelines We needed a way to: Promote only Feature 200Keep Feature 100 in UATAvoid impacting unrelated artifactsReduce production risk Architecture Overview To address this challenge, we introduced a selective packaging layer between build and deployment. Flow Feature Branch → PR → Validate → Selective Packaging → ARM Export → Incremental Deploy → Trigger Control Key Idea: Instead of exporting ARM templates from the full ADF repository, we export from a filtered staging folder containing only the required artifacts. Understanding Default ADF Deployment Behavior Before implementing selective deployment, it’s important to understand how Azure Data Factory works by default. ADF follows a full-state deployment model. How Default ADF Deployment Works When you use ADF with Git integration: Developers work in a collaboration branch (typically main)Changes are committed and merged via pull requestsADF provides a Publish button in the UI When you click Publish, ADF generates ARM templates representing the entire factory state. These templates are stored in the adf_publish branch: In modern setups, instead of clicking Publish manually, teams often use @microsoft/azure-data-factory-utilities (npm-based export). This allows pipelines to validate ADF resources and export ARM templates programmatically. YAML - name: Validate ADF resources run: | set -euo pipefail FACTORY_ID="/subscriptions/${{ env.SUBSCRIPTION_ID }/resourceGroups/${{ env.RESOURCE_GROUP }/providers/Microsoft.DataFactory/factories/${{ env.SOURCE_FACTORY_NAME }" npm run build validate "${{ github.workspace }" "$FACTORY_ID" YAML - name: Export ARM templates (CI publish) run: | set -euo pipefail FACTORY_ID="/subscriptions/${{ env.SUBSCRIPTION_ID }/resourceGroups/${{ env.RESOURCE_GROUP }/providers/Microsoft.DataFactory/factories/${{ env.DEV_FACTORY_NAME }" npm run build export "${{ github.workspace }" "$FACTORY_ID" "${{ env.ARM_OUTPUT_DIR }" Whether you click Publish manually or use npm export in CI/CD, the outcome is the same: Full factory deploymentNo control over individual featuresAll changes get bundled together Selective Deployment Layer (Core Design) We can address this requirement and the associated challenges by introducing a workflow driven by a manifest to define the deployment scope, and a program to identify all necessary ADF dependencies for each manifest file. As a developer, I can now control which release is promoted to production, without worrying about releasing any other features that are not ready. The manifest controls which pipelines to deploy and which optional categories to include. Below is an example of a manifest file JSON { "pipelines": ["pl_ingest_population_selective"], "includeTriggers": false, "includeIntegrationRuntimes": false, "includeAllGlobalParameters": true, "includeLinkedServices": true, "validateLinkedServicesExist": true, "includeManagedVirtualNetwork": false, "includeManagedPrivateEndpoints": false } Workflow Explanation Let's understand the crux of the selective deployment workflow now. I am working in the release branch on my feature branch directly in ADF Studio. Since ADF Studio is integrated with Git, my development changes will be saved to my branch. Here are the steps I can take to promote my change to a higher environment. 1) Validation of ADF on PR validation This is an early validation step and a guardrail: if the PR fails, it's because objects are invalid and misaligned. This is equivalent to the "validation all" button in the ADF ui, here is this workflow Trigger: Pull requests targeting the branch selective_deployment. Purpose: Validate that the ADF JSON in the PR is valid in the context of the target factory. Main steps: CheckoutSet up Node.js 20npm installAzure login using OIDC (azure/login@v2)Validate with ADF Utilities: YAML FACTORY_ID="/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourceGroups/${AZURE_RESOURCE_GROUP}/providers/Microsoft.DataFactory/factories/${DEV_FACTORY_NAME}" npm run build validate "$GITHUB_WORKSPACE" "$FACTORY_ID" 2) Release build + selective deploy to DEV adf-release-build-selective-deploy.yml Triggers: Push to selective_deploymentManual run (workflow_dispatch) with optional manifest inputDefault: deploy/manifests/release.json This workflow has two jobs: Job A: adf-build (staging + export + sanitize + artifacts) Checkout (full history)Azure login using OIDCSet up Node.js 20Install build dependencies inside build/ (npm install in build)Stage selective subset python scripts/select_adf_subset.py <manifest>, a code snippet below for the complete script, refer to the GitHub repository link given Python import json import re import shutil import sys from pathlib import Path from typing import Dict, Set, Tuple, List from collections import defaultdict # Your repo layout has pipeline/, dataset/, linkedService/ at ROOT. REPO_ROOT = Path(".") STAGE_ROOT = Path("build/adf_subset") RESOURCE_DIRS = { "pipeline": REPO_ROOT / "pipeline", "dataset": REPO_ROOT / "dataset", "linkedService": REPO_ROOT / "linkedService", "dataflow": REPO_ROOT / "dataflow", "trigger": REPO_ROOT / "trigger", "integrationRuntime": REPO_ROOT / "integrationRuntime", "credential": REPO_ROOT / "credential", "managedVirtualNetwork": REPO_ROOT / "managedVirtualNetwork", } # Copy these if present so ADF utilities behave the same on staged subset. ROOT_FILES_TO_COPY = [ "publish_config.json", "arm-template-parameters-definition.json", "arm_template_parameters-definition.json", "package.json", "package-lock.json", ] Produces: build/adf_subset/ (staged tree)build/adf_subset_report.json (dependency report)Refer to logs below (showing output of stage selective subset and debug to view output generated after select_adf_subset.py )Export ARM templates from the staged subset via ADF Utilities: npm --prefix build run build -- export "adf_subset" "$FACTORY_ID" "ArmTemplate"Produces: build/ArmTemplate/ARMTemplateForFactory.jsonbuild/ArmTemplate/ARMTemplateParametersForFactory.jsonStrip infra-owned resources scripts/strip_arm_resources.py to produce a safe template: build/ArmTemplate/ARMTemplateForFactory.safe.json⚠️ Note on Infrastructure Components (Refer to the “Future Work & Next Steps” section for follow-up topics in this series) The step above intentionally strips infrastructure-dependent components from the generated subset to avoid overwriting existing shared resources such as linked services. This implementation focuses on developer-owned artifacts (pipelines, datasets, and triggers) and assumes that infrastructure components — such as Integration Runtimes, managed private endpoints, and linked services — are pre-provisioned and managed outside of this deployment workflow.Upload artifacts: ARM templates (adf-arm)metadata (adf-release-meta)subset report (adf-subset-report) Job B: deploy_dev (deploy safe template) Download ARM artifactAzure login using OIDCEnsure az Data Factory extension is installedValidate JSON files exist/parseDeploy via azure/arm-deploy@v2(Incremental) to DEV RG/factory: Template: ARMTemplateForFactory.safe.jsonParameters: ARMTemplateParametersForFactory.json + factoryName=<DEV_FACTORY_NAME> Lesson Learned Setting up selective deployment in ADF was more than a technical task. It made us rethink our approach to deployments, ownership, and CI/CD design. Here are the main things we learned: 1. The Problem Is Not Tooling; It’s Deployment Granularity At first, we thought the limitation came from the tools we used, like UI publish or npm export. However, both methods yielded the same result: full factory templates. The real problem was that we couldn’t control the scope of deployments, not how the templates were made. 2. Dependency Awareness Is Critical Selective deployment only works when every dependency is found and included. We learned that: Pipelines often reference multiple datasets and linked services. Missing even one dependency results in deployment failure You must automate dependency discovery. 3. “Incremental” Is Often Misunderstood Incremental deployment is important, but it doesn’t work like a patch. It reapplies the full configuration for all included resources. This means: Your generated templates need to be complete for all the artifacts you include. If you use partial definitions, deployments can fail. 4. Separation of Concerns Is Key Not all ADF artifacts are the same. We began to separate them into different groups: Application-owned artifacts: pipelines, datasets, triggers Infrastructure-owned artifacts: linked service, managed virtual networks, managed private endpoints, and integration-runtime, among others. This separation proved crucial for safe, scalable deployments. 5. Selective Deployment Adds Complexity, But It’s Worth It It’s true that implementing this approach brings in additional scripts, manifest management, and CI/CD complexity. But in exchange, we gained precise control over releases, reduced production risk, and faster hotfix deployments. Future Work and Next Steps While selective deployment solved a major gap in ADF CI/CD, it also opened up new areas for improvement and standardization. 1. Defining Infrastructure vs Application Ownership One of the biggest follow-up areas is clearly defining ownership boundaries. In our experience: Application teams should own pipelines, datasets, and triggers Platform or infrastructure teams should own linked services, managed virtual networks, and managed private endpoints, among other things. Future work can focus on: Enforcing this separation in CI/CD. Preventing accidental deployment of infrastructure components Integrating Terraform or platform pipelines for infrastructure provisioning 2. Governance Around Linked Services Linked services are often shared across multiple pipelines and teams. Future improvements include: Centralizing linked service management Using Key Vault and Managed Identity consistently Preventing direct modifications through application pipelines