从Demo到生产：Al Agent缺的到底是什么

From Demo to Production: What Exactly Is Missing in an AI Agent

Author | Trista Pan

Translator | Tian Cheng

Building an AI Agent demo might only take an afternoon. But making it production‑ready is a completely different story.

A production‑grade agent must stay online reliably, be secure enough, control costs, and when it inevitably encounters issues, you need sufficient observability to pinpoint what went wrong. These capabilities rarely come from the model itself; they stem from everything you build around the model: memory, tool access, model routing, guardrails, cost management, and the various traces you have to dig into when the agent misbehaves.

This layer is what we call the Agent Harness.

In the following article, we’ll introduce what Agent Harness is and its two main components: development and operations. Then, we’ll explore how teams decide which parts to manage themselves, ranging from fully managed Harness-as-a-Service to self‑managed technology stacks. Finally, I’ll build the same Agent—FinBot—in two different ways and compare how each achieves the same capabilities. After reading, you should have a clearer understanding of how to use the Harness capabilities described here to build truly production‑ready Agents.

“Agent Harness” only became popular this year, but the work behind it has existed for a long time. If you were working on agents last year, you were already doing these things: connecting various tools, iteratively tweaking prompts, adding retries and logging, and after waking someone up in the middle of the night, continuing to fix the issue the next day. What has really changed now is that people finally gave all this work a unified name. This name is useful because it lets you treat these scattered capabilities as a single whole in your design, rather than slapping a temporary fix into the system every time a problem arises.

The most common way I understand it comes from Vivek Trivedy’s article “The Anatomy of an Agent Harness” (https://blog.langchain.com/the-anatomy-of-an-agent-harness/): Agent = Model + Harness. In other words, if you’re not responsible for the model itself, what you do basically belongs to the Harness. Using a car as an analogy makes this easy to grasp.

The model is like an engine; the power comes from there. But no one would mount a bare engine on a tray and ship it directly to a customer. The chassis, brakes, dashboard, and seat belts are what make it a car you’d feel comfortable letting your family ride in. The model is just the engine; what truly makes the product safe, reliable, and user‑friendly is the Harness you build around it.

In my view, Harness can be roughly divided into two parts. The development side is responsible for extending what the model can do, including cross‑session memory, tools and MCP, retrieval, prompts, and orchestration. The operations side handles keeping the entire system stable after real users arrive, covering observability, evaluation, safeguards, routing, drift and cost monitoring, deployment, and scaling.

It’s worth noting that when you see a picture with obvious asymmetry on both sides, it’s easy to mistakenly think the model isn’t that important. Of course that’s not the case. The truly difficult cognitive work is still performed by the model, and the stronger the model, the more the higher‑level capabilities benefit. What I want to emphasize is that an AI Agent is not just a model; it ultimately becomes a product. Building a good product involves far more than wrapping the model in an API—most of the extra work that makes it actually usable by users falls on the Harness. If you’ve ever built an online service, many of these aspects will be familiar: especially the operations side, which is essentially DevOps under a different name.

One thing is best to clarify first: HaaS and self‑management solutions essentially address the same kind of problem—they are both the infrastructure for building and running agents. Underlying they share the same capabilities: model access, retrieval, tools and MCP, routing, safeguards, etc.

The real difference is how these capabilities get into your hands and how you use them. HaaS packages them as managed APIs that you configure; a self‑managed solution gives you the components to assemble, deploy, and maintain yourself. This distinction ultimately boils down to a few very practical questions: how quickly you can go live, how much control you retain, and how much Harness will cost in the end—on one side the vendor bill, on the other the infrastructure and engineering labor costs. The parts themselves haven’t changed; what changes is who assembles them and who gets woken up in the middle of the night when something breaks.

Strategy, budget, Prompt, tools, evaluation

Harness-as-a-Service is on one end, self‑managed on the other, and many teams end up in the middle, which is normal. There are many options between the two, and which one to choose mainly depends on the following questions:

• Team capability – can you maintain Kubernetes and Gateway long‑term and arrange on‑call duty? Or, do you actually prefer not to manage these at all?

• Existing cloud investment—are you fully committed to AWS, or do you intend to stay multi‑cloud?

• Cost—do you prefer a predictable vendor bill, or infrastructure costs plus the engineering time to run it yourself? HaaS makes budgeting easier; however, as you scale, if you inevitably need to invest manpower to operate the system, self‑managed may be cheaper.

• Governance—must guardrails, audit logs, and data paths remain within your own boundaries? Some enterprises require all of these to run within their own VPC.

• Portability—are you fine using a single provider, or do you need to stay cloud‑agnostic?

• Expected scale — To support this scale, how complex an operations system are you willing to take on.

• Tolerance for operational complexity — Are you really willing to handle cluster maintenance, upgrades, and occasional alerts that go off at 3 a.m. over the long term?

FinBot is a tiny agent, and both approaches use the same scenario. The user asks a financial question: “Summarize Q3 revenue.” FinBot retrieves the quarter’s financial report from document storage, feeds the numbers to a code interpreter, then calls the model to write a summary, and finally returns the answer.

To get this workflow truly into production, the key isn’t choosing a smarter model but what you build around the model. No matter how powerful the model is, it won’t remember the previous conversation, won’t fetch financial statements, won’t stop before a runaway loop generates a massive bill, and won’t tell you exactly where something went wrong after an error. These are the problems Harness aims to solve, and they all revolve around a single request path. Next, I’ll run the same request through two different solutions, completing the same task, each implemented in the most natural way for its own tech stack.

A real Harness consists of many cooperating components (you can review the two major sections earlier), and a single article can’t cover everything. I’ll pick a few items from each side: first, how FinBot itself is built, then three operational capabilities. In my experience, these often determine whether an agent meets production requirements. Guardrails, evaluation, prompt orchestration, deployment and scaling, and multi‑agent workflows won’t be discussed here. Those topics are indeed omitted from this article, but that doesn’t mean they can be ignored—if you want the demo to be truly production‑ready, those issues must also be addressed. Below I’ll first explain why the selected capabilities are important, so that Solution 1 and Solution 2 can each focus on how they implement them.

The development side only retains what FinBot truly needs: the System Prompt, the tools it can access (an MCP Server for financial reports and a container that runs the MCP code execution server), retrieval for these reports, and cross‑turn memory. This is where the difference between the two technology stacks is most evident. One side is a configuration object, the other side is a set of processes that you must deploy and maintain yourself.

Each model provider has its own SDK, authentication method, and response format, and self‑hosted models add another set. What you really need is a unified entry point, so whether adding a new model, conducting a canary test for a new model, or switching to another provider during a failure, you only need to change the configuration, not the application. Provider keys also only need to be stored in one place, rather than scattered across each service.

A buggy loop, or a poorly thought-out feature, can cause token usage to spiral out of control in an instant, ending up as a huge bill. You need measurement, budgeting, and mechanisms that can handle limits gracefully, such as setting a hard cap to terminate the loop, or switching to a fallback solution to continue service, rather than discovering the problem only when the bill arrives.

The agent itself is like a black box; if you connect multiple vendors, the situation only gets worse. Without a unified view that strings together the prompt, tool calls, and responses—plus latency and cost—you can’t even debug, let alone optimize.

Next, let’s see how to implement the same agent with the same three capabilities using two different approaches.

Before diving into the specific capabilities, let’s quickly see what the two solutions consist of. Both sides provide the same set of basic capabilities, only the organization and delivery methods differ. The next few sections will mainly discuss these differences.

Amazon Bedrock AgentCore provides production‑grade foundational capabilities such as Runtime, Memory, Identity, Gateway, Observability, Code Interpreter, and Browser. AgentCore Harness is a managed wrapper built on top of these, turning the task of “piecing these components together yourself” into “just filling in a configuration.” By describing an agent with the CreateHarness and InvokeHarness calls, Harness takes care of wiring the underlying components: each session runs in its own Firecracker microVM, plus managed memory, identity management, and automatic tracing.

When configuration alone cannot meet your needs, you can export the Harness with a single command into editable Strands code and continue running it on the original Runtime. 80% of generic functionality is handled by configuration, while the remaining 20% that truly belongs to you can be solved with code.

The self‑managed solution keeps FinBot as a regular LangChain Agent, running on your own cluster, and delegates the part of Harness responsible for model access to the Gateway on the request path.

Agent Router (formerly Envoy AI Gateway) is an open‑source gateway built on Envoy Proxy and CNCF Envoy Gateway, specifically designed for generative AI traffic: it provides a unified OpenAI‑compatible endpoint in front of multiple model providers and supports token‑based rate limiting, routing and failover, observability, as well as an ext_proc extension hook, while clearly separating the control plane and data plane. The request flow is easy to understand: FinBot makes an OpenAI‑style call to the Gateway; the data plane (Envoy Proxy plus an ext_proc service) first handles authentication, routes by model, enforces safeguards, tallies tokens, and injects upstream credentials, after which the request actually reaches the model provider. The control plane consists of a set of Controllers that watch your Kubernetes CRDs and push configuration via xDS. The entire system runs in your own cluster and is not tied to any cloud vendor. As long as it’s Kubernetes, it works—whether EKS, GKE, AKS, or on‑premises.

This example runs on EKS, with the backend connected to Bedrock. If you move the entire tech stack to another cloud, for example using GKE to connect to Vertex, this demonstrates the benefit of cloud‑agnosticism: the FinBot code and the Endpoint it calls do not need to change. You only need to reconfigure the underlying components, including the Backend Schema, model ID, credentials, Workload Identity, and deployment settings.

Option 1 runs FinBot on AWS AgentCore, i.e., the Harness Runtime operated by the vendor; Option 2 runs the same LangChain Agent in your own cluster and places it behind the Agent Router. Both approaches require the same four components: a System Prompt, accessible tools, financial report retrieval, and cross‑turn memory. On AgentCore, these items are all fields in a single control‑plane call:

import boto3control = boto3.client("bedrock-agentcore-control")runtime = boto3.client("bedrock-agentcore")harness = control.create_harness(    harnessName="finbot",    executionRoleArn=ROLE_ARN,    systemPrompt=[{"text": "You are FinBot, a finance assistant."}],    model={"bedrockModelConfig": {        "modelId": "us.anthropic.claude-sonnet-4-6",        "apiFormat": "converse_stream"}},    tools=[        {"type": "agentcore_code_interpreter", "name": "code"},        {"type": "remote_mcp", "name": "filings",         "config": {"remoteMcp": {"url": "https://mcp.internal/filings"}}},    ],    memory={"managedMemoryConfiguration": {        "strategies": ["SEMANTIC", "SUMMARIZATION"],        "eventExpiryDuration": 60}},)resp = runtime.invoke_harness(    harnessArn=harness["harnessArn"],    runtimeSessionId=SESSION_ID,    messages=[{"role": "user", "content": [{"text": "Summarize Q3 revenue."}]}],)

Retrieval itself is not a separate field of Harness: the Amazon Bedrock Knowledge Base is integrated as a Gateway Connector Target and attached just like other tools. In a self‑managed solution, the same four capabilities are assembled from multiple independent components that you deploy and maintain yourself:

from langchain.agents import create_agentfrom langchain_openai import ChatOpenAIfrom langchain_mcp_adapters.client import MultiServerMCPClientfrom langchain_core.tools.retriever import create_retriever_toolfrom langgraph.checkpoint.postgres.aio import AsyncPostgresSaverfrom psycopg_pool import AsyncConnectionPoolfrom psycopg.rows import dict_rowmodel = ChatOpenAI(model="finbot", base_url="http://ai-gateway/v1", api_key="unused",                   default_headers={"x-ai-eg-model": "finbot"})async def build_agent():                                  # once, at startup    pool = AsyncConnectionPool(DB_URI, open=False,             kwargs={"autocommit": True, "row_factory": dict_row})    await pool.open()                                     # long-lived, shared    checkpointer = AsyncPostgresSaver(pool)    await checkpointer.setup()                            # migration: run once, not per request    mcp = MultiServerMCPClient({"filings": {"url": "http://filings-mcp:8000/mcp",                                            "transport": "http"},                                "code":    {"url": "http://code-sandbox:8000/mcp",                                            "transport": "http"}})    tools = [*await mcp.get_tools(),             create_retriever_tool(vector_store.as_retriever(search_kwargs={"k": 4}),                                   name="search_filings",                                   description="Search quarterly filings.")]    return create_agent(model, tools,                        system_prompt="You are FinBot, a finance assistant.",                        checkpointer=checkpointer)async def answer(agent, question, session_id):            # per request    return await agent.ainvoke(        {"messages": [{"role": "user", "content": question}]},        {"configurable": {"thread_id": session_id}})      # memory keyed by session

The two sides still use the same four blocks of building blocks and follow the same request path. The difference is that on one side they are just a few fields in a configuration object; on the other side they are independent processes with their own lifecycles. Looking at these two identifiers, the only difference is the runtimeSessionId and thread_id.

Each vendor has a unified entry point in front, so adding a new model, performing a canary test on a new model, or switching during a failure only requires changing configuration, not modifying the application.

Specifying a model is just a configuration, and it can be overridden on each call. The AgentCore underlying layer can handle Bedrock, OpenAI, Gemini, and all providers compatible with LiteLLM; it can also switch providers mid‑session without losing context, and store third‑party keys in the AgentCore Identity’s Token Vault. The model field is a union type of four configurations (bedrockModelConfig, openAiModelConfig, geminiModelConfig, liteLlmModelConfig), which can be set on the Harness or overridden individually for each call:

resp = runtime.invoke_harness(    harnessArn=harness["harnessArn"],    runtimeSessionId=SESSION_ID,    model={"openAiModelConfig": {"modelId": "gpt-4o"}},   # swap provider per call    messages=[{"role": "user", "content": [{"text": "Summarize Q3 revenue."}]}],)

Agent Router provides a unified endpoint compatible with OpenAI, mapping model aliases to specific backends, with vendor credentials stored on the Gateway:

kind: AIGatewayRoutespec:  rules:    - matches:        - headers: [{ name: x-ai-eg-model, value: finbot }]      backendRefs:        - name: bedrock-claude          modelNameOverride: us.anthropic.claude-sonnet-4-6   # alias -> real model id---kind: AIServiceBackend            # translate OpenAI schema -> Bedrockspec: { schema: { name: AWSBedrock } }

FinBot can simply point a regular client to this endpoint, never touching the vendor’s keys; to switch models or vendors, just change the alias it sends. Credentials always remain in the Gateway. For Bedrock, this means using an AWSCredentials policy backed by EKS Pod Identity or IRSA, so no static keys need to be stored in the cluster:

apiVersion: aigateway.envoyproxy.io/v1beta1kind: BackendSecurityPolicyspec:  targetRefs:    - {group: aigateway.envoyproxy.io, kind: AIServiceBackend, name: bedrock-claude}  type: AWSCredentials                 # Bedrock uses AWS auth  awsCredentials:        region: us-east-1                  # credentials via EKS Pod Identity or IRSA

Both approaches provide a unified entry point. The entry for AgentCore is an API that you call with different configuration objects; the entry for Gateway is a Route that you deploy yourself. The real difference lies in where the provider key is stored: either in the Token Vault of AgentCore Identity, or attached to the Gateway via Pod Identity or IRSA by binding an IAM Role.

Cost control means two things: there must be a hard limit that can immediately stop a runaway loop; and there must be a budget that can be enforced before the bill arrives.

AgentCore directly tackles the runaway loop failure mode, offering hard limits at the single-call level, including maxIterations (default 75), maxTokens, and timeoutSeconds. It also provides Tags for cost allocation and Token metrics in CloudWatch:

aws bedrock-agentcore-control update-harness   --harness-id "finbot-UuFdkQoXSL"   --max-iterations 50 --max-tokens 8192 --timeout-seconds 1800

Gateway measures Token as the cost of a request. Here, we set a daily Token budget for each user and deduct it based on the actual consumption of the response. Once the budget is exhausted, Gateway will reject subsequent requests with HTTP 429 (although the field name is limit.requests, it actually limits the Token budget):

kind: AIGatewayRoutespec:  llmRequestCosts:    - { metadataKey: llm_total_token, type: TotalToken }  rules:    - matches: [{ headers: [{ name: x-ai-eg-model, value: finbot }] }]      backendRefs: [{ name: bedrock-claude }] ---kind: BackendTrafficPolicy               spec:  rateLimit:    global:      rules:        - clientSelectors: [{ headers: [{ name: x-user-id, type: Distinct }] }]          limit: { requests: 5000000, unit: Day }          cost:                 request:  { from: Number, number: 0 }                response: { from: Metadata, metadata: { namespace: io.envoy.ai_gateway, key: llm_total_token } }

AgentCore limits the Agent itself, while the Gateway limits the traffic. One can terminate the loop directly within a single call, the other will reject the next request after a tenant’s token budget is exhausted. Only the latter knows what a “tenant” is.

What you need is a unified view that strings together the Prompt, tool calls, and responses, along with latency and cost.

On AgentCore, this part is almost ready out of the box. Each call automatically sends traces, logs, and metrics to CloudWatch, including model invocations, tool calls, and memory operations, and displays them centrally. After a one‑time configuration at the account level, subsequent calls will be automatically traced:

# one-time, account levelaws logs put-resource-policy --policy-name MyResourcePolicy   --policy-document file://xray-to-logs.json   # logs:PutLogEvents for xray.amazonaws.comaws xray update-trace-segment-destination --destination CloudWatchLogs

You can also add an additional index rule to set the sampling rate, but this is not required.

If you want to know exactly how much work it saves you, you can compare it with the instrumentation you add to the Agent yourself. If the Agent is hosted outside the Runtime, you need to set AGENT_OBSERVABILITY_ENABLED, OTEL_PYTHON_DISTRO, and OTEL_PYTHON_CONFIGURATOR, and also wrap the startup command with opentelemetry-instrument. If you bring your own container onto the AgentCore Runtime, the Runtime will automatically set the ADOT defaults, leaving only the opentelemetry-instrument wrapper. Furthermore, if you deploy using the AgentCore CLI, it will automatically add instrumentation during the packaging stage, so you don’t even need to add this wrapper yourself. Harness developers don’t have to write any of this.

Agent Router outputs generative AI metrics according to OpenTelemetry’s semantic conventions, including token usage, first-token latency, and inter-token latency, which Prometheus can scrape directly:

scrape_configs:  - job_name: envoy-ai-gateway    relabel_configs:      - source_labels: [__meta_kubernetes_pod_container_port_name]        regex: "metrics|aigw-admin"        action: keep# exposed GenAI metrics (OTel semconv):#   gen_ai.client.token.usage#   gen_ai.server.time_to_first_token

Therefore, when an incident actually occurs, the two most frequently asked questions—who is burning tokens and who is the slowest—each require only a single query. Because the metric names use standard semantics, they remain usable even when switching model providers:

sum(gen_ai_client_token_usage_sum{gateway_envoyproxy_io_owning_gateway_name="finbot"})  by (gen_ai_request_model, gen_ai_token_type)

Metrics can tell you how much was consumed and where it’s slow. But if you want to know exactly what happened, you also need tracing. The overall architecture is simple: two data producers, a collector, and a backend.

Both sides will output the same set of gen_ai.* semantics, which is why their Spans can finally align. Instrumentation and backend can be chosen independently: Langfuse, Phoenix, and OpenLIT all provide a LangChain SDK as well as a backend, so you can use one for instrumentation and store the data in the other. On the Gateway side, just point ext‑proc to the Collector:

kind: GatewayConfigspec:  extProc:    kubernetes:      env:        - { name: OTEL_EXPORTER_OTLP_ENDPOINT, value: "http://otel-collector:4317" }        - { name: AI_GATEWAY_TRACING_SEMCONV,  value: "gen_ai" }   # gen_ai.* span names

These spans cover the model calls in the request path. The remaining parts—tool calls, the agent’s own control flow, prompt assembly—never leave the application process, so the application itself still needs instrumentation. OpenLIT requires only a single line of code and uses the same set of semantics:

import openlitopenlit.init(otlp_endpoint="http://otel-collector:4318")   # once, at startup

The OpenTelemetry project itself also provides LangChain Instrumentation, but as of the writing of this article it is still in Beta. No matter which approach you choose, there is one pitfall that almost everyone encounters: message content is not captured by default. Therefore, if you want the Prompt itself to appear in the Trace, you need to set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY on both sides, and you also need to carefully consider who will have permission to read this content once it is enabled.

Point both sides to the same Collector, and you will get a complete trace from Prompt to tool to response. This is essentially the same instrumentation work you would do when bringing your own code into the AgentCore Runtime; the real difference is that only Harness will automatically trace the entire Agent Loop for you.

This article introduces the concept of Agent Harness and showcases two building approaches: one using a Harness-as-a-Service hosted solution, and the other a self-managed solution. Due to space constraints, we only demonstrate a few capabilities of the Harness with FinBot, leaving the rest for you to explore further.

No matter which route you ultimately choose, building and running an Agent will ultimately face similar issues, and these are mostly architectural problems—not magic. You need to first clarify what the product is supposed to accomplish, what counts as done well, and which boundaries must never be crossed. The model provides reasoning capability, while Harness provides control, allowing that reasoning capability to become a truly deliverable product. You define the goals and boundaries, and the tool handles the work within that scope.

Original link:

https://www.infoq.com/articles/agent-harness-build-one/

 Conference Recommendation

Jensen Huang talks about Chinese AI companies: the scarcer the computing power, the more it forces them to perfect their architecture

Just now, Qianwen Office storms into e‑commerce operations! Store setup cost reduced from 30,000 to a few hundred yuan, one agent manages multi‑platform businesses

Just now, Google Gemini 4 goes full throttle: API half-price for a limited time, cache slashed by 95%, self-built simulation teaches robot to score 7 consecutive balls

Are you also “watching”? 👇