Code‑ification
Code-First
Summary
Author | Karthik Ramgopal Translator | Mingzhishan In recent years, agents built on large language models have undergone dramatic changes. Initially they were just temporary prompts cobbled together with glued code.
In recent years, agents built on large language models have undergone a dramatic transformation. What started as ad‑hoc prompts cobbled together with glue code has gradually evolved into what we call the agent Harness framework, also known as the structured execution framework—scaffolding built around probabilistic models that make them stable, observable, and production‑grade.
This evolution was not instantaneous but incremental, driven by the growing demands of real‑world business scenarios. Along this arc of change, three distinct paradigms have emerged, each reflecting a deeper understanding of how to constrain nondeterministic intelligence and turn it into reliable software systems.
The earliest agent systems were built around simple linear workflows, which we call “chains.” This approach is intuitive and quick to implement, allowing rapid deployment. You string together a series of prompts, using the output of the first step as the input to the second step, and so on. The Harness framework is merely glue code for tool calls and conversational memory, ultimately treating the LLM as a black‑box function: text in, text out, plus some syntactic sugar.
For example, on LinkedIn, one of our early generative AI agents was built on the LangChain framework using a fixed three‑step chain process:
Figure 1: Simple chain workflow with Retrieval‑Augmented Generation (RAG) (source: author illustration)
-
Routing: a classification step that determines whether a query falls within scope and which specialized AI “expert” should handle it. -
Retrieval: a recall‑oriented step that invokes appropriate tools (such as Bing web search or LinkedIn internal APIs) to gather specific information, creating a repository that provides factual grounding for Retrieval‑Augmented Generation (RAG). -
Generation: an accuracy‑oriented step that filters and synthesizes the retrieved data into a coherent final answer.
Although building and deploying is relatively fast, this simple architecture is fragile. The workflow (e.g., answering user questions through a series of tool calls based on the LinkedIn Economic Graph) relies entirely on the textual fidelity of the preceding step. An upstream error cascades, derailing the entire process, because the system lacks built‑in mechanisms to systematically detect or automatically correct mistakes.
This approach reflects the “do‑the‑best‑you‑can” nature of those early intelligent agent products. Building the initial 80% of the experience is relatively quick, but achieving the final quality required for production environments demands extensive iteration through rigorous evaluation, prompt optimization, and ongoing engineering effort.
As agent system use cases become increasingly complex, the inherent limitations and fragility of linear chain solutions can no longer be ignored. Developers are turning to graphs—a natural abstraction that allows branches, conditional logic, parallel execution, and retry mechanisms to be incorporated into workflows.
In this approach, the graph‑based Harness framework represents different tasks (such as LLM calls, tool invocations, or validations) as nodes and uses edges to control the flow. This shift from a linear structure to a graph mirrors the evolution of data pipelines and traditional distributed systems.
The key innovation of graph systems is not merely adding branches, but making the control flow explicit and declarative. Developers no longer embed logic implicitly in prompts or code; instead, they define it as an executable plan that is easy to view, modify, and reason about. This creates a reasoning loop control flow, allowing LLMs to use the graph structure to route and execute complex, long‑running tasks in dynamic environments.
For many complex tasks, monolithic agent applications are no longer feasible. Graph systems have given rise to multi‑agent systems, with a top‑level intelligent orchestrator managing task flows and dependencies. This shift to multi‑agents requires mature protocols to establish asynchronous communication between agents, such as A2A and ACP (the latter eventually merged into A2A).
Figure 2: Typical multi‑agent architecture (source: author illustration)
Because an agent’s reasoning ability is limited, a “Human‑in‑the‑Loop” (HITL) mechanism is essential. Agents are not fully autonomous; instead, they seek clarification, receive feedback, or request approval from humans at critical decision points, striking a balance between autonomy and controllability.
At LinkedIn, we reuse the existing LinkedIn messaging system to orchestrate communication between agents, handling long‑running tasks by breaking them into a sequence of messages that guarantee strict first‑in‑first‑out (FIFO) delivery and seamless message‑history lookup. This foundation provides:
-
Parallelism: achieved through multiple message threads. -
Resilience: provided by existing persistent retry and eventual delivery mechanisms. -
Flexible execution modes: agents can return results in a single response, stream synchronously, or stream asynchronously across multiple messages.
Graph-based workflows are implemented with LangGraph, offering several key features that enhance system robustness and manageability of complexity:
-
Conditional execution: tasks can be routed or rerouted based on intermediate results. -
Retry and fallback: the system can gracefully recover from failures. -
State management: the system goes beyond text, maintaining structured intermediate data throughout the workflow. -
Explicit validation: includes schema checks, business rules, or auxiliary models to avoid blindly trusting LLM outputs.
These agents also require advanced memory capabilities, enabling continuous learning from user interactions and supporting increasingly personalized, adaptive, and practical experiences over time. Based on a distributed multi-agent architecture, we design cognitive memory itself as an independent agent for integration by application agents.
These innovations drove the launch and subsequent iterations of Hiring Assistant — LinkedIn’s first AI agent designed for recruiters. Recruiters can choose to delegate time‑consuming tasks to Hiring Assistant, including sourcing candidates and assisting with applicant screening, allowing them to focus their energy on more strategic, people‑centric core work.
Over time, we realized that delivering a truly outstanding agent experience requires more than just a powerful model. Our agents need deep domain expertise, grounded in rich organizational context, and the ability to continuously iterate and improve. This calls for memory capabilities—to learn from interactions and provide recruiters with increasingly personalized, adaptive, and practical experiences.
Although graphs significantly expand the complexity of tasks that agents can handle compared to chains, they are essentially still automation programs with workflows predefined by humans. As these workflows and state spaces become increasingly complex, developers face mounting pressure to manually write large amounts of fault‑recovery logic, validation rules, and edge‑case handling code. The resulting process definitions and evaluation frameworks tend to become fragile, cumbersome, and hard to maintain. This rigidity also makes graph‑based approaches unsuitable for tasks that cannot be clearly expressed as deterministic workflows.
The most recent major breakthrough in agent systems comes from using code (instead of prompts) as the primary medium for reasoning. Rather than simply asking the model to “think” about a complex problem in natural language, we let the coding Harness generate, execute, and optimize code within a sandbox environment.
This breaks a fundamental limitation: developers no longer need to pre‑design complex workflows; instead, agents can “search” for solutions by generating code with verifiable outputs. Such Harnesses typically come equipped with tools for planning, task delegation, memory access, token management, and compression.
The natural‑language output of large models is probabilistic and, before subsequent verification steps, essentially unreliable. Code, however, is deterministic. It either compiles, runs, and produces the correct result, or it fails to run. This determinism of success or failure provides a tight feedback loop, which is crucial for dramatically improving system reliability.
To ensure safety and correctness, the architecture includes four core components:
-
Execution sandbox: a controlled isolated environment where generated code can run safely. -
Instrumentation and logging: captures all outputs, errors, and intermediate states during code execution, facilitating subsequent troubleshooting. -
Verification harness: unit tests, assertions, or checking rules that automatically verify whether the generated code’s output meets expectations. -
Iterative optimization loop: when the agent detects a failure, the system systematically modifies the generated code and automatically retries execution until correctness requirements are met.
At LinkedIn, we built an internal tool using this approach that lets developers, product managers, data scientists, and even business operators ask natural‑language analytical questions to an agent, which answers based on platform‑aggregated data and business insights. The agent runs within our existing privacy, permission, and security framework, follows role‑ and purpose‑based data access rules, layers automated compliance review into its output, and routes model interactions through approved compliance infrastructure.
The agent translates user queries into a combination of declarative SQL and imperative Python code, executes it on sample data, validates the output against strict constraints (including privacy, visibility, and compliance checks), and iterates until all correctness requirements are met.
Using a code‑based solution shortened time‑to‑launch because we didn’t have to painstakingly customize prompts, tools, workflow diagrams, and evaluation tuning. The resulting system is not only more reliable, but also significantly more auditable and reproducible.
No matter what stage your agent system is at—whether a simple chain or a complex coding Harness—some fundamental principles determine whether the system is robust or fragile.
-
Treat LLM output as untrusted input: because large language model outputs are probabilistic, they must be rigorously validated before influencing downstream logic, and never trusted without verification. Apply schema checks, business and compliance rules, or auxiliary models before proceeding. -
Decouple planning from execution: let the large model propose a plan, but execute it within a strictly controlled software framework. This separation is crucial because it significantly improves transparency and system control. -
Increase observability: if you cannot see the internal behavior of a complex, highly nondeterministic system, you cannot debug it. Strong observability must capture detailed information, including the agent’s execution state at each step, execution trace, and validation results, while appropriately redacting sensitive data.
Not every problem justifies introducing the operational complexity that comes with a Harness. Choosing the right abstraction essentially means balancing task complexity, reliability requirements, interpretability, latency, and cost. The basic principle is simple: use the simplest system that can reliably solve the problem.
For well‑defined linear tasks, a chain‑of‑thought approach is usually sufficient— a single or limited sequence of prompts can produce acceptable results. Examples include drafting customer‑service replies, summarizing documents, content moderation, or extracting structured data from predictable inputs. These systems typically have the lowest latency and computational overhead because they minimize model calls, orchestration costs, and state management. For many production workloads, a chained solution remains the most economical and efficient option, especially when rapid iteration or large‑scale deployment is required.
When a workflow requires conditional branching, retries, tool calls, or explicit state transitions, a graph-based approach is appropriate. Real‑world examples include multi‑step customer support issue handling, enterprise IT automation, compliance pipelines, or onboarding systems that trigger different actions based on intermediate outputs. While graphs improve resilience and operational control, they also introduce additional latency, often requiring multiple LLM calls, which increases service costs and system complexity. They sit in the middle ground: more reliable than chain‑style solutions, but with higher design and operational overhead.
Coding Harness is best suited for scenarios that demand extremely high accuracy, verifiability, and fault‑recovery capabilities. Examples include software engineering agents, financial analysis systems, legal document verification, scientific workflows, and infrastructure automation—where the generated output can be tested, compiled, simulated, or validated. Because these systems involve iterative reasoning loops, external tool execution, and verification cycles, they typically incur higher computational and latency costs. However, in situations where correctness far outweighs speed or cost, they provide the strongest guarantees.
As models continue to improve and costs keep dropping, the tendency toward over‑engineering remains a common cause of failure. More complex abstractions tend to increase computational overhead, latency, and operational complexity faster than they improve business outcomes.
The real advantage comes from a disciplined choice of abstraction layers: deploying the simplest system that meets reliability, speed, and cost requirements. Strong engineering organizations win not by maximizing architectural complexity, but by optimizing the capability gains from every unit of cost.
The rapid evolution of agent systems has triggered a fundamental shift in the developer role. We are witnessing a career transition from “prompt engineer” to “system architect.” In the early chain‑based LLM stage, success often depended on the skill of crafting perfect prompts. However, as systems evolve into complex graphs and code Harnesses, the focus has moved to rigorous software engineering: designing execution flows, managing state and dependencies, and building robust validation and recovery mechanisms.
Behind this shift lies a larger goal: moving from a “do‑best‑effort” mindset to “engineered reliability.” Early systems treated unpredictability as an inherent constraint and tolerated it. Modern agent Harnesses demand more, pursuing determinism whenever possible, using controlled nondeterminism only when necessary, and defining clear failure modes and recovery paths.
This process marks AI agents evolving from cutting‑edge experimental prototypes to top‑tier production‑grade systems. The novelty lies not in the components themselves, but in how they integrate probabilistic models into deterministic frameworks.
Our challenges and opportunities lie in: not treating these systems as magical black boxes, but handling them as software systems—just like other critical infrastructure, they require equally rigorous standards, careful design, and top‑tier engineering quality.
View the original English article:
https://www.infoq.com/articles/agentic-ai-architecture-harness-evolution/
Disclaimer: This article is original content from AI Frontline, does not represent the platform’s views, nor does it constitute investment advice; reproduction is prohibited without permission.
The momentum continues: See you at AICon Beijing in December
On December 18–19, the AICon Global Artificial Intelligence Development and Application Conference will be held in Beijing. This year’s conference focuses on exploring the boundaries of AI applications, featuring 13 technical topics that cover core areas such as post-training for large models, inference optimization, agent engineering, knowledge engineering, intelligent agent evaluation and safety, self-evolving systems, AI software delivery, native architecture, and organizational transformation. Leading technical experts will gather to share breakthrough AI technologies and real-world practices for enterprise-scale deployment.
Today’s Recommended Article
GPT-6 self-modified Qwen weights hailed as godlike? Prominent Meta researcher douses the hype
Are you also “watching”? 👇
Get MirrorSpeed
Get MirrorSpeed