What Exactly Is Distilled When You Distill an Agent? The First “System‑Level Review” Arrives.

From distilling a single model to distilling an entire Agent system.

The Agent’s task-solving ability comes from the combined effect of the model, reusable resources, and execution control, so knowledge distillation also needs to expand from the model level to the completeagent system.

This 57‑page review uses the retention location of knowledge in the student agent as the main thread, systematically outlining the definition, technical approaches, and evaluation framework of agent distillation, providing systematic guidance for the transfer and reuse of agent capabilities.

A programming Agent fixed the bug. When a new Agent takes over a new code repository, the original patches may no longer be applicable, but the methods for diagnosing issues, the testing order, and the rollback strategy when regressions occur are still worth reusing.

These experiences can be learned by the model, can also be distilled into callable skills, or transformed into control rules within execution processes.

To convey this task‑solving capability, knowledge distillation for agent systems needs to consider the model, reusable resources, and execution control simultaneously.

Focusing on this shift from a model-level to a full-agent-system perspective of distillation, a research team from Central South University, Beijing Academy of Artificial Intelligence, The Hong Kong Polytechnic University, and Beijing University of Posts and Telecommunications recently jointly released a system-level intelligent agent distillation survey.

This review treats complete agents as both teachers and students, focusing on how task-solving knowledge is extracted, retained, and applied to subsequent tasks, establishing a unified definition, taxonomy, and evaluation framework, and further discussing the transfer and composition of knowledge across different carriers.

Paper Title:

Distilling Agentic Systems: A Roadmap across Models, Artifacts, and Harnesses

Paper URL:

https://arxiv.org/abs/2609.36630

〓 Figure 1. From model distillation to agent distillation. Knowledge can be retained in models, reusable resources, and execution control. Source: Paper Figure 2

What remains of the experience after a single execution ends

In classic knowledge distillation, the teacher and student are usually regarded as two models. The student absorbs knowledge into its parameters by learning the teacher’s outputs, distributions, or reasoning process.

For agents that can invoke tools, maintain memory, and adjust plans based on execution feedback, this perspective needs further expansion.

The paper uses a running example throughout to illustrate the reason: teacher Agent adapts the code repository to the new version of dependencies, first attempts a large‑scale replacement of the old API. Local tests passed, but full testing revealed a regression.

Subsequently, it reverted the changes, switched to a locally compatible wrapper, and ultimately passed verification.

The entire record mixes temporary paths, repository‑specific code, and reusable insights. The distillation process needs to extract reusable practices from it, such as reproducing the error first, limiting the scope of changes, gradually expanding test coverage, and rolling back when regressions occur.

After switching to a different repository, the specific patches cannot be directly copied, but these strategies may still be useful.

Following this line of thought, the paper defines agent distillation (Agent Distillation) as: extracting task‑solving knowledge from the evidence provided by the teacher agent and permanently integrating it into the student agent, so that this knowledge can still be used for subsequent tasks after the original interaction ends and the teacher is no longer involved.

Thus, a single ad‑hoc hint or online assistance by itself does not constitute distillation; the key is whether it leaves a change that the student can use in the future.

A teacher can also be an earlier version of the same Agent. This enables cross‑Agent knowledge transfer and self‑evolution based on historical experience, allowing discussion under the same set of definitions.

〓 Figure 2. Derive a transferable debugging strategy from a code fix, then have students reuse it in a new task. Illustrative case (source: paper Figure 3)

The same experience can enter three types of knowledge carriers

The review classifies based on where knowledge is retained in students, organizing related methods into three main strands: Models, Artifacts, and Harnesses.

The criterion is how this knowledge participates in future execution, not what the training material is called, which optimization algorithm was used, or whether it was saved as a file.

Parametric Knowledge in the Model

The first approach is closest to the familiar model distillation: By training, the teacher’s experience influences the student model’s inference and decisions. In the code‑fix scenario, this means teaching the student to proactively verify after a change and adjust its strategy when regressions occur.

Agent tasks also bring an additional challenge: the student’s actions affect subsequent states and feedback. Learning only from the trajectories provided by the teacher may not cover the situations encountered when the student acts autonomously.

Therefore, the review further discusses how to incorporate teacher guidance into the interaction trajectories generated by the student themselves, helping the student learn to handle deviations and errors during actual execution.

Externally retrievable and callable knowledge

The second approach preserves experience as memory, examples, skills, code, or tools; the paper refers to these independently retrievable or callable resources as Artifacts.

Model parameters can remain unchanged, and students leverage teacher knowledge by using these resources. For example, abstract the fixing process into debugging skills with applicable conditions, or save them as validated auxiliary scripts.

The focus of these methods is on filtering, abstraction, and validation. Temporary information, outdated interfaces, and conflicting suggestions in the original trace can all reduce reusability. The repository also needs to be updated, merged, and pruned as the environment changes.

Executable Knowledge for Organizing Actions

The third path solidifies experience into the execution control system (Harness). It determines when to invoke resources, how to allocate budgets, which outcomes need verification, and how to retry, roll back, or terminate after a failure.

Similarly, a“when a regression occurs, roll back” experience can be documented in a skill guide, or implemented as a control branch that must be passed at runtime. The former is a readable resource; the latter directly governs execution.

The paper distinguishes Artifact and Harness accordingly, avoiding mixing all workflow files, scripts, and control logic into one category.

This also explains why file formats cannot determine classification: separately saved model adapters still belong to parametric knowledge; callable scripts are resources; the mechanism that decides when to invoke a script belongs to execution control.

Knowledge transcends models, bridging the gap between skills and processes

These three types of knowledge carriers can also be transformed into one another. For example, a debugging skill can serve as training material and be absorbed by the student model; the rules concerning checking, retrying, and rollback can further be converted into execution control logic.

The paper calls the conversion between such forms of knowledge “cross‑media transfer.” It focuses on whether the useful judgments and action strategies originally available can be retained after the knowledge changes its representation.

Cross-media transfer describes how knowledge is transformed from one carrier to another, the hybrid distillation result describes the state where knowledge is ultimately retained simultaneously in multiple student components.

Both can occur simultaneously: for example, training a student model with a diagnostic method from the teacher’s debugging skills, while converting its verification and rollback requirements into execution control rules.

Knowledge is migrated from reusable resources into the model and execution control framework, and remains continuously in these two components.

After the score improves, you still need to find evidence that experience matters

The increase in student scores indicates that the overall system performance has improved, but it does not alone reveal where the teacher’s knowledge is retained. The improvement may stem from model learning, or from added skills, better routing mechanisms, or a larger execution budget.

To this end, the survey builds an evaluation framework that links knowledge retention, causal contribution, and practical task benefits, systematically examining what the model retains, how these changes affect execution, and whether the associated gains persist in new tasks and deployment settings.

Specifically, while controlling other components, task conditions, and execution budget, you can restore a distilled component to its pre‑distillation state and observe how the student’s behavior and performance change.

Only by also combining new task testing, cost and failure analysis can we link “what was left”, “what actual effect it had”, and “how much benefit it brought” together.

Put knowledge where it belongs

This perspective ultimately points to an engineering problem: Which experiences are suitable for training into models, which should be kept as editable resources, and which should be transformed into execution control rules?

The review discusses several possible allocation principles:frequently used and relatively stable decision patterns, can be considered for parameterization; knowledge or specific operational specifications that change frequently, are suitable to be stored in updatable resources; validation, recovery, and permission constraints, can be implemented via explicit control logic.

The specific choice still depends on cost, update frequency, portability, and security requirements.

At the same time, knowledge becomes outdated, tools get upgraded, and permissions change. After preserving knowledge, you also need to record its source, applicable conditions, and dependencies, and support modification, revocation, and rollback.Continuously accumulating experience also means continuously maintaining it.

From model distillation to Agent distillation, the research target of knowledge transfer expands to a complete system composed of models, reusable resources, and execution control.

This review establishes a unified framework around how knowledge is extracted, retained, and reused, providing researchers with tools to analyze the transfer mechanisms of agent capabilities, and offering developers a basis for selecting knowledge carriers, designing distillation processes, and validating practical outcomes.

Read more

·