Can Your Model Edit Videos? Alibaba and Shanghai Jiao Tong Launch CutCraft, Testing Editing Commands Across Six Dimensions

Nowadays multi‑camera audio‑video generation models are becoming increasingly powerful, producing smooth video footage with coherent narratives.

But smooth footage ≠ professional editing.

Many AI video tools can produce smooth final footage, but they fail to meet basic editing requirements: controlling the number and duration of shots, achieving specific types of montage storytelling, implementing designated transition effects, and establishing specified audio‑visual relationships, among others.

Most existing evaluation benchmarks focus on general metrics such as visual quality, synchronization, and physical plausibility, but lack a dedicated assessment of the actual execution of editing commands.

To address this gap, researchers from Alibaba have introduced  CutCraft: the first evaluation benchmark for multi‑camera audio‑video generation that focuses on professional editing command execution capabilities, incorporating shot structure, transition syntax, audio‑visual editing relationships, montage techniques, and more into the evaluation scope.

Paper Title:

Beyond Coherence: Benchmarking Professional Editing‑Technique Execution in Multi‑Shot Audio‑Video Generation

Author affiliations:

Alibaba, Shanghai Jiao Tong University, Fudan University, University of Texas at Austin, Institute of Software, Chinese Academy of Sciences

Paper link:

https://arxiv.org/abs/2609.08275

Code repository:

https://github.com/AlibabaResearch/cut-craft-bench 

Dataset repository:

https://huggingface.co/datasets/ZTY01/CutCraft 

CutCraft Dataset: Expert-built, covering the full editing dimensions

CutCraft is not a simple text‑and‑video pairing dataset; the entire set of samples has been filtered through large models and multiple rounds of human review. From 1,500 candidate prompts, 295 structured prompt samples were finally selected, containing 1,228 shots and 933 shot‑to‑shot transitions.

Each sample consists of an overall video description, descriptions for each shot (content, cinematography, style, etc.), and transition descriptions between every two shots (effects, audio‑visual, camera work, etc.).

The dataset covers the full elements of film editing: 13 types of montage (narrative, expressive, conceptual, etc.), a rich transition system (camera dimensions, visual‑effects dimensions, audio‑visual relationships, etc.), shot‑level cinematic language, and diverse styles and content.

Layered Hybrid Evaluation Framework: Not Just Looking at Visual Appeal, But Whether Commands Are Executed

To address the issues of “generated video shots being scrambled, merged, or missing, making it impossible to directly score editing commands,” CutCraft designs a layered hybrid evaluation framework, first ensuring that the video under test aligns with the prompt storyboard through shot aggregation and alignment.

The evaluation covers 6 major dimensions and 16 sub‑dimensions: narrative execution, rhythm and tempo, montage execution, transition execution, cinematography and physical plausibility, and audio implementation. The entire evaluation is divided into three main execution paths:

Path 1: Direct Scoring by Expert Model

Using tools such as shot boundary detection, optical flow, audio separation, and visual features to produce objective, quantifiable metrics, e.g., shot count accuracy, shot duration precision, etc.

Path 2: Expert Signals + VLM Arbitration

When low‑level signals are insufficient for a decision, pass the evidence signals extracted by the expert model to the VLM for comprehensive judgment. For example, determining J‑cut/L‑cut audio‑visual relationships, various narrative logic transitions, etc.

Path 3: VLM Q&A‑Based Determination

For highly semantic editing concepts, such as causal coherence and event execution degree recognition, have the VLM answer preset multiple‑choice questions, using a Q&A approach to determine whether the model has fulfilled the editing intent.

Agentic Editing Intelligent Editing Baseline: Planning — Generation — Synthesis — Restoration

A review alone isn’t enough; the author also proposes an Agent editing baseline solution to test how to better apply generative models to editing commands.

The entire pipeline does not perform one‑shot end‑to‑end long‑video generation; it is broken down into “global planning — single‑shot generation — posterior synthesis — iterative refinement”.

Experimental results show that this Agent solution can significantly improve discrete controllable editing capabilities such as shot count, shot duration, and optical transitions. Other higher-level abilities, such as montage storytelling, are limited by the model itself and still have room for improvement.

Tested 13 mainstream audio‑video models (8 closed‑source, 5 open‑source) and applied the agentic baseline to 3 models, yielding four key conclusions:

1. Visual continuity ≠ editing capability: Many models can generate smooth video streams, but they fail to faithfully follow editing instructions to convey specific logic and emotion.

2. The model naturally favors continuous narrative: Its performance drops significantly for non‑continuous montages that involve temporal or spatial jumps (parallel montage, cross‑cut montage, psychological montage, juggling montage, etc.).

3. More shots make editing harder: As the number of shots increases from 2 to 5‑6, the ability to execute edits noticeably declines.

4. Aesthetic image quality is weakly correlated with editing compliance: A good-looking image quality does not guarantee good montage, transitions, or audio‑visual sync. Many models with high image‑quality scores have relatively low editing execution scores.

Common high-frequency failure types of the model: montage expression logic failure, shot structure disorder (discrepancies in shot count and timing), transition effect degradation (collapsing into direct cuts, making it difficult to achieve complex visual effects and audio‑visual relationships), etc.

CutCraft aims to advance multi‑camera audio‑video generation from “producing visually coherent footage” to “creating content with editing intent.”

Previous video benchmarks focused more on “how good it looks”; building on that, CutCraft also cares about “whether it can understand and execute editing commands.”

Its comprehensive dataset, hierarchical evaluation framework, and Agent editing baseline provide a complete research foundation for AI video generation in complex multi‑camera scenarios.