First public results for real-time video AI are out—top score only 15, all 37 models failed.

Real-time Video AI’s First Public Scorecard Released! The Highest Score Is Only 15 Points, All 37 Models Completely Failed

Exactly what you want!

Let the colleague keep performing, but replace the on‑screen presence with Ben Yao.

If I act coquettish, he has to be coquettish too; if I send a kiss, he has to send a kiss too.

(P.S.: The colleague handles the performance, I handle the on‑screen appearance—hehe, the editorial team’s division of labor just got a bit more detailed.)

How’s that, does it have something to it?

This issue isn’t really a problem for ordinary video generation.When a video is produced, you can tell at a glance whether it looks good. But real‑time video is different.

It’s a stream that keeps moving forward. At second 10 you want it to change clothes, at second 40 you want to switch back, at second 90 you decide the first outfit was best. Every time you intervene, the model has to figure out three things simultaneously:

• What needs to be changed now.
• What must stay unchanged.
• Whether previous changes still count.

So the question arises: how do you even evaluate this thing?

You can’t just grab a nice screenshot and claim the model understands me.

Then I discovered that someone actually took this seriously.

◽A publicly released exam paper

On October 20, a paper titled StreamEval: Evaluating Interactive Streaming Video Generation Models was posted on arXiv.

Paper link:

https://arxiv.org/abs/2610.28402

Look at the authors—an impressive lineup: Tsinghua, NVIDIA, CMU, UC Berkeley, Princeton, National University of Singapore, and Reactor.

StreamEval is the world’s first benchmark for real‑time interactive video.

Currently, this benchmark is open‑source and has done one thing: turned the “real‑time interaction quality,” which could only be judged by feel, into a public test.

Research institutions, corporate model teams, and individual developers can participate: integrate models, reproduce results, or study cases, and directly feed discovered issues and suggested test items back to the maintenance team.

The code, examples, and scoring rules are all public, so everyone can discuss in reverse: Is the problem set reasonable? Are the scores judged accurately? This way, model progress is traceable, and the evaluation itself can be continuously refined through collective participation.

This benchmark consists of 300 questions in total, a modest size, divided into five categories, each with 60 questions:

• Digital Human Dialogue
• Digital Human Performance
• World Generation
• Text Command Editing
• Reference Image Editing

It basically covers the three hottest real‑time video directions right now: digital humans, virtual worlds, and real‑time modifications of currently streaming video.

The paper tested 37 models and 47 combinations in one go; the earlier experiments I did—changing characters, changing appearances—correspond to reference image editing, while the “Xiaoyao version” digital human responding and performing actions corresponds to digital human dialogue and performance.

The interesting part is its question-posing method.

Each question includes three elements:

Starting material: character image, source video, audio, or reference image.

Input timeline: at which second a new request is issued.

Grading criteria: called “oracle” in the paper, clearly stating after each request what should change and what must be preserved.

The paper presents a very typical “real‑time cape editing” case study:

The timeline of this character video is set to 0 secondsStart using the “Red Velvet Cloak” reference image,10 secondsswitch to the “Gold Satin Cloak” reference image,20 seconds: Cancel the change and restore the current source frame.

There is a test trap here:

When canceling a modification after 20 seconds, the screen should revert to the source frame at that moment; the character and actions must never “roll back” to the original starting point.

As a result, in the test where two outputs were sampled during the golden reference image activation phase, the cape remained its original gray—while the background and scene were preserved, the new appearance requirement was never actually applied.

I also tried “continuous outfit changes” with Vidu S2: in the video I kept switching between the first, second, and third reference images while the video stream kept moving forward.

Besides task type, the challenges are also divided into five duration tiers: 15 seconds, 30 seconds, 60 seconds, 120 seconds, and 300 seconds.

Short ones test how quickly and accurately it reacts; long ones test endurance: after several rounds of requirements, does it still remember what was promised earlier?

◽Evaluation Criteria

StreamEval breaks down capabilities into 4 dimensions and 10 metrics, each scored from 0 to 100, covering the rules comprehensively:

• Task Execution
• State Consistency
• Visual Fidelity
• Audio-Visual Synchronization

Each task only uses metrics relevant to itself.

Take reference‑image editing as an example: it must simultaneously monitor three things—whether the appearance from the reference image was applied, whether unchanged parts and the original motion were preserved, and whether this edit counts as a complete success.

PS: So, in the clothing‑swap case above, even if the clothing is changed correctly, removing the cup the character was originally holding still counts as a failure under the “Edit Success” metric!

◽Evaluation Method

The scoring process consists of two steps:

1. Visual Language Model (VLM)
   
   First extract structured observations (e.g., what defect appears at a certain moment, whether the appearance matches, whether a specified action occurs);
2. Fixed Rules
   
   Combine the case-specified time limits and validation windows to convert observations into precise scores.

No output, or does not support a required operation, assign zero points directly.

There’s another point I didn’t get at first, but later found it absolutely right:

The five types of tasks are ranked separately, not merged into a single overall leaderboard.

Digital human chat and generating a virtual world are completely different things. Forcing them together explains nothing; testing them separately is more meaningful.

◽37 models, where is the gap?

The paper released the test results of each model:

Among the five task categories, Shengshu Technology’s Vidu S2-Avatar achieved the highest overall score in both digital human dialogue and performance:

S2-Editing leads in two types of video editing:

MemFlow, proposed by teams such as HKU and Kuaishou Keling, ranks first in world generation:

If we carefully break down the specific items, we’ll find the real challenges currently facing the real‑time video generation industry:

For example, Vidu S2‑Avatar’s performance stability is 97.25, and motion following is 36.90.

And 36.90 is already the highest score among the seven models for this task….

(I actually sensed this when I was playing with Vidu S2‑Avatar: although the digital human can stay very stable and the footage looks smooth, there’s a lot of room for improvement when making it raise a hand, turn its head, or perform specific actions.)

Strict editing (Edit Success) is as hard as climbing a mountain…

In the reference image editing task, the strictest “complete editing” metric was evaluated, and the highest score in the entire competition was only 15.00 points!

The other models that didn’t take the top spot also each have their own “misaligned advantages” when looked at separately:

• Tavus CVI Full
  
  Leads in audio‑visual synchronization within dialogues;
• LiveTalk 1.3B
  
  Shows the best performance in rendering quality;
• JoyAI-Video-Edit
  
  Better preserves content in the original video that should not be altered during instruction editing;
• Krea Realtime
  
  Although it lags behind in the overall world generation score, its instruction-following score reached the highest in the venue, 25.93.

With the misaligned advantage data evaluated by the StreamEval benchmark, it offers great value for model selection: you can see which mistakes real‑time virtual fitting fears most, which abilities chat characters rely on most, by referring to the corresponding metrics and output logs.

◈Why create a benchmark for real‑time video generation

I believe StreamEval’s greatest value to the entire industry lies in turning the “controllability” of real‑time video generation from an abstract concept into a set of public, reproducible, quantifiable standards.

When to modify, what to retain, how to recover, how to score—there are now clear guidelines for all of these.

Nitrogen, this benchmark indeed needs further validation.

For example, automatic scoring still needs to be compared with human judgment; beyond the 5‑minute case, we also need to add longer conversations, complex scenarios, as well as frame rate, latency, and cost testing in real deployments.

Currently, StreamEval has publicly released the evaluation code, case timelines, datasets, and scoring rules, with the link posted below:

GitHub:

https://github.com/StreamEval/Stream-Eval-Open 

Leaderboard:https://streameval.ai 

Paper:https://arxiv.org/pdf/2610.28402

If you have model or API development capabilities, you can directly refer to the GitHub repository’s integration documentation to set it up yourself and see how many points your model can achieve.

StreamEval will be maintained long‑term; each month we will evaluate new and updated models and publish the latest test results on the leaderboard.

Enterprises, research institutions, and individual developers can submit model information through the “Evaluate your model” entry on the official website, and may also attach self‑test results. After review and verification, the results will be included in subsequent rankings.

As more models and versions are added, a move or a switch missed today can be taken out for testing in the next release.

The industry’s accumulated records will also become a series of data that can be further studied and improved.

As for that family member in the comments, I’ll submit the assignment here first~

If you want to see what I’ll do next, ask the editor‑in‑chief’s most handsome male colleague whether he’s willing to act out hh.