From “Gaming the Rankings” to “Doing Research”: How Long Until AI Surpasses Scientists?

From “gaming the rankings” to “doing research”: How long before researchers are overtaken by AI?

Recently, the OpenRSI Index has attracted attention on X. So far, the project announcement post has been viewed over 133,000 times, and related topics have appeared in X’s “Today’s News” section.

Figure 1 OpenRSI-Index appears in X’s “Today’s News” section. X post:https://x.com/OpenRSI/status/2102831770458890626

As large language models approach saturation on some benchmark tests, frontier researchers face an increasingly unavoidable question: to what extent can progress on leaderboards translate into real advances in research?

Boosting rankings is not equivalent to effective research, and simply completing a task is even less indicative of the ability to sustain research progress.

From large‑scale data pre‑training to human‑feedback reinforcement learning (RLHF), model evolution has always relied on effective feedback signals. When research moves toward recursive self‑improvement (RSI), a key bottleneck emerges:

When one model tries to improve another model, or even attempts to improve itself, who defines the truly valid environmental feedback for it?

This is precisely the starting point of the OpenRSI Index: Can research agents, within clearly defined experimental boundaries, improve human‑designed model training schemes and discover better methods?

• Website: https://index.openrsi.foundation/index.html

• GitHub: https://github.com/OpenRSI-Foundation/OpenRSI-Index

• X: https://x.com/OpenRSI

If an agent uses thousands of GPUs,

what will happen?

The OpenRSI Index derives its challenges from influential open‑source research projects in the real world, handing existing models, code, data, and training recipes to agents so they can further improve upon these results. Projects such as Marin, Qwen, and GPIC therefore become test cases for research agents, covering pre‑training, fine‑tuning, and image generation.

These tasks require actually running model‑training experiments, with compute budgets ranging from several thousand to tens of thousands of H100‑hours. The three representative tasks shown on the website are:

• Pre‑training: improve Marin’s optimizer. Marin‑Scaling‑Ladder is based on the open‑source foundational model development project Marin and requires the agent to apply the same optimizer rules to six model scales, covering 550 million to 2.545 billion parameters, testing whether a method remains effective as model size changes. The website notes a single run size of 18,432 H100‑hours.

• Post‑training: designing better reinforcement‑learning tasks for Qwen. Qwen-122B-RL-Merge enables agents to generate verifiable training tasks, improving Qwen’s performance across mathematics, code, and other domains. A single run consumes about 20,864 H100·hours.

• Vision generation: learning better generation methods from GPIC data. Agents start from a shared model checkpoint and explore improvements to architecture, training objectives, or training schemes; each candidate is retrained on the same ten‑million images at most once. A single run consumes 5,815 H100·hours.

The three tasks are combined, corresponding to three questions:Can the method scale across levels? Will the agent organize experiments based on feedback? Can a single improvement be clearly attributed?

Figure 2 | Representative tasks of the OpenRSI Index cover pretraining, post‑training, and visual generation, with the H100‑hour scale for a single run of each task shown. Image source: OpenRSI Index official website.

If you only look at these numbers, the OpenRSI Index can easily be interpreted as “just a more expensive benchmark.” What it really aims to change is not only the computational scale of the evaluation but also the real‑world complexity retained in the assessment.

To obtain stable, comparable results, traditional benchmarks often fix the environment and narrow the tasks, compressing the problem into a well‑defined test with a clear answer. This design is certainly valuable, but it drifts farther from the trial‑and‑error, scale variations, limited budgets, and failed experiments that characterize real research.

These compute costs are incurred in real training and experiments. The figure below shows the publicly released experimental progress of the GPIC text-to-image task: the agent repeatedly adjusts the training plan and continues to improve based on the evaluation results of the generated images.

Figure 3 | Public experimental measurement record for the GPIC text-to-image task. The horizontal axis shows research time; the vertical axis FD‑DINOv2 measures the feature distribution difference between generated images and reference images, lower is better. Each dot represents a measurement, and the step line shows each agent’s best record up to that point. Image source: https://index.openrsi.foundation/tasks/gpic-leaderboard

The tasks also cover emerging open‑source research. For example, the Kev decision‑architecture task associated with Open‑Jev asks agents to improve small models that directly output decision probabilities, exploring decision structures and training methods. The public tasks also involve robotics, retrieval, reasoning, and low‑level system optimization.

What these tasks ultimately aim to answer is: can agents improve upon human‑designed training schemes and discover better methods?

Using an independent testing environment,

to address the research community’s “chain of suspicion”

Researchers who have conducted rigorous experiments know this feeling: an increase in test scores is not the end of the research.

• Is it because hyperparameter tuning unintentionally aligned with the test set distribution?

• Did a coincidence amplify a tiny perturbation?

• If we re‑validate, can this change be reproduced?

Data contamination, test set leakage, and irreproducible experiments are unavoidable issues when evaluating a model’s “self‑improvement.” Therefore, a key principle is: The AI experimental workspace must be isolated from the final acceptance environment.

In the OpenRSI Index, each task has clear resource and code boundaries: which modules can be modified, how many resources can be used, and under which environment the final acceptance occurs.

The solution published by the original author will be rerun in the same task environment as a control. Upon submission, the system pauses the workspace and saves a snapshot, which is then scored by a newly launched clean evaluation environment. Private tests will not be handed over to the AI conducting the research, and file changes in the evaluation environment will not be brought back to the workspace, but the scores and required feedback will be returned. If there are hard violations such as tampering with the reward path, leaking evaluation information, or exceeding the budget, a zero score will be assigned according to the rules. 

Figure 4 RSI‑Harness workspace, independent acceptance, and feedback process. Original project image:https://github.com/OpenRSI-Foundation/OpenRSI-Index/blob/main/assets/figures/rsi-harness.png

This mechanism actually consists of two loops.

The inner loop is the research loop: the Agent modifies code and runs experiments in a continuously existing workspace, and continues researching based on feedback.

The outer loop is the validation loop: each formal submission is frozen as a snapshot and handed to a newly launched independent Judge for re‑evaluation.

Under this operating mechanism, the OpenRSI Index does not require the Agent to “provide the correct answer in one go.” It allows the Agent to iterate and make mistakes. However, every claimed improvement must meet one condition: it must not only improve in the Agent’s own experiment logs but also remain valid in an environment beyond the Agent’s control.

Clear constraints and independent acceptance provide a verifiable foundation for the results. What a single effective submission can actually demonstrate still depends on the specific task, budget, and evaluation criteria. However, the verifiability of the method and the traceability of the results give researchers at least a more concrete basis to answer: Is the work I cite truly trustworthy?

Powerful Mentor Team

The mentor lineup of the OpenRSI Index brings together scholars from multiple research fields. 

Figure 5 | OpenRSI Index official website mentor lineup, sorted alphabetically by name. Image source: https://index.openrsi.foundation/

• Luke Zettlemoyer | University of Washington. Co‑author of RoBERTa, BART, and Toolformer, researching the extension of language model pre‑training to autonomous tool use by models.

• Karthik Narasimhan | Princeton University. One of the four authors of the original GPT paper and co‑author of ReAct, SWE‑bench, and SWE‑agent, researching across large models, agent methods, and real software engineering evaluation.

• Yejin Choi | Stanford University. Known for the commonsense reasoning benchmark HellaSwag, the commonsense knowledge base ATOMIC, and the knowledge generation model COMET, continuously exploring models’ commonsense understanding and reasoning abilities.

• Dawn Song | University of California, Berkeley. Co-author of classic evaluation benchmarks such as MMLU, MATH, and involved in Agents’ Last Exam (ALE), CyberGym, and ExploitGym, extending evaluation to complex scenarios like real-world professional tasks, vulnerability reproduction, and exploit development.

The advisory lineup also includes Wenhu Chen, Alvin Cheung, Jianfeng Gao, Pang Wei Koh, Ranjay Krishna, Hanqing Lu, Kunle Olukotun, Rameswar Panda, Radha Poovendran, Yu Su, Huan Sun, Diyi Yang, Ming-Hsuan Yang, Xiang Yue, Jian Zhang and Yu Zhang, covering research areas such as language, multimodal, systems, and security.

Returning to the open-source community:

Create a question for AI using your signature work

The team believes that research is a positive-sum game. By openly sharing research methods, tools, and experimental infrastructure, more researchers can verify and reuse each other’s results and pose new, thoughtful questions. 

For this reason, the OpenRSI Index makes the tasks, evaluation framework, verification tools, and execution traces publicly available:

• Teams with compute resources can reuse this benchmark environment, validate existing results, and continue exploring new research questions.

• Small teams and individual researchers lacking large clusters can see what AI changed, where it failed, and which directions remain unexplored, using the existing records as a starting point for their next attempts.

One failure is not enough to prove a direction cannot work; but a complete experimental record can help later researchers determine which conditions are worth changing and which attempts are worth continuing, letting taste guide trial and error and truly enhancing AI’s ability to do research.

“Chat with an agent for 1 hour = design an RSI task”

So the next question is: Who decides what the research Agent should study?

The answer provided by the OpenRSI Index is to let the people who actually conduct this research create the problems. Through RSI-Anything, researchers can gradually organize their open research work into tasks that an Agent can execute. According to the project description, by conversing with AI via RSI-Anything and confirming key settings, researchers can transform their own topics into runnable RSI task environments in about an hour. The AI will automatically compile the proposal, build the environment, and help scientists contribute new research tasks to the OpenRSI-Index. Subsequent experiments and reproductions still require the appropriate time and compute resources; after a task is reviewed, reproduced, and accepted, the contributor will be invited to become a co‑author.

Figure 6 | RSI‑Anything task contribution workflow. About one hour refers to the initial discussion and task preparation; subsequent experiment time varies by task. Original project image: https://github.com/OpenRSI-Foundation/OpenRSI-Index/blob/main/assets/figures/rsi-anything-pipeline.png 🌐 Website: https://index.openrsi.foundation/contribute

The community is also looking for compute partners, as well as leads and contributors in foundational models, biomedicine, law, 3D vision, scientific computing, and other fields. We want more researchers to bring real problems forward, so that successes, failures, and intermediate processes can be examined, reused, and further advanced — making RSI belong to everyone, and having everyone participate in building RSI evaluation standards, which is exactly what the OpenRSI Index team emphasizes: From the community, for the community.

© THE END 

For reprints, please contact this public account for permission

Submit articles or request coverage: liyazhou@jiqizhixin.com