AI做出的科学发现，谁来证明它是对的？

Scientific discoveries made by AI, who will verify they are correct?

From September 20 to 21, the PAIR Conference 2026 on the theme “Frontiers of Next‑Generation Physical Intelligence” was held in Shanghai. The conference, organized by the Shanghai Institute of Physical Intelligence and Robotics, invited numerous experts and scholars from universities, research institutions, and industry to discuss topics such as next‑generation intelligent computing systems, next‑generation intelligent carriers, next‑generation scientific discovery, and next‑generation computing paradigms.

At the “Next‑Generation Scientific Discovery” forum, several talks examined how AI can enter various stages of research from the perspectives of research infrastructure, bioengineering, drug development, and scientific imaging. The concluding round‑table, titled “How Model Results Become Trusted Scientific Evidence—Interdisciplinary Validation, Reproducibility, and Applicability Boundaries,” narrowed the discussion to a core question: When AI produces a seemingly valuable result, what kind of verification must it undergo to become trusted, reusable scientific evidence?

The round‑table discussion was chaired by Associate Researcher Guo Yuchen from the National Information Research Center at Tsinghua University, with four young researchers from diverse disciplinary backgrounds participating in the dialogue,who are Associate Professor Zhang Shuyi from the School of Pharmacy, Tsinghua University; researcher Bao Feng from Fudan University; young researcher Zhang Hongliang from Fudan University and founder of Dan Yuan Xing Tu; and Xie Tong, CEO of Rhythm Creatures and lecturer at the University of New South Wales.

The main theme of AI for Science in recent years has been to demonstrate that models “can do”: they can predict protein structures, screen thousands of candidate materials, and automatically plan synthesis routes. However, while the speed of generating candidates has increased by several orders of magnitude, the speed of experimental validation has not kept pace. This gap is becoming a key variable that determines the true value of achievements in this field.

Photo | (left to right) Guo Yuchen, Bao Feng, Zhang Shuyi, Xie Tong and Zhang Hongliang in a round‑table discussion (source: PAIR)

The following content is compiled from the round‑table discussion.

Predictions are fast, verification is too slow

After the roundtable begins, regarding which AI predictions are worth sending to the lab, the four guests’ answers, based on verification cost, happen to form a gradient from low to high.

Baofeng starts by discussing the methodology of machine learning. To determine whether a model can generalize to the real world, machine learning has a mature practice: split the data into training, validation, and test sets, and evaluate the model’s performance on data it has never seen. This approach works well in image recognition because there is plenty of data available. However, in many scientific scenarios, this premise no longer holds.

He uses the AI virtual cell, which has attracted much attention in the past two years, as an example. These models aim to predict cellular responses to genetic perturbations, and a classic task is to predict what changes occur when two perturbations happen simultaneously. Human cells have about 20,000 genes, yet the best dataset he found last year covered only a little over 100 perturbed genes. Even if a model trained on such data achieves impressive metrics, it is difficult to prove that its predictions are equally reliable for the remaining tens of thousands of genes.

Therefore, Bao Feng’s stance is, when pushing model results toward real-world validation, “stay skeptical and lower expectations.” Instead of hoping the model will directly provide the correct answer, it’s better to conduct more pilot experiments and preliminary analyses on the front end, using the model as an aid to maximize the probability of eventual success.

As for the protein design field, Zhang Shuyi believes that computation is no longer the primary bottleneck; the key is whether functional validation feedback can be obtained quickly on a standardized, high‑quality platform. Proteins have a natural advantage in this regard: the newly designed protein only requires a sequence rewrite, and the costs of DNA synthesis and protein expression have dropped to an acceptable level, so a small test tube can holdnan10¹⁰ order‑of‑magnitude sequences. Small molecules are different; a single change on a side chain can require the entire synthetic route to be redesigned.

He also links this advantage to biological evolution. Engineering learning is accustomed to standardization and modularization, whereas biological systems naturally possess evolutionary mechanisms that can self‑select among massive variants. If we can build a high‑throughput automated platform capable of handling this scale, and then use the data generated during evolution to train AI, life sciences might actually become the first field where AI design successfully completes the “design‑validate‑redesign” loop.

The situation in the materials field is more complex. Xie Tong mentioned that at the end of 2023, Google DeepMind’s material exploration graph network (GNoME) and the Lawrence Berkeley National Laboratory’s (LBNL) autonomous lab A‑Lab each published papers in Nature, leading many to believe that AI‑driven material discovery has arrived.

Among them, GNoME reported about 2.2 million crystal structures that lie below the known convex hull (i.e., thermodynamically potentially stable), of which roughly 380,000 were listed as new stable materials. Years later, Xie Tong’s assessment is that there remains a large gap between these computational predictions and the materials that are actually synthesized and usable in the lab.

He believes the problem lies in the “distribution.” Many predictions in the materials field still fall within existing theories and known chemical space; the models simply compute faster, making it hard to call them genuine scientific discoveries. This view is corroborated by earlier academic criticism: a UCSB team published in *Chemistry of Materials* in 2024, arguing that many of GNoME’s predictions are merely simple extensions of known compounds, with almost no examples that simultaneously satisfy novelty, reliability, and practicality.

At the same time, a turning point in reducing experimental costs is also emerging. When Xie Tong worked on perovskite materials early in his career, he could only produce 8 samples per week; now his laboratory can produce more than 250 samples in a single day. With the increase in throughput, experiments no longer have to be predicated on success, and researchers can explore materials outside the usual scope that no one has attempted before. Even if they fail, those results are highly valuable for improving models.

When it comes to the fusion and fission reactor materials studied by Zhang Hongliang, the verification costs rise another level.Materials for plasma and reactor cores must operate under a multi-field coupled environment of intense radiation, high temperature, severe corrosion, and high stress; for scholars in this field, a single irradiation assessment takes years, and usable data are extremely scarce.

Under these conditions, Zhang Hongliang’s premise for deciding whether to conduct an experiment is clear: the model must provide uncertainty analysis and a well‑defined applicability boundary, including the applicable temperature range, radiation dose, dose rate, and stress level. Only when the boundaries match real operating conditions and the predictions obey physical laws is it worth investing in the next round of experiments. “For us, model validation is not the last mile of AI model research, but an environment for stress‑testing the model.”

The value of “AI prediction” varies dramatically across fields: a protein prediction can be tested quickly and cheaply; but in the fusion materials domain, testing a single prediction may take years. Therefore, the more expensive the validation, the more the model needs “self‑awareness” to clarify its own capability limits.

Who validates the “validation”?

After filtering out the candidates worth pursuing, the next step is to design reliable experiments. This may sound redundant at first—run an experiment, look at the results, and the correctness is obvious. However, A‑Lab already has a cautionary tale of validation steps going wrong.

In a research paper, the team reported that A‑Lab autonomously synthesized 41 out of 58 target compounds during a 17‑day continuous run. The targets were sourced from the Materials Project database and DeepMind’s stability calculations; synthesis recipes were generated by a model trained on literature data, and product identification was also handed to algorithms: a deep‑learning model recognized phases from X‑ray diffraction (XRD) patterns, and a theoretical model fitted the measured patterns to confirm the conclusions.

Shortly after the paper was published, researchers at University College London (UCL) and Princeton University re‑examined these maps. In March 2024, an article in PRX Energy pointed out that the automated analysis of a large number of poorly fitted data, and the neglect of disordered structures where elements may share lattice sites, led to the conclusion that almost none of these products could be considered new materials at the time of publication.

The controversy has lasted more than two years. Earlier this year, the team corrected the A‑Lab related papers, removing manually analyzed cases that did not meet the criteria, and changed the wording about “new materials” to “new for the prediction platform, but not necessarily new to the scientific community.”

In Xie’s view, this case shows that when the same AI system is responsible for both generation and judgment, its blind spots will appear at both ends simultaneously.

Based on this, he proposed three principles for AI experiment design. First, validation criteria and procedures must be set before the experiment, independent of the results; you cannot define success after seeing the data. Second, representation and analysis can heavily rely on AI, but the validator and generator should be two separate models operating in completely independent environments. Third, ensure cross‑platform reproducibility. A material that performs well in a Shanghai lab may not yield the same result in Beijing, which is not uncommon in materials science.

His team is collaborating with laboratories in five countries to conduct the same set of chemical experiments, aiming to determine whether failures in materials science stem from the model or the experimental environment.

Zhang Hongliang’s team started building a model‑validation chain for reactor materials in 2023, and it has basically taken shape this year. Their approach is hierarchical: first, using phase diagram calculations, first‑principles simulations, molecular dynamics and other “dry” experiments to narrow down potentially tens of thousands of candidates to a few hundred; then conducting low‑cost, rapid “wet” experiments on mechanical properties, high‑temperature performance, etc., under non‑extreme conditions to filter down to dozens or about a hundred; finally, they move on to testing and simulation in truly extreme environments.

In addition, they conduct coupled and decoupled experiments to clarify the differences when irradiation, corrosion, stress, and high temperature act individually, in pairs, or under multi‑field coupling. The goal is to compress the target window as much as possible before the expensive final assessment, keeping the timeline and cost within a manageable range.

Zhang Shuyi, drawing on experience in protein design, emphasizes another dimension: validation scenarios should be as close as possible to the final application. Taking the engineering of intracellular catalytic enzymes in biomanufacturing as an example, the traditional approach has been to extract the enzyme, test its activity in vitro, and after improving activity, put it back into the cell. However, enzymes that perform well in vitro often fail to meet expectations when reintroduced into cells.

His current approach is to measure production directly in the biomanufacturing context and feed the intermediate process data to AI for learning. For materials and chemistry, the chain from computation to application is too long, making this difficult; but the underlying biological systems are easier to validate, providing an opportunity to compress the intermediate steps.

Bao Feng brings the issue back to the process organization: AI side has low cost and fast speed, while the experimental side has high cost and long cycles; the two should not be treated as separate sequential stages. The ideal arrangement is to re‑order the experiments, front‑load the critical intermediate results so the model receives feedback as early as possible; at the same time, invest more computation and time on the AI side, using relatively cheap compute to reduce the number of experimental iterations.

These approaches emphasize different aspects, but they all point to a key shift: once AI participates in the experiment, the validation plan must be redesigned and finalized before the results appear, otherwise it becomes “talking to oneself”.

This also raises questions about the prevailing way of evaluating models. Currently, many AI for Science works habitually use benchmark scores to demonstrate performance. But Baofeng’s experience is that scientific tasks often cannot be reduced to classification or regression problems. His approach is to first visualize from multiple angles, ensuring that the benchmark measures the properties researchers care about, and then look at the model scores.

When can we say “It doesn’t work”?

In practice, it is normal for experimental results and model predictions not to match. Zhang Hongliang estimates that about 95% of cases in his work exhibit some deviation. After AI enters the research workflow, the source of the deviation becomes harder to pinpoint—whether it is an error in the model’s judgment, a mistake in the agent’s scheduling, a bias in experimental execution, or unconsidered boundary conditions between different platforms.

When the Xie Tong team built a real‑world experimental benchmark for materials chemistry, they categorized failures into four layers from easy to hard. The easiest to troubleshoot is the execution layer, such as differences in raw material batches; next is the characterization layer, for example when equipment is not calibrated in advance after a material system change; above that is the agent layer, where the agent might call the wrong standard, causing the design to be off from the start; the hardest is the model layer, where the pre‑training data itself is problematic.

Zhang Hongliang’s troubleshooting order is similar: first check whether the deviation falls within the model’s own error margin, then examine data collection and experimental conditions, and repeat the experiment; if the problem persists, only then revisit the model to see if it lacks certain physical descriptors, and involve peers for multi‑platform replication.

Zhang Shuyi questions the definition of “failure.” From an engineering perspective, a molecule or material that does not meet the set criteria is considered a failure; from an AI perspective, they are just another form of data, often occupying the model’s weakest regions and prompting model researchers to supplement the corresponding training content.

The “hallucinations” that early protein design models were often criticized for have now become a novel design approach. As early as 2021, David Baker’s team at the University of Washington used network “hallucinations” to design entirely new proteins starting from random sequences.

Zhang Shuyi notes that in recent years, some have used hallucinations to generate a large, diverse set of starting points, then screened and optimized functions, achieving results that even surpass single-path engineering designs. He draws an evolutionary analogy: mammals had no advantage during the age of dinosaurs, but as long as they survived, a change in environment turned them into winners.

This view has solid scientific backing, but it also raises a problem. Host Guo Yuchen pointed out that if both success and failure are counted as success, wouldn’t this field lose its basis for error correction? He quoted a Nature editor: “You see too many successful cases, while I see too many failures.” He then further asked the guest: when lacking which evidence should a model not be declared effective.

Zhang Hongliang replied that he would not dismiss a model because of a single deviation, but if a model only performs well on its own training and test sets and has not been independently validated in its intended environment—especially through cross‑platform independent experiments—he would not consider it usable.

Baofeng believes that every conclusion in papers that the “model performs well” is qualified and only holds under specific scenarios and conditions, and that the real contribution to science and society requires a longer time to assess.

For example, Google DeepMind’s protein structure prediction model AlphaFold. Its first version debuted in 2018, and the second‑generation model in 2020 pushed prediction accuracy to a level close to experimental determination.

At that time, people widely expected it to open new avenues for drug discovery. But Baofeng searched last year for which recently approved drugs were developed based on structures predicted by AlphaFold, and he said the answer is clear (hardly any).

Zhang Shuyi then explained the reason. AlphaFold’s success is largely due to the protein database (PDB). This public database aggregates about 200,000 protein structures solved by structural biologists worldwide over decades. AlphaFold learned from all of these protein data at once, but the functional data needed for drug development lacks such a foundation and can only be accumulated slowly over time, so progress is naturally slower.

A more common misconception is conflating a model’s performance in a limited scenario with people’s expectations of it. In a constrained setting, as long as the test conditions are clearly stated, it’s easy to judge the model’s quality; the real difficulty lies in the fact that people often hope the model from a limited scenario will naturally generalize to broader contexts such as drug development or biomanufacturing, overlooking the need for additional retraining and development in between.

Xie Tong believes that ranking first in benchmark tests at most indicates that this is a decent model. To prove that it has made a scientific discovery, at least three things are still needed: “independence,” “foresight,” and “extrapolation.” Validation must be performed by a party independent of the developers; it must be forward‑looking, making predictions first and then testing them experimentally—in Xie Tong’s words, you should aim at the target before shooting the arrow, not draw the target after the arrow is fired; and it must be extrapolative, with effective results extending beyond the known space.

If a model can achieve all three of these criteria, the recursive self‑improvement (RSI), autonomous scientific discovery, and foundational scientific models that people discuss may finally become a reality.

As the discussion draws to a close, Guo Yuchen summarizes that there is still no clear consensus in the field about what new standards should accompany a new research system.

The four guests offered different approaches, but the discussion reveals that AI‑generated results are already ahead, while the rules for judging their credibility have not kept up: who will verify, what criteria the verification must meet, whether thresholds should differ across disciplines, and who should bear the cost of verification. These issues are moving from academic debate to practical concerns in research management and industry decision‑making.

Operations/Formatting: He Chenlong

Note: Cover/lead image generated with AI assistance