• An AI-generated research paper by Tokyo-based startup Sakana AI has passed peer review at ICLR 2025, raising questions about AI’s role in scientific publishing.
  • While the paper met peer review standards, reviewers noted contradictions and imprecisions, highlighting both AI’s potential and its current limitations.

An AI-generated research paper, created entirely by Tokyo-based startup Sakana AI, has passed peer review, marking a significant milestone in artificial intelligence’s role in academic publishing.

The paper, titled “Compositional Regularization: Unexpected Obstacles in Enhancing Neural Network Generalization,” was submitted to a workshop at the International Conference on Learning Representations (ICLR) 2025.

The peer reviewers, who were unaware of which submissions were AI-generated, evaluated the paper based on the same standards applied to human-authored research.

Sakana AI’s submission was one of three fully AI-generated papers among 43 total submissions. Of these, only the Sakana AI paper met the acceptance threshold, with an average reviewer rating of 6.33, just above the cutoff of 6. The company posted the paper and supporting materials on GitHub.

Challenges in AI-Authored Research

Despite passing review, the AI-generated paper had notable weaknesses. Reviewers highlighted definitional imprecision, contradictions between claims and data, and a lack of nuanced understanding.

One reviewer pointed out that the paper needed to “be more precise” about embedding hidden states, while another noted conflicting claims about attention mechanisms.

The study itself examined how neural networks handle compositional generalization—the ability to understand novel combinations of familiar components.

The abstract states:

"Neural networks excel in many tasks but often struggle with compositional generalization—the ability to understand and generate novel combinations of familiar components. Our experiments on synthetic arithmetic expression datasets reveal that models trained with compositional regularization do not achieve significant improvements compared to baseline models."

Sakana AI’s “AI Scientist-v2” system was responsible for generating the entire research process, from hypothesis formation to manuscript writing. The company regards this as a step toward automating scientific discovery.

Impact on Scientific Publishing

The success of an AI-generated paper raises concerns about trust and credibility in academic research. While AI can produce research at lower costs and potentially higher volumes than human researchers, questions remain about accuracy, originality, and the integrity of scientific discourse.

Sakana AI believes AI’s role in academic publishing will continue to expand, predicting that AI-generated research will eventually meet or exceed human standards.

“Advanced AI systems may eventually produce papers ‘at and beyond human levels,’ even at the highest tiers of scientific publishing,” the company stated.

However, academic institutions remain cautious. Various organizations, including the U.S. National Institutes of Health (NIH), the National Science Foundation (NSF), and the Committee on Publication Ethics (COPE), have set guidelines for AI use in academic research. These policies address concerns about integrity, authorship, and ethical considerations.

Edited by Harshajit Sarmah