Tel Aviv and Stanford Researchers Pretrain AI Models Without Any Human Data

The paper, "Self-Play Pretraining with Zero Data", was posted to arXiv on 24 September. Its authors are Aditya Cowsik, Kfir Dolev, Michael Y. Li, G. Bruno De Luca, Nourya Cohen, Noah D. Goodman and Yoav Levine. The paper says the first three contributed equally. According to a Ynetnews report, Dolev did the work as part of his postdoctoral research under Levine, who is at Tel Aviv University's School of Computer Science and AI.

The work addresses a known constraint in AI development. Today's large language models are pretrained on huge amounts of human-written text, and developers are approaching the limits of the high-quality human data available. A method that can build general predictive ability from data the system generates itself would loosen that constraint, though the authors say it cannot replace real data.

In the method, both models start from random initialisation. A generator writes short computer programs, which are run to produce sequences of bytes. A learner is trained to predict the next symbol in those sequences using the standard objective for language models. The generator is trained with reinforcement learning and rewarded for producing programs at the edge of what the learner can currently handle, so the difficulty rises as the learner improves. The authors describe the approach as a search over all computable data-generating processes, inspired by Solomonoff induction, a theoretical framework for prediction.

The researchers then tested the learner on natural data without further training. According to the paper, its zero-shot loss on several natural datasets showed predictable scaling with the compute spent on self-play, meaning more compute reliably brought better predictions. Ynetnews said the datasets covered text, images, sound, music, computer code and DNA sequences. The authors also report that the learner could pick up rules from examples given at test time, a form of in-context learning. They found that the generated sequences included recognisable mathematical patterns such as the Fibonacci sequence. In a separate experiment, a model pretrained this way learned faster when it was later trained on real-world data.

The team is cautious about its results. "This is still a proof of concept using small models," Levine told Ynetnews. Whether the method scales to the large models behind today's commercial AI systems is an open question. The paper also says this kind of universal pretraining cannot supply contingent facts about the world, such as who holds an office or what a company sells. The authors present it as a complement to natural data, not a replacement.

The next step, according to the researchers, is to work out whether and how models pretrained on self-generated data can be combined with real-world knowledge, and whether the scaling trend holds at much larger sizes.