Text-to-video-Ai flowers with new metamorphic video functions

With the help of time -lapse videos as training data, computer scientists have developed video technicians who simulate the physical world more precisely.

While text-to-video models for artificial intelligence such as Openas Sora quickly metamorphoses before our eyes, they have tried to produce metamorphic videos. Simulating a tree or simulating a flower blossom is more difficult for AI systems to generate than other types of videos, since the knowledge of the physical world is necessary and can vary greatly.

But now these models have taken an evolutionary step.

Computer scientists at the University of Rochester, Beijing University, the University of California, Santa Cruz and the National University of Singapore developed a new AI-Text-Video model that learns physical physics from real physics from time-lapse videos. The team outlines its Magictime model in a paper that was published in IEEE transactions for pattern analysis and machine intelligence.

“Artificial intelligence was developed to try to understand the real world and to simulate the activities and events that take place,” says Jinfa Huang, a doctoral student who was supervised by Professor Jiebo Luo von Rochester department for computer science and both are among the authors of paper. “Magictime is a step towards AI that can better simulate the physical, chemical, biological or social characteristics of the world around us.”

Previous models generated videos that usually have limited movement and poor variations. In order to train AI models in order to improve metamorphic processes more effectively, the researchers developed a high-quality data record of more than 2,000 time-lapse videos with detailed captions.

The open source U-net version of Magictime currently generates 512 to 512 pixel clips (with 8 frames per second) for two seconds, and an accompanying diffusion transmission architecture extends this to ten seconds. The model can be used to simulate not only biological metamorphosis, but also buildings that undergo in bread or bread in the oven.

While the generated videos are visually interesting and the demo can be fun, the researchers see this as an important step towards more demanding models that could offer scientists important instruments.

“We hope that, for example, biologists could one day use generative videos to accelerate the preliminary research of ideas,” says Huang. “While physical experiments for the final review remain indispensable, precise simulations can shorten the iteration cycles and reduce the number of living tests required.”