Seaweed, short for "Seed-Video," is a research initiative aimed at building a foundational model for video generation. It includes diffusion transformers with approximately 7 billion (7B) parameters, trained using an average of 1,000 GPUs. Seaweed learns world representation from massive amounts of multi-modal data such as video, image, and text. It allows for creating videos of various resolutions, aspect ratios, and durations from text descriptions. In this article, we present its generated videos and highlight its hallmark capability as a foundational model capable of supporting a wide range of downstream applications.
Our model is highly adept at generating lifelike human characters that exhibit a diverse array of actions, gestures, and emotions.
Seaweed excels at generating a wide variety of landscapes. With intricate detail and dynamic composition, it can create visually stunning environments that enhance storytelling.
Seaweed is capable of generating highly realistic animals and lively beings with remarkable detail and accuracy.
Seaweed can also generate stunning fantasy scenes, including futuristic environments, mystical landscapes, and otherworldly settings. It can bring complex, imaginative worlds to life, complete with intricate details and striking visuals.
We demonstrate our model's generative capabilities through a short film. All the videos are generated and the only manually added components are the background music and the ending titles.
Our video generation model offers enhanced controls that allow users to precisely create the content they envision. By providing an image as the first frame, users can direct the model to generate the rest of the video with consistent motion and style. This grants users full control over the visual aesthetics, making it ideal for applications where accuracy and creative direction are crucial.
We demonstrate our model's image-to-video generation capability, where the same input image can produce diverse videos based on different prompts.
Our model can also condition on both the first and last frames, allowing it to generate interesting transition videos for greater creative control.
Our model can also be finetuned to generate videos based on reference images, offering flexible input options for users. Whether it's a human reference image, an object reference image, or a combination of multiple reference images, the model can synthesize them into dynamic video sequences.
Learn more about PhantomSeaweed is adapted to generate content conditioned on audio inputs by Omnihuman, enabling the creation of realistic human characters that perfectly match the voice in the audio. The model ensures synchronized lip movements and body gestures that align with the tone and timing of the audio, creating a seamless and lifelike interaction.
Seaweed is also capable of generating both audio and video together. The audio generated is synced to reflect the action, scene, tone, rhythm, and style of the video. The audio complements and elevates the visual storytelling, providing a seamless multimedia experience.
Seaweed supports natively generating a single shot lasting 20 seconds without any extension technique. With the extension, it can generate videos up to a minute long.
Seaweed is capable of generating consistent, multi-shot, long-form stories, maintaining continuity across scenes and shots. Users can provide both a global text description for the overarching narrative and fine-grained text descriptions for each individual shot.
Learn more about Long-Context-Tuning Learn more about VideoAuteur
Overview of the forest.
Close-up shot of the trees.
The boy follows the girl.
Cut to the front of the girl.
The boy follows.
Another overview of the forest.
The girl talks to the boy.
The boy becomes serious.
The girl becomes nervous.
The girl walks forward.
Drone view.
Wide angle ground view.
Camera dolly in.
A house appears in front.
Close-up shot of the house.
Close-up view of the characters.
The door. Dolly out.
The boy tries to open the door.
Inside the empty room.
The characters walk inside.
They look around in the house.
They walk into a new room.
An old bookshelf.
Close-up shot on the shelf.
The characters walk to a table.
A glowing ball floating on a map.
They look at each other.
Seaweed natively supports generating videos up to 1280x720 resolution. The result can also be further upsampled to 2K QHD (2560x1440) resolution. The super-resolution module can be separately applied to existing videos for upsampling and restoration.
Learn more about SeedVRSeaweed can also generate videos in real-time at 1280x720 resolution and 24fps. This is particularly valuable for real-time and interactive applications, where immediate video generation is essential.
Learn more about Seaweed-APTSeaweed can natively support camera control through text descriptions, offering a range of options for different camera movements. Users can specify camera actions, such as panning, tilting, zooming, or more complex movements like tracking and orbiting.
Seaweed can be utilized for modeling precise camera control through defined trajectories, providing not only enhanced creative direction but also an interactive way for users to explore the simulated world. With its real-time generation capability, Seaweed also serves as a foundational model for advanced research in world simulation.
Learn more about CameraCtrl-IISeaweed can also be post-trained on synthetic video rendered via computer-generated imagery (CGI), enabling it to enhance the physical consistency in video generation while preserving photorealism. Below, we showcase generated videos with superior 3D consistency and precise human pose integrity in complex actions, alongside the synthetic videos used for training.
Feng Ling, Huafeng Kuang, Huixia Li*, Jerry Duncan, Jiashi Li*, Junda Zhang, Junru Zheng, Li Sun, Manlin Zhang, Renfei Sun, Rui Wang*, Shu Liu*, Xiaojie Li, Xin Xia, Xuefeng Xiao*, Xuyan Chi, Yanghua Peng*, Yuxi Ren*, Zhongkai Zhao, Zuquan Song
Bingchuan Li, Chao Liang, Chongyang Ma, Deyao Zhu, Gaojie Lin, Gen Li, Haibin Huang, Hao He, Jianwen Jiang, Jianyi Wang, Jiaqi Yang, Jiawei Liu, Junfei Xiao, Lijie Liu, Qian He, Siyu Zhou, Tianxiang Ma, Xiaobin Zhuang, Xiaohui Shen, Xinglong Wu, Yuping Wang, Yuwei Guo, Yuxuan Wang, Zerong Zheng, Zhuo Chen, Zhuowei Chen