| Category | Duration | Tool / Material |
|---|---|---|
| ML/AI; experiment | April 2024 - Present @ University of Washington | Python |
We will compare different Text2Video tools and models with a slightly different set of prompts.
In addition to the test with Text2Video-Zero, one of the
DXARTS pretrained models collaborated with Juan Pampin will be used. The prompts for this experiment are not the same as the
others since we have customized the prompt in our model. Please check the section Text2Video-Zero & DreamBooth.
Overview
| Name | Version / Pipeline | Year | OSS |
|---|---|---|---|
| Runway | Gen-2 | Feb 2023 | ❌ |
| Pika | 1.0 | Dec 2023 | ❌ |
| Haiper | N/A | Mar 2024 | ❌ |
| AnimateDiff | MotionAdapter; AnimateDiffPipeline | Jul 2023 | ✅ |
| ModelScope | DiffusionPipeline | Aug 2023 | ✅ |
| Text2Video-Zero | TextToVideoZeroPipeline | Mar 2023 | ✅ |
| Text2Video-Zero & DreamBooth | TextToVideoZeroPipeline & StableDiffusionPipeline | Mar 2023 & Aug 2022 | ✅ |
Prompt 1
A Zen citrus tree surrounded by mechanical butterflies in a futuristic botanical garden, a citrus is in the middle giving a speech, (impressionist paintings), (((highly dynamic))), big movements, (watercolour style), blending of colours, soft gradient, Chinese painter Wu Guanzhong, Edward Hopper
Settings
motion 10 seed 3764983004 upscale no interpolate yes
Prompt 2
Mechanical butterflies in a futuristic botanical garden, (impressionist paintings), (((highly dynamic))), big movements, (watercolour style), blending of colours, soft gradient, Chinese painter Wu Guanzhong, Edward Hopper
Settings
motion 10 seed 1304017825 upscale no interpolate yes
Prompt 3
Mechanical butterflies
Settings
motion 10 seed 592815518 upscale no interpolate yes
Prompt 1
A Zen citrus tree surrounded by mechanical butterflies in a futuristic botanical garden, a citrus is in the middle giving a speech, (impressionist paintings), (((highly dynamic))), big movements, (watercolour style), blending of colours, soft gradient, Chinese painter Wu Guanzhong, Edward Hopper
Settings
strength of motion 1 seed 8542545107064007 consistency with the text 15
Prompt 2
Mechanical butterflies in a futuristic botanical garden, (impressionist paintings), (((highly dynamic))), big movements, (watercolour style), blending of colours, soft gradient, Chinese painter Wu Guanzhong, Edward Hopper
Settings
strength of motion 1 seed 3467453598348163 consistency with the text 15
Prompt 3
Mechanical butterflies
Settings
strength of motion 1 seed 3467453598348163 consistency with the text 12
Prompt 1
A Zen citrus tree surrounded by mechanical butterflies in a futuristic botanical garden, a citrus is in the middle giving a speech, (impressionist paintings), (((highly dynamic))), big movements, (watercolour style), blending of colours, soft gradient, Chinese painter Wu Guanzhong, Edward Hopper
Settings
seed 2226327552
Prompt 2
Mechanical butterflies in a futuristic botanical garden, (impressionist paintings), (((highly dynamic))), big movements, (watercolour style), blending of colours, soft gradient, Chinese painter Wu Guanzhong, Edward Hopper
Settings
seed 2164124837
Prompt 3
Mechanical butterflies
Settings
seed 1965428839
Prompt 1
A Zen citrus tree surrounded by mechanical butterflies in a futuristic botanical garden, a citrus is in the middle giving a speech, (impressionist paintings), (((highly dynamic))), big movements, (watercolour style), blending of colours, soft gradient, Chinese painter Wu Guanzhong, Edward Hopper
Negative Prompt
text, watermark, copyright, blurry, low resolution, blur, low quality
Settings
num_inference_steps 200 num_frames 24
Prompt 2
Mechanical butterflies in a futuristic botanical garden, (impressionist paintings), (((highly dynamic))), big movements, (watercolour style), blending of colours, soft gradient, Chinese painter Wu Guanzhong, Edward Hopper
Negative Prompt
text, watermark, copyright, blurry, low resolution, blur, low quality
Settings
num_inference_steps 200 num_frames 24
Prompt 3
Mechanical butterflies
Negative Prompt
text, watermark, copyright, blurry, low resolution, blur, low quality
Settings
num_inference_steps 200 num_frames 24
Prompt 1
A Zen citrus tree surrounded by mechanical butterflies in a futuristic botanical garden, a citrus is in the middle giving a speech, (impressionist paintings), (((highly dynamic))), big movements, (watercolour style), blending of colours, soft gradient, Chinese painter Wu Guanzhong, Edward Hopper
Negative Prompt
text, watermark, copyright, blurry, low resolution, blur, low quality
Settings
num_inference_steps 200 num_frames 24
Prompt 2
Mechanical butterflies in a futuristic botanical garden, (impressionist paintings), (((highly dynamic))), big movements, (watercolour style), blending of colours, soft gradient, Chinese painter Wu Guanzhong, Edward Hopper
Negative Prompt
text, watermark, copyright, blurry, low resolution, blur, low quality
Settings
num_inference_steps 200 num_frames 24
Prompt 3
Mechanical butterflies
Negative Prompt
text, watermark, copyright, blurry, low resolution, blur, low quality
Settings
num_inference_steps 200 num_frames 24
Prompt 1
A Zen citrus tree surrounded by mechanical butterflies in a futuristic botanical garden, a citrus is in the middle giving a speech, (impressionist paintings), (((highly dynamic))), big movements, (watercolour style), blending of colours, soft gradient, Chinese painter Wu Guanzhong, Edward Hopper
Negative Prompt
deformed, disfigured, poor details, bad anatomy
Settings
seed 2390 num_inference_steps 100
Prompt 2
Mechanical butterflies in a futuristic botanical garden, (impressionist paintings), (((highly dynamic))), big movements, (watercolour style), blending of colours, soft gradient, Chinese painter Wu Guanzhong, Edward Hopper
Negative Prompt
deformed, disfigured, poor details, bad anatomy
Settings
seed 2390 num_inference_steps 100
Prompt 3
Mechanical butterflies
Negative Prompt
deformed, disfigured, poor details, bad anatomy
Settings
seed 2390 num_inference_steps 100
Prompt 1
(A Zen citrus tree surrounded by mechanical butterflies in a futuristic botanical garden, a citrus is in the middle giving a speech, (impressionist paintings), (((highly dynamic))), big movements, (watercolour style), blending of colours, soft gradient, Chinese painter Wu Guanzhong, Edward Hopper) printed [A] banknote design
Negative Prompt
deformed, disfigured, poor details, bad anatomy
Settings
seed 2390 num_inference_steps 100
Prompt 2
(Mechanical butterflies in a futuristic botanical garden, (impressionist paintings), (((highly dynamic))), big movements, (watercolour style), blending of colours, soft gradient, Chinese painter Wu Guanzhong, Edward Hopper) printed [A] banknote design
Negative Prompt
deformed, disfigured, poor details, bad anatomy
Settings
seed 2390 num_inference_steps 100
Prompt 3
(Mechanical butterflies) printed [A] banknote design
Negative Prompt
deformed, disfigured, poor details, bad anatomy
Settings
seed 2390 num_inference_steps 100