Working on a Short Video Pipeline
Why?
I have always wanted to make videos and it seems like the time has to come to finally do it. I am soon going to start an internship at MotherDuck in the DevRel team. And I wanted to test myself at communicating around the product which will help me understand the needs of the team better.
Fully AI Generated Path
You might have noticed the flood of AI made videos appearing on your feed recently. Indeed, LLM's have become very good at animating videos through html/css/javascript. The first thing I tried was to
- find prompts that were proven to generate good animated videos
- take a subject (here: Why postgres fails to scale for analytical workloads)
- define the style (I just linked the MotherDuck website)
- Give an API key for TTS providers (ElevenLabs and Cartesia) We then had a grilling session with the Agent but essentially I asked it to adapt the prompt to that MotherDuck style and to use that subject.
You might have heard about HyperFrames or other such frameworks that are designed to animate videos through React. But having tried it in the past, I feel like starting from example prompts from this website1 that uses plain Javascript worked better for me. Funnily enough, many people actually try to sell a magic workflow for hundreds of euros that outputs videos weirdly similar to what you get using the prompts from the above.
I got the first draft and here are my first observations.
- The word delivery is very bad, not natural, has pauses when it shouldn't have, rushes when it should'nt rush.
- In general the script is also very robotic (as expected), it lacks all connectors such as "however", "as opposed to",...
- The animation looks amazing
Hybrid Path
As good as AI voices are getting, to me it just feels so much better to have an actual human talk to you and see his face. So I put the AI voices in the bin and started working on a new workflow that kept what worked in the previous one (the style and animation method).
A simple workflow looks like this:
- I pick a subject (e.g. a technical document)
- The AGENT generates a script
- I review the script
- If I accept, the script, the AGENT generates a storyboard
- I record a video of me narrating the script.
- The AGENT parses the transcript of my speaking and converts to text and timestamps
- The AGENT edits the video, cuts silences,...
- The AGENT adapts the storyboard around the video and animates it
The part where it struggles the most seems to be the script. This is what I tried to tell the Agent and why:
- You have to specify the audience (or ask it to suggest different audiences and pick one)
- If the technical document is too long, tell the agent it can make one or more short scripts, otherwise it will probably fail to make a cohesive story, it will try to rush to attain the 60 seconds mark with way too much content to summarize and struggle to keep what's really interesting. I found that giving it an escape path (make more than one scripts if you need) helps in that case.
- Make a script that's speakable. Otherwise it comes back with awkward sentences like: "Because none of them change how Postgres stores data: row by row, one order at a time."
- Review the script using independent subagents and iterate. (That's just a review loop)
The Hardest Part
In the end I found that making your own script from the summary the Agent gives you is much easier, You use your own words and it feels natural to read the script out loud.
But the hardest part is none of the above... Speaking naturally to a camera is very difficult, even harder when it's in not your native language. So I have been practicing in the past couple of days and it's getting better. But it takes years to become consistently good at this.