Storytelling: From Idea to Production

How to go from an idea to an animated video of a story you want to tell? Does it differ now with AI tools?

In my (limited) experience, no. AI makes more tools available that can help along the way, but the same creative process that has been used by animation production houses for years still seems relevant today.

Well, maybe I should back up there a little. If you just want to create AI slop, then you can get AI to generate a story, generate images, and automate a complete pipeline. I have no interest in that. I am interest in human stories, but sharing them on YouTube. By that I mean stories thought up by a human. Sure, you can use AI to review your story and make suggestions, but ultimately the decisions are human ones on what to include.

Here is the Pixar pipeline (source: https://renderman.pixar.com/stories/pixars-usd-pipeline).

• You start with a story idea, kick it around, flesh out the characters, work out the main plot points and what makes the story interesting

• You write a screenplay, crystalizing ideas and dialog

• You develop storyboards, for me blocking out shots using stick people

• I use text-to-speech AI (like ElevenLabs) or voice cloning for voices, making it possible to create a first crude version of the video based on dialog and stick figures. Its enough to help verify pacing which often leads to adding or removing shots. If you can work out changes in shots earlier, that can significantly save effort.

• I am interested in series, so I tend to visually design characters, sets (locations), and props and then reuse them across episodes as much as possible

• With AI, modeling and shading is not quite the same as with 3D animation, but you scan still create Loras which are conceptually similar - fine tunings for models to increase character consistency

• The main rendering pipeline differs between 3D and AI rendering, but you do still need to block out shots, worry about how to control the animation from a starting shot, apply any other effects, and finally consider issues such as lighting

So while there are differences in the individual steps, the overall flow for 3D animation is very similar to that for Pixar 3D rendering. In both cases, it is desirable to get a feeling of an episode early, then refine it. Start with bullet points for a story outline, flesh it out into a script, then storyboards, then rough cuts, each iteration improving the quality of the shot, but identifying pacing issues as early as possible to avoid more expensive later rework.

Originally I thought AI generation would speed up creating shots for scenes that some steps could be skipped. Don’t sketch out storyboards - jump straight to AI to generate images for scenes. To be clear, you can do this, but I found it less efficient than rough sketches for shots. The rough sketches I found gave me a better feel earlier, allowing me to identify and fix problems sooner.

While I am using ComfyUI to do shot rendering, I have found the sheer number of files and workflows at times overwhelming. As such, I have been building features into my web app to reduce the total effort.

• Storyboards start with text panels. I use AI to suggest “cards” (text only panels) from the screenplay, but then manually reorder, insert, and delete cards.

  

• I then upgrade panels from “cards” to “boards”, sketching out crude storyboards with a mouse.

  

• Later, when the final images are worked on, I finally promote panels from “boards” to “shots” with full video clips.

  TODO: replace with YouTube embed (Substack-hosted video; not included in the export)

In my case, I want to release content incrementally, so I am more interested in many short episodes rather than fewer complete movies. I can learn and release as I go. But it also increases the importance of keeping organized. As such, I create checklists (sometimes several) per episode. It helps keep me on track, as well as the “feel good” sensations of ticking things off a list as done.

I use markdown to allow easy formatting, but I create whatever list of tasks I need for each episode. There are common patterns, but also uniqueness especially when learning a new technology.

But returning to the original topic, the focus is on storytelling. I find it easier to visualize the final end-to-end result by going through phases from quick to complete to more lengthy:

• Text “cards” that are really quick to created, then drag them around and edit them until you are happy.

• Image “boards” with quick sketches, possibly upgrading with static AI generated images.

• Boards to “shots” is the next phase, with final AI generation of videos.

At each stage of refinement, the full episode can be previewed to check the story’s pacing and interest level, which can help reduce wasted rework by identifying issues in the story earlier.

Because it does not matter how good shot video clips look, if the underlying story is not interesting, people won’t stay or return.