QWEN Image Edit from Sketches

Previously I tried rough sketches of poses with Google Gemini’s Nano Banana model. It worked pretty well, but had a tendency to center things. (Easy workaround - crop the resultant image.) So I thought I would try an open source model, QWEN Image Edit, in case it surprised me. TL;DR: sigh!

First a reminder of why I care about rough sketches. The normal workflow for an animated movie is you start from a story, turn it into a script, draw up story boards, then go from the storyboards to setting up and rendering out the rest of the movie. (Yes, I skipped lots of details in the last step! See my previous post for more details.)

Related: Storytelling: From Idea to Production

The idea is speed - try out ideas and refine quickly. Also, to be clear, I am interested in tools to help humans create their vision of a story. I am not trying to just create videos - I am trying to create videos that match the creator’s vision. I want control. I want a fast way to visualize what something may look like before creating the real version, then provide fast ways to iterate and improve each part. I don’t want to render out a full video sequence, only to find that the basic story is lacking. So early previews matter. Quick sketches are one way to get that rough “feeling”.

QWEN Image Edit accepts 3 input images which you can refer to as “image 1”, “image 2”, and “image 3”. So I fed the images in and…

Err, yeah! Not quite what I was after.

It may be my prompt could be improved, but some observations. If you look at the guy, it CLOSELY followed the outline I drew. The arms were stretched and angled in the same way as my rough sketch. Even the stare on the chest maintained the same look. So the problem was not that it did not understand the sketch, the problem in my case here is it followed it too closely! Even if I could get it to use the robot instead of making up a new character, this precision is too great.

I did try beefing up the text description, but it still struggled, with the same results for the sketch - following it too accurately.

Note: I still like QWEN Image Edit, I am only saying rough sketches may not be its forte. Instead, there are other approaches like turning the outline into an inpainting mask and asking a model to infill may work better. (Or learning to draw! Lol!)

Here is an example workflow that uses dwpose to extract a pose skeleton, then use that as a control net to feed into QWEN Image Edit. But it needs a reference pose to start from.

Note that there are different ways control nets have been trained to follow poses:

• Using skeletons like the above

• Using edge detection (Canny is a popular model), which may be what QWEN Image Edit above decided to do (treat my sketch as edges to be followed)

• Using 3D depth maps where variations in grey indicate depth

The thing I like about skeletons is using Google MediaPipe, I can take a webcam image in a webapp I am building and convert it live to a 3D skeleton, which I can also rotate and position in a scene. Edge detection and 3D depth maps I fear may try to follow the outline too closely, meaning the image needs to follow the actual character. Skeletons capture the core detail I want - the pose - not the details.

Ideas for future experiments:

• Use Nano Banana to convert sketch into a posed character, extract a pose skeleton, then use that with a model like QWEN Image Edit to compose a frame.

• Don’t use sketches - take photos of humans acting out a scene instead.

• Build a library of poses and use them to create sketches, more like dropping clipart to build up a storyboard.

• Or don’t try to go from storyboard artwork to a more detailed image. Just use it as inspiration and discard it.