It's a fantastic option for creating AI animations, if and/or when you can actually get it to do something interesting for you!
A week or so ago Deep Dream Generator released their new Runway animation generating option. It was announced on their front page along with an embedded video which showed an eXample featuring a woman walking down a street under a 'Starry Night' sky.
The sample video was smooth and realistic, with quite a wow factor, and I couldn't wait to try it out for myself. So off I went to browse through my gallery to see what image I'd like to have animated.
I settled on my 'Deep Rooted Sorrow' picture and uploaded it into the Runway option. Then I wrote a brief prompt to eXplain that I wanted the tree roots and branches to wiggle and writhe as they worked their way up to cover the face.
Here is what I got in return:
Oh my, instant disappointment and regret for spending the energy points! The AI totally ignored my written instructions and just did its own thing. Instead of moving the branches and roots around, it just panned the camera a bit and moved the background to give the effect of a 3-D image.
BOOOOOOO! This wasn't anywhere near what I had envisioned in my mind, nor anywhere near to the movement that DDG's introduction video had displayed. WTH DDG?! Was this just some sort of grift devised to get me to waste my energy points needlessly?
I thought I was done with it all, but then I noticed some other people's videos where they were indeed getting some awesome movement in them. DDG was having a sort of 'sale' on making the videos, ("only" 25 energy points, for a limited time only), so I thought I'd try again.
This time I chose my 'Babushka with a Buzz' image. It was already very psychedelic, and I thought that if I could get those squiggly and swirly lines to animate I'd have a drug user's dream-come-true that I could present to the masses!
Again I uploaded the image and again I wrote a well thought out yet brief prompt to go with it, and again I was very eXcited to see the outcome.
Here's what they gave me:
So yeah, AGAIN I was very disappointed with the outcome. The movement and morphing was interesting, but nothing at all like what I had envisioned in my mind, not even close! I had now wasted 50 energy points and was done, my grand animation career apparently over.
But then I saw someone else's Runway video that had amazing movement in it again. What were they doing to achieve this? What was I doing wrong to NOT achieve it? It was an irritating mystery that got me to spend yet another 25 points, but 3 strikes and I'm definitely out!
So again I went back to my gallery to pick out an image. This time I chose my 'Down in the Daisy Field' picture. There was no specific reason why I chose the image, I just happen to like it.
Once again I uploaded it into the system, and this time I didn't even bother writing a prompt to go with it. Why should I? The AI apparently could care less what the prompt says, anyway.
This is what I got back:
AT LAST!!! I finally got an animated video that was worth the points I had spent to make it! It was FABULOUS, (if I do say so myself), and I have to admit, I was totally floored with the outcome!
In fact, I was so impressed with the video that it made me break my 3+ month hiatus from publishing anything publicly at DDG! Other folks there needed to see THIS!
But why was I so successful this time around while my other attempts were such dismal failures? I scrutinized my images for an answer, then went and scrutinized the videos of others, both successful videos and the not so much so, to see if I could figure it out.
I'm not sure if I'm right or not, but I do now have a theory about it all and why some images work well while others do not.
I think that the static images we upload into Runway have to have an obvious sense of movement in them that the AI can understand, or "see" for itself. My first 2 images didn't have this movement, but I think the 3rd did.
For instance, in the 3rd image I uploaded, the woman's hair is up in the air, like a sort of 'action shot', and the daisies are all over the place, some of them just scattered in the sky around her. I think the AI was then able to deduce that her hair should fall down, as well as the daisies, and made the animation accordingly.
I think it's also the reason that DDG's promotional video had such great animation to it. The static image must have shown the full body of a person walking down the street, so the AI deduced that their legs should be moving to walk, and the area around the person also should be moving to show that the figure is moving forward in their surroundings.
Of course this is just my theory, but I think I'm correct. If you want to try out this new option, make SURE there is movement in your static image that is easy to 'see' by the AI, otherwise you'll be left with disappointment at the outcome, like I was with my 1st two tries.
This is actually something that I'd really enjoy eXperimenting with further, but the sale is now over. This morning I checked and it had gone up to 50 points for a 5 second video instead of the 25 points introductory offer. Just now I checked again, and it's gone up another 20 energy points from this morning and is now 70 energy points for a 5 second video! YIKES!
This is way too pricey for eXperimentation purposes, especially since you just don't know what you're going to get in return. If I were guaranteed to get a good outcome I *might* be inclined to try again at these prices, but that's just not the case so I ain't playin' with it no mo'!
But I really am quite delighted with my 3rd video attempt's outcome and have to give DDG's Runway video model 2 thumb's up for what it can do...if it happens to like the image you feed into it!
I imagine that future video creating features that are still in the worX for upcoming release will blow our soX off! I'm looking forward to those, and if they're on sale again at reasonable prices, I will once again eXplore them while I can afford to! Of course I'll share my findings here again in the X-Files should that happen :)
Until then, I hope you're having a wonderful & animated day!
"X"
October 28th, 2024