Today, Meta Platforms Inc. showed off a high-tech “generative artificial intelligence system” that is meant to help artists show off their creativity better.
The “Make-A-Scene” system is meant to show how AI could make it possible for anyone to make their ideas come to life. The user can simply describe and draw their vision using a combination of text and freehand sketches, and the AI will come up with a stunning representation of it.
In a blog post, the company says that some artists already use generative AI to help them be more creative. Some examples are avatars with feelings, animating children’s drawings, making virtual worlds in the metaverse, and making digital art with only text descriptions. But the systems we have now are still pretty simple, and artists don’t have much control over the end result.
The problem is that most of them are just text. So, a text prompt like “a picture of a zebra riding a bike” could mean many different things. The AI could put the zebra on the left or right side of the picture, or it could be much bigger or smaller than the bicycle. It could be facing the camera or be turned to the side. A lot of the process of making something new is random.
The research team at Meta AI said, “It’s not enough for an AI system to just make content, though.” “For AI to fulfill its potential to advance creative expression, people should be able to shape and control the content a system makes. It should be easy to understand and use, so people can use speech, text, gestures, eye movements, or even sketches to bring their visions to life in whatever mediums work best for them, such as audio, images, animations, video, and 3D.
This is where Make-A-Scene comes in. It lets artists use text prompts and simple sketches to better show what they have in mind. The system works like other generative AI models. It learns the relationship between images and text by training on millions of example images.
Once trained, Make-A-Scene uses a new method to capture the layout of a scene as shown in a rough sketch. It then uses the text prompts to fill in the details. It has also been taught to pay attention to things like animals, people, and objects that are likely to be more important to the artist. Because of this, users have a lot more say over the image they ask the system to make.
Sofia Crespo, Scott Eaton, and Alexander Reben, all well-known AI artists who have worked with generative AI before, put Make-A-Scene to the test.
Crespo is a generative artist who looks at how nature and technology work together. She used Make-A-Scene to take pictures of different artificial lifeforms that have never existed anywhere but in her mind. Make-A-Scene took her sketches and text prompts and came up with a bunch of strange creatures, like a flower-shaped jellyfish.
Crespo said that using freeform drawing lets her try out new ideas much more quickly than before. “It will help move creativity along a lot faster and give artists easier-to-use interfaces,” she said.
Eaton, a “creative technologist” and contemporary artist, used Make-A-Scene to carefully put together scenes and try out different variations using different text prompts, such as “skyscrapers sunk and falling apart in the desert,” to show how climate change affects the world.
Eaton said, “Most text-to-image models are very good at giving you some part of what you asked for, but you don’t have control over where the image is placed.” “This is a very powerful way to interact with the model if you want to make an AI-generated picture with a planned composition.”
Meta said that Make-A-Scene is part of a larger research project that is looking into how AI can help people be more creative in many different ways, such as bringing 2D sketches to life or using natural language to make 3D objects or even a whole virtual space.
Meta AI’s researchers said, “It could one day make possible completely new forms of AI-powered expression that put creators and their ideas at the center of the process.” This could be an art director coming up with ideas for their next creative campaign, a social media influencer making more personalized content, an author making unique illustrations for their books and stories, or a person sending a fun, unique birthday greeting to a friend.
Mark Zuckerberg, the CEO of Meta, said that he is excited about art made by AI, especially when it comes to his main focus, the metaverse. “Our research team made a prototype that lets you make pictures with a simple sketch and some text,” he said. “This is a great tool for creators, especially as they build more and more immersive 3D worlds.”