A new artist-in-residence has arrived at Carnegie Mellon University’s Robotics Institute, and it’s a robotic arm with a paintbrush. FRIDA, which stands for Framework and Robotics Initiative for Developing Arts, uses artificial intelligence to collaborate with humans on works of art.
How FRIDA Works
- FRIDA can be directed by inputting a text description, submitting other art pieces to inspire its style, or uploading a photograph to paint a representation of it.
- The robot uses AI models similar to OpenAI’s ChatGPT and DALL-E 2, which generate text or an image in response to a prompt.
- FRIDA simulates how it would paint an image with brush strokes and uses machine learning to evaluate its progress as it works.
- The robot spends an hour or more learning how to use its paintbrush and then uses large vision-language models to understand the input.
- FRIDA uses real2sim2real, reducing the simulation-to-real gap between what it composes in simulation and what it paints on canvas.
Exploring Human and Robotic Creativity
The FRIDA project explores the intersection of human and robotic creativity. The robot is using AI models developed for captioning images and understanding scene content, but applying it to an artistic generative problem.
- The team is addressing limitations in current large vision-language models by continually refining them and providing them with a sense of what is happening in the world.
- The multicultural collaboration effort includes contributions from countries such as China, Japan, Mexico, Nigeria, and Norway.
- Once the user specifies a high-level concept of the painting, FRIDA uses machine learning to create its simulation and develop a plan to make the painting.
The Final Product
FRIDA’s final products are impressionistic and whimsical, with bold brushstrokes. If the robot makes a mistake, it incorporates the errant paint into the final result.
- The whole process takes hours and includes capturing images of the painting to evaluate its progress and refine the plan, if needed.
Artists, Don’t Worry!
Despite the impressive capabilities of FRIDA, it is not meant to take artists’ jobs. The main goal of the FRIDA project is to collaborate with artists and provide them with a new tool for creating art.
“FRIDA is not generating the ideas to communicate. FRIDA is a system that an artist could collaborate with. The artist can specify high-level goals for FRIDA, and then FRIDA can execute them,” said Peter Schaldenbrand, a Ph.D. student in the Robotics Institute working with FRIDA.