Right now it feels like artificial intelligence is the hot topic in the media industry and beyond. From chatbots creating metadata to the metaverse and synthetic humans, there are countless ways the technology is being employed. And one of the newer ways is using AI to either de-age or even bring a celebrity back to life.
This month’s Indiana Jones and the Dial of Destiny uses artificial intelligence to bring a younger Harrison Ford to the screen. Ford has spoken about the process, explaining that LucasFilm used AI to go back through all the films he has made for the company, including footage that’s never been seen before. “They could mine it from where the light is coming from, the expression,” said Ford. “But that’s my actual face. I put little dots on my face and I say the words and they make it. It’s fantastic.”
Metaphysic is a company looking to bring this kind of technology to a wider audience. Viewers of America’s Got Talent have already seen it in action after the company’s Chris Ume and Tom Graham showcased the technology on the show. They worked with previous AGT finalist Daniel Emmet to make it look as if Simon Cowell was singing on stage. Cowell and the judges were so impressed, Metaphysic made it all the way to the final, finishing fourth.
Going back to the start
Metaphysic was formed in 2018, but the company has seen real momentum in the last 18 months. “We have north of 50 employees now,” explains co-founder Martin Adams. “We brought in some excellent people, including Sam Head (aka Shamook), who’s a bit of a legend in the space. We’ve really grown out both the tech and the creative team.”
Adams has worked in the cultural industries for almost 20 years, including setting up one of the world’s first digital transformation consultancies for talent, working with people like Barack Obama, Will Smith, and Hugh Jackman. “I started to understand very quickly the use of data could help them be more effective in their comms and their online presences,” he says. “I’ve always had this schizophrenic background between the cultural industries and law and intellectual property. I studied for a Master’s at Harvard Law School, which is where I met my co-founder Tom, who’s CEO of the business, and I became obsessed with intellectual property because I realised that for modern businesses to create value they didn’t need to just make an impact but retain the value of that impact by having intellectual property.”
After Harvard Adams and Graham set up codec.ai together, which uses machine learning and AI to make brands act more like entertainers and produce content that would engage audiences online.
The pair partnered with Chris Ume, who famously created @deepTomCruise on TikTok and now has 5.2 million followers, to launch Metaphysic with the aim to use artificial intelligence to create content. “I’ve always been interested and committed to using technology to impact culture, which is quite rare. Most technology companies are really about efficiency, and doing things a faster way. Tom and I have always been interested in being able to get a more exciting and magical creative output from using AI,” explains Adams.
“It’s important to us that audiences are charmed and delighted by what they’re seeing and consuming. We wanted to birth a company that would take that technology on to the next level, but build it with ethics and permission and consent at the heart of the business model, which was and is quite a rare approach.”
That means that the Metaphysic team is principally in control of the technology. They work with key people in the entertainment industry, with the current focus on Hollywood and film studios. “We insist upon full and informed consent from them contractually, but also from any talent that they’re looking to have the technology applied to,” states Adams.
Recently it was revealed that Keanu Reeves has a clause in his contracts that bans studios from digitally editing his performances. “Good on him,” says Adams. “It’s his data. It’s his agency to do that. We would always insist on permission.”
How it works
The technology works by training AI models based on captured or existing historic data, which allow a set of human behaviours of an underlying performer to be enhanced with what the model thinks needs to happen to match that person. Or, in simpler terms, a model is created of the talent who have given permission for their identity to be captured. That can be either historic footage or captured live, such as movements and dimensions of their face, emotions, how they smile, how they might wince, how they might look angry, and all of that information is fed into the model.
“Once you’ve got the model of an actor, you have an underlying performance, and the technology will fill in all of the gaps,” explains Adams. “When the actor is smiling, the AI-generated character will smile in the same way, or when they’re frowning, it will do that. The algorithms jump the bridge between the actor and their character in their reactions and performance.”
The technology needed to capture the actor’s performance can have various degrees of sophistication. “Higher fidelity and more angles are valuable, but it’s certainly not restrained to that,” says Adams. “On some level, being able to capture it with a phone is helpful. It’s not even a total dependency, if you have enough publicly available data of a deceased star or a current film actor or musician, you can capture them just using historical data.”
As mentioned earlier, Metaphysic showcased what they could do on America’s Got Talent, both with ‘live talent’ and using footage of Elvis Presley. “What’s interesting is the technology has got to the point where what you might think would be the hardest thing, such as resurrecting Elvis live on stage, is not the case anymore,” Adams adds. “Actually, you’re back into traditional business in terms of negotiating with estates making sure you have full permission, making sure they’re comfortable with what is happening on stage and that it fits the brand that the estate is trying to uphold.
“To be honest, there’s a genuine, unfair advantage on some level that we feel we have in Metaphysic in that we are a mixture of absolute cutting edge, innovative and creative technologists and machine learning experts, but also that we have the commercial and the legal experience and background to put those pieces together and build innovatively but appropriately and sustainably.”
A seismic shift
The technology has caught the attention of Hollywood, and is currently being used at Pinewood Studios on the set of Robert Zemeckis’ latest film, Here. It stars Tom Hanks and Robin Wright with the actors portraying their characters over the course of their lifetimes. To do this, Zemeckis is employing hyperreal AI-generated face replacements and de-aging into the storytelling, without the need for further compositing or VFX work.
“We’re hugely excited to be working with a truly world-class and frontier-pushing director and team,” states Adams. ”There’s an extra special feeling of contentment that comes with seeing maybe the most famous and best actor in the world, Tom Hanks, falling in love with this technology. It’s really quite special for us.”
Zemeckis is employing Metaphysic Live, a brand new technology developed by the company that live streams high-resolution face swaps for multiple characters in real-time. “That means there’s no further visual effects or compositing work that’s required,” explains Adams, “and both the director and the performer can see the results live as the performance is being delivered. The feedback loop allows the actor to perform with more humanity and to fine-tune their performance.
“Robert Zemeckis using it as his primary feed, he’s essentially directing from that perspective, which is magic in a sense,” he adds. “You’re changing reality, and you’re building empathy, you’re actually building a relationship as a director and being able to direct that character live. That’s a seismic shift.”
As well as film production, the technology could be used to take Hanks in the metaverse. “The environments that content sits within are changing and therefore, the content itself will adjust as well,” says Adams. “We think those structural shifts will be generally moving towards more personalisable content. So there’s kind of another bridge to be built between the performance and the audience member consuming that content. In a simple sense, you can absolutely have Tom Hanks in a powered-up digital environment, performing live, and you have a live synthetic representation of him as a different person or as a different gender or a different age or whatever it might be, playing with all of those variables that are available to us.
“The other piece of the metaverse vision is that the technology that is available to studios now, we ultimately want to put it in the hands of normal people,” he continues. “We believe that audiences will be powering themselves up into more immersive, more media-rich sort of environments to consume content. We want to give the power to people to kind of capture their identity and control its usage so that they can be on the receiving end of those amazing hyperreal virtual immersive performances that will happen in the metaverse.”
If that all sounds very Ready, Player One, well you’re not alone. “Reality often follows science fiction on some level,” admits Adams. “I hope it will be less violent, and I believe that it will. I think if you stick to the principles that have generally guided society, which is respecting what people own and respecting that it is theirs and giving them some sort of incentive to participate in its usage, if you stick to the principles of informed consent and permission for working with someone’s identity, and your ultimate goal is to enhance the creative industries rather than ‘disrupt’, then I think you will have a metaverse that is healthy, enriching, educational, and entertaining.”
The rate of technological innovation at which the media industry is accelerating, especially around uses of artificial intelligence, is unprecedented. For the team at Metaphysic it’s been even faster. As they moved through the rounds of America’s Got Talent, the quality of the models the team were able to create was constantly increasing while the time it took them to create those models decreased.
Asked where the technology will be in 18 months’ time, Adams says he would expect to see two things: “Breakneck rates of change in terms of the time to get to hyperreal quality, because of the investments that we are making as a company into proprietary technology. We are building out our own patent portfolio around this. My background, as I mentioned, is as an intellectual property lawyer, so we’re taking that very, very seriously.”
The second thing Adams expects to see is the audience’s hunger to have content experiences within the metaverse or immersive environments. “There’s a general willingness to experience things digitally, and so I think that when it comes to consumption of the major entertainment forms of music and film, we’ll see both the capabilities and the technology continue to advance at this breakneck speed,” he says.
“The direction of travel is categorically towards hyperreality. Too much of the current market is about cartoony representations of these digital experiences. We believe that as the audience and market mature, people will want to get hyperreality in those immersive experiences and that’s a big part of our vision, and our investments.”
