From Deepfakes to Facial Capture: How Artificial Intelligence Is Already Changing Hollywood

7:00 AM 5/13/2020

by Carolyn Giardina

AI looks poised for use in all phases of making a movie, from analyzing viewers' reactions to creating digital humans: "Decades from now, an AI algorithm will make your movie from the text of the script."

Walt Disney Studios Motion Pictures/Photofest; Adobe;Marvel Studios; Columbia Pictures Corporation/Photofest
  • Operations

    'Star Wars: The Rise of Skywalker'
    'Star Wars: The Rise of Skywalker'
    Walt Disney Studios Motion Pictures/Photofest

    One of the hum-drum housekeeping tasks that AI already handles at VFX studio Industrial Light & Magic is overseeing the workflow of rendering animation and visual effects. "We use it for scheduling render jobs on the render farms to get our workloads done every single night in a predictable fashion," says Rob Bredow, senior vp, executive creative director and head of ILM, whose recent work includes Star Wars: The Rise of Skywalker. "One of the things that's interesting about machine-learning algorithms is, even when they're only 80 or 90 percent effective, sometimes their guesses tend to pretty much align with if you had put a person there and had them make that guess."

  • Editing

    Adobe Premiere Pro Auto Reframe uses AI.
    Adobe Premiere Pro Auto Reframe uses AI.
    Courtesy of ADOBE

    Bones editor Harry B. Miller relates that AI is used for certain tasks in various editing systems including Avid's Media Composer, "which has a software add-on that matches dialogue spoken in dailies to the script, allowing the editor to quickly find every take and angle for that part of a scene." Adobe's Premiere Pro can also use AI "to reframe video for different aspect ratios," adds Miller, though he sees limitations. "A computer may know what a joke is, but it can't tell what is funny," he says. "Stanford University has worked on software that takes motion picture dailies and assembles shots in script order. But it can't tell what is the best dramatic scene. To know what 'works,' for now, takes a human."

  • Visual Effects

    A facial capture system with AI helped create Josh Brolin's Thanos.
    A facial capture system with AI helped create Josh Brolin's Thanos.
    ©Marvel Studios

    VFX house Digital Domain used AI as part of its Masquerade facial capture system to superimpose Josh Brolin's performance onto the villain Thanos in Avengers: Infinity War. "We've seen huge advancements in that area of research lately," says Darren Hendler, director of the studio's digital human group. "Real-time rendering together with our AI-driven, real-time facial capture system is changing the timelines for creating movie creatures. Soon we'll be able to generate realistic Thanos-like performances live on set." Hendler says a suite of deep-learning technologies on the horizon "can potentially allow us to quickly and realistically transform a stunt double into an actor or change an actor's lines."

  • Digital Humans

    Axelle/Bauer-Griffin/FilmMagic; BRENDAN SMIALOWSKI/AFP via Getty Images

    AI is having a "drastic" effect in the world of digital humans, says Christopher Nichols, director of Chaos Group Labs and key member of the Digital Human League, a research and development group: "The massive challenges that were put on artists and tools to find all those subtle nuances that make a human look real are actually much easier to solve through deep learning." The improvements have also led to deepfakes, such as the viral faux PSA using FaceSwap AI in which Jordan Peele ventriloquizes former President "Barack Obama" calling President Trump a "complete dipshit." Says Nichols: "A great number of people want to stop the technology and outright outlaw it. [But] the tools themselves have huge potentials that we should all continue to explore." He concludes, "What I believe is that we need more education on the subject and better tools that help detect what is real and not real. We have come to accept that Photoshop can manipulate images very well and easily, and we are not banning that as a tool. But we have come to understand and doubt every image that is presented to us.”

  • Analytics

    AI helped predict which moviegoers 'Star Wars: The Rise of Skywalker' would find 'Once Upon a Time' appealing.
    AI helped predict which moviegoers 'Star Wars: The Rise of Skywalker' would find 'Once Upon a Time' appealing.
    Columbia Pictures Corporation/Photofest

    Vista Group's Movio is a marketing data analytics firm that focuses on the cinema industry, applying AI to analyze audience behavior and box office trends for releases like Hobbs & Shaw and Once Upon a Time in Hollywood. With clients including all the major studios, co-founder and CEO Will Palmer says that AI helps Movio "to predict which movies will appeal to each and every moviegoer" and offer "insights used to drive marketing and distribution strategy and support creative decision-making. Ultimately, AI removes complexity and continuously learns more about moviegoer behavior."

  • How About a Movie From Scratch?

    "Decades from now, an AI algorithm will make your movie simply from the text of the script," even in a particular director's style, predicts Paul Debevec, senior scientist at Google. "Both scripts and features are just strings of 0's and 1's on a computer. If we provide a sufficiently large neural network and enough training pairing scripts and movies, it will learn the mapping and generate an entirely new movie from an entirely new script. You could even ask it to make your new film — or remake an old one — in the style of Kubrick, Tarantino or Scorsese by emphasizing the correlations between each of their scripts and movies."

    This story first appeared in the May 6 issue of The Hollywood Reporter magazine. Click here to subscribe.