Adobe Firefly and AI video editing tools in Premiere Pro

What we’ll see in the new Adobe Firefly for video and would Adobe add AI features in its upcoming Premiere Pro update? Let’s find out.

Apart from Adobe Firefly for AI image generation introduced in March, Adobe has announced its expansion to video and audio generation. Additionally, we’ll see new handy features for Premiere Pro, including a new Texted-Based Editing and automatic Color Match.

Adobe Firefly

Something everyone is excited to see - Adobe Firefly, as an image synthesis mode will expand to include video and audio-generation tools.

Currently, performing certain special effects or editing tasks can be labor-intensive. Adobe aims to streamline that process with generative AI assistance. In a demo reel posted on its blog, Adobe showed"Firefly for Video" generating a background soundtrack, generating sound effects, changing the visual season in a video, creating a visual effect on the word“molten," locating and brightening a person's face, and generating audio transcripts via text commands.

An additional tip Adobe shared with the community - you can use pre-made sounds and effects, generated by the creators right before. You can search through the library and pick the right sound from suggested - it seems that this “suggestion” is made by AI, considering the objects identification or the overall atmosphere in your clip:  

What we can also see from the trailer, is that you can easily change the color grading of the clip by writing a prompt:

This would be an amazing addition, as color grading in Premiere Pro hasn’t always been an easy task.

And something that really blows our minds - you would be able to create any object just from a prompt, including animated 3D words with effects. Here we can see, that Adobe generated an animated word “molten” and it looks really nice:

There would be no need to cut out and replace your A- and B-rolls, Adobe Firefly will do it for you in seconds:

Apart from these features, you’ll get a script-generator that will make a storyboard with visualized shots and color grading suggestions:

Now Adobe Firefly is in closed beta testing and we might probably not see it among May updates, but the tool was promised to land“later this year”.

Adobe Firefly for video sounds very promising and innovative, as editors would be able to create their own effects, background music, and sounds based on the text prompt.

Such features made a lot of noise half a year ago when promised by Runway in its teaser, but here Adobe was faster.  

Text-Based Editing

Text-Based Editing will be coming to Adobe Premiere Pro in May and represents a convenient shift in post-production workflows. Adobe is referring to this new feature as of simple as copying and pasting text. Powered by Adobe Sensei, Text-Based Editing automatically analyzes and transcribes clips so editors can copy and paste sentences into any order they want and instantly see them appear on the timeline. Transcripts can be searched in the transcript window to quickly find exact words and phrases.

Adobe initially previewed a similar text-based video-editing capability as part of"Project Blink", half a year ago, before it became a part of the Adobe Sensei framework. Whether the would be a technical difference or just a new addition to Adobe Sensei, is unclear.

It would be a handy built-in feature for editors, though it’s not as revolutionary as Adobe says - some companies on the market already offer a similar feature for video and podcast editing. Still, great to have it in Premiere.  

Color Match

The new automated competitor to LUTs is something that Adobe is going to launch in May. Automatic Tone Mapping and Log Color Detection will help editors and colorists to mix and match HDR footage from different sources into the same SDR project. This means you can take color grades from the clip you like and apply it to your own.

It would be a great time saver, as you won’t need to manually balance footage, including your own shots within the same scene. Easy to set and adjust the result, so we believe many editors will actively use this feature. Additionally, you can enable the Face Detection tool if your and a reference clip contain people - the Color Match will pay attention to the skin tones.

We can’t help noticing, that Adobe takes a look at its competitor, Blackmagic design - such a feature is already available in DaVinci Resolve.  

To Sum Up

We’re extremely excited to see how Adobe Firefly would work with its video and audio generation tools.

It looks like a massive space of abilities for content creators - many are struggling to find the right background music for their YouTube video, or just a suitable video effect. But this is almost nothing, in comparison to video generation and visualized storyboards - such tools would be game changers for the entire industry.

Log color detection would also help a lot - imagine how much time you can save if the color grading would be generated automatically.

Firefly for image generation proved to be a nice tool for graphic designers, but this update could be an even better solution for us, video editors.