n8n workflow template · 14 nodes · OpenAI
Narrating over a Video using Multimodal AI is a free n8n workflow template that starts when you run it manually in the n8n editor and works with OpenAI and Google Drive. It has 14 nodes, 3 of them AI nodes (OpenAI Chat Model, OpenAI and Basic LLM Chain). The notes inside the workflow say: “This template was inspired by Processing and narrating a video with GPT's visual capabilities and the TTS API” To run it, you add your own credentials for Google Drive OAuth2 and OpenAI API.
The text below comes from the sticky notes inside the workflow. We removed links, code samples and anything that looks like a key or token.
This template was inspired by Processing and narrating a video with GPT's visual capabilities and the TTS API
Sample the finished product here
Join the Discord or ask in the Forum!
Learn more about the HTTP Request node
In this demonstration, we'll download a stock video from pixabay using the HTTP Request node. Feel free to use other sources but ensure they are in a format support by OpenCV (See docs)
The video used in this demonstration is © Coverr-Free-Footage via Pixabay
Learn more about the Code node
We need to think of videos are a sum of 2 parts; a visual track and an audio track. The visual track is technically just a collection of images displayed one after the other and are typically referred to as frames. When we want LLM to understand videos, most of the time we can do so by giving it a series of frames as images to process.
Here, we use the Python Code node to extract the frames from the video using OpenCV, a computer vision library. For performance reasons, we'll also capture only a max of 90 frames from the video but ensure they are evenly distributed across the video. This step takes about 1-2 mins to complete on a 3mb video.
Using large videos or capturing a large number of frames is really memory intensive and could crash your n8n instance. Be sure you have sufficient memory and to optimise the video beforehand!
Read more about the Basic LLM node
To keep within token limits of our LLM, we'll need to send our frames in sequential batches to represent chunks of our original video. We'll use the loop node to create batches of 15 frames - this is because of our max of 90 frames, this fits perfectly for a total of 6 loops. Next, we'll convert each frame to a binary image so we can resize for and attach to the Basic LLM node. One trick to point out is that within the Basic LLM node, previous iterations of the generation are prepended to form a cohesive script. Without, the LLM will assume it needs to start fresh for each batch of frames.
A wait node is used to stay within service rate limits. This is useful for new users who are still on lower tiers. If you do not have such restrictions, feel free to remove this wait node!
Read more about the OpenAI node
Finally with our generated script parts, we can combine them into one and use OpenAI's Audio generation capabilities to generate a voice over from the full script. Once we have the output mp3, we can upload it to somewhere like Google Drive for later use.
Have a listen to the finished product here
The 14 nodes in file order (sticky notes excluded). Branches and connections determine execution order in n8n:
Configure your own credentials after importing. Create each credential in n8n, then select it in the node.
Do you need somewhere to run it? Read our guide to self-hosted n8n.
We can adapt this workflow to your apps, data and process, then deploy and support it.
Contact Us