Aantraa.site: AI audio video translation, caption generator and shorts cutter tool
Under the Hood
I run a small YouTube channel. I am not a content creator, but YouTube is a solid platform to gain traffic for your online work/business/project/idea.
Aantraa is what I made in a week, and the main concept is simple
- Video translation tool into multiple languages
- Audio translation, including text to audio, and get an MP3 file for Premiere Pro
- Convert a YouTube long-form video to shorts
At that time, only 3 features were needed, so not much work in the website development was required.
Heavy work was building APIs, backend infrastructure to integrate AI into video and dealing with heavy storage.
Breaking the execution into steps: How I made Aantraa?
AI LLM Layering and Provider
Aantraa is heavily dependent on AI APIs, and we need reliable and solid infrastructure for AI LLM APIs or providers.
Openrouter, Portkey, Vercel AI SDK labs and individual APIs for Anthropic, Deepseek, and OpenAI are the absolute available options.
I prefer OpenRouter for Aantraa for one reason: multiple models support (To acknowledge the cheapest available models for each job on Aantraa). Easy to integrate, community support, FREE models accessibility and more.
AI LLM APIs are needed on almost every stage in the backend, as explained
- Understanding video context and creating a script
- Translating the script into languages
- Recording the script into MP3 or WAV format
- Summarising the video
- Generating captions
- Cutting videos into shorts
Building APIs and Servers
Each layer needs heavy AI LLM context and prompt engineering. Loop engineering is the new trend, which is required over here.
For example, for video translation, the AI API works in multiple connected steps.
Video translation API breakdown
- AI first understands the video, which is fed into the AI LLM via the ffmpeg module.
- Step is generating a script/caption from the video
- AI creates the translation into the desired language (Script writing into a new language)
- AI then generates an audio MP3 or WAV format of the new translation
- AI then glued the audio and video using ffmpeg in the end
Each step is dependent on the previous step, making it difficult to track sometimes if it breaks in production.
Solution: Track each process usage tokens, estimated time, errors and response.nSimilarly, we need the same steps for audio translation, viral clip cutter, and caption generator.
Infrastructure and Servers
Local API development is not a heavy task until we release it in production.
Framework: Honojs
Backend hosting: Vercel Edge or Fly servers
Devops: Docker, simple Git CI/CD
Database: Firebase/Supabase
Storage: Uploadthing
Interestingly, I've found new server side and client SDK alternative to AWS and firebase or supabase storage, uploadthingUploadthing provides SDK for both client and server side to quickly upload files of 5MB in size at a time in a FREE plan.
We need storage heavily because every layer of AI doesn’t have its own storage or memory, so every audio/video generated needs to be stored in storage.
One more thing, the heavy packages to work with video and audio are FFMPEG, which has limitations with serverless functions and Vercel Edge, which brings us to switch to fly.io, railway, or render as the alternative.
Video translation into 90+ Languages
Aantraa supports 90+ languages for the translation of video and audio.
AI can translate scripts, text or text on an image or a video very well.
AI only need to understand the video through the script and each frame, FFMPEG helps with the same.
AI then generates a script, translate into the desired language and creates audio in that language to finally glue the audio and video together to generate a new translated video.
Each step needs debugging, AI prompt engineering and FFMPEG integration with AI.
Finally, the video upload on the storage to return the URL to the client side for Download and more capabilities.
Audio to MP3 into languages
Aantraa should not only provide video translation, but we also need more capabilities. One requirement I have as a creator is to convert blog or text into audio for readers to listen to as a podcast.
We need simple text-to-speech that provides an MP3 or WAV format to download.
Aantraa audio translation tool did a few more things in total
- Text to audio format
- Text audio supporting 90+ languages
- Video to audio supporting 90+ languages
- Support for multiple languages at the same time
This makes aantraa not just a video but a supporting platform, convert one recording into 90+ languages, download the MP3 for one-click sharing 😸.
YouTube videos to Shorts
This is the main feature: convert long-form YouTube videos to shorts, viral shorts to directly upload to your YouTube channel.
API breakdown in steps
- AI understand the entire video context, creates a summary and script
- AI breaks the scripts as per the timestamp into the desired number of Shorts
- AI using FFMPEG cuts the clip and API upload to storage
The entire process looks easy, but a few tweaks are needed: FFMPEG as the module, AI context understanding, file size limit, more than 10MB size video will cost time and money.
Video Translated AI Examples
Reach out to aantraa.site/examples, to check 10+ translated videos into multiple languages, including Spanish, Hindi, Bengali, Gujarati, Marathi, Tamil, French, English, Japanese and Chinese.
90+ languages make aantraa a universal global platform 😄
Conclusion
First version is LIVE. We also provide APIs to Business agencies and startup teams.
aantraa.site examples screenshot
Website: aantraa.site
Blog: https://aantraa.site/blog
Examples: https://aantraa.site/examples
Pricing: https://aantraa.site/#pricing
Do check the website and let me know in the comments section what you think.
First early signups will get 1 minute FREE AI credit!!
Cheers
Shrey