AI Video Clipper & 9:16 Shorts Generator
Autonomous 9:16 Shorts Generator & FastMCP Video Engine
An autonomous video clipping engine that turns long-form landscape videos (YouTube, Vimeo, podcasts, interviews) into high-retention 9:16 vertical shorts with intelligent OpenCV speaker tracking and animated karaoke subtitles.
About AI Video Clipper & 9:16 Shorts Generator
AI Video Clipper & 9:16 Shorts Generator is an open-source, production-grade video automation framework engineered to transform wide landscape content (such as YouTube videos, podcasts, keynotes, and interviews) into viral, high-retention 9:16 vertical shorts optimized for TikTok, YouTube Shorts, and Instagram Reels.
Traditional short-form editing requires hours of tedious manual scrubbing, transcription, face re-framing, keyframing, and subtitle synchronization. AI Video Clipper automates this entire pipeline autonomously using hexagonal architecture, speech recognition, computer vision, and modern large language models.
The engine operates in dual modes: as an intuitive standalone Command-Line Interface (CLI) directly from your terminal, or as a standardized Model Context Protocol (MCP) Server that plugs directly into AI agents and IDEs like Claude Desktop, Cursor, and Antigravity. It can run with state-of-the-art cloud AI models (Google Gemini 3.8 Flash, OpenAI GPT-4o) or completely offline without any API keys or network costs.
Key Highlights & Capabilities
Model Context Protocol (MCP) Ready
Exposes native FastMCP tools (analyze_video_highlights, render_highlight_clip, auto_clip_video) enabling AI assistants to orchestrate end-to-end video production directly in chat.
Intelligent Face & Speaker Tracking
Uses OpenCV Haar Cascades and Exponential Moving Average (EMA) smoothing to frame speakers naturally, eliminate camera jitter, and snap seamlessly across scene transitions.
Dual-Speaker Split-Screen Layout
Automatically detects conversational video scenes with two active participants and composes a dynamic stacked vertical split-screen view in real time.
Libass Animated Karaoke Subtitles
Generates dynamic word-by-word highlighted ASS subtitles with TikTok and Instagram safe-zone protection to prevent UI overlap.
Zero-AI / 100% Offline Mode
Full standalone execution without API keys or token costs using local Whisper speech cadence, silence boundaries, and heuristic scoring.
Instant Range HTTP Downloads
Leverages yt-dlp section range requests to fetch only the required 30 to 60 second video slices, eliminating multi-gigabyte source file downloads.
Features & Technical Capabilities
Autonomous Virality Scoring & Highlight Detection
Evaluates transcribed speech against established virality rubrics, ensuring only high-impact moments are extracted.
- •Hook verification within the first 3 seconds to maximize retention.
- •Emotional intensity and contrarian argument detection.
- •Punchline and insight payoff identification with strict sentence boundary snapping.
- •Zero-AI fallback mode powered by punctuation and speech density analysis.
Computer Vision 9:16 Smart Dynamic Framing
Translates 16:9 widescreen footage into polished 9:16 portrait frames without distortion or manual keyframing.
- •Continuous facial bounding box tracking to center active speakers.
- •Exponential Moving Average (EMA) filter smoothing out minor micro-movements.
- •Scene-cut detection that instantly resets viewport coordinates without jarring slides.
- •Automated dual-speaker vertical split-screen stacking when both participants speak.
Burned-in Animated Karaoke Subtitles
High-impact, word-level highlighted typography rendered directly onto the video stream using libass.
- •Four built-in color presets: karaoke_yellow, neon_cyan, clean_white, and bold_magenta.
- •Millisecond-accurate word-level timing generated from faster-whisper transcription.
- •Enforces 250px vertical bottom safe margin to keep text clear of TikTok, Reels, and Shorts interface icons.
FastMCP Model Context Protocol Integration
Native integration for agentic AI workflows, allowing assistants to analyze, select, and render clips interactively.
- •analyze_video_highlights(url, max_clips): Inspects audio, returns viral scores, timestamps, and reasoning JSON.
- •render_highlight_clip(url, start_time, end_time, subtitle_preset): Renders exact time slice to MP4.
- •auto_clip_video(url, clip_count): Macro tool performing complete ingestion, analysis, cropping, and rendering.
Lightweight Ingestion via HTTP Range Requests
Bypasses heavy bandwidth overheads by streaming only what is strictly necessary.
- •Downloads audio stream first to evaluate virality scores quickly.
- •Extracts pinpoint 30–60s video slices using HTTP byte-range queries.
- •Avoids multi-gigabyte downloads of full 2-hour podcasts or webinars.
- •Automated ephemeral directory cleanup ensuring zero disk bloat.
Hexagonal Architecture & Comprehensive Test Suite
Built according to strict software engineering standards with zero hardcoded business logic.
- •Fully decoupled adapters for audio downloaders, transcribers, vision trackers, and renderers.
- •Typed configuration management using Pydantic settings and .env overrides.
- •Comprehensive automated test suite covering unit, property, and integration validation.
How It Works
Ingestion & Audio Extraction
Input a YouTube URL, Vimeo link, or local media file. The tool extracts audio rapidly via yt-dlp to evaluate the full conversation without downloading heavy video files.
Word-Level Transcription & Virality Scoring
faster-whisper extracts verbatim transcripts with millisecond word timestamps. Gemini, OpenAI, or local heuristics analyze narrative arcs, emotional hooks, and contrarian punchlines.
Face Detection & 9:16 Viewport Calculation
OpenCV processes video frames to locate speakers. The viewport dynamically follows facial positions using EMA smoothing, switching to split-screen if multiple speakers interact.
Subtitle Generation & Hardware Encoding
Word-by-word karaoke styles (.ass) are compiled with platform safe-zones. FFmpeg composites audio, cropped video, and animated subtitles into high-definition 1080x1920 MP4 files.
Quickstart & Configuration Reference
# Run end-to-end auto clipping for top 3 viral moments
python main.py "https://www.youtube.com/watch?v=Gy_kI0S3UgA" -c 3 -p neon_cyan
# Run 100% offline with zero API keys or external costs
python main.py "https://www.youtube.com/watch?v=Gy_kI0S3UgA" --no-ai
# Analyze highlight timestamps only without rendering video
python main.py "https://www.youtube.com/watch?v=Gy_kI0S3UgA" --analyze-only{
"mcpServers": {
"ai-video-clipper": {
"command": "python",
"args": ["/path/to/ai-video-clipper/main.py", "--mcp"]
}
}
}Technical Specifications & Compatibility
- Primary Language
- Python 3.10+
- Core Protocols
- Model Context Protocol (FastMCP stdio)
- Computer Vision
- OpenCV (Haar Cascade Classifiers & EMA Filtering)
- Speech Transcription
- faster-whisper (CTranslate2, CPU / CUDA)
- Video Compositing
- FFmpeg with --enable-libass support
- Supported LLM Providers
- Google Gemini 3.8 Flash, OpenAI GPT-4o, Ollama, Offline Heuristic
- Output Resolution
- 1080 x 1920 (Vertical 9:16 @ 30 FPS)
- Subtitle Presets
- karaoke_yellow, neon_cyan, clean_white, bold_magenta
Who Is AI Video Clipper & 9:16 Shorts Generator Built For?
Content Creators & YouTubers repurposing long podcasts into viral TikToks and Shorts.
Media Companies & Agencies needing high-volume automated vertical video pipelines.
AI Engineers & Developers integrating autonomous video generation into MCP agents and LLM tool chains.
Educators & Keynote Speakers extracting memorable soundbites for social distribution.
Frequently Asked Questions
Does AI Video Clipper require paid OpenAI or Gemini API keys?
No! The engine supports a 100% offline heuristic mode (--no-ai). When running offline, it uses local faster-whisper transcription and sentence cadence density to detect key moments at zero cost. If you do provide a Gemini or OpenAI key, it performs enhanced virality rubric scoring.
How does the Model Context Protocol (MCP) server work?
By running with the --mcp flag, the tool launches a FastMCP server over standard I/O. AI assistants like Claude Desktop, Cursor, and Antigravity can directly call analyze_video_highlights, render_highlight_clip, and auto_clip_video, enabling conversational AI to produce videos autonomously.
Why is libass required for FFmpeg?
Libass is the industry-standard subtitle renderer for Advanced SubStation Alpha (.ass) files. It allows pixel-perfect karaoke animations, custom font weights, colored highlights per spoken word, and precise margin offsets that match TikTok and Instagram UI overlays.
Can it handle videos with multiple speakers?
Yes. The computer vision engine detects multiple active faces in a frame and can compose a dynamic dual-speaker vertical split-screen stack, ensuring both participants remain clearly framed without awkward cuts.
What platforms can I upload the generated videos to?
Every generated video is encoded as an H.264 / AAC 1080x1920 MP4 file at 30 FPS with enforced platform safe-zones. They are ready to publish directly to TikTok, YouTube Shorts, Instagram Reels, and LinkedIn Video.
Experience AI Video Clipper & 9:16 Shorts Generator
Explore the official deployment, clone the repository, or inspect the comprehensive documentation to see how it can enhance your workflow.