mcp-video-analyzer
About
Turn any video — YouTube, Instagram, TikTok, Loom, direct URLs, or local files — into transcripts, key frames, OCR text, and metadata for AI agents.
Details
- Author
- guimatheus92
- Categories
- Productivity, Other, Web Scraping
Jump to
Setup
Install mcp-video-analyzer in your MCP client (Claude Desktop, Cursor, Windsurf, and others).
Repository: https://github.com/guimatheus92/mcp-video-analyzer
Follow the installation instructions in the repository README, then restart your MCP client.
Turn any video — YouTube, Instagram, TikTok, Loom, direct URLs, or local files — into transcripts, key frames, OCR text, and metadata for AI agents.
Turn any video — YouTube, Instagram, TikTok, Loom, X, Vimeo, direct links, local files — into transcripts, key frames, OCR text, and metadata for AI agents.
No existing video MCP combinestranscripts + visual frames + metadatain one tool. This one does — across Loom, the major yt-dlp platforms (YouTube/Vimeo/TikTok/Instagram/X/Twitch/Dailymotion/Facebook), direct video URLs, and local files.
- Node.js 22.12+— required to run the server vianpx
- yt-dlp—requiredfor YouTube/Vimeo/TikTok/Instagram/X/Twitch/Dailymotion/Facebook URLs; optional for everything else (improves Loom download quality). Install withpip install yt-dlp
- Chrome/Chromium(optional) — fallback for frame extraction if yt-dlp is unavailable
Without yt-dlp or Chrome, direct URLs and local files still get frames — the bundledffmpeg-staticdoes the extraction, and Loom falls back to its own CDN download. Platform URLs (YouTube etc.) degrade to a clear "install yt-dlp" warning. Transcripts, metadata, and comments never require either.
There are three ways in: the/videoplugin(Claude Code — slash command + MCP server auto-configured), a plainMCP serverconfig (any MCP client), or theportable skill + CLI(Codex, Cursor, Copilot, and any agent with a shell — no MCP required).
Claude Code —/videoplugin (recommended)
/plugin marketplace add guimatheus92/mcp-video-analyzer /plugin install video@mcp-video-analyzer
This adds the/videoslash commandandauto-registers the MCP server — noclaude mcp addneeded:
/video https://youtu.be/jNQXAC9IVRw what happens at 0:10? /video ~/Movies/screen-recording.mp4 when does the UI break?
Other agents — Codex, Cursor, Copilot, Gemini CLI, …
npx skills add guimatheus92/mcp-video-analyzer
Installs thevideoskill (Agent Skillsformat) into every agent detected on your machine. Agents without the MCP server configured fall back to the bundledCLIautomatically — zero configuration.
claude mcp add video-analyzer -- npx mcp-video-analyzer@latest
Then restart Claude Code or start a new conversation.
- VS Code:File → Preferences → Settings → search "MCP"or edit~/.vscode/mcp.json/%APPDATA%\Code\User\mcp.json(Windows)
- Cursor:Settings → MCP Servers → Add
{ "servers": { "mcp-video-analyzer": { "type": "stdio", "command": "npx", "args": ["mcp-video-analyzer@latest"] } } }
Then reload the window (Ctrl+Shift+P→ "Developer: Reload Window").
- macOS:~/Library/Application Support/Claude/claude_desktop_config.json
- Windows:%APPDATA%\Claude\claude_desktop_config.json
{ "mcpServers": { "video-analyzer": { "command": "npx", "args": ["mcp-video-analyzer@latest"] } } }
The same engine is exposed as a one-shot command — this is what thevideoskill uses on agents without MCP, and it works standalone in any terminal:
npx -y mcp-video-analyzer@latest analyze "https://youtu.be/jNQXAC9IVRw"
stdout is a single JSON document —metadata,transcript,ocrResults,timeline,warnings,frameCount, andframesas{ time, filePath, mimeType }entries pointing at JPEG key frames copied to--out(default: the per-user cache dir —%LOCALAPPDATA%on Windows,~/Library/Cacheson macOS,$XDG_CACHE_HOMEor~/.cacheon Linux — undermcp-video-analyzer/<url-hash>/; setMCP_CACHE_DIRto an absolute path to relocate it). Unlike the temp dir this used to live in, nothing reaps that location, so frames persist until you delete them — the directories are created0700. Progress streams on stderr, sostdoutcan be piped straight into a JSON parser. Partial failures land inwarningswith exit code 0; only hard failures exit 1.
Run with no arguments (npx mcp-video-analyzer@latest) to start the MCP stdio server — the CLI is purely additive.
Analyze this video: https://www.youtube.com/watch?v=jNQXAC9IVRw
(also works with an Instagram/TikTok/Loom link, a direct.mp4URL, or a local file path). If the server is connected, it will automatically call theanalyze_videotool.
Eight tools — the AI picks the cheapest one for the job and calls it automatically. Click any tool to expand its parameters and examples.
Extracts everything from a video URL in one call:
> Analyze this video: https://www.youtube.com/watch?v=abc123...
- Transcriptwith timestamps and speakers
- Key framesextracted via scene-change detection (automatically deduplicated). For static clips with no scene cuts — e.g. talking-head Reels/Stories where only an on-screen text overlay changes — it automatically falls back to uniform temporal sampling so you still get frames (and OCR) instead of an empty result.
- OCR textextracted from frames (code, error messages, UI text, prices/dates/CTAs visible on screen)
- Annotated timelinemerging transcript + frames + OCR into a unified "what happened when" view
- Metadata(title, duration, platform)
- Commentsfrom viewers
- ChaptersandAI summary(when available)
The AI willautomaticallycall this tool when it sees a video URL — no need to ask.
- detail— analysis depth:"brief"(metadata + truncated transcript, no frames),"standard"(default),"detailed"(dense sampling, more frames)
- fields— array of specific fields to return, e.g.["metadata", "transcript"]. Available:metadata,transcript,frames,comments,chapters,ocrResults,timeline,aiSummary
- maxFrames(1-60) — cap on extracted frames. Default scales with video duration atstandarddetail (~12 for ≤30s up to 60 for >10min); fixed 60 atdetailed, 0 atbrief. An explicit value always wins
- threshold(0.0-1.0, default 0.1) — scene-change sensitivity
- forceRefresh— bypass cache and re-analyze
- skipFrames— skip frame extraction for transcript-only analysis
- model/language/initialPrompt— per-call Whisper overrides for the transcription fallback (overrideWHISPER_MODEL/WHISPER_LANGUAGE/WHISPER_PROMPTfor this call only — pick a heavier model or a domain glossary for one hard clip without restarting the server)
Runsanalyze_videoover a list ofsourceswith aconcurrencylimit (default 2), returning onestructured result per source— counts + warnings on success, or a per-itemerroron failure (one bad file never aborts the batch). Frame images are not inlined and full transcript/OCR/timeline are returned only whenfieldsis set; otherwise you get counts. Pair withMCP_WRITE_SIDECARS=1(below) so each video's result persists to disk and a re-run resumes instead of recomputing.
Quick transcript extraction. Falls back to Whisper transcription when no native transcript is available. Accepts the same per-callmodel/language/initialPromptoverrides asanalyze_video.
Returns metadata, comments, chapters, and AI summary without downloading the video.
> Extract frames from this video with dense sampling
- Scene-change detection(default) — captures visual transitions
- Dense sampling(dense: true) — 1 frame/sec for full coverage
> Analyze what happens between 1:30 and 2:00 in this video
Combines burst frame extraction + filtered transcript + OCR + annotated timeline for a focused segment. Use when you need to understand exactly what happens at a specific moment.
> Show me the frame at 1:23 in this video
The AI reads the transcript, spots a critical moment, and requests the exact frame to see what's on screen.
> Show me 10 frames between 0:15 and 0:17 of this video
For motion, vibration, animations, or fast scrolling — burst mode captures N frames in a narrow window so the AI can see frame-by-frame changes.
Results are cached in memory for 10 minutes. Subsequent calls with the same URL and options return instantly. UseforceRefresh: trueto bypass the cache.skipFramesis part of the cache and sidecar key, so a transcript-only analysis and a framed one of the same URL never answer for each other.
Persistent sidecars (resumable bulk processing)
The in-memory cache is lost on restart, which makes reprocessing a large local corpus costly. SetMCP_WRITE_SIDECARS=1to also persist resultsnext to each local videoso the work survives restarts and can resume:
- <stem>.vtt— the transcript,onlywhen it was generated by the Whisper fallback (an existing<stem>.vttfrom your own pipeline is never overwritten). A later call reuses it via the normal sidecar reader and skips Whisper entirely.
- <stem>.analysis.json+<stem>.frames/— the full result (frames + OCR + timeline), keyed by the video'smtime:sizeand the analysis params. On a later call with a matching stamp + params, the result is returned straight from disk (no extraction, no OCR).
This makesanalyze_videosover thousands of files resumable, and lets an external GPU transcription pipeline and this MCP share results through the filesystem: the pipeline writes<stem>.vtt, and the MCP picks it up instead of running Whisper.
Loom frames: transcript, metadata, and comments come straight from Loom's API with no extra tooling. Frame extraction is different — Loom serves most videos as separate DASH video+audio streams, which onlyyt-dlp(pip install yt-dlp) fetches and merges. Merging uses the bundledffmpeg-static, so no system ffmpeg is required. Without yt-dlp a direct-CDN fallback still covers some videos; when it can't, you get transcript + metadata + comments plus a warning explaining why frames are missing.
Local files: pass an absolute path (e.g.,/Users/you/clip.mp4) or afile://URI as theurlargument to any tool. Relative paths are rejected — the server's working directory is unpredictable from the MCP client. Note that any caller of the MCP server can ask it to read any file the server process has access to.
Sidecar transcripts: if aclip.vtt,clip.srt,clip.en.vtt, etc. lives next toclip.mp4, it's used as the transcript automatically — no Whisper roundtrip needed. SRT is converted to VTT in-memory.
Embedded subtitles: if no sidecar is found and the container has an embedded subtitle stream (common in.mkv/.mov/.mp4from screen recorders), it's transmuxed to VTT via ffmpeg and used as the transcript.
Recognized extensions(local files and direct URLs):.mp4.mov.mkv.webm.avi.m4v.wmv.flv.mpeg.mpg.m2ts.mts.3gp.ogv. The extension only gates routing — ffmpeg does the actual demuxing, so most common containers work..tsis excluded to avoid colliding with TypeScript source files.
Platform URLs via yt-dlp (YouTube, Instagram, TikTok, …)
Single-video pages on major platforms route throughyt-dlp(pip install yt-dlp— required for these URLs). Playlists, channels, and profile pages are rejected by design; pass individual video URLs (batch them withanalyze_videos).
Set theTWELVELABS_API_KEYenvironment variable to analyze direct video URLs withTwelveLabsPegasus. Pegasus analyzes the video server-side (visualsandits own audio) and returns anAI-generated, timestamped transcriptplus an AI summary as text — capabilities theDirectAdaptercan't provide (a raw.mp4URL has no transcript or summary on its own), and withno Whisper key required.
The transcript is best-effort LLM output, not a deterministic ASR dump: Pegasus ispromptedto emit[MM:SS] linerows, and lines that don't match that shape are dropped, so wording and exact timestamps depend on the model's prompt adherence. Failures (bad key, timeout, API error) surface in the tool'swarnings[]rather than silently returning an empty transcript.
The biggest win is on the text-only paths:get_transcriptandget_metadatareturn a Pegasus transcript and summary for direct URLs — a few KB of text, no frame images, no per-frame token cost.analyze_videoatdetail: "standard"/"detailed"still extracts frames in addition (usedetail: "brief"to stay text-only).
Long videos: the summary and full transcript share a single capped completion (max_tokens= 16384), so for very long videos the transcript may be truncated. For multi-hour content, chunking by time window is the better approach.
It's fully opt-in and non-breaking: whenTWELVELABS_API_KEYis set theTwelveLabsAdapterhandles direct video URLs (it registers the public URL with TwelveLabs — no upload); when it's unset, theDirectAdapterhandles them exactly as before. Loom URLs are unaffected. Get a key atplayground.twelvelabs.io.
When a source has no native transcript (no sidecar.vtt/.srt, no embedded subtitles, no platform captions), the audio track is transcribed with Whisper via a graceful fallback chain (in execution order):
Silent tracks: before any Whisper run, the audio is probed with ffmpegvolumedetect(first 2 minutes). A present-but-mute track — common in muted Reels/Stories — skips transcription entirely and emits a warning that the empty transcript isexpected content, not an error, saving a pointless Whisper run.
- @huggingface/transformers(JS-native, zero external deps) —opt-in only: this strategy runsfirst, butonly whenWHISPER_HF_MODELis explicitly set. When it's unset (the default) the strategy is skipped entirely, so the CLI below wins and itsWHISPER_MODEL/WHISPER_LANGUAGEsettings are never silently overridden.
- whisperCLI— used when awhisperexecutable is found (pip install -U openai-whisper). PointWHISPER_BINat the executable if it isn't onPATH. Model viaWHISPER_MODEL, language viaWHISPER_LANGUAGE. The bundledffmpeg-staticis put on the CLI'sPATHautomatically, so no system ffmpeg is required.
- OpenAI Whisper API— used whenOPENAI_API_KEYis set.
No backend configured?If none of the three is available (nowhisperonPATH/WHISPER_BIN, noOPENAI_API_KEY, noWHISPER_HF_MODEL), transcription tools return an empty transcriptwith a warning telling you how to enable one— rather than a silent "no transcript". Installopenai-whisperor set one of the keys above. (The CLI is spawned withPYTHONUTF8=1so non-English/CJK transcripts don't crash the Python process on Windows.)
The defaulttinymodel is fast but weak for non-English audio. For Portuguese (or other non-English) sources, install the CLI and setWHISPER_MODEL=small(ormedium) +WHISPER_LANGUAGE=ptfor much better accuracy. AddWHISPER_PROMPTwith a domain glossary (brand/place names) to fix proper nouns. You can also overridemodel/language/initialPromptper callonanalyze_video/get_transcript/analyze_videos— no restart needed.
GPU (faster-whisper):whisper-ctranslate2(pip install -U whisper-ctranslate2) is a drop-in CLI with the same flags plus--device cuda/--compute_type/--beam_size. PointWHISPER_BINat it and setWHISPER_DEVICE=cuda(+ optionallyWHISPER_COMPUTE=float16). These GPU flags areenv-gated— they're only passed when set, so plainopenai-whisper(which rejects--compute_type) keeps working when they're unset.
Windows note:pip installswhisper.exeinto the PythonScripts/dir, which is oftennoton thePATHthat GUI-launched MCP clients inherit. If transcripts come back empty, setWHISPER_BINto the full path ofwhisper.exe.
Frame extraction uses a two-strategy fallback chain — no single dependency is required:
The fallback is automatic — if yt-dlp is not available, the server tries browser-based extraction viapuppeteer-core. If neither is available, analysis still returns transcript + metadata + comments, just no frames.
After frame extraction, the pipeline automatically applies:
The OCR step requirestesseract.js(included as a dependency). If it fails to load, analysis continues without OCR — no frames or transcript are lost. OCR preprocessing is on by default; setMCP_OCR_PREPROCESS=0to OCR the raw frames instead.
OCR always reads thefull-resolutionframe, not the copy emitted to the client. The two have different jobs: the emitted frame is capped for token cost, while recognition needs every pixel it can get.
Emitted frames are capped at 800 px wide, which suits the common case — talking-head clips, Reels, bug repros — where the subject fills the frame.
It is the wrong size for adense UI capture: a terminal, dashboard, IDE or spreadsheet recording, where the meaning lives in small text. An unscaled 1920×1080 screen recording lands at 800×450, and a 15 px UI font drops below what a vision model can resolve.
PassmaxWidthper call to keep more (or all) of the source resolution —0disables the cap:
get_frames(url, { maxFrames: 8, maxWidth: 0 }) // source resolution get_frame_at(url, "2:14", { maxWidth: 1920 }) analyze_video(url, { detail: "standard", maxWidth: 1568 })
Supported onanalyze_video,analyze_videos,analyze_moment,get_frames,get_frame_atandget_frame_burst, and on the CLI as--max-width <px>.
Native frames cost several times more context than the default, so raise the cap deliberately —get_framesreturns up to 20 frames andanalyze_videoatdetailedup to 60.
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
