Vidlizer

by arizawan

152 downloads

Not rated

GitHub

About

Extract structured JSON from video, images, and PDFs using local LLMs (Ollama, LM Studio, oMLX) or via OpenRouter. Runs fully offline.

Details

Author: arizawan
Downloads: 152
Categories: Developer Tools, Other

- Supports local videos, images, PDFs, and URLs (YouTube, Loom, Vimeo, Twitter)
- 4 providers: Ollama (fully offline), LM Studio, oMLX, OpenRouter — auto-detected
- Cross-provider fallback if primary model fails
- Automatic JSON repair for malformed model output
- 3 output formats: JSON (default), Markdown, plain-text summary
- Audio transcription via Apple MLX Whisper merged into each step
- MCP server for use with Claude Code, Cursor, Claude Desktop
- In-memory cache, cost guard (MAX_COST_USD), and live progress indicators

Setting up with Highlight

This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:

Download and install Highlight from highlightai.com/download
Navigate to the plugins tab and select "Add Custom Plugin"
Configure the plugin with the settings below

Plugin Name Vidlizer

Command (node, npx, python, etc.)

Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
Enable "Start Automatically" if you want the plugin to start when Highlight launches

Setting up with Cursor

Open Cursor's settings
Navigate to the MCP section
Click "Add new Global MCP server"
Copy and paste the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

Setting up with Claude Desktop

Open Claude Desktop's settings
Go to the Integrations tab
Click "Add MCP Server"
Use the configuration below based on your operating system

Note: Make sure Claude Desktop is running before attempting to connect.

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

Setting up with Windsurf

Open Windsurf's preferences
Go to the Integrations section
Click "Add MCP Integration"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

Setting up with Cline

Open Cline's settings
Navigate to the Integrations section
Click "Add MCP Server"
Copy and paste the configuration below based on your operating system

Note: Make sure to restart Cline after adding the configuration.

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

Setting up with Goose

Open Goose's preferences
Go to the Integrations tab
Select "MCP Servers"
Click "Add New Server"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

From the repository

Install with uvx vidlizer, pipx install vidlizer, or pip install vidlizer, then run vidlizer setup to auto-detect providers and write your .env file. After setup, run vidlizer <file> (e.g., vidlizer demo.mp4, vidlizer "https://youtube.com/watch?v=...") to analyze the input. Use --provider and --model flags to choose a specific provider or model, or run with no arguments for an interactive file picker.

Claude Desktop / Cursor

Paste into your MCP client config file to install this server.

{
    "mcpServers": {
        "vidlizer": {
            "vidlizer": {
                "type": "stdio",
                "command": "uvx",
                "args": [
                    "--from",
                    "vidlizer[mcp]",
                    "vidlizer-mcp"
                ],
                "env": {
                    "PROVIDER": "ollama",
                    "OLLAMA_HOST": "http://localhost:11434",
                    "OLLAMA_MODEL": "gemma4:2b"
                }
            }
        }
    }
}

McpServers

{
    "vidlizer": {
        "type": "stdio",
        "command": "uvx",
        "args": [
            "--from",
            "vidlizer[mcp]",
            "vidlizer-mcp"
        ],
        "env": {
            "PROVIDER": "ollama",
            "OLLAMA_HOST": "http://localhost:11434",
            "OLLAMA_MODEL": "gemma4:2b"
        }
    }
}

vidlizer

Point it at a video, image, or PDF. Get structured JSON — scene by scene.

demo

</div>

---

vidlizer pulls frames out of any video, image, or PDF using ffmpeg, sends them to a vision LLM, and returns a flow array — one entry per scene. Each entry tells you what happened, who was on screen, what text was visible, and what changed. If the video has audio, it transcribes it with Apple MLX Whisper and merges the speech into each step.

Runs fully local via Ollama or any OpenAI-compatible server (LM Studio, vLLM, oMLX) — no API key, no data leaving your machine. Or connect OpenRouter for cloud models. vidlizer setup detects what you have installed and writes your config in under a minute.

vidlizer demo.mp4
vidlizer "https://youtube.com/watch?v=..."
vidlizer screenshot.png
vidlizer document.pdf

---

✨ Features

- Any input — local video, image (jpg/png/webp/…), PDF, or URL (YouTube, Loom, Vimeo, Twitter)
- 4 providers — Ollama (fully offline), LM Studio (port 1234), oMLX (Apple Silicon, port 8000), OpenRouter (cloud) — auto-detected in that order
- Cross-provider fallback — primary model fails → automatically switches provider (e.g. oMLX → OpenRouter)
- JSON repair — malformed model output is re-sent to the model to fix before skipping; recovers from partial JSON
- Free-model guard — :free OpenRouter models auto-force concurrency=1 to stay within rate limits
- 3 output formats — --format json (default), summary (plain text by phase), markdown (step-per-section doc)
- Usage tracking — --stats shows per-model token + cost breakdown across all runs; get_usage_stats() MCP tool
- Auto transcript — detects audio, transcribes with Apple MLX Whisper (Neural Engine), merges speech into each flow step
- Perceptual dedup — removes near-duplicate frames before sending (saves tokens)
- analyze_moment — --start/--end flags to focus on a time range
- In-memory cache — repeat runs on the same file skip the API call
- Cost guard — aborts if spend exceeds MAX_COST_USD (default $1.00)
- Live progress — Rich streaming indicator shows elapsed time and token count per batch
- MCP server — use from Claude Code, Cursor, Claude Desktop; provider/model locked via env vars; result includes model_used + provider_used
- Auto-install — missing ffmpeg is brew-installed; mlx-whisper bundled in default install (macOS)
- doctor --fix — interactive repair wizard: installs missing ffmpeg/Ollama/LM Studio via Homebrew, re-runs vidlizer setup for .env, upgrades mlx-whisper
- mcp-setup — one-command MCP config wizard: detects vidlizer-mcp, reads .env, writes editor config or shows a claude mcp add-json one-liner
- Mac-native — file picker dialog, Apple MLX transcription, handles macOS Unicode filenames (e.g. "11:26 AM")

---

📦 Requirements

- macOS (Apple Silicon recommended for transcription speed)
- Python 3.10+
- Ollama mode: Ollama installed + a vision model pulled (5 GB+ RAM)
- LM Studio mode: LM Studio 0.3.16+ with a vision model loaded
- Cloud mode: An OpenRouter API key

ffmpeg is installed automatically via Homebrew on first run if missing.

---

🚀 Install

Option 1 — uvx (no install, run directly)

uvx vidlizer setup

Option 2 — pipx (isolated, globally available)

pipx install vidlizer
vidlizer setup    # interactive wizard: detects providers, writes .env

Option 3 — pip / virtualenv

pip install vidlizer
vidlizer setup

Option 4 — from source

git clone https://github.com/arizawan/vidlizer.git
cd vidlizer
python -m venv .venv && source .venv/bin/activate
pip install -e .
vidlizer setup    # or: cp env.sample .env

First-run wizard

vidlizer setup detects all installed providers, lets you pick primary + fallback, and writes a .env for you. It also offers to pull a vision model for Ollama if none is installed.

$ vidlizer setup Detected providers: 1. Ollama → qwen2.5vl:3b 2. OpenRouter → google/gemma-3-27b-it:free Primary provider (1–2): 1 Fallback (1–1, Enter to skip): 2

✓ .env written → /your/project/.env

Health check

vidlizer doctor          # shows ffmpeg, .env, provider, mlx-whisper status
vidlizer doctor --fix    # interactive repair: brew-installs ffmpeg/Ollama/LM Studio, re-runs setup

Manual provider setup

Ollama (fully offline, no API key):

```bash

No reviews yet — be the first

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.

About

Details

Jump to

Setting up with Highlight

From the repository

Claude Desktop / Cursor

McpServers

vidlizer

✨ Features

📦 Requirements

🚀 Install

Option 1 — uvx (no install, run directly)

Option 2 — pipx (isolated, globally available)

Option 3 — pip / virtualenv

Option 4 — from source

First-run wizard

Health check

Manual provider setup

Install

More like this

Sequential Thinking

Sentry

Github

Filesystem

EverArt

Google Drive