Ingero
Description
eBPF-based GPU causal observability agent with MCP server. Traces CUDA Runtime/Driver APIs via uprobes and host kernel events via tracepoints to build causal chains explaining GPU latency. 7 MCP tools for AI-assisted GPU debugging and root cause analysis. x3C2% overhead…
About
eBPF-based GPU causal observability agent with MCP server. Traces CUDA Runtime/Driver APIs via uprobes and host kernel events via tracepoints to build causal chains explaining GPU latency. 7 MCP tools for AI-assisted GPU debugging and root cause analysis. x3C2% overhead, production-safe.
Details
- Author
- ingero-io
- Downloads
- 326
- Categories
- Other, Infrastructure, AI, Developer Tools
Jump to
- eBPF-based causal tracing across 4 layers (CUDA API, host kernel, system context)
- Production-safe with <2% overhead and zero code changes required
- Python source line attribution (3.10/3.11/3.12) via DWARF
- SQLite storage with 10 GB rolling cap
- Kubernetes support (DaemonSet, Helm chart, pod metadata)
- Single binary, no runtime dependencies
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
IngeroCommand (node, npx, python, etc.)Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
Install the single Ingero binary, then start the MCP server with ingero mcp --db ~/.ingero/ingero.db for stdio transport (Claude Code, Cursor) or ingero mcp --db ~/.ingero/ingero.db --http :8090 for HTTPS. No code changes are required. Use the provided MCP tools to diagnose performance.
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"ingero": {
"ingero": {
"command": "ingero",
"args": [
"mcp"
],
"env": []
}
}
}
}McpServers
{
"ingero": {
"command": "ingero",
"args": [
"mcp"
],
"env": []
}
}Ingero is an eBPF-based agent that provides production-safe, kernel-level causal tracing for GPU workloads. It answers: "Why is my GPU training/inference slow right now?"
## MCP Tools (7)
- get_check — System diagnostics (kernel, BTF, NVIDIA, CUDA, GPU processes)
- get_trace_stats — Per-operation p50/p95/p99 latency stats
- get_causal_chains — Root cause analysis with severity ranking and fix recommendations
- get_stacks — Resolved call stacks (Python source file, function, line number)
- run_sql — Read-only SQL queries against the event database
- run_demo — Run synthetic demo scenarios (no GPU or root needed)
- get_test_report — GPU integration test results
## How It Works
Ingero traces 4 layers via eBPF:
1. CUDA Runtime API (libcudart.so) — cudaMalloc, cudaFree, cudaLaunchKernel, cudaStreamSync
2. CUDA Driver API (libcuda.so) — cuLaunchKernel, cuMemcpy, cuCtxSynchronize, cuMemAlloc
3. Host kernel tracepoints — sched_switch, mm_page_alloc, oom_kill, process lifecycle
4. System context from /proc — CPU, memory, load, swap
These produce causal chains: System context + host event → CUDA call → root cause.
## Usage
```bash
# Start MCP server (stdio — for Claude Code, Cursor, etc.)
ingero mcp --db ~/.ingero/ingero.db
# Start MCP server (HTTPS — for remote clients)
ingero mcp --db ~/.ingero/ingero.db --http :8090
Key Features
- <2% overhead, zero code changes, single binary
- Python 3.10/3.11/3.12 source line attribution via DWARF
- SQLite storage with 10 GB rolling cap
- Kubernetes support (DaemonSet, Helm chart, pod metadata)
- Tested on A10, A100, H100, GH200, RTX 3090, RTX 4090
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
