---

Your AI, Your Choice

- Supports many AI providers and MCP
- OpenAI or compatible API ¹ for best performance
- OpenRouter for access to 400+ models from all providers
- Ollama and LM Studio for best privacy (local LLM)
- Integrates with your workflow via a built-in MCP server.

<small>¹ Includes xAI / Grok and Claude OpenAI compatible APIs.
Simply adjust the URL settings and overwrite OPENAI_API_KEY.</small>

- Fully resumable parallel processing
- Processes multiple files at once using optimized multi-threading.
- Uses AI cache and generous request retry logic for all network requests.
- Leverages AI structured output with high-quality prompts and schemas.

graph LR
%% Ingestion Pipeline

  subgraph Ingestion

    A[<b>Track and Commit Files</b><br>Supports Profiles] --> B[<b>Ingest Files</b><br>Automatic OCR Strategy<br>Image Entity Extraction]

    B --> C[<b>Semantic Chunking</b><br>with Context Headers<br>Lines per Block: 100<br>Words per Chunk: 200]

    C --> D[<b>Embed Chunks</b><br>Model: text-embedding-3-large<br>Vector Size: 3072]

    D --> E[<b>Store Chunks</b><br>Local Qdrant database]

  end
%% Query Pipeline

  subgraph Query

    F[<b>Ask Question</b>] --> G[<b>Embed Question</b><br>Model: text-embedding-3-large]

    G --> H[<b>Retrieve Nearest Chunks</b><br>Score Min: 0.1<br>Chunks Max: 30]

    E --> H

    H --> I[<b>Rerank by Relevance</b><br>Chunks Max: 8]

    I --> J[<b>Expand Context</b><br>Chunks Radius: 1]

    J --> K[<b>Generate Answer</b>]

    K --> L[<b>Get Answer</b><br>in CLI, GUI, MCP]

  end

Archive Agent has been tested with these configurations:

- Ubuntu 24.04 (PC x64)
- Ubuntu 22.04 (PC x64)

If you've successfully installed and tested Archive Agent with a different setup, please let me know and I'll add it here!

To install Archive Agent in the current directory of your choice, run this once:

git clone https://github.com/shredEngineer/Archive-Agent
cd Archive-Agent
chmod +x install.sh
./install.sh

Archive Agent is now installed!

👉 Please complete the AI provider setup next.
(Afterward, you'll be ready to Run Archive Agent!)

Archive Agent lets you choose between different AI providers:

💡 Good to know: You will be prompted to choose an AI provider at startup; see: Run Archive Agent.

If the OpenAI provider is selected, Archive Agent requires the OpenAI API key.

echo "export OPENAI_API_KEY='sk-...'" >> ~/.bashrc && source ~/.bashrc

💡 Good to know: OpenAI won't use your data for training.

If the OpenRouter provider is selected, Archive Agent requires an OpenRouter API key.

echo "export OPENROUTER_API_KEY='sk-or-...'" >> ~/.bashrc && source ~/.bashrc

💡 Good to know: You can customize the models in the Archive Agent settings. OpenRouter supports structured outputs, embeddings, and vision across many models. Browse all available models at openrouter.ai/models.

If the Ollama provider is selected, Archive Agent requires Ollama running at http://localhost:11434.

ollama pull llama3.1:8b             # for chunk/rerank/query
ollama pull llava:7b-v1.6           # for vision
ollama pull nomic-embed-text:v1.5   # for embed

💡 Good to know: Ollama also works without a GPU.
At least 32 GiB RAM is recommended for smooth performance.

If the LM Studio provider is selected, Archive Agent requires LM Studio running at http://localhost:1234.

meta-llama-3.1-8b-instruct              # for chunk/rerank/query
llava-v1.5-7b                           # for vision
text-embedding-nomic-embed-text-v1.5    # for embed

💡 Good to know: LM Studio also works without a GPU.
At least 32 GiB RAM is recommended for smooth performance.

Archive Agent currently supports these file types:
- Text:
- Plaintext: .txt, .md, .markdown
- Documents:
- ASCII documents: .html, .htm (images not supported)
- Binary documents: .odt, .docx (including images)
- PDF documents: .pdf (including images; see OCR strategies)
- Images: .jpg, .jpeg, .png, .gif, .webp, .bmp

📌 Note: Images in HTML documents are currently not supported.

📌 Note: Legacy .doc files are currently not supported.

📌 Note: Unsupported files are tracked but not processed.

Archive Agent processes files with optimized performance:
- Surgical Synchronization:
- PDF analyzing phase is serialized (due to PyMuPDF threading limitations).
- All other phases (vision, chunking, embedding) run in parallel for maximum performance.
- Vision operations are parallelized across images and pages within and across files.
- Embedding operations are parallelized across text chunks and files.
- Smart chunking uses sequential processing due to carry mechanism dependencies.

For PDF documents, there are different OCR strategies supported by Archive Agent:

- auto OCR strategy:
- Attempts to select the best OCR strategy for each page, based on the number of characters extracted from the PDF OCR text layer, if any.
- Decides based on ocr_auto_threshold, the minimum number of characters for auto OCR strategy to resolve to relaxed instead of strict.
- Trade-off between cost, speed, and accuracy.

⚠️ Warning: The auto OCR strategy is still experimental.
PDF documents often contain small/scattered images related to page style/layout which cause overhead while contributing little information or even cluttering the result.

💡 Good to know: You will be prompted to choose an OCR strategy at startup (see Run Archive Agent).

💡 Good to know: This smart chunking improves the accuracy and effectiveness of the retrieval.

📌 Note: In rare cases where a chunk exceeds the embedding model's token limit (typically 8192 tokens), Archive Agent automatically truncates it as a last resort with progressive 10% reductions (up to 10 attempts) until it fits.

📌 Note: Splitting into sentences may take some time for huge documents.
There is currently no possibility to show the progress of this step.

To ensure that every chunk can be traced back to its origin, Archive Agent maps the text contents of each chunk to the corresponding line numbers or page numbers of the source file.

Archive Agent retrieves chunks related to your question like this:
- The question is turned into a vector using AI embeddings.
- Points with similar vectors are retrieved from the Qdrant database.
- Only chunks of points with sufficient score are kept.
- If retrieve_knee_enable is enabled, an adaptive cutoff trims low-relevance chunks when there is a clear score drop-off.

📌 Note: Adaptive cutoff uses the Kneedle algorithm on the sorted similarity scores. Set a higher
retrieve_knee_sensitivity to make the cutoff more conservative, and use retrieve_knee_min_chunks
to enforce a minimum floor.

💡 Tuning tips:
- If retrieval feels too short or misses context, increase retrieve_knee_min_chunks (e.g., 3–5) or raise retrieve_knee_sensitivity.
- If retrieval feels too long or noisy, lower retrieve_knee_sensitivity slightly (e.g., 0.8–1.0) or reduce retrieve_knee_min_chunks.
- If you want to disable the adaptive cutoff entirely, set retrieve_knee_enable to false.

Archive Agent filters the retrieved chunks .

Archive Agent answers your question using the reranked and expanded chunks like this:
- The LLM receives the chunks as context to the question.
- LLM's answer is returned as structured output and formatted.

💡 Good to know: Archive Agent uses an answer template that aims to be universally helpful.

📌 Note: Always use quotes for the pattern argument (to prevent your shell's wildcard expansion),
or skip it to get an interactive prompt.

Add excluded patterns

To add one or more excluded patterns, run this:

📌 Note: Always use quotes for the pattern argument (to prevent your shell's wildcard expansion),
or skip it to get an interactive prompt.

List included / excluded patterns

To see the list of included / excluded patterns, run this:

archive-agent patterns

Resolve patterns and track files

To resolve all patterns and track changes to your files, run this:

archive-agent track

List tracked files

To see the list of tracked files, run this:

archive-agent list

📌 Note: Don't forget to track your files first.

List changed files

To see the list of changed files, run this:

archive-agent diff

📌 Note: Don't forget to track your files first.

Commit changed files to database

To sync changes to your files with the Qdrant database, run this:

archive-agent commit

To see additional information (vision, chunking, embedding), pass the --verbose option.

To bypass the AI cache (vision, chunking, embedding) for this commit, pass the --nocache option.

To automatically confirm deleting untracked files from the database, pass the --confirm-delete option.

💡 Good to know: Changes are triggered by:
- File added
- File removed
- File changed:
- Different file size
- Different modification date

The Qdrant database is updated after all files have been ingested.

📌 Note: Don't forget to track your files first.

Combined track and commit

To track and then commit in one go, run this:

archive-agent update

To see additional information (vision, chunking, embedding), pass the --verbose option.

To bypass the AI cache (vision, chunking, embedding) for this commit, pass the --nocache option.

To automatically confirm deleting untracked files from the database, pass the --confirm-delete option.

Search your files

archive-agent search "Which files mention donuts?"

Lists files relevant to the question.

📌 Note: Always use quotes for the question argument, or skip it to get an interactive prompt.

To see additional information (embedding, retrieval, reranking), pass the --verbose option.

To bypass the AI cache (embedding, reranking) for this search, pass the --nocache option.

Query your files

archive-agent query "Which files mention donuts?"

Answers your question using RAG.

📌 Note: Always use quotes for the question argument, or skip it to get an interactive prompt.

To see additional information (embedding, retrieval, reranking, querying), pass the --verbose option.

To bypass the AI cache (embedding, reranking) for this query, pass the --nocache option.

To save the query results to a JSON file, run either:

- --to-json with a specific filename:

  archive-agent query "Which files mention donuts?" --to-json answer.json

  

- --to-json-auto [DIR] to auto-generate a clean filename from the question
(max 160 chars, truncated with [...] if needed)
and write to directory DIR if provided (defaults to current directory .; creates directories in path if not existing):

  archive-agent query "Which files mention donuts?" --to-json-auto Output/

  # Creates: Output/Which_files_mention_donuts_.json

  

📌 Note: As of Archive Agent v12.2.0, a corresponding Markdown file (.md) containing the answer is also created when using the --to-json or --to-json-auto options. (There is currently no way to opt out of this.)

Launch Archive Agent GUI

To launch the Archive Agent GUI in your browser, run this:

archive-agent gui

To see additional information (embedding, retrieval, reranking, querying), pass the --verbose option.

To bypass the AI cache (embedding, reranking) for this query, pass the --nocache option.

To save the query results to JSON files, run this:

- --to-json-auto [DIR] to auto-generate clean filenames from the questions
(max 160 chars, truncated with [...] if needed)
and write to directory DIR if provided (defaults to current directory .; creates directories in path if not existing):

  archive-agent gui --to-json-auto Output/

  

📌 Note: As of Archive Agent v12.2.0, corresponding Markdown files (.md) containing the answers are also created when using the --to-json-auto option. (There is currently no way to opt out of this.)

📌 Note: Press CTRL+C in the console to close the GUI server.

Start MCP Server

To start the Archive Agent MCP server, run this:

archive-agent mcp

To see additional information (embedding, retrieval, reranking, querying), pass the --verbose option.

To bypass the AI cache (embedding, reranking) for this query, pass the --nocache option.

To save the query results to JSON files, run this:

- --to-json-auto [DIR] to auto-generate clean filenames from the questions
(max 160 chars, truncated with [...] if needed)
and write to directory DIR if provided (defaults to current directory .; creates directories in path if not existing):

  archive-agent mcp --to-json-auto Output/

  

📌 Note: As of Archive Agent v12.2.0, corresponding Markdown files (.md) containing the answers are also created when using the --to-json-auto option. (There is currently no way to opt out of this.)

📌 Note: Press CTRL+C in the console to close the MCP server.

Standalone STRICT OCR

To OCR a PDF file using the STRICT strategy and output a Markdown file, run this:

archive-agent standalone-ocr-strict "~/Documents/scan.pdf"

Each page is rendered as a full-page image and processed through AI vision OCR.
The result is written as a .md file next to the PDF (e.g., scan.md).

To set custom rendering resolution, pass the --dpi option (default: 150).

To see additional information, pass the --verbose option.

To bypass the AI cache for this operation, pass the --nocache option.

📌 Note: This command does not require a running Qdrant database.