SensorMCP Server

ASensorMCP Model Context Protocol (MCP) Serverthat enables automated dataset creation and custom object detection model training through natural language interactions. This project integrates computer vision capabilities with Large Language Models using the MCP standard.

SensorMCP Servercombines the power of foundation models (like GroundedSAM) with custom model training (YOLOv8) to create a seamless workflow for object detection. Using the Model Context Protocol, it enables LLMs to:

- Automatically label images using foundation models
- Create custom object detection datasets
- Train specialized detection models
- Download images from Unsplash for training data

[!NOTE] The Model Context Protocol (MCP) enables seamless integration between LLMs and external tools, making this ideal for AI-powered computer vision workflows.

- Foundation Model Integration: Uses GroundedSAM for automatic image labeling
- Custom Model Training: Fine-tune YOLOv8 models on your specific objects
- Image Data Management: Download images from Unsplash or import local images
- Ontology Definition: Define custom object classes through natural language
- MCP Protocol: Native integration with LLM workflows and chat interfaces
- Fixed Data Structure: Organized directory layout for reproducible workflows

- uvfor package management
- Python 3.13+(uv python install 3.13)
- CUDA-compatible GPU(recommended for training)

git clone <repository-url> cd sensor-mcp

- Set up environment variables(create.envfile):

UNSPLASH_API_KEY=your_unsplash_api_key_here

{ "mcpServers": { "sensormcp-server": { "type": "stdio", "command": "uv", "args": [ "--directory", "/path/to/sensor-mcp", "run", "src/zoo_mcp.py" ] } } }

- list_available_models()- View supported base and target models
- define_ontology(objects_list)- Define object classes to detect
- set_base_model(model_name)- Initialize foundation model for labeling
- set_target_model(model_name)- Initialize target model for training
- fetch_unsplash_images(query, max_images)- Download training images
- import_images_from_folder(folder_path)- Import local images
- label_images()- Auto-label images using the base model
- train_model(epochs, device)- Train custom detection model

Define ontology for "tiger, elephant, zebra"

Set base model to grounded_sam Set target model to yolov8n.pt

Fetch 50 images from Unsplash for "wildlife animals"

sensor-mcp/ ├── src/ │ ├── server.py # Main MCP server implementation │ ├── zoo_mcp.py # MCP entry point │ ├── models.py # Model management and training │ ├── image_utils.py # Image processing and Unsplash API │ ├── state.py # Application state management │ └── data/ # Created automatically │ ├── raw_images/ # Original/unlabeled images │ ├── labeled_images/# Auto-labeled datasets │ └── models/ # Trained model weights ├── static/ # Web interface assets └── index.html # Web interface template

- GroundedSAM: Foundation model for object detection and segmentation

- YOLOv8n.pt: Nano - fastest inference
- YOLOv8s.pt: Small - balanced speed/accuracy
- YOLOv8m.pt: Medium - higher accuracy
- YOLOv8l.pt: Large - high accuracy
- YOLOv8x.pt: Extra Large - highest accuracy
- Create an account atUnsplash Developers
- Create a new application
- Add your access key to the.envfile

- mcp[cli]- Model Context Protocol
- autodistill- Foundation model integration
- torch&torchvision- Deep learning framework
- ultralytics- YOLOv8 implementation
- Fork the repository
- Create a feature branch
- Make your changes
- Add tests for new functionality
- Submit a pull request

If you use this code or data in your research, please cite our paper:

@inproceedings{Guo2025, author = {Guo, Yunqi and Zhu, Guanyu and Liu, Kaiwei and Xing, Guoliang}, title = {A Model Context Protocol Server for Custom Sensor Tool Creation}, booktitle = {3rd International Workshop on Networked AI Systems (NetAISys '25)}, year = {2025}, month = {jun}, address = {Anaheim, CA, USA}, publisher = {ACM}, doi = {10.1145/3711875.3736687}, isbn = {979-8-4007-1453-5/25/06} }

This project is licensed under the MIT License.

For questions about the zoo dataset mentioned in development:Email:yq@anysign.net

Integrates with MLflow, enabling AI assistants to interact with experiments, runs, and registered models.