Model Context Protocol (MCP) Servers are lightweight, stateless services that expose machine learning models as standardized, versioned endpoints. They manage model loading, context provisioning, and inference lifecycle, enabling consistent model serving across environments and frameworks. Use an MCP Server when you need predictable, scalable, and interoperable model inference—especially when deploying models from multiple training pipelines, serving models with dynamic context (e.g., embeddings, metadata, user profiles), or integrating models into microservices with varied client expectations.

Deploying an MCP Server

An MCP Server is deployed as a single HTTP service, typically running in a container. It listens on a single port and serves two primary endpoints: /model for inference and /health for liveness.

Start the server with mcp-server --config config.yaml --model-dir /models.

Configuration

The server reads a YAML configuration file with the following keys:


port: 8080
model-dir: /models
default-context: default-context.json
max-concurrent-inferences: 16
timeout: 30s
logging:
  level: info
  format: json
  output: stdout
  • port: TCP port to bind on (default: 8080).
  • model-dir: Directory where model artifacts are stored, each as a subdirectory named by model ID.
  • default-context: Path to a JSON file used as a fallback context when no context is provided.
  • max-concurrent-inferences: Maximum number of concurrent inference requests (default: 16).
  • timeout: Maximum duration a request may take before timing out (default: 30s).
  • logging: Controls log output format and destination.

Model Directory Structure

Each model is a subdirectory under model-dir with a model.json and a model.bin file:

/models
  /image-classifier-v1
    model.json
    model.bin
  /text-embedding-v2
    model.json
    model.bin

model.json contains metadata:

{
  "id": "image-classifier-v1",
  "version": "1.2.0",
  "framework": "pytorch",
  "input_schema": {
    "type": "object",
    "properties": {
      "image": {
        "type": "array",
        "items": { "type": "number" },
        "description": "Flattened 224x224 RGB image tensor"
      }
    },
    "required": ["image"]
  },
  "output_schema": {
    "type": "object",
    "properties": {
      "labels": {
        "type": "array",
        "items": { "type": "string" }
      },
      "probabilities": {
        "type": "array",
        "items": { "type": "number" }
      }
    }
  },
  "context_schema": {
    "type": "object",
    "properties": {
      "user_id": { "type": "string" },
      "session_id": { "type": "string" },
      "embedding": {
        "type": "array",
        "items": { "type": "number" },
        "description": "Precomputed user embedding"
      }
    }
  }
}

The context_schema defines the shape of the context object expected in each inference request. The server validates context against this schema and merges it with the default-context file.

Inference Requests

Clients send POST requests to /model with a JSON body containing model_id, inputs, and optional context.

Request Format

{
  "model_id": "image-classifier-v1",
  "inputs": {
    "image": [0.1, 0.2, 0.3, ..., 0.9]  // 150528 values
  },
  "context": {
    "user_id": "u_12345",
    "session_id": "s_abcde",
    "embedding": [0.15, -0.3, 0.02, ...]
  }
}

The model_id must match a subdirectory in model-dir. The inputs object must conform to the input_schema of the model.

Response Format

On success, the server returns a 200 OK with a JSON body:

{
  "model_id": "image-classifier-v1",
  "version": "1.2.0",
  "outputs": {
    "labels": ["cat", "dog", "bird"],
    "probabilities": [0.82, 0.15, 0.03]
  },
  "context": {
    "user_id": "u_12345",
    "session_id": "s_abcde",
    "embedding": [0.15, -0.3, 0.02, ...],
    "inference_time_ms": 42
  },
  "request_id": "req_98765"
}
  • outputs contains the model’s prediction.
  • context is the full context, including merged input context and additional metadata like inference_time_ms.
  • request_id is a unique identifier assigned per request.

Context Management

Context is a first-class concept in MCP Servers. It is passed from client to server and persisted between requests. The server merges context from three sources:

  1. The default-context file.
  2. The context object in the request.
  3. Internal state (e.g., model-specific metadata, session state).

Context Merging Rules

  • Keys in the request context override keys in the default context.
  • If a context key is missing, it is populated from the default.
  • The server may enrich the context with metadata (e.g., inference_time_ms, model_load_time_ms).

Context Schema Validation

The server validates the context object against the context_schema defined in model.json. If validation fails, it returns a 400 Bad Request with error details:

{
  "error": "invalid_context",
  "message": "Context failed schema validation",
  "details": [
    {
      "field": "user_id",
      "expected": "string",
      "actual": "number",
      "path": "user_id"
    },
    {
      "field": "embedding",
      "expected": "array of numbers",
      "actual": "array of strings",
      "path": "embedding"
    }
  ]
}

This error is critical: clients must validate context before sending requests. A missing or malformed user_id in a request can silently degrade performance.

Error Handling

MCP Servers return standardized error responses for all non-200 responses.

400 Bad Request

Returned when the request body is invalid or context schema validation fails.

Example:

{
  "error": "invalid_request",
  "message": "Model ID not found",
  "details": {
    "model_id": "text-classifier-v3"
  }
}

404 Not Found

Returned when model_id does not exist in model-dir.

{
  "error": "model_not_found",
  "message": "Model with ID 'image-classifier-v2' not found",
  "details": {
    "model_id": "image-classifier-v2",
    "available_models": [
      "image-classifier-v1",
      "text-embedding-v2"
    ]
  }
}

500 Internal Server Error

Returned when an internal model error occurs—e.g., model binary fails to load, inference crashes.

{
  "error": "internal_error",
  "message": "Model failed to compute inference",
  "details": {
    "exception": "RuntimeError",
    "message": "Failed to load model state",
    "traceback": [
      "File \"model.py\", line 45, in predict",
      "    model.load_state_dict(state)",
      "RuntimeError: size mismatch at index 0",
      "  Expected: 256, got: 255"
    ]
  }
}

503 Service Unavailable

Returned when the server is overloaded or under high latency.

{
  "error": "service_unavailable",
  "message": "Maximum concurrent inferences reached",
  "details": {
    "current_concurrent_inferences": 16,
    "max_concurrent_inferences": 16,
    "queued_requests": 5
  }
}

High Availability and Scaling

MCP Servers are designed for horizontal scaling. Each instance is stateless and shares a common model-dir (e.g., via NFS or S3).

Health Checks

Clients and orchestration tools (e.g., Kubernetes) use /health endpoint:

GET /health HTTP/1.1
Host: mcp-server.example.com

Response:

{
  "status": "healthy",
  "timestamp": "2024-04-15T10:30:00Z",
  "uptime_seconds": 3412,
  "loaded_models": [
    "image-classifier-v1",
    "text-embedding-v2"
  ],
  "memory_usage_bytes": 104857600,
  "model_load_times_ms": {
    "image-classifier-v1": 120,
    "text-embedding-v2": 85
  }
}

A status field of "unhealthy" triggers a rollout or restart. The model_load_times_ms field helps diagnose slow startup.

Load Balancing

A load balancer distributes requests across multiple MCP Server instances. Because each server maintains its own model cache, it is common to see:

  • Warm-up latency: First request to a model may take 100–300 ms to load the model binary.
  • Cache misses: Model not yet loaded into memory, requiring disk I/O.

Model Versioning and Rolling Updates

When updating models, clients can target specific versions via model_id or use versioned IDs:

  • image-classifier-v1 (latest stable)
  • image-classifier-v1.2
  • image-classifier-v1.2.1

During a rollout, the server supports multiple versions of the same model ID. The model.json file includes a version field used to route requests.

Clients may send model_id with a version suffix:

{
  "model_id": "image-classifier-v1.2.1",
  "inputs": { "image": [...] }
}

If image-classifier-v1.2.1 does not exist, the server falls back to image-classifier-v1.

Silent Failures and Sharp Edges

Model Binary Not Found

If model.bin is missing in a model directory, the server logs:

ERROR: Failed to load model 'image-classifier-v1': model.bin not found

Clients receive a 500 error with error: model_load_failed.

Context Schema Mismatch

If the request context contains a field not defined in context_schema, the server logs:

WARN: Unknown context field 'metadata' in model 'text-embedding-v2'

The field is included in the response context but not validated.

Schema Version Mismatch

The server does not validate schema versions. If a client sends context_schema version 1.0 but the model expects 1.1, the server uses the 1.1 schema and logs:

WARN: Context schema version mismatch: expected 1.1, got 1.0

This leads to subtle bugs: a new embedding field added in 1.1 may be ignored if the client uses 1.0.

Timeout and Concurrency Limits

Requests exceeding timeout are logged and may be canceled:

INFO: Request req_98765 timed out after 30s

If max-concurrent-inferences is reached, new requests are queued and returned as 503. The queue depth is not exposed in the response.

Model Binary Corruption

If model.bin is corrupted (e.g., truncated), the server returns:

{
  "error": "model_corrupted",
  "message": "Failed to deserialize model binary",
  "details": {
    "exception": "EOFError",
    "message": "Unexpected end of file",
    "expected_size": 1048576,
    "actual_size": 1048570
  }
}

Default Context Not Applied

If default-context.json is missing or malformed, the server uses a minimal default context:

{
  "user_id": "anonymous",
  "session_id": "default_session",
  "embedding": []
}

Clients should validate this default behavior in integration tests.

Monitoring and Debugging

Use curl to probe the server:

curl -X POST http://localhost:8080/model \
  -H "Content-Type: application/json" \
  -d '{
    "model_id": "image-classifier-v1",
    "inputs": { "image": [0.1, 0.2, 0.3] }
  }'

Inspect logs for error patterns:

ERROR: Model image-classifier-v1 failed to load
  Exception: RuntimeError: Failed to load model state
  Traceback:
    File "model.py", line 45, in predict
      model.load_state_dict(state)
    RuntimeError: size mismatch at index 0
      Expected: 256, got: 255

Use systemctl to manage the server:

sudo systemctl start mcp-server
sudo systemctl status mcp-server
sudo journalctl -u mcp-server -f

Configuration Reload

The server supports hot configuration reload on SIGHUP:

kill -HUP $(pgrep mcp-server)

On reload, it re-reads config.yaml, updates logging, and applies new max-concurrent-inferences.

Summary of Key Points

  • The MCP Server is a single, stateless HTTP service for ML model inference.
  • Each model is a directory with model.json and model.bin.
  • Context is a first-class object, merged from default, request, and internal state.
  • Errors are standardized and include error, message, and details.
  • High availability is achieved via horizontal scaling and shared model-dir.
  • Silent failures include missing model binaries, context schema mismatches, and timeout handling.
  • Monitoring uses curl, systemctl, and structured logs.

This page was rewritten on 11 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at hello@vibecodewiki.ai and we will correct it.