AI Development in Monorepos

AI development in monorepos centralizes machine learning models, data pipelines, training scripts, and serving infrastructure under a single codebase. This approach is essential when multiple teams maintain interdependent models, share datasets, and deploy services that rely on consistent model versions and feature engineering. It matters when a change to a core feature extractor must be validated across three downstream models, or when a new data preprocessing step needs to ripple through training, evaluation, and inference pipelines.

Managing Model Versions Across Services

Model versioning in monorepos requires coordination between model artifacts, metadata, and deployment manifests. Each model is stored in a models/ directory, with a model.yaml file in its root. The file defines the model’s name, version, input schema, and dependencies.

name: recommendation
version: 2024-03-15
input_schema:
  - name: user_id
    type: integer
    required: true
  - name: item_ids
    type: array
    items:
      type: integer
    required: true
dependencies:
  - torch==2.1.0
  - scikit-learn==1.4.0
  - pandas==2.1.0

A model.yaml file is the canonical source of truth for a model’s identity. When a model is trained, the output is a model.tar.gz file placed in models/recommendation/2024-03-15/artifacts/. The training pipeline expects this structure and writes a model.json file alongside it.

// models/recommendation/2024-03-15/artifacts/model.json
{
  "training_date": "2024-03-15T12:00:00Z",
  "metrics": {
    "accuracy": 0.876,
    "f1_score": 0.851
  },
  "dataset_version": "v3.2",
  "training_duration_seconds": 4217
}

To register a model, run:

./scripts/register_model.sh models/recommendation/2024-03-15

This script reads model.yaml, copies the artifact, and registers the model in a shared metadata store via HTTP.

curl -X POST http://registry.api/models \
  -H "Content-Type: application/json" \
  -d @- << EOF
{
  "name": "recommendation",
  "version": "2024-03-15",
  "artifact_path": "models/recommendation/2024-03-15/artifacts/model.tar.gz",
  "metadata": {
    "training_date": "2024-03-15T12:00:00Z",
    "metrics": {
      "accuracy": 0.876,
      "f1_score": 0.851
    }
  },
  "dependencies": [
    "torch==2.1.0",
    "scikit-learn==1.4.0",
    "pandas==2.1.0"
  ]
}
EOF

The registry returns a model_id and updates a models/index.json file in the monorepo root.

Common Pitfalls

• The model.json file is often written before the model.tar.gz is finalized, leading to mismatched metadata.
• The model.yaml file is not versioned with the model artifact; a model version may refer to a model.yaml from a prior commit.
• The register_model.sh script fails silently if the artifacts/ directory does not exist, but no error is logged.
• When the model registry is down, the script retries once but does not roll back the model directory.

Sharing Features and Data Pipelines

Feature engineering is centralized in features/ and shared across models. Each feature set is defined in a feature.yaml file and built by a build_features.py script.

# features/user_profile/feature.yaml
name: user_profile
description: "Aggregated user behavior over 30 days"
input: "raw/user_events.parquet"
output: "features/user_profile/2024-03-15"
dependencies:
  - features/common/standardize.py
  - features/common/encode_categorical.py

The build script is invoked with:

python features/user_profile/build_features.py \
  --input raw/user_events.parquet \
  --output features/user_profile/2024-03-15 \
  --config features/user_profile/feature.yaml \
  --parallel \
  --cache-dir .cache/features

The build_features.py script reads feature.yaml, loads input data, applies transformations, and outputs a Parquet file with the feature set.

The key challenge is ensuring feature consistency across model training and inference. A model trained on features/user_profile/2024-03-15 will fail at inference if the build_features.py script is run with a different version of feature.yaml.

Versioning the Feature Pipeline

The feature.yaml file is versioned alongside the feature data. When a new feature is added, the pipeline is updated and tested in a dev branch. Once validated, the feature is promoted to main and a new version is tagged.

git checkout -b feature/new_user_engagement
# modify feature.yaml
git add features/user_profile/feature.yaml
git commit -m "Add user engagement score to profile features"
git push origin feature/new_user_engagement

A CI job runs build_features.py and compares the output to the expected schema using schema.json.

// features/user_profile/2024-03-15/schema.json
{
  "fields": [
    {
      "name": "user_id",
      "type": "int64"
    },
    {
      "name": "total_sessions",
      "type": "double"
    },
    {
      "name": "avg_session_duration",
      "type": "double"
    },
    {
      "name": "engagement_score",
      "type": "double"
    }
  ]
}

The CI job fails if the schema differs from schema.json. It logs the diff and attaches the generated Parquet file.

Silent Failure Mode

The build_features.py script assumes that input and output paths are absolute. If the script is run from the monorepo root but feature.yaml uses relative paths, the input file is not found, and the job exits with no error message.

# In monorepo root
python features/user_profile/build_features.py --config features/user_profile/feature.yaml
# Expected: raw/user_events.parquet
# Actual: features/user_profile/raw/user_events.parquet

The file is not found because the relative path resolves from the script’s location, not the current working directory.

To fix this, the script must set os.chdir(os.path.dirname(__file__)) at startup, or the feature.yaml must use absolute paths.

Synchronizing Training and Inference

Training pipelines and inference services must use the same model version and feature set. The train.py script reads training.yaml and calls register_model.sh.

# training.yaml
model_name: recommendation
feature_set: user_profile
train_date: 2024-03-15
epochs: 100
batch_size: 256
learning_rate: 0.001

The training script runs:

python train.py --config training.yaml --features features/user_profile/2024-03-15

After training, it calls register_model.sh with the output directory.

./scripts/register_model.sh models/recommendation/2024-03-15

Inference services consume the model via a REST API. A service running in services/recommendation-api mounts the model directory and serves it with uvicorn.

uvicorn app.main:app \
  --host 0.0.0.0 \
  --port 8080 \
  --workers 4 \
  --model-dir /models/recommendation/2024-03-15

The service expects the model to be in a specific structure:

/models/recommendation/2024-03-15/
├── artifacts/
│   ├── model.tar.gz
│   └── model.json
├── model.yaml
└── schema.json

The service logs the model version at startup:

{
  "service": "recommendation-api",
  "version": "1.2.0",
  "model_version": "2024-03-15",
  "model_id": "rec-45812",
  "started_at": "2024-03-15T14:30:00Z"
}

The Model Version Mismatch

The most common production failure is a mismatch between the model version used in training and the one deployed to inference. The training script writes model.yaml and model.json in models/recommendation/2024-03-15, but the inference service loads the model from /models/recommendation/2024-03-15 — a directory that may contain an older model.

The service starts up with a model version that does not match the training output. The logs show:

2024-03-15 14:30:01 INFO  [recommendation-api] Model version mismatch: expected 2024-03-15, found 2024-03-10

The root cause is that the register_model.sh script copies the model directory but does not update the model.yaml file in the deployment directory.

To fix this, the deployment process must:

1. Copy the new model directory to /models/recommendation/2024-03-15
2. Create a symbolic link current pointing to the latest version
3. Update model.yaml in the deployment directory to reflect the current version

# After registration
cp -r models/recommendation/2024-03-15 /models/recommendation/2024-03-15
ln -sf /models/recommendation/2024-03-15 /models/recommendation/current
sed -i 's/version: .*/version: 2024-03-15/' /models/recommendation/current/model.yaml

Without this step, inference services may fail silently, using a model that was trained on data not available during deployment.

Coordinating CI and CD Across AI Services

CI pipelines run on main branch and on pull requests. Each service has a ci.yml file in .github/workflows/.

# .github/workflows/ci.yml
name: CI
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install -r features/requirements.txt
          pip install -r services/recommendation-api/requirements.txt
      - name: Run tests
        run: |
          python -m pytest tests/
          python features/user_profile/test_features.py
          python services/recommendation-api/test_api.py
      - name: Build features
        run: |
          python features/user_profile/build_features.py \
            --config features/user_profile/feature.yaml \
            --output features/user_profile/2024-03-15 \
            --parallel
      - name: Train model
        run: |
          python train.py --config training.yaml --features features/user_profile/2024-03-15
          ./scripts/register_model.sh models/recommendation/2024-03-15
      - name: Upload model artifacts
        run: |
          tar -czf model.tar.gz \
            models/recommendation/2024-03-15/artifacts/model.tar.gz \
            models/recommendation/2024-03-15/model.json \
            models/recommendation/2024-03-15/model.yaml
          github upload-artifact --name model-2024-03-15 --path model.tar.gz

The CI job produces a model.tar.gz artifact and uploads it to GitHub. The CD pipeline then deploys the model to staging and production.

The Hidden Cost of Parallel CI

When multiple services depend on a shared feature set, a PR that updates the feature pipeline may trigger CI jobs in multiple repositories. The user_profile feature is used by both the recommendation-api and personalization-api services.

A PR that modifies features/user_profile/feature.yaml runs CI in both repositories. The recommendation-api service builds features and trains a model, but the personalization-api service uses an older feature set because its CI job runs before the PR is merged.

The solution is to use a feature-lock.json file that pins the feature version to a specific commit.

// features/user_profile/feature-lock.json
{
  "commit": "a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2",
  "version": "2024-03-15",
  "dependencies": {
    "standardize.py": "3a2b1c0d",
    "encode_categorical.py": "4b3c2d1e"
  }
}

The build_features.py script reads feature-lock.json and ensures that the exact version of each dependency is used. When a service is deployed, it includes feature-lock.json in the deployment package.

CI Failure Patterns

• The CI job fails to install dependencies because requirements.txt is not updated with new versions.
• The build_features.py script uses a cached version of the feature set, but the cache is not invalidated across runs.
• The train.py script outputs model.json but does not validate that model.yaml is present.
• The register_model.sh script assumes model.yaml exists in the model directory but fails silently if it does not.

Automating Monorepo Releases

Releases are coordinated via a release.sh script that updates version numbers, creates tags, and pushes to remote.

./scripts/release.sh \
  --version 2024.03.15 \
  --models recommendation \
  --features user_profile \
  --services recommendation-api

The script:

1. Updates package.json and pyproject.toml with the new version.
2. Creates a git tag v2024.03.15.
3. Commits all changes.
4. Pushes to origin.
5. Runs ci.yml on main.

The release script also generates a RELEASE.md file with a changelog, model versions, and feature set versions.

# Release v2024.03.15

## Models
- recommendation: 2024-03-15 (trained on user_profile v2024-03-15)

## Features
- user_profile: 2024-03-15 (based on feature-lock.json)

## Services
- recommendation-api: v1.2.0
- personalization-api: v0.8.1

## Changelog
- Add engagement score to user profile features
- Improve recommendation model accuracy by 2.3%
- Fix missing feature in inference pipeline

The release script is idempotent: running it again with the same version number does not re-create the tag or commit.

Release Pitfalls

• The release.sh script assumes git is available in the PATH but fails silently if it is not.
• The script updates package.json but not pyproject.toml, leading to mismatched version numbers.
• The changelog is generated from git log, but the log does not include commits from models/, features/, and services/ directories.
• The script does not validate that all model artifacts exist before creating the release.

The release process is only as strong as the release.sh script. A broken release script leads to inconsistent versions across the monorepo, making it hard to reproduce a deployed system.

This page was rewritten on 11 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at hello@vibecodewiki.ai and we will correct it.