Skip to content

Ingest Service

This document defines the unified ingest service architecture that handles camera Detection ingestion in both edge and central deployments.

Document Status: Living Document Last Updated: 2025-12-30

Table of Contents

  1. Overview
  2. Deployment Modes
  3. Architecture
  4. Configuration
  5. API Endpoints
  6. Deployment

1. Overview

The ingest service is a single application that handles camera Detection ingestion. It can run in two modes:

  • Edge mode: Deployed on Raspberry Pi at each site, pulls/receives from local cameras
  • Central mode: Deployed on central infrastructure as fallback for sites without edge devices
┌─────────────────────────────────────────────────────────────────┐
│                      Ingest Service                             │
│                                                                 │
│   ┌─────────────┐     ┌─────────────┐     ┌─────────────┐      │
│   │  Adapters   │     │   Buffer    │     │  Uploader   │      │
│   │ (pull/push) │────►│  (SQLite)   │────►│  (HTTPS)    │──────┼──► Central
│   └─────────────┘     └─────────────┘     └─────────────┘      │    Main App
│         ▲                   │                                   │
│         │              edge mode only                           │
│   ┌─────────────┐                                              │
│   │  Receiver   │◄─── Camera webhooks (push mode)              │
│   │ (FastAPI)   │                                              │
│   └─────────────┘                                              │
└─────────────────────────────────────────────────────────────────┘

Key principle: One codebase, two deployment modes. Mode determines default behavior.

2. Deployment Modes

Edge Mode

Deployed on Raspberry Pi at each site. Handles all cameras at that location.

Characteristics: - Pulls from cameras on local LAN (and/or receives pushes) - Buffers events locally (survives network outages) - Uploads to central over internet - Loads cameras.yaml for camera definitions

Use when: Site can accommodate a Raspberry Pi.

Central Mode

Deployed on central infrastructure. Fallback for sites without edge devices.

Characteristics: - Receives camera webhooks only (no pull - can't reach cameras over internet) - No buffer (or minimal) - forwards immediately - Forwards to central main app (internal network or localhost) - Does NOT load cameras.yaml - cameras self-identify via webhook URL

Use when: Site cannot deploy edge device; camera pushes directly to central.

Mode Comparison

Feature Edge Mode Central Mode
Pull adapters Enabled Disabled
Push adapters (receiver) Enabled Enabled
Buffer Enabled (7 days) Disabled
cameras.yaml Loaded Not loaded
Typical upload target Remote central server localhost or internal

3. Architecture

Component Overview

Status: Proposed structure. Implementation tracked in Phase 5 (Edge Collector Core).

shared/ingest/
├── adapters/              # Camera vendor adapters
│   ├── base.py            # Abstract CameraAdapter
│   ├── hikvision.py       # Hikvision ISAPI (pull + push)
│   ├── unifi.py           # Unifi Protect (pull + push)
│   └── factory.py         # Adapter factory with vendor routing
├── normalizer.py          # Detection normalization
├── receiver.py            # FastAPI webhook endpoints (push mode)
├── buffer.py              # SQLite buffer with retention policy
├── uploader.py            # HTTPS client to central main app
├── config.py              # Layered config loader with mode defaults
└── main.py                # Application entry point

Data Flow

Edge mode (pull): 

Camera ◄──GET── Adapter ──► Normalizer ──► Buffer ──► Uploader ──► Central
         (poll)                            (SQLite)    (HTTPS)

Edge mode (push): 

Camera ──POST──► Receiver ──► Adapter ──► Normalizer ──► Buffer ──► Uploader ──► Central
        (webhook)            (parse)                     (SQLite)    (HTTPS)

Central mode (push only): 

Camera ──POST──► Receiver ──► Adapter ──► Normalizer ──► Uploader ──► Central Main App
        (webhook)            (parse)                      (HTTPS)

4. Configuration

Layered Config System

The ingest service uses layered configuration:

  1. Mode defaults - Applied based on mode setting
  2. ingest.yaml - Base configuration (always loaded)
  3. cameras.yaml - Camera definitions (edge mode only)
  4. Environment variables - Secrets and overrides

Mode Defaults

Setting Edge Default Central Default
buffer.enabled true false
buffer.retention_days 7 N/A
pull_adapters.enabled true false

Explicit config values override mode defaults.

Config Files

ingest.yaml (always loaded): - mode - edge or central - buffer - buffer settings (optional, uses mode defaults) - upload - central server URL and credentials - receiver - webhook server settings - logging - log level and format

cameras.yaml (edge mode only): - List of camera definitions - Per-camera: vendor, mode (pull/push), connection details - Credentials via environment variable references

Minimal Configuration

Edge mode requires only: mode, upload.url, upload.api_key

Central mode requires only: mode, upload.url, upload.api_key

All other settings use mode-appropriate defaults.

Implementation Details: See Ingest Service PRP for config schema and implementation patterns.

5. API Endpoints

Webhook Receiver

POST /api/v1/ingest/{vendor}

Receives camera webhook events. Routes to vendor-specific adapter for parsing.

Path Parameters: - vendor - Camera vendor: hikvision, unifi, auto

Query Parameters: - camera_id - Optional camera identifier (for logging/metrics) - key - API key (central mode, for internet-facing deployments)

Health Check

GET /api/v1/ingest/health

Returns service health including buffer depth, camera status, upload status.

6. Deployment

Edge Deployment (Raspberry Pi)

Deployment artifacts in deploy/edge/:

File Purpose
install.sh Deployment script for Pi
systemd/alpr-ingest.service Systemd service definition
config/ingest.yaml.template Config template
config/cameras.yaml.template Camera config template

The install script copies templates, prompts for configuration values, installs systemd service.

Central Ingest Deployment (Docker)

Deployment artifacts in deploy/central-ingest/:

File Purpose
Dockerfile Container image
docker-compose.yml Service definition
config/ingest.yaml Central mode config

Can be added to main central docker-compose.yml or run on separate host.

Implementation Details: Deployment scripts to be created during Phase 5 (Edge Collector Core).

Document Maintainer: Architecture Team Review Cycle: Update when adding ingest features