How We Turned Dumb CCTV Footage into a Smart API

The business request sounds simple: accept CCTV footage and identify when and where a rule violation occurred. A vision model, however, returns labels, scores, and coordinates. The API must turn those uncertain observations into a reviewable incident with a timestamp, camera, mapped zone, supporting evidence, and lifecycle state.

• Input contract: an authenticated upload intent, camera identity, capture time range, content metadata, checksum, and purpose.
• Job contract: a durable identifier and explicit states for upload, processing, review, completion, expiry, cancellation, and failure.
• Incident contract: zero or more candidates with evidence, confidence, mapping rationale, review status, and stable identifiers.
• No-result contract: distinguish a completed analysis with no incident from unsupported media, model failure, or expired input.

Video is too large and processing too variable for a single request-response transaction. The API should create a job and issue a time-limited upload capability. The client uploads directly to object storage, and a verified completion event makes the job eligible for processing.

Amazon S3 presigned URLs are one dated implementation example of time-limited object access. Equivalent object stores can provide the same architectural boundary. The upload key must be unique, scoped, short-lived, checksum-validated, and unusable for arbitrary overwrite.

A video is a time-indexed sequence, not merely a folder of pictures. Sampling too sparsely can miss a brief event; sampling every frame multiplies decoding, inference, storage, and review cost. The right strategy depends on event duration, camera motion, scene stability, and model behavior.

OpenCV VideoCapture can read video files, streams, and image sequences. Its documentation also notes that effective behavior depends on the backend, operating system, driver, and hardware, so timestamp and seeking assumptions need tests against the deployed codecs.

The detector should return a vendor-neutral observation: frame timestamp, class, confidence, bounding polygon, model version, and preprocessing version. This keeps the downstream mapping and incident rules independent of one vision service.

• Version every prediction. Store the model, threshold set, class map, and frame-selection configuration used.
• Retain bounded evidence. Preserve only the frames or short clips required for review and evaluation, subject to retention policy.
• Calibrate confidence. Treat raw model scores as ranking signals until tested against the real cameras and target events.
• Support multiple observations. A single frame can contain multiple objects, and a video can contain multiple incidents.

Google Cloud Vision object localization is a dated managed-service example that returns object names, scores, and normalized bounds. A generic object detector still requires task-specific data and evaluation before it can recognize a domain event such as falling waste.

A bounding box does not contain a flat number. The application needs a calibrated map for each camera: image dimensions, active crop, camera revision, zone polygons, blind spots, and the rule for selecting the relevant point on a detected object or trajectory.

1. Normalize coordinates. Convert detections into a consistent coordinate system after rotation, crop, and resize transformations.
2. Choose the event point. A centroid, lower edge, trajectory crossing, or impact point can map to different zones.
3. Match versioned polygons. Camera movement or replacement invalidates old calibration, so mappings need effective dates.
4. Return the rationale. Include the matched zone, overlap or distance, calibration version, and evidence frame.

Repeated detections across adjacent frames are usually evidence for one event, not dozens of incidents. Aggregation groups observations using time windows, track identity, spatial proximity, class, and camera.

• Deduplicate temporally. Merge observations that describe the same continuous event.
• Separate concurrent events. Different tracks or zones can produce multiple incidents in one source video.
• Score the event. Combine supporting frames and mapping consistency rather than selecting only the highest frame score.
• Create stable evidence references. Review and appeal flows should address the same immutable candidate package.

{
  "job_id": "job_01J...",
  "status": "review_required",
  "incidents": [
    {
      "incident_id": "inc_01J...",
      "camera_id": "camera-east-02",
      "occurred_at": "2026-07-29T09:14:32.480Z",
      "candidate_locations": [
        { "zone_id": "flat-7b", "confidence": 0.82 }
      ],
      "model_confidence": 0.88,
      "evidence": {
        "frame_ids": ["frame-1834", "frame-1842"],
        "calibration_version": "east-02-v4"
      },
      "review": { "status": "pending" }
    }
  ]
}

Clients should not infer job state from missing results or HTTP timeouts. A state model gives the CRM, workers, operators, and retention tasks one vocabulary.

Transitions need idempotency and preconditions. An upload event delivered twice must not create two analyses. A late worker must not overwrite a cancelled job. A retry should continue from stored artifacts rather than repeating every completed stage.

A queue decouples ingestion from processing, but it does not guarantee that work happens exactly once. RabbitMQ’s reliability guidance describes acknowledgements, redelivery, publisher confirms, and the need for idempotent consumers.

• Control backpressure. Bound upload size and duration, queue depth, per-tenant concurrency, worker memory, and inference rate.
• Classify failures. Retry transient storage, network, and model-service errors with limits; dead-letter unsupported or repeatedly failing media.
• Evaluate at event level. Measure missed incidents, duplicate incidents, false candidates, location accuracy, review correction, and time-to-result.
• Stratify the dataset. Include camera angles, day/night, weather, compression, occlusion, event duration, and empty footage.
• Monitor drift. Camera movement, scene changes, new codecs, and model updates can invalidate previous thresholds and calibration.

Surveillance data can expose residents, staff, visitors, homes, routines, and disputes. A technically accurate pipeline can still create unacceptable privacy or governance risk. The NIST Privacy Framework provides a general approach for identifying and managing privacy risk; applicable law and local policy require specialist review.

• Define purpose and authority. Document which incidents the system may process, who authorised it, and which uses are prohibited.
• Minimise collection and retention. Process only required cameras and time windows; delete source video, frames, and derived identifiers on defined schedules.
• Restrict access. Separate upload, processing, review, administration, and export permissions; record every evidence access.
• Require human verification. A trained, authorised reviewer confirms evidence before enforcement or communication.
• Support correction and appeal. Preserve review rationale, notify through an appropriate process, and allow disputed results to be re-examined.
• Provide notice. Signage, policy, and user communication should explain the system’s purpose, retention, contact, and complaint path.

The worker should read a durable job record, claim one processing attempt, store intermediate artifacts by stable identity, and finish through a guarded state transition.

def process_video(job_id: str, delivery_id: str) -> None:
    job = jobs.claim(
        job_id=job_id,
        delivery_id=delivery_id,
        allowed_states={"queued", "retryable"},
    )
    if job is None:
        return

    media = object_store.open_verified(job.object_key, job.checksum)
    calibration = calibrations.for_camera(
        camera_id=job.camera_id,
        captured_at=job.capture_started_at,
    )

    observations = []
    for frame in sampler.frames(media, policy=job.sampling_policy):
        for detection in detector.detect(frame.image):
            observations.append(
                normalize(detection, frame, calibration)
            )

    candidates = aggregate_events(observations)
    incidents = [
        map_and_package(candidate, calibration)
        for candidate in candidates
    ]

    jobs.complete(
        job_id=job.id,
        attempt=job.attempt,
        incidents=incidents,
        next_state="review_required" if incidents else "completed",
    )

1. Version the contract. Job, observation, incident, evidence, and review schemas evolve independently of one model.
2. Test real media. Validate codecs, timestamps, rotations, variable frame rates, corrupted files, and camera-specific calibration.
3. Exercise queue failure. Duplicate, delay, reorder, retry, cancel, and dead-letter jobs without duplicating incidents.
4. Evaluate event outcomes. Tune the complete pipeline against reviewed incidents and empty footage.
5. Review privacy controls. Confirm purpose, notice, access, retention, deletion, review, appeal, and evidence-export behavior.
6. Keep enforcement separate. The API produces reviewable evidence; authorised business processes decide what happens next.