Unified NVR System - Engineering Architecture
System Overview: The Unified NVR is a multi-vendor Network Video Recorder built with Flask backend, ES6+jQuery frontend, and Docker containerization. It provides unified streaming, PTZ control, motion detection, recording, and two-way audio (talkback) across Eufy, Reolink, UniFi, Amcrest, and SV3C cameras via WebRTC (~200ms), LL-HLS (~2s), HLS, MJPEG, snapshot polling, and Neolink (Baichuan) protocols. Features include playback volume control with per-camera persistence and go2rtc integration for ONVIF AudioBackChannel. Mobile-optimized with iOS snapshot polling and Android MJPEG for grid view. Last updated: January 26, 2026.
3
Motion Detection Methods
Level 1: System Overview
High-Level Architecture: User browsers connect to a Flask web application which orchestrates camera streams via vendor-specific handlers. FFmpeg processes transcode RTSP sources and publish to MediaMTX, which serves multiple delivery formats: WebRTC (WHEP) for ~200ms latency, LL-HLS for ~2s latency, and RTSP re-export for motion detection and recording services. This allows multiple consumers without opening additional camera connections. Credentials are managed via AWS Secrets Manager. Neolink provides Baichuan-to-RTSP translation for cameras with broken RTSP (e.g., Reolink E1 series).
graph LR
A[Web Browser
WebRTC + HLS.js + MJPEG] -->|HTTP/WebRTC| B[Flask App
app.py:5000]
B -->|API Calls| C[StreamManager
Orchestrator]
C -->|Strategy Pattern| D[Vendor Handlers
Eufy/Reolink/UniFi/Amcrest/SV3C]
D -->|RTSP Input| E[FFmpeg
Dual-Output Transcoder]
E -->|RTSP Publish| F[MediaMTX
WebRTC:8889 HLS:8888]
F -->|WebRTC WHEP ~200ms| A
F -->|HLS Segments ~2s| A
F -->|RTSP Re-export| J[Motion Detection
Recording Service]
G[(AWS Secrets
Manager)] -.->|Credentials| D
H[(cameras.json
Config)] -.->|Camera Metadata| B
I[IP Cameras
17+ Devices] -->|RTSP/P2P| D
K[Neolink
Baichuan Bridge] -->|RTSP| D
I -->|Baichuan:9000| K
style A fill:#e1f5ff
style B fill:#667eea,color:#fff
style C fill:#17a2b8,color:#fff
style D fill:#28a745,color:#fff
style E fill:#fd7e14,color:#fff
style F fill:#6f42c1,color:#fff
style G fill:#fff3cd
style H fill:#fff3cd
style I fill:#d4edda
style J fill:#dc3545,color:#fff
style K fill:#20c997,color:#fff
Recording/Motion Services
Technology Stack
- Backend: Python 3.11 + Flask + Threading
- Frontend: ES6 + jQuery + WebRTC (WHEP) + HLS.js + Axios
- Video Processing: FFmpeg 7+ with dual-output support (sub + main streams)
- Stream Delivery: MediaMTX for WebRTC (~200ms), LL-HLS (~2s), and RTSP re-export
- Protocol Bridge: Neolink v0.6.2 for Baichuan-to-RTSP translation
- Containerization: Docker + Docker Compose
- Credentials: AWS Secrets Manager integration
- Database: PostgreSQL + PostgREST (recording metadata, PTZ latency learning)
- Motion Detection: Reolink Baichuan, ONVIF PullPoint, FFmpeg scene detection
Level 2: Component Architecture
Internal Components: The application initializes multiple services at startup: CameraRepository for config access, StreamManager for stream orchestration, vendor-specific credential providers, ONVIF PTZ controllers, recording services with pre-buffer support, motion detection services (Baichuan/ONVIF/FFmpeg), segment buffer manager, and client-side health monitoring with escalating recovery.
graph TB
subgraph Init["Application Initialization (app.py)"]
A[Flask App] --> B[CameraRepository
./config/cameras.json]
A --> C[StreamManager]
A --> D[RecordingService]
A --> E[SnapshotService]
A --> F[ONVIF PTZ Handler]
end
subgraph Credentials["Credential Providers"]
G[EufyCredentialProvider]
H[UniFiCredentialProvider]
I[ReolinkCredentialProvider]
J[AmcrestCredentialProvider]
J2[SV3CCredentialProvider]
end
subgraph Handlers["Stream Handlers (Strategy Pattern)"]
K[EufyStreamHandler]
L[UniFiStreamHandler]
M[ReolinkStreamHandler]
N[AmcrestStreamHandler]
N2[SV3CStreamHandler]
end
subgraph Services["Background Services"]
O[BridgeWatchdog
Eufy P2P Monitor]
P[SegmentBufferManager
Pre-Buffer Recording]
Q[RecordingMonitor
Daemon Thread]
R[ReolinkMotionService
Baichuan Protocol]
end
subgraph Motion["Motion Detection"]
S[ONVIF Event Listener
PullPoint Subscription]
T[FFmpeg Motion Detector
Scene Detection]
end
C --> Handlers
Credentials --> Handlers
B --> C
B --> D
D --> E
D --> P
A --> Services
A --> Motion
Motion --> D
style A fill:#667eea,color:#fff
style B fill:#17a2b8,color:#fff
style C fill:#28a745,color:#fff
style D fill:#6f42c1,color:#fff
style E fill:#6f42c1,color:#fff
style F fill:#fd7e14,color:#fff
style S fill:#dc3545,color:#fff
style T fill:#dc3545,color:#fff
style P fill:#20c997,color:#fff
Component Responsibilities
- CameraRepository: Loads and manages camera configurations from
cameras.json, provides filtering by type/capability
- StreamManager: Orchestrates FFmpeg processes with dual-output (sub/main), manages stream lifecycle, implements watchdog restarts
- Credential Providers: Fetch credentials from AWS Secrets Manager per vendor (Eufy, Reolink, UniFi, Amcrest, SV3C)
- Stream Handlers: Build RTSP URLs, FFmpeg parameters using Strategy Pattern; SV3C added Dec 2025
- RecordingService: Manages continuous, motion-triggered, and manual recording with pre-buffer support
- SegmentBufferManager: Rolling 5-second segment buffer for pre-motion capture
- SnapshotService: Periodic JPEG capture from MediaMTX RTSP output
- ONVIF PTZ Handler: Standardized pan/tilt/zoom/preset control across Amcrest, Reolink, SV3C
- Motion Detection: Three methods: Reolink Baichuan (push), ONVIF PullPoint (poll), FFmpeg scene detection (universal)
Level 3: Streaming Data Flow
Protocol Pipeline: Camera RTSP sources are ingested by FFmpeg, transcoded with vendor-specific parameters, then published to MediaMTX via RTSP. A single FFmpeg process produces dual outputs: sub-stream (transcoded, scaled for grid view) and main-stream (passthrough for fullscreen). MediaMTX provides multiple delivery options: WebRTC via WHEP protocol (~200ms latency), LL-HLS segments (~2s latency via HLS.js), and RTSP re-export for motion detection and recording services—all without opening additional camera connections.
flowchart LR
subgraph Camera["Camera Source"]
CAM[IP Camera
RTSP/P2P]
NEO[Neolink Bridge
Baichuan:9000]
end
subgraph FFmpeg["FFmpeg Dual-Output Process"]
IN[RTSP Input
-rtsp_transport tcp
-reconnect 1]
SUB[Sub Output
-c:v libx264 -vf scale
Transcoded 320x240]
MAIN[Main Output
-c:v copy
Passthrough 1280x720]
end
subgraph MediaMTX["MediaMTX Packager"]
INGEST[RTSP Ingest
:8554]
PACK[LL-HLS Packaging
part_duration: 250ms
segment_duration: 500ms]
HLS[HLS Output
:8888/*/index.m3u8]
WEBRTC[WebRTC Output
:8889/*/whep ~200ms]
RTSP_OUT[RTSP Re-export
:8554 for consumers]
end
subgraph Browser["Browser"]
WEBRTCJS[WebRTC Player
RTCPeerConnection]
HLSJS[HLS.js Player
lowLatencyMode: true]
VIDEO[HTML5 Video]
HEALTH[Health Monitor
Blank Frame Detection]
end
subgraph Services["Backend Services"]
MOTION[Motion Detector
FFmpeg/ONVIF]
REC[Recording Service
Pre-Buffer Support]
end
CAM --> IN
CAM -->|Broken RTSP| NEO
NEO --> IN
IN --> SUB
IN --> MAIN
SUB --> INGEST
MAIN --> INGEST
INGEST --> PACK
PACK --> HLS
INGEST --> WEBRTC
INGEST --> RTSP_OUT
WEBRTC --> WEBRTCJS
HLS --> HLSJS
WEBRTCJS --> VIDEO
HLSJS --> VIDEO
VIDEO --> HEALTH
RTSP_OUT --> MOTION
RTSP_OUT --> REC
style CAM fill:#d4edda
style NEO fill:#20c997,color:#fff
style IN fill:#fd7e14,color:#fff
style SUB fill:#fd7e14,color:#fff
style MAIN fill:#fd7e14,color:#fff
style INGEST fill:#6f42c1,color:#fff
style PACK fill:#6f42c1,color:#fff
style HLS fill:#6f42c1,color:#fff
style WEBRTC fill:#00b894,color:#fff
style RTSP_OUT fill:#6f42c1,color:#fff
style WEBRTCJS fill:#00b894,color:#fff
style HLSJS fill:#e1f5ff
style VIDEO fill:#e1f5ff
style HEALTH fill:#17a2b8,color:#fff
style MOTION fill:#dc3545,color:#fff
style REC fill:#dc3545,color:#fff
Latency Characteristics
- WebRTC (Lowest): ~200-500ms via WHEP protocol; best for real-time PTZ control and monitoring
- LL-HLS: ~2-4 seconds due to segmentation and browser buffering; more compatible than WebRTC
- Classic HLS: 4-8 seconds; maximum compatibility, good for archive playback
- MJPEG: Sub-second latency via direct proxy; works for cameras with native MJPEG support
- Dual-Output Design: Single camera connection, two MediaMTX paths (sub/main) for grid and fullscreen views
- WebRTC LAN-Only: Current config uses direct ICE without STUN/TURN (add STUN for remote access)
- Neolink Bridge: Adds ~100-200ms for Baichuan-to-RTSP translation; required for cameras with broken RTSP
Audio Architecture (Jan 19, 2026)
Key Discovery: WebRTC only supports Opus audio codec. MediaMTX explicitly skips AAC/MPEG-4 audio tracks for WebRTC sessions.
- Camera Native Audio: Most cameras (UniFi, Reolink) output AAC LC @ 16kHz mono
- WebRTC Requirement: Must transcode AAC → Opus for audio to work
- FFmpeg Config:
"c:a": "libopus", "b:a": "32k" in cameras.json audio section
- Video Passthrough: Video stays as copy (
-c:v copy) - only audio is transcoded (minimal CPU)
- Both Streams: Sub and main streams both use Opus (fullscreen also uses WebRTC)
| Stream Type |
Audio Codec |
Notes |
| WebRTC |
Opus required |
AAC skipped by MediaMTX; must transcode |
| LL-HLS / HLS |
AAC preferred |
Opus has limited Safari support in HLS |
| MJPEG |
N/A |
Video only, no audio track |
Config Location: config/cameras.json → per-camera ll_hls.audio section
Code Location: streaming/ffmpeg_params.py lines 386-395 (sub stream) and 417-425 (main stream)
flowchart TD
Start([Stream Request]) --> Check{Stream
Active?}
Check -->|Yes| Return[Return Existing URL]
Check -->|No| Reserve[Reserve Slot
status: 'starting']
Reserve --> Thread[Spawn Background Thread]
Thread --> GetCam[Get Camera Config]
GetCam --> GetHandler[Get Vendor Handler]
GetHandler --> BuildURL[Build RTSP URL
with Credentials]
BuildURL --> Protocol{Stream
Protocol?}
Protocol -->|MJPEG| Skip[Skip FFmpeg
Use MJPEG Proxy]
Protocol -->|RTMP| RTMP[Start FFmpeg
-f flv stdout]
Protocol -->|WEBRTC| WEBRTC[Start FFmpeg
Publish to MediaMTX
Browser uses WHEP]
Protocol -->|LL_HLS| LLHLS[Start FFmpeg
Publish to MediaMTX]
Protocol -->|HLS| HLS[Start FFmpeg
Write Local Segments]
RTMP --> Register[Register Active Stream]
WEBRTC --> Register
LLHLS --> Register
HLS --> Register
Register --> Watchdog[Start Watchdog Thread]
Watchdog --> Monitor[Monitor Process Health]
Monitor --> Healthy{Process
Running?}
Healthy -->|Yes| Monitor
Healthy -->|No| Restart[Trigger Restart]
Restart --> BuildURL
style Start fill:#28a745,color:#fff
style Check fill:#ffc107,color:#000
style Protocol fill:#ffc107,color:#000
style Healthy fill:#ffc107,color:#000
style Register fill:#667eea,color:#fff
style Watchdog fill:#17a2b8,color:#fff
style Restart fill:#dc3545,color:#fff
Thread Safety Architecture
- Master Lock:
_streams_lock (RLock) protects active_streams dictionary
- Per-Camera Locks:
_restart_locks prevent concurrent restart attempts
- Slot Reservation: Stream slots marked 'starting' before thread spawn to prevent duplicates
- Lock Discipline: Never sleep() while holding locks to prevent deadlocks
- Race Condition Fix (Jan 2026):
get_active_streams() no longer stops 'starting' streams during status queries
Level 4: Vendor Handler Strategy Pattern
Design Pattern: StreamManager delegates vendor-specific logic to handler classes implementing a common interface. Each handler knows how to build RTSP URLs, FFmpeg input/output parameters, and LL-HLS publish commands for its vendor.
classDiagram
class StreamHandler {
+build_rtsp_url(camera_config, stream_type) str
+get_ffmpeg_input_params(camera_config) List
+get_ffmpeg_output_params(stream_type, camera_config) List
+_build_ll_hls_publish(camera_config, rtsp_url) Tuple
+validate_camera_config(camera_config) bool
+get_required_config_fields() List
}
class EufyStreamHandler {
-credential_provider: EufyCredentialProvider
-bridge_config: Dict
+build_rtsp_url() str
}
class ReolinkStreamHandler {
-credential_provider: ReolinkCredentialProvider
-reolink_config: Dict
+build_rtsp_url() str
}
class UniFiStreamHandler {
-credential_provider: UniFiCredentialProvider
-protect_config: Dict
+build_rtsp_url() str
}
class AmcrestStreamHandler {
-credential_provider: AmcrestCredentialProvider
+build_rtsp_url() str
}
class SV3CStreamHandler {
-credential_provider: SV3CCredentialProvider
+build_rtsp_url() str
+_build_ll_hls_publish() str
}
StreamHandler <|-- EufyStreamHandler
StreamHandler <|-- ReolinkStreamHandler
StreamHandler <|-- UniFiStreamHandler
StreamHandler <|-- AmcrestStreamHandler
StreamHandler <|-- SV3CStreamHandler
class StreamManager {
-handlers: Dict
-active_streams: Dict
-camera_repo: CameraRepository
+start_stream(camera_serial, stream_type)
+stop_stream(camera_serial)
+get_stream_url(camera_serial)
}
StreamManager --> StreamHandler : uses
| Vendor |
RTSP URL Format |
Authentication |
Special Handling |
| Eufy |
rtsp://user:pass@bridge:554/live0 |
Per-camera credentials via Eufy Bridge |
P2P bridge required for camera access |
| Reolink |
rtsp://user:pass@cam:554/h264Preview_01_sub |
Camera local credentials |
Supports Baichuan protocol for motion events |
| UniFi |
rtsps://console:7441/proxy_url |
Protect console API token |
TLS required, session management |
| Amcrest |
rtsp://user:pass@cam:554/cam/realmonitor |
Camera local credentials |
ONVIF PTZ (primary), CGI API (fallback) |
| SV3C |
rtsp://user:pass@cam:554/12 (sub) /11 (main) |
Environment variables |
ONVIF PTZ, digital zoom only, single RTSP connection limit |
| Neolink |
rtsp://neolink:8554/{serial} |
Baichuan protocol credentials |
Bridge for cameras with broken RTSP (Reolink E1 series) |
flowchart TB
subgraph FFmpegParams["FFmpegHLSParamBuilder"]
Builder[FFmpegHLSParamBuilder
camera_name, stream_type
camera_rtsp_config, vendor_prefix]
BuildRTSP[build_rtsp_params
Four-tier config priority]
BuildLLHLS[build_ll_hls_publish_output
MediaMTX publish args]
end
subgraph ConfigSources["Configuration Sources"]
CamJSON[cameras.json
rtsp_input, rtsp_output, ll_hls]
FFMap[ffmpeg_names_map.py
Key translation]
end
subgraph OutputParams["Generated Parameters"]
Input["-rtsp_transport tcp
-timeout 30000000
-analyzeduration 1000000"]
Output["-c:v libx264
-preset veryfast
-vf scale=640:480
-r 18"]
LLHLS_Out["-an -c:v libx264
-f flv rtmp://nvr-packager:1935/path"]
end
CamJSON --> Builder
FFMap --> Builder
Builder --> BuildRTSP
Builder --> BuildLLHLS
BuildRTSP --> Input
BuildRTSP --> Output
BuildLLHLS --> LLHLS_Out
style Builder fill:#667eea,color:#fff
style CamJSON fill:#fff3cd
style FFMap fill:#fff3cd
FFmpeg Parameter Resolution
- stream_type filtering:
resolution_sub vs resolution_main selected based on request
- Key translation: Project keys (e.g.,
frame_rate_grid_mode) mapped to FFmpeg flags (-r)
- Codec handling:
"c:v": "transcode" resolves to -c:v libx264
- Scale filter:
resolution_sub: "320x240" becomes -vf scale=320:240
Level 5: Recording Architecture
Recording System: RecordingService supports continuous (24/7), motion-triggered, and manual recording with optional pre-buffer capture. Sources tap MediaMTX RTSP output (avoiding additional camera connections for single-connection cameras). Segment buffer maintains rolling 5-second segments for pre-motion capture. Recordings organized in per-camera hierarchical directories (CAMERA_NAME/YYYY/MM/DD/). Metadata stored via PostgREST API to PostgreSQL.
flowchart TB
subgraph Triggers["Recording Triggers"]
Manual[Manual Start
UI Button]
Motion[Motion Event
Baichuan/ONVIF/FFmpeg]
Continuous[Continuous Mode
Auto-start on boot]
end
subgraph Buffer["Segment Buffer (Pre-Buffer)"]
SegBuf[SegmentBufferManager
Rolling 5s segments]
TempSegs[/recordings/buffer/
seg_*.ts files/]
end
subgraph RecService["RecordingService"]
StartRec[start_motion_recording
start_manual_recording
start_continuous_recording]
PreBuf{Pre-Buffer
Enabled?}
Concat[FFmpeg Concat
prebuf + live = final]
SpawnFFmpeg[Spawn FFmpeg Process]
Track[Track in active_recordings]
end
subgraph Sources["Recording Sources"]
MediaMTX_Src[MediaMTX RTSP
rtsp://nvr-packager:8554/path]
end
subgraph Storage["Storage"]
FS[File System
CAMERA/YYYY/MM/DD/
*.mp4]
PG[(PostgreSQL
Recording Metadata)]
PostgREST[PostgREST API
:3001]
end
Manual --> StartRec
Motion --> StartRec
Continuous --> StartRec
SegBuf --> TempSegs
MediaMTX_Src --> SegBuf
StartRec --> PreBuf
PreBuf -->|Yes| Concat
PreBuf -->|No| SpawnFFmpeg
Concat --> SpawnFFmpeg
TempSegs --> Concat
MediaMTX_Src --> SpawnFFmpeg
SpawnFFmpeg --> Track
SpawnFFmpeg --> FS
Track --> PostgREST
PostgREST --> PG
style Manual fill:#17a2b8,color:#fff
style Motion fill:#fd7e14,color:#fff
style Continuous fill:#28a745,color:#fff
style RecService fill:#667eea,color:#fff
style FS fill:#6f42c1,color:#fff
style PG fill:#e1f5ff
style SegBuf fill:#20c997,color:#fff
style Concat fill:#dc3545,color:#fff
Motion Detection Methods
- Reolink Baichuan: Proprietary TCP push protocol (port 9000), instant event notification, lowest latency
- ONVIF PullPoint: Standardized subscription-based event polling, works with ONVIF-compliant cameras (Amcrest, SV3C)
- FFmpeg Scene Detection: Universal fallback using
select='gt(scene,X)' filter; auto-adjusts threshold for LL-HLS streams (0.01 vs 0.3)
Recording Directory Structure
- Path Format:
/recordings/{type}/{CAMERA_NAME}/YYYY/MM/DD/{serial}_{timestamp}.mp4
- Recording Types:
motion, continuous, manual, snapshots
- Pre-Buffer:
/recordings/buffer/{camera_id}/seg_*.ts (rolling segments, auto-cleanup)
- Camera Name Normalization: Uppercase, spaces to underscores, special chars removed, max 50 chars
Level 6: Frontend Architecture
flowchart TB
subgraph HTML["streams.html"]
Grid[Camera Grid Container]
Fullscreen[Fullscreen Overlay]
Settings[Settings Panel]
Modal[Recording Settings Modal]
end
subgraph Streaming["Streaming Modules"]
StreamJS[stream.js
Stream lifecycle + recovery]
WebRTCJS[webrtc-stream.js
WHEP RTCPeerConnection]
HLSJS[hls-stream.js
HLS.js wrapper]
MJPEGJS[mjpeg-stream.js
MJPEG img refresh]
HealthJS[health.js
Blank frame detection]
end
subgraph Controllers["Controllers"]
PTZJS[ptz-controller.js
ONVIF PTZ + latency learning]
RecJS[recording-controller.js
Recording UI + settings]
FullscreenJS[fullscreen-handler.js
Sub/main stream switching]
ConnMon[connection-monitor.js
Server disconnect detection]
end
subgraph Utils["Utilities"]
Logger[logger.js
Console logging]
Loading[loading-manager.js
Spinner states]
EufyAuth[eufy-auth-notifier.js
2FA prompt]
end
Grid --> StreamJS
StreamJS --> WebRTCJS
StreamJS --> HLSJS
StreamJS --> MJPEGJS
WebRTCJS --> HealthJS
HLSJS --> HealthJS
HealthJS -->|Escalating Recovery| StreamJS
Fullscreen --> FullscreenJS
FullscreenJS -->|Switch to main| StreamJS
Settings --> PTZJS
Modal --> RecJS
ConnMon -->|Server down| Grid
style StreamJS fill:#667eea,color:#fff
style WebRTCJS fill:#00b894,color:#fff
style HLSJS fill:#28a745,color:#fff
style HealthJS fill:#fd7e14,color:#fff
style ConnMon fill:#dc3545,color:#fff
style PTZJS fill:#17a2b8,color:#fff
Health Monitoring System (Nov 2025 Refactor)
- Blank Frame Detection: Canvas sampling detects frozen/black streams every 6 seconds
- Escalating Recovery: Tier 1 (attempts 1-3): standard refresh; Tier 2 (attempts 4+): nuclear stop+start cycle
- Nuclear Recovery: Forces UI stop → 3s wait → UI start to clear stuck backend FFmpeg state
- 60-Second Failure Window: Sliding window for escalation decisions; clears on success
- Configurable Max Attempts:
UI_HEALTH_MAX_ATTEMPTS in cameras.json (0 = infinite)
- Fullscreen Optimization: Health monitors detach for paused background streams, reattach on exit
Connection Monitor (Jan 2026)
- Health Polling:
/api/health every 10 seconds, 10 failures before redirect
- Graceful Shutdown: Server sends 503 when shutting down for immediate redirect
- Offline Modal: Shows spinner with auto-retry when server unreachable
- Cache-Busting:
location.reload(true) on recovery to bypass browser cache
PTZ Controller Features (Nov 2025 - Jan 2026)
- ONVIF Protocol: Standardized pan/tilt/zoom/preset control for Amcrest, Reolink, SV3C
- Adaptive Latency: PostgreSQL-backed per-camera latency learning (rolling avg of 10 samples)
- Race Condition Fix: Fire-and-forget move commands with acknowledgment tracking before stop
- Touch Device Support: Document-level touchend handlers, mobile-optimized controls
- Draggable Controls: PTZ panel draggable in fullscreen mode with position persistence
- GotoHomePosition: ONVIF command for PTZ stepper recalibration
Level 7: Mobile Streaming Architecture (Jan 2026)
Problem: iOS Safari has unreliable multipart MJPEG parsing and strict video decode limits (~4-8 simultaneous HLS streams). Android handles MJPEG well but also has resource constraints on mobile.
Solution: Device-specific streaming strategies with snapshot polling for iOS grid view.
flowchart TB
subgraph Detection["Device Detection (stream.js)"]
UA[User Agent Check] --> iOS{iOS?}
UA --> Android{Android?}
UA --> Desktop{Desktop?}
end
subgraph iOSFlow["iOS Flow"]
iOS -->|Yes| GridiOS[Grid View]
GridiOS --> Snapshot[SNAPSHOT Mode
1s polling via /api/snap/]
Snapshot --> ImgTag["<img> element
No video decode limit"]
GridiOS --> ExpModal[Expanded Modal]
ExpModal --> SnapModal[Still SNAPSHOT
Buttons visible]
ExpModal --> FSiOS[Fullscreen]
FSiOS --> HLSMain[HLS Main Stream
High quality + audio]
FSiOS --> PauseOthers[Pause other cameras]
end
subgraph AndroidFlow["Android Flow"]
Android -->|Yes| GridAndroid[Grid View]
GridAndroid --> MJPEG[MJPEG Mode
Multipart stream]
MJPEG --> ImgTagA["<img> element"]
GridAndroid --> FSAndroid[Fullscreen]
FSAndroid --> HLSMainA[HLS Main Stream]
end
subgraph DesktopFlow["Desktop Flow"]
Desktop -->|Yes| GridDesk[Grid View]
GridDesk --> HLSSub[HLS Sub Stream
or WebRTC]
HLSSub --> VideoTag["<video> element"]
GridDesk --> FSDesk[Fullscreen]
FSDesk --> HLSMainD[HLS Main Stream
High resolution]
end
style iOS fill:#ff9500,color:#fff
style Android fill:#3ddc84,color:#fff
style Desktop fill:#0078d4,color:#fff
style Snapshot fill:#ff6b6b,color:#fff
style HLSMain fill:#4ecdc4,color:#fff
Streaming Behavior by Device & Mode
| Device |
Grid View |
Expanded Modal |
Fullscreen |
| iOS (iPhone/iPad) |
Snapshots (1s polling) |
Snapshots |
HLS main stream |
| Android |
MJPEG |
MJPEG |
HLS main stream |
| Desktop |
HLS/WebRTC sub |
HLS/WebRTC sub |
HLS/WebRTC main |
iOS Snapshot Implementation Details
- Endpoint:
/api/snap/<camera_id> - Returns latest JPEG from frame buffer
- Frame Sources: Checks reolink_mjpeg_capture_service → mediaserver_mjpeg_service → unifi frame buffers
- Polling Interval: 1 second (configurable via SnapshotStreamManager)
- Element Swap:
<video> → <img class="stream-snapshot-img"> for grid view
- Fullscreen Switch: SNAPSHOT → HLS main stream with element restoration
- Resource Management: All other cameras' snapshot polling paused during fullscreen
sequenceDiagram
participant Browser as iOS Safari
participant StreamJS as stream.js
participant SnapMgr as SnapshotStreamManager
participant API as /api/snap/
participant Buffer as Frame Buffer
Note over Browser,Buffer: Page Load - Grid View
Browser->>StreamJS: Load page
StreamJS->>StreamJS: isIOSDevice() = true
StreamJS->>StreamJS: Swap video→img elements
StreamJS->>SnapMgr: startStream(cameraId, imgEl, 1000ms)
loop Every 1 second
SnapMgr->>API: GET /api/snap/{cameraId}?t=timestamp
API->>Buffer: Get latest frame
Buffer-->>API: JPEG data
API-->>SnapMgr: image/jpeg
SnapMgr->>Browser: Update img.src
end
Note over Browser,Buffer: Fullscreen Entered
Browser->>StreamJS: Click fullscreen button
StreamJS->>SnapMgr: stopStream(cameraId)
StreamJS->>StreamJS: Restore video element
StreamJS->>StreamJS: hlsManager.startStream(main)
StreamJS->>SnapMgr: Pause all other cameras
Note over Browser,Buffer: Fullscreen Exited
Browser->>StreamJS: Exit fullscreen
StreamJS->>StreamJS: Stop HLS
StreamJS->>StreamJS: Create img element
StreamJS->>SnapMgr: Resume all cameras
Mobile UI Behavior
- Grid Mode: All control buttons hidden for clean thumbnail view
- Expanded Modal: Tap camera to show larger view with all buttons visible (fullscreen, audio, PTZ toggle, controls toggle)
- Fullscreen Mode: Full device screen with HLS video and all controls
- PTZ/Stream Controls: Hidden in grid mode via CSS
.stream-item:not(.css-fullscreen):not(.expanded)
- Pagination Disabled: iOS previously limited to 6 cameras/page for video decode limits; disabled since snapshots use
<img> tags
iOS WebRTC Support (DTLS Enabled - Jan 18, 2026)
- DTLS-SRTP Enabled: MediaMTX configured with
webrtcEncryption: yes for iOS Safari WebRTC support (~200ms latency)
- Configuration:
cameras.json has webrtc_global_settings.enable_dtls: true
- nginx Proxy: WHEP endpoint proxied via HTTPS (
/webrtc/ → https://nvr-packager:8889 with SSL verify off)
- Grid Mode Options: iOS defaults to snapshot polling (1fps), but experimental "Force WebRTC Grid" setting available
- Fullscreen: iOS now uses WebRTC (~200ms) instead of HLS fallback (~2-4s)
Mobile-Specific Limitations
- iOS Safari MJPEG: Multipart parsing unreliable; snapshot polling is more stable
- iOS Video Decode Limits: Safari limits concurrent video decodes (~4-8 streams); Force WebRTC Grid is experimental
- Audio Autoplay: Both iOS and Android block autoplay with audio; video starts muted by default
- Background Tab: iOS aggressively suspends background tabs; streams may need refresh on return
Level 8: Audio Architecture (Jan 2026)
Playback Volume Control
Per-camera volume control with browser persistence:
- UI: Click speaker icon → popup with slider (0-100%) and mute toggle
- Storage: localStorage
cameraAudioPreferences[cameraId] = { volume: 75, muted: false }
- Implementation:
static/js/streaming/stream.js - event handlers for slider input/change
- Styling:
static/css/components/stream-volume-popup.css
Two-Way Audio (Talkback)
Browser microphone → camera speaker communication:
- Eufy (Working): Browser → WebSocket → Flask →
TalkbackTranscoder (FFmpeg: 16kHz mono AAC ADTS 20kbps) → Eufy P2P Bridge → Camera
- ONVIF (via go2rtc): Browser → WebSocket → Flask → go2rtc API → ONVIF AudioBackChannel → Camera
- go2rtc Service:
nvr-go2rtc container handles ONVIF backchannel while MediaMTX serves video (avoids dual-connection issues)
go2rtc Configuration
File: config/go2rtc.yaml
- API: Port 1984 - Web UI and REST API
- RTSP: Port 8555 - Internal RTSP server
- WebRTC: Port 8556 - WebRTC signaling
- Streams: ONVIF-only connections for cameras with AudioBackChannel capability
- Note: Credentials injected via environment variables from AWS Secrets Manager
| Camera Type |
Protocol |
Status |
Configuration |
| Eufy |
Eufy P2P Bridge |
Working |
two_way_audio.protocol: "eufy_p2p" |
| SV3C |
ONVIF AudioBackChannel |
Configured |
two_way_audio.onvif.go2rtc_stream |
| Amcrest |
ONVIF AudioBackChannel |
Configured |
two_way_audio.onvif.go2rtc_stream |
| Reolink |
Baichuan |
Future |
Not yet implemented |
| UniFi |
Protect API |
Future |
Requires Protect API integration |
Level 9: Power Management (Jan 2026)
Camera Power Control
Remote power cycle capability for cameras connected to smart outlets or PoE switches:
- Hubitat Integration: Cameras connected to Hubitat-controlled smart outlets (Zigbee/Z-Wave)
- UniFi PoE Integration: Cameras powered via UniFi switch PoE ports
- Manual Trigger: Power button in stream controls (immediate cycle)
- Auto Power-Cycle: Optional feature when camera goes OFFLINE (disabled by default)
Power Cycle Safety (CRITICAL)
Auto power-cycling requires explicit opt-in per camera:
- Config:
power_cycle_on_failure.enabled: true (default: false)
- Cooldown:
power_cycle_on_failure.cooldown_hours: 24 (configurable)
- UI Settings: Power Management section in camera settings modal
- Manual Override: Manual power button always works regardless of opt-in setting
| Power Source |
Service |
Configuration |
Notes |
| Hubitat Smart Outlet |
services/power/hubitat_power_service.py |
power_supply: "hubitat"
power_supply_device_id: "123" |
Uses Hubitat Maker API |
| UniFi PoE Switch |
services/power/unifi_poe_service.py |
power_supply: "unifi_poe"
power_supply_port: 5 |
Uses UniFi Network API |
Architecture Summary
NVR Engineering Architecture Overview
The Unified NVR system represents a sophisticated multi-vendor video surveillance solution with modular architecture:
- Strategy Pattern: Vendor-specific handlers (Eufy, Reolink, UniFi, Amcrest, SV3C) enable clean separation of camera protocols while sharing common streaming infrastructure
- Dual-Output Streaming: Single FFmpeg process per camera produces sub-stream (grid view) and main-stream (fullscreen) to MediaMTX
- MediaMTX Hub: Central stream multiplexer providing WebRTC (~200ms), LL-HLS (~2s), and RTSP re-export for motion detection and recording
- Protocol Bridge: Neolink v0.6.2 translates Baichuan protocol for cameras with broken RTSP
- Thread-Safe Design: RLock-protected shared state with per-camera restart locks; race condition fixes for status queries
- Credential Security: AWS Secrets Manager integration with vendor-specific credential providers
- Escalating Health Recovery: Two-tier frontend recovery (standard refresh → nuclear stop+start) with 60-second failure window
- Motion Detection: Three methods - Reolink Baichuan (push), ONVIF PullPoint (poll), FFmpeg scene detection (universal)
- Pre-Buffer Recording: Rolling 5-second segment buffer for motion recordings; FFmpeg concat for final output
- ONVIF PTZ: Standardized control with adaptive latency learning stored in PostgreSQL
- Two-Way Audio: Eufy talkback via P2P bridge; ONVIF backchannel via go2rtc for SV3C/Amcrest
- Playback Volume: Per-camera volume slider with localStorage persistence
- Power Management: Hubitat smart outlets and UniFi PoE integration for remote power-cycle with safety opt-in
- Config Sanitization: Pre-commit hook auto-generates example configs with fake IPs/serials/credentials
- Docker-First: Full containerization with docker-compose; timezone sync for accurate timestamps
Known Limitations & Vendor Quirks
- WebRTC LAN-Only: Current WebRTC config uses direct ICE without STUN/TURN (add STUN server for remote access)
- Dual-Output Status: Single FFmpeg with dual outputs (sub/main) implemented Jan 2026; previous composite key approach reverted Nov 2025
- Single-Connection Cameras: SV3C, Eufy, some Reolink support only one RTSP connection; must use MediaMTX RTSP re-export for motion/recording
- Eufy P2P Dependency: Eufy cameras require eufy-security-ws bridge; frequent WiFi disconnects observed
- SV3C PTZ Quirks: Digital zoom only (no optical motor); ONVIF GetStatus always returns (0,0)
- Neolink Buffer Overflow: v0.6.3.rc.x has regression; must use v0.6.2
- Scene Detection Thresholds: LL-HLS streams need 0.01 threshold vs 0.3 for direct RTSP due to
scenecut=0
- Hardware Decoding: Browser hardware video decoding is heuristic-based, cannot be forced programmatically
File Structure Reference
- Entry Point:
app.py - Flask application and service initialization
- Stream Orchestration:
streaming/stream_manager.py
- Vendor Handlers:
streaming/handlers/{eufy,reolink,unifi,amcrest,sv3c}_stream_handler.py
- Credential Providers:
services/credentials/{eufy,reolink,unifi,amcrest,sv3c}_credential_provider.py
- FFmpeg Config:
streaming/ffmpeg_params.py
- Camera Config:
config/cameras.json
- Recording Config:
config/recording_settings.json
- go2rtc Config:
config/go2rtc.yaml
- Talkback Transcoder:
services/talkback_transcoder.py
- Recording Service:
services/recording/recording_service.py
- Segment Buffer:
services/recording/segment_buffer.py
- Storage Manager:
services/recording/storage_manager.py
- Motion Detection:
services/motion/ffmpeg_motion_detector.py, services/onvif/onvif_event_listener.py
- ONVIF PTZ:
services/onvif/onvif_ptz_handler.py
- Power Services:
services/power/hubitat_power_service.py, services/power/unifi_poe_service.py
- Power Controller:
static/js/controllers/power-controller.js
- Neolink Config:
config/neolink.toml
- Frontend Streaming:
static/js/streaming/*.js (includes snapshot-stream.js for iOS)
- Frontend Controllers:
static/js/controllers/*.js
- Templates:
templates/streams.html
- Database Schema:
psql/init-db.sql
Recent Changelog (Nov 2025 - Jan 2026)
- Nov 2-3, 2025: ONVIF PTZ implementation, UI Health Monitor with escalating recovery
- Nov 8, 2025: Infinite retry fix, nuclear recovery for stuck backend state
- Nov 15, 2025: Recording UI modal, manual recording controls
- Nov 22-24, 2025: Composite key refactor (reverted); dual-stream investigation
- Dec 31, 2025: SV3C camera integration, per-camera recording directories, motion detection fix
- Jan 1, 2026: MediaMTX centralization, pre-buffer recording, FFmpeg scene detection threshold fix, PTZ latency learning
- Jan 2, 2026: Neolink Baichuan bridge, NEOLINK → MediaMTX LL-HLS integration, dual-output FFmpeg, connection monitor
- Jan 3-4, 2026: CameraStateTracker service (MediaMTX API polling, exponential backoff), StreamWatchdog unified service (replaces per-stream watchdogs), UI recovery detection (auto-refresh on backend recovery), WebRTC streaming via WHEP (~200ms latency)
- Jan 18, 2026: iOS snapshot polling (replaces unreliable MJPEG), expanded modal mode (tap-to-expand), Force MJPEG desktop setting, UI health re-enabled with tuned thresholds, PTZ/controls visibility fixes for grid mode, DTLS enabled for iOS WebRTC support (~200ms latency), Force WebRTC Grid experimental setting for iOS
- Jan 19, 2026: WebRTC audio restoration - AAC→Opus transcoding for all cameras (WebRTC requires Opus, MediaMTX skips AAC), fixed ffmpeg_params.py main stream hardcoded -c:a copy, user-stopped stream tracking (localStorage prevents watchdog auto-restart), .gitignore fix for config/cameras.json tracking
- Jan 22-25, 2026: PTZ reversal settings for upside-down cameras, PTZ double-action bug fix, PTZ Home/Recalibration buttons, Eufy bridge direction mapping fix, timeline playback timestamp fix, Hubitat power service (auto power-cycle), UniFi POE power service, device picker modal, power button in stream controls, WebRTC fullscreen quality recovery
- Jan 26, 2026: Power cycle safety (opt-in requirement + 24h configurable cooldown), FFmpeg params null/none handling fix, SV3C RTSP stability (15s timeouts, reconnect options), playback volume slider with localStorage persistence, SV3C direct HTTP snapshots for /api/snap, config sanitization pre-commit hook (auto-generates example files)