Immersive 360° Livestreaming
HOA + 4K 360° live streaming toolchain and audio-video synchronisation research.
Opens in a new tab. Use headphones. Chrome recommended.
A freely-accessible toolchain for live streaming music events with Higher-Order Ambisonics audio and 4K 360° video, deployed for the 70th anniversary concert of the Faculty of Electronics, Telecommunications and Informatics at Gdańsk University of Technology (March 2022). The capture chain used a Zylia ZM-1 ambisonic microphone and an Insta360 Pro 2 camera, streaming 3rd-order Ambisonics (16 channels) with dynamic head-rotation-linked binaural rendering synchronised to 360° video in the browser. A dedicated event page with the recording remains accessible.
In 2026 the full stack was containerised into a single Docker Compose deployment, hosted at stream.bmroz.eu, and used as a research platform to characterise the audio-video synchronisation behaviour of MPEG-DASH under varying segment durations. The containerised version replaces the original manual install process with reproducible, server-ready infrastructure; the rebuild also surfaced and resolved several non-obvious streaming and A/V issues during the 2026 containerisation, documented below.
Engineering findings
Rebuilding the stack as a reproducible VP9/Opus MPEG-DASH pipeline surfaced several non-obvious failure modes, each now fixed and documented in the repository:
- Live-edge gap jump. On load the player skipped ~10 s forward, because the DASH muxer advertised only a ~3 s presentation delay while the player tried to start at the live edge. Raising the advertised presentation delay (patched into the muxer) and pinning the player’s live delay to 30 s removed the jump.
- Full-range VP9 breaks browser playback. 4K VP9 encoded in full colour range decodes correctly in isolation but fails inside the browser’s Media Source Extensions pipeline on GPU-accelerated decoders, looping silently – and passes automated headless (software-decode) testing, so it only shows up on real machines. Emitting limited-range video, matching the live transcode, is the fix.
- Keyframe size ceiling at the ingest. 4K keyframes on the H.264 contribution leg (~1.2 MB) exceeded nginx-rtmp’s 1 MB default message limit, crashing and restarting the ingest roughly once a minute. Raising the limit stabilised long sessions.
- A/V desync from an edit list browsers disagree on. The live path played video 1.6–2.0 s ahead of audio on Chromium-based browsers only, with the offset changing on every encoder restart. Root cause: a subscriber joining a live nginx-rtmp stream receives audio immediately but video only from the next keyframe, so the transcoder records the start skew as a leading empty edit (
elst,media_time = -1) in the DASH init segment. Firefox applies that edit through Media Source Extensions and stays in sync; Chromium and Safari drop it (while both apply it in plainvideo.srcplayback), a genuine cross-browser interoperability gap. Fixed at both ends: the player now parses the edit list and schedules audio itself, and two nginx-rtmp directives (wait_key,wait_video) align the track starts at the source so no edit list is written at all – verified by a restart drill (20/20 runs clean, tracks aligned within 3 ms). Published as a minimal cross-browser repro with live demo, interoperability issues filed against Chromium, Firefox and Safari/WebKit, escalated as a spec-clarification issue with the W3C Media Working Group (where the correct behaviour was confirmed to be implicitly mandated by the Web Platform Tests, which only Chromium fails), and an upstream fix to Envelop Earshot, now merged.
A/V synchronisation measurements. Practical experience with the Insta360 Pro 2 revealed temporal instability in its internal clock, causing drift in longer recordings. The 2026 Docker deployment introduced a combined-MPD DASH path (audio and video in a single manifest) and was used to measure the effect of segment duration on A/V offset, buffer depth, and bitrate. Measurements were taken with a headless-Chromium harness over GOP-matched re-encodings of the same 4K master (one keyframe interval per segment duration), three runs each. The 0 ms offset follows from the combined-MPD design: audio and video share a single media-element clock under MSE, so they cannot drift relative to each other regardless of segment length. A small constant audio delay (tens of milliseconds, in the perceptually forgiving direction) remains from the binaural HRIR convolution in the browser and is not captured in the table.
| Segment duration | A/V offset | Buffer depth | Bitrate |
|---|---|---|---|
| 0.5 s | 0 ms | 0.4–0.6 s | ~43 Mbps |
| 1 s | 0 ms | 3–5 s | ~11 Mbps |
| 2 s | 0 ms | 9–12 s | ~7.5 Mbps |
| 4 s | 0 ms | 15–20 s | ~5.8 Mbps |
Segment duration vs. effective bitrate and playback buffer depth. A/V sync outcome is identical across all variants; the tradeoff is buffer size and bitrate overhead.
Segment duration had no effect on A/V sync in the combined-MPD path across all tested values. The shortest segments are not viable at 4K: a keyframe every half-second pushes the VP9 encoder roughly 6× past its bitrate target and leaves the playback buffer under one second – the only setting that stalled in testing. 2-second segments were chosen as the operating point: buffer depth deep enough for stable playback, bitrate efficient enough for server hosting, and latency acceptable for a recorded-concert use case.
The original drift problem also motivated a separate research thread on automatic synchronisation of ambisonic and 360° video streams from separate devices. The result is AudioSync, a VST3 plugin (C++/JUCE) that decodes SMPTE LTC timecode from audio tracks, computes per-track delays, and applies them in real time. It runs as a Master/Slave pair – one instance on the reference LTC track, one per additional track – communicating via shared memory. When LTC quality degrades, it falls back transparently to audio-based delay tracking using normalised cross-correlation on novelty curves. Up to 8 slave instances per group are supported. Initial LTC decoding work by T. Dalhou (B.Sc.Eng); plugin developed by F. Lewiński (M.Sc.Eng thesis).
AudioSync plugin UI -- Master instance displaying decoded LTC timecode, Slave instance showing computed delay.
Directions
- Standalone-headset playback over NDI. An optional NDI output is being designed to drive untethered VR headsets directly: a Meta Quest 3 running Twinkle VR Player can receive the 16-channel ambisonic stream over the local network and render head-tracked binaural audio in-headset, taking the browser out of the in-venue path. This suits teaching-lab and exhibition use, where several headsets receive one stream wirelessly.
- Edge deployment. The stack already builds for ARM64; validating a Raspberry Pi 5 as a low-cost self-contained streaming node is the next deployment target.
- Capture-side synchronisation. Folding the AudioSync timecode plugin into the capture chain would align ambisonic audio and 360° video from separate devices automatically, addressing the Insta360 clock drift at the source rather than in post.
The repository contains the full Docker Compose stack (rtmp-ingest, Earshot, DASH muxer, hoast-player), Nginx config, and measurement and packaging scripts. Runs on AMD64 and ARM64; validated on WSL2 Ubuntu and Ubuntu Server 22.04. Raspberry Pi 5 (ARM64) is a planned target. See the README for quick-start and deployment notes for Azure Container Apps.
Compose files and scripts: Apache 2.0. Bundled components (HOAST360, Earshot, nginx-rtmp) retain their own licenses — see the README for the full table.