Immersive 360° Livestreaming

A freely-accessible toolchain for live streaming music events with Higher-Order Ambisonics audio and 4K 360° video, deployed for the 70th anniversary concert of the Faculty of Electronics, Telecommunications and Informatics at Gdańsk University of Technology (March 2022). The capture chain used a Zylia ZM-1 ambisonic microphone and an Insta360 Pro 2 camera, streaming 3rd-order Ambisonics (16 channels) with dynamic head-rotation-linked binaural rendering synchronised to 360° video in the browser. A dedicated event page with the recording remains accessible.

In 2026 the full stack was containerised into a single Docker Compose deployment, hosted at stream.bmroz.eu, and used as a research platform to characterise the audio-video synchronisation behaviour of MPEG-DASH under varying segment durations. The containerised version replaces the original manual install process with reproducible, server-ready infrastructure; the rebuild also surfaced and resolved several non-obvious streaming and A/V issues during the 2026 containerisation, documented below.

Engineering findings

Rebuilding the stack as a reproducible VP9/Opus MPEG-DASH pipeline surfaced several non-obvious failure modes, each now fixed and documented in the repository:

• Live-edge gap jump. On load the player skipped ~10 s forward, because the DASH muxer advertised only a ~3 s presentation delay while the player tried to start at the live edge. Raising the advertised presentation delay (patched into the muxer) and pinning the player’s live delay to 30 s removed the jump.
• Full-range VP9 breaks browser playback. 4K VP9 encoded in full colour range decodes correctly in isolation but fails inside the browser’s Media Source Extensions pipeline on GPU-accelerated decoders, looping silently – and passes automated headless (software-decode) testing, so it only shows up on real machines. Emitting limited-range video, matching the live transcode, is the fix.
• Keyframe size ceiling at the ingest. 4K keyframes on the H.264 contribution leg (~1.2 MB) exceeded nginx-rtmp’s 1 MB default message limit, crashing and restarting the ingest roughly once a minute. Raising the limit stabilised long sessions.
• A/V desync from an edit list browsers disagree on. The live path played video 1.6–2.0 s ahead of audio on Chromium-based browsers only, with the offset changing on every encoder restart. Root cause: a subscriber joining a live nginx-rtmp stream receives audio immediately but video only from the next keyframe, so the transcoder records the start skew as a leading empty edit (elst, media_time = -1) in the DASH init segment. Firefox applies that edit through Media Source Extensions and stays in sync; Chromium and Safari drop it (while both apply it in plain video.src playback), a genuine cross-browser interoperability gap. Fixed at both ends: the player now parses the edit list and schedules audio itself, and two nginx-rtmp directives (wait_key, wait_video) align the track starts at the source so no edit list is written at all – verified by a restart drill (20/20 runs clean, tracks aligned within 3 ms). Published as a minimal cross-browser repro with live demo, a Chromium interoperability issue, and an upstream fix to Envelop Earshot.

A/V synchronisation measurements. Practical experience with the Insta360 Pro 2 revealed temporal instability in its internal clock, causing drift in longer recordings. The 2026 Docker deployment introduced a combined-MPD DASH path (audio and video in a single manifest) and was used to measure the effect of segment duration on A/V offset, buffer depth, and bitrate. Measurements were taken with a headless-Chromium harness over GOP-matched re-encodings of the same 4K master (one keyframe interval per segment duration), three runs each. The 0 ms offset follows from the combined-MPD design: audio and video share a single media-element clock under MSE, so they cannot drift relative to each other regardless of segment length. A small constant audio delay (tens of milliseconds, in the perceptually forgiving direction) remains from the binaural HRIR convolution in the browser and is not captured in the table.

Segment duration	A/V offset	Buffer depth	Bitrate
0.5 s	0 ms	0.4–0.6 s	~43 Mbps
1 s	0 ms	3–5 s	~11 Mbps
2 s	0 ms	9–12 s	~7.5 Mbps
4 s	0 ms	15–20 s	~5.8 Mbps

Segment duration vs. effective bitrate and playback buffer depth. A/V sync outcome is identical across all variants; the tradeoff is buffer size and bitrate overhead.

Segment duration had no effect on A/V sync in the combined-MPD path across all tested values. The shortest segments are not viable at 4K: a keyframe every half-second pushes the VP9 encoder roughly 6× past its bitrate target and leaves the playback buffer under one second – the only setting that stalled in testing. 2-second segments were chosen as the operating point: buffer depth deep enough for stable playback, bitrate efficient enough for server hosting, and latency acceptable for a recorded-concert use case.

The original drift problem also motivated a separate research thread on automatic synchronisation of ambisonic and 360° video streams from separate devices. The result is AudioSync, a VST3 plugin (C++/JUCE) that decodes SMPTE LTC timecode from audio tracks, computes per-track delays, and applies them in real time. It runs as a Master/Slave pair – one instance on the reference LTC track, one per additional track – communicating via shared memory. When LTC quality degrades, it falls back transparently to audio-based delay tracking using normalised cross-correlation on novelty curves. Up to 8 slave instances per group are supported. Initial LTC decoding work by T. Dalhou (B.Sc.Eng); plugin developed by F. Lewiński (M.Sc.Eng thesis).

AudioSync plugin UI -- Master instance displaying decoded LTC timecode, Slave instance showing computed delay.

Directions

• Standalone-headset playback over NDI. An optional NDI output is being designed to drive untethered VR headsets directly: a Meta Quest 3 running Twinkle VR Player can receive the 16-channel ambisonic stream over the local network and render head-tracked binaural audio in-headset, taking the browser out of the in-venue path. This suits teaching-lab and exhibition use, where several headsets receive one stream wirelessly.
• Edge deployment. The stack already builds for ARM64; validating a Raspberry Pi 5 as a low-cost self-contained streaming node is the next deployment target.
• Capture-side synchronisation. Folding the AudioSync timecode plugin into the capture chain would align ambisonic audio and 360° video from separate devices automatically, addressing the Insta360 clock drift at the source rather than in post.

The repository contains the full Docker Compose stack (rtmp-ingest, Earshot, DASH muxer, hoast-player), Nginx config, and measurement and packaging scripts. Runs on AMD64 and ARM64; validated on WSL2 Ubuntu and Ubuntu Server 22.04. Raspberry Pi 5 (ARM64) is a planned target. See the README for quick-start and deployment notes for Azure Container Apps.

• GitLab: git.pg.edu.pl/p829296/hoa-360-stream
• GitHub: github.com/mormegil6/hoa-360-stream

Compose files and scripts: Apache 2.0. Bundled components (HOAST360, Earshot, nginx-rtmp) retain their own licenses — see the README for the full table.

@inproceedings{mroz_commonly-accessible_2023,
  address = {Huddersfield, UK},
  title = {A commonly-accessible toolchain for live streaming music events with higher-order ambisonic audio and 4k 360 vision},
  url = {http://www.aes.org/e-lib/browse.cfm?elib=22204},
  booktitle = {2023 {AES} {International} {Conference} on {Spatial} and {Immersive} {Audio}},
  publisher = {Audio Engineering Society},
  author = {Mróz, Bartłomiej and Odya, Piotr and Danowski, Przemysław and Kabaciński, Marek},
  month = aug,
  year = {2023}
}