Ambisonic Box: Immersive 360° Livestreaming

srt://ingest.bmroz.eu:8890?streamid=<any-name>&latency=2000000&pkt_size=1128

The capture rig: Zylia ZM-1 (top) and Insta360 Pro 2 (below), WETI 70th-anniversary concert, March 2022.

A freely-accessible toolchain for live streaming music events with Higher-Order Ambisonics audio and 4K 360° video, deployed for the 70th anniversary concert of the Faculty of Electronics, Telecommunications and Informatics at Gdańsk University of Technology (March 2022). The capture chain used a Zylia ZM-1 ambisonic microphone and an Insta360 Pro 2 camera, streaming 3rd-order Ambisonics (16 channels) with dynamic head-rotation-linked binaural rendering synchronised to 360° video in the browser. A dedicated event page with the recording remains accessible.

In 2026 the full stack was containerised into a single Docker Compose deployment, hosted at stream.bmroz.eu, and used as a research platform to characterise the audio-video synchronisation behaviour of MPEG-DASH under varying segment durations. The containerised version replaces the original manual install process with reproducible, server-ready infrastructure; the rebuild also surfaced and resolved several non-obvious streaming and A/V issues during the 2026 containerisation, documented below.

Ambisonic order: why 16 channels and not 25

The stack carries 3rd-order Ambisonics, 16 channels, ACN/SN3D, end to end: the audio is never downmixed between the microphone and the listener’s headphones. The ceiling sits on the RTMP contribution leg rather than on delivery or rendering, because legacy RTMP/FLV cannot carry Opus and the AAC layouts it can carry are named ones, so 4 and 16 channels pass and 9 and 25 are refused. Delivery and the player are not the constraint: the on-demand path has been verified at 4th order end to end. The full argument, including what a sender must do to feed the stack at other orders, is in the repository’s ambisonic-order notes.

The player reads the ambisonic order out of the stream rather than being told it: the patched HOAST360 fork takes the channel count from the DASH manifest and re-checks it on the first decoded frame, so a 1st, 2nd, or 3rd-order feed plays without reconfiguration.

The end of that chain is a standalone headset with no native app involved. Below, the stack’s own capability probe runs inside Oculus Browser on a Meta Quest 3, reporting that both the 16-channel 3rd-order and the 25-channel 4th-order Opus streams decode, with the 360 test card playing behind it. This is the case the format is meant for, and it works on the device where it matters, in the browser.

Oculus Browser on a Meta Quest 3, playing the 360 test card. The overlay is the stack's capability probe: 16-channel (3rd-order) and 25-channel (4th-order) multichannel Opus both decode in a stock mobile browser.

Safari and iOS playback

It works. Third-order Ambisonics, 16 channels, plays on an iPhone and on macOS Safari from the same DASH stream every other client receives – video, spatial audio, head tracking, fullscreen – verified on an iPhone Xs (A12, iOS 18.7) and macOS Safari 27, on-demand and live. No iOS-specific variant, no server-side downmix, no extra lossy generation.

An on-demand clip on an iPhone Xs (left) and the live stream on macOS Safari 27 (right), both decoding the same 16-channel Opus.

The reason it needed work at all. Every native decode route Safari offers refuses this stream’s 16-channel Opus, on both macOS and iOS – measured directly rather than assumed:

This is measured as a structural gap in WebKit’s audio pipeline, not a regression with a fix in flight like the Chromium field trial above – nothing found while measuring it points at an upcoming change. A separate, permanent ceiling sits on the AAC side: Apple’s own encoder and CoreAudio’s channel-layout tags cap out at 8 channels platform-wide, so 3rd-order Ambisonics has no AAC route on Apple hardware regardless of what ffmpeg or WebKit do.

What plays it instead: the stream’s own segments, decoded in WebAssembly. opus-decoder, the same library whose minified bundle silently dropped multichannel support – see Engineering findings – built from source and fed the live DASH segments directly through one stateful decoder:

A weaker fallback – four parallel 4-channel AAC streams, decoded natively with no WASM – also works on macOS but loses its gapless path specifically on iOS (InternalAudioDecoderCocoa decoding failed on the stateful decode), which is why WASM is the primary route rather than a mild preference. Making it work also took two platform-specific fixes with no browser bug behind them: iOS classifies a muted video element as ambient sound unless navigator.audioSession.type is explicitly set to 'playback', and WebKit suspends a backgrounded video element within about two seconds unless it carries a decodable, unmuted audio track – solved with a silent 8 kbit/s AAC track riding alongside the real 16-channel audio, at a measured, constant cost of 0.37 s to first sound.

Two intermittent failures once looked separate; they turn out to share a trigger, and both are now root-caused. A media-session teardown: dash.js’s own StreamController detaches and re-attaches the ManagedMediaSource on certain transitions, and a closing source emits endstreaming on its way out, which latches dash.js’s own request gate off with nothing left to reopen it – zero segment requests forever, no error on the element or the bus, nothing logged. Deterministic and reproducible with no iPhone at all: element.removeAttribute('src'); element.load() at sourceopen reproduces it in seconds on any public DASH stream. hls.js hit the identical WebKit event years ago and already guards its handler against it; dash.js had none of those guards. This player’s own watchdog already recovered a foregrounded tab within about 30 seconds; filed as dash.js #5127 with the actual fix offered as #5128, verified in real Safari recovering full playback within tens of milliseconds of the detach, backgrounded tabs included.

The other is a WebKit audio-output fault, where every API reports a healthy session – context running, samples confirmed by an analyser node – yet nothing is audible. The mechanism is now identified in WebKit’s own source, and it is the same trigger as the dash.js teardown above: a GPU-process relaunch leaves WebKit’s cached audio-session category stale, and the fresh GPU-side session defaults to a category iOS hard-mutes at the Ring/Silent switch while rendering keeps running untouched underneath it. Confirmed directly on the iPhone Xs: flipping the physical switch to ring restores audio immediately, no reload; flipping back mutes it again. A page-side workaround follows from the same mechanism – cycling navigator.audioSession.type through a different value and back to 'playback' forces WebKit to resend the category and heals the session in place, no reload needed. Filed as WebKit 323104, triaged into WebKit’s Media component and CC’d to WebKit media engineers including Jer Noble. The full measurement set, including the gesture and fullscreen handling this needed, is in docs/IOS-SAFARI.md; the same probes run live at stream.bmroz.eu/iphone-test/.

Engineering findings

Rebuilding the stack as a reproducible MPEG-DASH pipeline surfaced several non-obvious failure modes, each now fixed and documented in the repository:

• Live-edge gap jump. On load the player skipped ~10 s forward, because the DASH muxer advertised only a ~3 s presentation delay while the player tried to start at the live edge. Raising the advertised presentation delay (patched into the muxer) and pinning the player’s live delay to 30 s removed the jump.
• Full-range VP9 breaks browser playback. 4K VP9 encoded in full colour range decodes correctly in isolation but fails inside the browser’s Media Source Extensions pipeline on GPU-accelerated decoders, looping silently – and passes automated headless (software-decode) testing, so it only shows up on real machines. Emitting limited-range video, matching the live transcode, is the fix.
• Keyframe size ceiling at the ingest. 4K keyframes on the H.264 contribution leg (~1.2 MB) exceeded nginx-rtmp’s 1 MB default message limit, crashing and restarting the ingest roughly once a minute. Raising the limit stabilised long sessions.
• A/V desync from an edit list browsers disagree on. The live path played video 1.6–2.0 s ahead of audio on Chromium-based browsers only, with the offset changing on every encoder restart. Root cause: a subscriber joining a live nginx-rtmp stream receives audio immediately but video only from the next keyframe, so the transcoder records the start skew as a leading empty edit (elst, media_time = -1) in the DASH init segment. Firefox applies that edit through Media Source Extensions and stays in sync; Chromium and Safari drop it (while both apply it in plain video.src playback), a genuine cross-browser interoperability gap. Fixed at both ends: the player now parses the edit list and schedules audio itself, and two nginx-rtmp directives (wait_key, wait_video) align the track starts at the source so no edit list is written at all – verified by a restart drill (20/20 runs clean, tracks aligned within 3 ms). Published as a minimal cross-browser repro you can run in your own browser, with a plain-language explanation of what it shows, interoperability issues filed against Chromium, Firefox and Safari/WebKit, escalated as a spec-clarification issue with the W3C Media Working Group (where the correct behaviour was confirmed to be implicitly mandated by the Web Platform Tests, which only Chromium fails), and an upstream fix to Envelop Earshot, now merged.
• A test card exposes a renderer flaw the video never had. The 360 image is textured onto a faceted sphere in the browser, and UV coordinates are exact only at the mesh vertices: between them the equirectangular-to-sphere mapping is interpolated linearly where the true mapping is trigonometric, so a straight line bows inside every quad. The delivered bytes are unaffected, which is why photographic 360 content hides the artefact and a line-based test card does not. The player shipped the upstream default of 32 by 32 segments, about 11 degrees per quad, where the bowing was plainly visible on grids and concentric circles; raising the mesh to 256 by 128, about 1.4 degrees per quad, removed it with no re-encode and no change to the DASH package. The error falls roughly quadratically with mesh density, so the chosen setting sits at the knee of the curve and finer meshes spend triangles on an error already well below one screen pixel. The fix is player-side and applies to any equirectangular stream through this player, not only to this stack.
• Passthrough silently hands segment duration to the contribution master. Under video passthrough (-c:v copy), Earshot’s own -g GOP setting is inert – the DASH muxer can only close a segment where the untouched contribution stream already carries a keyframe, so the master’s keyframe interval sets segment duration, not the encoder flag. A master re-encoded to fix colour range shipped without an explicit -g; x264’s 250-frame default produced 8.3 s video segments against the documented 2 s target for roughly three hours live, unnoticed by any existing check. Closed with a direct assertion on the largest keyframe gap in a rolling window, verified to fail at the regressed 8.342 s and pass at the corrected 2.002 s.
• The player’s own decoder was scrambling higher-order channels. Building a binaural render for the codec study below surfaced a six-year-old bug in HOAST360’s filter loader: concatBuffers() always read the first channel of each higher-order group as the impulse response for every channel in that group, so 10 of the 3rd order’s 12 higher-order channels decoded with the wrong filter – only channels 5 and 13, first in their groups, were correct. It degrades timbre rather than localisation (≤0.17 dB left/right shift), which is why nobody caught it by ear: individual channel errors ran up to 14.7 dB, and on real programme material the defect alone cost more measured binaural quality than the worst codec setting tested against it below, with zero compression involved. Fixed (one line: getChannelData(0) to getChannelData(j)) and shipped as rf19; sent upstream to the original project. The same update also fixes a display-scaling bug – device-pixel-ratio was never applied to the WebGL canvas, softening the image by half in each dimension on any high-density display (47.5 dB PSNR at the corrected 1600px canvas vs. 40.2 dB at the old ~800px one) – opt-in via ?beta for now, since doubling the pixel ratio roughly quadruples shader cost on top of the 16-channel binaural convolution already running.
• A Chromium field trial silently breaks multichannel Opus. DirectOpusAudioDecoding, a field trial rolling out in Chrome 151, fails every Opus decode above 2 channels in both decodeAudioData and MSE playback – exactly the codec and channel count this stack delivers. It is a regression, not a gap: multichannel Opus was deliberately enabled in Chromium M62, and this trial silently withdraws it. Forcing the same trial on in Brave and Edge fails identically, ruling out Chrome-specific packaging as the cause. Filed against Chromium (fixed: a corrected channel-mapping landed in OpusAudioDecoder, reviewed by a Chromium media engineer, and cherry-picked into the M153 stable branch, verified there by Google QA (2- and 16-channel playback both passing)), with a minimal reproduction and a live check anyone can run. The player itself now gates on a real decode probe rather than an advisory capability check that a browser can pass and still fail to decode, so it names the field trial and the flag that restores playback instead of hanging on a spinner – submitted upstream to HOAST360.
• A minified library bundle silently broke multichannel decode. eshaz/wasm-audio-decoders’ published Opus decoder mangles its own constructor option names when minified for the browser – channels, streamCount, channelMappingTable and others simply aren’t in the shipped bundle, so a 16-channel configuration is silently ignored and the decoder falls back to stereo with no error reported. Confirmed by counting symbol occurrences in the shipped bundle (absent) against src/ (present); the same packets decode correctly when built from source instead. Easy to miss: stereo is exactly what the ignored options default to anyway, and multichannel Opus is separately broken in several browsers right now, so a quietly-smaller channel count reads as the platform’s fault rather than the library’s. Filed upstream and fixed the same day (#129, released in opus-decoder@0.7.12); surfaced while investigating Safari/iOS playback.
• Shaka Packager writes a malformed Opus box. Packaging on-demand highlight clips through the shaka service (the stack’s offline VOD path) produced segments Chromium’s MSE flatly refused, while the same clips played fine as plain <video src>. Root cause: OpusSpecific::ReadWriteInternal copies the RFC 7845 OpusHead body into the ISO BMFF dOps box verbatim, but OpusHead stores PreSkip, InputSampleRate and OutputGain little-endian where dOps requires big-endian – a 48 kHz stream reports InputSampleRate as 2159738880, still a structurally valid box, just numerically wrong. General to any Opus content Shaka packages from WebM, not specific to this stack’s 16-channel audio – unfixed in every release including v3.9.3, unnoticed since a 2016 commit because the read path carries the mirror defect: an existing Opus MP4 comes out looking correct, since the two errors cancel exactly, and only WebM-to-MP4 packaging exposes it. VOD packaging here now runs a patched build until the fix lands upstream; patch written and verified against Shaka’s own unit tests and Chromium’s MSE parser, offered as a PR – #1627.

Not a screenshot: both panels are rendered by reproducing the player's own sampling against the test card, so the comparison is exact rather than eyeballed. Nadir view, the artefact's worst case, since converging meridians and sliver quads make the same angular error read far louder there than at the equator.

Where the server’s time actually goes is also worth recording, because it is the opposite of the usual assumption. Decoding the oversized contribution feed and rescaling it dominate the cost; the output encode, normally treated as the bottleneck, is the cheapest stage of the chain. The practical consequence is that having the contributor send the delivery resolution removes most of the server’s load, which is a larger lever than choosing a different output codec.

Signal chain. The contribution leg is H.264 with 16-channel AAC by protocol necessity; from the transcode onward the audio is 16-channel Opus, never downmixed.

A/V synchronisation measurements

Practical experience with the Insta360 Pro 2 revealed temporal instability in its internal clock, causing drift in longer recordings. The 2026 Docker deployment introduced a combined-MPD DASH path (audio and video in a single manifest) and was used to measure the effect of segment duration on A/V offset, buffer depth, and bitrate. Measurements were taken with a headless-Chromium harness over GOP-matched re-encodings of the same 4K master (one keyframe interval per segment duration), three runs each. The 0 ms offset follows from the combined-MPD design: audio and video share a single media-element clock under MSE, so they cannot drift relative to each other regardless of segment length. A small constant audio delay (tens of milliseconds, in the perceptually forgiving direction) remains from the binaural HRIR convolution in the browser and is not captured in the table. These are initial measurements, taken through an earlier build of the player; a re-run against the current build is queued.

Segment duration vs. effective bitrate and playback buffer depth, measured June 2026. A/V sync outcome is identical across all variants; the tradeoff is buffer size and bitrate overhead.

Segment duration had no effect on A/V sync in the combined-MPD path across all tested values. The shortest segments are not viable at 4K: a keyframe every half-second pushes the VP9 encoder roughly 6× past its bitrate target and leaves the playback buffer under one second – the only setting that stalled in testing. 2-second segments were chosen as the operating point: buffer depth deep enough for stable playback, bitrate efficient enough for server hosting, and latency acceptable for a recorded-concert use case.

Audio codec quality measurements

Two further studies, run the same way as the segment-duration measurement above: a stated question, a controlled method over five excerpts from the seven-year HOA recording corpus (solo piano, orchestra, dense ensemble, live concert, outdoor ambience), scored with AMBIQUAL against an uncompressed reference, and a decision that follows from the numbers rather than convention.

AAC contribution bitrate. The contribution leg’s 96 kbit/s-per-channel setting has always been a borrowed convention – AAC-LC’s classic ~64/channel, EBU Tech 3324’s ~75/channel for 5.1, YouTube’s 64/channel for ambisonics – never measured on higher-order ambisonic material itself. Each excerpt was AAC-encoded from 32 to 160 kbit/s/channel and scored twice: the contribution leg alone, and the full cascade a viewer actually receives (AAC decoded, then re-encoded to Opus, since the transcoder always re-encodes downstream).

AMBIQUAL quality vs. contribution AAC bitrate, AAC-alone and the full AAC-then-Opus cascade a viewer receives. Neither curve plateaus in the range measured; the cascade runs below the contribution leg at every rate, and the gap widens as bitrate rises.

From 64 to 128 kbit/s/channel, AAC-alone quality rises 1.22–1.4× faster than the cascade does over the same span: the downstream Opus re-encode acts as a partial ceiling on some of what extra contribution bitrate buys. It is a tax, not a wall – neither curve plateaus inside the range measured, and 96 is not a knee: nothing in the data marks it as the point where further contribution bitrate stops paying. What the data supports is the weaker statement that every extra kbit/s above it returns less to the viewer than it costs on the uplink, which is what the production setting rests on rather than a measured ceiling. Localisation degrades far more than quality at low bitrate, the same shape the Opus study below found: at 32 kbit/s/channel it sits at 26–29% of the quality score, rising to 82–87% by 160.

A second, independent metric – BAM-Q + GPSMᵠ, which scores binaural signals and is validated against listening tests, unlike AMBIQUAL’s ambisonic-domain scoring – was run on the same conditions decoded through HOAST360’s own binaural renderer, to check the result against a genuinely different model family:

BAM-Q agrees the cascade returns less, and puts it more strongly than AMBIQUAL does: monaural quality improves from 96 to 128 kbit/s/channel on all five excerpts for AAC alone, but the cascade gains less on four of them and goes negative on two. Where the two metrics diverge is more interesting than where they agree. BAM-Q finds the damage almost entirely monaural (binaural score stays 97–100 across every codec condition); AMBIQUAL finds the opposite from the ambisonic side – localisation runs 33–39% below quality on the AAC leg at 96 kbit/s/channel and 19–27% below at 128, wider still on the cascade (41–47% and 30–40%). The ambisonic model finds the spatial dimension the worst-hit where the binaural model finds it the least-hit; damage to the directional HOA channels does not have to survive binaural decoding as an interaural cue error, and can arrive as spectral colouring instead. Neither metric alone bounds the question, which is the argument for running both. Building this binaural cross-check is what surfaced the decoder bug above: scored on the same conditions, the uncoded reference decoded through the buggy filter alone came out worse on binaural quality (92–97) than the worst codec setting tested against it (97–99) – with no compression involved at all.

Same five excerpts, per-item rather than averaged: the decoder defect (red) sits outside the cluster of codec conditions on both binaural quality and interaural level error, for every excerpt.

Opus compression effort. Nothing in this stack sets libopus’s -compression_level, so every Opus encode runs at the slowest, most thorough setting, 10. Lowering it toward 5 is a standard way to buy back encoder CPU for a difference that’s usually hard to hear – worth checking here since it would apply to 16 channels on every route, including the Raspberry Pi 4 deployment target.

Quality change per excerpt at compression levels 5 and 0 against the default, level 10. Every excerpt but solo piano sits inside the band drawn at the largest same-direction excursion; piano is the only one that leaves it.

Left unset. There’s no meaningful CPU to reclaim – level 5 saves 0.9% of one core over level 10 (1.41 s vs. 1.68 s to encode 30 s of 16-channel audio), an order of magnitude less than the 59% of one core that removing the AAC contribution re-encode already saved elsewhere in this pipeline. Quality points the same way without needing to carry the argument alone: every excerpt but solo piano moved upward at level 5, which can’t be a genuine gain from spending less encoder effort, so that spread is the metric’s own noise on this material rather than a real one. Piano – the corpus’s single most common content type, 8 of its 23 sessions – is the only excerpt that moves the other way, by 2.0× that noise floor on quality and 3.2× on localisation, and localisation degrades further than quality on piano specifically too, by 2.4×, the same shape the AAC study above found. Sparse tonal material with long decay is exactly where encoder search effort should matter most, and it is the one case objectively measured to lose something by skipping it.

Capture-side synchronisation

Everything above corrects timing in the delivery path. The drift itself starts earlier, at capture, when the ambisonic recorder and the 360° camera each keep their own clock, so it also motivated a separate research thread on aligning the two streams automatically. The result is AudioSync, a VST3 plugin (C++/JUCE) that decodes SMPTE LTC timecode from audio tracks, computes per-track delays, and applies them in real time. It runs as a Master/Slave pair – one instance on the reference LTC track, one per additional track – communicating via shared memory. When LTC quality degrades, it falls back transparently to audio-based delay tracking using normalised cross-correlation on novelty curves. Up to 8 slave instances per group are supported. Initial LTC decoding work by T. Dalhou (B.Sc.Eng); plugin developed by F. Lewiński (M.Sc.Eng thesis).

AudioSync plugin UI. The Master instance displays decoded LTC timecode; the Slave instance shows the computed delay.

Folding the plugin into the capture chain itself is the remaining step, which would correct the drift at the source rather than in post.

Operational monitoring

The deployment reports its own state rather than being inspected by hand. A telemetry service samples the running stack and publishes both a dashboard and a small public status document, which is what the demo button on this page reads to decide whether the stream is live, idle, or unreachable. Segment freshness is tracked per representation, so a stalled video encoder cannot hide behind an audio track that is still being written.

The stack reporting on itself: stream state, per-service health, encoder headroom, and thermal history.

The transcoder carries its own monitor, which is the view used while a concert is actually running: it shows the RTMP connection and the sixteen ambisonic channels arriving discretely, alongside Earshot’s own dash.js preview player and its manifest/client settings – a separate embedded player for monitoring the transcode, distinct from the DASH manifest the site’s own HOAST360 player consumes.

The transcoder's monitor during a live concert: all 16 ambisonic channels and the equirectangular video, alongside Earshot's own dash.js preview player and its client-side manifest settings.

On-demand reference clips

Alongside the live path the stack serves a small set of reference clips, packaged offline into the same combined-MPD DASH shape the live path uses and played by the same browser player. One is a purpose-built 360 test card: a full equirectangular chart with labelled directions, grids, and a Siemens star, which is what made the sphere-tessellation artefact above visible in the first place. They double as a fixture for checking whether a given browser can decode multichannel Opus at all.

The clips play in the browser here and the masters are downloadable from the GitHub release, published as release assets rather than committed to the repository. If playback fails, the browser capability check reports whether multichannel Opus decodes at all on that machine.

The direction-cue clip: an equirectangular test card with the rendered sound-energy direction overlaid, used to confirm that audio and image agree about where a source is.

Directions

• Standalone-headset playback over NDI. An optional NDI output drives untethered VR headsets directly on the venue network, taking the browser out of the in-venue path: a Meta Quest 3 running Twinkle VR Player receives the 16-channel ambisonic stream over the local network and renders head-tracked binaural audio in-headset. Both questions this depended on have now been answered on hardware: the headset player does render third-order ambisonics, and all 16 channels survive the NDI hop and are recognised as third order at the receiver. What remains is a run on adequate networking, since the first test shared one radio between sender and headset and was bandwidth-bound rather than design-bound. This suits teaching-lab and exhibition use, where several headsets receive one stream wirelessly; remote viewers are unaffected and continue to be served over DASH to the browser player.
• Edge deployment. The stack already builds for ARM64. A Raspberry Pi 4, the slower of the two candidate boards, has been characterised on the committed default workload and is currently being brought up as a self-contained streaming node; a Pi 5 would buy portability and headroom rather than new evidence.

Repository

The repository contains the full Docker Compose stack (SRT gateway, RTMP ingest, Earshot transcoder, HOAST360 player, telemetry, and an offline packaging profile), Nginx configuration, and the measurement and packaging scripts. It runs on AMD64 and ARM64, and is validated on WSL2 Ubuntu and Ubuntu Server 22.04. See the README for quick-start and deployment notes. Links to both mirrors are in the resources list below.

Compose files and scripts: Apache 2.0. Bundled components (HOAST360, Earshot, nginx-rtmp) retain their own licenses — see the README for the full table.

@inproceedings{mroz_commonly-accessible_2023,
  address = {Huddersfield, UK},
  title = {A commonly-accessible toolchain for live streaming music events with higher-order ambisonic audio and 4k 360 vision},
  url = {http://www.aes.org/e-lib/browse.cfm?elib=22204},
  booktitle = {2023 {AES} {International} {Conference} on {Spatial} and {Immersive} {Audio}},
  publisher = {Audio Engineering Society},
  author = {Mróz, Bartłomiej and Odya, Piotr and Danowski, Przemysław and Kabaciński, Marek},
  month = aug,
  year = {2023}
}