Immersive 360° Livestreaming

A freely-accessible toolchain for live streaming music events with Higher-Order Ambisonics audio and 4K 360° video, developed for the 70th anniversary concert of the Faculty of Electronics, Telecommunications and Computer Science at Gdańsk University of Technology (March 2022). The system combined a Zylia ZM-1 ambisonic microphone with an Insta360 Pro 2 camera, streaming 3rd-order Ambisonics (16 channels) with dynamic head-rotation-linked binauralisation synchronised to 360° video.

The toolchain uses open-source components: OBS Music Edition, Envelop Earshot for ambisonic RTMP streaming, and HOAST360 for the web player. A dedicated web page with the recording remains available. Developed with Dr P. Odya; collaborators include P. Danowski (Chopin University of Music) and M. Kabaciński (Zylia). Published at the AES International Conference on Spatial and Immersive Audio, Huddersfield, 2023.

Audio-video synchronisation research. Practical experience with the Insta360 Pro 2 revealed temporal instability in its internal clock, causing audio/video drift in longer recordings. This motivated a dedicated research thread on automatic synchronisation of ambisonic and 360° video streams captured by separate devices. The proposed hardware solution uses Tentacle Sync E devices — one injected into the camera’s mic-in port, one on the audio interface — establishing a common SMPTE timecode reference. A software plugin calculates the drift between streams and applies sample-accurate synchronisation, with a machine learning fallback for cases where timecode is absent or corrupted. Developed as student projects by T. Dalhou (B.Sc.Eng) and F. Lewiński (M.Sc.Eng thesis, ongoing).

Future work includes containerising the full livestream stack as an Azure-deployable service for reproducible immersive livestream infrastructure.

@inproceedings{mroz_commonly-accessible_2023,
  address = {Huddersfield, UK},
  title = {A commonly-accessible toolchain for live streaming music events with higher-order ambisonic audio and 4k 360 vision},
  url = {http://www.aes.org/e-lib/browse.cfm?elib=22204},
  booktitle = {2023 {AES} {International} {Conference} on {Spatial} and {Immersive} {Audio}},
  publisher = {Audio Engineering Society},
  author = {Mróz, Bartłomiej and Odya, Piotr and Danowski, Przemysław and Kabaciński, Marek},
  month = aug,
  year = {2023}
}