On desktop Safari, adding AudioContext fixes the issue:
const AudioContext = window.AudioContext || window.webkitAudioContext;
const audioCtx = new AudioContext();
I found out by accident, so I have no idea why it works, but this removed the delay on my app.