If only the Linux desktop stopped getting offended when it’s not treated like a server and has to shut down. “Wait, you had audio settings that I was supposed to remember? Cool story bro…”
That can happen when there's a mismatch between the sample rate your audio device expects and what it receives. One way to fix this is to force the system to only allow one sample rate. I forget which files need to be edited for this, perhaps someone else will know, but you have a list of accepted and fallback sample rates, and you need to delete all except one.
I can't say that it will solve your specific issue, but it solved mine and I had the same symptoms.
Ultimately it’s all open source, you can make your own distro. If something doesn’t work, fork it and fix it yourself. That’s the beauty of Linux, with all that’s good and bad about it.