Skip Navigation
Self Hosted - Self-hosting your services. @lemmy.ml tarneo @lemmy.ml

Automatic updates: a cautionary tale

tarneo.fr Renn.es Was Down for Eight Hours Because of an Automatic Docker Update - Tarneo's blog

Programming, Linux, self-hosting, ergo keyboards...

Tl;dr: Automatic updates on my home server caused 8 hours of downtime of all of renn.es' docker services including email and public websites

17
17 comments

  • I don't want to seem rude, but in my opinion automated unattended updates on Gentoo is a bad idea.

    • That's what I learned :-)

      Edit: no saying that isn't rude

  • Blind automatic upgrades are a bad idea even for casual home users. You could run into a Linus Tech Tips "do as I say" scenario where it uninstalls half your system due to a dependency issue. Or it could accidentally uninstall part of your system that you don't notice.

    I'm not sure how stable Gentoo's default branch is but I know that daily upgrades on Arch Linux is close to suicide - you have a higher chance of installing a buggy package before it's fixed if you install every package version as it comes in.

    I'm surprised this strategy was approved for a public server - it's playing with a loaded revolver and it looks like you were finally shot.

    • I'm surprised this strategy was approved for a public server

      The goal was to avoid getting hacked on a server that could have many vulnerable services (there are more than 20 services on there). When I set this up I was basically freaked out by the fact I hadn't updated mastodon more than a week after the last critical vulnerability in it was found (arbitrary code execution on the server). The quantity of affected users, compared to the impact it would have if hacked, made me choose the option of auto-updates back then, even if I now agree it wasn't clever (and I ended up shooting myself I'm the foot). These days I just do updates semi-regularly and I am subscribed to mailing lists like oss-security to know there's a vulnerability as early as possible. Plus I am not the only person in charge anymore.

      • I'm not a real sysadmin so take it with a grain of salt, but in all reality this is probably why you would choose something like Debian for a server instead a bleeding-edge distro. Debian quickly backports security updates and fixes but otherwise keeps everything else stable and extremely well-tested, which pretty much 100% prevents serious bugs from reaching its Stable branch. You may still need to figure out an appropriate strategy for keeping your Mastodon container updated, but at least the rest of your system isn't at risk of causing catastrophic errors like this. Also, Debian Stable does allow you to auto-upgrade security patches only, if you still want that functionality.

    • [This comment has been deleted by an automated system]

      • Right, it was clearly LTT's fault for not reading, but automatic upgrades are the same thing as not reading. I've been using Linux for a very long time now, and I've seen Apt try to do some very stupid things before. Maybe it's better nowadays but I don't know if I'll ever shake the gut instinct to not allow Apt to do whatever it thinks is right.

  • What is the reason to shy away from Ubuntu? It is pretty solid in terms of automatic updating and rebooting. I used to be hardcore centos but I gave up after all of the hubbub around 8. I just need to server to update, reboot when necessary and keep running all my stuff so I don't have to touch it. In my old age, I don't care to tinker anymore - I just want my services running and I want reports given to me about health and status.

    Also, if you're concerned about privilege escalation, running a MAC is probably a good idea. SELinux saved my hide one a dozen years ago with a php bug where I did not sandbox an app properly. Thankfully, SELinux caught this and prevented anything bad from happening.

17 comments