How do you set up a server? Do you do any automation or do you just open up an SSH session and YOLO? Any containers? Is docker-compose enough for you or are you one of those unicorns who had no issues whatsoever with rootless Podman? Do you use any premade scripts or do you hand craft it all? What distro are you building on top of?
I'm currently in process of "building" my own server and I'm kinda wondering how "far" most people are going, where do y'all take any shortcuts, and what do you spend effort getting just right.
I'm a lazy piece of shit and containers give me cancer, so I just keep iptables aggressive and spin up whatever on an Ubuntu box that gets upgrades when I feel like wasting a weekend in my underwear.
I get paid to do shit with rigor; I don't have the time, energy, or help to make something classy for funsies. I'm also kind of a grumpy old man such that while I'll praise and embrace Python's addition of f-strings which make life better in myriad ways, I eschew the worse laziness of the all the containers attitude that we see for deployment.
Maybe a day shall come when containers are truly less of a headache than just thinking shit through the first time, and I'll begrudgingly adapt and grow, but that day ain't today.
After many years of tinkering, I finally gave in and converted my whole stack over to UnRAID a few years ago. You know what? It's awesome, and I wish I had done it sooner. It automates so many of the more tedious aspects of home server management. I work in IT, so for me it's less about scratching the itch and more about having competent hosting of services I consider mission-critical. UnRAID lets me do that easily and effectively.
Most of my fun stuff is controlled through Docker and VMs via UnRAID, and I have a secondary external Linux server which handles some tasks I don't want to saddle UnRAID with (PFSense, Adblocking, etc). The UnRAID server itself has 128GB RAM and dual XEON CPUs, so plenty of go for my home projects. I'm at 12TB right now but I was just on Amazon eyeing some 8TB drives...
Proxmox, then create LXC for everything (moslty debian and a bit of alpine), no automation, full yolo, if it break I have backup (problems are for future me eh)
I used to do the same, but nowadays I just run everything in docker, within a single lxc container on proxmox. Having to setup mono or similar every time I wanted to setup a game server or even jellyfin was annoying.
That's usually enough for me. I have all my docker compose files in their respective containers in the home directory like ~/red-discordbot/docker-compose.yml.
The only headache I've dealt with are permissions because I have to run docker as root and it makes a lot of messy permissions in the home directories. I've been trying rootless docker earlier and it's been great so far.
Right now, I just flash ubuntu server to whatever computer it is, ssh and yolo lmao. no containers, no managers, just me, my servers, and a vpn, raw dogging the internet lmao. The box is running a nas, jellyfin, lemmy, and a print server; the laptop a minecraft server, and the pi is running a pihole, and a website that controls gpio that controls the lights. In the pictured setup i dont have access to the apartment complex's router, so i vpn through a openvpn server i setup in a digitalocean server.
i didnt even know what a container was until i setup the lemmy server, which i just used ansible for.
People like to diss running kubernetes on your personal servers, but once you have enough services running in your servers, managing them using docker compose is no longer cut it and kubernetes is the next logical step to go. Tools such as k9s makes navigating as kubernetes cluster a breeze.
About two years ago my set up had gotten out of control, as it will. Closet full of crap all running vms all poorly managed by chef. Different linux flavors everywhere.
Now its one big physical ubuntu box. Everything gets its own ubuntu VM. These days if I can't do it in shell scripts and xml I'm annoyed. Anything fancier than that i'd better be getting paid. I document in markdown as i go and rsync the important stuff from each VM to an external every night. Something goes wrong i just burn the vm, copy paste it back together in a new one from the mkdocs site. Then get on with my day.
I tend to prefer installing Debian on a server, but recently I did install Ubuntu's recent LTS on a box because I was running into an issue with the latest version of Debian. I didn't want to revert to an earlier version of Debian or spend a bunch of time figuring out the problem I was having with Python, so I opted to use Ubuntu, which worked.
Ubuntu is based on Debian, so it's like using the same operating system, as far as I'm concerned.
I use Unraid and their docker and VM integration, Works great for me as a home user with mixed drives. Most of the dockers i want already have unraid templates so require less configuration. Does everything i want and made it a bit easier for me with less configuration and the mixed drive support.
I've set up some godforsaken combination of docker, podman, nerdctl and bare metal at work for stuff I needed since they hired me. Every day I'm in constant dread something I made will go down, because I don't have enough time to figure out how I was supposed to do it right T.T
I have a git repository with all my compose files sorted neatly into directories, i.e. my "stack". Portainer allows adding stacks using a repository, so it's essentially one click deployment once the compose file is on a remote git server.
I run unraid on my server box with a few 8tb hdd and nvme for cache.
From there it is really easy to spin up Docker containers or stacks using compose, as well as VMs using your iso of choice.
For automation, I use Ansible to run one click setup machines; it is great for any cloud provider work too.
A bunch of old laptops running Ubuntu Server and docker-compose. Laptops are great; built in screen, keyboard, and UPS (battery), and more than capable of handling the kind of light workloads I run.
I have a single desktop running Proxmox with a TrueNAS VM for handling my data and a Debian VM for my Docker containers which accesses the NAS data through NFS.
I use a heterogeneous environment with some things hosted in various cloud providers and others locally. Often times, I can usually find the package I need - but if I can't, I usually go for Docker and docker-compose. This is often the case in Oracle Linux on OCI - where docker just makes things so much easier.
For my static stuff I just use Cloudflare Pages and forget about it.
On my homelab it is Arch Linux with my own set of scripts. I used to do VFIO gaming a lot (less now), so I had the host only be a hypervisor and used a separate Arch VM to host everything in a docker-compose stack. The VM makes my server operations a lot more tidy.
My RPI is using dietpi and is natively running the pihole software and a couple other things.
I know some folks swear by UnRaid and Proxmox, but I've always found those platforms limited me vs building things my way. Also borking my own system unintentionally on occasion is a thrilling opportunity to learn!
i do this, mixed with a little docker run inside of Makefiles. i store all my ansible playbooks in a repo, along with other repos for different projects and purposes. i store all of those in git repos that i clone via ssh from a server that acts as a NAS backed by zfs.
If doing a fresh server external, I'd go for debian as base(don't need to update it too often + stable)
For apps it's mostly docker-compose to set up portainer/nginx-proxy then from ther just manage the rest from portainer/nginx-proxy web-ui. ony log on the server for the occasional docker updates / pruning for space.
I see a lot of guys going the full kubernetes route and it's something I'm hoping to get into at some point but it seems like a lot to unpack for now.
I have a base Debian template with a few tweaks I like for all my machines. Debating setting up something like terraform but I just don't spin up VMs frequently enough to wan tto do that. I do have a few Ansible playbooks I run on a fresh server to really get it to where I want though.
Proxmox and shell scripts. I have everything automated from base install to updates.
All the VMs are Debian which install with a custom seed file. Each VM has a config script that will completely setup all users, ip tables, software, mounts, etc. SSL certs are updated on one machine with acme.sh and then pushed out as necessary.
One of these days I’ll get into docker but half the fun is making it all work. I need some time to properly set it up and learn how to configure it securely.
I'm all in on docker-compose + rootless podman. Definitely not no issues, but I've got the hang of the kinds of issues it presents at this point. They're mostly around SELinux and networking, though generally the networking only gets problematic on exotic compose setups - jitsi was a huge pain for me.
Raw server with SSH and an immutable OS too. I'm using fedora IOT for my homeserver, and apart from some initial issues with GPU drivers because of layering issues (now working) that's been basically flawless.
I was on OpenSuse MicroOS, but I had huge problems with BTRFS and decided to give it up in favour of EXT4 + XFS. That necessitated moving distro, because MicroOS uses BTRFS snapshots as the basis for its auto-updating/green/blue system. Fedora IOT uses rpm-ostree instead, and works on any filesystem.
For a while I tried to run k8s (k3s mostly), then I did run nomad for a while. Now I am just running docker compose on Ubuntu (still have one box running Proxmox, but that will be decommissioned eventually, and mostly just runs one VM running Ubuntu).
I am building a few things to solve specific problems I have with this:
Some basic ansible scripts to set up ssh, users, basic packages, etc
Docker label-based service discovery/announcement that traefik can consume. (currently working!)
Deployment: getting the compose files, config files, and docker images to the right machine and getting them running. (in progress)
At some point I will probably get around to automating deployment of the rest of the above via Ansible when it is more stable.
Most of my server hardware is oriented toward having a bunch of disks plugged into them (I am 100% guilty of being a data hoarder), and I am running gluster to glue that all together, so that is something I install onto the servers to share their physical disks and/or mount the logical disks.
I forgot to mention I also have a samba share running on it and it's sooooooo sloooooow. I might need to reflash the thing just to cover my bases but it's unusable for large or many files.
Fedora-server with Podman and Quadlet on btrfs drives. Although I must admit I often use rootful mode in Podman as it works better with Containers made for Docker. Ah and you might want to turn off SElinux in the beginning as it can get frustrating fast.
Sorry for the delayed response. I do also see the inconsistency between looking at your post directly vs lemmy.ml. I have noticed, however, that every now and the lemmy.ml throws a bad gateway error, which would imply it's getting overloaded again. That might create situations where lemmy.ml has all comments marked as federated, while some of them were actually dropped mid transit. Same applies to lemmy.one.
I don't know of any workarounds for that, unfortunately. Feels a lot like a bug.
In regards to subscriptions - you're right, the pending state does seem to actually impact federation. Some of my subscriptions to beehaw have been pending since day one but I can see the content just fine. I've written this off for another bug in the software.
I had a look at your profile - I can definitely see the posts you've created as well as the comments. I've noticed some UI bits fail to get refreshed - things like notification status, etc. I found forcing a page refresh helps with that.
Usually Debian as base, then ansible to setup openssh for accessandd for the longest time, I just ran docker-compose straight on bare metal, these days though, I prefer k3s.
Up until now I've been using docker and mostly manually configuring by dumping docker compose files in /opt/whatever and calling it a day. Portainer is running, but I mainly use it for monitoring and occasionally admin tasks. Yesterday though, I spun up machine number 3 and I'm strongly considering setting up something better for provisioning/config. After it's all set up right, it's never been a big problem, but there are a couple of bits of initial with that are a bit of a pain (mostly hooking up wireguard, which I use as a tunnel for remote admin and off-site reverse proxying.
Salt is probably the strongest contender for me, though that's just because I've got a bit of experience with it.
For me it’s Ubuntu Server as the OS base, swag as reverse proxy and docker-compose for the services. So mostly SSH and yolo but with containers. I’d guess having something like Portainer running would probably be useful, but for me the terminal was enough.
As folder structure I just have a services directory with subfolders for each app/service.
NixOS instances running Nomad/Vault/Consul. Each service behind Traefik with LE certs. Containers can mount NFS shares from a separate NAS which optionally gets backed up to cloud blob storage.
I use SSH and some CLI commands for deployment but only because that’s faster than CICD. I’m only running ~’nomad run …’ for the most part
The goal was to be resilient to single node failures and align with a stack I might use for production ops work. It’s also nice to be able to remove/add nodes fairly easily without worrying about breaking any home automation or hosting.
A series of VPSes running AlmaLinux, I have a relatively big Ansible playbook to setup everything after the server goes online. The idea is that I can at any time scrape the server off, install an OS, put in all the persistent data (Docker volumes and /srv partition with all the heavy data), and run a playbok.
Docker Compose for services, last time I checked Podman, podman-compose didn't work properly, and learning a new orchestration tool would take an unjustifiable amount of time.
I try to avoid shell scripts as much as possible because they are hard to write in such a way so that they handle all possible scenarios, they are difficult to debug, and they can make a mess when not done properly. Premade scripts are usually the big offenders here, and they are I nice way to leave you without a single clue how the stuff they set up works.
I try to have most of the common parts setup with ansible. Over time, keep adding more and more. This is useful specially for things you may not do, or need, often and that is not as fresh on your mind how you set it up last time.
Any configuration management system would work; I find ansible is very approachable and fast to get productive with it.
Proxmox + mostly Debian + currently documenting my builds for future automation.
Lots of snapshots and clones/backups, for in case I want to roll back, or in case I want a head start in the future.
For example, I have a couple LAMP stack VMs backed up. If I need another LAMP VM, I clone (restore-as-unique) the backup in Proxmox, twiddle a few settings to make it actually unique, and go.
I don't do Docker or anything like it currently, and eventually I'm sure I'll learn, but having a crapload of VMs (true VM or LXC) suits me just fine for now. I will likely learn how to do my deployments with Ansible before learning Docker et al.
I have a bunch of different stuff, a dedicated server with Debian, 4 raspberry Pis + 1 micro computer that acts as a LB/Router/DHCP/DNS for the Pis.
In general I would say that my logic is as follows:
Every OS change is done through Ansible. This sometimes is a pain, you want to just apt install X and instead you might need to create a new playbook for it, but in the long term, it paid off multiple times. I do have some default playbook that does basic config (user, SSH key provisioning, some default packages) and hardening (SSH config, iptables).
I then try to keep the OS logic to a minimum, and do everything else as code. On my older dedicated server I run mostly docker-compose with Systemd + templated docker-compose files dropped by Ansible. The Pis instead run Kubernetes, with flux and all my applications are either directly managed via Flux or they have Helm in between. This means I can destroy a cluster, create another way, point it to my flux repository and I am pretty much back where I started.
Oh, I am there with you on that. I got used in my previous job, where everything was done with Ansible, but I still find myself copy pasting and changing most of the times. I actually like way more a declarative approach a-la-terraform.
Overall though there is a lot of community material, and once the playbooks are written it's quite good!
Web servers: unikernel (nanos) + fastly to backblaze
Mail servers: tailscale (for admin ssh), fail2ban, letsencrypt nginx, docker compose, mailu, heavy use of vps firewall
Raspberry pi "servers": k3s, managed with lens desktop pro, but honestly learning kubernetes isn't exactly enjoyable and feels painful compared to a shell script that launches dead simple API deployments
I used to do a lot of Ansible to manage a lot of servers but most of the time the thing I need to fix is manual, and a setup.md in my personal notes is all I need for "how do I make a new minio backup host" or whatever.
Probably the odd one here with Arch Linux + docker compose with still a lot of manual labor
updating it after maximum 4 weeks is enough, container more often
I've recently switched my entire self hosted infrastructure to NixOS, but only after a few years of evaluation, because it's quite a paradigm shift but well worth it imho.
Before that I used to stick to a solid base of Debian with some docker containers. There are still a few of those remaining that I have yet to migrate to my NixOS infra (namely mosquitto, gotify, nodered and portainer for managing them).
Cloud vps with debian. Then fix/update whatever weird or outdated image my vps provider gave me (over ssh). Then setup ssh certs instead of password. I use tmux a lot. Sometimes I have local scripts with scp to move some files around.
Usually I'm just hosting mosquitto, maybe apache2 webserver and WordPress or Flask. The latter two are only for development and get moved to other servers when done.
I don't usually use containers.
I'm better at hardware development than all this newfangled web stuff, so mostly just give me a command line without abstractions and I'm happy.
I'd like to use rootless podman, but since I include zerotier in my containers, they need access to the tunnel device and net_admin, so rootless isn't an option right now.
Podman-compose works for me. I'd like to learn how to use Ansible and Kubernetes, but right now, it's just my Lemmy VPS and my Raspberry Pi 4, so I don't have much need for automation at the moment. Maybe some day.
You can add net_admin to the user running podman, I have added it to the ambient capability mask before, which acts like an inherited override for everything the user runs.
Only ssh,nvim,htop, and screen.
Rest all are whatever is required.
I like to keep things minimal until i really need the server to do anything specific.
I resort to docker only if i need the application temporarily or the application setup is awkward/annoying.
I have a stupid overcomplicated networking script that never works. So every time i set up a new server I need to fix a myriad of weird issues I've never seen before. Usually I setup a server with a keyboard and mouse because SSH needs networking, if it's a cloud machine its the QEMU console or hundreds of reboots.