Which kind of hardware (used computer/single-board computer/dedicated server machine/NAS/something else) and operating system do you recommend for a home server like this?
Which kind of hardware (used computer/single-board computer/dedicated server machine/NAS/something else) and operating system do you recommend for a home server like this?
Software: First and foremost: must be unix-like, must be able to communicate in both ways with an open-wrt router firmware distro and the devices on the local network (android, windows, linux, ipadOS systems). Must be very secure, like enterprise-grade or almost like that. Must be free and open-source. Must be somewhat fault-tolerant (so no Arch or gentoo or anything like that, i don't feel like recompiling the server's system daily). Must have these in base repos or easily installed in other methods: secure ssh client (like openSSH or such caliber), a software that enables me to securely control and see the gui of the server from android (Rustdesk? or such), (optionally i2p, dnscrypt, vpn clients, not needed if the router has them, just in case of emergency), ip camera management software, high-security intrusion-detection system, https server with css and js support (preferably command-line). Window manager: must support a very easy to use and lightweight tiling window manager (like i3wm) or if not, its installation and configuration needs to be possible and documented.
Hardware: affordable, x86_64 architecture, should be able to handle all of these at the same time, without freezing or overheating (i live in Hungary, so should be able to handle up to 40°C air temperature with stock fans or there should be space for more fans. liquid cooling is no-go).
I have considered these operating systems. Are any of these bad ideas? What you recommend that is not here?
AlmaLinux Alpine Linux Ubuntu Server Rhino Linux (unofficial ubuntu rolling) Debian Testing Void Linux FreeBSD
I would always recommend good old Debian for a mostly „it just works“ experience. You‘ll find debian packages for most if not all the things you mentioned. Alternatively you could go the steeper route and use an immutable OS like Fedora CoreOS or Fedora Silverblue for a more desktopy experience.
Hardware wise I‘ve been told the Intel NUC kits work wonders, or similarly specced boxes from Asia. You might get like 32GB RAM, a fairly recent CPU for <400€. Personally I‘m using a 12 year old Mac Mini until it dies, running debian.
Debian.
i have all of those things on my ubuntu server; but i wouldn't characterize all of them as enterprise-grade . my ubuntu server it's based off of off-the-shelf hardware from a couple years ago and it does all of those things that you described plus more.
it's my wifi router; my data storage backup; my home made security system; my media server inside; & my cat's favorite warm spot all within a tiny case the size of a toaster with lots of harddrives. it uses 2 kvm/qemu based virtual machines on top of the bare iron and they both use pci-passthrough; the first virtual machine is based off of the pfsense soft firewall & router and also serves to air-gap the bare iron server from the internet and the second virtual machine is windows 10 and serves to provide wifi 6 & 7 speeds with the windows ap driver.
i wouldn't describe any of it as enterprise grade since they're a bit hacky: for example, the server is mostly headless; but i did install the xserver & vnc because i use the motion project along with a bunch of old androids to create a homemade security monitoring system and that requires a browser. this means that i can now access the server's gui anywhere than i want; but it's subject to vnc's limitations.
however the things that come from the soft firewall are definitely enterprise grade: the vpn works well and i can use both it and ssh from anywhere in the world to access my home network and i could theoretically add in a remote check in capability from a new project that reacts to incoming connections.
the only thing i don't think it could do i the high temperatures; the case is compact so i doubt that its thermals are any good.
so i should avoid mini and compact pcs then, right?
for heat: you're either going to want a blade serve with lots of fans or big, mostly empty case.
Proxmox has been pretty good to me. I have an ancient office PC that has proxmox installed as the hypervisor. It's based on debian but everything is done via a web interface (you can ssh or whatever into it too if you needed to). Then I have debian with docker containers, TrueNAS, and home assistant all installed as VMs. Benefits to this means you can put mission critical stuff on the "boring" debian and then have fun with whatever you want to tinker with on an entirely different os/Virtual Machine. I also use wireguard easy which is stupid simple to setup a VPN with. I would strongly recommend keeping all management of the server on the local network and use a VPN to connect. This will get you the "enterprise grade" security. Anything public should go through a reverse proxy/DMZ VM if you host something on the Internet. Use cloudflare or similar as an extra layer if you need a domain name and want a buffer between users and your network. Keep that device and software up to date and you should have a great defense.
IDS wise, it's a lot of work. You're better off spending that time working on building security by design by doing the above and ensuring anything that touches the public Internet has as little permissions as possible (no running the web server as root/user account), firewall management, etc. If you do want the challenge, or are Interested in learning something like security onion, wazuah or whatnot, don't let it stop you.
Hardware wise, affordable and uptime could mean it might be cheaper to have a backup machine. Proxmox has features to support high availability where if one of your physical servers go down, another can take over (2 physical servers that are copies of each other). You could have a decent workstation and then a used PC or whatnot as the backup. More important is probably a UPS and some workstation gear unless you want a screaming server jet in whatever room it goes in. Nothing you've mentioned seems too performance heavy so technical PC recommendations are going to vary based on expected traffic or use cases. My 2014 DDR3 office PC manages just fine but it's for very few people and in air conditioned space. You could probably price out mid grade consumer equipment for the main server and a used office PC for redundancy.
is it a big problem if i don't use virtualization? And i think if i ever need a public website, i will use an another machine to host that, or a docker. Also, what kind of cpu is needed and how much ram? i don't want a headless server, since survallience stuff needs graphical enviroment, my best bet would be a lightweight x11 window manager
Virtualization can be nice in that you can tinker and not worry about dependencies. Plus you can have one resource that's stable on FreeBSD, another that works well on Unix, etc.
Headless servers can run surveillance stuff via web interfaces or API/app integrations. Plus you can use the GUI via vnc, spice or another service to get to your x11 environment. I find proxmox easier than docker/containers as most of my troubleshooting is there. I've got security cameras linked to home assistant and it's all headless. You could plug a monitor in and pass that to a virtual machine to get the desktop experience.
Hardware recommendations are going to need more information. Number of users? Number of cameras/tasks the server is expected to do concurrently, will you have media/NAS hosting and if so, how much space and how fast do you want that to be?
Your use case in the OP for less than 4 users could probably be run on a potato (my potato is bottlenecked by wifi @ 10MBps). 10-15 users streaming media or 20 cameras constantly streaming to a monitor could easily eat up a decent chunk of resources.
If you're not exposing anything to the Internet, you probably don't need an IDS. It's a lot of effort to reduce false positives/tune it and the benefits are probably not worth it unless this is a business use case. Enterprise IDS/SIEMS used by actual companies is typically not FOSS because it needs that support provided by the vendor.
I run Slackware on all my servers
i heard it is extremely hard to use. is it true?
Preface: Not the person you responded to.
I've never used Slackware myself, but it's probably the oldest distribution out there. It's supposed to be stable AF, doesn't "fix" what ain't broken, and is very old school in its efficiency mindset. This means it's indeed not likely to hold your hand through things, but it's also very thoroughly documented at this point, and any help you find online is much more likely to still (mostly) work regardless of it's age - unlike most other more frequently updated distros. It's meant to be reliable, not fancy.