My home lab has a mild amount of complexity and I'd like practice some good habits about documenting it. Stuff like, what each system does, the OS, any notable software installed and, most importantly, any documentation around configuration or troubleshooting.
i.e. I have an internal SMTP relay that uses a letsencrypt SSL cert that I need to use the DNS challenge to renew. I've got the steps around that sitting in a Google Doc. I've got a couple more google docs like that.
I don't want to get super complicated but I'd like something a bit more structured than a folder full of google docs. I'd also like to pull it in-house.
Thanks
Edit: I appreciate all the feedback I've gotten on this post so far. There have been a lot of tools suggested and some great discussion about methods. This will probably be my weekend now.
Well, whatever you end up using for documentation, print it out and actively maintain an up to date paper hard copy in a 3-ring binder somewhere. That way when all your shit falls over and you have a nonfunctional LAN you can still remember how everything was set up. Don't ask me how I know.....
Documentation is not worth much if you can't access it when needed. So yes, either print it out or store it somewhere else what you can access even if your own hardware is completely dead.
I've been using Obsidian for a lot of other purposes for a couple years now, so I was comfortable adding my documentation into my existing vault there. I made a couple templates that I fill out for any hardware/software/networking equipment.
Since the app's selling point is storing all your notes in plain text I wouldn't put anything security-related in there without some encrypted container. I use KeePass for that part, and keep the file it generates in the same folder as Obsidian so I can link to it within notes. Click the link in the note, KeePass opens the vault and asks for its password.
I left everything in, so no doubt there's stuff in there specific to my vault you won't need like metadata - adjust these to your needs or use them as a starting point for something new. There's no network device template, I usually use the hardware one and just delete the irrelevant bits.
Mine is, er, self-documenting, and my partner has instructions, in the event that I die, to plug the wifi router into the modem and unplug all the other crap. The router has a sticker saying ROUTER and the modem has a sticker saying MODEM.
As I move to more self hosting, it's becoming more and more important to create a "what to do if I die" procedure for my wife (or even children) to follow. I mean it's not big deal if the plex server goes down and doesn't come back up, I'm thinking more along the lines of all of our photos, important documents, password manager, those type of things. I have 3 - 2 - 1 backups for the important stuff and have tested them, but that means nothing to my wife if I wasn't around to get that stuff back if something happened.. I wonder some days if I should document it all and put a print out with a step by step guide on how to get everything back that a semi tech savvy person could follow.
Yes, I really need to do this too. It's a fairly daunting task because my significant other is not at all comfortable with even slightly complicated operations on a computer. I need to figure out a way to make a really easily accessible backup of everything.
+1 for WikiJS. As a bonus you can have WikiJS back itself up to plain text MarkDown files, so if things explode you can always just read those from wherever.
Another great feature I use is to have WikiJS back itself up into git. If I am going to a place with no internet access I can do a quick git pull and have a complete copy of my wiki including files on my laptop.
Same, but I have no plan to change. My setup isn't very complex so there's not much to track. Most of it is just one docker compose file. It would just be one more thing to maintain without much benefit.
ansible, self-documenting. My playbook.yml has a list of roles attached to each host, each host's host_vars file has details on service configuration (domains, etc). It looks like this: https://pastebin.com/6b2Lb0Mg
Additionally this role generates a markdown summary of the whole setup and inserts it into my infra's README.md.
Manually generated diagrams, odd manual maintenance procedures and other semi-related stuff get their own sections in the README (you can check the template here) or linked markdown files. Ongoing problems/research goes into the infra gitea project's issues.
I'm only just starting to dip my toes in docker. Most of my stuff are kvm vms. I have a decent set of Ansible roles to setup a new vm when I spin it up but I'm not to the point where the specifics of every system is in Ansible yet.
You can full well deploy docker stacks using ansible. This is what I used to do for rocket.chat: [1][2] (ditched it for Matrix/element without Docker, but the concept stays valid)
Iโm not to the point where the specifics of every system is in Ansible yet.
What I suggest is writing a playbook that list the roles attached to your servers, even if the roles actually do nothing:
# roles/application-x/tasks/main.yml
- name: setup application-x
debug:
msg: "TODO This will one day deploy application-x. For now the setup is entirely manual and documented in roles/application-x/README.md"
# roles/application-x/tasks/main.yml
- name: setup service-y
debug:
msg: "TODO This will one day deploy service-y. For now the setup is entirely manual and documented in roles/service-y/README.md"
#...
This is a good start for a config management/automated deployment system. At least you will have an inventory of hosts and what's running on them. Work your way from there, over time progressively convert your manual install/configuration steps to automated procedures. There are a few steps that even I didn't automate (like configuring LDAP authentication for Nextcloud), but they are documented in the relevant role README [3]
Joplin has been my note-taking app of choice. Free (OSS); no accounts needed; simple; local; synchronizes through my Nextcloud instance to Linux, Windows and Android; markdown-based, etc. It's been a good workflow for me and makes taking and searching through notes quick and pretty painless.
I document my setup, backup and restoration procedures (done rarely enough that it's good to have it written down), etc. with it.
+1 for Joplin. I have a different setup since I don't use Nextcloud: Run Joplin server in a docker container and back up the volumes mapped to it (as well as those of other containers) with rsync.
What is it that you all are documenting? I'm just getting started and have so far just set up docker and several containers (Plex, the *arrs, qbittorrent, gluetun) and not sure if I should be writing something down?
I document my proxmox configuration, and again for each service running on it; ports, fqdn, letโs encrypt etc. Any problems or deviations from the standard installation and how I resolved it. How I mounted my media drive and and made it accessible in Plex and Jellyfin. Any configuration that I had to search for the solution, including the actual steps and not just links that might be gone in the future. The more services you accumulate the harder it is to start the documentation, so itโs a good idea to start sooner than later.
I'd like to document what I have, and what I need to do it spin it back up with minimal effort should I need to. Esp, anything that's unique to my setup or that I did to work around a problem. I've shot myself in the foot many a time where I go back to something 6 months later and I've long since forgotten what I did. So, if it's not commented in a config I'm figuring it out all over again.
I run wiki.js for documentation for my home lab, but also things like the custom rom setup for my phone. However it's hard to keep it up to date as I forget it exists. I mostly use it to document setting up windows server core with different roles as I don't need to do that often, but most tutorials on the web are SEO optimised with low quality
I'm not very good at deciding on where to document things, so I have a mix of BookStack, Dokuwiki, and Obsidian currently.
I really like Dokuwiki but I like the UI/UX of BookStack better so I'm working on a plugin to sync bookstack and obsidian. I'll probably get rid of Dokuwiki after that.
The main reason for syncing with obsidian is that I want documentation that isn't stored on the thing it's about, in case my servers completely die.
In another thread, someone reminded me that TiddlyWiki still exists, it's also a pretty cool little tool.
Ibrun my own gitlab instance where i have a repository with all my configs, scripts etc.
As well as a wiki that contains the admin guide with (hopefully) all the relevant infos in form of text and PlantUML graphs.
The nice thing is.. You can just 'code' the diagrams and use a PlantUML instance to render the graphs live
I use trillium that gets backed up every hour to my pc.
I also do a lot of python development so my project ideas get written down there too.
I'm not a fan of code is documentation because what happens when you step away for a month and you need to figure something out? In trillium I have a search bar. What do you have in the code?
I will second trilium. I use their sync server in a VM (which is backed up with the rest of my VM's so its easy to drop back down should something happen). The app appeals to me, even after using Obisdian for the past 6 months (i'm a fan of markdown as well).
I use the wonderful https://draw.io to sketch up my homelab and which device hosting what service. More fun when it's vidualized the way I want it ๐
I went with dokuwiki forever ago. Super stupid simple single container to run (no DB) and writes down to plain text files. I sync it with git every now and then.
My only gripe about it is the dokuwiki syntax and not using normal markdown. I do now have a plugin for that but it's still just ok. But at this point I might be too engrossed in it to ever really switch. But other than that it works well, is lightweight, has other plugins (email, mermaid flow charts, etc. etc.) and really is pretty maintenance-free.
If there is some better one that is accessible via a browser and doesn't require a DB then I'd be interested.
I'm using netbox, and I'm in the process of info dumping my brain to a media wiki. I may add ansible into the mix in the near future.
My lab is a bit large and complex and I'm currently in the process of trying to train help to run it, from the pool of people that make use of it. They know how the front end works, a few of them need to learn the back end.
+1 for bookstack.
I also selfhost a kanban with the services basic info and it's related status (pilot/test, production and to be decommissioned). At the beginning I used Planka, but now switched to Nextcloud Deck.
In all honesty, it is a hodge podge. Some are in my dokuwiki, some are plain text, some are markdown, some in my phone, lots on scraps of paper. Just about the time I get it all in one place I scrap my systems and start over.
I use logseq to record any manual steps as well as any administrative actions that I take on a service. That being said, all of my homelan infrastructure is codified and stored in git in various ways so, it can be recreated as needed. There are very few manual steps in reconfiguring any of my services.
I use a combination of netbox for the physical/logical network and server connectivity, and outline for text documentation of the different components.
I think I looked at netbox a while back. I may circle back to it for the actual physical layer. If I remember the ipam didn't include network scanning tho.
I know that I would keep forgetting to update the docs, so my documentation are the ansible playbooks and docker-compose.yaml files that I use to set it all up.
That leaves anything that has to be done in some Ui undocumented, so I try to keep that to a minimum, which isn't always easy (I'm looking at you authentik!).
Ideally I'd like to move in that direction. I have some Ansible roles that I use for initial configuration but I haven't kept current. I'd like to get better about that as part of this project.
This is interesting. I already just keep a collection of markdown files.. might as well make it an internal documentation site so it's easier to browse ๐ค
I use Ansible, Docker, and Emacs OrgMode files committed to Git. Diagrams are a mix of Miro and Graphviz. There's also a few markdowns in there too. Joplin is used for rough notes only.
I made myself a wiki in my helpdesk system - I use osticket to send me various email alerts to so I can track issues I need to fix, and they have a little wiki option.
Then one day that host was down and I needed some info and I was very irritated. Now all of those notes are in my Apple notes backed up in iCloud and searchable on whatever Iโve got handy so if I need info I can get the info
I played with GLPI just long enough to realize that was way more than I wanted or needed. I'd like to track changes but I don't want to run a full ticketing/chg mgmt system to do it.
This seems pretty vanilla based on what other have suggested but I use regular markdown files in a git repo.
For data flows or diagrams, mermaidJS syntax within the markdown file works wonders and when I need to link one document to another or one section to another, you can use the normal link syntax of markdown.
Easy to use, hardly any setup and easily accessible.
I'm defining my service containers via GitLab and I deploy them via tagged and dockerized GitLab Runners.
If something fails, I change the runner tags for a service and it will be deployed on a different machine.
Incl case of a critical failiure, I just need to setup a Debian, install docker, load and run the GL runner image, maybe change some pipelines and rerun the deployment jobs.
Some things aren't documented well, yet. Like VPN config...
Ah yes, my router is able to access GitLab as well and pull the list of static routes etc. from it.
As someone not super familiar working in Git Iโd love more details about your documentation for your setup. I have most of my containers organized in compose stacks that make sense (eg all the Arrs are in a single compose with the download client) but actually documenting the structure is โฆ well nonexistent.
The thing is it's not really a "documentation" but just a collection of configs.
I have organized my containers in groups like you did ("arrs", web server, bitwarden, ...) and then made a repository for each group.
Each repository contains at least a compose file and a Gitlab CI file where a aimple pipeline is defined with basically "compose pull" and "compose up". There are alao more complicated repository where I build my own image.
The whole "Git" management is really transparent, because with Gitlab you can edit directly on the platform in a hosted VSCode environment where you can directlY edit your files and when your satisfied you just press commit. I don't do weird stuff with branches, pushing and pulling at all. No need for local copies of the repository.
If you want to fulltext search all your repos, I can recommend a "Sourcegraph" container, but use version 4.4.2 because starting with 4.5.0 they have limited the number of private reositories to 1. But this is something for later, when your infrastructure has grown.
There are tons of tutorials around, but the basic gist is that you only use a couple of commands (or even a good frontend) in git, especially when it's a one (wo)man show.
Gotcha. Git is useful in so many way, but it can be confusing to learn. I donโt have a guide on hand but searching for โgetting started with gitโ will get you pretty far.
Another great way to do this that I just thought of this second is using Notion. It is in markdown.
I deploy as much as I possibly can via Ansible. Then the Ansible code serves as the documentation. I also keep the underlying OS the same on all machines to avoid different OS conventions. All my machines run Debian. The few things I cannot express in Ansible, such as network topology, I draw a diagram for in draw.io, but that's it.
Also, why not automate the certificate renewal with certbot? I have two reverse proxies and they renew their certificates themselves.
Why not have the reverse proxy also do renewal for the SMTP relay certificate and just rsync it to the relay? For a while I had one of my proxies do all the renewals and the other would rsync it.
I write down everything I built so for plus future plans in OneNote. This kind of defeats the purpose of self hosting but I want to keep a written copy complete off site in case if a complete loss. Plus I like OneNote. Itโs actually a well designed product.
Scripts, docker compose files and such are in GitHub.
I document steps for configuration or installation of a service. Normally, only if it's not pretty obvious and I have to combine multiple sources to figure out what I need to be doing.
I also document all of my static IP address in one place. That way I know which service is on which IP and I have a list of used IPs so I don't try to reuse them.
Some stuff is in Joplin, some stuff is in wiki.js.
Joplin lacks organization features. Wiki.js stores stuff in database and has problems with search, both are possible to fix, I believe...
Occasionally I remember about problems with this setup, but I'm too lazy to fix or replace it
Frankly the only thing I'd save in Google Docs are encrypted archives. Otherwise they'll profile the documents to send ads to you. But it is a good back up in case lightning strikes your home or something.
I don't save all my documents. Just my self-hosting, servers infraestructure notes. I don't want to have the recovery intructions in the same machine I'm recovering
Are you writing to Google drive directly from the cli? If so how? I regularly need to search, edit, copy, and paste to and from my notes; backup config files; save a neat little script I wrote; etc. all from the CLI. It would be awesome to have this searchable and online from a web browser too for when I'm not working in the terminal. For example, piping an error message to a file and grabbing/sanitizing that error to search later. I have ways, but their all a lot clunkier than simply have a Dropbox. I'm basically looking for something that works just like Dropbox, is not self hosted, and not as cumbersome to setup as NextCloud and the like.