Skip Navigation

Search

Uptime monitoring in Windows

> Disclaimer: This is for folks who are running services on Windows machines and does not have more than one device. I am neither an expert at self hosting nor PowerShell. I curated most of this code by doing a lot of "Google-ing" and testing over the years. Feel free to correct any mistakes I have in the code.

Background

TLDR: Windows user needs an uptime monitoring solution

Whenever I searched for uptime monitoring apps, most of the ones that showed up were either hosted on Linux or containers and all I wanted was a a simple exe installation file for some app that will send me alerts when a service or the computer was down. Unfortunately, I couldn't find anything. If you know one, feel free to recommend them.

To get uptime monitoring on Windows, I had to turn to scripting along with a hosted solution (because you shouldn't host the monitoring service on the same device as where your apps are running in case the machine goes down). I searched and tested a lot of code to finally end up with the following.

Now, I have services running on both Windows and Linux and I use Uptime Kuma and the following code for monitoring. But, for people who are still on Windows and haven't made the jump to Linux/containers, you could use these scripts to monitor your services with the same device.

Solution

TLDR: A PowerShell script would check the services/processes/URLs/ports and ping the hosted solution to send out notification.

What I came up with is a PowerShell script that would run every 5 minutes (your preference) using Windows Task Scheduler to check if a Service/Process/URL/Port is up or down and send a ping to Healthchecks.io accordingly.

Prereqs

  1. Sign up on healthchecks.io and create a project

  2. Add integration to your favorite notification method (There are several options; I use Telegram)

  3. Add a Check on Healthchecks.io for each of the service you want to monitor. Ex: Radarr, Bazarr, Jellyfin

    When creating the check, make sure to remember the Slug you used (custom or autogenerated) for that service.

  4. Install latest version of PowerShell 7

  5. Create a PowerShell file in your desired location. Ex: healthcheck.ps1 in the C drive

  6. Go to project settings on Healthchecks.io, get the Ping key, and assign it to a variable in the script

    Ex: $HC= "https://hc-ping.com/<YOUR_PING_KEY>/"

    The Ping key is used for pinging Healthchecks.io based on the status of the service.

Code

  1. There are two ways you can write the code: Either check one service or loop through a list.

Port

  1. To monitor a list of ports, we need to add them to the Services.csv file. > The names of the services need to match the Slug you created earlier because, Healthchecks.io uses that to figure out which Check to ping.

Ex:

"Service", "Port" "qbittorrent", "5656" "radarr", "7878" "sonarr", "8989" "prowlarr", "9696"

  1. Then copy the following code to healthcheck.ps1:

Import-CSV C:\Services.csv | foreach{ Write-Output "" Write-Output $($_.Service) Write-Output "------------------------" $RESPONSE = Test-Connection localhost -TcpPort $($_.Port) if ($RESPONSE -eq "True") { Write-Host "$($_.Service) is running" curl $HC$($_.Service) } else { Write-Host "$($_.Service) is not running" curl $HC$($_.Service)/fail } }

> The script looks through the Services.csv file (Line 1) and check if each of those ports are listening ($($_.Port) on Line 5) and pings Healthchecks.io (Line 8 or 11) based on their status with their appropriate name ($($_.Service)). If the port is not listening, it will ping the URL with a trailing /fail (Line 11) to indicate it is down.

Service

  1. The following code is to check if a service is running.

    You can add more services on line 1 in comma separated values. Ex: @("bazarr","flaresolverr")

    This also needs to match the Slug.

$SERVICES = @("bazarr") foreach($SERVICE in $SERVICES) { Write-Output "" Write-Output $SERVICE Write-Output "------------------------" $RESPONSE = Get-Service $SERVICE | Select-Object Status if ($RESPONSE.Status -eq "Running") { Write-Host "$SERVICE is running" curl $HC$SERVICE } else { Write-Host "$SERVICE is not running" curl $HC$SERVICE/fail } }

> The script looks through the list of services (Line 1) and check if each of those are running (Line 6) and pings Healthchecks.io based on their status.

Process

  1. The following code is to check if a process is running.

    Line 1 needs to match their Slug

$PROCESSES = @("tautulli","jellyfin") foreach($PROCESS in $PROCESSES) { Write-Output "" Write-Output $PROCESS Write-Output "------------------------" $RESPONSE = Get-Process -Name $PROCESS -ErrorAction SilentlyContinue if ($RESPONSE -eq $null) { # Write-Host "$PROCESS is not running" curl $HC$PROCESS/fail } else { # Write-Host "$PROCESS is running" curl $HC$PROCESS } }

URL

  1. This can be used to check if a URL is responding.

    Line 1 needs to match the Slug

$WEBSVC = "google" $GOOGLE = "https://google.com" Write-Output "" Write-Output $WEBSVC Write-Output "------------------------" $RESPONSE = Invoke-WebRequest -URI $GOOGLE -SkipCertificateCheck if ($RESPONSE.StatusCode -eq 200) { # Write-Host "$WEBSVC is running" curl $HC$WEBSVC } else { # Write-Host "$WEBSVC is not running" curl $HC$WEBSVC/fail }

Ping other machines

  1. If you have more than one machine and you want to check their status with the Windows host, you can check it by pinging them

  2. Here also I use a CSV file to list the machines. Make sure the server names matches their Slug

    Ex:

    "Server", "IP" "server2", "192.168.0.202" "server3", "192.168.0.203"

Import-CSV C:\Servers.csv | foreach{ Write-Output "" Write-Output $($_.Server) Write-Output "------------------------" $RESPONSE = Test-Connection $($_.IP) -Count 1 | Select-Object Status if ($RESPONSE.Status -eq "Success") { # Write-Host "$($_.Server) is running" curl $HC$($_.Server) } else { # Write-Host "$($_.Server) is not running" curl $HC$($_.Server)/fail } }

Task Scheduler

For the script to execute in intervals, you need to create a scheduled task.

  1. Open Task Scheduler, navigate to the Library, and click on Create Task on the right
  2. Give it a name. Ex: Healthcheck
    1. Choose Run whether user is logged on or not
    2. Choose Hidden if needed
  3. On Triggers tab, click on New
    1. Choose On a schedule
    2. Choose One time and select an older date than your current date
    3. Select Repeat task every and choose the desired time and duration. Ex: 5 minutes indefinitely
    4. Select Enabled
  4. On Actions tab, click on New
    1. Choose Start a program
    2. Add the path to PowerShell 7 in Program: "C:\Program Files\PowerShell\7\pwsh.exe"
    3. Point to the script in arguments: -windowstyle hidden -NoProfile -NoLogo -NonInteractive -ExecutionPolicy Bypass -File C:\healthcheck.ps1
  5. Rest of the tabs, you can choose whatever is appropriate for you.
  6. Hit Ok/Apply and exit

Notification Method

Depending on the integration you chose, set it up using the Healthchecks docs.

I am using Telegram with the following configuration:

Name: Telegram Execute on "down" events: POST https://api.telegram.org/bot<ID>/sendMessage Request Body: { "chat_id": "<CHAT ID>", "text": "๐Ÿ”ด $NAME is DOWN", "parse_mode": "HTML", "no_webpage": true } Request Headers: Content-Type: application/json Execute on "up" events: POST https://api.telegram.org/bot<ID>/sendMessage Request Body: { "chat_id": "<CHAT ID>", "text": "๐ŸŸข $NAME is UP", "parse_mode": "HTML", "no_webpage": true } Request Headers: Content-Type: application/json

Closing

You can monitor up to 20 services for free. You can also selfhost Healthchecks instance (wouldn't recommend if you only have one machine).

I've been wanting to give something back to the community for a while. I hope this is useful to some of you. Please let me know if you have any questions or suggestions. Thank you for reading!

4

Problem connecting to host from Docker container

I followed this Guide to setup headscale with caddy. And tried to add Keycloak with this guide from the same guy.

Sadly my docker containers do not seem to be able to connect to the keycloak server. What happens is that if i try to download the openid configuration from the host (via wget) or from my local PC it just works. But the headscale server gets a timeout when trying to connect to the endpoint. When i use the internal docker name to connect to the keycloak container the connection works fine but then i get an error because its not the external url.

I experimented a bit and managed to reproduce the issue with a different container (running an ubuntu container and also getting a timeout when trying to download the config from keycloak). If i run the container with the host network i works just fine.

Does anyone know how to fix this?

PS: i also tried the example from the guide with gitea an its also the same problem

Update: I tried most suggestions and for some reason it just didn't work. My solution that is working now is that I bind the container ports to localhost only (by using p.e.: ports: -"127.0.0.1:4567:8080") and using the caddy server in host network mode. Now all containers can connect like expected and are working flawlessly. Thanks for all your suggestions :)

9

Guide to Self-Hosting Lemmy with Individual Containers &amp; Existing NGINX Instance.

Problem Statement

The official docker-compose and Docker documentation for self-hosting Lemmy is not suitable for my use-case. It:

  • Spins up its own single-use containers for pictrs, postgres and nginx.
  • Makes a bunch of assumptions about the deployment network topology that doesn't always work in a more managed setting.

I'm not a pro nor an expert in sysadmin, Docker or web technologies, so it took many hours of deciphering the (very) sparse documentation to figure out how to make Lemmy fit my deployment scenario. Here, I'd like to just share my own docker-compose, lemmy.hjson and my NGINX reverse proxy configuration, and hope it helps someone out there.

How I Host My Services

  • Each service is single-instance, multiple-use. For example, my postgres container serves not just Lemmy, but other containers that require a DB service as well.
  • I have an existing reverse proxy with nginx, already provided by the awesome swag image.
  • Each container gets an assigned internal LAN hostname, assigned internal LAN IP, and specified MAC address for house-keeping and manageability purposes.
  • I have full control and authorship of all my services, and only services are exposed to the public internet through either CloudFlare or nginx. Hence, storing key values in my docker-compose is not a major security risk. If my LAN is breached, then I have bigger things to worry about besides a few passkeys being compromised. If you are operating in a multi-user LAN environment where security is paramount, then please use Docker Secrets instead of storing your secrets in plaintext.

Some Parameters You'll Need

My template values are assumed as such. For API keys and passwords, use your own generator or some UUID generation service. If you're using Linux and have the uuidgen package, just generate keys on your terminal with uuidgen -r | sed 's/-//g'; The second command just removes the - character from stdout. All provided values below are dummy values! Please generate your own whenever applicable.

To generate MAC addresses, use any MAC address generator tool, or an online service.

Needless to say, change the following parameters to suit your own deployment.

General Networking

  • Your internal DNS IP: 192.168.0.1
  • Your LAN subnet mask: 192.168.0.0/16
  • Your Docker container subnet mask: 192.168.1.0/24
  • Your Docker host IP: 192.168.1.1
  • Your localhost domain name: .local
  • Your Docker bridge name (this must be an existing bridge): custom_docker_bridge

Your SMTP settings

Here, I am assuming you have a gmail account that you want to use as your mailbox to send admin emails. Follow this guide to generate an app password for Google to authenticate you.

Your Lemmy Site Name &amp; Admin Account

  • admin_username: admin
  • admin_password: c97f337aaa374d8a9c47fce0e197fd29
  • site_name: lemmy.yourowndomainname.yourtld

For pictrs

  • PICTRS__SERVER__API_KEY: e7160a506a9241abb1e623d4180d6908
  • Container IP: 192.168.1.2
  • Container MAC: 30:b1:fb:dd:af:ee
  • Container Hostname: PICTRS.local
  • Persistent Volume: /some/host/directory/pictrs

For postgres

  • POSTGRES_USER: postgres_admin
  • POSTGRES_PASSWORD: eefb3bce7ea54b8497307d0e0234b6c8
  • POSTGRES_DB: postgres_db
  • Container IP: 192.168.1.3
  • Container MAC: a9:95:c4:a3:e5:4f
  • Container Hostname: POSTGRES.local
  • Persistent Volume: /some/host/directory/postgres

For lemmy

  • Container IP: 192.168.1.4
  • Container MAC: 77:26:eb:bf:c9:f7
  • Container Hostname: LEMMY.local
  • Persistent Volume (for your lemmy.hsjon): /some/host/directory/lemmy/lemmy.hjson
  • DB name: lemmy_db
  • DB user: lemmy_admin
  • DB password: ebd3526474cf4cc6af752971f268d0f3

For lemmy-ui

  • Container IP: 192.168.1.5
  • Container MAC: bd:77:70:e6:ca:d8
  • Container Hostname: LEMMYUI.local
  • LEMMY_UI_LEMMY_EXTERNAL_HOST (this is your public-facing domain name that points to your Lemmy UI): lemmy.yourowndomainname.yourtld
  • LEMMY_UI_LEMMY_INTERNAL_HOST (match with above): LEMMY.local:8536

Templates for docker-compose

For pictrs

version: "3.7" services: pictrs: container_name: pictrs image: asonix/pictrs:0.4 environment: - PICTRS__SERVER__API_KEY=e7160a506a9241abb1e623d4180d6908 ports: - 8080:8080 restart: unless-stopped hostname: PICTRS.local dns: 192.168.0.1 mac_address: 30:b1:fb:dd:af:ee networks: custom_docker_bridge: ipv4_address: 192.168.1.2 volumes: - /some/host/directory/pictrs:/mnt networks: custom_docker_bridge: external: true name: custom_docker_bridge

For postgres

This assumes you don't already have a postgres instance. version: "3.7" services: postgres: container_name: postgres image: postgres:latest # This is the default postgres db that is created when you spin up a new postgres container. This will not be used by Lemmy, but the credentials here are important in case you ever lose your password to `lemmy_admin`. environment: - POSTGRES_USER=postgres_admin - POSTGRES_PASSWORD=eefb3bce7ea54b8497307d0e0234b6c8 - POSTGRES_DB=postgres_db ports: - 5432:5432 restart: unless-stopped hostname: POSTGRES.local dns: 192.168.0.1 mac_address: a9:95:c4:a3:e5:4f networks: custom_docker_bridge: ipv4_address: 192.168.1.3 command: [ "postgres", "-c", "session_preload_libraries=auto_explain", "-c", "auto_explain.log_min_duration=5ms", "-c", "auto_explain.log_analyze=true", "-c", "track_activity_query_size=1048576", ] volumes: - /some/host/directory/postgres:/var/lib/postgresql/data networks: custom_docker_bridge: external: true name: custom_docker_bridge

For lemmy Backend

version: "3.7" services: lemmy: container_name: lemmy image: dessalines/lemmy:latest hostname: LEMMY.local dns: 192.168.0.1 mac_address: 77:26:eb:bf:c9:f7 ports: - 8536:8536 networks: custom_docker_bridge: ipv4_address: 192.168.1.4 restart: unless-stopped volumes: - /some/host/directory/lemmy/lemmy.hjson:/config/config.hjson:Z networks: custom_docker_bridge: external: true name: custom_docker_bridge

For lemmy-ui Frontend

version: "3.7" services: lemmy-ui: container_name: lemmy-ui image: dessalines/lemmy-ui:latest environment: - LEMMY_UI_LEMMY_INTERNAL_HOST=LEMMY.local:8536 - LEMMY_UI_LEMMY_EXTERNAL_HOST=lemmy.yourowndomainname.yourtld - LEMMY_UI_HTTPS=false - LEMMY_UI_DEBUG=true hostname: LEMMYUI.local dns: 192.168.0.1 mac_address: bd:77:70:e6:ca:d8 networks: custom_docker_bridge: ipv4_address: 192.168.1.5 restart: unless-stopped networks: custom_docker_bridge: external: true name: custom_docker_bridge

Template for lemmy.hjson

``` { database: { uri: "postgres://lemmy_admin:[email protected]:5432/lemmy_db" } pictrs: { url: "http://PICTRS.local:8080/" api_key: "e7160a506a9241abb1e623d4180d6908" } email: { smtp_server: "smtp.gmail.com:587" smtp_login: "[email protected]" # Password to login to the smtp server smtp_password: "abcdefghijklmnop" smtp_from_address: "[email protected]" tls_type: "tls" }

These will be used for the first-ever time the container is created. This is the admin account used to login to https://lemmy.yourowndomainname.yourtld and manage your Lemmy instance.

setup: { # Username for the admin user admin_username: "admin" # Password for the admin user. It must be at least 10 characters. admin_password: "c97f337aaa374d8a9c47fce0e197fd29" # Name of the site (can be changed later) site_name: "lemmy.yourowndomainname.yourtld" # Email for the admin user (optional, can be omitted and set later through the website) admin_email: "[email protected]" } hostname: "lemmy.yourowndomainname.yourtld"

Address where lemmy should listen for incoming requests

bind: "0.0.0.0"

Port where lemmy should listen for incoming requests

port: 8536

Whether the site is available over TLS. Needs to be true for federation to work.

tls_enabled: true } ```

Template for nginx

This assumes your NGINX's http directive is pre-configured and exists elsewhere. ``` server { listen 443 ssl; listen [::]:443 ssl;

server_name lemmy.*;

# Assuming your ssl settings are elsewhere include /config/nginx/ssl.conf;

set $lemmy_frontend_hostname lemmyui.local; set $lemmy_frontend_port 1234;

set $lemmy_backend_hostname lemmy.local; set $lemmy_backend_port 8536;

set $upstream_proto http;

location ~ ^/(api|pictrs|feeds|nodeinfo)/ { set $prox_pass $upstream_proto://$lemmy_backend_hostname:$lemmy_backend_port; proxy_pass $prox_pass; }

location / {

# Default to lemmyui.local set $prox_pass $upstream_proto://$lemmy_frontend_hostname:$lemmy_frontend_port;

# Specific routes to lemmy.local if ($http_accept ~ "^application/.*$") { set $prox_pass $upstream_proto://$lemmy_backend_hostname:$lemmy_backend_port; } if ($request_method = POST) { set $prox_pass $upstream_proto://$lemmy_backend_hostname:$lemmy_backend_port; } proxy_pass $prox_pass; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } ```

Conclusion

That's it! If you're looking for more in-depth tutorials for how each of these work, it is unfortunately out of scope for this post. Hope this helps someone in their journey to self-host Lemmy. Cheers.

Edit #1 - (2023-07-29) nginx.conf needed some additional parameters for proxy_http_version and proxy_set_header, otherwise Lemmy's root_span_builder will start to throw Incoming activity has invalid signature errors. I believe the important line is proxy_set_header Host $host;

21