Migrated to new host succesfully.
Endlesstalk has been migrated to a new host.
There did seem to have been some caching issues, causing some certificate issues, but it seems to be fixed(Atleast for me).
Let me know, if there are any issues after the migration
EDIT: The 525 ssl certificate error shows up intermittently. I'm looking into it.
EDIT2: Fixed. Was an issue with the nginx loadbalancer.
Downtime caused by server going down. Migration to new host tommorow at EDIT: 20:00 UTC
Earlier today one of the servers, where endlesstalk is hosted went down. After some time, the server came back up again, but there were some unknown issue and the server was unstable. So preparation to migrating endlesstalk to a new host began. However after setting the new servers up, there was success with getting one of the "old" servers up and running again.
Tommorow at 18:00 20:00 UTC the migration to the new host will begin. See local time here. There will be some downtime with this, probably around an hour or less.
EDIT: Server went down again, but should be back again now.
EDIT2: 20:00 UTC, since I forgot I have something from 17-19 UTC.
Successfully upgraded to 0.19.5
The upgrade went smoothly and everything seems to work.
Let me know, if there is anything that dosn't work after the upgrade.
Upgrading to 0.19.5 on 22th June at 13:00 UTC
I have found the issue with the database migration, so the upgrade to the latest version of lemmy can proceed.
0.19.5 brings a lot of smaller bugfixes. See release notes here for more information. I will also upgrade the database to a newer version(postgres 16).
For this upgrade there will be downtime and I expect it to last around 1 hour or less. If there are any major issues with the upgrade, you can check the uptime site here or site status here
Local time can be seen here
Upgrading to 0.19.4 postponed.
The database migration to 0.19.4 failed, because the database schema doesn't align with the state the migrations want. The reason is probably because it didn't restore correctly from a previous backup, but I don't actually know the cause.
I thought I could create a new database with a correct schema and then import the data from the currrent database into the new one. This might still be possible, but it simply takes too long and it has gotten too late for me(03:00 in the night).
I will look into a fix for the migration and when I have a fix I will announce a new date for the upgrade to 0.19.4.
Updating to 0.19.4 on 15th June at 19:00 UTC EDIT: Issue with database migration. Will take a bit longer
0.19.4 brings a lot of changes. See release notes here for more information.
There should be no downtime or very minimal downtime. If there are any issues, check the uptime site here or site status here
Local time can be seen here
Note: An update to postgres 16 and pictrs 0.5 is also comming soon, which will bring some downtime. Don't know when yet, but will post an update, when I know.
EDIT: There was an issue with migrating the database, while upgrading to 0.19.4, so will take longer.
EDIT2: The database is in a different state, than the migration to 0.19.4 expects. The cause is not clear, but I'm looking into it.
[Fixed] Instability of site
Hello
I have noticed that the server have been going down a lot for 10 - 20 mins.
Unfortunately, I'm currently on vacation, so I don't think I will have the time to fix it.
I will be back tomorrow evening and will look into it and hopefully fix it then.
EDIT: There was a misconfiguration of the auto scaling setup. This scaled the system up and used all of its CPU, which caused the site to be unresponsive.
This should be fixed now, but I will keep monitoring it.
Approx 6 hours downtime, caused by deletion of all data.
While working on a small fix to lemmy, that was causing some unneeded cpu usage, I made a change that unfortunately caused the db and pictrs service storage to be deleted.
Thankfully I have backups of everything, so I went to restoring from a backup. However the restoring was very slow, since I used an unoptimal way to backup the db(raw sql dump). After the first backup completed, I found out that it was missing data, so I tried an older backup, but that didn't work either. It was missing data as well. So I tried a backup from another server(Since I backup to 2 different servers), which finally worked.
Usually restoring from backups, haven't taken too long previously, since my backups are faily small, but I will need to look into a quicker way to restore backups for lemmy, since the backup size of lemmy is much bigger.
NOTE: Data from ca. 2 hours before the site went down(16-18 UTC) will be missing and I'm unable to restore it.
I restarted the federation containers and my test site now only receives 1 activity, when I upvote something. Before I received 2 activity requests, where nginx returned 400 for the first request and then 200 for the second request.
Is it fixed for now for you as well @[email protected]?
For federation I run 3 containers as lemmy.ca seems to do, however the indexes are set to the correctly value. So that doesn't seem to be the cause, as it has been for lemmy.ca
I run 2 lemmy-ui and backend containers, so maybe that has something do with it? I run a test site, so I will look into it and see if I can figure out the issue.
Thanks for notifying me of the issue :)
Images/pictrs not working. UPDATE: Moved to new host.
The s3 host that pictrs use, have gone down.
I might have to move to another s3 host, which will mean it will take a bit before images are working. This will cause a loss of images for the last 2-3 hours before pictrs stopped working.
EDIT: Have moved to new s3 host. Unsure how many images were lost during the outage.
Updating to 0.19.2 on the 14th January at 15:00 UTC EDIT: Sucessfully updated!
Local time can be seen here
0.19.2 contains fixes for outgoing federation and a few other things. See release notes here for more information.
There should be no downtime or very minimal downtime. If there are any issues, check the uptime site here or site status here
EDIT: The server went down / was very very slow to respond. I'm not quite sure why.
Updating to 0.19.1 on the 23th December at 15:00 UTC. EDIT: Success
Local time can be seen here
0.19.0 introduces a lot of new exciting changes. See release notes here for more information.
0.19.1 fixes some bugs found in 0.19.0 version.
As long as the database migration doesn't interfere too much, there should be no downtime or very minimal downtime. If there are any issues, there are an uptime site here and a site status here
EDIT:
Unfortunately, the database migration caused some errors but I fixed it and afterwards the migration had to run for about an hour. Everything seems to work now, but let me know, if there's anything that doesn't seem to work.
Note: There has been a significant change to the federation setup, so if there's anything relating to federation that seems amiss, please let me know.
The update to 0.19
The 0.19 version is out and I expect to update sometime during the next week. Since it is a big release, I will need to spend some more time testing, that everything still works fine and ensure that the migration works without problems as well.
I will update with another post, when I know, when the update will take place.
I selfhost PhotoPrism.
I use a folder sync app, to upload all my pictures from my phone to a directory on my server, which PhotoPrism imports from every 15 mins(Might be a bit more or less).
I have used it for a couple of years and it has worked fine for me.
Moving to new server tomorrow(24/09) at 9:00 UTC. Update: Success
I expect a very minimal downtime of ca. 5-15 mins.
Downtime from 12:00 to 15:40 UTC
I had made some config changes to the database earlier in connection with the move to a new server, that caused the storage usage of the database to grow a lot. Then when it had no more space left, it crashed which caused the downtime. Unfortunately it happened at a time, where I wasn't available to fix immediately, which is why it was down for so long.
It is now fixed and I will keep a watch(setup an alert for database disk usage) to make sure it doesn't happen again.
Unstable/downtime from 18:00 to 19:00 UTC
Seems to be caused by low amount of available storage. This caused k8s to evict/delete pods -> site went down,, then it would fix itself -> site goes up again but then k8s would evict again -> site goes down. This continued untill it stabilized at some point.
This should be fixed and there should be more space available, when I move the server to a new host. I expect to move to a new server sometimes in the comming week. Will annonce the date, when I know, when it will happen.
EDIT: Spoke a little bit too soon, should be fixed now though.
EDIT2:
There was something that kept using storage, so ran into the issue again. Then the volume/storage for the image service(pictrs) stopped worked for some unknown reason(Thankfully, I have backups) and there shouldn't be any images lost.
Good news is that I have reclaimed a lot of storage, so shouldn't be in danger of running out of space for a long time.
Downtime from 23:32 - 23:56
While preparing for migration to a new host, I had to setup the db, but during that I deleted a resource in k8s to force a reload of settings in the db. This caused the db to use a different volume and it took a bit before I could revert it back to using the old volume.
No data should have been lost. Let me know, if anything is missing.
Images for the last 8 hours deleted because of CSAM.
Unfortunatly the tool for scanning CSAM didn't detect the image, so to ensure there are no CSAM on the server, images for the last 8 hours has been deleted.
Hmm. I can find the community when searching for [email protected] and I can also subscribe to it.
I can see that I'm the only subscriber and when a community hasn't been subscribed/searched, it will take a minute to show up before it has been pulled by the instance.
Thank you for clarifying(and sorry for the late answer).
For you (and anyone interested) I will answer the questions you asked.
Are you going to close upon any legal threat, or only upon a certain degree or size of threat?
If the legal threat is real(they have a real chance to win in a court) and there is nothing I could do(come to an agreement, move the hosting to another server etc), then I would close the server.
From the US only, or from any country?
Server is hosted in Germany
Will you close instantly, or will you guarantee a Minimum Survivability Timeframe for eg.: helping users to migrate away, like Mastodon’s covenant does?
As long as there aren't fines for keeping the website up or I get arrested, I would give a notice, so users can move to another server.
Lemmy.world does a great job(with a little downtime) and it is the same software we are hosting, so I can only see a couple of ways to make a better "offering"
- Better uptime as you mentioned
- Different deferation/moderation policy.
- Visual customizations(eg. different theme)
I try to provide a better uptime and a different deferation/moderation policy. I don't have the skills to make visual customizations, but I have added multiple frontends(like lemmy.world).
I'm open to suggestions/ideas if there is anything else that could be done to improve the "offering".
As an alternative to lemmy.world I would like to suggest my own instance endlesstalk.org.
I have no plans to deferate or ban anything releated to piracy. Only thing that might change my mind would probably be a company taking legal action against me.
I also strive to have as little downtime as possible and keep everything running smoothly.
You(and everyone else) are welcome at endlesstalk.org.
I prioritize uptime and have setup monitoring to notify, if anything goes down(See here). I have no problem banning/defederating Nazi's and the like, but I generally don't take action unless, an instance/users aren''t following the rules. I'm open for requests and discussion though.
Currently I'm the only admin, but I'm in the process of changing the setup of the site, so other admins would be able to fix the site, if anything should happen.
Think they were clapping because the star vessel died yeah. I don't think they cared who killed the star vessel.
My instance endlesstalk.org is hosted in Germany via Hetzner, so that should fit the bill.
Defederations are only done, when it would have a negative effect on the instance's users, but I'm always open to discuss, if anyone disagrees with a deferation.
Anyone is welcome to join my instance at endlesstalk.org. Its running on a pretty beefy server, so it can take a decent amount of users.
There are multiple frontends like lemmy.world and I have setup a seeder(lemmony), so the all feed should be pretty filled.
How powerful should my hardware be?
I think 512mb ram is probably a bit too low. There is a risk for the pc to crash at some point. I think 1 GB+ should be fine.
Which ports do I need opened to operate a lemmy instance for my personal use (i’m only going to be connecting from home)?
To avoid opening ports you can use Cloudflare tunnel as you mentioned.
Only other option is probably renting a cheap vps. Here is a cheap vps options for 2.5$
If you have the ability then self hosting is the best option
Otherwise I think instances with less users and content will be quicker, but then the all feed will probably be smaller.
I will also recommend my own instance endlesstalk.org. It is very small(10 users) and the server is located in Germany. Hopefully that is close to where you live, so the latency is low.
About 2 months, 62 comments, 17 post. The post are mostly server updates for my instance though.
Still a big improvement over reddit though, since I just lurked there. Think I have more comments now, than in all my 9 years on reddit.
For a very small instance(10 users), you can try mine at endlesstalk.org. It also has multiple frontends hosted like lemmy.world and many others.
Otherwise I would recommend lemm.ee, lemmy.sdf.org and lemmy.zip
It is on my to-do list, but it would take some work to add support for multiple people to have access to everything required and I would also need a lot more documentation, than I currently have.