Site will be down for the upgrade sometime today as I run the scripts we have and see what happens and then troubleshoot. Hopefully not too long.

Rule

FB Marketplace is a strange place.

Today on 22 April 2024 lemmy.myserv.one reached >99% uptime. We are also in the top 350 most reliable lemmy servers now as measured by this site: https://lemmy.fediverse.observer/list - currently on page 11 if sorted by uptime.

The server is also now almost 1 year old. It will be 1 year on June 2024. Thanks for sticking with us!

Next goal is top 100 servers and/or 99.5% uptime.

Thanks everyone:)

Another apology to everyone.

I upgraded the instance to 0.19.1 today, and the upgrade unfortunately had a major issue. This caused the Database to become fairly broken and it has taken several hours to repair the damage and the database.

I have found the root cause of the issue and will be implementing some changes to the backup routine to avoid this issue in the future for future upgrades. This is thanks to some very welcome help from the admins on the Beehaw instance who are incredible and supportive.

As a result this will be improved and shouldnt happen again in the future when upgrading. I still need to run a vacuum on the postgres database but I will do this tomorrow, but there will be a 15 minute window tomorrow when the site is unresponsive while this is run.

Hopefully after that there are no more issues and the site can finally come back to being reliable again.

Kind regards,

• P

Unfortunately the server went offline today for approximately 12 hours. This is due to a deficiency in the backup routine, where if the vps hoster I use turns off the server/has an interruption, during the window of about 30 minutes where the backup runs then the script does not put everything back in place to ensure the server can start up again.

The good news is, this new issue that was not thought about should be resolvable with a reboot script, that uses an if or statement to check if the backup did do what it was supposed to and if not take some fairly simple logic to put things back as they should be.

In the next couple of weeks I will implement and test such a solution.

However it does raise another issue in that lemmy had some growth and then shrinking, that has stabilised and left us with a constant but slow increase in users, lost some of the admins that used to help check this server and they have abandoned their lemmy deployments for whatever reason and there are less people around actually checking of things are working.

Sadly I didnt notice the server was offline for 12 hours as I was working and in meetings all day and only got around to checking lemmy later in the afternoon. This is disappointing as the fix only took a few minutes - less than 10 to login, check what happened and correct it.

Had I been made aware of an issue earlier, many hours of downtime would have been saved.

To be clear, this server is run extremely cheaply. This means we dont have as many automated checks as other more expensive lemmy servers and we must rely on our members to tell us if things go wrong. This design choice was chosen in case lemmy becomes less popular and so the cost of the server is no issue. Essentially I can pay and run it out of my pocket just as a public service.

What would be great is if anyone can join the admin team to do some actual man hours work like, respond to new users joining the server, checking its functional and messaging me if not, blocking a user here or there if we get a complaint and so on. All of these tasks have become harder as the admins we used to rely on together have lost interest.

Finally if nobody is able to help, and the expectation is this simply continues as is for free indefinitely then that is totally fine also. The server is in no danger of being turned off or anything like that, requests and issues that come up will just take longer to be fixed as a result. To be clear I dont mind, if I cant use the lemmy server I just login to it and fix it and then carry on my merry way. This mostly affects people who cant do that.

Just wanted to make an appeal for some admins to improve the areas we are weakest but also clarify nobody is under and obligation to do anything and the server remains free regardless. The weakest area is not enough humans helping by messaging and opening communication lines if there are issues. If you can type a message and send it to someone, use this server and want to make it better then you are qualified to help.

Any help is of course greatly appreciated. My commitment is that you will never be expected to do anything to ‘keep it going’ but I also sleep, work, and so on so this is why I am asking. Totally up to whatever anyone feels comfortable with. Zero obligations.

Thanks -P

A short while ago I asked about a youtube alternative. While I was happy to try odyssey a small subset of commenters here were fairly cynical that any solution could be developed that could challenge a big company like YT and some even suggested just paying google.

This is the original question: https://lemmy.myserv.one/post/3116340

I am now happy to learn that in fact a solution has been developed which I will be trying. In fact it is presented in a YT video: https://youtu.be/5DePDzfyWkw?si=eMGHNGsxICcLmFOW

It warms my heart to see that solutions you difficult problems can be found and that we as thinking people and as humans still have the ultimate ability to change the course of the ship. I will be giving this new video app a try that similar to the messenger apps looks like it has a chance to begin the slow eroding of a single entity having all the content. Many years I am sure but feel like my question was answered even if not on Lemmy.

Thanks everyone:)

Looks like its over for me and youtube. Being told I cant watch because of an ad blocker.

Where is everyone moving to and using instead of youtube? I will just move to the same place.

Maybe some of you can speed it along so we can see what it is.

If you like Debian and like to help make it better, and also want to use testing please review this guide where I try to cover many common things and improve your experience:

https://makedebianfunagainandlearnhowtodoothercoolstufftoo.computer/doku.php?id=start

Thanks for being a Debian testing user :)

Server didn’t need more memory but I upgraded the memory anyway to 8gb just because I felt like it.

This server is still very underutilised and I havent even had to upgrade to 8gb ram yet which is something I have been waiting for. Uptime is now continuously above 99% so if anyone has any friends they want to invite please do. The server doesnt take much work and is cheap and simple so no reason to fill it.

Any issues let me know. For pictures loading this will be improved when stable version of pictrs is released as I will change to image proxying. This is not currently supported in 4.0 and only arriving in 5.0.

Any other performance issues let me know. Im not aware of any.

In order to keep lemmy.myserv.one clean and the rest of the fediverse, it would be helpful to have members who are interested to assist in checking new user accounts that are created are not troll/spam accounts.

Currently when a new user signs up, an email is generated to admin staff with their username so that after a suitable delay we can review their comment history/posts and delete any user who is just a spambot or a troll looking to harm communities.

So far just one user has had to be removed in this way, but I expect there to be more as time goes on. In addition you may be contacted by other instances admins/mod teams from time to time via DM asking to ban a user if they have egregious behavior. This has never happened yet, but eventually one day I am sure it will.

If you are interested in helping please DM me so I can review your account (ie you are a normal person) and have been posting/using the server and have a vested interest in helping out.

Kind regards P

Unfortunately there will be some downtime later as I work on optimisation of the database which is a problem for lemmy instances at the moment and will mean the site is inaccessible for an hour at some point later while I take a dig around and aee what O can change.

Maintenance time has been moved from 3AM daily to 3:16AM daily. Maintenance takes about 10 minutes. EDT time. This was done because I needed to script something else first. Tnx.

To combat email issues over the past week I have upgraded the email setup and security so it will hopefully be more reliable.

P

Currently this instance is run by just me, however there is currently an effort started to figure out what happens if I am away on holiday and there is an emergency/site down situation with other trusted instance admins.

It will be a long work in progress, starting simple but eventually enabling another trusted admin to do everything to get the server back online including start/stop the vps via api calls if it has died.

To begin with I am working on this with just 1 other admin for now - the admin from boulder.ly - who is creating a docker image that will allow us to connect to each other instances and resolve basic problems. This will be extended as we have time to add them.

Also I am having to create the workflow and documentation on what to do in various scenarios and the commands and processes and procedures to do them.

Eventually also (although not being worked on now) we will need to consider how bills are paid if another admin dies/drops off the face of the earth and how that process might look. For now we will have to leave this until the more basic stuff is in place.

This is a complex procedure for various reasons especially revolving around access rights and permissions so I will keep you updated as we progress. It will be some weeks before the basic stuff is ready but main thing to know is its a problem being worked on so that we are not in the situation where an instance is run by just 1 person who cant be available 24/7 and there will be eventually some way to protect users if real life happens to the owner of the site.

As there is a lot of work to get even a simple setup going please be patient, it is being worked on slowly and will be a constant work in progress for some time.

Have been getting some training from other instance admin on how to configure this. Will require a lot of changes to accomplish this change. So will be looking into setting this up later next week once I have accommodated all the basic stuff that needs to be done first. Once this is ready to be done I will let you know then deploy it to this site and (hopefully) not cause downtime during the change.

Pete