Lemmy.ca scheduled maintenance Saturday July 6th - Complete
Lemmy.ca scheduled maintenance Saturday July 6th - Complete
Hi everyone,
We'll be undergoing maintenance on Saturday morning from about 11am - noon PST.
We've been having some stability issues over the past few days that I've tracked down to database performance, so I'll be doing some tuning and also applying all our needed system updates at the same time.
Cheers!
Update: This is all done now!
Thanks for the notice and also for keeping this whole enterprise functional.
Thanks for your hard work.
Thank you for all you guys do to keep this amazing instance running
o7
Thank you for your awesome work!
Thanks for the heads up!
Thank you for the hard work! 💪
We've been having some stability issues over the past few days
I was gonna ask about that. I had noticed several slow downs over the last couple days, but pretty brief. Glad to see it's being looked into/sorted out.
Chears, m8!
🙌
Thanks! ... great work in maintaining everything ... it's much appreciated!
PST? Do the lemmy.ca servers happen to be hosted in the Vancouver area?
I'm somewhat of a network specialist, though in wireless technology. Now I'm the head of my company's IT department so I'm a jack of all trades but would love to get some experience working on database infrastructure.
If you want some free labour, let me know how I can get in contact.
Server is in Beauharnois, but I'm in Vancouver.
Unless you're already a postgres tuning expert, I think we're all good. Thanks for the offer though!
No worries, always looking for ways to help out the community.
Cheers!
Howdy! I know y'all probably have a plan for Saturday, but was hoping you could peek at another possible issue.
While I believe this might be a client issue (Connect), some communities from other instances are not being listed properly. So, being logged into lemmy.ca and then trying to list /c/cat on lemmy.world shows a strange list of posts that resembles /c/all.
This is not a widespread spread issue and may not happen when logged into other instances. (196 is another with a similar issue here, which is annoying as I prefer to monitor that community from this instance.)
Unfortunately, the Connect dev has been missing for a bit and we cannot verify where the issue actually is. (It would suck to switch client apps .. )
Thanks in advance!
You'll have to wait for the connect dev for that unfortunately
It was kind of a stretch for it being an instance issue anyway, but thanks for the response. (These issues started after the last instance update, which does point the finger at the client in many cases. Not all, but many.)
Unfortunately, my manipulation skills are top tier. Unless you happen to have the memory span of a carrot, this issue report is now planted in your head as something to associate with any possible future issues that you might encounter. (I know how engineers think, bud.)
Ok, maybe I just have a little too much time on my hands at the moment to type out extended comments. ;)
I would love to see a write up on what was done. PostgeSQL is my favourite system and I would like to know more.
Nothing too crazy. My discovery this week is that our custom postgres conf actually wasn't being applied and I had never noticed. It got fixed a while ago but I haven't been following the upstream ansible repo - https://github.com/LemmyNet/lemmy-ansible/commit/22aa431a3d1ed8437a418633e9d92ae4af45bbf5
So just fixing up postgres to actually use the conf with our tuned wal & worker settings, along with the usual apt-get update/upgrade/reboot.