I'm aware that some of you have been getting some errors loading this instance. This was a configuration that I needed to be adjusted which has since been.
Do be patient if we run into other issues, I'll be continuously working in the back-end with others to better your overall experience.
Still a lot of capacity available!
edit: For those that are interested in the configuring changes. I was using the default lemmy configuration for the nginx worker_connections value. This value needed to be raised.
Shout out to @[email protected] over at lemmy.world for helping me out.
Just out of curiosity, would it help if when posting images we use services such as imgur (or alternatives)? I’m assuming if there is storage issues those type of posts are the biggest culprit.
As someone who is still hosting a forum, I would not suggest this. I have also tried to reduce footprint of the forum using external image hodting services and this has always ended up badly (images get lost as one hosting service changes policy and maybe disables direct linking or just closes it's doors, whatever).
This is one of the reasons why I'm not tempted to open up a lemmy server, even though my hosting plan allows 3 subdomains. Used space will rise quickly, even if it's just images... if videos are allowed as well, than all hell will break loose, even if they are processed and reduced in quality and resolution server side. I've seen it happen before, it's a nightmare to revert things afterwards (users complain), not to mention you can't revert the damage, that space is taken and that's that.
My estimate is that, if only images are allowed and images are processed in webp, it'll take about a week for a quite busy instance to reach the 1GB mark... probably a lot faster if it's an NSFW instance (a few days). Think about this from a migration perspective after 1 year - it will be a nightmare.
I figured out the storage issue last night, the instance is only at about 20% utilization so we should be good to take on a good amount more. We probably will need to do some more tweaking as we grow but for now its looking pretty good!
Think long term. Don't take too many new users or you'll end up with loads of new content (mainly images). Storage is not expensive nowadays, but you can't load new disks to the arrays indefinitely.
Thanks for the update! Any plans for a separate channel outside of sh.itjust.works like Discord or Mastodon, just in case to give people a heads up if the server goes down or is in maintenance?
Not sure if you already did or the default config does it but if you only have one backend in an nginx reverse proxy it might also make sense to configure the max_fails and possibly the fail_timeout options so nginx won't consider your backend down for a few seconds every time it receives a TCP error connecting to that (single) backend. max_fails=0 in the named upstream section (see http://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream ) is what you want here.
Thanks @TheDude! Scaling systems like this is always a challenge and you really get to learn the performance quirks of the code. Thanks for all your work.
@[email protected] does Lemmy support a distributed configuration with multiple database and app servers, or are you limited to a single instance of everything?
The official supported deployments are single instance based, that being said things are already broken up into separate docker containers so it should be pretty easy to do. Would need to do some testing before hand. If this instance continues growing this way I'll need to look into scaling horizontally instead of vertically
On desktop it will eventually load more stuff, after minutes if I just let it sit there. On Jerboa, nothing at all. Other sorting methods seem to work, but are weird or hard to use. New makes everything keep moving so I can't keep my place. Active is just weird.
I guess I can make an count on another instance as a backup.
Absolutely. At the same time, when you have a bunch of users migrating from a centralized platform who don't quite get the fediverse stuff yet, they're likely going to go to well-populated instances.
Hopefully as things progress messaging around this point will penetrate with new folks (like me!)
Thanks for the great server!
I've read your sticky post saying the storage was the most unexpected ressource going up. Can you share what kind of storage space your talking about? I could in the future be interested in spinning up an instance, and I haven't seen any documentation or info on that.