Huge thank you! I had a feeling something like this was going on but had no idea how to troubleshoot/fix.
My pictrs
and lemmy
containers were the biggest between 3-8 GB (significant for a smaller instance) after a couple weeks.
For anyone who finds this, in addition to what OP provided here, another command I found helpful (since I am a docker noob π) to find the name of the currently running containers:
docker ps --format '{{.Name}}'
After some tinkering, **yes ** this indeed was my issue. The logs for pictrs
and lemmy
in particular were between 3 and 8 gb only after a couple weeks of info
level logging.
Steps to fix (the post above has more detail but adding my full workflow in case that helps folks, some of this wasn't super apparent to me) - these steps assume a docker/ansible install:
-
SSH to your instance.
-
Change to your instance install dir
most likely:
cd /srv/lemmy/{domain.name}
- List currently running containers
docker ps --format '{{.Name}}'
Now for each docker container name:
- Find the path/name of the associated log file:
docker inspect --format='{{.LogPath}}' {one of the container names from above}
- Optionally check the file size of the log
ls -lh {path to log file from the inspect command}
- Clear the log
truncate -s 0 {path to log file from the inspect command}
After you have cleared any logs you want to clear:
- Modify docker-compose.yml adding the following to each container:
logging:
driver: "json-file"
options:
max-size: "100m"
- Restart the containers
docker-compose restart
UPDATE:
If anyone else is running into consistently rising disk I am pretty sure this is my issue (RE logs running with no cap):
Trying out ^ and will update with my findings if it helps.
Gotcha thanks! Thats good to know. Based on the originating ticket: https://github.com/LemmyNet/lemmy/issues/1133
Sounds like it might be safe for me to purge that table a bit more often as well.
Dumb question, how are you profiling (RE your mention of getting a better idea of which tables might be bloated) your DB? Just SSHing into your box and direct connecting to DB? Or are there other recommended workflows?
Tips for "Care and Feeding" of my Instance - Disk Usage Rising
Any helpful tips for general care and feeding I should be doing on a regular basis?
I know I need to keep an eye on updates and re-run my ansible setup form time to time to stay up to date.
But I have also been keeping an eye on my VPS metrics to see when/if I need to beef up the server.
One thing I am noticing is a steadily increasing disk utilization (which mostly makes sense except its seeming a bit faster than I expected as most all media is links to external sites rather than uploading media directly to my instance).
Anything I can do to manage that short of just adding more space? Like are there logs/cached content that need to be purged from time to time?
Thank you!
Classic HC Plans?
Folks been planning their HC toons/profs for when official drops?
Been trialing a few on unofficial Bloodsail and having a blast.
Feeling good about Priest with ench (until GMW) and then tailoring/skinning.
Anyone else a D4 alt-o-holic?
I get to like 20/30 and then want to try another class. Loving the mechanics on all of them! Hard to settle.
π man ima dumb dumb. Most excellent, thank you!
Any way to "allow all" on a new instance?
I have just been hand-jamming lemmy domains I know about in the allowed instances
section of my new instance.
Is there a way to just allow all? Or maybe a running list somewhere of domains I can copy/paste?
Seeing a bunch of:
WARN Error encountered while processing incoming HTTP request: lemmy_server::root_span_builder: Domain is not in allowlist
Flowing through my logs and from what I can tell, it doesn't seem to tell me WHICH domain I should be trying to add π₯²
Any thoughts? Thank you!
How to make NSFW Communities Discoverable?
It is my (perhaps incorrect) understanding that NSFW communities are hidden from the global search unless you are registered/logged in at that instance.
This makes it tough for federated users to find your stuff.
Any tips on making that easier? For example, I am registered at lemmy.ml
but wanna see y'alls
goodies
ποΈπποΈ
I also run an instance that allows NSFW content so curious for helping discoverability of that instance as well. Was thinking of maybe adding a 'site map-esque' concept to a general/welcome community's sidebar with the appropriate search strings.
ie, something like:
If you'd like to subscribe to a community put this in your search bar...
Community 1 is blah
!{community_name_1}@{instance_url}
Community 2 is blem
!{community_name_2}@{instance_url}
Community 3 is shmlep
!{community_name_3}@{instance_url}
Any better way?
Thanks!. Yeah I gotta figure that out. Something funky going on.
There are several communities in there atm but I'm getting some "odd" federation behavior.
If you are interested in federating there are two ways I have seen it work (again, warning, note that it is NSFW π ):
-
use your search feature on your main instance and enter
[email protected]
then you can subscribe -
or, from your main instance, alter your url to:
{your main instance}/c/[email protected]
so for example if your main instance islemmy.ml
that would be:https://lemmy.ml/c/[email protected]
^ butt_frenzy
is one of the communities on booty.world
for example.
Gotta be a better way, and I am probs doing something wrong. Will work on it. Open to feedback if anyone has any advice though haha.
Thanks.
Noice many thanks! I'll give it a shot.
Oh snap, yes I am using Digital Ocean.
Looks like you might have forgotten to add the link, mind chucking that in when you get a chance?
Thanks, this seems like a good lead as well.
Cool thanks! Last thing for now, and then I'll go do some debugging.
Dumb Q: The lemmy install includes an SMTP server correct? Like I don't need to use an external service for that?
Nice! Thanks your mention of the config.hjson
makes me wonder lol. I probably goofed that too.
I did the ansible install which I believe just adds orchestration on top of the Docker install. I'll ssh in and try the docker-compose
command.
Do you know, if I did goof the email config, can I just tweak the config locally and then re-run ansible? Or do I need to do some manual tweaks to the deployed solution? Or I suppose at this point it might be easier to just blow the instance away and start fresh.
Help Troubleshooting Email Verification on Self Hosted
Any good resources/Discord servers (to discuss with other devs) for troubleshooting user login/email verification issues?
I have a self-hosted instance and was able to make the admin user, but when trying to create a new test user, the user does not get the verification email.
I assume its related to this issue because I see the same issue but on login rather than sign up: https://github.com/LemmyNet/lemmy-ui/issues/1080
I am able to create the new user but get no email verification and then when I try to login the login button gets stuck with the spinner. No frontend console logs so I assume something is dying server-side and not bubbling the error.
Really just looking for some guidance/links to docs on how I can access logs to troubleshoot server-side errors (and my gut tells me maybe an email issue but I am a novice in this space so not sure).
Any help is appreciated.
Thank you.