Search

cross-posted from: https://lemmy.cafe/post/1403198

> #### Overview > > This is a quick write up of what I had spent a few weeks trying to work out. > > The adventure happened at the beginning of October, so don't blindly copy paste > queries without making absolutely sure you're deleting the right stuff. Use > select generously. > > When connected to the DB - run \timing. It prints the time taken to execute > every query - a really nice thing to get a grasp when things take longer. > > I've had duplicates in instance, person, site, community, post and > received_activity. > > The quick gist of this is the following: > > - Clean up > - Reindex > - Full vacuum > > I am now certain vacuuming is not, strictly speaking, necessary, but it makes me > feel better to have all the steps I had taken written down. > > \d - list tables (look at it as describe database); > > \d tablename - describe table. > > \o filename\ - save all output to a file on a filesystem. /tmp/query.sqlwas my choice. > > ___ > > ####instance> > You need to turnindexscanandbitmapscanoff to actually get the duplicates > ``` sql > SET enable_indexscan = off; > SET enable_bitmapscan = off; > ``` > > The following selects the dupes > ``` sql > SELECT > id, > domain, > published, > updated > FROM instance > WHERE > domain IN ( > SELECT > domain > FROM > instance > GROUP BY domain > HAVING COUNT(*) > 1 > ) > ORDER BY domain; > ``` > > Deleting without using the index is incredibly slow - turn it back on: > ``` sql > SET enable_indexscan = on; > SET enable_bitmapscan = on; > ``` > > ``` sql > DELETE FROM instance WHERE id = ; > ``` > > Yes, you can build a fancier query to delete all the older/newer IDs at > once. No, I do not recommend it. Delete one, confirm, repeat. > > At first I was deleting the newer IDs; then, after noticing the same instances > were still getting new IDs I swapped to targetting the old ones. After noticing > *the same god damn instances* **still** getting new duplicate IDs, I had to dig > deeper and, by some sheer luck discovered that I need toreindexthe database > to bring it back to sanity. > >Reindexingthe database takes a *very* long time - don't do that. Instead > target the table - that should not take more than a few minutes. This, of > course, all depends on the size of the table, butinstanceis naturally going > to be small. > > ``` sql > REINDEX TABLE instance; > ``` > > Ifreindexingsucceeds - you have cleaned up the table. If not - it will yell > at you with the first name that it fails on. Rinse and repeat until it's happy. > > Side note - it is *probably* enough to onlyreindexthe index that's failing, > but at this point I wanted to ensure *at least* the whole table is in a good > state. > > ___ > > Looking back - if I could redo it - I would delete the new IDs only, keeping the > old ones. I have no evidence, but I think getting rid of the old IDs introduced > more duplicates in other related tables down the line. At the time, of course, > it was hard to tell WTF was going on and making a *wrong* decision was better > than making *no* decision. > ___ > > ####person> > The idea is the same for all the tables with duplicates; however, I had to > modify the queries a bit due to small differences. > > What I did at first, and you **shouldn't** do: > > ```sql > SET enable_indexscan = off; > SET enable_bitmapscan = off; > > DELETE FROM person > WHERE > id IN ( > SELECT id > FROM ( > SELECT id, ROW_NUMBER() OVER (PARTITION BY actor_id ORDER BY id) > AS row_num > FROM person) t > WHERE t.row_num > 1 limit 1); > ``` > > The issue with the above is that it, again, runs adeletewithout using the > index. It is horrible, it is sad, it takes forever. Don't do this. Instead, > split it into aselectwithout the index and adeletewith the index: > > ```sql > SET enable_indexscan = off; > SET enable_bitmapscan = off; > > SELECT > id, actor_id, name > FROM person a > USING person b > WHERE > a.id > b.id > AND > a.actor_id = b.actor_id; > ``` > > ``` sql > SET enable_indexscan = on; > SET enable_bitmapscan = on; > > DELETE FROM person WHERE id = ; > ``` > >personhad dupes *into the thousands* - I just didn't have enough time at that > moment and started deleting them in batches: > > ``` sql > DELETE FROM person WHERE id IN (1, 2, 3, ... 99); > ``` > > Again - yes, it can probably all be done in one go. I hadn't, and so I'm not > writing it down that way. This is where I used\oto then manipulate the output to be in batches using coreutils. You can do that, you can make the database do it for you. I'm a better shell user than an SQL user. > >Reindexthe table and we're good to go! > > ``` sql > REINDEX table person; > ``` > > ___ > > ####site, communityandpost> > Rinse and repeat, really.\d tablename, figure out which column is the one to > use when looking for duplicates and delete-reindex-move on. > > ___ > > #### received_activity> > This one deserves a special mention, as it had *64 million rows* in the database > when I was looking at it. Scanning such a table takes *forever* and, upon closer > inspection, I realised there's nothing useful in it. It is, essentially, a log > file. I don't like useless shit in my database, so instead of trying to find the > duplicates, I decided to simply wipe most of it in hopes the dupes would go with > it. I did it in 1 million increments, which took ~30 seconds each run on the > single threaded 2GB RAM VM the database is running on. The reason for this was > to keep the site running aslemmybackend starts timing out otherwise and > that's not great. > > Before deleting anything, though, have a look at how much storage your tables > are taking up: > > ``` sql > SELECT > nspname AS "schema", > pg_class.relname AS "table", > pg_size_pretty(pg_total_relation_size(pg_class.oid)) AS "total_size", > pg_size_pretty(pg_relation_size(pg_class.oid)) AS "data_size", > pg_size_pretty(pg_indexes_size(pg_class.oid)) AS "index_size", > pg_stat_user_tables.n_live_tup AS "rows", > pg_size_pretty( > pg_total_relation_size(pg_class.oid) / > (pg_stat_user_tables.n_live_tup + 1) > ) AS "total_row_size", > pg_size_pretty( > pg_relation_size(pg_class.oid) / > (pg_stat_user_tables.n_live_tup + 1) > ) AS "row_size" > FROM > pg_stat_user_tables > JOIN > pg_class > ON > pg_stat_user_tables.relid = pg_class.oid > JOIN > pg_catalog.pg_namespace AS ns > ON > pg_class.relnamespace = ns.oid > ORDER BY > pg_total_relation_size(pg_class.oid) DESC; > ``` > > Get the number of rows: > > ```sql > SELECT COUNT(*) FORM received_activity; > ``` > > Delete the rows at your own pace. You can start with a small number to get the > idea of how long it takes (remember\timing? ;) ). > > ``` sql > DELETE FROM received_activity where id < 1000000; > ``` > > **Attention!** Do let the autovacuumfinish after every delete query. > > I ended up leaving ~3 million rows, which at the time represented ~ 3 days of > federation. I chose 3 days as that is the timeout before an instance is marked > as dead if no activity comes from it. > > Now it's time toreindexthe table: > > ``` sql > REINDEX TABLE received_activity; > ``` > > Remember the reported size of the table? If you check your system, nothing will > have changed - that is because postgres *does not release* freed up storage to > the kernel. It makes sense under normal circumstances, but this situation is > anything but. > > Clean all the things! > > ```sql > VACUUM FULL received_activity; > ``` > > *Now* you have reclaimed all that wasted storage to be put to better use. > > In my case, the *database* (not the table) shrunk by ~52%! > > ___ > > I am now running a cronjob that deletes rows fromreceived_activity` that are > older than 3 days: > > sql > DELETE FROM > received_activity > WHERE > published < NOW() - INTERVAL '3 days'; > > > In case you're wondering if it's safe deleting such logs from the database - > Lemmy developers seem to agree > here and > here.

Woke up in the morning and my selfhosted Lemmy server was basically braindead. I installed it in a Proxmox lxct container using the ansible playbook. It spams my logs with this: target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=8206186d-eaf9-486d-99ad-d9c5def188c8", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:08.001260Z WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum AnnouncableActivities lemmyohaaxyz-lemmy-1 | 0: lemmy_apub::activities::community::announce::receive lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/community/announce.rs:46 lemmyohaaxyz-lemmy-1 | 1: lemmy_server::root_span_builder::HTTP request lemmyohaaxyz-lemmy-1 | with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=dee66940-4048-4424-88c8-51cb58851eb0 lemmyohaaxyz-lemmy-1 | at src/root_span_builder.rs:16 lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum AnnouncableActivities, context: SpanTrace [{ target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 46 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=dee66940-4048-4424-88c8-51cb58851eb0", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:08.442900Z WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum AnnouncableActivities lemmyohaaxyz-lemmy-1 | 0: lemmy_apub::activities::community::announce::receive lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/community/announce.rs:46 lemmyohaaxyz-lemmy-1 | 1: lemmy_server::root_span_builder::HTTP request lemmyohaaxyz-lemmy-1 | with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=ba977ad6-03ea-46b3-a7d8-c0eb05355358 lemmyohaaxyz-lemmy-1 | at src/root_span_builder.rs:16 lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum AnnouncableActivities, context: SpanTrace [{ target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 46 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=ba977ad6-03ea-46b3-a7d8-c0eb05355358", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:10.626474Z WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum PageOrNote lemmyohaaxyz-lemmy-1 | 0: lemmy_apub::objects::comment::verify lemmyohaaxyz-lemmy-1 | at crates/apub/src/objects/comment.rs:127 lemmyohaaxyz-lemmy-1 | 1: lemmy_apub::fetcher::post_or_comment::verify lemmyohaaxyz-lemmy-1 | at crates/apub/src/fetcher/post_or_comment.rs:68 lemmyohaaxyz-lemmy-1 | 2: lemmy_apub::activities::voting::vote::verify lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/voting/vote.rs:57 lemmyohaaxyz-lemmy-1 | 3: lemmy_apub::activities::community::announce::receive lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/community/announce.rs:144 lemmyohaaxyz-lemmy-1 | 4: lemmy_server::root_span_builder::HTTP request lemmyohaaxyz-lemmy-1 | with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=0aeb2c01-23ac-492c-9693-3a9bb171a900 lemmyohaaxyz-lemmy-1 | at src/root_span_builder.rs:16 lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum PageOrNote, context: SpanTrace [{ target: "lemmy_apub::objects::comment", name: "verify", file: "crates/apub/src/objects/comment.rs", line: 127 }, { target: "lemmy_apub::fetcher::post_or_comment", name: "verify", file: "crates/apub/src/fetcher/post_or_comment.rs", line: 68 }, { target: "lemmy_apub::activities::voting::vote", name: "verify", file: "crates/apub/src/activities/voting/vote.rs", line: 57 }, { target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 144 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=0aeb2c01-23ac-492c-9693-3a9bb171a900", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:14.796561Z WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum PageOrNote lemmyohaaxyz-lemmy-1 | 0: lemmy_apub::activities::voting::vote::verify lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/voting/vote.rs:57 lemmyohaaxyz-lemmy-1 | 1: lemmy_apub::activities::community::announce::receive lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/community/announce.rs:144 lemmyohaaxyz-lemmy-1 | 2: lemmy_server::root_span_builder::HTTP request lemmyohaaxyz-lemmy-1 | with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=552c4fb1-c65c-4220-8919-430243d720cd lemmyohaaxyz-lemmy-1 | at src/root_span_builder.rs:16 lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum PageOrNote, context: SpanTrace [{ target: "lemmy_apub::activities::voting::vote", name: "verify", file: "crates/apub/src/activities/voting/vote.rs", line: 57 }, { target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 144 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=552c4fb1-c65c-4220-8919-430243d720cd", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | 2023-08-22T05:27:00.779726Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | 2023-08-22T05:30:09.026861Z WARN lemmy_server::root_span_builder: Timeout occurred while waiting for a slot to become available lemmyohaaxyz-lemmy-1 | 0: lemmy_apub::objects::person::read_from_id lemmyohaaxyz-lemmy-1 | at crates/apub/src/objects/person.rs:66 lemmyohaaxyz-lemmy-1 | 1: lemmy_apub::fetcher::user_or_community::read_from_id lemmyohaaxyz-lemmy-1 | at crates/apub/src/fetcher/user_or_community.rs:47 lemmyohaaxyz-lemmy-1 | 2: lemmy_server::root_span_builder::HTTP request lemmyohaaxyz-lemmy-1 | with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=0a4c7741-ce73-4108-a323-e7f5cc5b92ae lemmyohaaxyz-lemmy-1 | at src/root_span_builder.rs:16 lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_apub::objects::person", name: "read_from_id", file: "crates/apub/src/objects/person.rs", line: 66 }, { target: "lemmy_apub::fetcher::user_or_community", name: "read_from_id", file: "crates/apub/src/fetcher/user_or_community.rs", line: 47 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=0a4c7741-ce73-4108-a323-e7f5cc5b92ae", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | 2023-08-22T05:35:46.568991Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for hot ranks update: could not translate host name "postgres" to address: Try again lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | 2023-08-22T05:42:37.481187Z WARN activitypub_federation::activity_queue: Queueing activity https://lemmy.ohaa.xyz/activities/announce/df1ccbd5-3971-49e1-96b4-df6e73f39420 to https://pcglinks.com/inbox for retry after connection failure: Request error: error sending request for url (https://pcglinks.com/inbox): operation timed out. Sleeping for 216000s and trying again lemmyohaaxyz-lemmy-1 | 2023-08-22T05:42:51.370663Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | 2023-08-22T05:46:36.393798Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | 2023-08-22T05:53:44.661098Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for hot ranks update: could not translate host name "postgres" to address: Try again lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | 2023-08-22T06:00:21.684837Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | 2023-08-22T06:10:18.172432Z WARN lemmy_server::root_span_builder: Timeout occurred while waiting for a slot to become available lemmyohaaxyz-lemmy-1 | 0: lemmy_apub::objects::person::from_json lemmyohaaxyz-lemmy-1 | at crates/apub/src/objects/person.rs:134 lemmyohaaxyz-lemmy-1 | 1: lemmy_apub::activities::verify_person_in_community lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/mod.rs:62 lemmyohaaxyz-lemmy-1 | 2: lemmy_apub::activities::voting::vote::verify lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/voting/vote.rs:57 lemmyohaaxyz-lemmy-1 | 3: lemmy_apub::activities::community::announce::receive lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/community/announce.rs:144 lemmyohaaxyz-lemmy-1 | 4: lemmy_server::root_span_builder::HTTP request lemmyohaaxyz-lemmy-1 | with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=b7159d57-eaae-4992-8d37-0ab87f9f3adb lemmyohaaxyz-lemmy-1 | at src/root_span_builder.rs:16 lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_apub::objects::person", name: "from_json", file: "crates/apub/src/objects/person.rs", line: 134 }, { target: "lemmy_apub::activities", name: "verify_person_in_community", file: "crates/apub/src/activities/mod.rs", line: 62 }, { target: "lemmy_apub::activities::voting::vote", name: "verify", file: "crates/apub/src/activities/voting/vote.rs", line: 57 }, { target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 144 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=b7159d57-eaae-4992-8d37-0ab87f9f3adb", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | 2023-08-22T06:16:50.839271Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for active counts update: could not translate host name "postgres" to address: Try again lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | 2023-08-22T06:26:51.471879Z WARN lemmy_server::root_span_builder: Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | Caused by: lemmyohaaxyz-lemmy-1 | Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT lemmyohaaxyz-lemmy-1 | 0: lemmy_server::root_span_builder::HTTP request lemmyohaaxyz-lemmy-1 | with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=57b8b802-fc7e-4f14-87ac-89de2b0d7770 lemmyohaaxyz-lemmy-1 | at src/root_span_builder.rs:16 lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | Caused by: lemmyohaaxyz-lemmy-1 | Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=57b8b802-fc7e-4f14-87ac-89de2b0d7770", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | thread 'actix-server worker 5' panicked at 'read local site data: LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=ab4b5ccf-f8b9-4acd-9e4b-3c7966a17fd5", file: "src/root_span_builder.rs", line: 16 }] }', crates/apub/src/lib.rs:45:8 lemmyohaaxyz-lemmy-1 | note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace lemmyohaaxyz-lemmy-1 | thread 'actix-server worker 3' panicked at 'read local site data: LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=4e4ba6d1-d716-4d48-93de-e3d7772243d7", file: "src/root_span_builder.rs", line: 16 }] }', crates/apub/src/lib.rs:45:8 lemmyohaaxyz-lemmy-1 | connection error: db error: FATAL: terminating connection due to administrator command lemmyohaaxyz-lemmy-1 | federation enabled, host is lemmy.ohaa.xyz lemmyohaaxyz-lemmy-1 | Starting http server at 0.0.0.0:8536 lemmyohaaxyz-lemmy-1 | connection error: db error: FATAL: terminating connection due to administrator command lemmyohaaxyz-lemmy-1 | federation enabled, host is lemmy.ohaa.xyz lemmyohaaxyz-lemmy-1 | Starting http server at 0.0.0.0:8536

After using Ansible to upgrade Lemmy following the instructions (https://github.com/LemmyNet/lemmy-ansible) I am now getting the error

Error: LemmyError { message: Some("Cannot have both private instance and federation enabled."), inner: Cannot have both private instance and federation enabled., context: "SpanTrace" }

I am not sure how to change either private to requiring approval or public OR disable federation now. I don't see anything about it in the relevant documentation (https://join-lemmy.org/docs/en/administration/configuration.html) but I could be missing something.

The docker container keeps trying to restart and fails each time. Any help with this would be really appreciated!

EDIT: solved it by ssh'ing into the postgres docker container, running psql and UPDATE local_site SET private_instance = 'f' WHERE id = 1;

Also, I changed it back to private in the admin settings and it's fine now... really weird. Going to need to remember to do this every update I guess.

looks like the build of diesel failed. I'm using a fresh install of Ubuntu 20.04. Which Ubuntu release are y'all using?