I managed to get it to run. I checked in postgresql logs and it seems it was caused by trying to add into site. Apparently the site name cannot be more than 20 chars but this is mentioned nowhere in the documentation.
And when this happens, it causes this cascade of somehow trying to recreate the admin again and again, which is a red herring after all.
Yeah, I encountered this too, but only when deciding to rename the site. When I renamed it through the lemmy UI I was able to find the character limitation and work around it. I bet it has something to do with the federation protocol, and how much memory is allocated to a site's name.