Tiff @reddthat.com Self Proclaimed Internet user and Administrator of Reddthat
regreSSHion: CVE-2024-6387
Update your openssh, now
Oh I was wrong, after further reading this looks to be a lot better than what I was thinking.
I must have been thinking about another methodology of attempted privacy over a dataset.
Before I start reading, if this has anything to do with differential privacy, I'm going to be disappointed.
Yes, you can upload images to Reddthat when posting, commenting, etc.
We have a CDN In front and aggressively cache all of the images.
3rd party images are fetched to generate a thumbnail, and to cache the image. The problem with this is, catbox can be slow at times and if that happens when you post it can't generate them.
Some clients also only open the direct links instead of showing the cached image, resulting in images not loading, or taking forever to load.
I say if you are posting on Reddthat, I'm happy for people to use the features provided by Reddthat. So upload here, if you so wish.
Just remember that the images you upload are linked to your account.
Ooooh I love the temp change idea! I'll have to find a new supplier for that, and think of a cool idea for it! 😉
A faster db. Just the regular performance benefits, https://www.postgresql.org/about/news/postgresql-16-released-2715/
Also, Lemmy is built against v16 (now) so at some point it will eventually no longer JustWork
The script will be useless to you, besides for referencing what to do.
Export, remove pg15, install pg16, import. I think you can streamline with both installed at once as they correctly version. You could also use the in place upgrade. Aptly named: pg_upgradeclusters
But updating to 0.19.4, you do not need to go to pg16... but... you should, because of the benefits!
This should be fixed now. After a re-deploy of 0.19.4 (as they repushed it ~6h after the initial tag)
We re-deployed 0.19.4 which had fixes. The emoji popup now shows up (for me on mobile)
Another one @[email protected] but not "breaking" for the 0.19.4 release
@[email protected] I've confirmed it, on voyager too (I'm away from my computer and don't have access to git, hopefully you see this)
@[email protected] I've confirmed it. (I'm away from my computer and don't have access to git, hopefully you see this)
Yeah I havn't promoted it. As moderating one platform is already enough of a time sink in life.
I wanted to promote our single platform rather than also having the option for people to come to the instant-chat platform.
But as you've asked for it has been delivered! (Also sorry for kicking you! I invited you to the wrong one. I've fixed up the links for everyone else to join)
Assuming you are talking about our regular web UI.
I managed to replicate it, once, but then i clicked on any of the other options, deleted my text and now it "works". But still loads on the far left 😩
Whats even worse, is that our current build is basically the production level of complete for version 0.19.4
Happy Reddthat Day!
It's our Birthday! 🎂
It's been a wild ride over the past year. I still remember hearing about a federated platform that could be a user driven version of Reddit.
And how we have grown!
Thank you to everyone, old and new who has had an account here. I know we've had our ups and downs, slow servers, botched migrations, and finding out just because we are on the otherside of the world Lemmy can't handle it, but we are still here and making it work!
If I could go back and make the choice again. Honestly, I'd probably make the same choice. While it has been hectic, it has been enjoyable to the n'th degree. I've made friends and learnt a lot. Not just on a technical level but a fair amount on a personal level too.
We have successfully made a community of over 300 people who regularly use Reddthat as their entryway into the Lemmyverse. Those numbers are real people, making real conversations.
Here's to another amazing year!
Tiff
PS: I'm still waiting for that first crypto donation 😜
PPS: Would anyone like a Hoodie with some sort of Reddthat logo/design? I know I would.
Two years ago, something very strange happened to me while working from my home network. I was exploiting a blind XXE vulnerability that required an external HTTP server to smuggle out files, so I spun up an AWS box and ran a simple Python webserver to receive the traffic from the vulnerable server.
Unbelievable...
How's everyone's weekend going?
Recently we have been smashing my internal goals for Reddthat.
- ✅ Fix the beta issues plaguing the site
- ✅ Fix the database issues for upgrading (oops)
- ✅ Fix LemmyWorld federation
- ✅ Finalise a new release for lemmy-ansible
- ✅ Solid uptime & page speed loads (baring the unfortunate beta issue, which unless we had found would have made it into production for everyone)
With all these stability changes now done, I am looking at ways in which we can improve Reddthat further and we are open to any ideas so please feel free to post away!
I'm toying with ideas to use the local only community feature to provide a point of difference in the landscape of Lemmy servers. Weekly events? New communities?
Either way I'm enjoying some coffee, drafting our monthly post to go into more details on the LW federation, and hopefully finding some time to enjoy some autumn sunshine!
Cheers, Tiff
v0.19.4 - Beta Bugs & Fixes
As we are running the latest and greatest Beta currently. This post will be a list of all known bugs & fun things that we are dealing with.
Please comment if you have a bug so we can help make the next Lemmy version.
Known Bugs
- Video links are not embeddable when they have a thumbnail. Issue: (UI-#2418)
- FIXED: Local and All pagination (Clicking Next) results in 25-30 seconds load times. Issue: (Lemmy-#4618)
Good News: This will be fixed VERY SOON! PR: #4662
- Federation with LemmyWorld
- We have created a server in Amsterdam which is close enough to LemmyWorld where we can ingest all of the Activities, and then batch-send them to Reddthat. Currently it is performing very well. It is still early in testing but all signs point to solving our activity backlog issue!
- Private messages are broken #2439-ui
Brane Dump: How I Tripped Over the Debian Weak Keys Vulnerability
Recent HN Thread: https://news.ycombinator.com/item?id=39976225
Reddthat Update: April 2024
April is here and we're still enjoying our little corner of the lemmy-verse! This post is quite late for our April announcement as my chocolate coma has only now subsided
Donations
Due to some recent huge donations recently on our Ko-Fi we decided to migrate our database from our previous big server to a dedicated database server. The idea behind this move was to allow our database to respond faster with a faster CPU. It was a 3.5GHz cpu rather than what we had which was a 3.0GHz. This, as we know, did not pan out as we expected.After that fell through we have now migrated everything to 1 huge VPS in a completely different hosting company (OVH).
Since the last update I used the donations towards setting up an EU proxy to filter out down votes & spam as a way to try and respond faster to allow us to catch up. We've purchased the new VPS from OVH (which came out of the Ko-Fi money), & did the test of the separate database server in our previous hosting company.
Our Donations as of (7th of April):
- Ko-Fi: $280.00
- OpenCollective: $691.55 (of that 54.86 is pending payment)
Threads Federation
Straight off the bat, I'd like to say thank you for those voicing your opinions on the Thread federation post. While we had more people who were opposed to federation and have since deleted their accounts or moved communities because of the uncertainty, I left the thread pinned for over a week as I wanted to make sure that everyone could respond and have a general consensus. Many people bought great points forward, and we have decided to block Threads. The reasoning behind blocking them boils down to:
- Enforced one-way communication, allowing threads users to post in our communities without being able to respond to comments
- Known lack of Moderation which would allow for abuse
These two factors alone make it a simple decision on my part. If they allowed for comments on the post to make it back to a threads user then I probably would not explicitly block them. We are an open-first instance and I still stand by that. But when you have a history of abusive users, lack of moderation and actively ensure your users cannot conduct a conversation which by definition would be 2 way. That is enough to tip the scales in my book.
Decision: We will block Threads.net starting immediately.
Overview of what we've been tackling over the past 4 weeks
In the past month we've:
- Re-configured our database to ensure it could accept activities as fast as possible (twice!)
- Attempted to move all lemmy apps to a separate server to allow the database to have full use of our resources
- Purchased an absurd amount of cpus for a month to give everything a lot of cpu to play with
- Setup a haproxy with a lua-script in Amsterdam to filter out all 'bad' requests
- Worked with the LW Infra team to help test the Amsterdam proxy
- Rebuilt custom docker containers for added logging
- Optimised our nginx proxies
- Investigated the relationship between network latency and response times
- Figured out the maximum 3r/s rate of activities and notified the Lemmy Admin matrix channel, created bug reports etc.
- Migrated our 3 servers from one hosting company to 1 big server at another company this post
This has been a wild ride and I want to say thanks to everyone who's stuck with us, reached out to think of ideas, or send me a donation with a beautiful message.
The 500 mile problem (Why it's happening for LemmyWorld & Reddthat)
There are a few causes of this and why it effects Reddthat and a very small number of other instances but the main one is network latency. The distance between Australia (where Reddthat is hosted) and Europe/Americas is approximately 200-300ms. That means the current 'maximum' number of requests that a Lemmy instance can generate is approximate 3 requests per second. This assumes a few things such as responding instantly but is a good guideline.
Fortunately for the lemmy-verse, most of the instances that everyone can join are centralised to the EU/US areas which have very low latency. The network distance between Paris and Amsterdam is about 10ms. This means any instances that exist in close proximity can have an order of magnitude of activities generated before they lag behind. It goes from 3r/s to 100r/s
- Servers in EU<->EU can generate between 50-100r/s without lagging
- Servers in EU<->US can generate between 10-12r/s without lagging
- Servers in EU<->AU can generate between 2-3r/s without lagging
Already we have a practical maximum of 100r/s that an instance can generate before everyone on planet earth lags behind.
Currently (as of writing) Lemmy needs to process every activity sequentially. This ensures consistency between votes, edits, comments, etc. Allowing activities to be received out-of-order brings a huge amount of complexity which everyone is trying to solve to allow for all of us (Reddthat et al.) to no longer start lagging. There is a huge discussion on the git issue tracker of Lemmy if you wish to see how it is progressing.
As I said previously this assumes we respond in near real-time with no processing time. In reality, no one does, and there are a heap of reasons because of that. The biggest culprit of blocking in the activity threads I have found (and I could be wrong) is/was the metadata enrichment when new posts are added. So you get a nice Title, Subtitle and an image for all the links you add. Recent logs show it blocks adding post activities from anywhere between 1s to 10+ seconds! Since 0.19.4-beta.2 (which we are running as of this post) this no longer happens so for all new posts we will no longer have a 5-10s wait time. You might not have image displayed immediately when a Link is submitted, but it will still be enriched within a few seconds. Unfortunately this is only 1 piece of the puzzel and does not solve the issue. Out of the previous 24hours ~90% of all recieved activities are related to votes. Posts are in the single percentage, a rounding error.
> This heading is in reference to the 500 miles email. > Requests here mean Lemmy "Activities", which are likes, posts, edits, comments, etc.
So ... are we okay now?
It is a boring answer but we wait and enjoy what the rest of the fediverse has to offer. This (now) only affects votes between LemmyWorld to Reddthat. All communities on Reddthat are successfully federating to all external parties so your comments and votes are being seen throughout the fediverse. There are still plenty of users in the fediverse who enjoy the content we create, who interact with us and are pleasant human beings. This only affects votes because of our forcing federation crawler which automatically syncs all LW posts and comments. We've been "up-to-date" for over 2 weeks now.
It is unfortunate that we are the ones to be the most affected. It's always a lot more fun being on the outside looking in, thinking about what they are dealing with and theorising solutions. Knowing my fellow Lemmy.nz and aussie.zone were affected by these issues really cemented the network latency issue and was a brilliant light bulb moment. I've had some hard nights recently trying to manage my life and looking into the problems that are effecting Reddthat. If I was dealing with these issues in isolation I'm not sure I would have come to these conclusions, so thank you our amazing Admin Team!
New versions means new features (Local Communities & Videos)
As we've updated to a beta version of 0.19.4 to get the metadata patches, we've already found bugs in Lemmy (or regressions) and you will notice if you use Jerboa as a client. Unfortunately, rolling back isn't advisable and as such we'll try and get the issues resolved so Jerboa can work. We now have ability to change and create any community to be "Local Only".
With the migration comes support for Video uploads, Limited to under 20MB and 10000 frames (~6 minutes)! I suggest if you want to shared video links to tag it with [Video] as it seems videos on some clients don't always show it correctly.
Thoughts
Everyday I strive to learn about new things, and it has certainly been a learning experience! I started Reddthat with knowing enough of alternate technologies, but nearly nothing of rust nor postgres. 😅 We've found possibly a crucial bug in the foundation of Lemmy which hinders federation, workarounds, and found not all VPS providers are the same. I explained the issues in the hosting migration post. Learnt a lot about postgres and tested postgres v15 to v16 upgrade processes so everyone who uses the lemmy-ansible repository can benefit. I'm looking forward to a relaxing April compared to the hectic March but I foresee some issues relating to the 0.19.4 release, which was meant to be released in the next week or so. 🤷
Cheers,
Tiff
PS. Since Lemmy version 0.19 you can block an instance yourself without requiring us to defederate via going to your Profile, clicking Blocks, and entering in the instance you wish to be blocked.
Fun Graphs:
Data Transfers:
The xz sshd backdoor rabbithole goes quite a bit deeper
@bl4sty: the xz sshd backdoor rabbithole goes quite a bit deeper. I was just able to trigger some harder to reach functionality of the backdoor. there's still more to explore.. 1/n it requires sending a properly craf...…
The PoC thickens
I made a graphical IRC client that runs in UEFI. It’s written in Rust and leverages the GUI toolkit and TrueType renderer that I wrote for axle’s userspace. I was able to develop it thanks to the vmnet network backend that I implemented for QEMU. You can connect to an IRC server, chat and read messa...
UEFI IRC, the perfect companion to asking why your Linux boot partition no longer exists #joke
xzbot: Notes, honeypot, and exploit demo for the xz backdoor (CVE-2024-3094)
notes, honeypot, and exploit demo for the xz backdoor (CVE-2024-3094) - amlweems/xzbot
