[Done] New try at upgrading to 0.18.1 July 1st 20:00 CET
[Done] New try at upgrading to 0.18.1 July 1st 20:00 CET
We'll give the upgrade new try tomorrow. I've had some good input from admins of other instances, which are also gonna help troubleshoot during/after the upgrade.
Also there are newer RC versions with fixed issues.
Be aware that might we need to rollback again, posts posted between the upgrade and the rollback will be lost.
We see a huge rise in new user signups (duh.. it's July 1st) which also stresses the server. Let's hope the improvements in 0.18.1 will also help with that.
You know, maybe I'm being overly dramatic, but I really feel like we're at the beginning of something special. I realize that Lemmy has existed for some years, but the whole Reddit thing is really pushing it into the limelight, and it's new to me (I joined just before the blackout).
Seeing all the developers and admins dealing with the inevitable bugs, working together to keep the federated instances operating together during a huge surge in users, like a flotilla of disparate ships lashed together facing a major storm - it makes me excited, nervous, and hopeful. I fully intend to support developers and instances financially, but also want to give a big "yay team" shout-out.
Not dramatic at all. This feels like starting fresh, and the enthusiasm and excitement makes it feel like the early 2010s of the internet again. I love it 😍
Totally agree. Alas, I'll need to find time to help with time/knowledge. (that's my best chance, cash is an issue)
Time and knowledge are probably more important. A lot of well meaning people are copying over a bunch of Reddit content into communities here, but I think that's only useful in a small subset of them. If you're looking to scan through a bunch of memes, okay that's great. But a lot of people are looking for discussions and content to interact with, and a giant number of posts with no comments isn't helping that.
So starting or contributing to real discussions (like this one!) is a big contribution.
I think the solving the bugs and server issues is great to see, but I also think Lemmy is facing some deep issues about its design in the future, and it's going to take more than technical know-how to fix it.
Yeah, I agree. I think they can be worked out, but I don't think they have been yet. I think there's a lot of potential in the distributed architecture, and am excited to see it progress.
This is like that scene in a sci-fi movie where they're trying to repair the ship mid-battle. Good luck!
I used to work with a guy who was a sysadmin for a moderately sized webhost in the mid-2000s when things like containers and cluster orchestrators didn't exist and high availability/multi-master database systems were only really accessible to banks.
He once described patching the servers "like trying to change all the tires on a car without being allowed to pull over".
...as an IT guy who started my career as containers starred to be a thing, some of the stories from the late 90's early 2000's are bonkers. We have it so much easier with things like k8s handling the orchestration.
Watching lemmy react realtime is hella interesting, good luck y’all!
Admins be like:
PSA from Admin Team: The update completed roughly two hours ago. Since that time, the Admin team (and other site admins) have been working on the noted performance issues. We believe we have found a solution, but we still need time to test this out. You may still see brief outages and differences in performance as we are testing different configurations. We are trying to prevent rolling back.
While I know this can be frustrating - especially today - please keep in mind we have a team of volunteer techies (from around the globe!) collaborating on this issue. It is an inspiring situation. Also keep in mind that lemmy.world is quite a bit larger (and more active than any other instance). As such, we are a bit of a ‘test instance’ in regards to high volume requests. This is just part of the growing pain. We appreciate your understanding.
@[email protected] will provide a debrief once we have completed testing.
Godspeed. I really hope this is the one and the age of randomly redirecting to sports game threads will be over!
Surprise MLB got you down? Too bad, we have orbital space lasers!
*zap!*
Yeah the random refreshes on desktop is almost unusable. Fingers crossed!
the age of randomly redirecting to sports game threads
Glad I wasn't the only one experiencing that.
That should be a conspiracy.
Thank you for the heads-up. 🤙
Is it wise to do a big deployment on the day when you may receive a massive influx of new users from Reddit? The API changes take affect on the 1st.
It's a bit of a lose lose situation tbh because if we don't upgrade then the first experience those new people are gonna have of Lemmy is an unusable All feed and a broken Jerboa.
My other account is on lemm.ee and the admin there contributes a lot of fixes to the software (plus has been working with Ruud to figure out our federation issues). He upgraded us a couple days ago and it's been amazingly stable given that it was only a day after the rc1 disaster.
Ultimately I trust that he and Ruud know what they're doing, but sometimes with an untestable situation like the size of .world you just have to pull the trigger and hope it works.
Probably not, but surprisingly some 3P apps still work. I can confirm that Joey for Reddit and Infinity for Reddit are still able to access reddit. Seems like they don't pull the plug in one go? Or cherry picking the most demanding apps first?
For context, it's July 1st 16:27 p.m., GMT+9 where I am.
There might be less users influx than expected, though we'll see in the next few hours or days.
All of the apps will work until the devs pull the plug. Maybe those devs have decided to try to eat the cost and offset with subscriptions or something. I know the Apollo dev said it would cost him twenty million dollars per month to pay the API fees, so hopefully the devs of those other apps don't go bankrupt overnight.
Well website crashing all the time and jerboa crash loops. Woop, at least I've found a working alternative client now
Which one?
Hey man if y'all have planning meetings or anything for this stuff I'm willing to jump on and read the source code. 15 years engineering and dev, willing to put time in for you guys.
You might want to DM @[email protected]
Sent thanks
0 years of experience willing to do the same. Me basically this meme https://i.kym-cdn.com/photos/images/newsfeed/001/446/009/d10.jpg
The lemmy code is on github.
Good luck. This is my favorite instance so I will be eagerly awaiting. Lemmy has been incredible and strangely what I needed. I would doom scroll reddit and see the same posts and the same comment chains and just redundancy hell.
Lemmy feels so different and the community is so welcoming. Thanks again everyone for making this lost Redditor a found lemming (is that the correct term lol).
I would doom scroll reddit and see the same posts and the same comment chains and just redundancy hell.
Ugh, so true. On Lemmy, it feels like it's actually worth it to qualitatively engage with people.
Agreed. I think the user base is a big part of it. Almost by definition, this place is full of people that used to use third party Reddit apps. That group would have been dominated by people that signed up before Reddit launced their official app in 2016, so accounts that are at least seven years old, and in many cases, even older (my first Reddit account dates to 2011). That means (1) users who were teens or adults in 2016 that are now seven years older and hopefully seven years wiser, and (2) users that participated in Reddit when it was more discussion forum oriented rather than the current form of quick dopamine hit memes.
I'm nearly 40, and I suspect a decent amount of the users over here are too (i.e., the group of twentysomethings that discovered Reddit shortly after the Digg situation). Children and teenagers that signed up for Reddit within the past two or three years would be on the official app without a reason to migrate, so hopefully, that means this place is mostly adults willing to have respectful conversations with one another. We'll see if it lasts, but that's the impression I get now. It's refreshing.
Fingers crossed. Would love for the auto refreshing to finally stop.
FYI, you can mitigate this by specifying page zero in the URL. As an example, if you're looking at the "All" page on Lemmy.world, the URL looks like this:
https://lemmy.world/home/data_type/Post/listing_type/All/sort/Active/page/1
Just change the
1at the end to a0and hit Enter. No more refresh bug.
I can only wish you the best. You are an amazing person for signing up for all of this.
Thanks for all your hard work Ruud and team. This place is seriously shaping up well to completely replace my Reddit addiction.
Looking forward to it. The lag has been brutal.
Keeping my eyes crossed for you!
For those that were about to Google it... 20:00 CET is 12pm PDT and 3pm EDT
18:00 UTC would be a better time to report for an international server. (on the other hand, people who can fund us can DDG the timezones as well ;) )
I'm pretty sure OP means CEST, since we're in summer. So that means you need to subtract one more hour.
Waiting for the update so I can sign into Jerboa.
Lemmy looks like it's shaping up nicely to take over my Reddit habit
Same here. Thought I was doing something wrong, being a very fresh reddit-refugee.
Meanwhile I'm trying out connect for lemmy. Seems pretty good so far, too, in case anyone wants to give it a try as well.
Use liftoff for now
I hope it works! I'm so over posts just spawning into a page that is already loaded.
Performance is extremely uneven. Sometimes loads instantly, sometimes I get a timeout. Upvotes don't show up until reload. Still a lot better than the last attempt.
The server hamsters seem to be on fire, but I won't hold it against the service/community/individuals (except spez; fuck you spez). Lemmy is in the unfortunate position of being forced into a development model called "fuck it, we'll do it live!"
edit: commenting and editing seems to be fine.
Latest version of Jerboa on Google Play doesn't work with servers below 0.18;
For those who want to use Jerboa to access lemmy.world before the upgrade to 0.18.x happens, you can install a specific release from GitHub instead of Google Play. I found version 0.0.34-alpha to work (using it rn to post this). https://github.com/dessalines/jerboa/releases/tag/0.0.34
Edit: since the update is done, it's probably best to go back to latest version. Glad I helped someone!
Godspeed you absolute madlads.
o7
@[email protected] Your energy is amazing and commendable! Your Fediverse hobby is like what most people call a job! Thank you so much for all the hard work and attentive management, and also thanks to those that are helping! I hope more folks take a moment to look into the options for donating to the cause via Patreon and the other options you have. Excellent work ethic!
Godspeed. I really hope this is the one and the age of randomly redirecting to sports game threads will be over!
This should be some sort of conspiracy.
May the odds be ever in
yourour favor.Good luck! Really looking forward to the new version
This is our home now. May the road rise to meet us. And FUCK Spez
Good luck! Posted 19:59 CET.
Y'all are so nice and transparent here. This place is lovely, unlike Reddit and their admins lol.
I sadly couldn't use the web version of lemmy.world at all cause of the websocket issues. 0.18+ is gonna be an amazing upgrade fore this instance! All the devs and staff who made this happen have my undying respect.
Bon courage mon ami with the upgrade. You guys rock.
Hopefully third time's the charm and the upgrade will be successful.
Most issues were caused by browsers trying to access cached code. A recent change introduced changed paths for each new release, so browsers will fetch the new files. At least those issues should be a thing of the past.
o7
Thanks for giving me a new home.
It’s awesome to see some teamwork between server admins. Best of luck with the upgrade, hope it goes smoothly for you this time!
Good luck! 🍀
I’m going to begin supporting this instance on Patreon starting this month, I guess symbolic of the changing times and server demands.
Fingers crossed and thanks for all the transparency!!
Looks like it works, but it's very slow - 25 seconds to load the page. https://i.imgur.com/ghiofDC.png
o7
May god help us all
looking good so far, the instance is back, testing to post!
GL! And while at it, a big thank you for running (and fixing occasional server oopsies) this instance. :) I started to donate for the server costs, even if 1€ is mostly symbolic
If everyone on the server donated $/€1 per month we'd be Gucci.
Looks like the update is showing, but it's been slow since. Hopefully not permanent. Also having issues upvoting.
My outdoor BBQ Im hosting starts at the same time as the lemmy upgrade. See you all on the other side and good luck LW admin!
o7
let's fucking go!!!
Seems to have worked! Typed from Jeboa :)
Just a heads-up. I ended up having to create a new login at lemm.ee because even after the improvements in speed and so on with the upgrade, for all intents and purposes it looks like my account here may have been wiped. Can't login from anywhere, getting "Record not found" whenever I try to sign in on Connect, "incorrect login" from Jerboa, and a spinning button and then nothing on a PC browser, basically treating my account as if it doesn't exist. Some folks appear to be able to login though. Is this an ongoing server issue, or did my account actually get wiped? Same username (Grangle1) as on lemm.ee.
Stay on target.....
Crossing my fingers, excited for the update
We have almost the same nickname!
So it was you took my normal username haha
Thanks for everything you do, [email protected]
Crossing my fingers for everything to work out well. I’m looking forward to being able to log into Jerboa.
Me too! I miss Jerboa.
Good luck! And thanks for keeping us updated with what you're all doing, I don't know why but it makes me feel part of something big :D
Something tells me it will work this time.
Third times the charm, baby!
Looks like it worked! Thanks for all you do @Ruud!
I am probably one of many refugees coming from Reddit. Here is hoping lemmy grows and grows well.
Amen, me as well.
Success?! Hello from 0.18 :)
The UI is a lot smoother. Feels good man.
Edit: servers are very sluggish though. Hopefully this gets fixed without needing to roll back.
May the surf be calm and the sailing smooth.
o7
Goodluck and thanks for the effort in all of this guys.
Wow, this is much faster now that caches are built.
Thanks Ruud. In June I donated to the Lemmy Devs and in July I’ll donate to Lemmy.world I appreciate all of your work.
Did I just check lemmy right after the reboot? Good timing huh
Godspeed, commander.
I am probably one of many refugees coming from Reddit. Here is hoping lemmy grows and grows well.
Finally able to register. Glad to be here.
o7
See you on the other side
Is this happening now? Everything seems up and running
EDIT: absolutely perfect timing
Good luck!!!
Great! Let's hope it goes well since many users will default to jerboa when migrating.
Good luck!
Hey man if y'all have planning meetings or anything for this stuff I'm willing to jump on and read the source code. 15 years engineering and dev, willing to put time in for you guys.
Good luck.
Can't login anymore. I'll just use my .ca account for now while things get sorted. Not a big deal. These things take time.
I'm here to stay 🤟
No problem if you have to wait, it is the best instance. Do what you have to do.
No problema si toca esperar, es la mejor instancia. Haz lo que tengas que hacer.
Seems to have worked! Typed from Jeboa :)
Godspeed
Good luck, we’re all counting on you
Muuuch faster now. Thanks for all the hard work guys!
Seems to have worked! Typed from Jeboa :)
Good luck! Thanks for all your hard work
God speed
May the force be with you. Live long and prosper.
Thanks for the heads up, and good luck!
Update went good so far ;)
Looking forward to the update. Jerboa for Android as available on F-Droid has 0.18 as a minimum requirement. Really wanna get that one set up for this Account.
looking good so far, the instance is back, testing to post!
🤘🤞
The last few minutes the site seems to be very fast! Let's hope it stays this way!
Best of luck!
Thank you so much. Im happy to be here
Just looked. First 1/2 loads were slow but after that it's lighting fast! I think by not everyone establishing a Websocket connection and just loading once performance should increase a tad bit.
My lemmy server is also running the same versions are you are now. I was getting timeouts with lemmy.world federation so I recompiled lemmy_server with timeout changes:
diff --git a/crates/utils/src/lib.rs b/crates/utils/src/lib.rs index e5d07db2c..e2c592d82 100644 --- a/crates/utils/src/lib.rs +++ b/crates/utils/src/lib.rs @@ -18,7 +18,7 @@ use std::time::Duration; pub type ConnectionId = usize; -pub const REQWEST_TIMEOUT: Duration = Duration::from_secs(10); +pub const REQWEST_TIMEOUT: Duration = Duration::from_secs(13); #[macro_export] macro_rules! location_info { diff --git a/src/lib.rs b/src/lib.rs index cc77ca48f..45c621a7c 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -37,7 +37,7 @@ use tracing_subscriber::{filter::Targets, layer::SubscriberExt, Layer, Registry} use url::Url; /// Max timeout for http requests -pub(crate) const REQWEST_TIMEOUT: Duration = Duration::from_secs(10); +pub(crate) const REQWEST_TIMEOUT: Duration = Duration::from_secs(16); /// Placing the main function in lib.rs allows other crates to import it and embed Lemmy pub async fn start_lemmy_server() -> Result<(), LemmyError> {Good luck with this!
thanks for keeping us informed :)
yay, instance is back! trying to post timed out two times, this is the third try.
Happy to be here. o7
Hoping the update works well this time !
Hey man if y'all have planning meetings or anything for this stuff I'm willing to jump on and read the source code. 15 years engineering and dev, willing to put time in for you guys.
o7
Thanks for the heads up! Fingers crossed indeed!
Super responsive, and I can login using Jerboa 37!
Here for it! This is the last chance to catch the wave of signups, if there's gonna be one.
Working for me currently on connect for Lemmy.
Thanks
The progressive web app had ui issues after the update so I uninstalled it and now I can't figure out how to reinstall it 😕 tried wefwef but that doesn't load anything and can't log in. Happily staying patient though, hoping things smooth out!
Thank you so much for your hard work! I deeply appreciate it.
Can't wait
Thanks for all your hard work Ruud and team. This place is seriously shaping up well to completely replace my Reddit addiction.
Good luck!
Page me if you need any halp
o7
Good luck!
Thanks for all the work you're doing, best of luck!
Working super slow. Like 45 seconds per page load.
o7
Good luck
New update sure is fire, post making process page is all cleared up!
Best of luck! I really appreciate the transparency on here that you keep providing.
Best of luck!! “crossing fingers”
yay, instance is back! trying to post timed out two times, this is the third try.
Thanks and good luck!
Good luck! I know you and the team will do the best possible work to get everything updated
Thank you for all of your hard work!
my fingers are crossed for you, good luck. o7
Thanks for all the work you're doing, best of luck!
Hell yeah
Good luck!
wew. noice!
May the git repos bless the server 🙏
Good luck, hope it works this time
You got this ruud.
o7
o7
Thoughts and prayers! /s
Seriously though, good luck!!
I hope it helps with the speed. It's like trudging on molasses right now, but can't complain much.
Thank you for all of your hard work!
O7
o7
GLHF
Good luck!!!
Nice work with everything so far, really liking it. What all is being changed/fixed in the new revision? Is it just more efficient code to support higher site traffic?
Super responsive, and I can login using Jerboa 37!
It's working! Writing from jerboa.
It's working! Writing from jerboa.
I get an error message when replying to comments tho.
Thanks everyone for your hard work!
It worked this time, nice!! Feels a lot better, much more responsive.
Things look better. Thank you for the updates!
Instance got faster now, but still impossible to log in after you've cleared cookies.
A lot of improvements, but some weird problems. Jerboa finally allowed me to log in but not much else. Connect is working better, but won't allow me to reply to one particular post ("incorrect language" or something). The browser version and wefwef are slow and glitchy - my comment about birds ended up in a conversation about Lemmy apps. I don't know anything about tech stuff but I get the impression that people are working very hard to fix all the issues. And the spezidus wouldn't be making things easy. Looking forward to a bigger better Lemmy!
How loaded are the servers? It would be good to see it maybe in the instance sidebar or a website
Looking awesome! faster, more responsive, new sorting types, tighter layout, no auto-load of random posts 🥳 🥳
Just need to sort out the wefwef issue, and we're back in business...
Was it succesfull?
Nice work!
o7
Pages loading slowly here and sometimes not loading content. Could be due to the number of new users.
I like the new layout better, just hope everything keeps working
Super responsive, and I can login using Jerboa 37!
entire instance proceeds to fucking obliterate itself
Srs though, hope it works well.
Good luck!
Looking better! Let's go!
Good luck, and Godspeed.
o7
🙏
Godspeed friend haha
I feel like that time. On that day. Is a bad time to do something that will likely end up with people creating accounts being turned.
How this goes through this time.
You couldn't have picked a worse day to try something risky. There's going to be a huge flood of newcomers tomorrow, and if things aren't working, it could send them straight back to Reddit.
Things aren’t working perfect right now either. The random sports score bug is quite confusing
I don't have that in Jerboa or Liftoff... Just lots of random crashes mainly
I feel like that time. On that day. Is a bad time to do something that will likely end up with people creating accounts being turned away.
Oh.... that's why it's unstable right now.
How this goes through this time.
How this goes through this time.
How this goes through this time.
How this goes through this time.
Bailing out, you are on your own. Good luck.
