This Week In Destiny 6/29/2023
This Week In Destiny 6/29/2023
Lots of info in this one about the past month or so of server instability and some insight into what's been causing it.
I'll try to TL;DR/ELI5 this but I encourage everyone to read the article:
-
There is a service called "Claims" that helps run Destiny 2. It "syncs" the Destiny 2 game client to the game servers to "save" your progress. Changes were made to Claims before Lightfall to help it handle more traffic but it's not working as expected. Claims keeps getting interrupted for various reasons, which puts your game client out of sync with the server, forcing a disconnect.
-
"...Claims handles a tremendous amount of volume, routing every single kill, orb, or unit of Glimmer in Destiny 2 to the correct recipient."
-
Normally to fix issues like this they do a rolling restart to the Claims service, which causes disconnects but not downtime. This isn't working and they have to bring the whole Claims system down at the same time to restart. Without Claims online Destiny 2 cannot run, hence the repeated emergency downtime.
-
They are working to fix this issue and have listed specific patches over the next few months where they're targeting fixes:
- Mid Season 21 Update (7.1.5)
- Season 22 Launch (7.2.0)
- Season 23 Launch (7.3.0)
-
They're emphasizing that this is not something they can fix overnight or quickly. They also emphasize that the process of fixing it could potentially introduce more instability. They're implementing a lot of tracking, alerting, and testing tools to help them figure out a fix.
Next week will have increased drop rates for Exotic fish!
After that the TWID goes into the regular community stuff.
Sounds like they really screwed up with whatever they did pre-Lightfall if it's taking them nearly a year to fix. I'm curious to learn what the root cause was and how it was missed during testing. As an IT person it's interesting for me to see these kinds of case studies and the kind of stuff you have to deal with when you're working on something that has the scale of Destiny 2.
Glad to see them talking about this finally. I've never been too bothered by the server issues; of course it's rather annoying, but I know these problems are way more difficult than basically the entire player base realizes, and that Bungie 100% doesn't want this happening either. I hope that many of those people will settle down a bit now because I've seen so many insane overreactions on Reddit and other places over the server stability.
As an infrastructure engineer myself, this smells of corporate mismanagement. When you cut corners on development to make deadlines or quarterly earnings, you push out your best developers, and the ones left don't have time to learn the old codebase and do the job properly.
Bungie lost a lot of quality engineers in the 2019 split, and this is right about the timeline that old code needs updates to meet new architectural demands and those "business decisions" come back to bite the company.
All the sympathy for the devs, but I have zero sympathy for bungie as a company. I will absolutely not be buying marathon.
Every game that releases in a broken state comes out broken because of corporate mismanagement.
People really don’t understand the complexity and scale of live service games. Hell even running something like the Lemmy instance we’re currently talking on gets nuts when you hit a certain size. This stuff is insanely complex and there’s so many interacting parts besides just the code on the game. This post kind of gives us insight as to how the game works and I think it’ll get some people to calm down.
Bungie kind of have themselves to blame for the community acting how they do. They made an intentionally addicting game and now they’re dealing with the addicts who can’t get their fix.
On one hand, I feel that frustration from server issues when I just want to play Destiny.
On the other hand, nobody's forcing me or anyone else to play Destiny. Just our own sense of FOMO that devs love taking advantage of.
Best of luck to the backend team. They're probably putting up with loads of technical debt and other problems.
im sorry, it took them 8 months to put in logging for the networking problems?
The recap of last weeks TWID is longer than the TWID from last week lol.