How do you balance rapid iteration and merging/upgrading?

Semantic versioning. Moving from v2.3 to v2.4 shouldn’t require major changes, but moving to v3.0 can.

Yeah for this situation, versioned APIs are the answer. If, for example, you look at the kubernetes ecosystem, the entire thing is based on APIs and every resource starts by specifying an api version on the very first line.

apiVersion: v1
kind: Namespace 
metadata: 
    name: example-namespace

This is how they can make upstream changes and not break existing environments in the process.

One approach I’ve seen (from a user pov, not dev, so I’ve no idea of the code bloat it might cause) is to pass the API version number in the call. Then your api can be backwards compatible for 2 or 3 versions, giving other users time to upgrade their code. It de-couples things to give you all a bit of slack for both rapid iteration and stability.

But it also depends on the ‘contract’ between you and the users so be very clear how long / how many versions will be available. Probably will involve a ‘use by’ date.

Context:

I'm a dev that consumes company wide libraries, not an author of such libraries. So the following comes from that perspective.

A couple questions:

1. Is development and consumption of your library happening in parallel? It sounds like you use the users to vet new features or major changes... is that correct? (They are iterating with you and reporting back on issues or desired changes)
2. Is your library made up of a group of isolated components? Or is it a library that does one or two major things and so a breaking change literally changes the whole of what the library does?
3. How are the consumers of your library when it comes to adopting changes? Do they readily do it? Is there a good bit of inertia?

My thoughts:

First off, SemVer is definitely going to be important. Also, it sounds like you're working toward API stabilization which is going help iterating in the future.

My idea 1:

If your library is made up of several isolated components, what about doing major releases (ex 2.x.x -> 3.x.x) more frequently? Only include a small subset of breaking changes for one or two components rather than jamming a whole bunch in there just because it's a "major version release". The result is you could move quickly and iterate while also minimizing the impact on ALL of your users every release. Some of your users may be able to upgrade to the latest without having to touch much or any of their code.

My idea 2:

Do frequent major release (ex 2.x.x - 3.x.x) but always start with an "alpha" release that early adopters could implement and provide feedback on. This would shield the majority of your consumer's code from having to iterate frequently but would also require you to enlist a group of committed early adopters that are diligent about iterating their code as often as you release.

Feedback on the original option 1 and 2

Option 1

This could work if your users are excited about your releases. But, it could result in people NEVER upgrading because it's too much work to do so. (I've seen this happen. No one upgrades until they absolutely have to.)

Option 2

Depending on the size of your company, this will be a lot of work for you and will slow you down. If you're using your users to vet out new features, then everyone is going to have to iterate frequently (like you said) if experimental changes don't work out.

I'd say the ideal thing to aim for is case 2-ish. Building in isolation isn't great and dog-fooding your lib (via their projects) can help reduce the amount of future breaking changes (getting things right the first time). I.e. Ideally your helping people upgrade when needed and seeing their problems. Even if your forking their projects to try your experimental changes out for them and then provide a PR? That's probably a lot easier said than done though xD

A dependecy management system to let consumers know when a new version is available could go a long way, or you pushing for them to update by talking to them (as this is all internal?).

Basically reducing the distance between teams and getting the tightest possible feedback loops should be the goal.

That's my rushed 2p while waiting for a haircut anyway ;)

Get good at the three point turn.

• Add the new code path/behavior. Release - this can be a minor version in semver.
• Mark the old code path or behavior as deprecated. Release - this can be another minor version.
  • In between here, clean up any dependencies or give your users time to clean up.
• Remove the old code path or behavior. Release. If you're using semver, this is the major version change.

This is a stable way to make changes on any system that has a dependency on another platform, repository, or system. It's good practice for anything on the web, as users may have logged in or long running sessions, and it works for systems that call each other and get released on different cadences.

Users will become dependent on anything you release.

Microsoft has to be careful with private and internal members as users will use reflection and become dependent on them. Meaning internal changes break customer code.

Admittedly that is over kill in most cases. I take the stance if they are dependent on private state it's their fault if a release breaks it.

But the point remains you can't easily change existing APIs once released.

As mentioned above semantic versioning is a good solution to this. At least then they know when an upgrade will cause breaking changes.

If it can be avoided don't put out anything that will likely be retired. Releasing experimental features that are likely to be replaced is always going to be bad.

If you release a feature you need to plan to support it essentially.

Are there any good strategies for avoiding painful upgrades?

If you're not already doing so, hold design reviews with your users. Breaking API changes should be communicated early and in a way that makes it clear how the users benefit from the change. If the users don't benefit, you should reconsider why you're making changes in the first place.

Some companies use feature flags for this. You add features, sometimes add feature flags, and you slowly deprecate the old API at the same time.

I don't have multi-user library maintenance experience in particular, but

I think a library with multiple users has to have a particular consideration for them.

1. Make changes in a well-documented and obvious way
   1. Each release has a list of categorized changes (and if the lib has multiple concerns or sections, preferably sectioned by them too)
   2. Each release follows semantic versioning - break existing APIs (specifically obsoletion) only on major
   3. Preferably mark obsoletion one feature or major release before a removal release
   4. Consider timing of feature / major version releases so there's plannable time frames for users
2. For internal company use, I would consider users close and small-number enough to think about direct feedback channels of needs and concerns and upgrade support (and maybe even pushing for them [at times])

I think "keeping all users in sync" is a hard ask that will likely cause conflict and frustration (on both sides). I don't know your company or project landscape though. Just as a general, most common expectation.

So between your two alternatives, I guess it's more of point 1? I don't think it should be "rapidly develop" though. I'm more thinking doing mindful "isolated" lib development with feedback channels, somewhat predictable planning, and documented release/upgrade changes.

If you're not doing mindful thorough release management, the "saved" effort will likely land elsewhere, and may very well be much higher.