Cargo's missing stability guarantees

By: Georg Semmler
Posted:

The Rust 2024 edition was recently published. It contains a lot of shiny new features. As always this was made possible by the work of many great people in the Rust project. This is usually a reason to celebrate.

I personally see the release of the Rust 2024 edition with some disappointment as the edition again contains changes that can result in build failures. In my opinion, these changes are in direct contradiction to what was agreed on by the Rust project in the corresponding edition RFCs.

I would like to use this blog post to write about which problems are caused by these changes. Also I would like to show how these changes in my opinion contradict the edition RFCs. This discussion will include examples from the last two editions, including specific code examples on what is made much harder to express by these changes. Later on I present several possible ways to address these issues. Finally I would like to talk a bit about how certain members of the Rust project reacted to raising these issues.

Resolver Changes

Both the 2021 and the 2024 edition changed the behaviour of the resolver used by cargo to determine which dependency should be compiled with which configuration:

Notably this kind of change affects not only the crate opting into this behaviour, but also all it's dependencies, including subdependencies.

resolver = "2" changes the way cargo unifies features between different instances of the same crate version. With resolver = "1" cargo unified features for any instance of the crate version no matter if the specific instance was a host or a target dependency. With resolver = "2" this behaviour changed, so that cargo now unifies features for host and target dependencies independently. Overall this is a good change as this enables using features in your host dependencies that you cannot use in your target dependencies, which is an important feature for embedded Rust users.

resolver = "3" changed the way cargo resolves dependency versions. With resolver = "1/2" cargo always resolved a dependency version to the latest available semver compatible crate version. So if you wrote serde = "1" (or serde = "1.0.0") you would eventually get something like serde = "1.0.218" as actually used dependency version. This works well as long as you use the latest Rust version. It becomes problematic as soon as any crate in your dependency chain requires a newer Rust version than the one you are working with. It essentially bumps that version for the whole dependency tree. resolver = "3" changes this behaviour to also consider the rust-version field in the dependencies Cargo.toml file. Now it resolves the dependency version to the latest semver compatible crate version that is supported by your Rust version. So for the serde example, if you compile that with Rust 1.30.0 you would get serde = "1.0.179" instead as dependency version. Overall this changes simplifies the usage of older Rust versions.

Problematic Cases with Resolver Changes

Overall, both changes seem to represent a clear improvement compared to the old behaviour. They enable new use cases while maintaining compatibility with the old behaviour. Unfortunately, this is not 100% true, because with both changes there are cases in which code no longer compiles after they have been introduced.

resolver = "2"

For resolver = "2" things get tricky around proc-macro crates. These are always considered to be host dependencies, as proc-macros are executed during compilation. On its own that is not problematic, but many crates couple proc-macro crates with their main crate, which means the proc-macro generated code refers to code from the main crate. Now it can happen that the main crate forwards a feature to the proc-macro crate to conditionally generate code based on the feature flags. This generated code can then conditionally refer to feature gated code in the main crate. If you now depend on the main crate with different feature flags from your host and your target dependencies, cargo will compile two instances of the main crate with a separate feature set for host and target. It will also compile a single version of the proc-macro crate with a union of all features enabled by both the host and the target version of the main crate. This in turn leads to a compilation error at the usage point of a proc-macro item that doesn't expect these feature flags. An example for such a crate combination is diesel and diesel_derives. For example the following dependency declaration will result in a build failure in diesel itself, as diesel internally uses derives from diesel_derive to generate code depending on feature flags:

[dependencies]
diesel = { version = "2.2.7", features = ["sqlite"] }

[build-dependencies]
diesel = { version = "2.2.7", features = ["postgres"] }

You end up with a dependency tree like that for this particular example:

[dependencies]
diesel 2.2.7 feature "sqlite"
├── diesel_derives 2.2.7 feature "sqlite"
…
[build-dependencies]
diesel 2.2.7 feature "postgres"
├── diesel_derives 2.2.7 feature "postgres" (+ feature "sqlite" enabled by the ordinary dependency)
…

It's important to note here that you can enable both features together for both diesel and also diesel_derives, as long as the same features are enabled for both crates. The later point was ensured by `resolver = "1".

A more severe variant of this breakage was present at the time of the edition = "2021" release for diesel versions older than 1.4.7. The diesel team almost had to perform a breaking change and therefore release a new major version just to address that incompatibility. We avoided it back then by finding a different, rather hacky solution to the problem. The compiler shipped a lint specifically for diesel to make people aware of the required update.

The stablization report lists several more broken crates that did not get this kind of special handling. It's also an issue that causes problems for other crates as well from time to time according to various posts at users.rust-lang.org and the cargo issue tracker (Example 1, Example 2 (note the linked comment from the other user) ).

resolver = "3"

For resolver = "3"things get complicated around minimal dependency versions. Up until now it was a convenience to just write out dependency declarations in your Cargo.toml files as serde = "1" as this is just shorter to type. Cargo then would automatically pick the latest dependency version for you. Often you would end up using functionality from the latest version without noticing that it was just recently introduced.

resolver = "3" changes this behaviour to pick a potentially older version of that dependency to keep your dependency tree compatible with your current Rust version. The downside of that is that now you might get a dependency version that doesn't have the functionality you used before. This does not only happen for your crate itself, but for any crate in your dependency tree. So you might suddenly be stuck with an error that points to some crate deep in that dependency tree with no straightforward way to fix it. Now one could argue: "All these dependency declaration were already wrong before, so that is clearly a bug in the relevant crate" and that's not wrong. The problem here is: There was no easy way to test for this specific error case before resolver = "3" was stable. Even with resolver = "3" a sufficient number of Rust releases might be required to have a large enough version space to trigger this problem.

With resolver = "2" there existed only one way to downgrade a dependency: Explicitly calling cargo update -p your_dependency --precise 0.1.2. This made it really hard to even test if you declared one of your minimal dependency versions incorrectly. There is the unstable -Z minimal-versions flag for cargo update to automatically downgrade all dependencies to their lowest supported version and there exist various third party solutions like cargo-minimal-versions which essentially use this flag internally. These solutions all required a nightly compiler.

There is another case where resolver = "3" can break things, even without a formally incorrect dependency deceleration: Inter-crate dependencies. Consider a case where your crate a depends on both crate b and b-traits, which are both versioned independently and have different supported minimal Rust versions. Now crate b-traits provides a set of traits implemented for types from crate b, so both must always be used in a compatible version combination. If a downstream crate, like our crate a, wants to support multiple major versions of crate b and b-traits it can fall back to using version ranges like b = ">=0.2,<0.5", to allow multiple versions to be used. This works as long as the API actually used by crate a stays stable in all declared versions. What now can happen with resolver = "3" is that you end up in a situation where cargo selects incompatible versions of crate b and b-traits depending on how they declare their minimal supported Rust versions. After all one of the main goals of the Rust version aware resolver is that each crate can declare it's own independent minimal supported Rust version. To illustrate this with a specific example: Assuming there are the following versions:

crate b:

  • 0.2, which declares Rust 1.80.0 as MSRV
  • 0.3, which declares Rust 1.83.0 as MSRV

crate b-traits:

  • 0.2, which declares Rust 1.81.0 as MSRV
  • 0.3, which declares Rust 1.84.0 as MSRV

We now assume that our crate a depends on the version range >=0.2,<0.4 for both crates and that the version of crate b and b-traits must be the same for both crates to be compatible. If we now build any crate depending on crate a with Rust 1.83.0 we will end up with crate a depending on crate b = "0.3" and b-traits = "0.2". I'm not aware of any way to declare dependencies to enforce this version coupling in such a way that it is respected by cargo.

This case is hard to discover with local or automated testing as you would essentially need to build your crate with all supported Rust versions. It is not sufficient anymore to just test with the current stable version and possibly the minimal supported Rust version (the later approach is that was is currently common in the ecosystem). If this particular dependency is behind a feature flag you end up with even more combinations to check.

It's worth to note that this kind of breakage only happens if dependencies are actually resolved with the new resolver. That means if you have an existing Cargo.lock file in your repository you won't notice any changes until you actually run cargo update (or similar commands) to change the dependencies in your lock file. On one hand that sidesteps the described problems, as existing projects continue to work. On the other hand it makes it again harder to actually discover what's the underlying problem, as the build error might occur only at a later point in time and is then not directly correlated with the resolver change. This becomes more severe given that the cargo team now encourages users to always include a Cargo.lock file in their repository. The direct consequence of that recommendation in combination with the resolver change is that now you only notice such problems if some user reports them by adding your crate to their project. That's because your local Cargo.lock file likely won't change much even while switching Rust versions, while the Cargo.lock files of crates depending on your released version will always be regenerated if a new dependency (like your crate) is added.

It's hard to be sure if that change will have a large impact or not given that it is relatively new and that it needs a certain version space size so that these problems can actually occur. As an upper bound I would like to make an estimation by looking at recently published crates on crates.io and see how many of them might be affected by this change. At the time of writing the 10 most recently published crates are:

So out of the examined 10 crates, 8 contain potentially affected version declarations. Now merely having an imprecise lower version bound does not necessarily mean that the crate is affected. It would need additional checking, based on the minimal supported Rust version by these crates and other factors, to be sure. I would also assume that more well known crates do suffer less from this kind of issue, as they are used in far more setups and therefore different combinations, than newer less well known crates. Additionally, I would also assume that older crates are more affected than newer crates, due to the addition of tools like cargo add which always specify the full dependency version. Overall this at least demonstrates in my opinion that this is a problem that might affect a non-zero share of crates hosted on crates.io.

Overall Problematic Changes

As demonstrated by the examples above both changes have edge cases where they break legitimate use cases that existed before they were introduced. By itself that is no huge problem as long as such changes are coupled with some way to figure out if something might be problematic beforehand and by providing workarounds for existing working use cases. In my opinion, this is unfortunately not the case in the two cases presented.

The development experience for users of crates that have such issues is also not great. They get an compilation error message in code somewhere out of their control, without an indication what caused this issue. From their point of view the crate with the broken compilation is just broken in general. Depending on the exact dependency configuration, such effects can happen in seemingly unrelated crates, which makes it really hard for users to debug this kind of issue. This often leads to users filling issues with the crate that failed to compile.

As far as I'm aware there is no workaround available that I can use as crate author to prevent any of the described issues happening for crates depending on my code. For my specific use cases that unfortunately means that I'm either stuck with a crate that is broken in certain edge cases or the requirement to actively remove support for features that are part of stable crate releases on their own.

In both cases you can instruct the users of your crate to perform actions to workaround this kind of issue. For resolver = "2" that involves adding features from the target compilation unit to the host crates and vice versa. Depending on your use-case that might not be desirable, as it might pull in unwanted dependencies like native libraries. For resolver = "3" you might be able to workaround this kind of issues by manually adjusting your dependency tree via cargo update -p affected-crate --precise 0.1.2 to select a better matching version of the dependency. Depending on the number of affected crates this might be a lot of manual work, that also needs to be repeated after each update. As consequence this kind of change puts a lot of pressure on maintainers of affected crates to fix this issue in "their code" even if that is not 100% possible.

Edition Rules

At this point I would like to shortly look into what the relevant edition RFCs actually write about what's allowed to be changed between editions and what is not allowed. The relevant RFCs are

RFC-2052 introduces the general edition mechanism, proposes a 2018 edition and outlines a general set of rules. Notably it contains the following hard constraint:

TL;DR: Warning-free code on edition N must compile on edition N+1 and have the same behavior.

RFC-3085 extends the edition mechanism to be something that regularly happens, proposes a 2021 edition and outlines a lot of corner stones and common rules for editions. Most notably the RFC claims that:

  • Editions do not split the ecosystem, by specifying that "the decision to migrate to a newer edition is a "private one" that the crate can make without affecting others" (Source)
  • Edition migration is easy and largely automated, by using cargo fix, although it acknowledges there are edge cases that cannot be migrated automatically (Source)
  • Users control when they adopt new editions (Source)

RFC-3501 proposes a 2024 edition and largely adopts the constraints set by RFC-3085 for that.

Now one common theme of all these RFCs is that they promise a certain level of stability even with the kind of large changes editions bring to the ecosystem. They specify what is an acceptable change and they also call out what is not an acceptable change. They all highlight that the update to a new edition must be easy (by providing warnings beforehand, by automating the migration) and that it is a private choice each crate can make independently.

It's fair to note here that all three RFCs read like they were written with the compiler and the language in mind. Cargo is a different project with different requirements. It's also important to note that RFC-3501 was written by a cargo team member, so they would have certainly been able to adjust the rules to better fit cargo's needs there.

In my opinion any resolver change violates the rule that migrating to a new edition is a "private one" that doesn't affect others. This follows the interpretation that each crate is independent and does not include their dependencies as well. Also there is no warning or automated fix provided for any of the described issues. I would like to explicitly clarify that publicly adjusting these rules to match the requirements of these changes is certainly a way to resolve this conflict.

Possible Ways Forward

I personally do not feel that any of that is so critical that it cannot be fixed or requires rolling back these changes. I see several generic and specific solutions to migrate the particular problems outlined above and also address the general concerns whether these changes should have been done around an edition boundary.

The first rather obvious suggested change would be to adjust the edition RFCs to reflect that such resolver changes are in fact in scope of edition changes. I personally would argue that this is one of the large mistakes the cargo team made here. They could have easily adjusted at least RFC-3501 to reflect that.

The second suggestion is to spend time working on improved diagnostics for both failure cases. I know that it is hard for cargo to detect this, nevertheless I believe that there are several heuristics that can be used to improve the situation. For the failure cases of resolver = "3" cargo could look for imprecise lower version bounds like serde = "1" and warn against them. For resolver = "2" cargo could look for cases in the dependency tree where a host dependency gets different features from other host and build dependencies. This kind of information can later on be used to emit an additional note or hint if the compilation fails in a related part of the dependency tree. Sure that won't catch all possible edge cases, but it might provide a much better starting point for users to understand why heir compilation might be broken.

On top of these general suggestions there are additional fixes imaginable for both error cases.

For resolver = "2" the main problem is the propagation of features between host and target dependencies to shared host dependencies. That problem could be side stepped by being even more strict about separating both kinds of dependencies. This would mean building one version of the shared host dependency with only the features from the host dependency side and another version with only the features from the target dependency side. Another way to address the problem is to provide another, better working solution for the underlying use case. For the presented case above diesel generated feature flagged code with the help of a proc-macro. If now proc-macro crates were coupled more tightly to their main crates and if the proc-macro runtime had an interface for querying the features of the main crate, passing the features at compilation time to the proc-macro crate would not be necessary in the first place.

For resolver = "3" I suggest providing more tools/flags to easily test different configurations of the dependency tree. Sure that's something you won't want to do for every build, but that's certainly something that maintainers might want to enable in their CI setups to make their overall test coverage for different dependency versions better. For that such a tool/flag needs to be much more accessible than "You need to manually tweak dependencies via cargo update or use a third party tool". The second part of the presented problem could also be addressed by having a way that certain dependencies need to have compatible versions, by possibly having a way to declare that in the consuming crate.

The Social Aspect

Some readers may wonder why I choose to write this blog post rather than raising these issues with the relevant teams. I did that several times and what happened there is in fact more concerning to me than any of the technical issues above. After all, the technical problems are merely bugs that can be fixed.

For the resolver = "2" change, I opened a discussion thread on internals.rust-lang.org. This resulted in the compiler implementing a specific warning for the affected Diesel versions. Neither a more general solution was implemented nor were safeguard put in place for future edition changes with similar impacts.

For resolver = "3", I raised concerns multiple times in relevant GitHub discussions:

Each time, my technical concerns about compatibility and more importantly RFC requirements were not addressed with technical counterarguments. Instead, they were dismissed with statements that either:

  • The dependency declarations were "already broken" (despite being functional with the older resolver versions)
  • The issue wasn't relevant because Cargo.lock files would prevent problems (despite that being arguably only true if such a lock file exists and is not modified)

What I find particular troubling isn't just the technical dismissal, but how the discussion itself was handled. For both the "Stabilizing Edition 2024" PR and the seperate issue to track the missing warning a moderator stepped in to shut down the discussion. The later issue was additionally locked as "too heated" despite containing only technical discussion without personal attacks or similar non-technical content.

In private communication, a member of the moderation team indicated that continuing to raise these concerns publicly could result in being banned from the rust-lang repositories. Meanwhile, a cargo team member characterised my evidence-based concerns as "spreading FUD" in a direct message.

These actions effectively prevented technical discussions from taking place in public forums where other members of the community could assess the validity of the arguments.

To be clear, I understand that teams may disagree with my interpretation of the RFCs. What I find problematic is not the disagreement itself, but:

  • The refusal to acknowledge that there is a legitimate question about whether these resolver changes (as implemented) violate the letter or the spirit of the edition RFC's hard constraints
  • The use of moderation tools to end technical discussion

The Rust project has long prided itself on transparent governance and decision-making. When technical concerns about compatibility—backed by specific examples and reference to project RFCs—cannot be discussed openly, something important has been lost.

Personal Consequences

After careful reflection on these experiences, I've made some difficult decisions regarding my involvement with the Rust ecosystem.

Given the response from the cargo and moderation team members, I no longer feel that contributing to the Rust compiler/language development is a productive use of my time and expertise. This is particular disappointing as I had planned to work on improvements to the #[diagnostic] namespace and other areas where I believe I could add value.

My support for the crates.io team through Diesel-related assistance will also be discontinued. I've greatly enjoyed collaborating with the crates.io team in the past, which makes this decision especially difficult. Given that some of the people involved are part of several teams including the moderation team and leadership council, I not feel safe to interact with any part of the project in a productive way.

I will continue maintaining Diesel, but with an important change: we will be updating our stability policy to explicitly permit breaking changes when necessary to work around compatibility issues originating from the Rust project itself. Unfortunately, this exception is now necessary to ensure that we can adapt to unexpected changes in behaviour, such as those documented above.

I'm sharing this experience not just as a personal statement, but because I believe it represents a concerning pattern. Other maintainers in the Rust ecosystem may wish to add similar exceptions in their stability guarantees, given the precedent set by these changes.

For those considering contributing to the Rust project, I feel obligated to share that my experience suggests the environment has changed from the welcoming environment it once was. The technical barriers to contribution remain low, but the governance barriers to see concerns addressed have grown significantly, particularly when those concerns challenge decisions already made by established teams.

I hope that by documenting these issues publicly, positive change might eventually occur. I would gladly reconsider my involvement if:

  • The project leadership acknowledges that resolver changes at edition boundaries deserve the same careful warning mechanisms as other potentially build breaking changes
  • The moderation process evolves to better distinguish between code of conduct violations and technical disagreements
  • The project recommits to transparency in decision-making, especially around stability guarantees

Until then, I'll be focusing my energy on projects where I can more effectively contribute and where technical concerns can be openly discussed without risk of moderation action.