Very true

Unfortunately this is not a universal truth. Many libraries do not follow SemVer approach and there are multipl counter-examples in both directions (0.* versions being stable and patchlevels introducing breaking changes. Even `pip` itself does not follow it. `pip` actually uses modified CALVER version (with patchlevel) and version number (except patchlevel which is only for bug fixes) does not give you any indication whether the changes are breaking or not. 22.3.0 version might contain breaking changes - 22.3 is merely 3rd minor release in 2022.

Yep. We badly need that as well and I am happy to join any conversation out there.

We have a rule in Airflow that we comment any upper-bounded depednencies - why it is upper-bounded and what needs to happen to remove the limit (often linking to particular issue opened by one of the maintainers).

I highly recommend that - we run such tests for Airlfow - we install everything (we build production image using packages prepared in CI - airflow packages come from build sources but all the other packages are used from PyPI and we run tests against that image for every PR. This helped us to get early warning when some dependencies cause backtracking. We also have a tool that allows us to track the root cause of backtracking issue: https://github.com/apache/airflow/blob/main/dev/TRACKING_BACKTRACKING_ISSUES.md. Any such issue will fail "canary" main builds of ours and when we notice it (we usually do in 1-2 days) we have a way to see which one of our dependencies were released since the last successful build and we can track wich one triggered backtracking - via trrial-and-error but it's better than nothing).

In arflow we automatically upgrade our dependencies (with eager upgrade) and push new constraints when all tests pass. This means that manual updates are only needed in a few cases where things break. This helps us to manage 620+ depeendencies.

I personally think this is the only sustainable approach. Lack of consistency is not a problem if you have clear rules describing it (causing=trouble -> added) and keep a comment in your deps which clearly states what was the reason for pinning and link to an issue that should solve the problem when solved (issue created by one of the maintainers to get notified when it changes).  Then your approach is consistent with the rules, not in general and it is also self-sustainable - i.e. via the issue, you can control removal of such exception.

We could do this on a case-by-case basis. e.g. we currently limit minor version for pandas, because even minor version bumps can break our tight integration through the DataFrame API

This is exactly what we did in Airlfow. The rule is described here: https://github.com/apache/airflow/blob/main/README.md#approach-to-dependencies-of-airflow. We identified the dependencies that need to be upper bounnd for a good reason and we comment them in setup.cfg/setup.py with information when they can be removed. And we review the dependencies before major releases and see if we can get rid of them (sometimes even ad-hoc, and once we have it described - with link to issues sometimes even our contributors (not maaintainers) open an issue about upgrading those dependencies - especially when they have the need to do so). Example PR: https://github.com/apache/airflow/issues/24421

I think there is no need to assign owners. The list will be rather short. In our case it happens almost naturally that different people approach to fix the boundaris when needed or before release. Assigning owners will mean that others will wait for them rather than attempt to fix it IMHO. The key is to document (as comment in the code) why the boundary is and what conditions should be fulfilled (link to open issue ideally) to get rid of it. Then any contributor at any time can check if the condition is fulfilled and prepare a PR without asking anyone.

I think this is the key. We currently have very similar problem in Airlfow (google-ads vs few other libraries) https://github.com/apache/airflow/issues/25929 . I am working with Cloud Composer team to upgrade the old libs ASAP because this will hit us hard if we don't do it.

In GCP Java, we don’t use ranges. One release of 140 GCP client libraries are built and tested with the same set of dependencies (GAX, Google Auth Library, gRPC, Protobuf). Other versions of the dependencies “may” work, but we cannot guarantee the compatibility without tests. It’s impossible to test all possible combinations created by ranges.

Happy to help with working out good approach with the Google team. If you will be talking to the Google client team, I am super happy to join the conversation.

I can help with this. Just let me know.

timeline is debatable

Let me know if I can help with this.