Python Secret Weblog for 2008 March 5 (entry 0)

(4) Wed Mar 05 2008 00:36 Please don't break library compatibility in Python 3:

Update: Guido appears to share my opinion on this.

I've heard murmurs from a bunch of projects over time concerning their transition plans to Python 3.0. These transition plans are naturally vague at this stage. There's one theme I've heard a few times in these murmurs that worries me enough to speak out on it.

People sometimes reason that since Python 3 is going to break people's code anyway, now is a good time to make backwards compatible changes in their libraries too. The reasoning is understandable: people are facing a big change anyway with Python 3, so why break backwards compatibility for them once then, and then another time for your library? Better do it at the same time and make people suffer only once.

There are other reasons that drive projects to break backwards compatibility at this point. As far as I understand, the Python C API will change with Python 3, so that means projects that bind to C libraries need to be adjusted anyway. This might tempt people to make larger scale changes.

I just heard that the current WSGI spec isn't entirely compatible with Python 3. If that is so, and a new version of the spec is needed, then that might give some frameworks that use WSGI an opportunity to clean up some things and break backwards compatibility.

While very understandable, the negative consequence of all this is that the Python 2 to 3 conversion script strategy stops working for code that depends on these libraries and frameworks. This official transition strategy aims to make the transition between Python 2 and Python 3 easier by offering a script that convert your Python 2 code to Python 3 code automatically.

If lots of important libraries and frameworks however also break backwards compatibility in their Python 3 versions, this means that any converted code that depends on them has no chance of working. Instead the developer would need to track all the API changes and make adjustments there too before the code would work again: a much harder task.

So my advice to library and framework maintainers would be: please do not use this opportunity to break backwards compatibility in your library or framework too. I'd like to ask you to resist this temptation. The language changes already make this a major transition for everybody, and you'd make it that much harder by taking away people's ability to automatically convert their Python codebases.

Comments:

Posted by anonymous@189.160.155.7 at Wed Mar 05 2008 18:54

You are urging people not to change their API when that's what they want to do. The point is that this *is* the best moment to break compatibility. Of course there is no point in breaking API just for the sake of it but if you are planning to do it you may as well do it now.

It is not only the "break once" opportunity that you observed, but the fact is that new language features are the best incentive for rewriting and changing both library AND client code so IFF you are going to break your API this is the time.

Sure it won't help adoption of Python 3, sure the automagical 2to3 tool will not be enough (hint it was never enough, unit tests are still necessary after running 2to3), but so what? You are making it seem as if it was the job of framework and library developers to push Python 3 into the community.

If Python 3 is so good let it stand on its own.

Posted by Ian Bicking at Wed Mar 05 2008 21:06

One thing that I probably will do is remove a lot of functionality in libraries. This is stuff that has replacements, and I've been meaning to remove for a long time, but there's little incentive to remove working code. With the move to Python 3 there will be incentive, as everything will require maintenance.

Unfortunately the result will probably be code breaking for other people. But I don't really see a way around that. As with WSGI, with lots of things there won't be a clear path forward, instead updates will require thoughtfulness and choices, and these have to be expressed to anyone consuming the library who upgrades to Python 3. That's painful. Keeping (or making) the surface area of the library small will be essential, at least to me when I consider what I am capable of maintaining.

Posted by Christopher Armstrong at Wed Mar 05 2008 23:32

I agree. More projects need to take a more serious stance on backwards compatibility. There are LOTS of ways with Python to maintain backwards compatibility while preventing it from getting in the way. Please, add deprecation warnings to code that you want to get rid of and leave it there for a good while before breaking it.

Posted by Martijn Faassen at Thu Mar 06 2008 00:08

anonymous: 2to3 is not enough, but 2to3 with unit tests *and no API breakage in major dependencies* has a chance of working for people. 2to3 with API breakage will make it much harder to determine whether a test failure is due to a library breakage or an API breakage. The job becomes a lot less tractable as you're dealing with more than one problem at the same time.

*If* framework and library developers are going to port over their code to Python 3 (and I expect many projects will; that's simply a reality we have to deal with), I'd like to at least ask them to seriously consider *not* using this as an opportunity to break APIs just because it seems to be the right moment. If it's necessary, fine. It may be necessary for WSGI. But if you avoid it, you'll make everybody that depends on you a lot happier.

Ian, to you I will simply repeat the point: if you want to break APIs and see Python 3 as an opportunity to do so as a developer, just realize that I think for many of your users Python 3 is the *wrong* moment to do so, if those users are at least interested in porting over existing code. I think having to deal with a conversion script, and, say, unicode issues, is hard enough for most people without shrinking away APIs. In general if this work can be shifted to the Python 2.6 world as far as possible, it'll make *gradual* changes possible for users, instead of mixing in two problems at once.

I agree with Christopher that deprecation should be taken seriously. Of course there is a right time and place to start moving to deprecation warnings - it depends on how much code there is and the time of evolution of the project, but it should be considered seriously by more projects.