Collin Winter writes: "I’m getting pretty sick of seeing blog posts and mailing lists threads endlessly bemoaning that, “the core developers…are causing a huge risk to the Python community by splitting it asunder for a period of years“. Gloom, doom, pox and peril, blah blah blah." Collin's quoting me, so that means he's getting pretty sick of me, among others. I don't think I've been exactly endless about this though. I've been talking about this for a few days now, so Collin in getting sick already seems to be displaying his rather fragile constitution. :) I also see the following in the comments to Collin's blog by a certain Matt: "The complainant you reference obviously won’t be happy until Guido drops the Py3K idea altogether, or at least delivers a personal apology. The rest of us will move forward with our clear migration plan, and our optimism for the next version of Python." Clearly he hasn't read what I said very well. I'll quote my previous blog entry: "There is a significant cost in doing this. I think the Python community can bear this cost. I am not sure whether the cost is worth the gain, but Guido thinks it's a good idea, and he does have enough credit with me to trust his judgement. I understand the attraction of cleaning up the language in a backwards incompatible fashion." Again, I understand that the core developers feel it necessary to break backwards compatibility. I can see where this is coming from. I sympathize with the goals. (besides that, radical cleanups are fun and exciting to language developers too. that's fine) I remember talking to Guido years ago about the kind of unicode reformation that Python 3 is going to introduce. I also know that Python 3 is not dropping out of the sky, and I have been fully aware that backwards compatibility is going to be broken for some time now. My comments are a note of caution amongst enthusiasm. I'm not saying "stop in your tracks", I'm just trying to inform the community that Python 3 is not only a good thing, but also carries significant risks. I think it is good to have a bit wider awareness of that. I also think that Collin better fortify himself with some vitamins, because he will likely remember this as the gentle beginnings. :) A bit of community push-back to the core language developers about this doesn't seem to be an entirely bad thing. After all, we're users of this great language and you're going to ask a lot from the community with this change. Now Collin writes that he has worked on technology to mitigate the risk. I applaud his efforts. It also shows to me that Collin is taking the difficulty seriously. Unfortunately the tone of his blog entry is rather dismissive, which is less good, though I understand why. He's been working getting this done, and now he gets random people like me that complain before they've even tried things. Still, I'd prefer a slightly different tone expressing what I trust is the true attitude of the language develpers, that is: "we hear you, we agree that this is a very serious issue, we realize what we're going to put the community through, and we're working on it." Now to a few points where I think we could learn from each other's perspective. Collin writes: "I have absolutely no pity for anyone trying to migrate to Python 3 without a test suite; you’re doing something fundamentally stupid and we will not bend over backwards to save your dumb ass." I see a lot of Python code out there in the real world being used in production that doesn't have very good test coverage. I think a little bit of pity for the dumb asses who decided to commit to the Python language without writing proper unit test suites would do the core developers good. Some notes on test driven development: widespread adoption of test-driven development in the Python world goes back to 2001 or thereabouts - the unittest module made it to the library about then, I believe. Python is about a decade older than that, and developing without a test suite was a normal enough way to write Python code throughout the 90s. Lots of actual production code is not very well tested. You can say people who wrote this code are stupid, but I don't think that's very productive position to take. Sometimes people writing the code have reasonable excuses for less than 100% test coverage. For instance, it is difficult to properly test user interface code, which can take up a significant portion of a codebase. A good example of this is game code. There is quite a lively subcommunity of game developers in the Python world, and I don't think many games have good automated test coverage. Sometimes people act out of expediency, or are in a hurry, or don't know any better even though they should, or simply disagree about testing altogether - do we want to say those people shouldn't be part of our community? I must also note that Python has for a long time been actively promoted to beginners who are learning how to program. Beginners are unlikely to be very test-happy yet. They may also have produced code bases they consider to be valuable. To alienate this audience would be against the spirit of Python. (By the way, I wonder what the plan is for Python C extensions. Are they going to break? Are people to rewrite their C code? Is SWIG going to work? I know that Pyrex will need a major update, for instance, so that would mean difficulty in porting lxml.) Instead of calling lots of people dumb and having no pity for them, which sounds like the language developers would like to ignore their problems altogether, it might be better treat these people with a bit of understanding. Hopefully there are at least minor things that can be done to help these people migrate towards Python 3. I'd like to ask the language developers to keep their eyes open for these possibilities. Just showing some patience and understanding will get you a long way even if you can't do anything to help them concretely, by the way. Now on to preparedness. The porting difficulties and risks lead me to believe that we are going to have to deal with a situation of two Pythons for quite a few years. Codebases written for one Python will be inaccessible to the other Python without additional effort to port things back and forth across the barrier. Let us, as a community, go into this situation with open eyes. Let us be aware of the risks and the costs. Let the language developers display patience and understanding, because they realize they are asking a lot from the communities that use this language. A community prepared will be able to handle the transition difficulties a lot better than a community going into it with blind optimism. Colin writes: "If you think you can do better, show us the code. Talk is cheap." I'm not saying I can do better. I'm trying to inform the community that we're in for a ride, that this ride is going to take a while, and the ride won't be all be fun, even if we believe the destination is worth it. I can show Collin a lot of code, by the way, but I don't need to: just study the cheeseshop. Talk is cheap, indeed. All that code is not.
![[Comments]](/blog-content//themes/Default/comments.png "10 comments")
(10) Sat Jun 23 2007 00:26 the purpose to my "whinging" about the transition to Python 3:
- Comments:
Posted by Collin Winter at Sat Jun 23 2007 03:16
"""Collin's quoting me, so that means he's getting pretty sick of me, among others. I don't think I've been exactly endless about this though. I've been talking about this for a few days now, so Collin in getting sick already seems to be displaying his rather fragile constitution. :)""""""I also think that Collin better fortify himself with some vitamins, because he will likely remember this as the gentle beginnings."""I wasn't addressing you specifically, but you were the one who triggered my post. The python-dev and python-3000 mailing lists have seen this kind of traffic for at least a year now, so while you may see your post as something to set off a landslide, you're just the latest in a long string of would-be Cassandras. We get a lot of people complaining that we're not doing enough, that we're not sufficiently considerate of the community, that the tools aren't 100% perfect, none of whom, suspiciously, are willing to contribute their time or their efforts to correct what they see as problems with the transition plan. They say what they're doing is "alerting the community" or somesuch nonsense -- as if the Python devs have somehow been hiding this!We've tried to be upfront about the difficulties and the pitfalls involved in this. We've been active and aggressive about consulting with major projects such as Twisted about what their concerns are, what we can do to make their lives easier, etc. Anyone who still needs to be warned at this point hasn't been paying attention.
"""I see a lot of Python code out there in the real world being used in production that doesn't have very good test coverage. I think a little bit of pity for the dumb asses who decided to commit to the Python language without writing proper unit test suites would do the core developers good....Lots of actual production code is not very well tested. You can say people who wrote this code are stupid, but I don't think that's very productive position to take. Sometimes people writing the code have reasonable excuses for less than 100% test coverage.
"""Migrating from Python 2 to Python 3 is little different from a large refactoring: without a good test suite, it's likely to be error-prone and tricky going. As a result, the first thing anyone considering a large-scale refactoring is generally advised to do is write a test suite, even if all you're doing is codifying current behaviour. Would you be overly sensitive to the complaints of someone refactoring without a test suite? I wouldn't. Testing is one of the basic principles of software engineering; the fact that a certain percentage of people tend to ignore that principle doesn't make it any less valid or any less important.
"""The porting difficulties and risks lead me to believe that we are going to have to deal with a situation of two Pythons for quite a few years."""No-one has ever said that Python 3 will completely replace Python 2 overnight. The process of migrating old Python code to the new Python language will be gradual, and some projects may never make that leap. Python 2 will go into maintenance mode, while Python 3 becomes the platform for the next generation of language and library development. We will not ask you to port your code, we will not demand it; we will merely offer a compelling feature set and cleaner syntax and hope that that is a sufficient lure and an adequate motivation.Posted by rgz at Sat Jun 23 2007 09:27
Well, since this is his blog, he can talk about his fears of a broken python world, the fact that other people have already complained about this doesn't mean nobody can blog about it ever again.I'm too terribly unimpressed by 3000, the warning about total compatibily brakeage is an overkill intented to prevent anyone from ever trying to run a 2.* program in 3.* but the fact of the matter is that i can see most 2.* programs running in 3.* just fine, honestly the only change most scripts need is changing print to function and dropping some iter methods, i know there are other changes, but it is mostly details, i think the 2.* still has much to offer and that most of py3k's feature could ( and will ) be backported so we could pretend Py3k doesn't exist for the next 2 years, then the change will be less hard.I'll only really complain if newbies are directed to python 3.0 by default from the official download pages.I know they want to get people to the 3.* branch ASAP, but they will be doing a disservice to the beginers if they are told to use the problematic version with less libraries and extensios and supported by less frameworks.Let the early adopters play with it, let them port their favorite extensions to Py3k.Let's pretend Py3k isn't here, I'm sure 2.6 and 2.7 (And I hope a 2.8) will make the transition painless...
Posted by Lennart Regebro at Sat Jun 23 2007 14:18
My thoughts on the subject:
http://regebro.wordpress.com/2007/06/23/python-3000-and-compatibility/Posted by Martijn Faassen at Sat Jun 23 2007 15:16
Collin, my initial blog post about this a few days ago was in response to some very positive expressions on Python 3 (people could hardly wait and were very excited) in some other blogs about Python. I haven't seen much discussion on planet.python.org at all about the dangers of this transition. Not everybody reads python-dev or certainly python-3000-dev. You can also expect the Python-dev community to be looking at this from a very very different perspective than other people. I'd recommend a strategy of continued patience, by the way, as this is going to last for years more. Lots of Python programmers don't read blogs either.Concerning test-suites: I realize having a test-suite when porting to Python 3 is a very good thing to have. You are the one introducing backwards incompatibility here though, and then you're the one calling people without a test-suite who want to move to Python 3 "dumb ass". Or do you not want these people to port their code to Python 3 at all? What is going to happen to them then? What about "sorry we had to break all this code, but we think it's for the best" instead of calling these people names or claiming it all doesn't matter in the end? What kind of attitude is that to take from people who are breaking so much code?Concerning aggressively consulting large projects: Who else has been consulted besides some discussions with Twisted at last PyCon when they expressed their concerns?You may think this is old news, but to my surprise my first blog post on this made it the top voted entry on programming.reddit.com for a while. It also sparked a sizable discussion. Perhaps people were not as well informed of the consequences of this as you suspect? Breaking backwards compatibility is one thing - perhaps not everyone suspected the consequences were as big as they likely will be and were surprised?I'm surprised you see me as a Cassandra preaching doom. I'm not predicting doom. I'm predicting a lot of trouble and confusion for the Python community for the years to come due to this transition, probably for 4 or 5 years at least. It certainly worries me. Two things: I don't remember much information from the language developers about how much time they expect the transition to take. What are their estimates about this? I also don't see this as doom, just lots of trouble. It won't destroy the Python community, as it's strong enough to deal with this and eventually we will emerge from this again. It's just a realistic consequence to this transition.
Posted by Martijn Faassen at Sat Jun 23 2007 15:19
rgz: I agree with your recommendations. I expect and hope there will be a Python 2.8. I don't expect that the Python 3 developers will show restraint in trying to get people over to the new platform - it's a chicken and egg problem they want to solve - the more developers they get on the Python 3 platform the more chance good libraries will be ported over. Since lots of the library and framework developers won't be very quick in porting things over, we can expect to see a lot of advocacy for the use of Python 3.I agree that this unwise and people should be recommended over to Python 2.(in the zope community we have lots of experience with transition issues like this, though the jump from zope 2 to zope 3 is in many ways larger than from python 2 to python 3. some of this looks awefully familiar)
Posted by Brett Cannon at Sat Jun 23 2007 21:54
I was going to post a reply here, but it just became too long and I wanted it to be a more general reply to people in the community who hold your sentiments, Martijn, so I put up on my blog (http://sayspy.blogspot.com/2007/06/python-dev-does-care-about-2x-30.html). I wanted to post here to make sure you knew of it and to doubly make sure you don't feel singled out as the response is meant to apply to a large swath of people and thus not all of it is directed at you.
Posted by Martijn Faassen at Sun Jun 24 2007 00:03
Thanks Brett for listening. I've replied there. In summary, I think things got more polarized than was my original intent. I do think I have some perspectives on software transitions that are valuable in this discussion.
Posted by Collin Winter at Mon Jun 25 2007 04:24
Martijn:"""Not everybody reads python-dev or certainly python-3000-dev...Lots of Python programmers don't read blogs either."""Then what are we supposed to do? Go around, knocking on doors, handing out fliers?
"""You can also expect the Python-dev community to be looking at this from a very very different perspective than other people."""Why? You think we don't use Python? You think we're ivory tower academics, "language designers" sitting around, pulling new syntax out of thin air for our own amusement? The improvements we're pushing are driven by our own, everyday, on the job experiences with this language. We use it professionally, we see how certain design decisions play out when scaled to multi-million-line codebases. The ideas that don't work die and those that do prosper and live forever. The community tells us what we've done right and what we've done wrong, and we adjust our plans to better suit that broad base.
"""Concerning test-suites: I realize having a test-suite when porting to Python 3 is a very good thing to have. You are the one introducing backwards incompatibility here though, and then you're the one calling people without a test-suite who want to move to Python 3 "dumb ass". Or do you not want these people to port their code to Python 3 at all?"""This is no different than changing your application to use a different version of MySQL or upgrading from Apache 1 to Apache 2. If you don't have a test suite, and you can pull it off anyway, good for you, well done. If you try it without a test suite and you fail, I have no sympathy for you, and you're stuck with old dependencies. Tough luck.
"""What is going to happen to them then? What about "sorry we had to break all this code, but we think it's for the best" instead of calling these people names or claiming it all doesn't matter in the end? What kind of attitude is that to take from people who are breaking so much code?"""We are not breaking your code. We are not sneaking into your version control systems in the middle of the night and screwing with your source files. What will happen to people who choose not to upgrade? Nothing. The code they've written for Python 2 will continue to run on Python 2. The 2.x series will continue to receive maintenance and bug-fix releases as long as we feel it is prudent to do so. We will not yank your Python interpreter out from under you, we will not switch things when you are not looking. New features will go into Python 3, new enhancements and better performance, and anyone who does not need those features will simply not upgrade -- they'll have no reason to. Things will work well enough where they are.What I don't understand is this perception that there will be two communities, as if the people using Python 2 and the people using Python 3 will refuse to speak to each other at parties, as if they will have nothing in common. The core ideas of the language, its idioms, conventions and philosophies, will remain unchanged. What's happening to Python is not what's happening to Perl, where they're reinventing absolutely everything anew for Perl 6. In Python 3, some things will be spelled differently, but the heart and soul of the language will remain unchanged.Posted by Paul Boddie at Mon Jun 25 2007 12:36
Martijn: "Not everybody reads python-dev or certainly python-3000-dev...Lots of Python programmers don't read blogs either."Collin: "Then what are we supposed to do? Go around, knocking on doors, handing out fliers?"Do not underestimate the reach of the python.org front page. A few more announcements on the python-announce-list would also reach a few people who don't read all the different lists. Remember that people using Python aren't following all the discussions about language changes; many of them probably use other languages whose definitions are stable over a long period of time, and they may not have considered the existence of such discussions for Python.Collin: "The community tells us what we've done right and what we've done wrong, and we adjust our plans to better suit that broad base."For what level of community? Note that a number of respondents to Guido's blog entry provided wish lists for Python 3000 - clearly they are users of Python but aren't aware of the processes and deadlines, so they clearly aren't participating in any usual debate around Python 3000.Collin: "What I don't understand is this perception that there will be two communities"You have to accept that even if there aren't two communities now, that parts of the community already differ substantially. For example, there are people who actively follow the python-dev, python-3000, python-ideas discussions and those who don't. Perhaps people in the latter group don't have time even to read lists about actually using Python, let alone those lists about developing the language itself.For a large number of Python users, their usage of the language may have arisen because it's a language that they get with their operating system distribution and that they've seen and liked other people's code written in the language, or perhaps their boss or supervisor just told them to use it, or perhaps they have to use a tool or a library written in Python. Their view of the language may be rooted in its place in the distribution or workplace and in the bundled documentation, not in the Python "movement" where everyone knows who the celebrities are, what people on the blogs are saying, and so on. For such people, getting extensions might be about running apt-get or yum, not easy_install, for example. As Python enthusiasts, it's easy to forget that not all people frame everything in terms of Python.I still think this is great panel material for EuroPython, and I see that Collin, Martijn and Guido will all be there. If only Brett Cannon could have made it there, too... ;-)
Posted by Martijn Faassen at Mon Jun 25 2007 12:48
Collin: about "ivory towers". I'm completely unconvinced that people developing Python 3 are "normal" Python developers. Are you maintaining that the language developers are a cross-section of average Python developers? Language developers are just random python hackers who ran into some annoyances with Python and said "oh, wait, string formatting isn't ideal" and then go and spend a lot of time and effort improving the core language? People are drawn to language development for more reasons than real-world concerns in their codebases.python language developers will likely to be:* be strongly interested in language design and implementation* be better than average programmersTo claim that all this work is done merely for the reason that it will solve issues in your daily work is unconvincing to me. This is one aspect but not the only one. Another important aspect is a strong interest in programming language design and implementation. This strong interest will inevitably result in a different evaluation of priorities for you than for people with less interest in these issues. I am not saying that this is *wrong* - it's natural and it's how it should be. It also explains why you will get different feedback from people without your set of priorities.I understand the argument that this is open source, and Python 2.x is not going away, so you're not breaking my code. This is exactly what I fear: lots of people will switch, lots of people won't, and their code won't interoperate without significant efforts. It makes channels of cooperation in the community less efficient. This isn't a prediction of *disaster* - it is a worry about future troubles.It's not the people who won't speak to each other at parties, but the modules not speaking to each other in codebases that I'm worried about. And this will of course affect the people.Anyway, you're not going to take my worries away. I think it's good to have some worries about this process. I have learned a lot about what will happen when I express them, too. I wasn't asking anyone for anything, I wasn't saying anything was doing anything wrong (though I wasn't pleased with the tone of some of the feedback either, just like you're not pleased with my tone). I was just talking about my feelings about Python 3, which are, if you will please forgive me, not entirely positive but mixed with worries. I must say it hasn't been an unalloyedly pleasant experience.
