Briefly, some of my Python 3000 thoughts. I see quite a bit of enthusiasm in some other blogs. I'm not very enthusiastic. While I understand the need to be able to break backwards compatibility, I am worried about Python forking into two parallel versions (2.x and 3.x) for many, many years. Such a fork can't be good for the Python community. Why do I worry about Python forking? Because porting code to Python 3 will be hard. The porting instructions discuss the need for a good unit test suite before getting started. This is very wise. In practice of course, while I write unit tests for code all the time, lots of the code I work with doesn't have a great test coverage. There's lots and lots and lots of valuable Python code like that out there. What is going to happen to it? How is it going to be ported? Even with test coverage, porting over code is going to be a massive amount of work for larger code bases. Considering that Zope 3, a clean, modern, extremely well-tested code base (one of the best test coverages in the Python world) is still on Python 2.4 as it takes time and effort to port to Python 2.5, what is going to happen with Python 3? What does this say about codebases with far less tests? The result in many cases is that people will put off porting code, as it's too costly. It won't be easy to motivate a customer to pay for porting activity that will bring no new features to them whatsoever. People will therefore continue to run this code on Python 2.x. Since Python 2.x code doesn't work on Python 3.x, it won't be accessible to people who made the jump. Since Python 3.x code doesn't work on Python 2.x, it won't be accessible to those with existing code bases who can't make the jump any time soon. As a result, two Python communities for a period of what I expect to be 5 to 10 years. I realize there are no answers to these worries beyond those that have already been given. Breaking backwards compatibility may be necessary. It just doesn't make me have a "birthday feeling", as I saw described in another blog. Python 3 is a serious risk to the Python community. Not by far as much as, say, Perl 6, is to the Perl community, but a significant risk nonetheless. (I also am worried that Abstract Base Classes are throwing away some key benefits of zope.interface, but that is not really a blocker - at least zope.interface can be made to work on Python 3).
![[Comments]](/blog-content//themes/Default/comments.png "34 comments")
(34) Wed Jun 20 2007 12:10 Brief Python 3000 thoughts:
- Comments:
Posted by Philipp von Weitershausen at Wed Jun 20 2007 12:58
Thanks for writing the blog post I wanted to write about Py3k, you saved me some time :).I am not too excited either. Sure, Py3k sets a few things right (iterator views on dicts, etc.) which improve the language from an aesthetics point of view (an important aspect in the Python world). A lot of other stuff worries me, though. ABCs sound like a lame duck compared to interfaces. Many people also go nuts about Py3k's ability to specify types and such. I'm skeptical as to whether that'll turn out to be a good thing. I'm intrigued by the idea of generic functions, though (we have something similar in Zope already with adaption).As far as the migration is concerned, I'm afraid that Python might make the same mistake Zope made. I certainly hope that the Python 2.x line will *evolve* to the Python 3.x syntax, gradually allowing code changes and allowing a large cross-section of code to be executed on both versions. Your slogan "Evolution, not revolution" has worked well for us in the end, but we're only slowly overcoming that separation of Zope 2 and 3 after 4 or so years (well, there were other problems involved, such as an identity crisis, as Tres put it).
Posted by Brian Hocklan at Wed Jun 20 2007 14:29
Py3000 seems mostly like a marketing release. That may be good. But from a coding standpoint, the benefits are simply not worth the transition for existing code. Not until we see some benchmarks. Sure, the purity of "pythonic" has improved, but not more than 5%. People get excited and say big words before the thing is done. Just look up some old comments about how pypy would be 10 times faster than C, how would perl6 bring the apocalypsis and GNU Hurd would rule the OS world by 1998.The emperor has no clothes?
Posted by Bill Mill at Wed Jun 20 2007 15:30
I *am* excited about Python 3000. I think it introduces a bunch of cleanups to the language that will ease everyone's life (new-style/old-style classes, anyone?) and provide a base for future enhancements.I also think that they're doing it exactly the right way to prevent a Perl-style stasis. The next version of Python will have a mode where you can get warnings on what parts of your code will break. There will be a PyLint-style tool to help you migrate your program. There are specific release dates that we can trust will be more-or-less adhered to.Will it take a while to get everyone to Py3k? Yup. But I think it'll take just slightly longer to do so than it took to get people to Py 2.5, which introduced C API changes.I'm very hopeful.
Posted by Michael at Wed Jun 20 2007 15:31
I'm worried too because i think python2 and python3 may be two diffrent languages.
Posted by Bob at Wed Jun 20 2007 15:35
I hope it doesn't end up like the Perl 6 fiasco. Also Perl 6 is further along than most people give it credit for, the new version has diverted resources from the production version, Perl 5, and they are different enough to be close to different languages.It's causing serious harm to Perl's acceptance. So I hope it doesn't happen to Python. Then both of my favorite languages would be in a bad state.
Posted by Martijn Faassen at Wed Jun 20 2007 15:48
Bill Mill, why do you think a transition to Python3k will take "just slightly longer to do so than it took to get people to Py 2.5"? There are fundamental changes to the way the language works. Practically all Python code (not just C extensions) is going to break on Python 3. Porting production code means reviewing all of it, which is a large amount of work in the case of incomplete test coverage (which is frequent). Who is going to port all that code? Who is going to pay for all this work? I think it will be much, much, more work than the transition to Python 2.5, and note that this transition is still underway three quarters of a year after its release.I think in this case the burden of proof is on those who think the transition will be relatively quick, not on those who think it will take years. I base my expectations on experience porting code from one iteration of a framework to another. This can be a significant amount of work even in the face of minimal API changes. Porting code to another language (even if a very much related one) is more involved than that. Some of your most basic assumptions suddenly break.If porting costs are high enough, people will put it off. This means the porting process will drag out even longer than you'd expect from the amount of work that actually needs to be done.
Posted by René Pijlman at Wed Jun 20 2007 16:18
Python Five Point Five: Python 3 integrated into the Python 2.5 interpreter (python -v3), so we can use new features in old code. We can then run Plone 3 using Zope 3 features through Five with Zope 2, on Python 2.5, while still claiming to ship Plone with Python 3 :-)
Posted by Paul Boddie at Wed Jun 20 2007 16:45
I'd be interested to see whether any of the other Python implementations ever decide to aim for full 3.x compatibility.I guess IronPython has a lot of momentum, but I don't see much commitment there beyond 2.5 or 2.6 - perhaps they'll wait and see. The PyPy people seem (very wisely) to be targetting 2.4.x still. I guess Jython will make it somewhere between 2.4 and 2.6 at some point, as will CLPython (which is apparently only a few things short of 2.5 right now).I can see the motivation behind some of the changes, but if I were maintaining a Python implementation, I wouldn't feel any serious desire to track a large part of that list, especially if there were no serious benefits in terms of people actually being able to run contemporary programs on it.
Posted by Doug Napoleone at Wed Jun 20 2007 16:50
I am not sure why you see Py3K as a marketing release (or even a corporate release). This is the ultimate language geek release cleaning up a lot of the cruft hard core python users have been struggling with for quite some time.Porting will be the biggest hurdle. Getting all the third party components upgraded in particular. SciPy will defiantly be upgraded as many changes in Py3K are directly from that camp. The larger integration frameworks will make the jump as much of the API/isinstance/ect. work is something they have been very vocal about for quite some time. It is the smaller projects which only have one or a few people working on them. Many of these projects will just skip over 2.5 (I know we are). It will definatly take more than a year. I would estimate 1.5 to 2 years from the first Py3K alpha release before things completely settle.I see history repeating here. All these same discussions came about back when 2.0 was released and broke compatibility with 1.5.9. There are still many die hards who view 1.5.9 as the last version of python and all later releases as selling out and adding too much garbage to the language. There the upgrade was easier as less had to be changed, but there were no tools to help you either.While concern over forking is warranted I do not believe the risk is that high. Yes there is much more python out there, and yes many projects will not make the jump, just like last time (remember xstruct? xre? webtools? pyhttpd?). I do not believe that there will be enough which are not upgraded to cause a fork. One of the reasons for there being a year between the initial alpha and the release is to give maintainers a chance to do the upgrade. The second are the new features. And the third is the community.The carrot of new features in 3K and beyond will bring people forward. Many people do not see these features as important or useful to them (yet), much in the same way many people did not believe new style classes were useful, or iterators, or generators, or properties, or more recently decorators. If you look at any project out there under active development you will notice all those features being used. As more projects move onto Py3K, the others will follow. Python has one of the best communities out there. I believe the community will band together and drive the upgrade, helping each other and making the entire ecosystem healthier. This is by belief based on past experiences. Only time will tell.
Posted by Peter Fein at Wed Jun 20 2007 16:57
"Considering that Zope 3, a clean, modern, extremely well-tested code base"You lost all credibility with me right there. Most of the non-Zope Python programmers feel that Zope is a horribly ugly hack that is only nominally a Python. A lot of us feel the same way about Twisted, which hasn't released support for 2.5 yet either AFAIK.I suspect a lot of proprietary code will *never* get ported to Py3K, but rather be maintained as legacy apps on 2.x. When it gets scraped (as almost all code does, eventually), the rewrite'll be to Py3K. You'll still be able to share data between 2.x & 3.x programs (whether via database, xml, pickle or text file). So some code never gets converted, what's the big deal? We all support legacy apps, some of which even run on the current version of python. ;-) Libraries are perhaps a bit trickier. The popular ones will certainly get ported by the author/community and those that don't were probably on their way out anyway.
Posted by Fredrik at Wed Jun 20 2007 17:06
"I see history repeating here. All these same discussions came about back when 2.0 was released and broke compatibility with 1.5.9. There are still many die hards who view 1.5.9 as the last version of python and all later releases as selling out and adding too much garbage to the language. There the upgrade was easier as less had to be changed, but there were no tools to help you either."What on earth are you talking about?
Posted by shwag at Wed Jun 20 2007 18:23
You are right, but this is the same problem php went through when going from PHP4 to PHP5. The way it worked out was for the first year,..yah, most people stuck on PHP4. But the after a little bit of time, all the major projects made their code compatible with PHP4 and 5. And all the new projects just started using PHP5. Now the transition is complete. These things tend to work pretty well actually I think. Maybe the linux kernel 2.4 to 2.6 draws some analogies too.
Posted by Martijn Faassen at Wed Jun 20 2007 20:03
Peter Fein, you say: "You lost all credibility with me right there. Most of the non-Zope Python programmers feel that Zope is a horribly ugly hack that is only nominally a Python."You've lost all credibility with me there, so that's mutual. :) Zope 2 is an acquired taste, and one that definitely is showing its age (it goes back to 1997). It's excessive to call Python's most-used web framework as horrible hack, but it's definitely unpleasant in places so I can at least see where you're coming from. I don't agree, but I can see where you're coming from.You seem to have absolutely no idea what the Zope 3 codebase is like (under development since 2002 or thereabouts). You can disagree with its design decisions, but to call a modular framework with extensive test coverage (I bet it'd rank in the 5 of Python frameworks) a "horrible ugly hack" is really uncalled for. Did you know by the way that the work on new-style classes in Python was pushed for by Zope Corporation? (Guido did this work while he worked there)You also seem to imply in your comment that you think Twisted is a horrible ugly hack that is only nominally Python, so I guess you have some rather extreme views.I definitely hope you are wrong about the existence of such massive misunderstandings among other non-Zope programmers. And since I've lost all credibility with you, I definitely recommend against using lxml if you ever need an XML processing framework in Python. (I recommend it to everybody else).
Posted by Doug Napoleone at Wed Jun 20 2007 20:10
@FredrikSorry, the version of python is 1.5.2, not 1.5.9.
I might be mis remembering things as google is not finding any of the relevant discussions I remember. Maybe Guido or Barry or other code developers from the time remember better than I.This was back when Guido left CNRI. Jim Hugunin and Guido both worked at CNRI, and Jim created Jython. Guido left CNRI in the middle of python 1.6 development and took all later python development with him under a new non-CNRI license. Jim and Jython were still at CNRI. There was much talk of a split in the python community back then especially with all the new class changes in python 2.0. There were problems with extension models as the PyObject* base changed dramatically. There was also a big GPL discussion over the old CNRI license which still has not been resolved.One of the people who went to the Boston python meetup two months ago is still using python 1.5.2 and is adamant about not switching. I know many others like him as well.I wish I could find the old discussions on the eminent breakup of python. Google is not helping unfortunately.Posted by Peter Fein at Wed Jun 20 2007 20:11
Eh, I was flamebaiting a bit, thanks for not rising to it. ;-)Yeah, I'm not familiar with Zope 3. My point about Zope/Twisted being marginal Pythons is that they are kind of in somewhat separate worlds from traditional/mainstream Python programming and don't always play well with others. But that' really neither here nor there for 2vs3.Anyway, I came off a bit harsher than I intended. But that's what Usenet's for, isn't it? :-PFunny, I was looking at lxml yesterday. ;-)
Posted by Martijn Faassen at Wed Jun 20 2007 20:15
Doug (and Fredrik): Fredrik's point is mine too. Python 2.0 did not break backwards compatibility with Python 1.5. I don't understand where you're coming from there. The transition to Python 2.0 was a relatively large hump, but it's nothing like breaking backwards compatibility in the language definition.shwag: Interesting comparison with PHP's transition. I'm wondering how possible it will be to have ones project's code compatible with Python 2.x and Python 3.x at the same time. Since large frameworks often already target a single Python 2.x version as its recommendation it would certainly mean a significant additional development burden.Frameworks, libraries and applications with large, active open source communities are going to get ported to Python 3 (eventually). What about the frameworks, libraries and applications that have smaller communities? Some of these will die. Sometimes that's fine, but one does wonder how much useful code is going to waste sooner than it needs to this way.This is not to say that I disagree with the whole idea of Python 3. Breaking backwards compatibility is hard, and sometimes it needs to be done. That doesn't mean we should be simplistically optimistic and ignore the significant risks to our community. We need to be aware of the risks in order to reduce them.
Posted by Martijn Faassen at Wed Jun 20 2007 20:24
Peter Fein: I saw in your blog you had a run-in with the Twisted community on irc. The Zope community is not like the Twisted community and we handle flame-bait differently. :)It's true Twisted and Zope are in their own worlds. That's a price of age, and of popularity. I think you should also be aware that Zope has been working on breaking out of its own world for many years now. This is a continuing process that will likely never end, and one that we actively work on.To give some examples, Zope 3 is using Twisted as its default web server now, instead of its home-grown system. It's also WSGI compatible and we have experimental integration with Paste. Moreover, Zope 3 is being split into many separate libraries (installable as eggs), and while we're in transition now some of the components should start being useful outside Zope proper within the coming year or so. Components already useful outside of Zope are zope.interface and the extremely powerful Zope Object Database. Note that the prime motivation for the rewrite of Zope was to become more Pythonic. We continue to work to stay integrated with the rest of the Python community, as we consider this to be valuable to Zope (and also to the Python community. We're clever people who have learned a few lessons over the years).
Posted by Kumar McMillan at Wed Jun 20 2007 21:32
I'd like to argue that any python project without at least moderate unit test coverage will be slow to make any changes made to, not just porting to py3k. Without tests it is hard to upgrade 3rd party libraries, hard to add new features, and hard to move to even a new point version of python. If that code continues to survive in 2.x land [probably on life support], you know what? the next developer hired to maintain it is going to rewrite it in py3k, with automated tests ;)
Posted by Martijn Faassen at Wed Jun 20 2007 21:47
Kumar: Of course Python projects should have unit test coverage and I agree a good test coverage will help tremendously when doing any changes. But I think there are quite a few real-world code bases written in Python with active users and developers without such test coverage. User interface code, for instance, tends to be hard to test so even applications with test coverage elsewhere might not be so well tested there. These code bases might not be very pretty, but they can be quite valuable to people.The next developer might decide to rewrite it in Ruby instead. :)
Posted by Michael Dillon at Wed Jun 20 2007 22:56
Memory is getting cheaper every day. Why can't Python3000 include a complete binary of the latest Python 2.6 (in a dynamically-loadable library of course) and use that to run any uncoverted code. In order to distinguish Py3k code, require all source files to have a header like the UTF-8 encosing header. For instance, if a source file starts with:
# -*- coding: utf-8 -*-
then you can use any unicode characters in strings and comments as long as your text editor supports UTF-8 encoding. We could require that Py3k files begin with
# -*- PythonVersion: Python3000 -*-
otherwise they will be passed to Python 2.6 for execution.Posted by Michael Dillon at Wed Jun 20 2007 23:03
Or look at it from another point of view.The work on PyPy is very well developed. This work could be the basis of automated translation tools that could be used to translate Python 2.x code into Python3000 code.Imagine a PyTrans 2.4 which runs translates and runs Python 2.4 code in a Python 3k environment. And PyTrans 2.5 which does the same for Python 2.5 code. And so on. Remember that Python both compiles source code into byte code, and runs the byte code. As long as Python 3000 can contain the byte-code support needed for running Python 2.x programs, then such translators would be feasible.And then there is the possibility of simply emitting Python 3000 source code from a translator so that translation is a one-time task.
Posted by Martijn Faassen at Wed Jun 20 2007 23:14
Michael Dillon: either dual-runtime support or a smart PyPy based translator tool would be very useful in easing the pain of the transition. The dual-runtime support is something similar to what I actually did myself for Zope 2, enabling Zope 3 to be used from within it, when I started the Five project. It's going to be tricky but if it can be made to work that would be good.But that does mean someone needs to actually build a dual-runtime system or a translator tool. These are non-trivial efforts and more importantly, I didn't hear Guido talk about tools like that at all when describing the transition process. No commitment to build such tools therefore seems to exist.
Posted by Joe Cooper at Thu Jun 21 2007 00:04
Interesting that the state of Perl 6 is being viewed as something to avoid...particularly given that Perl 6 doesn't break backward compatibility, and that's the core complaint you have with Python 3k. I assume this is merely a misunderstanding of the goals of Perl 6.Perl 6 will be able to run 5 and 6 code at the same time, so it will be entirely feasible to migrate a codebase one file at a time from 5 to 6 with no "everything's horribly broken" moments along the way. No Perl programmer would accept the loss of all of CPAN, even for all of the cool new stuff in Perl 6.I'll also mention that I have Perl code I wrote 10 years ago that runs unmodified on the most recent Perl5 (though looking at it makes me a bit itchy)...I can't say the same of Python code I wrote even four years ago (the jump from 1.5 to 2.x killed it). Seems to me that Perl ought to be held up as a shining example in this discussion, rather than something to heap scorn upon, since it is guilty of none of the sins you're railing against. Sure, it's taking a long time, but Perl 5 is still being actively developed and maintained, so Perl coders will never be without a working and stable base.Otherwise, I agree whole-heartedly with your premise. Python 3k is troublesome. It sounds nice, and I'd probably be just as happy working in Python daily as I am working in Perl, but wholly losing backward compatibility is a major failing.
Posted by Martijn Faassen at Thu Jun 21 2007 01:37
Joe: Interesting about Perl 6. I was assuming Perl 6 breaks backwards compatibility. I read somewhere today that it accomplishes (will accomplish?) backwards compatibility by basically combining the old and runtimes together, is that correct?I think Guido's comment about Perl 6 was mainly about it taking an extremely long time to get developed. The assumption is that this very drawn-out process brings much insecurity to the Perl community. Python 3 has more modest goals *for the language implementors*, even though it breaks backwards compatibility - backwards incompatibility is its primary reason to exist.
Posted by Joe Cooper at Thu Jun 21 2007 02:45
Dual runtimes appears to be one model under development. There is also a Perl 5 Parrot implementation. I'm not sure which of those two approaches is the officially sanctioned one that will ship when Perl 6 goes live.It has been a very, very long dev cycle, no doubt about that. But I don't think most Perl developers think much of it. Our company isn't waiting around for Perl 6 in any way. We'll continue developing in Perl 5 predominantly until Perl 6 has a large installed base (a couple of years, probably), since most of our software is installable on the end user's machine. I guess because of the relative age of Perl and our codebase, we don't agonize too much about it as a language...we aren't going to rewrite 300k lines of code under any circumstance. But I guess for new projects at the "So what language?" phase, it probably is hurting Perl's uptake to have a relatively slow moving, and somewhat crufty in places, stable branch.
Posted by Chris McDonough at Thu Jun 21 2007 03:20
God knows it can be done successfully (Apache 1 -> Apache 2). I suspect it's easier to swallow when you're not so close to it. ;-)I think most people who are brand new to Python will appreciate Python 3000 (having separate types for unicode and 8-bit strings sucks, the new-style vs. old style class split kinda sucks, etc). But people who have an intestment in existing Python 2 code will likely be using it for at least a few years after the first release of Python 3000 and maybe indefinitely. The good thing about this is that there are a lot of them, and therefore Python 2 will have a long life ahead of it.Recently I used Adobe's Flex 2 framework to write a flash app. I'd never written a flash app before but I knew I was trodding on the backs of some poor fuckers who had put up with the Flash IDE for years to build applications. I felt a little guilty for having it so easy, but I was glad that I didn't need to do that. ;-) I think for new users an easier Python is a good thing, and I think there's enough people in the Python community to maintain the older stuff and do interesting crossovers (like Five did for Zope 2 to Zope 3).
Posted by Chris McDonough at Thu Jun 21 2007 03:28
Wrt to the specific things that the Py3K PEPs introduce: - The new string formatting syntax and behavior proposals has a number of
pretty fundamental issues. If you care about this, I'd suggest reading
http://www.python.org/dev/peps/pep-3101/ and commenting about it
on the Py 3K list. - ABCs seem to ignore the use case of wanting to ascribe behavior to
instances (as opposed to classes).
Maybe this is OK but I really like that feature of Zope interfaces
(directlyProvides). I also think inheritance is the wrong vector
through which to describe behavior. I think Zope interfaces have it
more right here.Posted by Ycros at Thu Jun 21 2007 16:15
Take a look at Guido's recent post on py3k status: http://www.artima.com/weblogs/viewpost.jsp?thread=208549He's saying that 2.6 will be backwards and to some extent, forwards compatible, and there is also a python 2 to 3 conversion tool in the works.
Posted by Martijn Faassen at Thu Jun 21 2007 17:04
Ycros: I have read Guido's recent post and in part this is what I was responding to it. He clearly states that you will have to have a pretty good unit test coverage of your code base to attempt conversion. That excludes all those code bases that do not have such coverage. Additionally, since the explicit goal of Python 3.x is to break backwards compatibility with Python 2.x it is *impossible* to fully support the important changes Python 3.x makes in any Python 2.x version. Finally, don't put too much faith in conversion tools, especially a tool that is already described as not very smart.Even code bases that have excellent test coverage are not always trivial to port to even a 2.x update of Python. Language changes that break backwards compatibility will certainly be a large challenge. Therefore I suspect that many communities surrounding larger code bases will put off any conversion for a significant period of time. My conclusion is that we are in danger of a fork that will last many years. We may want to accept this risk as a community, but we should not pretend it doesn't exist.
Posted by Jon Rosebaugh at Thu Jun 21 2007 23:19
Chris, what are the fundamental issues with that new syntax? It looks a bit ugly to me, but workable.What I really wish we could have would be Ruby-style string interpolation:"This is a string with Ruby-style interpolation. #{someval} is the value from the local scope."And anything in #{} is just evaluated and put into the string. It's self-documenting!
Posted by Chris McDonough at Fri Jun 22 2007 15:17
Hi Jon,Take a look at this thread on the Py3K maillist:http://mail.python.org/pipermail/python-3000/2007-June/008479.html
Posted by Guido van Rossum at Sun Jun 24 2007 01:46
Martijn, will you be at EuroPython? We should talk there.
Posted by Martijn Faassen at Sun Jun 24 2007 15:03
Yes, I'll be at EuroPython. I'd be happy to talk. (conversations about this topic will be inevitable at EuroPython anyway :)
Posted by Michael Dillon at Tue Jul 03 2007 21:20
I must admit that I see no reason to upgrade from Python 2.4 at present. 2.4 has enough of the newer useful features like iterators and list comprehensions, but it does not suffer from the lack of binary modules that newer versions suffer from. If you are on a Windows platform, you basically need to have the developper of a binary module provide a binary that supports your version of Python. 2.4 has the widest support by developers.Also, 2.4 doesn't suffer as many EGG problems, for instance you can always tell when a module is binary, not just source code wrapped in an EGG.I for one would be happy if Python forked at Py3000 so that we can continue developing an EGG-free and compatible 2.x stream.
