Python Secret Weblog for 2008 April 14 (entry 0)

(12) Mon Apr 14 2008 16:22 Porting Zope to different Pythons:

Zope has been under development since 1996. It wasn't called Zope back then yet: it was called bobo, and later pieces were called Principia, and only in 1998 did it become Zope.

When Zope was born, there was just one Python implementation: CPython. CPython is important. It's the main line of Python evolution, has the most libraries available to it, and can integrate with all sorts of C and C++ libraries, and even other languages such as Java can be integrated with it.

Then there was another implementation, born in 1997: Jython (originally JPython), Python on the JVM. Unfortunately after a good start, Jython's development lagged behind CPython's for some time. More recently development of Jython picked up speed again, and received actual backing by Sun. Jython is very interesting as it can speak to Java libraries natively. Java has lots of libraries.

Meanwhile, in 2004, the originator of Jython, Jim Huginin, came up with another implementation of Python: IronPython, on top of Microsoft's common language runtime infrastructure. Microsoft hired him to work on it further and IronPython has come far since then. IronPython is also interesting as it can speak to lots of .NET platform libraries natively. In addition, IronPython runs on top of Mono too, meaning there is what looks like an open source implementation of this stack (no license flamewars please, though. I may be wrong). Not very open sourcy is that last I heard, Microsoft developers cannot accept patches to IronPython from the outside. Still, it's very interesting technology.

Then there's PyPy, in development possibly longer than IronPython actually, since 2003. PyPy is, in theory, very compatible with CPython. The PyPy Python interpreter can, in theory, be translated to run on the "C platform", on the JVM, and on the CLR, among other things. There is, in theory, potential for tremendous speed boosts once JIT technology is integrated into it, as is intended. It also allows, in theory, all kinds of interesting features in the interpreter that other Python implementations don't allow.

You noticed I used the words "in theory" a lot concerning PyPy. Unfortunately so far PyPy's coolness, though they have lots of working code, remains mostly theory. You can dream about PyPy but not really use it. The PyPy developers are not showing public signs of actually working towards a release we can all use. They are not actually showing public signs of working towards any release at all: their last release was over a year ago and a new release doesn't appear to be in sight. They seem to prefer developing in their SVN repository, organize sprints and present at conferences. In my book, if it's not actually used for real-world projects it can still be cool, but only theoretically cool. This makes PyPy rather a dark horse that may become useful one day. I hope it does, but I'm not holding my breath anymore.

This year, 2008, Python 3.0 will be released. It's actually the least compatible version of Python of the list so far, which is risky. It will most likely become important as significant amounts of people start using it as their main development platform.

The varieties of Python don't end there. There are Python interpreters that aim to be small in some sense, such as TinyPy and PyMite. There are the Python implementations geared towards translation to efficient C or C++ code: Pyrex and its spinoff Cython, and Shedskin. PyPy's RPython subset can also do things like this, but is only used for PyPy's implementation itself so far.

One other variety of Python that received lots of hype in the last week is the Google App Engine. It's of course CPython 2.x, but it doesn't allow write access to the file system, or opening sockets, nor the installation of libraries that contain C-language components (unless Google does it for you). If you want to store data, you will have to use Google's way to do it. This means that porting code to the App Engine is indeed that: a porting activity, just like to the other Python versions.

Zope, today, runs on CPython 2.x only. Given all these other Pythons around, this is a situation that probably can't and shouldn't last forever. When I say "Zope", I mean the Zope libraries as much as the framework itself - Zope has been moving to a reusable-library approach for some years now.

We have already seen the first work towards spreading Zope technology beyond CPython already. Some porting work was done, I believe by people from the Twisted team, who also use it, to translate one important core package (zope.interface) to Jython.

The Jython project is hoping to start a Google Summer of Code project this year to port more Zope 3 packages, such as zope.component to Jython. The Zope Foundation is very grateful to the Jython project for taking this initiative; I believe it's of great importance to Zope as a whole. So, we hope that project gets accepted. The Zope Foundation going to make sure the project is well-supported by the Zope community.

Hopefully the porting work to Jython can be done without the need to fork the packages involved. Forking packages is bad, so the hope is that the plain Python components can be tweaked to work with Jython while they remain compatible with CPython. This leaves the C-based components. Luckily Zope has maintained parallel Python versions of much of its C code, which will hopefully make it easier to port these packages. That said, it may well be that some drift has happened between the C versions and the Python versions, and the Python code may need to catch up. There is also C code that doesn't have a Python equivalent.

One of the hardest packages to port will be the ZODB, the Zope Object Database. It does significant meta-class trickery and has a significant amount of C code that doesn't have Python equivalents. Luckily most Zope libraries aren't dependent on the ZODB at all. We also devising ways for Zope 3 to run completely without the ZODB. Instead, object relational mappers (ORM) can be be used instead. This means that even without porting the ZODB, we can still use Zope much as it is.

There is another option that might help us allow the use of the ZODB on Jython: RelStorage. This is a ZODB backend that is backed by a relational database backend instead of by the usual big file in FileStorage, the most-commonly used ZODB backend. It may be that RelStorage is easier to port to the Jython platform than FileStorage, but perhaps not. Let the experts speak out!

Porting to the Jython platform will also teach us important lessons about porting to other platforms. If we can make Zope work on Jython, it may also be easier to make it work on IronPython. Or PyPy.

What about the current hot topic, the Google App Engine? Can we port Zope to that? Zope's C libraries won't work, just like with Jython. That's the same problem as we have with Jython! There is a synergy between these porting projects. Hopefully the Jython port (cross fingers that it'll happen!) will help make sure Zope works without the C extensions, and therefore also on the Google App Engine.

With the Google App Engine, we will also need to use Google's way to store data. We can integrate with that like we integrate with an ORM. There has also been some talk about porting the ZODB on top of it - the ZODB supports multiple backends, so why not Google's? I'd love to hear from Shane Hathaway, the author of RelStorage, to see whether RelStorage's approach could be used for this.

If we can make Zope run without the need for C extensions, and we have ZODB backends for the various platforms out there, we may be looking at a future where many run Zope applications on the Java platform, .NET, the Google App Engine, or maybe even PyPy.

---

Comments:

Posted by Laurence Rowe at Mon Apr 14 2008 18:02

The difficult part of porting ZODB would be the `persistent` package (which has no pure python equivalent). The storage implementations in themselves are pretty simple, they just deal with storing bytes.

The Google App Engine's idea of transactions is a little different to Zope's `transaction` module, but this would not be a major problem for application code, probably confined to the publisher. I would however question the wisdom of bothering to port ZODB to the App Engine, I'm sure it would be more efficient to use the inbuilt persistence machinery.

Posted by Martijn Faassen at Mon Apr 14 2008 18:24

Thanks for the answer Laurence! What do you mean by using the inbuilt persistence machinery, though? Built-in where?

Posted by Rocky Burt at Mon Apr 14 2008 19:01

I suspect what Laurence is referring to is google's inbuilt persistence machinery. In other words, it'd be far more efficient to build a Zope 3 app that accessed google data store directly instead of zodb-on-top-of-google-data-store.

Posted by Martijn Faassen at Mon Apr 14 2008 19:13

Ah, all right, understood. Yes, we could simply use Zope without the ZODB. That'd definitely be easiest and have faster performance.

But Zope, with its ZODB, is actually in a unique position here. If we port the ZODB, we could run the same web application in the Google App Engine as elsewhere. That is a benefit that other web application frameworks that are ORM-based don't have. This could be a significant advantage.

Posted by Justin Ryan at Tue Apr 15 2008 01:19

I agree with a lot of the sentiments here, except for the triviality of PyPy. The last release of PyPy is pretty significant, and there is some great work coming up in GSoC. I don't know how much *influence* I have, but I'd be glad to help cut releases here and there - I certainly have trouble getting a working CLI/.NET interpreter all the time, though SVN usually works.

I'm not sure if there are good tags that people aren't aware of, or if perhaps tagging would at least be helpful. PyPy is certainly a research project, but IMO, and in many ways, esp with CarbonPython, has a lot more promise and license difficulties than IronPython.

I do think that getting the ZODB to work is important, though not crucial as a first step. Having recently dug into this challenge, I've found the most guidance in PEAK's Persistent package, though it is deprecated, it is in Python and deals with _p_changed adhering to the interface, afaik. I believe also that in older versions of Zope there may be a pure Python Persistent meta/super.

I would like to point at a project I am working on as an Honors project for my C# course this semester. In some ways it is a Reductio ad absurdum, or Reduction to Absurdity, of implementing Zope in C#. My underlying goal is to determine if key portions of Zope could be implemented as IL code, generated from RPython using an advanced version of the CarbonPython concept.

IL is "Intermediate Language", the object-oriented assembler at the heart of .NET/Mono/CLR, which has some capability not exposed by C#, though I have found C# to be a decent language in many respects, if extremely verbose.

http://tinyurl.com/5wp42s

Posted by Justin Ryan at Tue Apr 15 2008 01:20

er, I meant "less license difficulties", in concert with "more promise", on the front of PyPy. whups. :)

Posted by Sidnei da Silva at Tue Apr 15 2008 01:51

Just pointing out that I raised some similar ideas about a week ago. Martijn must have missed it:

http://awkly.org/2008/04/08/plone-on-google-app-engine-anyone/

Posted by Martijn Faassen at Tue Apr 15 2008 02:24

Justin: I never claimed PyPy is trivial. PyPy is certainly not trivial. The last release of PyPy, over a year ago, was not useful for use in a real-world setting; I don't think the PyPy people claim it is. I don't know what the current plans are for a new release that is, or whether they actually have plans.

Sidnei, I didn't miss your blog entry; I don't claim credit for the ideas, and my apologies if I seemed to, I didn't intend to. All the Google App Engine posts were swimming together in my head. My post is about more than just about the Google stuff. I think the Jython port is at least as interesting.

Posted by Shane Hathaway at Tue Apr 15 2008 05:20

This is a nice overview of Python implementations, although I'm sure there are other interesting ones, such as the new tinypy. (Maybe tinypy on a microcontroller could run some of Zope. ;-) )

Laurence is right about the persistent package. If that can be ported, creating a storage similar to RelStorage won't be hard. OTOH, that's still probably not the right goal.

I would suggest that many of the other things built into Zope are more important than porting ZODB, such as object publication, interfaces, and the component architecture.

Posted by Shane Hathaway at Tue Apr 15 2008 05:21

Ahem, I see you did in fact mention tinypy before I did. Oops. :-)

Posted by Martijn Faassen at Tue Apr 15 2008 17:20

I agree that we should focus on the other bits first before tackling the ZODB. This makes sense for Jython. That said, if we can port the ZODB to these other platforms, we can run our code as it is on them.

This is something a ORM can do as well on most platforms as long as they offer relational database integration, but not for the Google App Engine for instance.

Posted by Frank Wierzbicki at Wed Apr 16 2008 18:24

In response to forking Zope to run on Jython: I would also hate to do any kind of forking of Zope components, I think that would be very counter-productive. We would certainly hope to contribute any adjustments needed to get Zope working on Jython back to the Zope community.