Python Secret Weblog for 2007 September 26 (entry 0)

(5) Wed Sep 26 2007 14:53 the challenges of version management in an eggified world:

Zope 3, and Grok in the last few months have been switching to a brave new eggified world of installation. The idea is that you compose your Zope application from a large amount of smaller packages, each providing their own components. I've sometimes described this Zope as an integrated megaframework. Zope is an integrated framework where packages follow common coding conventions, and the component architecture defines a way for packages to work with each other. Grok tries to step up by aiming for an integrated feel for developers. At the same time, Zope is a megaframework, allowing you to swap in best of breed components as they come available. Don't like zope.formlib? Swap in z3c.form for your form generation needs instead.

So how does eggified Zope work? You use zc.buildout to manage your development project. This buildout gathers eggs together in one place, looking at requirements in the setup.py of the various packages, and sets other programs like a start server script, a test runner, and so on. eggs that aren't installed locally yet will be downloaded from the Python cheeseshop and other locations. eggs aren't installed system-wide, keeping the system python nice and clean. What's more, different projects can easily use different versions of the same library. Since zc.buildout is easily extensible with new recipes, many complicated needs can be covered. To make initial installation of a buildout easier, Philipp von Weitershausen has developed zopeproject and grokproject to help you set up new Zope 3 or Grok (pick your flavor) projects.

Being at the forefront with eggs and buildouts means we also have some interesting challenges. I'll describe one that has been biting the Grok project more than once recently. This post describes the various concerns that we have with version management, and a proposed solution.

So, what is our problem? A while back, we made the 0.10 release of Grok. Grok is a framework and depends on many Zope 3 packages (among others). This is specified in the setup.py of Grok, like this:

install_requires=[
   ... long list ...
  'zope.proxy',
   ... more dependencies ...
   ]

Unfortunately, this approach has a problem. If someone releases a new version of, for instance, zope.proxy to the cheeseshop, newer installations of Grok will try to use this version instead of the version that we tested Grok with.

This is asking for trouble: we have made a release, but what people actually install keeps changing! No wonder that we've had several breakages of Grok 0.10 as people accidentally broke backwards compatibility, or mistakenly released broken eggs. Since these packages are also used by other Zope 3 applications besides Grok, we cannot ask these people to stop making such releases - this is a megaframework and individual packages should be allowed to evolve at a different pace.

How to go about fixing this problem? The simplest approach would be, whenever we release a new version of Grok, to hardcode a full list of the packages we depend on with exact version numbers in the install_requires section of Grok's setup.py, like this:

install_requires=[
   ... long list ...
  'zope.proxy == 3.4.0',
   ... more long list ...
   ]

Doing this would mean that anyone who installed Grok would get exactly those versions, nothing else. If someone tells us: I used Grok 0.10, we know exactly what that means.

Unfortunately it also locks in application developers into those versions exactly. If a bugfix release of zope.proxy comes out, the application developer that uses Grok cannot start using this new version, but instead will need to wait for a new release of Grok that depends on this newer version of zope.proxy. While that's often a good approach anyway, hardcoding the version dependencies does limit the developers that build their applications (or frameworks) on top of Grok.

There's another problem. Grok isn't the only Zope 3 package that uses these packages. zope.component for instance depends on zope.interface. If zope.component hardcodes a dependency on a particular version of zope.interface, the Grok developers would need to wait for a new release of zope.component in order to get a bugfix in zope.interface too. And remember where we came from: the whole idea of our megaframework approach was to have the flexibility to recombine components, and this would be blocking it.

Components, and frameworks, ideally should have weak dependency requirements to be maximally usable, allowing individual developers or framework developers to use the versions they want to. But on the other hand, if someone uses a framework, it should continue to work tomorrow. If someone releases a framework, it should remain installable tomorrow. If someone communicates to someone else about framework versions (important in open source software), they shouldn't have to give a list of 50 version numbers, but just one.

We therefore have two different requirements pulling in different directions. On the one hand you don't want to lose flexibility, on the other hand if you want to have a community working and reusing chunks of software, you want to be able to rely on stability, and frequently you even want to count on bug for bug compatibility.

To allow flexibility, instead of hardcoding version numbers in install_requires in setup.py, you only loosely specify them. You say, for instance, that zope.component requires zope.interface, but not which version. If you know that your version of zope.component needs a feature that's only in zope.interface 3.2 or later, you'd write zope.interface >= 3.2.

Now we're back at our original problem, however: we got flexibility, but damage stability. What if someone makes a new release of some dependency of Grok?

zc.buildout has a feature that can help us pin versions down for our particular application. We could ask all the people who use Grok for their applications to put the following section in their application's buildout.cfg:

[buildout]
...
versions = grok-0.10

[grok-0.10]
... long list ...
zope.proxy = 3.4.0
... more dependency specifications ...

This can be made work well if you don't use a framework like Grok but instead develop an application from scratch that uses a long list of components. But in the case of Grok we want the framework to specify these dependencies. We don't want to require all application developers to replicate a long list of dependencies in their buildout.cfg. It's easy to make mistakes, it's hard to communicate about such lists to everybody, and what do people do when they need to perform an upgrade to a newer version of Grok? They'd need to get a new long list and edit their buildout.cfg again. It'd be a lot nicer if they only had to deal with the change of a single version number instead.

Zope 3 has a culture of where making developers figure things out for themselves is okay if this leads to maximum flexibility for everybody. Grok has a different approach: it tries to make things easy, too. Telling developers to maintain long lists of version numbers is not good enough for Grok, and probably not good enough for other frameworks built on Zope 3 either.

So, we need a way to have the Grok framework developers maintain this list of versions in a central place, and allow all application developers that use Grok to reuse this list.

zc.buildout does have a feature to help us here too: it can include bits of external buildouts into your own, using a URL. You can use a pattern like this in your buildout.cfg:

extends = http://grok.zope.org/releases/grok-0.11.cfg

What this would mean is that developers that use Grok place this in their own buildout.cfg, and we maintain the list of versions under that URL. When a new release of Grok happens, we create a new URL and ask people who want to upgrade to update their buildout.cfg to reuse that:

extends = http://grok.zope.org/releases/grok-0.12.cfg

That should be the only modification they need to make if they didn't hardcode any dependencies in their setup.py. And if they want to override a version number they can still do so in their own versions section, so we retain flexibility.

This is likely the approach we are going to use in the near future. It's pretty good, but not ideal, so I'm going to go into some of the drawbacks of this next.

For one, this doesn't work in locations you don't have internet access, such as on a train. Now this problem exists for egg downloads as well, but in typical buildout setups, you'd have a lot of eggs available in your home directory available that you downloaded previously, so there's a good chance you can still create a new Grok project even while you're on a laptop in the train.

Another problem is that the release managers of Grok will have to deal with two release artifacts instead of one: besides the usual, easy, automatic package upload using python setup.py sdist upload whichg places the new version on the cheeseshop, we will now also need to maintain a list of dependencies somewhere and create a new URL whenever we release Grok. We also need to communicate this new URL to our userbase, and this is different from the usual Python dependency mechanism, which is defined in setup.py. This isn't a major problem, but it makes the release process more cumbersome nonetheless so it's less than ideal.

There is another potential drawback to this approach. Dependency relationships form a tree. Grok may depend on zope.component, which in turn depends on zope.interface. In order to pin down the version for zope.interface, we would need to do this inside Grok. This it not a big problem for such a small dependency, but when frameworks start to depend on frameworks it will start to be cumbersome to create a unified list of dependencies. This may sound theoretical, but in the Zope world it's common to have frameworks that depend on frameworks: Plone depends on CMF which depends on Zope, for instance. If someone were to write a CMS in Grok, they would need to maintain and publish their own list of version requirements for that CMS, which would include the entire list for Grok. It'd be nicer if they could just say: here's my list, and for the rest, please reuse Grok's list.

I think many of these problems could be resolved if we could specify this list in a package's setup.py instead of on a URL. A package would have an optional extra section in their setup.py besides install_requires, perhaps something like install_recommends. This section would contain version number recommendations that have been known to work with this package. Tools like buildout could then choose to make use of this information, but the developer is also free to ignore it. This would solve our "URLs cannot be accessed on a train" problem. It would allow us to do simple release management again with a single release artifact: all the information will be available in the egg instead of on a separate URL. It would also open the door for smart tools which can combine version recommendations from various packages into a longer list.

Jim Fulton, creator of zc.buildout, told me that I'd need to convince the distutils-sig for the need for a install_recommends section first, and wished me luck. So I hope I can get some of the distutils people to read this blog entry. :)

Update: I've just made an install_recommends proposal to the distutils-sig.

---

Comments:

Posted by Charl at Wed Sep 26 2007 22:12

Here is a heartfelt plea for consolidating effort.

When talking about megaframeworks; sometimes one must consider an even larger "framework" which is the deployment environment as a whole. For example, some of us also have Python components that depend on specific versions of Python C libraries or other (non-Python) executables or libraries; or even depend on specific tools that have to be built, just to be able to build the application.

(For example, sometimes it is attractive to use SMC (smc.sourceforge.net) to generate Python state machine code. SMC bugs in outputting Python code get fixed from time to time, and break existing workarounds).

I think there is much benefit to examine lessons learned with other buildout tools, and to incorporate their evolution into the "python component, software config management" thought process. Config management / build tools aren't trivial, and the more weight in the community, the better chance we will end up with a widely useful, well-maintained, stable solution.

For example, BitBake is a Python-based build tool, which when used with the OpenEmbedded metadata (openembedded.org), is probably the premier solution for building entire (linux) software distributions for embedded systems. This includes picking a suitable combination of Python components, picking the proper support libraries, compiling executables (incl the Python interpreter), compiling the dependencies for those executables, etc. Even down to compiling tools which don't go into the distro, but are needed to build other things (like the cross-compiler). The idea is somewhat akin to Gentoo ebuilds, with OpenEmbedded's focus on building software that has to run on a different box entirely.

For example, in goes a bunch of source code for a mobile phone (which might include Python code), out pops a firmware file that may include rootfs, maybe kernel, and maybe even bootloader. The advantage being that you can version control the source for *every* single piece that goes in, and every tool that had to be built to facilitate the rest of the build. Thus you can *really* reproduce the entire build at any time, in a way that I don't think eggs/zc.buildout really enables.

Of course BitBake is not focused primarily on Python-based code. Still, conceptually, there is quite a bit of overlap with zc.buildout (esp where buildout is also used to build things like Varnish cache), while differing holes remain in both stories. E.g. BitBake/OpenEmbedded imo allows for much better sandboxed approach, where every build can end up with its own islolated Python interpreter & install; and it is easy to fall back to local storage when downloads aren't possible; yet the finer-grained component dependencies remain tricky.

I guess what I am trying to say, is that it is frustrating to see two different Python communities trying to solve the same class of problem, and not seeing any dialog or sharing of ideas.

For example, I would love to see more egg-ish capability in BitBake (expanded along the lines you are talking about) to standardize better finer-grained Python component selection and dependencies. And I would love to see the capability of building an entire Zope server deployment rootfs (maybe a complete EC2 image, for example), which BitBake is probably better suited to than zc.buildout.

And I'm frustrated that, (just counting eggs, zc.buildout and BitBake), we have three different ways of expressing metadata geared to solving variations on the same config management problem.

Posted by Martijn Faassen at Wed Sep 26 2007 22:24

Thanks for this plea. Consolidation would certainly be nice. It also depends on how much use cases diverge.

What is bitbake's primary use case? zc.buildout is used to set up development projects primarily, in which a developer will work on python software. It can actually be taught to install C tarballs too with the appropriate recipe, though it is likely far less solid than bitbake in that area.

bitbake as far as I understand is a tool used to put together linux distributions. Typically we have like 20 buildouts sitting on a developer's machine, which all may use different versions of the same packages, and which typically have one or more packages actually installed in development mode. What does this look like with bitbake?

Concerning metadata: zc.buildout and eggs share the same distutils defined metadata. There is already a lot of consolidation here; zc.buildout just allows you to more precisely control which eggs you get and where they go.

What would be interesting to find out is what kind of dependency management system bitbake includes, and whether this would be reusable. This is not a very strong point of the current buildout/setuptools world.

Besides development issues there are of course also deployment issues. zc.buildout is currently not trying to solve those primarily, though I suspect it will be used that way more as we go into the future...

Posted by Charl at Thu Sep 27 2007 00:54

BitBake (BB) is essentially just a "task executer", that knows how to read a set of recipes and figure out how to resolve dependencies to "bake" a requested "result".

In one invocation, you tell it that you want to perform a particular action (say "build") on a recipe.

In another invocation, you just tell it that you need a particular result. (For example, "I need 'sox' "). It will then figure out what recipe provides that result (specific version can be requested), what other results (leading to recipes) are depended on, and what sequence of actions have to be taken with those recipes to produce each interim and the final result.

(Recipes in turn can utilize common "classes"; these can contain or be python code).

As such its primary use case is to be a generic a dependency solver / task executer. A "result" can really be anything.

It is the *metadata* (recipes, classes, config files) which really gives it purpose and targets it to a specific use case. For example it is the OpenEmbedded (OE) metadata specifically which purposes it to building embedded linux distros (and this metadata defines, for example, even basics such as that the typical sequence for a package is "fetch, install src, configure, compile, populate staging, install into install area, make pkg etc"). Given other metadata, there is no reason why BB couldn't be used to build a book or a piece of music. (Ok maybe this is a little far fetched, but I'm trying to underline that its purpose is to be generic).

This makes BB difficult to understand initially, because on its own it is essentially abstract. Still, BB needs to be considered as distinct from a collection of metadata (such as OE).

For example, the OE metadata structure not only allows me to build an entire rootfs -- I can ask for it just to build a specific app, even to build me a package (e.g. ipkg, rpm, deb or even possibly an egg) of that app or component. Typically BB will then interpret the OE recipes to discover it must download some code, maybe run some executables over them, add some files and package it up in a specific format (in the process maybe including dependency information, extracted from its own metadata, converted into whatever format that package needs it in). Or just a subset of these, depending on what its done before.

So for example, say I had a component "abc.xyz". I might represent the analog of an egg with a recipe that say:
- this is where you find the source files (locally, svn, cvs, tarball)
- this recipe depends on these other "results" to run (translation -- e.g. "abc.def" also needs to be installed for "abc.xyz" to work)
- this recipe depends on these other "results" in order to produce this result (E.g., maybe some code needs pre-processing by a custom executable; find another recipe that describes how to produce that pre-processor).

Conceptually this is similar to the metadata encoded in an egg; theoretically I could now ask BB to produce me an actual egg that contains the source and encodes the "required to run" dependency information. A consumer of the egg need not know BB was involved at all.

But just as easily, the BB recipe could further describe that if I did want to install (say) this component, that it needs to use distutils (or theoretically setuptools). (Typically just add a line to the recipe saying that it inherits from the distutils recipe class).

So in the OE setting, I've used this in practice to encode the dependency information that would these days go into the egg, into a BB recipe. For example I might have a recipe for "dateutils". If I ask BB to produce just dateutils, it will first build a brand new python interpreter (if it hasn't already), then install dateutils into that environment. (And also produce just the dateutils as an ipkg / rpm / or whatever package format I might be interested in. In fact typically it produces the end rootfs by building ipkgs, then installing all the relevant ipkgs into the rootfs area. But this is a function of the OE metadata collection, not BB. We've even done classes that use distutils but generate and keep only the "pyo" files to save space).

Then in the recipe for my app, I would encode that it is dependent on dateutils (and whatever other components I need). If I then ask for my app as result, BB/OE will ensure I have an isolated Python, with dateutils installed into it, plus everything else I've asked for, plus my app. And I could also end up with an ipkg / rpm (or theoretically egg) that contains just my app and encodes the dependency information.

In encoding that a dependency depends on specific results, I can encode dependency on specific versions of results (which in turn will look for a recipe that promises to provide that specific version of the result).

I can even define several different alternatives that will provide the same result, and arrange that if certain recipes are involved, a specific "provider" from this list of alternatives gets used. (So for example I could have several different recipes that have alternate ways of "providing" "abc.xyz" version 1.1.2).

So it feels to me that the BB recipe metadata format can represent what's currently in the egg metadata, as well as the further "combine eggs into something larger" of zc.buildout recipes, as well as the "give me this type of combination" typically in the buildout.cfg, but in a single metadata format.

(E.g. the recipe for a mobile phone in BitBake may be just a recipe that says "I provide mobile phone 1.1, I depend on bootloader 1.0, latest kernel, application 5 to run". Albeit that there are still often choices as to whether things are selected in a recipe, a class or in a config file).

So whether I'm spelling out the dependencies / recipe for a component, or combining components into an application, or asking for an application to be created within a repeatable deployment environment, or even asking for that deployment environment to be produced as an EC2 image -- its just more recipes written in one style in a single metadata collective.

Compared, egg metadata, zc.buildout recipes, and buildout.cfg feel to me a bit divergent -- each like a different metadata format (this is just perception on my part, not necessarily reality ;)

With BB I could have one (or multiple) copies of the metadata collection, and run bitbake/OE to produce 20 different "working folders" on my PC -- each in a folder that will end up containing everything that I would want in a self contained environment. The difference with the workingenv debate etc, is that you will end up with 20 separate Pythons, all completely isolated. Which makes it really easy to have Python 2.3.2 + xyz 1 in one, Python 2.4.4 + xyz 2 in another, and Python 2.5.1 and xyz 3 in yet another, all isolated from the system python. Or the same versions, but in one case some (or all) in development / debug mode, while another is used to rebuild the same but in production mode. Or if some of them didn't need the "datetime" module (say), it could be omitted.

As to what zc.buildout is used for, maybe this is slightly philosophical, but it seems to me the desired outcomes are:
- one developer wants to work on code and easily be able to reproduce consistent builds of something (say a whole Plone site)
- multiple developers are working on the same project and want to make sure that each can reproduce the same result (e.g. whole Plone site) on their dev systems
- these developer(s) want to be able to stage this out to somewhere in a repeatable fashion
- and hopefully eventually something must be put into production, in a repeatable fashion.

So my perspective on zc.buildout is that the idea is to allow developers to set up and work on something in "dev" mode that resembles closely (or can easily be turned into) the "production" environment, in a repeatable manner, without polluting the dev system itself, so several of these can co-exist. This is exactly how we've used BB/OE in the development of python code for embedded systems!

It seems to me that even substituting 'whole Plone site' above with 'single python component', BB (with appropriate metadata) is applicable as a buildout facilitator for development.

When we start talking about combining components into apps, apps with other apps, apps with caches, producing from scratch a full server install (down to EC2 image), multiple images (load balancers, zeo clients, zeo server etc), I have an increasing sense that BB has already laid the groundwork and attempting to address this with zc.buildout is reinventing the wheel on a hard problem.

I'm interested in the comment about zc.buildout not primarily trying to solve the deployment problem. Maybe so, but is it not true that a buildout tool for use in development looses value and engenders frustration, if it doesn't make final deployment to production / staging just as easy as buildout during the dev process? (Incidentally I've seen comment that this is a huge irritation with RoR, but I can't say from own experience). But I can certainly see that people will be attracted to zc.buildout because they also expect it to facilitate deployment of code to production servers. (Not that it doesn't currently).

What BB doesn't do, is the "make" type of dependency of "This single input has changed, so the following 3 results have to be rebuilt, and the end-result can then be achieved from that combo plus the other 9 results previously built." In BB if a result has been "baked" once it is deemed done and available to any other recipe that needs it -- unless one explicitly "cleans" that baked result away again. But again how this fits together is dependent on the metadata.

Ironically, prior to eggs gaining momentum, I sometimes constructed BB recipes to put small Python components into specific paths, and the embedded system loader to make sure to set the PYTHONPATH correctly. So I am not all discounting that there may be use for zc.buildout in some fashion from inside BB or its metadata!

So I would really recommend looking at what BB (as a tool of its own right) has already achieved; plus to look at a set of its established metadata (ie OE) to get a sense both for how BB could be used, as well as for alternate ideas on how its metadata is already trying to solve related problems, and the holes BB/OE themselves are trying to address.

As I noted earlier, I get this distinct sense of python communities with overlapping interests, without any current dialog. I do think there are significant synergies which could be leveraged off one another to provide even more traction so that we have an even better idea of "best" or "proven" practice.

Posted by Lennart Regebro at Thu Sep 27 2007 09:02

Just some loose ideas:

The install recommends seem to be a good thing. But There is another thing that might help, especially in resolving recommendations as well as possible: Somehow telling other eggs that the API has changed.

Basically, there should be some way to tell if two packages are completely exchangeable, ie the only difference is bugfixes, or if it has some new features or change in the API. Basically, there needs to be an API version number.

I have myself used the following version numbering for packages for packages to be used as libraries: x.y.z
z: Increases for every release.
y: Increases if there are new features.
x: Increases if I break backwards compatibility

This way we could specify v 3.4 and be reasonably sure that we'll get bugfixes but no new stuff. In fact, as long as we specify version 3, it should work, even if a 3.5 gets out, because if we end up having to break BBB or deprecate stuff, then the next version should be 4.0.0, and the specification of 3.x.x shouldn't use it.

This partly breaks with the version numbering used so far for Zope 3, where 3 was fixed an immovable, but we could of course use 3.x.y-z or simply just 3.x.z, or any other system.

The main problem with this is to get people to not break anything without increasing the numbers, but at least, if somebody released a 3.5.7 that break everything, what could be done is to quickly remove that package from CheeseShop, change the version number to 4.0.0 and re-release it, and everything should start working again.

Am I barking up a bad tree, or does this have some benefits?

Posted by Martijn Faassen at Thu Sep 27 2007 13:17

Lennart, I think there are two related but slightly different issues:

1) we need to take care when releasing a package, making sure to follow various guidelines to make sure that a new release is good and won't break things.

2) we need to make sure that if I use package x.y.z in my software (framework), when you install it, you use exactly that x.y.z too.

Point 1) helps everybody who develops software and wants to upgrade to a newer version for whatever reason. It increases predictability. It's good to follow some guidelines and pay attention. Mistakes *will* still be made however, as that's inevitable. Even non-mistakes ("this was just a bugfix!") might for all kinds of interesting reasons cause trouble for packages that rely on it ("our workaround broke and thus our package broke!").

That's why we have requirement 2): we want to make sure that no matter what happens, it doesn't affect any existing releases in any way. You can use my software today, and in 10 years time.