Python Secret Weblog for 2009 January 29 (entry 0)

(6) Thu Jan 29 2009 21:18 Cleaning up Zope 3's dependencies:

This week a bunch of us (myself, Christian Theune, Wolfgang Schnerring, Brandon Rhodes, Jan-Wijbrand Kolman and Sylvain Viollon) have been sprinting in my house at the "Grok Cave sprint". We've been working on cleaning up Zope 3's dependency structure, which in places is very hairy. This meant that you could often pull in one fairly innocent looking Zope 3 package and as a result pull in almost all of them. This makes it difficult to reuse packages and upgrade code. Loosely coupled code and all that.

See also Mark Ramm's talk at the Django Conference about why having a sane physical dependency structure between packages is a good thing.

One major part of this dependency reduction project has involved extracting a new package zope.container out of zope.app.container . Another part involved reversing the dependency relationship between zope.traversing and zope.location [UPDATE: I mistyped zope.security instead of zope.traversing here previously]. We've also started extracting zope.site from zope.app.component and moved some code from zope.app.security and zope.app.component into zope.security. In addition we created two new tools (z3c.recipe.compattest and an addition to the Zope test runner) to help us keep track of things. We've also made a lot of use of an existing tool to track dependencies between packages called tl.eggdeps.

That's all gobbledygook to most people. So here are the before and the after pictures (of the zope.container and zope.location/zope.traversing work in particular).

Here is the before, the dependency graph of zope.app.container (with core packages zope.interface and setuptools excluded as they don't depend on anything and clutter up the graph):

And here is the after, zope.container, which can be used instead of zope.app.container almost everywhere (and zope.app.container in fact uses it too), again with zope.interface and setuptools not shown:

We believe this is significant progress! It's still a lot of packages of course, but we can at least motivate the existence of a dependency relationship on them in most cases.

All this work is far from done. It's been a lot of work to get this far. There are still many dependencies more to clean up. It will take more work after the sprint to get to a good dependency structure for the complete set of Zope 3 libraries. We're starting to see some light at the end of the tunnel now though.

Comments:

Posted by whit at Thu Jan 29 2009 22:05

Nice work yall!

Any plans to decouple ZODB too? I know kapil has a working container implementation that is rdb backed.

Posted by Martijn Faassen at Thu Jan 29 2009 22:12

whit: I've got one of those too. Actually SQLAlchemy has got most of it already and megrok.rdb simply reuses that.

To make it independent of zope.container you'd need to extract the interfaces from zope.container into a new package (IContainer and friends), and perhaps a few adapters that only rely on those interfacees. Not a high priority for me but someone could do that.

The nice thing isn't just zope.container's simplification though; we also simplified the dependency structure of the graph below it, and much of that stuff is used throughout Zope 3.

Posted by Stephan Richter at Thu Jan 29 2009 22:55

Looks good guys. I will soon start the Zope 3.5 series and we will put the new versions of the packages through a test.

Posted by Jeff Shell at Fri Jan 30 2009 10:59

I've noticed some of the benefits of this work, just using the 3.4 era-ish packages that were updated this week. I have a small utility that I use to spit out Working Set info and ran it on a site before and after updating our internally-managed 'versions.cfg' and saw a couple of long-deprecated packages like 'zope.app.session' disappear from the working set completely. Nice!

Posted by Vicent at Sat Jan 31 2009 01:47

Hello.

Could you tell me wich software did you use in order to draw those "Graphs" or charts??

Thank you in advance!

Posted by Martijn Faassen at Sat Jan 31 2009 11:03

Jeff, in fairness, I think you're seeing the benefits of earlier work in this department, not our work this week. It's been a long and slow process (though I believe this week has managed to speed it up).

Vincent, we're using 'graphviz' and in particular the 'dot' tool. We also used 'tl.eggdeps' to generate the input for the charts.