Chris McDonough just posted a capsule criticism of the Zope project and culture to zope-dev in a discussion I started. I believe Chris and I have been "violently" agreeing on most many issues in this discussion... I thought this characterization is quite interesting and I'd like to share it with the wider world. I agree with it so much and disagree so much at the same time. Even though I disagreed with the decision to include underwear as a logo on a (now rejected) design for a new zope.org homepage, I do think it's good to sometimes focus on our dirty laundry as it can help with a cultural renewal I think the Zope community needs and is ready for. I think this information can also be interesting for developers of other web frameworks. Look at the stuff we deal with after having been around so long! Don't let this post mislead you: I see a lot of value in Zope technology and its community otherwise I wouldn't have stuck with it for more than 10 years. Chris obviously sees value in it too, otherwise he wouldn't be arguing with me on zope-dev and extracting so many of its concepts into Repoze components. I hope he'll forgive me for quoting him here: As a background, Chris is characterizing the Zope culture in such an extreme way as it contrasts with the Repoze project he leads that tries to these things differently. The Grok project shares quite a few of these values as well, and wants to get closer. I agree with much of this in the sense that it's a useful caricature of what's wrong with the Zope community and what needs to be improved. It's amusing that I look for ways to apply the lessons of Repoze and Grok (among others) to Zope we end up arguing so much. Chris interpreted my proposals for cultural renewal as a way to codify all things bad he describes, while my intent was more or less the opposite. I have to improve the way I express myself! Let's go into the details: 9 years is a long time, and while I agree that some cultural deficiencies such as bad presentation of modern Zope technologies to developers and a bad installation story, have lasted a very long time without enough of an effort to fix them, other deficiencies we're aware of and we're making progress on. This characterizes the current Zope 3 codebase quite well, but at the same time doesn't characterize our goals or efforts. We had an effort in 2007 to split up our large interconnected codebase into small components. These are now the many zope.* and related areas on PyPI. Some of these are reusable independently,
but far too many still pull in way too many dependencies (basically all the others), due to some seriously circular dependencies between them. Recently I and others have spent quite a bit of time in trying to make these dependencies better, and this is part of an on-going process. I think this is a mischaracterization therefore in the sense that is something the community knows about and is actively working on. I agree that we have erred on the side of too much backwards compatibility. That increased the overhead of changes tremendously and blocked innovation. That said, I also see a lot of value of having a lot of components that can work together, and we do have a large collection of those in the Zope ecosystem. This is why Grok is so careful to stay compatible with Zope 3, so we can share that pool of components. I'm in favor of an evolutionary approach where backwards compatibility on occasion is broken and it's clearly documented what developers should do to fix their code. We've found this worked quite well for changes in Grok. I'm also in favor of an approach where due to proper dependency factoring we can dump whole chunks of code that we aren't using (such as the Zope 3 ZMI) in a large step. I agree that most package-level documentation could be improved tremendously by focusing on writing real documentation instead of half-test/half-documentation stuff. That said, we also have a tremendous level of package-level documentation and interface documentation, and it's a mischaracterization of the values of the Zope project to say we haven't cared about documentation at all. We innovated with interface-level documentation and doctests, as well as making those doctests available on PyPI. Chris has said in the past that this is a sort of "false optimum" that stops people from really fixing documentation issues, and I agree with him there that this is wrong (even though I do value doctests). We should make an effort to change our culture and redirect our documentation efforts to go beyond doctests. We've seen the adoption of Sphinx in our community in the last year, and I have good hopes we will make a lot of progress on this in the coming year. I'll also note that documentation for the whole system has traditionally been lacking (how to get started, install it?). For this my answer is Grok. If you want to use the Zope 3 technology stack, it's by far the easiest way to get started. I'm not sure whether this characterization is accurate or not. Because Zope was there sooner than many other Python web frameworks, it's probably true we've not studied the competition as much as if we'd been there later and had more chance to compare and contrast. I've personally been quite interested in seeing how the cultures surrounding other web frameworks work and trying to adopt lessons from this (DRY and convention over configuration for Grok, for instance, and proper documentation). I've been able to apply the lessons I've learned from other web frameworks far better in the context of Grok than I have been in the context of the wider Zope community, and I wish that would change. In fact I'm trying to apply some lessons I've observed from Chris' efforts, Repoze, to Zope. So, we should do more of this, indeed. It's hard to judge this one, because what is "real cooperation"? We've tried reaching out quite a few times over the history of Zope, but I do think we can do better. Of course reaching out like this is one of the main things that Repoze is trying to do, so it's unlikely we'll be able to get up to the level of the Repoze folks any time soon. The culture of cooperation between other Python web frameworks has started really taking off surrounding WSGI. Zope has tried to integrate with WSGI (and Twisted before then), but with moderate success in gaining community benefits from this. This means we need to try harder. I believe Grok's upcoming 1.0 release will help us a lot with the adoption of this technology, as we've been working quite hard to make sure it works out of the box. I think we should do our best to integrate other technology in our own stuff, and we've had some progress with things like WSGI, Twisted and SQLAlchemy. Maybe Repoze is next, but I hear they think very badly of us indeed, probably because they know us so well. :) I share that concern very much. Part of the reason we don't attract more users is our lack of attention to documentation, proper web presentation, and our "here's a giant toolbox, it's flexible, you figure it out" approach. Grok has been one answer to this, and we've had a lot of good progress in bringing the old Zope 2 documentation up to speed as well. It's good that the Zope technology is so central to other projects which do attract new users (Grok, Zope 2, especially through Plone) so we still have an influx of new users that way. We also get an influx of users of our individual libraries such as zope.interface and zope.component, and we want to encourage that happening more. Besides this, I'm always surprised we are able to attract new users of the Zope 3 app server directly itself as well - there are more than you'd think given our rather bad public presentation. There must be some value in it after all then! :) I think we should recognize the position of the Zope technology as central to Zope web frameworks that do attract users. I want to call that technology the "Zope Framework". It's not something users install directly, but it's something that is used to build Grok, Zope 3 and Zope 2, that can be installed. We need to manage the Zope Framework as such. In it, there is a tension between the concerns of the individual libraries (the parts) and the whole (the integrated set of them that's traditionally been called Zope 3, but is also used by Zope 2 and Grok). We need to think of the Zope Framework as a whole, and as parts. In order to make whole better we need to improve the parts, but in order to improve the parts we often need to think about how they fit into the whole as well. We need to manage it so that we can resolve this tension so that we can have both good individual libraries and a better integrated experience. I'm optimistic we can resolve this tension to the betterment of the whole and the parts. We need to look at ways to make the Zope Framework smaller, composed of more easily digestible parts, and being a whole that's easier to comprehend. In reality we're not managing one big thing, but a tree of libraries that depend on each other, and people can approach parts of the trees as well as the whole. Breaking the tree metaphor, branches or nodes of the tree can be adopted into other trees such as repoze.bfg and Twisted. The Zope Framework, like Chris' description, is in a way a caricature of something more complex. It's a handy concept to organize a community around. That community is the Zope community. Here's our dirty laundry. We're washing it so you can use it too. And we'll need to wash it again in the future. We're used to doing laundry. We've been at it for over a decade, and we won't be going away any time soon. Care to join us? :)
![[Comments]](/blog-content//themes/Default/comments.png "8 comments")
(8) Tue Mar 03 2009 23:12 Zope Criticisms:
I have no faith whatsoever that staying on the course we've been on
for the last 9 years (large interconnected codebase, backwards
compatibility at all costs, lack of any consumable documentation at
a package level, not much curiosity about how other Python web
frameworks work, not much real cooperation with folks
that maintain other Python web frameworks, a constitutional
inability to attract new users) will bring any sort of positive
change.
I have no faith whatsoever that staying on the course we've been on
for the last 9 years
large interconnected codebase,
backwards compatibility at all costs,
lack of any consumable documentation at a package level,
not much curiosity about how other Python web frameworks work,
not much real cooperation with folks that maintain other Python web
frameworks,
a constitutional inability to attract new users
- Comments:
Posted by whit at Wed Mar 04 2009 16:08
I just don't know. I get the feeling repoze has lapped grok and zope3, maybe several times. Grok is better than zope2, plone, zope3, etc, but it's still mostly a silo that eventually leads to learning zope3 innards that are only applicable to zope3. There is alot of beauty in there, but you have to be wicked smart to reuse it elsewhere.LOC is the enemy here: too much code to document clearly, wrap your head around etc. Zope is a tragedy in the commons of attention. It is too big and too complicated and too interconnected even after rewriting as a component architecture to be able to easily know it and know the rest of the world. there are simply too many concepts for mere mortals to pay attention to both.I don't do zope anymore even though I could get paid for it if I wanted. I don't miss it.though I miss the people, the code hurt me. many of decisions represented by the code hurt me. and to be honest, the inertia and groupthink of zope programmers as a whole hurt me and made it unpleasant or unsatisfactory to do the good work that you and chris have done reaching out to others. It's worth noting the caliber of person it takes to bridge these worlds. I had neither the brains nor the patience.repoze is simply the right idea. decompose the canon of learning that is zope into bite size pieces people can digest one by one and reject the bad bites and let evolution run it's course. make the silo be web + python. let zope die and let the good ideas live on. grok could take it's "the code is the configuration" idea and apply it to repoze and instantly be a system that scales down to simple concerns rather than being a RAD lightweight framework hitched to white dwarf of a library.In the end, developer reach the point where they usually remove all the framework bits in a search for optimization. A framework that makes this hard is courting hatred. code must scale from 0 to 100 and back down to 0 again. I think repoze can do this, zope3 can only if you have a phd, and forget it with zope2 or anything that runs on it unless you are Chris and Tres.just my .02.-w
Posted by Peter Bengtsson at Wed Mar 04 2009 17:17
> Part of the reason we don't attract more users is our lack of attention to documentation, proper web presentation, and our "here's a giant toolbox, it's flexible, you figure it out" approach. I very much agree with the last point. I've done plenty of Grok AND plenty of Django and my feeling is that with Grok you spend half your time understanding eggs, buildout, virtualenv and grokproject before you can spend the 5 minutes it takes to write your Hello World app. Django feels much "flatter", simpler and (dare I?) stupider but at least it gets right to the thing I'm there to do: write application code. Beginners care about "componentization" but care more about getting things done and writing applications that actually does something. The best is the enemy of the good. Before you bash me, getting things done is not for everyone. But let's face it, Zope is severely suffering from lack of new users.
Posted by Martijn Faassen at Wed Mar 04 2009 18:23
Whit, I see a lot of value in the huge pile of components that *does* exist for Zope that I'd need to rewrite for Repoze. And a lot of value in the community of people that are thinking them up. Finally, I have codebases to maintain.Grok could decide to radically break with the past and give all that up and port Grok onto something else. But at this point I'm not willing to do that as I don't want to give up that value. That means I'm trying to change the Zope community and technology so it can be *more* like Repoze, with simpler libraries and clearer dependency relationships *and* the stuff that's already there.I want to break up *Zope* into bite-sized pieces that can be digested one by one and reject the bad bits and let evolution lead its course. Zope is a platform that has offered some evolutionary path forward for the last decade, and I am betting it can do it another decade and come out for the better.I believe that this effort has a chance of success, but if I decide it doesn't I'm going to consider alternatives. I don't want to be wasting my time.Peter, we're doing something wrong if you have to spend half your time understanding eggs, buildout, virtualenv and grokproject to get "hello world" code Grok going. And I really believe that using grokproject isn't that hard and that you're exaggerating a lot, sorry.We do need to solve the perennial "the platform installed incompatible versions of things Grok used that are clogging up the works" problem once and for all. Right now that's where virtualenv comes in making things far more complex than it should be. I don't use virtualenv with Grok at all, myself.Last week I was at a Dutch gathering where various Django people were talking about how they adopted buildout for their Django projects by the way!I'll also note that in my experience Grok isn't suffering severely from a lack of new users, though we could always use more.
Posted by Kevin Teague at Wed Mar 04 2009 23:15
I agree with Whit, "repoze is simply the right idea. decompose the canon of learning that is zope into bite size pieces people can digest one by one and reject the bad bites and let evolution run it's course. make the silo be web + python. let zope die and let the good ideas live on."That said, I'm still very happy using Grok as a framework. But any radical refactorings or efforts to push it to a *much* smaller core I'd be very happy with. As well as any work towards widening the silo to be "web + python" and including more packages from outside the Zope world (e.g. using WebOb for request/response for example). Of course, there is already lots of work being done in this area (WSGI, dependency clean-ups) - but as more non-Zope stuff plays a more central roles in a Zope-ish web app, is it fair or accurate to call this a "Zope Framework"? Does calling something the "Zope Framework" preclude or bias against using or integrating with packages such as Paste, WebOb, PasteScript, Sphinx or SQL Alchemy?The application level code in Grok is very much a mixed bag. Especially as someone who's arrived later to the Zope 3 stack and doesn't have any investment in the creation of these parts - pretty much anything that depends upon ZODB for persistence (e.g. zope.app.* mostly) - a lot of times I feel like it's just easier to ignore that stuff and pull in parts from the wider Python web world or just roll-your-own for the simpler use cases.For example, I wanted to grant some simple permissions based on LDAP data in a Grok app recently. The zope.securitypolicy package requires annotations to be persistently stored in the ZODB. It seems like it might be possible to use this without a ZODB footprint, but doing so felt like it was in the category of "wizards". I spent quite a bit of time headscratching and trying to make it work, but finally gave up and just brute force copied the LDAP data settings into the ZODB. It took quite a bit of time to arrive at a working solution, and the end code is hacky and abusive :(I maintain Plone apps and Grok apps, so I can see the arguement towards using a common Zope Framework between the different Zope sub-communities. In fact this was part of the original appeal of Grok for me. But then I also train people and help get them up to speed on developing web applications, and it's painful having to teach new people developing new web applications about APIs that are overly complex or oversized due to a reluctance to break backwards compatability or to ditch cruft. BFG has a very appealing approach of having a *very* small core for the framework.I disagree with Peter on the Grok installation experience and the statement, "it gets right to the thing I'm there to do: write application code." I'd argue that the end reward of web applications is to not just write the code, but to deploy the code. Yes, it's easier at the start to just dodge this question by teaching people bad habits such as "sudo python setup.py install". But these bad habits become ingrained in people, and create resistance to making improvements to the Python web app deployment experience. As a sysadmin, I'd much rather deploy a Grok app than a Django app. As a developer on a team project, I'd much rather collaborate on a buildout-based project than struggle with "Works on My Machine(tm)" issues.
Posted by Martijn Faassen at Wed Mar 04 2009 23:21
Kevin, the whole idea behind naming the Zope Framework that is to make better governance and evolution of it possible. Before that naming we had a nebulous Zope 3 entity which was a framework and application both.Concerning granting permissions based on external data in Grok, it's actually quite possible by implementing your own IPrincipalMap adapter that implements the rule you want. Like this (let's see whether my blog mangles the code):class MyMap(grok.Adapter):
grok.context(Something)
grok.provides(IPrincipalPermissionMap) def getSetting(self, permission_id, principal_id, default=Unset):
if i_want_to_allow_this:
return Allow
return UnsetThere are various other methods you'd officially need to implement to implement IPrincipalPermissionMap, but this will work. I agree that this needs to be documented.Posted by Martijn Faassen at Wed Mar 04 2009 23:23
Oh, and naming it the Zope Framework means it's just the bits of any larger framework like Grok that the Zope developers take care of. It might very well build on other libraries, and Grok might of course too.Whether this *will* happen depends on whether volunteers do the work. I want there an infrastructure in place so that volunteers can feel safe to jump in and do work.
Posted by Jose Dinuncio at Fri Mar 06 2009 21:39
I've been in a long love-hate relationship with zope. I loved zope 2 and what I could do with it, as long as I don't have to extend it. Then came Zope 3 and I tried it each year or so. A wonderful piece of software. The infamous zope2 products were dead and you could extend it by mere python packages....But the entry cost is too high for me. Make a blog? Well, easy: I define the interfaces, the components, maybe via schemas, then retouch the templates... don't forget to put in place an authentication system, remake the whole look'n'feel and then... wait, where do I download wordpress? Each step is easy by itself, but there are so many of them and I need instant gratification, I'm a sysadmin after all.repoze.bfg has the beauty of simplicity, but maybe a big site ends needing half of zope3 anyway. Anyway it has the not-instant-gratification problem too.{deep breath} I miss zope2's Through The Web development. Here, I finally said. I have this dream, on which I see a Zope 3 with all its current interfaces and components and its actual way of development, but with the zope2's ZMI. I download it, install it and connect to http://localhost:8080/manage, edit the index.html object and voila! my new intranet has born. I visit manage/acluser and now I have users. I develop a crude TTW temporary version of a blog with the promise of replace it with a python package "as soon as possible". In my dream my intranet is built with lego parts (python packages) I downloaded and (lightly) configured. The work-flow of the site was configured TTW, the same thing with portlets. And it is not plone. It's just zope interacting with the zillions of zope packages on the net.
And after years of happiness I finally say to myself "well, now is the time to take that bunch of TTW python scripts and replace them with my own zope-package-based blog"Maybe I'm not the target of the web development platforms, maybe my love to zope is not corresponded. But that's the problem with love, I see all its potential and just keep saying "if only..." Or maybe I'm looking in the wrong place... Maybe TurboGears, or Pylons or... but no. Once you've used zope, once you have studied it, nothing more satisfies you anymore. Maybe if I keep searching... Maybe in Utah.Posted by Martijn Faassen at Fri Mar 06 2009 22:13
Jose: Grok is Zope 3 but without the culture to do everything up front. It's not perfect and some things are more complex than they should be, but it does have a much easier entry curve than pure Zope 3. Better online documentation too.http://grok.zope.orgConcerning TTW, I said something about it before:http://faassen.n--tree.net/blog/view/weblog/2008/09/19/0We'll see whether TTW development ever comes back, but it's at least something I think about.
