(20) Mon Sep 10 2007 15:24 Well-kept secrets of Zope:

Zope is a web framework that comes equipped with powerful, apparently secret, features. Some of the things Zope has had for literally years other web frameworks are only evolving today. And in other cases, Zope comes equipped with features that other web framework communities are currently only dreaming about.

If Zope has such powerful features, why are they a secret? I think it's a combination of an unfortunate (but partially well-deserved) reputation Zope has in large parts of the Python community, and Zope isolating itself in its own community (it managed to build a large community many years ago).

Before I start off yet another discussion with people burned by Zope 2 in the past: Zope 3 is not Zope 2. It's not crufty. It is hard to approach, but we're fixing this with Grok, which cuts down on the complexity hard.

Now let's go into some features that Zope has and that we seem to be keeping a secret very carefully from the rest of the world.

Comments:

Posted by Ian Bicking at Mon Sep 10 2007 20:49

Note that FormEncode is nearly as old, from around 2004 (with several systems preceding its design), and it serves as the basis of validation in TurboGears and Pylons. It has included htmlfill since that time, which IMHO is a superior technique for handling forms -- stable, scalable, and far more maintainable than anything I've seen based on widgets. So while the widget systems like ToscaWidgets are newer, there's also been a very mature and stable system available and in use.

Posted by Steve Holden at Mon Sep 10 2007 21:16

Martijn, the essence of hte problem is right there in your writing. Zope (3) people aren't good communicators (yourself excepted), they are of the opinion that "if you buildut, they will come" - this has never been true, and it's getting less true by the day.

When I first came to Python in the 1.5 days Zope had great visibility in the Python world, and was trumpeted as *the* significant large Python applications. I agree, you *should* be shouting Zope's benefits from the rooftops, then people like me might start to investigate it seriously. I already know that some great developers have been involved.

The fact remains that searching for information about Zope can be a frustrating experience. It appears that many people have left outdated and inaccurate stuff around on the web, much of it from the early Zope days, and in the absence of any authoritative guide the clutter makes search results frustrating at best and useless at worst.

It's somewhat ironic that the web site for Zope, a system designed to make it easy to keep web content up-to-date, only lists fifteen articles from 1999 and a single article from 2003 (http://www.zope.org/Resources/Articles/). This general failure to "eat its own dog food" leaves Zope with an unconvincing story. If it's so damned good, why does it have such an uninspiring web presence?

Posted by Martijn Faassen at Mon Sep 10 2007 22:02

Steve, I agree fully that we haven't been communicating very well as a community. It's what I was ironically referring to in my article - it's like we keep our features a secret, and that's stupid. Thank you for saying I'm an exception, but there are certainly many other exceptions besides me in the Zope community.

I agree Zope has an atrocious web presence. I've been calling attention to it and trying to do something about it for a while now. It's hard to fix, as there is so much to tell nobody knows where to even start. It seems to be extremely hard to get anybody to actually write content. The job seems overly intimidating. It's mostly just a job of going in and doing it, but it's understandable people aren't exactly lining up to volunteer for this. Note that Plone is actually doing quite well in this respect, so it doesn't have to be that way.

Note also that I think we're making reasonable progress with grok (http://grok.zope.org), though we have a lot more work to do.

Posted by Mikkel Høgh at Mon Sep 10 2007 22:37

So there you have it - black on white. Zope is, like, ancient. Dinosaur. Ready for replacement ;)

Posted by Martijn Faassen at Mon Sep 10 2007 22:40

Zope is a giant dinosaur stomping through your neighborhood kicking ass. :)

Posted by Martin Aspeli at Tue Sep 11 2007 00:38

Zope is a prime example of one of the fundamental tenets of Open Source: Community first, software second.

Zope has a terrible web presence because the Zope community doesn't include (enough) people who (a) care to (b) can and (c) have the time do something about it. Lots of people use Zope 3, but for various historical and social reasons, these people have no time to fix the atrocious web story.

A lot of the Zope (2 and 3) users are also in the Plone community, and care more directly about Plone. Plone has a better web presence, of course, and a larger community that embraces a wider audience. Sometimes, that "wider audience" is more likely to produce people who are good at creating an attractive external presence.

If I had infite time, I'd do some Zope charity. I'd ask for carte blanche, nuke zope.org to oblivion and create a nice (if small) web presence there. But I don't, and it's so far down my list of priorities relative to its sheer size that I probably never will. Sorry. :-)

This is of course i no way a reflection on the quality of the Zope community, which includes some of the most talented framework designers I've had the pleasure to meet. I love Zope 3. I miss Zope 3 when I'm working with non-Zope frameworks. Zope 3 blows my mind (sometimes in the wrong way), but it's definitely a framework aimed at the mid-to-upper end of the framework market.

Grok holds the promise of letting us hit the entry level and still let people draw on the Zope 3 goldmine when they get beyond the basics. But Grok is a small project that needs more people. So thank you, Martijn, for shouting. :)

Posted by Ian Bicking at Tue Sep 11 2007 01:02

I think the web presence should be fixed with documentation generated from source, with some manual indexes, and tutorials that do lots of deep linking into that generated documentation.

You have the docstrings already, and things like interfaces really increase what generated documentation can express. Though not trivial, it seems by far the easiest way to resolve the documentation issue that Zope has. And gives you a website that isn't awesome, but is useful, and at least won't look abandoned. And it's even a programming task, which is the most likely task to work for the Zope Community ;)

Posted by Martin Aspeli at Tue Sep 11 2007 10:53

Ian: I think that's an excellent idea, but it's only 50% of the solution. The otehr 50% is the general evangelising and presentation. zope.org does not feel approachable and friendly, and doesn't highlight the important features of Zope 3 like Martijn has done here.

Posted by makkalot at Tue Sep 11 2007 11:34

Fistly i used zope2 in a web gui project which was a failure because i was confused from where to start. Some people said to start with zope3 it is newer an better. An i decided ok zope 3 is cool. I purchased a book for zope 3 but it arrived after 2 months to my country :) And project failed, and began todo it with zope 2 it is still under development:)
Anyway we dont have a good start with zope but in my free time i digg in to zope 3 source code an try to learn something. It is now just a hobby for me :) I think the source code is the best documentation and for zope 3 it is the most updated one :)

Posted by Michael Watkins at Tue Sep 11 2007 16:55

I keep thinking that I should be a target "user" of Zope 3, and by that I mean a developer who would use the Zope 3 API and services. After all, I have bought into some concepts familiar to Zopistas:

- an object database (I use Durus but ZODB would be perfectly familiar to me)
- object publishing (for many years now I've been using either Quixote or, over the last couple, its newer cousin QP, both of which share a form solution that has been around for quite a few years too, by the way)

And professionally I come from the enterprise document and workflow management space. I know what's involved in marketing, selling, building and implementing significant DM/WM/ECM solutions. Naturally the promise of Zope has some appeal to someone like me.

Every six months or so something I read on one of the Zope or Plone feeds twigs me into downloading the current state of the art and I spend a day or so plugging through the code. While "grok" is an interesting approach to allow someone to test the waters, it still doesn't fully hide the enormity of Zope and with my technology evaluation hat on, I am left with nagging thoughts about how much real time and effort it takes to get someone who isn't already Zope aware up the learning curve.

As others have noted already, the lack of readily available, completely current, documentation and really useful info -- is a real, not imagined, issue.

We should see examples, working code, step by step tutorials that actually get updated over time in step with the seemingly constant refactoring of Zope itself, buildout howtos, etc -- ALL IN ONE PLACE, easily identified and lovingly maintained... why not use a content management system ;-) ... and get rid of all the out of date cruft that inevitably lands in our Google search results.

Zope is a big system and there is not - to my knowledge - any hitchhikers guide to it. With relatively small frameworks like Quixote/QP or even Pylons and Django I can get a decent understanding of the system just by wading through the code. That's not nearly as possible with Zope. Some hand-holding is required. Whenever I go looking for resources invariably I turn up more outdated info than useful material.

After realizing yet again that I'll have too many questions to ask, and no authoritative resource which I can *easily* retrieve my own solutions from, I find it impossible to commit myself to "diving in".

Ultimately while I remain somewhat interested in actually understanding Zope 3, I soon will do what I always do - nuke whatever I've downloaded (eggs, grok, etc) and will try again in another six months or so and see if the situation has changed.

Each time I go through this process I end up scratching my head wondering whether Zope is too clever by half, or I'm too stupid by double.

While I can appreciate that its a big framework capable of aiming for big things, my experience in working with big clients suggests to me that if I can't fully "grok" it in a few days, it'll be a hard sell to some of my clients - the subset that "say" they want to inherit the on-going support and management of a finished solution. Frequently that's an evaluation criteria for new technology, even if reality says most IT shops won't; and many shouldn't unless the technology really is very core to their operations.

Posted by Martijn Faassen at Tue Sep 11 2007 17:07

Michael - "We should see examples, working code, step by step tutorials that actually get updated over time in step with the seemingly constant refactoring of Zope itself" - have you seen http://grok.zope.org/tutorial.html ? It's a step by step tutorial that should be working. Lots of pages too, though still incomplete. We're working on more.

Anyway the point is well taken and understood. We do need to improve the documentation. I'm doing my bit.

Note by the way that Grok is not aiming to *hide* the "enormity of Zope". It's trying to make it easier to approach.

Posted by Martin Aspeli at Tue Sep 11 2007 17:54

Michael - "Zope is a big system and there is not - to my knowledge - any hitchhikers guide to it" - Philipp's book (http://worldcookery.com) is a very good guide to Zope 3.

Posted by Michael Watkins at Tue Sep 11 2007 17:58

I did spend some time with the Grok tutorial and thought that was well done indeed, and nary a line of ZCML had to be written :-)

I suspect some fine tuning of even the grok docs needs to be done (or I need to read them closer), to make the process bullet proof to someone newer to Python than I am. Dealing with easy_install (shouldn't installing easy_install for you be something that buildout does?), buildout (my first experience with it), what does the grok buildout do to a system if Zope is already installed (maybe I don't care), a different version of Python (2.4.x) than my system's default (these days current 2.5.x) -- all these things are stumbling blocks for someone new to the Python family.

On the other hand, the process, with a little intuition on my side, worked well and I had some time playing with grok and got enough out of it that it may not suffer rm -rf grok yet.

Your comment that grok isn't intended to "hide the enormity of Zope" but make it more accessible is an interesting one; perhaps you could explain (or point me to an explanation) what the roadmap for grok is and what would cause one to choose the native Zope API or the grokked version? Would someone heading down the grok path eventually stumble across barriers causing them to have to rethink and re-implement with the Zope API directly?

In the meantime I'll continue to watch for changes to the grok tutorial; am interested in seeing more of the XXX's flushed out. Perhaps the grok subsite should have its own RSS feed...

Posted by Martijn Faassen at Tue Sep 11 2007 18:12

In brief: the grok project aims to do a number of things:

* replace Zope's configuration mechanism (ZCML) with a different one which groks the existing Python code you need to write anyway. This isn't about hiding Zope 3, just providing the same abilities in a different way.

* look at typical patterns in Zope 3 code and try to provide for the common case, making it easier. You could see this as 'hiding'.

Since Grok is mostly Zope 3 with an alternate configuration layer, there aren't many reasons in my mind for someone to want to step away from Grok and suddenly implement everything with ZCML. I don't see barriers. The APIs are the same in Zope 3 as in Grok, though Grok makes a few convenience-imports - grok doesn't 'layer' over Zope 3 so much as it provides an alternate, hopefully more convenient, configuration of the existing Zope 3 infrastructure.

Posted by Michael Watkins at Tue Sep 11 2007 18:37

After reading more of the sources and grok site I'd come to understand that providing an alternative to ZCML was one of the principal features of grok. I do think its appealing, compared to the alternative - ZCML seems awfully verbose to an outsider peering in. Having an alternative which looks more naturally like Python is a plus.

Since Grok is mostly Zope 3 with an alternate configuration layer, there aren't many reasons in my mind for someone to want to step away from Grok and suddenly implement everything with ZCML.

Thanks - that's the sort of judgment that someone experienced with Zope can make that is useful to someone like me peering over the fence.

Posted by Ignas at Tue Sep 11 2007 21:31

Two possible answers ;)

1. Zope 3 is our secret weapon. Why should I advertise Zope? I don't need anyone to use Zope3, and there are a lot of very smart developers making Zope3 better for my own applications every single day! (Jim Fulton, Martijn Faassen, Stephan Richter to name a few.) Most of Zope3 contributors are using Zope3 to make money ;) There aren't that many programmers working on open source Zope3 projects for fun, are there? ;)

2. The task to create a simple Zope3 application is still difficult enough to be a test for new hires. There was only 1 guy who managed to not just create an application, but to create a custom skin (as in - the app was not ZMI driven) out of like 6 guys who succeeded in completing the task. (none of them managed to write a proper Zope3 application that would not be using outdated API, or doing things in non-Zopey ways, or using Zope3 features I didn't know existed) and the application needs like 4 content objects at most. And it takes 2-3 days of pure work to finish the task (install Zope3, learn Zope3 and write the small application).

I don't even know what is the standard way to install an up to date Zope3 these days :) so I had to *invent* one. Choices I know were: a trunk checkout, zopeproject, buildout, and apt-get install. And I could drone for hours about their downsides ;)

Posted by Martijn Faassen at Tue Sep 11 2007 22:36

I know you're half tongue in cheek there, Ignas, but both those answers suck:

1. Zope 3 should not be secret. Zope 3 may be good when "secret", but it would definitely be even better with a larger community, *especially* if we use it for commercial projects.

2. You'd better devise another test for new hires. It should be *easy* to get started with Zope 3. Not just because new hires would have an easier time, but because *I* don't want to remember a lot of stuff just to get "hello world" in my web browser. Anyway, Grok has fixed this issue already.

Zope 3 may be in flux on how to install things, but Grok is not. Grok is about eliminating unnecessary choice and unclarity (while leaving the underlying flexibility intact). Everything has downsides, but that shouldn't stop us from making choices. The way to install Grok is to use grokproject, which is documented.

Posted by Ignas at Tue Sep 11 2007 23:43

I saw the way you should install grok just that I very very very dislike any tutorial that has:

1. Have root account on the system.
2. Install stuff that you can't uninstall into your system.

as it's first steps. Though it seems that most developers don't have any problems with that as most of python project tutorials have these 2 as first steps to get going.

So this personal flaw of mine is only causing a lot of frustration and work (to make schooltool buildout installable without root executing 1 maybe 2 shell commands) for me.

Posted by Martijn Faassen at Wed Sep 12 2007 01:12

You forget to mention that you only need to install grokproject this way (which I believe works in a workingenv, so you'd need no root). grok projects are installed anywhere you like as non-root.

We did it this way as using easy_install is the easiest way to get something going on someone's system. We could provide a .tgz I guess that you'd run buildout inside. Contributions are welcome. :)

Posted by Ignas at Wed Sep 12 2007 14:41

Hmm,
http://ignas.pov.lt/grokproject-buildout.tar.gz and http://ignas.pov.lt/grokproject-easyinstall.tar.gz

two takes on the problem. Both will work even on ubuntu with a broken version of setuptools and a conflicting version of pytz installed. Neither requires root. Downside - buildouts created with grokproject will share the same virtual python installation though, but i didn't have time to work around that yet. On the upside - buildouts will be more or less isolated from system python, so they will work even if the person installing grok has Zope3 installed in his system.

To try out - download, extract, make and run bin/grokproject to create a project