< megrok.rdf finally checked in
Grok's release mill >

[Comments] (16) Fri Sep 19 2008 17:35 What Zope can learn from Zope:

So we've had what Django can learn from Zope, what Zope can learn from Mozilla, and a few years ago already we had what Zope can learn from Ruby on Rails. (and www.zope.org still sucks, but grok.zope.org and repoze.org don't)

Since the Zope of almost a decade ago was mentioned by a TurboGears guy in a negative light to a Django audience a few of course took this opportunity to bash Zope, as the word Zope is like a red flag. Zope has accumulated long, rich history of good and bad ideas, so we must present a tempting target.

Zope can learn from Django, and is of course doing so. We've been around for a decade and you aren't around for that long if you don't learn and adapt.

Let's now talk about what Zope can learn from Zope itself. How can that be?

The Past from the Future

The obvious way for Zope to learn from Zope is for the past to learn from the future. We've presumably learned some things over time, after all. What could the old Zope 1 and Zope 2 hackers have learned from modern web development with Zope 3, Repoze, and Grok?

Filesystem development

Zope 2 has a user interface in which software development can take place. Limited, untrusted Python code is stored in the object database. This has two major drawbacks:

  • it's Python, but Python with a few twists: untrusted so normal imports and operations are not always possible, and implicit acquisition pulling things in everywhere. This sucks, we want to use Python the way it is designed to be.
  • you can't use normal file system tools to manage your code. Instead you have to use the web browser UI. That still isn't ideal even with modern AJAX goodness, and certainly wasn't ideal in 1999. It means reimplementing all the tools that already exists (editors, version control systems, searching, etc) on top of Zope. This was an attractive effort that many in the Zope community undertook, and most of that work is now lost.

Zope 2 development through the filesystem was also possible, but it has a completely different development model than the through the web development, and this accounts for part of the infamous Z-shaped learning curve of Zope 2.

Zope 3, and Grok, have filesystem-based development only. You just write normal Python code in normal Python modules which reside in normal Python packages. You use your favorite editor, and your favorite version control system.

Fine-grained components

Zope 2 introduced a powerful component programming model. Zope 2 is extensible through so-called Products, which often were used to create new ways to help developers program in the web UI. As an example, one product I wrote back in 2001 is called Formulator, which helps people construct web forms by putting together fields in the Zope UI.

Unfortunately Zope 2 components are rather coarse-grained. Single Zope 2 components would have to inherit from a lot of mixin classes in order to play within the Zope 2 framework, fattening up their APIs to unmanageable sizes. Components would be so coarse-grained it became hard to reuse them in other contexts, as they would make assumptions that were hard to override. It was hard to just use Formulator's widget classes and not its Form class for instance.

Zope 3 instead has a powerful component model that allows for fine-grained components. These components define interfaces which typically only provide a few methods, and this allows for loosely-coupled development and true reusable components. We distribute these components in many separate packages, which allows for a lot of flexibility - application developers may choose not to use some.

The Future from the Past

Now let's turn it around and consider what Zope 3 can learn from Zope 2. What can we learn from the successes of our past? What have we lost? What should we bring back?

A user-interface

So the Zope 2 user interface has drawbacks for developers. It locks code into a hard to manage Zope-only format. It was also Zope 2's killer feature.

Zope 2's through the web user interface contributed enormously to Zope 2's success. Zope 2 was being adopted by non-Python programmers who discovered they could be very productive using the Zope 2 UI. These developers often turned into full-fledged Python programmers later on.

A user interface offers discoverability: in Zope 2 you get a drop-down list of components you could add to your object database. See an unfamiliar one? Just try creating one of those, see what UI options it presents, and click the "help" button. The UI encouraged experimentation and learning.

The Zope 2 UI also allows less hardcore developers to easily tweak layouts here and there and do a bit of simple scripting.

The Zope 2 UI was also used as a crude CMS and often extended to build simple CMSes that were still quite powerful for their time. In this sense the Zope 2 UI was quite similar to the admin UI that is one of the killer features of Django.

Zope 3 has lost almost all of this. Zope 3 does have a UI modeled after Zope 2's, but it's rarely used. Zope 3's UI was intentionally cripped compared to Zope 2's to prevent unmaintainable code to be created, and some ideas existed for checking out code from the object database for development on the filesystem, but it never really went anywhere. As a result Zope 3 is only approachable for Python developers, and is harder for beginners to pick up.

Grok has made Zope 3 a lot more approachable already, and I think is competitive with the other modern Python web frameworks, but a user interface could bring many more people in beyond this.

The Zope 2 UI had the drawback that it was all UIs in one, and did none of them very well as a result. I think we should work on bringing back most of the features of the Zope 2 UI back to Zope 3:

  • an introspector UI. See what content objects are stored in your database, what APIs they offer, what views exist for them. There is a Zope 3 apidoc tool that does this. In addition, the Grok introspector was developed in last year's summer of code, and more work on it was done in this year's summer of code.
  • a management UI: install, uninstall, configure and monitor applications and the server. Grok has a simple UI that allows you to do this. There is also the ZAM effort.
  • a through the web development UI: we shouldn't be thinking about a full-fledged IDE here. This one should be tackled carefully, step by step. Tweaking page templates is something that could be presented in a UI quite well, for instance. Some work to this effect was done in the five.customerize package.
  • an "editing backend" - it would offer navigation through the object tree, would display the contents of containers and would allow form-based editing of schema-driven contents. This would help provide a "good enough" solution for many applications that need some form of editing backend.

A component with a user interface starts to look like the coarse grained components of Zope 2. We should therefore be careful: the user interface should be easily replaceable and de-installable. We shouldn't be locked into it, but it should be there when we need it.

The good news is that many of the pieces are already in place for this work, and we should continue these efforts.

Comments:

Posted by eric casteleijn at Fri Sep 19 2008 18:23

"The obvious way for Zope to learn from Zope is for the past to learn from the future." Uhmm, are you sure you meant to write it this way around? I think you may giving away too much, even though your weblog *is* secret... ;)

Posted by eric casteleijn at Fri Sep 19 2008 18:24

Ah never mind, you did mean it like that. I should have read the whole fine article... ;)

Posted by eric casteleijn at Fri Sep 19 2008 18:31

And a hearty yea on all of this: we need the distribution of labor that zope 2 gave us where designers can work with templates without learning python, and power users can customize some things through the web. If we get a clearer separation between what can/should be done through the web and what shouldn't than in Zope 2, so much the better, as long as the answer isn't: 'nothing, mostly.'

Posted by Chris McDonough at Fri Sep 19 2008 18:33

Aw do we have to? ;-)

I'm with malthe on this one...

Posted by Martijn Faassen at Fri Sep 19 2008 18:51

Chris, you're with Malthe on something, but what?

Posted by Chris McDonough at Fri Sep 19 2008 18:56

Oops.. sorry for the lack of content... http://plone.org/events/conferences/2008-washington-dc/agenda/toc/why-through-the-web-development-isnt .

That said, I don't want to contribute "stop energy" to TTW dev and "framework UI" if there's a clamor for it, I just personally think it would be better to concentrate on other things first.

Posted by Malthe Borch at Fri Sep 19 2008 21:11

There's limited resources and you can't support everything. So let's not waste resources supporting a mode of development which noone uses except for demonstration purposes. TTW dev is symptom treatment and comes out of the high entry barrier to developing plugins/extensions for Zope.

But there's more to this subject than can be summarized easily. The debate on TTW sits right there between division of labor, barrier of entry, deployment systems, version control, automated testing and quality assurance.

Posted by Raphael Ritz at Sat Sep 20 2008 09:36

First of all thanks to Martijn for keeping the ball rolling on this. I agree that these are important considerations.

To Chris and Malthe: don't through out the baby with the bath water (as we say in German at least). This is not (or rather just to a limited part) on TTW *development*.

Note that Martijn mentions 4 specific subjects:

1. introspection UI (!= development); no doubt that's useful
2. management UI: again this isn't about development at all
3. development UI: this is where you guys get caught but note that this is a huge area. Nobody is talking about reviving ZClasses but it is a big plus to be able to customize templates or quickly try out things TTW. It's also not about lost efforts *if* there is a sane way to move those things to the file system if needed. (e.g., that way, people who don't have file system/code repository access can still contribute via a web interface and their work can be transfered by more privileged people later on - many CMF/Plone-based projects use this pattern with great success)
4. a content editing backend: again, this is not about development at all.

While I can understand that TTW raises red flags for those who have been burnt by it it would be wrong IMHO to simply say: TTW isn't, been there; done that. That's too simple minded.

Just my 2 cents

Raphael

Posted by Malthe Borch at Sat Sep 20 2008 19:12

Raphael, I am simple-minded, but I can read. When you respond to comments, please try to argue against the points made in the comment, instead of trying to undermine the comment by obscuring its message. This is not politics, it's open-source.

Posted by Jon Stahl at Sun Sep 21 2008 02:59

I think it would be more useful to focus on what Zope and Plone *should* allow TTW, and how what's started TTW can be smoothly transitioned to the filesystem, rather than what they *shouldn't* allow TTW. And I think it's important to include less-experienced "integrators" in the conversation, too.

Posted by Raphael Ritz at Sun Sep 21 2008 17:23

@Malthe: while this is open source (of course) this is also politics to some extend but anyway:

Addressing your initial comment specifically:
- "There's limited resources and you can't support everything." This is always true - nothing to argue about
- "So let's not waste resources supporting a mode of development which noone uses except ...": What's considered "waste" can depend quite a bit on use case, practice, experience etc. "noone uses": I actually tried to sketch a concrete situation that's not demo purpose (people working on customization of templates without having server/repository access or local installations - this is actually not that uncommon from what I experienced)
- "TTW dev is symptom treatment ...": I don't agree; it's a different way of developing for which there are use cases.
- "... out of the high entry barrier to developing plugins/extensions for Zope." while this used to be true I think with tools like ArgGenXML (for Plone at least) or - more recently - ZopeSkel the barrier has been lowered quite a bit already but I still do see room for an intermediate level.
- "The debate on TTW sits right there between division of labor, barrier of entry, deployment systems, version control, automated testing and quality assurance.": that is exactly the point: (i) TTW provides one further option to organize a devision of labor (less priviledged people that you would never allow to mess with the real/final code can still contribute) (ii) for some TTW has a lower barrier of entry (how many people got into Zope exactly because of this?).

But you are of course right that for the "real" thing you want to use "real" tools and proven appraoches. To me, the entire argument is mainly about how to attract people and there I do think TTW can make strong points. (How many people customize Plone sites but are horrified if you mention svn or unit testing?)

Also note that there seems to be agreement that TTW only makes sense if there is a clear path out of the ZODB and into the file system. Nobody wants to repeat the mistakes of ZClasses.

Finally, noone askes you (or Chris) to care about TTW. But I'm getting tired of the bashing that comes up as soon as someone mentions this. Again, I do think bringing this up is worthwhile. That there are different opinions is to be expected given our history.

Raphael


Posted by Martijn Faassen at Sun Sep 21 2008 17:39

Thanks Raphael for arguing so clearly, I agree with what you're saying very much.

My experience is much like Raphael in that there is quite a potential audience for TTW customization, and in fact Zope *had* this audience in the early days but has since lost much of it. I think allowing TTW customization can attract new users to Zope. TTW customization should indeed not be a lock-in; it should be easy to transfer customized templates (for example) to filesystem development, just like it should be easy to customize a filesystem developed template through the web if the site administrator/developers enable it.

I also agree with Malthe that TTW development should not be a treatment of symptoms: if development on the filesystem is complicated and requires the tweaking of many knobs, a user interface shouldn't be added to make it easier. The focus should be to make it easier on the filesystem level. I think the goals of the Grok project give me enough credentials in this area... I do think that even given a nice filesystem development experience, a user interface can still be added to lower the barrier of entry for certain classes of users.

Posted by Santiago Videla at Mon Sep 22 2008 01:22

Hi there, just a comment/experience from a newbie on TTW development

I was be able to start working with Zope 2 because TTW, no doubt about that.

Then I started to work on Plone developments, and ZMI help me a lot to find the ways to do things.

These days, I rarely work with the ZMI but it´s still useful. For example for GenericSetup customization on Plone products, using the ZMI it´s very easy to customize some things and when you finish, just export it to a FileSystem product.

And back to pure Zope, I´m really want to work with Zope3, but I´m more like a caveman, so GROK is my best friend these days. And I think that something like grokui.admin could be very useful for new cavemans like me

Just my 0.1 cents

Posted by Malthe Borch at Tue Sep 23 2008 05:58

It's a double-edged sword, really. You might think you benefit from this mode of development, but the fractal complexity of software quickly unfolds.

Yes, through-the-web development lured many people into Zope in the first place and such attention is always good for a project. But I can't help thinking that they were sold a bridge to nowhere, much like the one Sarah Palin claims not to have supported up in Alaska.

@Martijn: I must say that I think the whole talk of using a content management system to develop a website and then export the "code" to the file-system is complex, backwards and not very elegant. If you can describe a story that does it differently, then I'm very keen to hear about it.

PS: It would be much easier to comment on your (always interesting and timely) posts if the website would notify by e-mail when a new comment is posted. Is it written TTW? :-)

Posted by Aroldo Souza-Leite at Tue Sep 23 2008 09:29

At the risk of repeating what has already been said, there is one thing I find important to underline: TTW touches a wound that aches far beyond just "programming". The problem of diversity of skills around software development remains yet to be tackled. In the past, the efforts of bringing sanity to software development and usage proved unbalanced, focusing too sharply on "real programmers".
I think this is changing to the better - at least in the Python community. The pursuit of pythonicity as a ruling principle is gradually transcending the realm of Python code. I perceive Grok as a prominent instance of this change in mentality. More than being just yet another web framework, Grok exemplifies a common intellectual endeavour towards building solid (and pretty!) bridges to interconnect an archipel of activities related to software development and usage.
I hope Grok establishes an example of applied pythonicity in the Zope scene. Then, a few of us will perhaps gather our shattered pieces of self-esteem and try some TTW again, just for fun.

Posted by Martijn Faassen at Tue Sep 23 2008 17:40

I'm not talking about a general content management system to write code; I think that is where Zope 2 had some problems, using the ZMI both for content and code and site management. I'm talking about a specific system that can help people do customization of limited code-like things. Templates are the most obvious example.

In the past, I've had Zope 2 customers ask for such features. I've also worked with people where the distribution of labour was between template designers and coders. Perhaps TTW is not necessary for these scenarios. Perhaps the requests came in part because they're already used to Zope 2's through the web customization abilities. In fact sometimes I have tried to lead the customer away from such a solution, as code through the web isn't so nice to maintain. In any case it would be good to make sure that the GUI development scenario maps well to the underlying non-GUI framework.

I do think GUIs can guide someone better than filesystem code can. If we can make the GUI good/limited enough so that Python developers are not scared of the consequences either and will use it themselves, or let people they're close to use it, we would be on the right track.

Besides a GUI for development, I also listed a GUI for easy content management, a GUI for introspection, and a GUI for application management. I also want to emphasize that even if you don't like a GUI for development purposes, that doesn't mean a GUI for these other purposes is useless.

[Main]

Unless otherwise noted, all content licensed by Martijn Faassen
under a Creative Commons License.