Python Secret Weblog for 2005 October 8 (entry 0)

(7) Sat Oct 08 2005 15:12 Zope and scaling down:

Ian Bicking posts about what he percieves is a focus of Zope 3 on modeling up-front:

Good development in the beginning means deferring choices as much as possible and focusing on results instead of abstractions. Abstractions should emerge from your functional goals, and if you spend a lot of time modeling in the beginning then you've made premature choices and designed code that you don't yet understand. You haven't just wasted time, you've introduced a liability.

I agree completely with this view of software development. This is how I try to develop software, learned through quite a bit of experience, just like Ian, I'm sure. And luckily enough, it's perfectly possible to do such a style of iterative development on top of Zope 3. I'm not sure what gave Ian the impression that you can't.

At Infrae we've been doing this for a few months now though, and the application we've worked on definitely evolved, sometimes quite drastically, in the face of customer feedback and a coming into focus of sometimes vague requirements. Since Zope 3 tries to stay out of the way of your Python code, you can refactor like you'd do with any piece of Python software. In fact, I talked about just this in an older post:

Jeff Shell already mentioned that Zope 3 makes it easier to build an extensible framework while actually building something useful for a customer; Zope 3 gives a lot of flexibility and extensibility right out of the box without much effort for the application developer. This I think is great news for the long term maintainability and extensibility of Zope 3 applications.

In addition, I can say that extraction of reusable code from Zope 3 projects into reusable libraries is much, much nicer than doing it in Zope 2. That doesn't mean it's actually easy; writing reusable code is always hard, but it's now much more doable. This is one of the coolest things about Zope 3.

[snip]

Framework extraction from practical applications is often the best way to build truly useful reusable components, so Zope 3's vastly improved extractability of reusable components is great news.

See also Jeff Shell's post.

Perhaps Ian gets the wrong idea because of Zope 3's focus on the concept of interfaces. While interfaces are common in the Zope 3 framework (which needs to be pluggable and flexible), Zope 3's component system allows you to register adapters and views for classes just fine. There's no need to start designing lots of interfaces right from the start. It's convenient to define (and evolve) your data layout of some content object in a schema, but I'm sure Ian as the creator of SQLObject wouldn't object too strenuously to this.

Or perhaps Ian gets the idea indeed from my comment on hello world. I used the word "small", but I meant really small; the context was "hello world". Zope 3 doesn't scale down well enough to tiny web applications of the "hello world" scale; the overhead of ZCML and such feels too big then. We need to do work there.

But as soon as there's enough code for your web app to be useful to do anything, the overhead of ZCML and interfaces quickly shrinks. I'm not saying it couldn't be improved further, but to say that Zope 3 does big-design up-front because "hello world" is slightly more difficult than it should be is a bit of a stretch. We're still programming in Python, after all!

---

Comments:

Posted by Ian Bicking at Sun Oct 09 2005 00:12

Well, when I think about starting from functional goals, I'm thinking about starting from actual HTML -- you have a page, you have some links, you have a form, the form has A, B, and C fields, etc. I haven't looked at Zope 3 docs or tutorials for a while, but the last ones I looked at started from Python classes and objects, doctests, getters and setters, etc.

It's a challenge -- PHP starts you solidly out in HTML, and I think that's a big part of its success -- you get into the UI right from the beginning. But it's also clear that's a lame design immediately after that first step. DTML/ZMI application design was similar, and similarly flawed. But I think there's some way to make it work; and certainly Zope 3 isn't deeply incapable of doing that at all -- it's really a matter of perspective. And the documentation I've seen takes a Python object perspective -- with fancy stuff like the ZMI and ZODB -- that is distinct from that UI perspective.

There's some minor details that I think could be better in Zope 3, but I don't know enough to say a huge amount. I think ZPT should have an easy Python escape in the model of Kid (which is *not* like the PHP/ASP model, and is far better IMHO). I'm not sure what I feel about the ZMI at all. And perhaps some of the stuff ZCML describes could be implied by convention, instead of being explicit in all cases. But I guess I'm thinking more about perspective than the particulars of the technology.

Posted by Leonardo Santagada at Sun Oct 09 2005 16:28

If only they did a screencast showing how to make a wiki like in the turbogears site maybe people could see how easy it is to make a Zope 3 app. For now I only see a lot of Zope 2 guys talking how easy zope 3 is. Well for most of us zope 2 was a nightmare to understand so we all turned to making our own web frameworks. To reunite the python programmers you zope crowd should make some easy to follow tutorials(I really think one or two screencasts will do) and some good documentation.

Posted by Martijn Faassen at Sun Oct 09 2005 20:06

Concerning functional goals, the application I'm talking
about in the article started with HTML mockups.

I agree that at least the Zope 3 tutorial slideshow (which Ian might've seen) is too heavy handed in telling people to do tests before they even get some basic code going. While I find the focus on tests very valuable, I also think it loses beginners and is thus not the right thing to advocate right at the start of a beginner-level tutorial.

Tests are very important for those who are framework developers, but the perspective of application developers can be quite different (even though I consider tests invaluble when developing applications as well). Anyway, Zope 3's framework developers recognize there are different styles of development, and different ways to do testing.

I also agree that a few screencasts would be useful (or even essential apparently in this day and age of Ruby on Rails :).

Anyway, I consider this more a documentation issue than anything else; Zope 3 has been developed iteratively and I don't think anyone's advocating Zope 3 applications shouldn't be developed iteratively. Iterative development is hard to catch in a tutorial though. And a tutorial on Zope 3 rightly tends to focus on explaining Zope 3 specific things, which inevitably means it ends up in abstraction & component space much sooner than you'd really do if you were developing an actual application.

Posted by Duncan McGreggor at Tue Oct 11 2005 02:20

Ian, that's an interesting perspective... I never think of apps coming from the UI perspective; I always think: what's a good API for my app that could be easily taken advantage of with a web UI. Hmm. Definitely good food for thought.

My first full-blow, non-toy z3 app was a custom-built CMS for managing a simple web site. Design (HTML, CSS, ZPT) took about two weeks (because of browser crap incompatibilities). The rest was building a custom content type (15 minutes), and letting zope know that the content type should be manageable through the web (ZCML/configuration), and fine-tuning. All the forms that I needed were auto-generating, all the data was handled automatically. The type of project that -- without the design phase -- has taken me 4-6 weeks to beat into submission through traditional web development frameworks took me less than a week, or less than three, including design.

I have to say how amazed I've been with it since I began using it. The first year or so I kept up with the releases, scanned the source, installed and diddled, but in the last six months, I have basically abandoned all other non-z3 projects or converted them to z3. It's the most impressive web framework I've ever used.

It's a good match for me because of how I solve problems and how I navigate my own headspace, how I organize information, etc. The z3 API makes so much sense to me (for the most part); I can get to just about any file I need intuitively. For me, intuitive APIs are a must. At first I hated ZCML -- I wanted everything in python. But it makes sense, now -- configuration really should be not be in code. I also didn't appreciate the complete, pervasive extent to which components and adapters have been used -- not only for the programmers, but down to the very structure of z3 itself (a structure which you can override at will with your own adaptors!). It's truly amazing stuff.

It's like a web framework designed by scientists and adapted to developers. It's beautiful, it's self-consistent, and it runs like a dream.

Serious python programmers can jump right in, no problems. Novices can too, but there has to be a suspension of deeper understanding. Once the coder has begun to get comfortable "living" in the component space, she or he can then start to look around and explore. With that exploration and growing familiarity, the true power and elegance of z3 will become apparent.

Posted by alan runyan at Wed Oct 12 2005 15:26

I have always leaned towards Ian's view. And I think Martijn's response completely skirts the issue at hand.
As of Zope 3.1 - the system is still not ready for
people who are not steeped in frameworks, abstractions
and rabbit holes. I have been trying to get a friend
of mine (whom was trying to use formlib) to get
up and going. And he was just lost - he spent a lot time. He has been doing web development for 5 years mainly with
ASP, VB.NET, PHP and mod_perl.

[I can barely write this because I want to launch into
a rant that will work against me trying to make
a point.]

Zope 3 does not scale down because the issues the framework
are trying to solve are inheriently sophisticated. Thats the problem. The core Zope 3 development, again, is aiming
at making the impossible fairly easy (versionable i18n
content with sophisticated workflow) but neglecting (to some degree) the basic programmer who WANTS to bang stones together. Lets make an analogy (I'm winging this).

Building a web based system is like making fire.
Zope 3 is a flamethrower. But most people have never
seen flamethrowers much less are versed in the way of
using them. So they want to either bang rocks (do everything
manually), use a match (php/everything inside a request and
template) or use a lighter (rails/turbogears).

My problem is that a lot of people in Zope 3 eitehr look
down on the neaderthals banging their rocks together talking
about inlined code in templates _OR_ need flamethrowers eventually (their domain is the high end) and they are
content with a large framework. Regardless either way
it leaves teh rock bangers out in the cold with no story.

The fact is Zope 3 has been in development for over 4
years and is now at Zope 3.1 and has almost no story for
people who want to:

- Write a HTML view

- Have some python code that will manually store the
values in RDBMS (or ZODB - but lets assume rdbms first)

- Does not mention Security, ZCML, Interfaces, Unit Tests
or anything else outside the initial scope of first task,
which is persisting html form values.

Yes. I want to do something about this. But honestly
this isnt interesting for me. If my friend comes down
and forces me to work with him - than I will help out.
And thats fundamentally one of the big problems in
the world of Zope: Rock Banging isnt
interesting but its necessary.

alan runyan

Posted by Martijn Faassen at Wed Oct 12 2005 16:04

I don't see how I skirt the issue at hand -- the issue at hand in this post was whether Zope 3 allows you to do
iterative development, and it does. I don't believe it
forces people into modeling up-front.

But if you want to change the topic, I agree Zope should scale down better at small and tiny applications, for the marketing and educational values you mention. I've been advocating this for a while now...

I agree we shouldn't mention automatic testing at all in a Zope 3 tutorial.

Concerning interfaces, I think most mention of interfaces can be avoided in a tutorial already, except as a "Hey, look, this is how you can quickly describe a schema" thing. Benji's tutorial tries this approach.

I don't know how to get rid of ZCML. We may make it shorter and easier and change the syntax away from XML over time, but that configuration information will have to go somewhere if you don't want to throw away the flexibility and evolvability aspects of Zope 3 development.

I'm worried about having to mention security early on too, though the usability of secure software development is a difficult topic.

I'll note that saying RDBMS should be chosen *first* above ZODB in the context of Zope 3 is I think to ignore the value of the ZODB. I see lots of new frameworks around the block say "Look, no SQL, your objects hide all the SQL!". Zope can just say "look, no SQL!", and that is valuable in itself. There are pros and cons, but chucking the ZODB as the primary storage medium wouldn't be my choice.

I do agree that deep RDBMS integration needs more attention from someone in the Zope 3 community. It won't be me, as I have enough on my plate already.

Posted by Roger Ineichen at Wed Oct 12 2005 16:20

Be careful,

and don't let us fall back to write code in page templates. Such inline code is a pain for developers.

Perhaps web developer how only can work with such a layer should better stay at PHP and not use a application server.

I agree to support a scripter level, but not in the presentation layer! Let us define such a layer and support scripters with a easy to use concept.

Regards
Roger Ineichen