Python Secret Weblog for 2005 December 15 (entry 0)

(7) Thu Dec 15 2005 13:34 The borderland between content and software:

Paul Everitt writes:

In the early days of Zope, you could design content "TTW" (through the web). You could answer questions about structure and suddenly, you had new kinds of content -- YOUR content -- that could be added to folders in the system. No programmers were involved, no special login permissions on the server, no database schemas to update.

First, a somewhat snarky question: Paul, if the olden days were so great, why did the Zope community move on from them? I'll go into what I see as some reasons why.

Anyway, I agree that this is a scenario we should support. I also am a software developer with many years of experience developing with Zope, so I know the trouble that this scenario can cause.

Paul is talking about the interesting dividing line between content and software, and between content authors and programmers. Let's keep it clear and call this the borderland, as calling it just software or just content warps our perspective -- we need to see it as both.

Paul also writes:

Alas, later in the history of Zope, the component folks decided that TTW was grotty and should be banished. There were good reasons for this...from their perspective.

I think these reasons are important to make explicit, and should not be swept under the rug, so let me describe some reasons from my personal perspective.

Paul attributes this development to the "component folks". I could be considered one of them, and they're Paul's favorite group of programmers that make his life complicated, but I think Paul will agree that the trend away from TTW was far broader than that. I think that this trend away from TTW to Product development is clear in Zope 2 as a whole, component folks or not, for the last five years or so. The question I asked in the beginning again applies again. Why did this happen? Why'd we move away from the apparently paradisical state Paul describes?

At Infrae, we have a lot of experience with people, end users and scripters writing software/content through the web, using facilities in Zope, Silva, and Formulator. While this is indeed very empowering for the end-user or scripter, it can also lead to an enormous increase in maintenance burden.

This gets noticed by the software developers and system administrators who will have to do the occasional maintenance (such as upgrades). They will have to deal not only with Silva and Zope (or Plone and Zope), but also the software/content that grew around it. This increases the complexity enormously. We can't do away with this extra code that grew around it; it's necessary for deployment of the software in that particular production setting. But we'd sure as hell wish it'd been more maintainable.

Programmers have tools and patterns to manage the complexity of software development and deployment. Editors and IDEs, test driven development, version control systems, release numbering, deployment tools, the works. These things help manage the complexity if an experienced programmer is using them, but a non-programmer or scripter doing development will not use them, and the programmer does often end up having to maintain code created that way.

As a side-discussion but exemplifying the pain: these tools work with the filesystem, but typically don't work with code developed through the web, such as in the ZODB. In the Zope community we've seen lots of attempts to make through the web work together with filesystem tools, with varying success, but I think nobody denies that this is a major pain. A general trend was to develop this stuff as code on the filesystem, as that made life a lot easier. Through the web development helps a class of users, but it can also be seen as a hindrance to the adoption of Zope as a software development platform by software developers. This is also an important audience to us.

So, letting users, customers, develop software as content is an extremely powerful concept. It's one we should support. It's also very dangerous.

The challenge is in reconciling the two. How do we empower users to develop things on the borderland of software and content, while not creating maintainability nightmares? People administrating large systems, as well as professional software developers, as many of us are in the Zope community, need a solution that answers that question.

I don't want to have to deal with solutions anymore that don't, as they're not complete solutions to me. They tend to shift or even increase the complexity on the longer term, something my company may end up paying for, as how do you explain this to a customer? The website works, right? Why is maintaining it so costly?

Let's try to figure this out. It'll likely take some smaller, careful, steps. Ideas are welcome.

---

Comments:

Posted by Alexander Limi at Thu Dec 15 2005 15:50

I think the TTW stuff should be supported, but as a configuration option in the Zope setup.

In other words, most of us would turn it off when we deliver stuff to customers, but leave it on out-of-the-box for something like Plone, that is designed to be tinkered with when you are new to it.

Posted by Martijn Faassen at Thu Dec 15 2005 15:57

I agree that any TTW development support should have the option of being turned off.

But the main question is: what should this TTW development actually support? How do we support it so that we aren't locked out with our filesystem tools? How do we support it so that the target audience doesn't get lost so easily? How do we support it so that a programmer can actually do something with the code produced by what could be a non-programmer?

I don't think that we necessarily have the right answers in the Zope 2/CMF world.

Posted by Peter Bengtsson at Thu Dec 15 2005 19:33

What are we talking about, really? Python Scripts, DTML and Page Templates? To me, a Python Script, Page Template or DTML Document IS content. It's not software. Software is the stuff in the space where there's no security restrictions eg. Python products.

Are people worried about allowing customers/clients do too scary stuff in the ZMI? If so, don't give them access to anything; you might have a complete design flaw on your hands if you can't trust your customers/clients to develop "advanced" content.

I love giving TTW options to people who "belong there" but I never join them there. My work is on the filesystem.

Posted by Martijn Faassen at Thu Dec 15 2005 22:22

Peter, if you think it's only software if there are no security restrictions, then Zope 3 doesn't contain much in the way of software -- security checks happen throughout the system. :)

Python Scripts, DTML and Page Templates can express almost arbitrarily powerful algorithms. They can form complicated interrelationships and interactions. That's why they're like software. Anyway, changing the definition of 'software' to mean one or the other doesn't make the problem I'm trying to describe go away.

I realize now that my original posting is probably a bit too less concrete. I'll post another entry that goes into a bit more detail.

Posted by Brett Haydon at Fri Dec 16 2005 03:24

In order to accommodate an SQL database I built an entire commerce/content management site TTW in Zope, which seemed simple and speedy to develop at the time.

Two years later and when I'm trying to maintain and change bits of the site, I bitterly regret going the easy way, as now I have to start from scratch when redeveloping it.

Bottom line is TTW works as a fast and dirty way way of doing things, just don't expect it to stand the test of time, and certainly don't expect that you'll be able to test it adequately.

Posted by Bill Seitz at Sat Dec 17 2005 13:13

Brett, you're assuming that 2 yrs from now you won't have to rebuild from scratch again. :)

Posted by Bill Seitz at Sat Dec 17 2005 13:22

I was thinking that maybe Zope needs a more-native way to synch objects with the file-system, so that some of those tools are available.

But maybe it makes sense to go the other direction - provide an API to the ZODB that makes it look like a filesystem.

http://www.itworld.com/Tech/2327/nls_ebiz_resistance051208/index.html