Malthe started an interesting discussion on the possibility to rewrite Plone from scratch. I have some thoughts on the topic of rewrites, evolution and revolution. Here are some of my scattered observations. While at Infrae, I led the development of Silva for many years. I also initiated the Five project. The creation of Five was driven by two concerns: I participated early on and enthusiastically in the development of Zope 3. I'm still a big fan of the project. At the time it was generally assumed that this was the next-generation platform that in time would replace Zope 2. The stories about how we were going to port Zope 2 code over were always vague. Luckily Silva can almost completely export its state as XML. I therefore assumed for quite a while that soon enough we'd end up rewriting Silva on the Zope 3 platform, and even without a way to port any code, we'd at least be able to carry over the content. As time progressed, it became clearer and clearer that the rewrite of Silva was not going to be happen. Of course not! Would any customer would pay for it? What would you tell a customer? Give us lots of money and we'll give you a new version with initially less features and more bugs? Infrae has wonderful, quite enlightened customers who care about Silva, but telling them that was unlikely to solicit very positive responses, and for good reason! Now Plone has the benefit of having a much larger community than Silva does, and it's full of all kinds of energy. But telling community members that you are going to rewrite Plone is going to be almost as hard a sell, and will give rise to inevitable confusion. Given the experience we had with the confusion surrounding Zope 3, I wouldn't announce you're rewriting Plone. Don't call it Plone, at least. You might get away with it if you are the sole dictator of a project. Apple could more or less do it with Macs, transitioning to Mac OS X. Microsoft could do it with Windows 95 and Windows NT. But even these use the same brand name and name the individual things differently - note how they use names that are not much like a version number. There is no Plone dictator. A rewrite project named Plone will certainly give rise to confusion in the community. You'll get questions like "sould I use Plone X or Plone Y?", ages before Plone Y is even ready. "Hey, why can't I run PloneAmazingProduct on top of Plone Y?" It's easy to avoid such confusion. If you want to create a new thing, give the thing a new identity. I even gave Five an identity by itself, and it wasn't really a new thing at all - it's whole idea was to combine existing things! I also learned the best way to make a new thing is to base it off some existing thing: Zope 2 and Zope 3 for Five. libxml2 and elementtree for lxml. Zope 3 for Grok. If you want to get a lift from Plone's community, by all means try. Closely ally it with Plone somehow. Make it easy for the two systems to work together, perhaps. That's the relationship we're aiming for between Grok and Zope 3. But naming it the same is a bad idea. We have namespaces in code for a reason; without namespaces confusion and bugs are more likely. I think the same applies to projects. It's true that reimplementing software is very attractive to software developers (it is to me). It's also true it's frequently not a good idea. I disagree with the position that rewrites from scratch are always a bad idea. I do agree that any idea to rewrite should be scrutinized aggressively. It's good that it's happening. This needs to be done, as rewrites have huge costs. Even backwards incompatible changes have a severe cost, especially where they are done low-down in the stack. Rewrites are like backwards incompatible changes with a lot more confusion to the community. If one would change the name for the new effort, is it a good idea for Plone? I think it could lead to useful things, but it could as well lead to nowhere. I don't know. When I was pushing Five, I used the phrase "evolution, not revolution". After a while I started hearing this back from others, so the meme was spreading. I still think this is a good idea for the position Plone is in - the same position Silva is in. So was the Zope 3 project, a near-complete rewrite, a bad idea? It's hard to say. I certainly like much about how it's turned out. It's one of the most advanced pieces of web technology out there. There's so much mostly untapped potential in it. The Grok project is now happily in the process of tapping it. So is the Zope 2 project! I believe we can go very, very, far with Zope 3 technologies. I think we'll surprise a few people with that... On the other hand, Zope 3 was incredibly disruptive to our community. False impressions as strong as promises were given, and much of what was generally expected never became a reality. We needed to do a lot of work to reforge the community in the aftermath of these disruptions. Then again, Zope 3 is also serving as a unifying force to our communities. Silva, Plone, and Zope 2 itself, have a more or less clear direction: toward the adoption of Zope 3 technologies. This saved the Zope community from breaking up into separate splinter groups. I think this is a great thing. Could a project more closely allied to Zope 2, a less disruptive project, reached the same thing? I don't know. I do think that by being independent from Zope 2, Zope 3 could make braver choices and helped us set a clear course. So Zope 3 was a curse to our community, and it's a blessing. It hurt us and reinvigorated us. Where will the evolution end up? Can Plone or Silva ever evolve to a purer Zope 3 base, using Five? That, after all, was also the idea behind Five. Can we slowly reengineer our applications so in the end they run on a pure Zope 3 core? I don't know. It remains to be seen. Certainly it appears quite distant in the future right now. Certainly also people are making great strides forward with every release. No matter whether we'll reach the end goal soon, later, or perhaps never: I am convinced that the direction we're going in, as a unified community, is a good one overall. Paradoxically, these days, I'm more in a revolution mood. I'm pretty free in what I can do now. I feel I paid my dues to Zope 2. So I'm aiming to change the world with Grok. Grok is purely Zope 3 based. It's Pythonic. It lets go of the past. No need to worry about Zope 2 cruft anywhere. A revolution! But note that Grok is pretty conservative in its revolutionary plans - Grok is Zope 3 in a different coat. Grok is not a rewrite but based on Zope 3 technology in development since 2001. It's revolutionary for me, as instead of combining two existing things I'm just building on one! It's hopefully revolutionary to others as we're putting a lot of power into the hands of many. We will end up going full circle. People keep asking me about supporting Grok technology in Zope 2. I think this will eventually happen; it's not hard, and so many people are asking about it one day someone will actually get up and do the work. Unless someone pays me a lot of money, I'm not going to do it myself. I'll support it though!
Grok needs Zope 2. Grok needs Plone. We can count on many of our best contributors to come from those communities. Perhaps the Zope community evolves by punctuated equilibrium. Long periods of seeming stability with intermittent fast spurts of new developments based on evolutionary innovations that took place almost in the hidden. Or perhaps that is just another metaphor that doesn't quite fit. We are about 10 years into a fascinating experiment. Can Zope continue evolve forward? Can we continue to be at the forefront of development? Will we continue to innovate? Can we reinvent ourselves while keeping our community whole? (and what a community! What other Python web framework has two foundations and a business partner network?) Can Zope continue to be relevant in the future? Yes, I believe it can, and very much so.
![[Comments]](/blog-content//themes/Default/comments.png "11 comments")
(11) Tue Jan 29 2008 15:02 On Revolution and Evolution:
- Comments:
Posted by whit at Tue Jan 29 2008 17:25
Rewrite is an ugly word. What Plone needs is a rethinking of sorts or maybe a continuation and manifestation of some of the thinking that has been going on for awhile. Malthe's project could be this sort of manifestation.It's unfortunate malthe tends to focus on the shortcomings of plone as if they were conscious mistakes rather than explore the processes and events that created them to avoid repeating them. also, we shouldn't be blinded by this shortsightedness and forget that what he is attempting can benefit everyone if we all have the proper attitude toward it. I think what is needed here is a slight reframing of perspective.To elaborate, Plone's strength is primarily in it's community and it's special breed of open source brand. There are no hard and fast rules that state that Plone must be a single application on a single platform. The sensibilities of plone can transcend and probably should.There is definitely a space for a well organized, lightweight, usable and performant CMS in the zope world (and the python world, and the larger world of web applications). It makes sense for this effort to borrow and learn as much as it can from Plone. and the plone community would also do well to foster such an effort under it's auspices, not as a replacement for the current Plone, but as a complement. after all, plone started as just a skin on an existing web solution and alot of it's best sensibilities live in that humble beginning. -w
Posted by Martijn Faassen at Tue Jan 29 2008 17:35
Whit, those are good points. I do support the creation of new CMSes taking the lessons learned from Plone (and Silva and so on). I'm interested in tackling that one myself one day, actually.I also agree that Plone's community and brand would be very useful for such a CMS. But the brand would need to be managed very carefully. Besides the possible namespace confusion (which can be mitigated if you manage it well), blessing a project that might never lead anywhere with your brand name would also be a bad idea. Better have it prove itself first.
Posted by Ian Bicking at Tue Jan 29 2008 19:38
Not necessarily apropos of this post, but kind of about the Zope 2/3/Five evolution, is that I think Repoze offers a different kind of evolution. Which might mean competing evolutions, which could be good. While Zope 3 (via Five) is about cleaner and more explicit object interactions, WSGI (via Repoze) moves towards using HTTP (aka WSGI) for interaction instead of objects.Or, instead of competition, it might mean that it's possible to wack away at Plone from two angles instead of one.
Posted by Martin Aspeli at Tue Jan 29 2008 19:56
I think it's very encouraging how much sensible stuff is being said here, by Carlos, by Paul (in his replies to Malthe's email), and by Malthe himself. It's a sign of maturity and experience, and I think we're largely pulling in the same direction.To Ian - Repoze (and WSGI) is very much on the horizon, and I suspect it'll be discussed at length. I think the Repoze/WSGI/eggs evolution is complementary to a deeper Zope 3/development concepts type evolution. We need (and want) both.The best part of Martijn's article, by the way, was this: " Zope 3 technology in development since 2001". That's seven years, folks, and I think Zope 3 has only really had relevance to third parties (i.e. not core Zope 3 developers) for the past two years or so. I'm not sure Plone (as a whole, including Archetypes and supporting libraries we maintain and depend on) is much smaller than Zope itself. Any rewrite would of course be smaller than the current codebase, but it would not necessarily take less time, relatively speaking, to get it right. There's a whole lot of evolution to be done in 5-7 years. ;-)Martin
Posted by Martin Aspeli at Tue Jan 29 2008 19:58
Sorry, I meant to write: "To Ian - Repoze (and WSGI) is very much on the horizon, and I suspect it'll be discussed at length" ... at the Plone planning summit next week.Martin
Posted by Martijn Faassen at Tue Jan 29 2008 20:04
Ian, I agree that projects to use WSGI with Zope (such as Repoze) are going to be important. Multiple dimensions (Zope 3, WSGI) for evolution are good. Solid, out of the box WSGI support for Grok is certainly on my agenda for this year. I think we're close; we just need to package an out of the box solution.I don't see them as competing but complementary. We will need to take a while to figure out what approaches are best for what problems. The community needs to gain experience with both technologies to determine whether Zope 3 components, WSGI middleware, or combination of both should be used to do a particular job.
Posted by Max M at Tue Jan 29 2008 20:38
Sometimes I love how easy it is to get start a project in Plone. At other times it makes me really furious.When I started programming (omg like 25 years ago) I was really fascinated about the idea of only having to do a boring job once, and then build on top of it.For the rest of my life the machine would do my boring job for me. Yeah right! In this respect Plone has failed miserably!About a year ago I had a relatively complex task of writing a code for importing adresses from xml files. Not that the task was that difficult in itself. But the details made it take about a week.It ran on a very old Plone version on Python 2.3.I knew that I would later have to reuse the code in an upcomming Plone version, so I wrote it very conservatively. I made a simple Zope 2 product as a tool, and saved tha data on that tool in OOBTrees, OOSets etc.The last week I had to rewrite it for Plone 3. I basically had to change the transaction semantics from get_transaction().commit() to transaction.commit(). It was the easiest rewrite I have done in years. Oh yeah and the 20K adresse ran so fast that it shook me to my AT bones.
This for me is the core of the problem with Plone. It is layer upon layer of backwards incompatible stuff. For every new Plone release my products have been broken. Everytime! I have spent far more time rewriting my products to new Plone versions than I have ever spent writing them in the first place.And I have made a lot of complicated products like webmail, workgroups, calendaring, contact databases with ics support, page aggregators etc.
Currently they are not being maintained anymore unless a customer specifically needs them and pay for the upgrade. It is simply to much of a bother to do it for free, as it will easily take several days per project.
The problem with every new layer that has been added to Plone, is that it doesn't really change the programming efficiency very much, but stops old code from working. And what worse is, is that you still need to understand every goddarn layer to code efficiently in Plone.
Plones style of 'sort of' backwards compatibility is a real serious problem for the project. A new layer of code should generally never break something that allready works, and new layers should be independent of the old layers.
I think that the best approach for Plone and Zope 3 would be to just develop them in parallel, so they can run on the same instance, on the same zodb, and then make shure that they do not break each other.
Death to deprecations!Posted by Martin Aspeli at Tue Jan 29 2008 21:13
Max, I think using Archetypes or other high-level tools to write your address import functionality would've been a mistake. In fact, I think even using Zope and CMF specifics like tools would be overkill. The truly re-usable way would be to make a pure-Python package that used simple Python data structures, and a thin, next-to-no-code Zope integration package if you really needed it.However, a lot of people can't or won't do that, because they're not really programmers. They're integrators who just need a new content type, a form and a theme. Plone's programming model (for third-party authors, not for core developers, who face different challenges) has been geared towards this.Obviously, stuff breaking with upgrades is a problem. I think we're getting better at this, having learned a whole bunch of painful lessons. Plone 3.1 is likely to be a lot less painful, as is 3.2 if and when it materialises. However, sometimes we *do* need to break something for two reasons:First, Plone is a near-infinitely customisable system. That means that the only way to have 100% compatibility with every single change people could make is pretty much impossible. Of course, there are APIs we told people to use and still broke - that's definitely a big problem, and something we've discussed at length.Second, Plone is first and foremost a product, not a general-purpose web development framework. Sometimes, we have to make changes (e.g. to UI conventions or libraries) to further the quality of Plone itself, which can cause some pain or deprecations. I don't see a feasible way around that.Incidentally, some of what we're talking about doing tries to shift this problem by doing customisation in a different way. If creating a new content type could be done with simple, declarative syntax or (XML) files, then backwards compatibility becomes a lot easier. If adding behaviour was more about using relatively stable interfaces and writing adapters and event handlers for those interfaces, then your custom code is much better separated from the framework code and thus better isolated from changes to it.Will that make Plone 4+ infinitely forwards and backwards compatible? No. But hopefully it'll ease the pain.One thing I've learned, actually, is that maintenance is the ugly side of programming regardless of what platform, tool or project you're on. There will be bugs. There will be changes. There will be things you need to migrate. I don't really know of (m)any non-trivial system where code written many years ago works, unmodified, on new versions of the underlying platform. Maintenance costs money, and yes - you either need to build a community who cares or have customers who pay in order to get things maintained. In many ways, that's the premise of the Plone project itself and the story we tell customers when they ask how and whether Plone will live on in two or five or ten years.Martin
Posted by Martin Aspeli at Tue Jan 29 2008 21:17
Oh, and in no way do I think any of that would be any easier with a complete rewrite or parallel project. A lack of a migration path is the same problem, whether it was implied and not delivered or simply discounted from the beginning.Sharing a Zope/ZODB instance between two ultimately different projects is not an important goal. Setting up two Zope instances is trivial and not that much overhead in real-world situations and is likely to be much more safe and stable.Having some kind of dump-transform-load migration story is important (between versions, servers and possibly different systems). It's also very hard in practice.Martin
Posted by Martijn Faassen at Wed Jan 30 2008 02:17
I don't agree with Martin that sharing a ZODB between Plone and Zope 3 code is not an important goal, as I disagree that these are *necessarily* different projects. I can see people write Plone content types in Zope 3 - Plone's folder would need to be equipped to branch off into pure Zope 3 land, and then Zope 3 would completely take over. It'd take some integration of the security machinery. Hopefully the long awaited __parent__ branch will make this possible.Five hurts especially where it has to bend over backwards to please Zope 2, especially in the area of acquisition. This makes a lot of hacks necessary. If you could have an easy way to branch off some Plone content into Zope 3 land, I think that would make our lives more easy.I don't think this goal is very far away. I think declaring it is not an important goal is a bad idea.
Posted by Max M at Wed Jan 30 2008 10:08
Martin Aspeli, I did not say that a parallel project is the best solution. I mean that we should bend over backwards to avoid backwards incompatibility. Just like Zope 3 is designed to do.One example is acl_users. How many versions of that have we seen in Plone? The problem is not the need for new ways to authenticate, it is how the need is adressed.If acl_users had not been replaced with another acl_users implementation but supplementet with a parallel acl_ploneusers or something like that, it would not have broken the backwards compatibility.but that is a somewhat naive example. What I want is the lessons of zope 3 implemented in Plone. I believe that the best way to do that is to have a parallel zope 3 / Plone instance.
