< New Year's Python Meme
ME GROK BOOK >

[Comments] (16) Sat Jan 02 2010 13:23 Python-based configuration in a Grok near you:

BFG 1.2 offers imperative configuration: doing configuration not in ZCML but in Python. The declarative configuration system is built on top of this. This is an interesting approach that has some merit.

It's interesting to compare this to Grok's configuration system, which also focuses on Python, not ZCML. Grok however offers a declarative configuration system in Python, not an imperative one.

It's important to note that from the start Grok's declarative configuration system allowed the user to be entirely explicit about how configuration happens. I don't know whether Chris was referring to Grok when he mentioned that in BFG no false choice between convention over configuration and ZCML is offered, but in case anyone had the wrong impression: Grok doesn't offer such a choice. You can be as explicit as you like. Relying on convention-based configuration patterns is an option when using Python-based configuration, but not an obligation.

Grok's declarative configuration system can be used in tests. If you have a component that you need to register, you can use grok.testing.grok_component to automatically configure it according to the standard Grok configuration rules.

Here is an example, a component:

import grok
class MyAdapter(grok.Adapter):
   grok.context(AdaptedFrom)
   grok.provides(IAdaptedTo)

This adapter when configured, be registered as allowing instances of class AdaptedFrom to objects providing interface IAdaptedFrom.

If we want to configure this adapter in our tests, we can use grok_component:

>>> from grok.testing import grok_component
>>> grok_component('MyAdapter', MyAdapter)
True

The grok_component function can also be imported from the reusable grokcore.component library, through grokcore.component.testing.

Interesting about this design is that Grok allows the use of the same configuration system for test-configuration as well as real configuration. There is no need to learn a different configuration approach when running tests.

Grok's focus on configuration as an area to challenge existing Zope assumptions is long-standing. Grok strikes a different balance than BFG as Grok aims to remain compatible with existing Zope-based systems.

Nothing's perfect. So, I'll close with some areas where we should improve Grok's configuration system:

Comments:

Posted by Chris McDonough at Sat Jan 02 2010 16:49

Yes, the "false choice" comment was a bit of a shot across the Grok bow, sorry.

While I really appreciate the ideas that came out of it (in particular, martian), It always struck me as a little odd that Grok made a big deal about providing a Python-based declarative alternative to ZCML.

While it's nice to have better locality of reference, the syntax used for declarative config has never really been a huge issue for me personally. Instead, I always thought it was a little strange to only have helper stuff for *declarative* configuration, like ZCML directives and grokker handlers. Obviously at the root there's some imperative thing going on. We polished this up, documented it, and made it available to end users, then based the declarative helper stuff on it.

I think when people complain about "ZCML" credibly (arguments not involving "I don't like XML", which is a weak argument, as all configuration syntax sucks in one way or another), they are often actually complaining about declarative configuration in general. Whether it's in Python or XML or YAML or whatever, some people just don't get it, and won't ever get it. Offering them an imperative alternative seems to make sense in a world where only developers are in charge of configuring an application. ZCML tries to extend the configuration of an application to a set of mythical "integrators", but Grok-style configuration doesn't. Once you're in a world where only developers interact with the configuration, purely declarative configuration isn't really necessary, and it's actually maybe a bit harmful, because doing something slightly out of the ordinary requires understanding and extending the configuration framework rather than injecting some if/elif/else clause in some set of imperative configuration steps.

But even if no one uses the imperative API, you can write your declarative directives in terms of this API too and it just helps you focus in such a way that the configuration code gets smaller, cleaner, and better-documented. At least that was my experience.

It also struck me as a little odd that Grok made a big deal out of "convention over configuration", because I never really thought that was much of a win; instead I thought it sort of hid a weakness. Grok does offer explicit, albeit mostly declarative, configuration. But it seems to make a big deal about convention over configuration in tutorials and docs. I *think* this is because Grok has gone down a path of providing a frontend to existing Zope 3 APIs and a lot of those Zope 3 APIs it's frontending can be fairly baroque. "Convention over configuration" is way to hide a lot of their complexity without actually *removing* any complexity.

I don't think that hiding complexity is really a good long-term solution vs removing complexity. I'd rather just have fewer features and less code than "convention over configuration" helper frontend to a maybe-too-large set of libraries, because the helper frontend is just more code you need to understand when it doesn't work right. You can't add simplicity; you can only take away code to make things simpler, IMO.

So, yeah. Some of this stuff gets at the heart of why BFG even exists today, I hope its not taken personally.

Posted by Martijn Faassen at Sat Jan 02 2010 20:48

To me reducing complexity and increasing regularity and learnability are related but not entirely identical goals. There are of course other goals that stand in some tension with simplicity like compatibility and features.

I think one interesting property of "convention over configuration" is that it encourages shared patterns of code, thus making code easier to understand. If the easiest way to hook up a view is to follow some convention pattern, it's more likely people's code will follow the same conventions.

(This in tension with flexibility of organizing code layout)

Another interesting property of convention over configuration is that it can help smoothening the learning curve. It provides sensible defaults for a lot of things. Compare to some JQuery widget where you can pass in a *lot* of configuration options but it still does something if you pass in only a few. This allows you to incrementally learn about controlling the API. Of course a better underlying API can make this less necessary as it's easier to learn, but sometimes you simply do want to offer a lot of configuration options.

(and documentation is of course a huge contributor to lowering the learning curve)

This is why I think patterns like sensible defaults can be helpful in making a system easier to learn and more regular to use. Whether what you accomplish by that can be labeled "simplicity" I don't know; the term has a wide range of interpretations.

You are of course right that removing complexity is also a good strategy - this is something where BFG has been a big inspiration to me.

You have of course noticed that the last year we've spent a huge effort to take away code to make things simpler in the Zope Toolkit. So far that has only been a relatively simple process of dependency cleanups that allowed us to isolate and eventually eliminate a ton of code that wasn't actually being used anyway. This means there is already a lot less to worry about.

Note that the grokcore.* libraries are not quite frontends. They are configuration libraries based on zope.component and zope.configuration - they are *alternatives* to ZCML-based configuration, not frontends on top of existing ZCML dirctive implementations. They just make sure that the configuration they do is compatible with the traditional ZCML directives they replace.

Hopefully in 2010 we can take the next steps towards taking away code. I think the long-standing publisher rewrite ideas could help a lot.

Posted by Chris McDonough at Sat Jan 02 2010 23:07

We've had this conversation before, of course.

Sensible default arguments are a good thing. I'm pretty sure this doesn't count as "convention over configuration", though. I think this is just sensible default arguments. "Convention over configuration" is something like encouraging that you name your templates after your class names, or creating tables based on your class names, and then bridging two otherwise unrelated systems together using the resulting mappings. It's not a feature provided by the language used to implement the framework.

I've noticed the ZTK work, yup. I've even done a good bit of it. But I fear that the pace is too slow, the set of changes too conservative, and the politics too brutal to outrun the curve of irrelevance that Zope currently finds itself behind. You talk about reimplementing the publisher; obviously we've already done this. Maybe you should think about trying to build a Grok-like system on top of BFG?

Posted by Martijn Faassen at Sun Jan 03 2010 13:01

Note I talked about sensible defaults, not sensible default arguments. Grok's convention over configuration system is based on defaults when a directive is missing.

Whether naming a view class the same as the template for that view, or a ORM class the same as a relational table that it maps is "sensible" or "otherwise unrelated" is in the eye of the beholder. I think one can make a reasonable case for such naming patterns, and if one can, then one can make a reasonable case for defaults based on those patterns.

As to curves of irrelevance, I think there are plenty of people to whom Zope is entirely relevant.

As to switching over to a completely different platform for Grok: there is a lot of code that I have or can use with Grok that I would prefer not to have to rewrite.

Posted by Martin Aspeli at Sun Jan 03 2010 15:12

Echoing Martin's statements - I'm pretty sure there are a lot more people using Zope (at least if you count people using it through Plone) than there are using a many other "relevant" Python web frameworks. That includes BFG, insofar as BFG is separate from Zope - BFG uses lots of Zope technology, which is of course a very good thing.

The Zope community is trying quite hard to shed unnecessary code and simply by subtraction, and are re-visiting many past decisions with the aim of making things easier to work with and more useful in general. Throwing out everything and starting from scratch, like BFG did, is not an option for many. It's also not necessarily a desirable path to take.

Finally, on politics - a lot of things have gotten done lately. I don't think there's evidence to suggest Zope is mired in politics - the pace of change is more likely constrained by limited volunteer resources. But, because Zope *is* relevant to many users, there are lots of stakeholders, and there is a process for ensuring people don't unilaterally make changes which are detrimental to others, at least not without some discussion first. I think that's healthy, even if zope-dev does sometime get a little long-winded. :)

Posted by Martin Aspeli at Sun Jan 03 2010 15:59

By "Martin" I meant "Martijn", of course, and not myself. :-p

Posted by Chris McDonough at Sun Jan 03 2010 20:15

As far as not being mired in politics... descriptive verbs and adjectives are hard. But recently Zope2 stopped using "the ZTK" after about ~30 messages were traded back and forth because a change was made to a "ztk.cfg" file that removed dependencies from a notional package that *hasn't yet seen a release* due to "backwards compatibility" concerns. So now "the ZTK" is this notional shared thing that is shared by exactly one system: Grok. And ironically this was the *best* outcome given the politics at hand; any alternative (like not changing the .cfg file or continuing to claim that Zope2 is a ZTK consumer) would have been worse.

I my judgment (which I know you do not share), this sort of thing is not progress: it's not "good judgment", "good process", "required discussion" nor "more grown up". It's not even "long-winded". It's just broken. And this is not really the fault of the people involved; it's just a lack of shared vision. I'm not sure the various folks involved should *need* to share a vision; at least not one implied by the current set of packages in the ZTK. Something smaller, sure. But the ZTK package set is still enormous, even without the recently removed dependencies, so shared vision is extremely difficult.

As far as Zope being relevant or irrelevant.. it's pointless to argue about the definition of relevant I guess. But we really do need to come to Jesus (or whatever messiah or non-messiah you have personally) here at some point. We need to stop pretending any version of Zope or any Zope-related system is even a blip on the radar to most people choosing a new web framework these days. It's just not and no amount of words we type to into our blog echo chamber will make it so. This has been the case since about 2005. If we can't agree on that, what can we agree on?

Zope's current relative obscurity may be a gift in disguise and incremental, careful, slow change may result in a well-factored, well-documented, reasonably sized system one of these days. But I suspect there will still be a raft of choices between better-factored, better-documented, better-sized system at that point too.

Maybe we don't care. But we need to realize that for most people it's a heck of a lot easier to just start using a different system than it is to keep up with happenings in the Zope world. It's even sometimes easier for Zope *maintainers* to do so.

Of course none of this has much to do with BFG. BFG isn't anywhere near relevance; it's even more niche and fringe than Zope. But it's got a sane set of dependencies, it's reasonably-documented, and it's Zopelike enough to feel familiar to Zope people, and Pylonslike enough to feel familiar to Pylons people. It seems like a reasonable choice to move to for a lot of people without a lot of pain.

Rather than spending another two years reinventing the publisher, begging folks for volunteer work to tease dependencies apart from each other, and writing massive amounts of documentation (or worse, not doing that), and generally trying to carve a fur coat out of the Zope beaver, another option might be to spend that two years porting the surrounding code that is important to you to something like BFG or Bobo, Pylons, or even Django. I think this is just as reasonable a path forward personally; maybe even more reasonable because the code isn't *that* hard to port, and the act of porting it makes it possible to do it right this time.

Maybe there's cultural differences or whatever, but given that it has aspirations to be lighter and have various features that other systems already have, I think it would be a mistake for Grok to not at least *consider* making some sort of move like this by evaluating the competing systems and making a rational choice about where to apply effort.

Posted by Martijn Faassen at Sun Jan 03 2010 20:34

Chris, I don't appreciate that biased summary of the recent ZTK discussion. Yes, we have passionate discussions and yes we make various mistakes. Various resolutions were also proposed that should make everybody happy enough, which makes the ZTK the smaller set with a backwards compatibility set maintained in zopeapp. We'll see how we can conclude that debate next week when more people are back from their vacations.

As to blips on people's radar and relevance, yes, there is a problem with Zope's relevance. But that doesn't mean the situation is hopeless.

I personally realized years ago that we have a problem keeping our community active and relevant. I've been working on that issue *for years*. And if I say so myself, I've helped made good progress here and there.

I know your negative evaluation of a lot of stuff helped push you to create BFG. All the better for you. But not everybody is you. I for one has been trying to *fix* the code and the process. That isn't easy and it is valuable to many people who use the code. And we're making progress. Frustrating and halting progress perhaps. Not enough progress to many, perhaps. But we're moving forward. Learning from other systems, including, for instance, your BFG.

I know you don't have the stomach for that frustration. Taking pot shots from the side lines isn't really helpful either, though.

Posted by Martijn Faassen at Sun Jan 03 2010 20:44

As to Grok considering competing systems such as BFG, a total port to a new system is a *big* deal. If not for anyone else, at the very least for me personally.

I have thought about it repeatedly nonetheless, but your potshots are certainly not helping. And I really don't want to go all "TurboGears 1 -> TurboGears 2" any time soon. Or all "Zope 2 -> Zope 3" for that matter.

Incremental progress is possible and is happening. If we can get a publisher that's at compatible enough with Zope 3's but simpler, that would be interesting. If we can adopt some of BFG's features, preferably by reusing its code but if not, by learning from BFG, all the better too. If you complain about reimplementation, I'll note that you yourself have made the choice to reimplement features that were already in Zope over and over again.

Posted by Chris McDonough at Sun Jan 03 2010 21:48

I don't consider any criticism I've made in any message here a potshot nor do I think I'm anywhere close to the sidelines. That's absurd. I'm right on the field next to you.

The way you word your responses seems to imply that you believe that there's some enormous silent majority here that you're speaking on behalf of, or maybe some poor minority you're trying to defend, or that maybe you're being truer to some existing old-guard contingent than I am. But as far as I can tell, there's "nobody here but us chickens", man. Unlike in 2002, when there were many, many cooks in the kitchen, and things were getting pretty out of control, now it's just you, me, Martin, and maybe a dozen other folks that care about *any* of this conversation (at least wrt to Grok and BFG; Zope2 and Plone are a different story). It's just you, me, and the wall. How do we bring things down to that scale? I think it would help.

If we can, we can use our shared relative obscurity as an *advantage*. In particular, it means that we become freer to have franker discussions and make larger changes than was previously possible. Deferring to existing process designed for an older world in such a scenario is a liability; it's just not necessary. It's like building a big city for ten people up front. It's wasting an advantage. Some code in the package set that makes up the ZTK doesn't deserve to be fixed; it instead deserves to die.

Over the last year, I've made a *lot* of checkins to packages that make up the ZTK to reduce dependencies, and I've participated in meaningful ways in all reasonable discussions on zope-dev about "big issues." I've put in a lot of effort towards trying to reuse Zope and Grok technologies, as well as trying to consolidate Zope technologies into the larger Python world. In both a Zope context and a BFG context I've spent a lot of time deconstructing dependencies, trying to fix design bugs, and generally thinking pretty hard about how to solve real problems that Zope has. All of this, I think has been pretty successful. I believe I have done *at least* as much as any other person in the Zope community to make Zope technologies consumable to a wider audience. I don't believe any of the stuff I've done has suffered from a lack of "the process"; you can check by asking any early adopter of BFG or repoze.who if they've had any meaniniful problems with backwards compatibility or low quality.

You seem to discount this form of progress because a lot of it took place outside "the process". Fair enough. It's still progress and I'm proud of it. I'd like to somehow not always be "fighting the process", but until that becomes easier, it seems like we're doomed to be on separate sides of the field. Is there any way to fix this?

Posted by Martijn Faassen at Mon Jan 04 2010 01:35

I fail to see how you're reading any idea of a "silent majority" into my statements. I know full well it isn't 2002. You're not the only one who is aware enough of the status quo to challenge it. I just do it in a different way than you do.

That doesn't make my approach wrong and useless. Neither does it make your approach wrong and useless. I have the sense I'm getting a lot of criticism about my approach. I'm telling you again: I'm seeing the things you are seeing. I am not blind, nor an idiot. I am just responding differently.

Concerning killing code in the ZTK, what in the world do you think we've been trying to do in the last year? To let a *lot* of code that was in Zope 3 die a graceful death. (we had a recent disagreement as to how we should kill it off, that's all). There's a tremendous amount of code that can go into deep freeze maintenance mode now. Thanks for your help by the way - I do appreciate it.

I am not discounting your work on Zope or BFG at all. You've done great work with BFG and with Zope. You've worked within the process and circumvented the process as well, to build a new better process. I applaud this - after all I've been trying to do the same with Grok and Zope. One big thing about the Grok project was all *about* circumventing the existing (lack of/broken) process.

Concerning opportunities because we're a smaller community: yes, you're right. We're a small community and can make a lot of progress. We can quibble about the details - I do know of a lot of code based on Zope (3) or Grok, but in the overall analysis we agree.

As to pot shots, I'll withdraw that remark. Sorry. But I will try to explain briefly what prompted me to say this. Hacking BFG is a luxurious position to be in for a developer in some ways. You can make a fresh start. You don't have to worry too much about a lot of stuff (code and politics and compatibility) that has has to worry about if you hack on, say, Grok. You can move things along with a lot of momentum. And I then I get the impression that you're criticizing Grok, Zope politics I'm involved in (it's painful enough as it is), that we should eliminate code to gain simplicity, and port over Grok to BFG, etc. Since I've been trying to fix a lot of these things, it gets a bit frustrating to keep hearing that. Call me jealous if you will.

I know you're just intending to have a dialog, and of course between it all I'm still learning a lot from you, but I'm worried enough about it all enough as it is, okay? I try not to do that with BFG. I realize though that probably from your perspective I do: it did and does bother me that BFG reimplements so much apart from Zope, but since I see what that strategy delivers I must acknowledge it's been working out pretty well. I'm trying to figure out ways to feed that back into Zope.

As to being on separate sides of the field: I'm not trying to be. I'm trying to learn from you and work with you where I can. I dearly wish we could use more of your code in Grok, and perhaps over the course of 2010 we will find out how.

I actually agree with you on many of these things, I share many of your frustrations, and I'm trying to solve many of the same problems as you do. So I think we'll just be fine. We've violently, frustratingly agreed about things in the past.

Posted by Chris McDonough at Mon Jan 04 2010 03:13

Martijn and I moved this to email, where it probably belonged.

Posted by Martin Aspeli at Mon Jan 04 2010 03:40

Chris,

I'd just say one thing, since it seems to come up a few times: Whenever a mailing list discussion goes above ten posts and people start resisting change, you seem to get very frustrated. I think you have a tendency to lose the energy to make your point (understandable!), and then assume that all is lost.

Don't.

In virtually all the discussions that have gone this way before, eventually the sane option has prevailed. The Zope 2 issue you mentioned above is an example, I think. Hanno was a bit premature to just unilaterally remove a bunch of packages which people may depend on. The namespace (which was probably at least somewhat arbitrarily chosen before) isn't the only determinant, and we can't just make changes without letting people at least have a say on how it may affect them. Zope-dev is now a place where representatives from many other projects are trying to collaborate, and the "frameworks" that matter are Grok, Zope 2, Plone, BFG (and possibly Zope 3). That means that invasive changes have a lot of stakeholders with a lot at stake. I think that's healthy, even if it means we sometimes get very long discussions about core things that touch all of us.

Zope 2 will use the ZTK. A very quick decision needed a few breaks put on it. No-one actually disagreed that removing these packages was desirable, but it's also a question of timing. That is the lack of "luxury" that Martijn is talking about. There are BBB concerns here, and I'm sure you'd be the first one to agree that that is both hard and important to get right.

It's been the same with most other things. I recall discussions about martian reading .pyc files, for example. That was fixed, without the need for a fork, even if some people had (valid) points to make about it. And in some other discussions, we've found that the original idea wasn't necessarily good (such as turning the component registry into a threadlocal dict, which at least would cause confusion in some places).

My advice would be - instead of giving up, just wait. People with the requisite energy will have the debate and a resolution normally emerges in a day or two. People like Martijn are also good at summarising decisions if there needs to be a vote. Read the summary emails, and in the meantime, have fun hacking something else. :)

And as for there only being a very small bunch of people using these technologies: look at number of people who've chimed into very important discussions on Zope-dev in the last few months. If there were only the three of us, well, then I'd take your point. But there are at least a dozen people - probably more - who make core contributions. Most open source projects don't have a dozen qualified core contributors.

Martin

Posted by Chris McDonough at Mon Jan 04 2010 05:28

Point taken. Although, really, let the people who care defend themselves.

I never get frustrated by conversations with people who have a valid, immediate, non-theoretical personal stake in a change being made. I work pretty well in that sort of conversation.

I do get frustrated, however, by conversations where people add stop energy purportedly for the sake some theoretical group of people without having any conceivable connection to that group of people. I don't work very well in that sort of conversation.

While I'm constitutionally, politically, sort of a Democrat (in the American political sense), I am a definitely a Republican when it comes to software development: I say fend for *yourself* instead of choosing some group to represent.

Posted by Martijn Faassen at Mon Jan 04 2010 08:48

I think Martin is someone who cares and therefore is defending himself as well.

Since you said before that you see me as am speaking for some silent majority, I'm not sure how to take your comment concerning stop energy for theoretical groups of people. The recent ZTK discussions were very much about a concrete problem I have upgrading my own apps and more abstractly, those of other Grok users (who can't be expected to maintain a zopeapp list of versions by themselves).

Posted by Chris McDonough at Mon Jan 04 2010 11:24

When I made that comment, I was thinking more of that insane conversation on grok-dev about grokking .pyc files actually.

But yes, in general, I think it would be *extremely* helpful if we brought conversations down to a level where it was "me get broke when you do X, me no like X" than it is to evoke "the process", even if it feels selfish to the person making the objection, because usually there's some reasonable solution to the problem. It's extremely frustrating to argue against "the process" rather than argue against concrete practical concerns.

[Main]

Unless otherwise noted, all content licensed by Martijn Faassen
under a Creative Commons License.