It seems to be a recent trend to point out things you don't like about doctests. There are two articles by Andrew (? - see update below) and one by Ned Batchelder. There's also one by Marius Gedminas. I take the doctest negativity as a sign of increased popularity of
doctesting in the Python world. Doctests are now being seen and read by a larger amount of Python programmers, so there are now more people to talk about the undoubted drawbacks of doctests. (Of course it is also a sign of people disliking aspects of doctests - Marius for one has been exposed to narrative doctests for years) I like narrative doctests, and have been using them for years now. They often constitute the bulk of the tests of my code. To Andrew at least, that probably means I "abuse" them. Why do I like them? Doctests are not an ideal testing tool. There are pros and cons. Read
the linked articles for some cons (and pros: Andrew points out they are easy to write). Narrative doctests aren't an ideal form of developer documentation either: a well-written, well-maintained dedicated text is better. The great thing about doctests is that you can write fair tests and fair developer documentation, at the same time. You can use doctests to provide reasonable test coverage suitable for solid, real-world code. Importantly, those same doctests then also provide developer-level documentation that may not always be great, but is still much better than the frequent alternative (nothing). One advantage of using doctests to describe your API is that you use the API of your code in the doctest before you actually
use it for real. As a result, the API of the code you write becomes
better as you are forced to think about it early on during the design
process. Unit tests of course have the same benefit: improved API
design is actually one of the great but rather underacknowledged
benefits of unit testing. But doctests encourage you to think about your API design more than plain unit tests, as you're actually writing prose that tries to explain the way your API works to the reader. If it's hard to explain, it may be time to change the design. Doctests often contain usage examples. Unit tests do too, but doc
tests have a narrative around them, including the often all-important
setup code. Instead of digging around to see which objects you're
supposed to create and what methods you're supposed to call in what order, you have a narrative in the doctest that tells you what to do. Another advantage of doctests is that they spell out the intent of
the tests better than a typical unit test suite does. An individual unit test can of course describe its intent by being well-named and by having comments, but nothing encourages you to do so. Doctests have an actual narrative, so this style of testing actively encourages writing down the intent. Here are some examples of narrative doctests I've written over the years. I don't find it particularly hard to work with doctest. In all cases below they form the bulk of the tests in the codebase. Is this abuse of the doctest format? Judge for yourself whether you like narrative doctests or not: Narrative doctests are not an ideal tool; no tool is. You have to actually write a narrative and if you don't, you are left with a testing tool that is in many respects worse than unit tests. I maintain that doctesting is an approach that's good enough to write good, solid software with reasonable developer-level documentation. You can enhance the reuse potential and API design of your library by writing a narrative doctest for it. Feel free to use unittest and doctest where appropriate, to your taste. But don't be scared off by the recent negativity that seems to surround doctests. Doctests have many benefits. Doctests are a good balance for me personally, and perhaps they will be for you as well. [update: I don't know why I gave Andrew a last name; I'm not sure where I got that from so I'll take it away again.]
![[Comments]](/blog-content//themes/Default/comments.png "17 comments")
(17) Mon Dec 01 2008 09:28 UTC0UTC I like doctests:
- Comments:
Posted by Philipp von Weitershausen at Mon Dec 01 2008 11:14
Well said.One should also point out that the mere usage of doctests doesn't make good narratives. If all you write "Test this" and "Now test that" in between code blocks, it's not a good story. And if the code blocks expose the API in an unnatural way (because you have the test in mind), doctests aren't good developer docs. In those cases, it would probably be more helpful to resort to a regular unit test.
Posted by Lennart Regebro at Mon Dec 01 2008 11:22
You already know what I think: Doctests are great, for testing documentation. :) And every module should have a README which is a doctest with an introduction of how to use the module.But the difficulty of debugging them means they are difficult to write and maintain. So for testing functionality I vastly prefer standard unit tests.
Posted by Martijn Faassen at Mon Dec 01 2008 11:29
Lennart: I evidently don't find it as difficult to write or debug doctests as you do. I imagine it's a matter of personal preference.
Posted by Martin Aspeli at Mon Dec 01 2008 12:22
A few observations: - Doctests help me think about my code and its API, and probably makes the API better. - Narrative helps me structure the testing. - Writing good narrative is probably as hard as writing good documentation in general (maybe even harder). The benefit, as you say, is more in terms of the way that the doctests "trick" developers into providing documentation and tests. In the ideal world, we'd have hundreds of unit tests, and lots of documentation, and a few doctests for everything. We don't live in an ideal world, though. :) - Doctests are easy to write for self-contained code that requires minimal surrounding infrastructure or test setup. - Doctests are hard to write when there's lots of setup or poorly defined APIs between the various modules. In this case, mock tests or integration tests may be more applicable.The last point is worth bearing in mind. I find it easy to write a doctest for a small, re-usable, Zope3-like module. For higher level (Plone) components that rely on a lot of framework (e.g. Archetypes), they are a lot more awkward.Martin
Posted by Peter Bengtsson at Mon Dec 01 2008 14:02
In Django I use doctests to test/narrate the models but when I write unit tests to test the views and other more complex data related ops.
A healthy mix. The right tool for the right job; ...or whatever the expression is.Posted by Chris McDonough at Mon Dec 01 2008 14:22
While having explanatory doctests is way better than no docs at all, it's never quite as good as actual narrative documentation because it needs to concern itself with the minutae of testing corner cases and invoking setup hair, etc. At least it does if it's also good tests. Thankfully, with Sphinx, and a bit of commitment, creating good developer documentation instead of pointing folks at tests is becoming a lot more reasonable. I suspect that the initial lack of documentation tools during Zope's growing-up phase has influenced Zope developers' thoughts about documentation a lot. Zope never had a strong documentation culture. This is understandable: in 2002 or even in 2004, writing "pretty" docs was a real chore. Writing doctests widely culturally excused Zope developers from needing to do that task. These days, the excuse is becoming a bit thin, at least if you want your software to compete for limited mindshare in a market where good documentation is becoming de-rigeur.
Posted by Martijn Faassen at Mon Dec 01 2008 14:32
I agree with most of what you say, though I must add though that I don't think Zope has a weak documentation culture as of the last 3 to 4 years compared to most other Python-based projects out there. There are a number of projects which have way better documentation than Zope 3, but on the other hand there are any number of projects that have very much worse documentation. I see doctests as a step forward, not as an excuse.We want to be where the projects with *great* documentation are though, and with the Grok project (and also with the Repoze project from what I've seen) we're trying to get there. Grok's got Sphinx docs too by the way (http://grok.zope.org/doc/current/).I think there's a huge niche for narrative doctests. There are a lot of useful libraries that aren't big enough to become projects all by themselves (though perhaps they can be as part of a larger project). Development on these libraries is intermittent. If you write such a library and at least want *some* chance people will start reusing it, and you're in a relative hurry (developers often are), a narrative doctest is a way to get to reasonable documentation without a large investment in documentation. Hopefully the users coming in can help bootstrap the process to better documentation. :)
Posted by Albertas Agejevas at Mon Dec 01 2008 15:49
You have to agree doctests are a compromise between tests and docs. These are conflicting targets. But you can use the technology for either, or at least shift the compromise significantly in the direction you want.Usually, I am a lot more interested in solid test coverage than in documentation. In fact, I prefer doctests to unittests as a unit test format. The short 'do this', 'test that', 'now we have to do foo in order to get bar' explanations make tests a lot easier to orient and read, and the failure reporting usually is a lot more informative than in unittest.I've seen doctests that have been biased to either part of the spectrum, but I notice that the middle-of-the road doctests often leave me unsatisfied both when I'm looking for a test to amend, and for the docs on "what is this, why, and how do I use it?"
Posted by Albertas Agejevas at Mon Dec 01 2008 15:53
Maybe it's because often when I need a girtty unittesty doctest I find a fluid narrative and additional edge case I want to add would spoil it, and when I need a gentle introduction I find the gritty edge-case ridden test.
Posted by Ian Bicking at Mon Dec 01 2008 16:41
Someone just needs to fork doctest, fix the things that we mostly all agree are broken, and then people can start using that. There are small but persistent problems which make it a less pleasant experience than it could be. Most of them have nothing to do with the core of what doctest is.
Posted by Paddy3118 at Mon Dec 01 2008 17:47
I am with Ian on his suggestion
"fork doctest, fix the things that we mostly all agree are broken, and then people can start using that"But to work out what is broken I think that you should remember that doctest is not primarily a quick way to write xUnit test cases (as you have found). It is chiefly a way to re-use command-line testing as tests by capturing command-line sessions as comments/docstrings/text files. It also excels at being an easy way to get people to write tests of any sort.Keeping the above in mind I think further development could keep doctests strong points whilst addressing its current short-commings, but I don't think it should be turned into yet another xUnit clone.It seems from a quick look around that improvements such as ways to abort testing the current file/docstring/everything on a failure; Better reporting of where errors occur; the ability to name a doctest/doctests; improvements to debugging doctests; and addressing some of the comparison loopholes are needed. But can it be done without loosing the spirit of doctest?Posted by Chris McDonough at Mon Dec 01 2008 17:55
There's almost no reason to start a project (even a small one) without accompanying Sphinx docs these days. It's just not a big deal to create them.While I appreciate the existence of doctests when they are the only form of documentation in a package, I sometimes see them as a barrier to getting to a place where creating good narrative and API docs is an *expected* part of development, because folks tend write the doctests and declare early victory on both the testing and documentation fronts, when in reality, the doctests as documentation are extremely hard to understand (see the zc.buildout docs), and the tests aren't testing all the edge cases (see any doctest written just to provide documentation). While this is of course a matter of degrees (the doctests you wrote and pointed at are quite good), it sure would be nice to have some uniformity culturally. I don't see that uniformity coming from doctests.
Posted by Half-baked Programmer at Mon Dec 01 2008 19:39
If it weren't for doctests I wouldn't be writing tests at all. In my tests, the coverage is poor and many edge cases are ignored until they come up in production. Still, just with that minimal effort: refactoring has become almost trivial; the code has become much better structured, with much better encapsulation, and much easier to understand; the actual development is faster since I started driving the coding from the tests.For an electrical engineer who has to pose as a half-baked programmer to get on with his job, doctests are a godsend.
Posted by Paddy3118 at Tue Dec 02 2008 11:34
I wonder if their are a lot more, but non-vocal people like "Half-baked programmer" out there for which its a case of doctests or nothing?- Paddy.
Posted by Jeff Shell at Tue Dec 02 2008 15:29
Doctests are better than no tests, and are better than most unit tests, but they are definitely no substitute for documentation. 'hurry.workflow' is a big case in point: I wanted to use it in an app, recently, but ultimately ended up writing my own much-lighter-weight tool. The docs (which were all doctest) didn't tell me anything about why I actually needed versioned workflow, or what utilities / components I would need to flesh out myself in order to actually use the system. That's where I fell down and had to back away given my time constraints.(Reason why there are "so many web frameworks" for Python: most Python projects tend to have such horrible documentation that it's easier to write your own tool than to read the docs for one that already exists. The 'doctest as documentation' syndrome is NOT helping this).Also: I love Buildout. It has been a life saving tool. But it took me more than half a year of fighting with its gigantic doctest-based docs and those of core recipes to understand it enough to actually use it. But I recognize that if it weren't for the doctests, there probably wouldn't be up-to-date documentation for it at all.As a testing tool, I don't mind them. The narrative structure one can put around them can be conducive to thinking through a problem (most beneficial in test-first design). But at other times, the narrative structure is so phony as to be almost laughable, and seldom is there any additional narrative about how one might do things in real integration-based usage.
Posted by Martijn Faassen at Tue Dec 02 2008 16:20
You're right about hurry.workflow: these were one of the first doctests I wrote and I was in a hurry. :) These aren't the best documentation in describing how to set up things. I will look into expanding the text at some point. I think I've gotten better at describing the context over time (see hurry.resource).As to the buildout doctests: well-known example of doctests being overwhelming and not a good introduction at all. I think doctests function better as component documentation than as application documentation.
Posted by Noah Gift at Thu Dec 04 2008 11:13
I am also a "closet" doctest lover. One of the main reasons is what you pointed out. Doing some basic testing and documentation of an API in one step is pretty cool.
