Much of the confusion stems I suspect from the Agile manifesto which has the following value statement: "Working software over comprehensive documentation". I think a number of people have taken this to mean, agile does not do documentation. I think that is a misunderstanding of the intent of the agile manifesto. The manifesto is not a methodology, it is a statement of the common principles which underlie the differing agile methodologies, of what it should require for a methodology to call itself agile. Martin Fowler has useful backstory on this. So the agile manifesto is what XP, Crystal, DSDM et al. have in common, not an agile methodology in itself.
To understand the the admonition it must be remembered that the non-agile methodologies were often characterized as document-driven development because they required the production of innumerable documents before a line of code could be written. In many cases the process steps were being performed by software development teams simply because the methodology told them that they had to be performed, even when they provided little value. As a reaction many development teams abandoned methodologies altogether. Agile was an attempt to remove the production of documentation without purpose, and focus once again on the key artifact of the software process: the code. Unfortunately many people who have never practiced a full on waterfall process like SSADM, do not recognize what is being rejected, and thus assume the limited documentation of ad-hoc or in-house processes was being targeted too.
In addition many development shops mistakenly equated agile with no methodology, and where they had no methodology began to self-label as agile as a form of justification for their lack of process. The inevitable failures gave agile a bad name in many organizations who never realized that Agile was actually a set of methodologies, all of which required discipline and adherence to practices. As a result though, many people equated the poor documentation practices of these ad-hoc processes with agile.
Crystal considers development to be a series of co-operative games, and the provision of documentation is intended to be enough to help the next win at the next game. The work products for Crystal include use cases, risk list, iteration plan, core domain models, and design notes to inform on choices. Crystal also defines the roles for these. Note however that there are no templates for these documents and descriptions are necessarily vague, but the objective is clear, just enough documentation for the next game. I always tend to characterize this to my team as: what would you want to know if you joined the team tomorrow.
As an example:
The common domain model is created by the designer-programmers as they understand the domain and incorporate increasing amounts of it into their design. It is reviewed by the business expert, who may help to construct it in the first place."
XP by contrast notes that documentation is not needed so much inside the team, as outside, and sees it as a costed story that needs to be delivered. The goal here is, I suspect, to reduce unnecessary documentation by forcing it to be evaluated alongside other feature sets. XP relies more heavily on programmer-to-programmer communication to distribute knowledge about the system than written documents, but mandates pair-programming to make this possible: because you share understanding with others through pairing, there are no ‘dark areas’, and less need for people to document what has happened. In addition the code and tests are seen as then primary repository of detail on how the software was implemented, for fear that documents always grow out-of-date and the natural inclination is to ‘use the source Luke’. Beck states that apart from code and tests you should ‘rely on social mechanisms to keep alive important history of the project’. Beck is unclear what these ‘social mechanisms’ are, but many projects I have worked on have used tools like wikis that create collaborative, constantly-edited, documents as repositories of history as one non-verbal, social mechanism, for recording project history. Also see this page for more on XP and documentation, but the summary is whatever the team needs or the customer wants.
As Crystal does not mandate pair-programming it needs to compensate by mandating a level of documentation to preserve project history. Crystal sees documentation as one of the work products and provides a list of what documents it would expect. Again though the documentation is purposeful – to ensure the next game is successful – and vague enough to encourage an only what you need for this project approach.
Personally, I am more aligned with the Crystal approach right now, as we do side-by-side programming, not pairing, so we need to preserve knowledge in writing. In addition, our structure requires handover of projects at times, so we need some documentation to effect that. I do have some warm feeling toward the idea of capturing history via wikis or even mailing lists so that we capture living information.
I have not used DSDM, though I worked for an organization that flirted with the idea in the ’90’s, but a review of the Wikipedia entry suggests that it has a significant number of artefacts produced throughout the development lifecycle.
The important thing to recognize here is that there agile covers a number of distinct methodologies, which while sharing common principles, differ in detail. You should consider adopting a methodology (especially if you have no experience with agile methodologies and thus lack the experience to generate your own in-house alternative), and choose a model which matches your needs. Documentation requirements are just one part of that decision, but the desire to have documents need not be at odds with adopting an agile development methodology.