Shop Class as Soulcraft: An Inquiry into the Value of Work
– Matthew B. Crawford

Crawford speaks eloquently about the value people feel in work that is tangible. He is primarily extolling the joy we feel in applying the manual arts well, and suggests that we should value those skills more highly.

He points out that a craft like plumbing is not subject to off-shoring, because it must be done on site. The Agile software community, of course, understands that working on site is far more effective for us, though it is not logically necessary in quite the way that plumbing is.

But there’s more for us in this book besides a good read. People do get joy and work harder, when their work is tangible. This has implications for how a developer can work toward greater joy, by focusing on delivering real features that people want.

There are implications for management as well. Because people are motivated to see their work come to fruition, we’re likely to get better results with feature-focused teams rather than teams divided along other lines, such as infrastructure levels.

This book is a quick and enjoyable read and it will give you valuable ideas.Give it a look!

Pragmatic Ajax

Justin Gehtland, Ben Galbraith, Dion Almaer programming

Ajaxian Maps

The book starts off strong. After a brief introduction, the authors take us through an implementation of a mapping application that’s very much like Google Maps, the web app that put “Web 2.0″ on the, um, pardon me, map.

I particularly like, in this part, that they show how the tiling geometry works. Folks always think that tiling and graphical ports and windows and such are difficult. Yet, approached cleanly, they are not, and Pragmatic Ajax gives a good example here of how straightforward such things can be.

By page 37, you’ve implemented a sweet little map application, that can be seen here. Neat.

The book moves right along, adding the zoom tool and push pins. They even show you what to do about the inevitable fact that Internet Explorer doesn’t support things like transparency correctly. With all the money in the world, why can’t they fix that stuff? But I digress …

The book shows the code you need, and of course the code is also available on the pragprog website. Delicious. These guys do good work.

Filling a Form

We move from the sublime to the practical, making fields on a form fill in without refreshing the whole page. Again, IE doesn’t correctly support XMLHttpRequest. Maybe next service pack. <sigh/>.

This chapter is putting off telling us how the server processes our calls to it, but takes us through things that I badly need to understand, like the way that state changes come back and get handled in our JavaScript.

But Wait! There’s More …

The book provides a brief but valuable explanation of some of JavaScript’s nicities. It briefly describes the Document Object Model that underlies modern browsers — a short and clear description that I really liked. I don’t like the DOM, mind you, but I do like the writeup.

Then the authors give us a look at some of the frameworks and toolkits that are available for Ajax. They introduce us to some of the UI libraries, and they give us some good simple advice about keeping our new-found power under the user’s control, not just our own.

They tell us about JSON (JavaScript Object Notation) and its RPC counterpart. They tell us about server-side integration using PHP, Rails, Spring, and ASP.NET. Best of all, they port their standard application (a Customer Relationship Manager) to each of those systems.

The book closes with a look at the future of Ajax as these authors see it, describing the <canvas> element, and Scalable Vector Graphics (SVG). They even provide a little page that displays a bar chart using <canvas>, and in SVG. They also point to other interesting canvas-based and SVG-based sites.

Bottom Line

I like this book. I don’t know whether, much less when, I might convert this site to use Ajax. Probably I’ll encounter clients who are using it before I do much more than an experiment here at home.

Justin Gehtland, Ben Galbraith, and Dion Almaer have done a very good job with this book. It’s well written, interesting, and best of all it includes some simple but powerful examples that can get you over that hurdle of putting up your first Ajax pages. Like all the PragProg books, it is attractive to read and well laid out.

If I had written this book, I’d be quite proud of it, and Justin, Ben, and Dion should be. Well done!

Refactoring Databases

Scott Ambler, Pramod Sadalage

In my consulting, I often run across clients who are getting pretty good at code refactoring, but who are hampered by the need to live with a database structure that is getting old and crufty. Sometimes I can give them some useful advice, since I have a lot of database work in my background, but I’ve never worked in a situation where a database needed refactoring, so mostly what I have been able to do is to cajole them and encourage them to figure it out.

Scott and Pramod have done a lot of the figuring out for us. They provide about 70 named refactorings, like Drop Column, Introduce Soft Delete, or Replace View with Method(s). Each refactoring includes a section on Motivation (why we might want to do this), Potential Tradeoffs, Update Mechanics, Data-Migration Mechanics, and Access Program Mechanics.

The latter two are particularly helpful, in my opinion. I’ve noticed that many database refactorings are deferred indefinitely because folks can’t figure out how to do them without breaking existing programs. Data Migration Mechanics addresses how to make the change in two steps, first putting in the new feature and making it available, while leaving legacy code running, and then later removing the legacy support after all programs have been updated. Some of the migration examples are quite elegant, and will help get you in the right frame of mind.

Access Program Mechanics sections address the changes you’ll need to make to the programs that access the data, pointing out various aspects that you’ll need to be concerned about.

If your project uses database technology, whether it’s a new project or an existing one, this book can help you become much more Agile with respect to database changes. I like the book, and I think you will also.

A Short History of Nearly Everything

Bill Bryson

If you’re interested in the history of science at all, this is one of the most enjoyable books on the subject that I have ever read. Bryson’s history and science is solid, at least as far as this layman can tell, covering both the history of scientific endeavor and discoveries, and the often very bizarre people who were involved.

Bryson’s writing is delightful. Some examples:

Looking for the unknown isn’t simply a matter of traveling to remove or distant places, however. In his book Life: An Unauthorized Biography, Richart Fortey notes how one ancient bacterium was found on the wall of a country pub “where men had urinated for generations” — a discovery that would seem to involve rare amounts of luck and devotion and possibly some other quality not specified.

Or this …

In the most literal way, cells also vary in liveliness. Your skin cells are all dead. It’s a somewhat galling notion to reflect that every inch of your surface is deceased. If you are an average-sized adult you are lugging around about five pounds of dead skin, of which several billion tiny fragments are sloughed off each day. Run a finger along a dusty shelf and you are drawing a pattern very largely in old skin.

Most living cells seldom last more than a month or so, but there are some notable exceptions. Liver cells can survive for years, though the components within them may be renewed every few days. Brain cells last as long as you do. You are issued a hundred billion or so at birth, and that is all you are ever going to get. It has been estimated that you lose five hundred of them an hour, so if you have any serious thinking to do there really isn’t a moment to waste.

Or this …

Bouyed by the success of leaded gasoline [which he had invented], Midgley now turned to another technological problem of the age. Refrigerators in the 1920s were often appallingly risky because they used dangerous gases that sometimes leaked. One leak from a refrigerator in Cleveland, Ohio, in 1929 killed more than a hundred people. Midgley set out to create a gas that was stable, nonflammable, noncorrosive, and safe to breathe. With an instinct for the regrettable that was almost uncanny, he invennted chloroflourocarbons, or CFCs.

Seldom has an industrial product been more swiftly or unfortunately embraced. CFCs went into production in the early 1930s and found a thousand applications in everything from car air conditioners to deodorant sprays before it was noticed, half a century later, that they were devouring the ozone in the atmosphere. As you will be aware, this was not a good thing….

Midgley never knew this because he died long before anyone realized how destructive CFCs were. His death was itself memorably unusual. After becoming crippled with polio, Midley invented a contraption involving a series of motorized pulleys that automatically raised or turned him in bed. In 1933, he became entangled in the cords as the machine went into action and was strangled.

Or this …

Perhaps nothing speaks more vividly for the strangeness of the times than the fate of the lovely little Bachman’s warbler. A native of the southern United States, the warbler was famous for its unusually thrilling song, but its population numbers, never robust, gradually dwindled until by the 1930s the warbler vanished altogether and went unseen for many years. Then in 1939, by happy coincidence two separate birding enthusiasts, in widely separated locations, came across lone survivors just two days apart. They both shot the birds, and that was the last that was ever seen of Bachman’s warblers. …

I mention all this to make the point that if you were designing an organism to look after life in our lonely cosmos, to monitor where it is going and keep a record of where it has been, you wouldn’t choose human beings for the job.

But here’s an extremely salient point: we have been chosen, by fate or Providence or whatever you wish to call it. As far as we can tell, we are the best there is. We may be all there is. It’s an unnerving thought that we may be the living universe’s supreme achievment and its worst nightmare simultaneously.

I truly enjoyed this book. Were it not for the author’s penchant for reminding use that we are overdue for a massive volcano eruption that will freeze the earth to a ball of ice, a meteor strike that will vaporize us at 60,000 degrees, or a disease that will carry us all off, it would be full of nothing but charm. As it stands, now I have to decide whether to save my money or get another BMW. Do you see a meteor coming from where you are?

Leadership and Self-Deception

The Arbinger Institute

I can’t really tell you what this book is about: the book itself will have to do that. Let me tell you what the book made me think about.

It made me think about the times when I have handled a situation badly, had interactions with people where I was trying to lead but somehow stepped on my own foot — or tongue? — and failed to make the situation better. Often I have made the situation worse, though fortunately people of good will often recover from such things.

The premise of the book is that we mess these things up, not from a lack of skill or “technique” but because we are deceiving ourselves about what’s going on, always in the same way. That may sound overly simplistic: could all these different problems really be due to a common cause? Still, I found the ideas to be quite compelling and certainly thought-provoking. Often I could look back on situations and see clearly that in at least that case, the book had me pegged perfectly.

The book is like a short novel or narration, about a new employee, Tom, and his interactions with his boss and others, talking about the company’s foundation for success, the recognition and elimination of self-betrayal and self-deception. It’s a very easy read, and if you let it, it will make you think. It’s not all smarmy slimy like some self-help things, nor is it cheerfully impractical like others. Still, it is offering some challenging ideas that deserve consideration.

You can check the Arbinger Institute Web site for more information and a look at a few chapters. The book has my very real recommendation. Thanks to Mark Graybill for calling it to my attention.

It is dangerous to read for anyone who does not understand what XP is, because it goes to such pains not to understand, to take out of context, and to distort.

Extreme Programming Refactored

Matt Stephens, Doug Rosenberg fiction

In its attempt to be a book full of satire while at the same time making good points regarding XP, Extreme Programming Refactored undermines itself. The book (I’ll call it XPR henceforth) makes some very good points about the value of some XP practices, and offers some good ideas about ways to improve it in situations where the practices cannot be done, will not be done, or are not sufficient. However, owing to the density of satirical distortions, quotes out of context, misinterpretations, and putatively amusing songs, it is almost impossible to find the good bits amidst the bad, even for someone who already understands XP (assuming that this reviewer does). For someone not already intimately familiar with XP, I believe it would be impossible.

A book satirizing XP might have been amusing to some people, but would have been informative to none. A book addressing XP’s limitations and how to deal with them would have been valuable to a wide range of people and might even have been made entertaining as a side effect. In this book, the intricate mixture of satire, error, and somewhat good advice is unmanageable by anyone who doesn’t already know the material, and largely unnecessary for anyone who does.

A Few Examples

Indented sections below are quotations from the book.

Doug once worked with a guy who had a technique that he called “Samurai debugging (this was back around 1981/1982). When you follow the Samurai debugging technique, you start with a blank screen. That’s not what you want, so you start debugging it, and you continue debugging it until your program does exactly what you want it to do.

It’s similar to Michelangelo’s “All you do is start with a block of marble and chip away everything that doesn’t look like David,” but it’s also not so unlike XP today.

Except entirely. XP is not about debugging — if anything it is about never needing to debug. XP isn’t about chipping away bugs, it is about synthesizing simple tests and code that does part of what we need, and building it up incrementally to come up with a more and more refined, well-designed, solution to the problem at hand.

This pattern, you’ll see, is typical. XPR presents a straw-man argument about what XP is like and then demolishes the straw man. The fact that XP is not like that is lost or distorted in aid of “satire”.

C3 project was floundering. So Chrysler brought in Kent Beck, who in turn brought in Ron Jeffries, who in turn brought in his crack team of XPers (except then, of course, they weren’t known as “XPers”.

Except not really. Kent did bring me in. I brought in no one. The project was done, for at least the first year, with members of the existing C3 team, not with new people. After that, a few new people were brought in. We built a team of XPers: we didn’t bring one in.

Much of the book relies on demolishing the reputation of the C3 project through mockery and satire. Unfortunately, much of the discussion is as poorly researched as this rather fundamental error suggests. Public and private offers to assist the authors with the facts of history and of XP were rebuffed, perhaps to the detriment of the book.

Normally it is the refactorer’s responsibility to change all the code that uses the interface that he has changed; therefore, all integrated code must be fully working. However, this does have the side effect … that code that was previously thought to be completed suddenly needs to be rewritten, possibly even redesigned, because a separate part of the system has been refactored. As we discussed earlier, many of the XP refactorings involve interface changes, so this is quite a high risk.

It is quite rare for a refactoring to require substantial “rewriting”, much less “redesigning” existing code. Changes induced by refactoring are largely lexical in nature rather than more deep or complex; they tend to be small and simple rather than tricky. The paragraph above suggests that there is often a lot of rework, of “doing over” when we use incremental design and refactoring. Based on my own experience and the reported experience of many other practitioners, this is simply not the case.

No long after Matt uploaded the first version of “The Case Against Extreme Programming” article onto his Web site, Ron Jeffries (in a rebuttal placed on the Agile Modeling mailing list) suggested that the author fears change and wants to suppress it. Then he added, “He’s afraid of integration.” Why fears? Why not “is wary of change,” “prefers a changeless approach,” “approaches change with caution”?

It wouldn’t be nearly as emotive. When “discussing” criticism of XP, XPers frequently shoot the messenger.

This observation isn’t entirely unfair. In informal web discourse, tempers get frayed, and often we do convert a bad idea into a bad person, and this is wrong. It happens to many people: falling into the same trap, Extreme Programming Refactored shoots the messenger with its own emotive distortions labelled as satire. Unfortunately, in this case, the messenger is the book itself.

This is another circular argument behind XP. XP is supposedly suited to a project with vague requirements, but the requirements are vague (not written down in detail and agreed upon by all project stakeholders) because you’re using XP.

Well, no. Actually, the requirements are not written down because they are vague, not the other way around. Instead, they are discussed, just as they would be in any reasonable process hoping to reach consensus and understanding. In XP, the difference is that the requirements are then committed, not to ambiguous text, but to far less ambiguous tests and code.

Things to Like

The book is not without content of merit, though unfortunately it is very difficult for a naive reader to tell what has merit and what does not. Some things that I like include these:

• David van der Klauw has written a very interesting short article on pair programming, addressing the return on investment from pairing in different kinds of work. I would like to see that article published somewhere on its own.
• The authors often point out the benefits of XP practices and values. That XPR then often goes on to distort the practices and values, and to present funny songs about them does not lessen the fact that the authors have clearly tried to include some good things to say when they had them.
• The authors raise one of the most important issues with XP, namely that it won’t work if you don’t do it. As far as I know, no process will work if you don’t do it, including the authors’ own favorite. While XPR is quick to point out, and rightly, that a so-called XP project will go off the rails if the team doesn’t do XP, the book does not address the same issue for other proposed processes and practices.
• The authors address the question of scalability fairly well, pointing out the difficulties which would arise if you tried to do XP with a 50 person team. We all agree that adaptations would have to be made for a project of that size, and it is certainly true that XP was devised with smaller teams in mind. If the conclusion of the book had been “don’t do XP with 50 people”, XPR would have made a fairly interesting case, albeit one that both experience and reason might argue with.

Conclusion

In this reviewer’s opinion, Extreme Programming Refactored has undermined its own position by presenting far too many cute songs and made up stories “satirizing” XP. The book thus presents something that isn’t really XP, and isn’t what XP people are talking about, recommending, or doing. This tragically reduces the book’s potential contribution.

For more valuable discussion of the limitations of XP and what to do about them, I would recommend either Pete McBreen’s Questioning Extreme Programming, or Barry Boehm and Richard Turner’s Balancing Agility and Discipline.

Balancing Agility and Discipline: A Guide for the Perplexed.

Barry Boehm, Richard Turner

The senior author’s name alone is enough to ensure that this book will be read by people we would like to influence. Fortunately Boehm and Turner have done a rather decent job of describing agile methods, plan-driven methods, how they work, where they are applicable, and how they can play together. There are things to question and perhaps disagree with, and there is much to agree with.

The authors make clear at the beginning and throughout the book that successful development, whether agile or plan-driven, requires discipline. They chose, I believe unfortunately, to use the word discipline in the title instead of plan-driven, the term used in the bulk of the book.

A Few Disagreements

I believe that the authors make a bit too much of the need for good people on an agile project. They hold that an agile project cannot sustain people of Alistair Cockburn’s “Level -1″ and that plan-driven projects can. And they hold that an agile project needs Level 2 or 3 people throughout, while a plan-driven project needs them only at the beginning. I’m not convinced that this is quite true. Or, it might be a matter of project size.

Boehm and Turner draw some conclusions from Amr Elssamadisy and Gregory Schalliol’s paper on “bad smells” in Extreme Programming, taking at face value some of Amr and Gregory’s conclusions about what was needed to shore up the areas where their project had problems. I acknowledge the problems, but believe that I would have tried some more agile solutions rather than falling back, as Amr and Gregory recommend, on more plan-driven approaches. On the other hand, they were there, and I was not.

The authors say, referring to Amr and Gregory’s paper:

In this case, the “customers” were one step removed from the actual end users. They compounded their limitations by doing less homework on the real needs in the early iterations, perhaps because they thought their other customer functions were more important. Putting blind trust in “the customer is always right” adage can be risky if the customer doesn’t fully understand and appreciate the end users’ needs. A good deal of early coaching is needed to get on-site customers to appreciate and perform the role of end-user representatives. Otherwise, the on-site customers can become examples of the point in Chapter 2 about the main source of tension in agile methods being between collocated and non-collated customers.

While this is true, the authors seem to conclude that a plan-driven fix would be appropriate to the situation. However, any such fix would imply a very different staffing and budgeting approach to the project. It might well be more effective and less costly to improve the ability of those in the customer role to perform it well.

I feel that in a number of places in the book, the authors fall into the common trap of “it can be done this way, therefore it should be done this way”, even though they rightly warn us all that this is a trap. When it comes down to it, our biases and expectations rule us all in ways we don’t always expect.

One more example, still referring to the same paper:

A second smell illustrates one of the likely side effects of fixed schedules and diseconomies of scale: “Everyone says that all the story cards were finished at the proper times, but it still takes 12 more weeks of full-time development effort to deliver a quality application to the customer.” This is related to the previous smell, in that there is a large gray area called “integration” in large systems between zero-defect story completion and acceptable system delivery. And there are other gray areas between “zero defects” and “zero defects plus documentation and standards compliance”.

The recommended solution [from Amr and Gregory] was to “Create a precise list of tasks that must be completed before the story is considered finished, and then police it regularly and honestly.” This conflicts with agile philosophies such as “Managers in organizations either have a fundamental trust of people or they don’t — there is no middle ground,” and “Trust people, mistrust communication” (where “communication” presumably includes plans and standards.) It illustrates the tension between the agile ideal of small teams, where there’s always enough time for everybody to talk through gray areas, and large-project practice, where this would just consume too much time, implying more efficient, objective approaches are needed.

Again, we all see that the problem occurred. However, there might be more “agile” solutions to try, such as improving automated tests for system integration. Certainly no one should object to having a task checklist, but might object to “policing” the list. It seems to me that the mantra of any team ought to be “Trust, but verify”, in spite of recent use of the phrase in a political arena we may find odious.

Many Agreements

Overall, the book is quite well-balanced, and makes points that we should all wish to make so clearly and well. Under the heading of “Future Applications Will Need Both Agility and Discipline”, we find:

Large projects can no longer count on low rates of changes, and their extensive process and product plans will become expensive sources of rework and delay. As the use of agile methods progresses from individual early-adopter projects to enterprise-coupled mainstream applications, the Brooksian software development werewolves of complexity and conformity will be waiting for them. Thus there will be a higher premium on having methods available that combine agility and discipline in situation-tailorable ways.

In aid of this, the authors offer six conclusions, of which the above is one. The full six are:

1. Neither agile nor plan-driven methods provide a silver bullet.
2. Agile and plan-driven methods have home grounds where one clearly dominates the other.
3. Future trends are toward application developments that need both agility and discipline.
4. Some balanced methods are emerging.
5. It is better to build your method up than to tailor it down.
6. Methods are important, but potential silver bullets are more likely to be found in areas dealing with people, values, communication, and expectations management.

A Good Book, Worth Your Time

This one is a keeper. While it is hard for a hard-core XPer like me to completely agree, the ideas are well thought out and well supported.

Get it.

Questioning Extreme Programming

Pete McBreen

After reading Software Craftmanship by Pete McBreen, and being impressed by many of the ideals it contained, I was anxious to see his take on XP. I approached Questioning Extreme Programming with an open mind, hoping to glean some legitimate viewpoints that challenge some of the concepts of XP. Worst case, I would at least know what to expect in terms of resistance when discussing XP. After reading the book, I was confused and disappointed.

Tainting the book immediately is the author’s admission in the preface that “I have never worked on a full XP project.” This becomes clear with some of the brief diatribes against XP practices that are not borne out in their practical application. By the book’s conclusion, I felt that the biggest point of book was to give the author (and perhaps the reader) plenty of excuses to never do XP. McBreen foreshadows his conclusions early in Chapter 1: “You are likely to discover a difference between what you think is important and the items that XP assumes are important.” (p10)

On page 12, McBreen says, “Changing processes is hard. Attempt it only when the benefits massively outweigh the costs.” This is an obvious assertion, yet it is repeated a few times in the book. Another example from page 53: “If your current approach to software development is not broken, it is unlikely that you can recoup the cost of changing to a different process.” But McBreen doesn’t appear to believe that most software development is broken, so it is clear what decision he has already made with respect to trying XP. A very brief list of “signs that your process is broken” appears on page 9. This list seems small, again suggesting that perhaps the processes out there are working just fine. However, McBreen doesn’t mention many other signs that you may encounter, such as:

• team members don’t improve in capability
• large amounts of dusty documentation lie around
• overstaffed or overworked help desk
• heavy volumes of defects; high-end defect tracking systems
• minimized communication; lots of headphones
• large number of chiefs involved in any given project
• disdainful comments from the business on the capability of the developers
• mistrust, evidenced by excessive “CYA” emails
• segmented knowledge that creates bad dependencies on individuals

From what I’ve seen, most software development is broken, or at least bent. But it is very hard for organizations to admit there is a problem, as Weinberg tells us in Secrets of Consulting. McBreen is suggesting that there is not a problem.

Chapter 4, “What do other methodologies consider important?” provides a good summary of the various development methodologies in use today. It compares and contrasts several examples of software engineering, open source, and agile processes.

Chapter 5, “What is important for your projects?” is an interesting discussion of how to assess the role of methodologies in your specific project. It ends with a list of key questions (p52) to ask yourself to determine whether an approach is right for your organization.

On page 60, the author talks about the XP planning game: “The accuracy of the estimates produced during the planning game needs to be investigated, especially for organizations that are just adopting XP. … I wonder how long it takes a new XP team to get good at the Planning Game.” Here McBreen misses the point of the heavy feedback cycles in the planning game. Initial estimates are going to be inaccurate in any process. In XP, the team has lots of opportunities to estimate and to learn how to do it well–the team hones their estimating skills every two weeks.

In any case, no other process provides a better solution to estimation. No matter how much planning, and how experienced a team is, software developers cannot predict the next hour accurately, let alone the thousands of hours that represent a typical project. The point of the planning game is to allow teams to get good at planning, something that is missing in processes that do not accommodate replanning.

Another theme in the book is related to XP’s attempts to fix things that the author admits are clearly broken. The author fears that certain changes just won’t happen, so why should XP even try? As an example, XP stipulates that developers have the right to make their own estimates, as opposed to being told–by a non-technical person–how long something should take. In response, the author states that “Imposed estimates are such an endemic problem in software development that I suspect that few organizations will be able to use the planning game successfully.” McBreen’s defeatist attitude is a key part of the problem.

Of the XP practices, McBreen reacts most critically to the short-cycle incremental design practices of simple design and refactoring. These disciplines emphasize constant care to ensure system design is always optimal, as opposed to producing a complete design up front and assuming it remains correct. But McBreen sees “parallels between undisciplined hacking and XP,” (p66) and appears to lean on the side of developers who view incremental design as “an inefficient, questionable practice, possibly bordering on malpractice.” (p68) Which is malpractice: presuming that significant up-front design is always perfect, or being vigilant and ensuring that the code and design always stay clean?

McBreen’s whole discussion on incremental design is where my confusion began. McBreen wrote an entire book, Software Craftmanship, discussing how what most of us do today is not software engineering; it is instead craft. The premise of that book is that software is malleable, and we can easily shape or craft it from nothing into a robust product. In contrast, engineering is presented as a paradigm where we do significant planning up front because either we must or because it is too costly to change things later. (Many real engineers will refute this statement, though.) And thus we do not engineer software, we craft it with care, largely because we are able to. Personally I loved the book.

Yet Questioning XP seems to refute the craft metaphor, as McBreen wishes for a process where we can do perfect design up front. He also wishes for developers not to be burdened by having to show constant love and care for their craft. Unfortunately, that’s what craft is all about: craft will fail if its practitioners are not disciplined. If McBreen wants us to believe him in Questioning XP, then he should refute his earlier conclusions in Software Craftmanship.

Besides, McBreen is just wrong, demonstrating his lack of personal experience with XP. XP does support up-front design–it’s just that it recommends that design always proceed in the presence of code. The more effort we spend on design without feedback, the more likely that our design is incorrect. Design is done during the planning game, in addition to being done constantly throughout development. XP by no means prohibits us from sketching out UML diagrams prior to beginning an iteration, a task, or a test case. But we learn as we repeat, and our need to physically sketch diminishes as our understanding of good OO design and the system increases.

The brevity and title of Chapter 8, “Are We Done Yet?” hints that the author is wearying of all the XP practices by this point. McBreen includes a brief discussion of test-first development and acceptance testing as part of the last few practices covered. This is unfortunate, since the implication is that testing is an afterthought in XP as well. In fact, these are the two technical practices that will make or break an XP effort, and they should really be viewed as the technical drivers in XP.

I also note that McBreen is inconsistent in his naming of the unit-testing practice, using “test-first design” once or twice earlier, and “test-first development” elsewhere. This is to be forgiven, since the XP community keeps changing the name. The preferred term is now “test-driven development,” which technically encompasses the older term test-first design–not test-first development–plus the refactoring practice. The important thing to note is the reason it is (was) called test-first design: writing tests first has a huge impact on design. Were McBreen to have gotten the name right and have done a bit more research, he could have helped explain in the previous chapter that test-first design is one more way that design is always happening in XP.

Some of the side arguments in the book are appalling. In the chapter “Truly Incremental Development,” McBreen argues that making code readable and maintainable will “dumb down” the software (p67). “Tight, well-written code … can look obfuscated to people with less knowledge of the programming language.” First, clever code has little place in modern software that must be maintained by people at all levels of ability. Second, XP does not promote dumbing down software to the level of incompetent developers. What it does promote is making code clear wherever possible. I’ve been burned by too many high-priced consultants, with an attitude like McBreen’s, who left behind unmaintainable messes. I’m sure most of you have too.

The discussion on clever code brings up the issue that there are few true craftsmen in the software development industry. The majority of developers are adequate at best. McBreen’s discussion on pair programming is brief, like the test-first design topic, and only scratches the surface of its huge role in an XP effort. Both practices do wonders for bringing up the level of competence on a team. Pair programming educates all developers far faster than individuals working alone in their cube. Even the most experienced developers learn more through pairing. Feedback from the quick cycles of test-first design also serve to bolster and reinforce the body of knowledge of a developer.

The bulk of the chapters in section 4, “Questioning XP Concepts” contain well-reasoned discussions of concepts such as test-first development, source code as design, and large teams. Chapter 16, “Requirements: Documentation or Conversation” re-emphasizes one of the author’s main concerns about XP: the reality of finding an on-site customer with enough knowledge to be able to accurately feed the development team. Indeed, it is difficult in many organizations to find such a resource. Yet, the ability to precisely pin down requirements is a key factor in making any project successful. Once again, the author seems to be dismissing an XP concept because it’s just too hard to try and fix things that even he admits are broken. If the organization values the ability to deliver what the business is looking for, then it must learn how to grow the body of people able to drive a project as customer.

In chapter 12, “Test First Development,” McBreen discusses the overall value of testing in the XP process compared to other processes. This is a valuable chapter that discusses automated testing vs. manual testing, as well as the relationship between the development team and the test team. McBreen admits that XP teams are producing software where “the quality of the resulting applications is better.” (p110) The chapter ends on the note that testers should be an integral part of software development, as XP promotes, “rather than after-the-fact commentators and critics, as in the traditional approach.” Excellent observation.

On page 130-131, the author states: “When all of the practices are being used effectively, XP teams report that their workdays are productive and enjoyable.” (emphasis added.) The good part about this admission is that McBreen answers his own question from page 66: “Is a high level of discipline sustainable in the long run?”

McBreen’s defeatism surfaces again in a discussion on documentation and XP maintenance teams (Chapter 17, “Is Oral Documentation Enough?”). McBreen admits that there is high value to acceptance tests and unit tests as documentation. But he doesn’t believe that maintenance teams will keep the tests up to date, based on his experience that shows him maintenance developers will not keep documentation up to date. Quite a leap of logic; nonetheless, why knock XP for something which there is supposedly no solution (if there is one, McBreen doesn’t tell us what it is).

Part V, “Understanding the XP Community,” talks about the divisive nature of XP jargon and developer culture. Granted, XP proponents are often overly enthusiastic, and are good at being provocative regarding other questionable practices, such as big design up front. Should we hold enthusiasm against them? McBreen seems annoyed that XP has its own terminology. What process does not?

Part V ends with a weird chapter that tries to come up with a way of yanking extreme programming away from developers, supposedly to force them into a more rigorous, heavyweight approach. McBreen states that this a difficult proposition because the developers love working the XP way so much, and because the customers love the productivity they get out of the development teams. If everyone likes XP, what’s the problem McBreen is trying to solve?

The final section, part VI, presents the author’s conclusions. The basic conclusion is that XP is too narrowly focused to be applicable to most people. McBreen presents a good checklist of ten criteria to use to determine whether XP is applicable to your efforts. But the answers given by McBreen do everything to convince you that the answer to most of the questions is no. Above all, fear seems to drive McBreen’s recommendations, and if we follow his lead, we are best off not changing anything.

Also included in the conclusions are assertions that aren’t discussed anywhere in the book. For example (p175): “If the majority of your projects involve writing life- or safety-critical embedded software, please don’t even think about using XP.” Please tell me why.

I appreciated Questioning XP, because it presents many good challenges that need to be addressed. But McBreen tarnished his effort with too many unverified claims. It also seemed like the primary audience was McBreen himself–perhaps he wanted to be convinced by book end that he had no reason to try XP. McBreen even hoped that XP will be soon supplanted by something else, so he wouldn’t have to hear the rhetoric any longer (p168).

In summary, Questioning XP did not appear to be backed by enough meaningful research or experience to provide a truly honest critique of XP. Its conclusions did not seem to be in line with the evidence presented in the rest of the book. However, I do recommend it for XP coaches–it does provide a thorough awareness of the issues that will be faced on an XP effort.

Software for Your Head

Jim and Michele McCarthy

I read a lot of books, and recommend a lot of books. I believe it’s possible to find something useful in almost every decent book. Still, over the many years of my life, there have been quite a few books and people that made a difference. I’m thinking of Dijkstra, Knuth, Weinberg, Brooks, Constantine, DeMarco, our own Kent Beck and Martin Fowler. I hesitate to list these authors, as surely I’m forgetting someone whose book made a big difference to my professional life.

Software for Your Head may be one of those books.

The book is based on the results of years of running the authors’ “Software Development BootCamp” courses, five intensive days where teams come together to envision a product, organize to make it, and then do so. From these simulations, done over a period of five years, the McCarthys have come up with that patterns that make up this book.

The ideas in the book are very challenging. The book is long. It is very readable yet hard to take in because some of the notions are hard to imagine. Can you imagine that it would be possible to reliably build teams where the members were fully present, committed to unanimously making the best decisions possible, committed to expressing their emotions and their ideas, committed to the productive confict of ideas, committed to true creativity, excellence, accomplishment, shipping the product? That’s what Software for Your Head asks you to imagine.

Often when people hear about XP, they get stuck because they think the world can’t really be that way. Believe me, Software for Your Head will do that to you in spades. Many of us have been on at least one great team in our lives; I hope most of us have that privilege at least once. Jim and Michele dare us to imagine that this experience could be repeated almost at will.

Is it possible? Will these patterns and protocols work? Will enough people study them, learn them, commit to them, adopt them? I don’t know. Certainly I hope they work and I hope people try them. I don’t know whether Jim and Michele are right, but I want them to be. I feel sure that they have at least a big piece of the truth.

What I’m sure of is this: if you read this book, really study and consider it, you will think thoughts you haven’t thought before, and you will likely learn something about yourself, your colleagues, and your projects. I read a lot of books and recommend a lot of books. This one is special. Do yourself a favor: buy it, read it, and give it deep consideration.

In this review: Java Tools for eXtreme Programming, Agile Software Development, Agile Development with Scrum, The Pragmatic Programmer, Slack, and Dynamics of Software Development.

Java Tools for eXtreme Programming Richard Hightower and Nicholas Lesiecki agile review by Mike Clark, of Clarkware.com A toolbox brimming with trusty tools can help teams deliver projects faster, and any budget can afford good-quality open source tools. This book effectively demonstrates how to use Ant, JUnit, Cactus, HttpUnit, and JUnitPerf to make the XP practices of automated testing and continuous integration possible. Each of these tools is dutifully put to work automating the build and test cycle of a sample J2EE application. Don’t let the extreme title or the fluorescent cover frighten you – it’s the price of admission for a ride on the XP bandwagon. This book should appeal to XPers and non-XPers alike who recognize that automated testing and continuous integration are good things for any project. To be fair, the book recycles a lot of good information already available online, but packages it into a cohesive, albeit gravity challenged, format. And although it includes simple examples for using every tool, those unfamiliar with J2EE will likely find the lengthy case study a bit heavy-handed and seek other references. The book is a good introduction for the uninitiated and a valuable reference for those plying their trade with these tools. Don’t miss an opportunity to easily automate your Java project and spend more time delivering business value!

Agile Software Development Alistair CockburnAlistair is software development’s archaeologist, or maybe I mean ethnographer. Whichever I mean, Alistair visits real projects and interviews them. He recommends methods based on two criteria: did the project ship, and would the team use the same method again. That, folks, is about as close as you get to facts in our business. Agile Software Development addresses a wide range of issues in agile development, both philosophical and practical. It includes the first book-published description of Alistair’s Crystal methodology, as far as I know. Be aware that some people think there’s not enough meat in this book. If you want rules, XP books have more rules. There is depth to this material in my opinion. I found it thought-provoking. Maybe you have to be nearing Level Three to get it.

Agile Development with Scrum Ken Schwaber and Mike BeedleScrum is a team approach to software development that relies on the “people in a room” approach to things. Scrum projects work in bursts of time, typically one month, delivering whatever they can in that period. The process description doesn’t go to the programming level as XP does: it assumes that the team knows some way to develop software and that they’ll do so. Scrum projects manage a “backlog” of desired features. There is a Product Owner, analogous to XP’s Customer, and there is a Scrum Master, who may be analogous to the XP Coach. The Scrum Master seems to have more power than a Coach. Scrum has been used successfully on a number of projects. The Scrum guys are in some ways more contentious than even I am. I like it when someone makes me look reasonable.

The Pragmatic Programmer Andy Hunt and Dave Thomas programmingOutstanding book! Andy and Dave are excellent programmers and excellent teachers, with one eye each on simplicity, skill, craftsmanship, learning, and quality. Where they get all those eyes, I’ll never know. This is a book on how an individual programmer can increase skills, and how to work effectively, alone or on a team. As with any such book, much of the advice is something you already know. Much of it is also something you have forgotten to focus on lately. Very very good stuff. I wish I could have written this book.

Slack Tom DeMarcoIf Tom DeMarco writes it, read it, that’s my advice. This one, billed as a “Handbook for Managers, Entrepreneurs, and CEOs”, is about “Getting past burnout, busywork, and the myth of total efficiency.” The basic premise is that when organizations run too “lean and mean”, they lose capacity for change and innovation. We’ve all see it happen: DeMarco documents and explains the effect and what to do about it. Tom reminds us that we all have responsibility to lead our companies out of the darkness. He feels that when we look at Dilbert, we need to ask ourselves, as Dilberts all, what we should do instead of the counter-productive responses that Dilbert too often shows. I think Tom is a little hard on Dilbert, who does try sometimes. Maybe Dilbert is just beaten down. We’re not beaten down yet: let’s change the world, one more time.

Dynamics of Software Development

Jim McCarthy

I’m embarrassed to report that Jim wrote to me recently and asked me why I hadn’t reviewed this book. I was embarrassed because it is one of my favorites but somehow got left out of the review stack. Sorry, Jim!

The book is a collection of “rules” for “delivering great software on time”. Many of them have entered the lexicon of programmers everywhere: “Don’t Flip the Bozo Bit”, and “Don’t Go Dark” are two of my favorites. I agree with most everything in here. Jim was Extreme before Extreme was cool! If you haven’t read this book, you should.