Ckwop's Journal

Meditation Driven Development

Sun, 15 Feb 2009 12:00:00 GMT

Meditation Driven Development

In the beginning

Tim is a veteran programmer. He works at a world renowned software company. Despite this, in his long career he has seen many projects fail and only a few succeed. After 15 years developing software, he decides he wants to get out of the trenches and start managing projects. He's tired of seeing projects fail for entirely preventable reasons. He realises that the only way to fix these things is to be the guy who makes the decisions on these wayward projects.

Tim, like most developers, is quite a eccentric man. He is an open source enthusiast and he has written a great variety of open-source software. Before writing any code on his projects, he usually spends an hour doing yoga followed by intense meditation. After doing his meditation he finds that he writes cleaner, better designed code.

One day, he is meditating when it strikes him: "Maybe this is the solution to the software crisis, everyone needs to meditate before they write a line of code." Meditation Driven Development (MDD) was born.

Tim goes to work and tells his team about his new methodology. Being good friends of Tim they decide to give it a try! Tim waits patiently for a project that he can try it out on.

The first project

It's day one of the implementation phase of the project. Tim rides his bicycle to work with extra-gumption. "Today will be the start of something great," he says to himself.

It's 9 AM and it's time of the first team meeting. Tim asks everyone to sit around on the floor in a circle and reflect deeply on a particular passage from the Bible. At first, his team are uncomfortable. They're unsure of what to do, but under Tim's careful direction he manages to get a result. His team manages a 45 minute meditation and afterwards they're energized and ready to write great code.

Tim mandates that all the team members must meditate twice a day, once when they come to work and once when they return from lunch. Each meditation session will be no shorter than 45 minutes long. This will be done without fail, every single day of the project.

The project was scheduled to last three months but Tim's team was done two weeks early. Not only that, his team's software contained 20% fewer bugs than the company average. Tim concluded that MDD must have been responsible. "MDD WORKS," Tim exclaimed!

Spreading the gospel

Given his early successes, Tim decides to roll out MDD across all his projects. His early successes were repeated. His boss is impressed and asks why he thinks his projects are enjoying so much success. For Tim, the answer is obvious: "MDD. MDD allows us to produce higher quality software, to budget, on time."

Tim convinces his boss that the whole company should use MDD. Tim sends out his disciples to other teams to convert them to MDD. Slowly but surely his company converts to MDD.

Tim realises that other people could benefit from MDD, so he makes a deal with O'Reilly and writes his transformational book, "Meditation Driven Development - The way to cheaper, more reliable software."

The initial reaction is positive. Before long, a line of people are queueing up at Tim's door to get in on on the new MDD craze sweeping the industry.

The MDD Manifesto

A year or two after Tim's book on MDD, it was clear that MDD was being misapplied in some companies. Some teams were having trouble making MDD work in their company. Some people were not following the strictures of the book correctly.

They were having 20 minute meditations instead of the mandated minimum of 45 and they were only having one of these per day! Tim exclaimed, "How could they possibly expect the methodology to work when they're only doing one meditation per day!?" Something needed to be done.

By this point MDD had fractured in to a series of closely related but slightly different methodologies. Some preferred to meditate to music, other liked to meditate on the words of famous poems. Others preferred four shorter sessions per day instead of the original two. The differences mattered less than the similarities. As long as there was an hour and a half of meditation per day, every day, it was MDD.

Tim and his disciples sat down and bashed out a manifesto that bound these various strains of MDD under a common political philosophy. They vowed to take forth the central message of MDD and transform the industry.

MDD becomes mainstream

It's now nearly a decade after Tim wrote his transformational book. Once a year he keynotes at "MDD-con" in San Diego. He has released a series of books on the subject and makes a substantial amount of money in consulting on everything and anything related to MDD.

MDD is now an established cottage-industry with a series of satellite conferences spanning the globe. It's now so popular that even the pointiest of pointy-haired bosses is starting to hear about it.

Yet, all is not well in the MDD community. Ordinary companies are having real trouble getting the benefits of MDD. Projects are still late, expensive and buggy. Tim's career has become a continual fire-fight. His consultancy gigs are no longer about getting people to adopt MDD but are about fixing problems in projects that already use MDD!

MDD fifteen years on

It is now over fifteen years since Tim wrote his award winning book. MDD is now thoroughly mainstream and used by a huge variety of teams. Despite Tim's initial success, MDD projects on average tend to be late, over-budget and low quality.

Tim still extols the merits of MDD but he finds dissatisfaction wherever he goes. A whole generation of programmers has now grown up with MDD and they're disillusioned by failed project after failed project. Tim can't understand why people were having such a bad time with MDD. He just knows that MDD works! He's seen it with his own eyes! In his own mind, he concludes that the problem must be that nobody is doing it right!

One day, Tim comes across a team led by a bright-eyed manager called Cathy. Cathy's projects are on time, to budget and much less buggy than the company average. Cathy was a developer for fifteen years and has only recently become a manager. Tim is suitably impressed by her success, he asks her what her secret is:

"It's our new methodology, QDD. QDD allows us to produce higher quality software, to budget, on time. It's going to transform our industry."

Afterword

Why did MDD work at first and then fail when it became mainstream? The reason that MDD worked initially had nothing to do with strictures of MDD. Tim was a talented developer working in a company filled with talented developers. Pretty much any methodology you care to think of was going to work with these people.

Moreover, Tim's initial team were the earliest of early adopters hand-picked by Tim himself. They were the very people that would develop high quality software naturally anyway. The only role the MDD had was in helping the team to gel, and once they had they looked like an impossibly productive team. Tim misidentified the source of this productivity as being the result of his methodology, rather than his competence of himself and his team.

As MDD became more widely used, the average quality of the developers using the methodology dropped and so did the success rate of using MDD. By the time it was used by the average team, MDD got average results.

I'll close with a quote from one of the greats of our field, Robert Glass:

People matter in building software. That's the message of this particular fact. Tools matter. Techniques also matter. Process, yet again, matters. But head and shoulders above all those other things that matter are people.

This message is as old as the software field itself. It has emerged from, and appears in, so many software research studies and position papers over the years that, by now, it should be one of the most important software "eternal truths." Yet we in the software field keep forgetting it. We advocate process as the be-all and end-all of software development. We promote tools as breakthroughs in our ability to create software. We aggregate miscellaneous collections of techniques, call that aggregate a methodology, and insist that thousands of programmers read about it, take classes in it, have their noses rubbed in it through drill and practice, and then employ it on high-profile projects. All in the name of tools/techniques/process over people.

Don't fall in to Tim's trap. Great developers create great software not because they follow the strictures of some methodology but because they are truly great, gifted artisans.

Do you proofread your code?

Sun, 11 Jan 2009 16:23:26 GMT

Do you proofread your code?

Image taken from www.yetiarts.com

When I write a blog post, such as this one, there are a few step I follow before committing any new entry to the world:

I type a first draft out in Google Docs.
When I'm reasonably happy with the post, I type-set the post in HTML.
I then proofread the post and fix any mistakes.
I put a copy of post up on the site but it is not linked from anywhere on the main site.
I then proof read again and fix any mistakes. This is the last chance to get it right and I usually leave it a day or so before doing this final proofread.
I make the blog post available to the world.

The process is not perfect. I'm sure there are many errors that survive right in to the final blog post. The fact that these errors survive is sort of beside the point. The point is that this process dramatically improves the quality of this blog.

I was told by my English teacher when I first started writing essays back in my high school days that I must proof read my work before I submit it. She said, "Proofreading your essay will save you time because you'll fix a lot of the mistakes before I find them. If you don't proofread your work then your essay will back to you covered in red ink and you will probably have forgotten what you'd written. As such, it'll take you much longer to fix the mistakes."

Doesn't that sound like everything we know about software defects?

And then it struck me. While I automatically proof anything I write on the web, how often do I proofread my code? In all honesty, I've never done it seriously. I've never taken a check-in I've made, printed it out, then sat down, worked out what each routine was meant to do, then come up with a proof as to why the code really does do its job properly. How often does anybody really proof read their code like that? In my entire (and fairly short) career, I've never seen anyone do it.

Moreover, how many people give that print-out to a colleague to examine? I'd bet that very, very few organisations do that. How many people sit around a table and proofread their code together? I guess it'd be even fewer.

With that in mind, is it any surprise that before we subject our code to testing there are typically eighty defects per thousand statements in our code?

Could you imagine an author publishing a book without proofreading it at least once? Could you imagine an author publishing a book without giving the manuscript to someone else to look over? Why is it acceptable to publish software without doing even this basic level of quality assurance?

Then I had another epiphany! This is why formal inspections remove up to 90% of the defects before the first test case is run. It's not that having inspections really finds the tough, race-condition style, kinds of bugs. Most of the time, they find trivial problems.

Think of if it like you're using a word processor to write an essay: If compilation finds the "spelling" mistakes in code, formal inspections find the "grammatical" mistakes. You're not finding the deep errors in your program, you're just finding the next layer of superficial defects. Yet these sorts of defects make up almost all of your bugs.

Proofreading in a small group is the closest thing we have to break-through technology in defect reduction. Robert Glass had this to say about formal inspections:

Rigorous inspections - the technique of pouring over a piece of software to identify any errors it contains - constitute a near-breakthrough. Research study after research study has shown that inspections can detect up to 90 percent of the errors in a software product before any test cases have been run. And that signifies an extremely effective process.

Furthermore, the same studies show the cost of inspections is less than the cost of testing that would be necessary to find the same errors. What we have here is an effective process that is also cost-effective.

According to the research, the key to successfully finding errors is to be as rigorous as possible in your proofreading. Do not mistake rigour for formality. It is entirely possible to do a formal inspection but be intellectually lazy about it. Any such approach is doomed to fail. Rigour means that you need to sit down and really concentrate hard on the code in question and not just go through the motions of a formal process.

If this sounds like hard work, it is. According to Glass it not uncommon to only be able to inspect a hundred lines of codes per hour. Worse, the process of going through the code so rigorously is mentally exhausting. An hour is the maximum that most people can do at a time.

But it's worth doing. The research does show that those that dare win. If you're willing to do that punishing work, you'll be able to develop quality software much cheaper than your competitors.

Without a proofreading programme, when a bug is fixed by a tester the bug is usually fixed leaving the design of the program entirely unchanged. By having a solid culture of peer review, it's much more likely you'll fix bad design decisions as well as the bugs. This means that your code base will probably be smaller, more efficient as well as being much less bug ridden.

If you're still not convinced, take this passage from Steve McConnell's Code Complete:

The combination of design and code inspections catch about 60 percent of defects, which is higher than other techniques except prototyping and high-volume beta testing. The results have been confirmed numerous times at various organisations, including Harris BCSD, National Software Quality Experiment, Software Engineering Institute, Hewlett Packard and so on.

The combination of design and code inspections usually removes 70-85 percent or more of the defects in a product. Inspections identify error-prone classes early, and Caper Jones reports that they result in 20-30 percent fewer defects per 1000 lines of code than less formal review practices.

Designers and coders learn to improve their work through participating in inspections, and inspections increase productivity by about 20 percent. On a project that uses inspections for design and code, the inspections will take up about 10-15 percent of project budget and will typically reduce overall project cost.

Just like an author subjecting their manuscript to review makes their book better and improves their ability to write, having others review your code makes your code better and it helps makes you a better programmer. Not only that, it makes you 20% more productive!

So perhaps a good New Year's resolution for any developer would be to set-up a proofreading programme at their company!

The Inverted Lemon

Sat, 20 Dec 2008 21:44:30 GMT

The lemon market was first described by economist George Akerlof in his seminal paper: "The Market for 'Lemons'." That paper won him the Nobel Prize in economics.

It has often been argued that software is a lemon market. In particular, Schneier made this point about security software in this article. The same argument has been applied not only to software security but to certain facets of software quality; such as, maintainability and testability. In this essay, when I'm talking about quality I'm really talking about these two facets of software quality.

The argument goes that Software is a lemon market because the people buying software do not have the skill nor the information to judge whether a given piece of software is high quality or not. As such, quality does not really impact their decision to buy the software at all.

However, the interaction between quality and price is much more complicated than it looks. I am going to argue that software is a lemon market but not in the traditional way. The suckers are not the customers, it's the companies that make low quality products.

What is software quality anyway?

I think of software quality as a list of "ilities" and while the list of which ones are priorities changes from application to application:

Security
Maintainablity
Testability
Performance
Portability
Usability

I don't want you to get too hung up on the order of this list. Different projects will weight these attributes according to their requirements. However, what is interesting is that the end user is only interested in a few of these attributes.

Usability
Performance
Security
Portability

Testability and maintability are gone. They don't even exist for a user of the software. Your code could be world-beatingly beautiful or it could be a giant ball of suck, they don't care. Worse still, on first examination the customer won't pay you a dime more either way.

As long as you put a shiny, sensible interface over that ball of mud, it's all good.

Why software is not like a car

The main theme behind the idea of the lemon market is that the asymmetry between buyer and seller in information about the cars leads to a situation where the buyer has no way to accurately gauge the quality of what they're buying.

The example that most people can relate to is the buying of a used car. The idea being that vendor usually has much better information than the buyer. This results in people getting ripped off, which if left unchecked, could result in a total break down of that market.

However, it is misleading to try and apply this definition directly to software. The problem with the car analogy is that physical items under go wear and tear. A piece of software will be the same quality on the day it was published to the day it finally falls out of use, provided no updates are made to it. In short, code doesn't rust.

If a user buys a low quality program, it is not a lemon in the same way that a car that barely works is. A poor quality car will randomly and expensively fail in various areas after purchase. A poor quality program may fail expensively but it will fail in precisely the same way every time.

The none-rusting property of code changes the quality calculation considerably. If software doesn't rust then the quality just has to be good enough. Good enough is simply a case of ensuring there are no show-stopper bugs; good enough is a synonym for "barely works."

This idea of "barely works" is an anthama to most software developers. Good developers are unhappy writing anything but the cleanest code they can possibly write. If Barely Works were a developer we'd stick it the stocks and throw rotten fruit at it.

But developers have a distorted view of the software. It's completely different to the view of a paying customer. Developers see all the bugs in the bug tracking database. Developers see all the suck in the code. Developers see all the poor architectural decisions and secretly think of ways to fix them.

The users see a reasonably good interface and only the bugs that stop them working.

Therefore, by simply fixing the bugs the users see, you can improve the impression of quality without doing a lot of work. As such, "barely works" makes a great deal of business sense.

In some cases, like the recent MySQL release, even shower-stopper bugs do not hold back a release.

Hold on a second, I thought that it is cheaper to write quality software?

Technical debt is a wonderful expression because it is succinctly explains to a laymen the insidious invisible cost of bad design. Unfortunately, the concept of technical debt also explains why convincing people to pay off that debt is a tough sell.

The problem is that the interest rate on technical debt is very small. The interest rate can't be any greater than five percent. In fact, it's probably a tenth of that value. As such, you can build up truly gargantuan amounts of technical debt and the cost of servicing that debt is tiny in comparison to the size of the principal. So why not just have an interest only mortgage and never pay back the principle?

It does seem futile, doesn't it? Say you kick off your project to pay back that giant debt, for the first few months you're going to be paying off nothing but interest and you won't make any appreciable dent on the principal. You're basically going to be thrashing around doing a lot of work and not achieving a great deal.

With most business applications, making the code base testable at all is a major multi-month project. It's a risky project too; you risk breaking a lot of working code without adding anything the business can see as an improvement.

If you eventually manage to pay off your technical debt and you keep it paid off then, yes, it will be cheaper overall to write quality software. The problem though is that getting there in the first place.

If getting to the promised land requires scaling Mount Everest without bottled oxygen, don't be surprised if you don't have many takers.

Inverting the Lemon

So far I've made the argument that quality, in the sense of maintainability and testability, doesn't matter. This is because the average user has no way to choose between a program consisting of a single gigantic main method and a program that is designed sensibly.

However, that is not the whole story. There are places in the industry where quality matters. Barely Works is not going to be acceptable in pace-maker software or missile control systems.

Even in general business software, there are some companies that have made quality their power-animal. The reason why quality works as a business strategy for them is because their audience understands that they're paying for quality. That's the point of their brand.

Take bugger trackers for example. Fogbugz is arguably the best bug tracker in the world. Everyone who buys Fogbugz knows they could just install Bugzilla but they don't. The reason they don't is that Fogbugz sucks less in pretty much every dimension.

If software really were a lemon market, Fogcreek would have gone bust years ago. In fact, if you look at every single one of Fogcreek's products, there is a mature open source competitor that they totally spank.

You see, the people who want quality will pay for it. Moreover, the people who demand quality know what they're looking for - they're informed buyers.

So few people develop quality software that the people who do can charge above the market rate for their product. Therefore, while it may be cheaper to develop high quality software it ends up being more expensive to buy. This makes it far more profitable to develop quality software!

Most software companies have got such a huge principal on their technical debt that trying to convert their business to a higher quality and thus more profitable outfit would put them out of business. This is the economic barrier that keeps quality software a niche.

And that, my friends, is the inverted lemon. The joke is not on the end user, they got exactly what they paid for, the joke is on the business who pays through the nose to maintain a crappy product - and they make less money selling that product while they're at it.

The death of the essay

Mon, 17 Nov 2008 20:25:29 GMT

Before the invention of the Internet, people used to write letters to one another. A century ago, people in the United States used to pay $0.05 to send a letter which, with inflation factored in, would cost $1.14 today.

The fact that letters took days to travel to their destination and sending one cost comparatively more than than it does today meant that when people did write, they wrote at least a few hundred words.

Today, we're living in a time where you can communicate instantly right across the world to thousands of people simultaneously for virtually no cost. In this climate, the impetus shifts from writing long, carefully written and carefully considered pieces to a focus on immediacy and the "now."

This change has been crystallising in front our eyes for the last decade.

I first experienced the wonders of electronic communication as a youngster when I signed up for my first e- mail account. All the cool kids were signing up to Hotmail when I was fifteen years old, so I got in on the deal. To cut a long and relatively boring story short, it quickly occurred to me that you could use this tool as a primitive instant messenger - provided the person you wanted to talk to was on-line at the same time. This was perhaps my first experience of the world of micro-messaging.

At the turn of the century, our household finally connected to the Internet. One of the first applications I fired up was mIRC and this was my first experience with the world of instant messaging. To me, this was a much more important experience than the first time I browsed the web. Here was a tool I could use to communicate to my friends instantly for virtually no cost. It was amazing being able to communicate in real time by typing messages out to each other. Here was a place where everyone could speak at once and yet, because the communication was written, everyone could be heard. It was one of the few times where I was genuinely amazed by computing.

Later, I got my first mobile phone and with that came the glorious innovation of SMS. This was another crucial step on the way towards micro-messaging. At first, SMS messages were limited to just 160 characters and it cost twelve pence to send a message¹. Therefore, your messages had to be short and each message had to be positively useful for it to be worth the cost of sending one. Of course, different people have different ideas of what useful meant; but I digress. Due to the lack of spam on mobile phones and the fact I always have my mobile to hand, this remains my primary form of communication, even today.

We fast forward to today and we see the likes of Twitter, or the all important news feed on Facebook and we've now got to the point where every tiny, inane detail, of one's life is logged for the world to see; and most people share these details to all and sundry without any consideration of the implications of what they're doing.

And I wonder if we've lost something. Have we dumbed down to the point where any thought that can't fit in to the Facebook news feed is automatically worthless? Has the world's attention span decreased by so much that nobody has the attention span to even read a letter any more, let alone write one?

I don't want to lay on the hyperbole too thick since I'm starting to border on the absurd but there is some sort of point there. I just wonder, if we don't have to invest any effort in talking to each other, if we encourage short brutish talk, then we don't really have any reason to value what we say. If we don't value what we have to say then what is the point of saying it?

So I ask, is Twitter the ultimate in onomatopoeia; the service that sounds like it is? The annoying drone of a thousand thoughtless monkeys battering away on keyboards producing nothing of value. I guess what I'm really trying to say is what is the point of Twitter? What is the point of micro-blogging to the world?

¹ Technically, they still are limited to 160 characters but from the user's perspective, you can seemlessly chain many messages together to make one much larger message.

² Atwood is major proponent of Twitter, which is why I've used his Twitter feed as an example. However, he is an unusual user of Twitter in the sense that he can actually write. His blog, Coding Horror, contains a number of interesting and well written articles. Yet, even for a talented writer like Atwood, it is impossible for him to use Twitter in a way that doesn't make him look like a moron. If he can't make it work, can anybody else?

Interviews considered harmful

Mon, 13 Oct 2008 18:58:29 GMT

Interviews considered harmful?

Introduction

I've been developing software professionally for six years now, so I think I have just about enough experience to talk about the interview. In that time, I've been the interviewer many more times than I've been the interviewee and I've come to the conclusion that interviews suck.

The idea of an interview is to try and work out whether a candidate is a good fit for your team. There are usually three things that you look for when interviewing:

The person in front of you is smart.
The person in front of you can actually write a program.
The person in front of you will compliment your pre-existing team.

The problem is that an interview is very good at weeding out a lot of the obviously bad people but is really poor at sorting the average candidate from an extraordinary candidate.

And it is common knowledge that extraordinary programmers are much better value for money than their much maligned mediocre counter-parts.

So the question is, how do you find the extraordinary programmers?

Finding someone smart

When looking for a potential programmer, you should try to find someone smart. Someone who is not just smart but smarter than yourself.

The problem is, this is a really difficult thing to asses in an interview. The problem is that knowledge and smart are not equivalent.

Then someone had the idea of setting little problem questions that the candidate would have to solve right there and then in the interview. The thinking goes, if they could solve these bizarre little puzzle, then clearly they must be smart and thus a good programmer.

Completely wrong.

The problem is that the answer to the puzzle is not the answer you'd want to hear from a real developer. For example, the title of a book about the puzzle question culture at Microsoft has the title: How would you move mount Fuji?

How would I move mount Fuji?

I wouldn't try and move it at all. Moving a mountain has never been attempted by anyone, ever. Any attempt to move a mountain would likely run vastly over budget. It's likely the technology doesn't even exist to do this in an environmentally friendly way; although the Russians did happen to experiment with nuclear weapons in civil engineering. Why do you want to move the mountain anyway? There's probably an easier way to do whatever it is you want to achieve and it will almost certainly cost a lot less money.

That is the answer I'd expect from such a question but no candidate will ever give you that answer. They'll try and solve the little brain-teaser you've set them because they believe that's what they're meant to do.

Then you get the questions that are nothing short of comical: "Why are man-holes covers round?" Well, if you don't live in the United States, there's a good chance they're not. There's literally thousands of square man-hole covers in my town alone. The cretin that came up with this question didn't realise that you can prevent the cover falling down the manhole by simply making the cover slightly larger than the hole it covers.

The people you are hiring are not logicians, they're engineers. You want them to think like an engineer. Being able to solve what is, in essence, nothing more than an elaborate Sudoku does not make you a great programmer. You might as well hire crossword competition winners!

Finding someone who can program

Most programming applicants can barely code. It's common knowledge that setting even simple problems, like FizzBuzz, will filter out 75% of applicants. FizzBuzz is great for filtering out the crappy candidates but we want to find great candidates. Assuming your candidate passes the FizzBuzz test, what do you give them next?

There is a tedency to lurch to "Implement Quicksort on the white-board" type of problems.

Why would you get them to write code on a white-board? That isn't how a programmer normally writes code! You can't execute scrawl written on a white-board. Code written on a white-board is likely to be chock full of errors that nobody in that room spots.

Even if you got them to implement at their computer, I find asking these sorts of problems as odd as asking a research chemist to recite the periodic table. It may show that you've got an interest in chemistry but that doesn't in any way tell you if they're up to the job at hand.

Presuambly the reason why you'd want them to implement Quicksort is that you want the candidate to be aware of the difference in run-time complexity between Quicksort and other slower sorts. Why not just ask them which sort they would use?

Ultimately, we want an engineer not a logician. If I had to choose between a person who knew nothing about run-time complexity but routinely profiled his code and a person who knew how to prove Euclid's GCD algorithm has a worst case running time of n² but never used a profiler - I'd take the guy who uses the profiler.

I'm not beating up on just Quicksort here, I'm taking about the whole shooting match; tree traversal algorithms, red-black trees, data structures etc. This stuff is certainly interesting but it's not at all useful in most programming jobs.

I've never implemented any of these things in my career and I suspect this is true for most of the development community. If I was asked to implement it, I'd just look it up just like I'd look up any other algorithm.

So what should you give them if you want to know if they're up to the job?

Where I work, we get them to use one of our products. We give them a couple of hours and get them to integrate with our web-services platform. They're producing the very code our customers would be producing; they're solving a real business problem that our customers have. This is infinitely more useful then getting them to implement something like Quicksort.

Development is 20% code, 80% personality

The most important attribute of a developer is their personality.

A person whose smarts are matched only by their laziness is completely useless to you. A person who lacks attention to detail is completely useless to you. A person who has no capacity to learn by their self is completely useless to you. A person who is an arsehole is.. well, I think you get the point. Even if they are none of these things, they could just be a poor fit for some other reason entirely unrelated to their aptitude.

This much is obvious. These are things you can't really judge until you actually work with them. Yet this stuff is what distinguishes a great hire from a poor one - yet the interview gives you no way to assess any of these important traits.

Are Interviews considered harmful then?

Just like the GOTO statement, interviews are not inherently bad. Properly deployed, they are useful. The best way to use an interview is to do a final sanity check on the candidate before offering them the job on a trial period. You then use the trial period to assess the candidate's fit with the team.

The trial period must be a trial period though. Too often, inertia sets in and there is a strong temptation to throw good money after bad. If the person stinks, it makes sense to get rid of them as fast as possible. You must have no qualms about firing someone who is sub-standard.

The interview must contain a programming test of some kind. This programming test must be relevant to what you actually develop.

A team is built around a shared sense of purpose and a shared sense of pride in what they do. Therefore, it must ultimately be the team the candidate works with who decides whether they stay or go. If anyone black- balls the individual then the candidate will have to go.

If this sounds expensive, well I guess it is. However, compared to the cost of getting hiring wrong it's a bargain.

Don't test to remove bugs

Thu, 21 Aug 2008 20:33:29 GMT

Don't test to remove bugs!

Testing is to defect location as trial division is to finding primes. Just as it's very slow to show a number is prime by doing trial division it is very expensive to expose bugs in your program simply through blind testing.

Yet I've found that a lot of people do think that this sort of blind testing is the best way to find and remove bugs from their program.

Just like there are mathematical short-cuts that allow you to find primes quickly, defects have properties you can exploit that help you find them faster.

There are a number of useful facts about software defects:

80% of the defects come from 20% of the code.
High cyclometric complexity is a good predictor of defects.
Even if your test coverage was 100%, roughly 35% of defects emerge from missing logic paths.
Another 40% of defects come from the execution of a unique combination of logic paths.
There are errors that most programmers tend to make.
The longer you leave a bug, the more expensive it is to fix it.

These facts paint a pretty important picture; there is a much more cost effective way to a lower defect rate.

I've come to the opinion that the biggest generator of defects in a piece of software is the design of the program. The fact that 80% of the bugs live in 20% of the code suggests that this must be true. What's special about that 20%? Why is it so buggy?

It's because this code contains bad abstractions, low cohesion, high cyclometric complexity and tight coupling. It's badly designed!

Fortunately, these facts give us a clear strategy for reducing defects. All we have to do is improve the design of the program. The better your design, the simpler your code will be. The simpler your code will be, the less bugs it will contain.

The simplification of one abstraction may result in the simplification of other dependent abstractions. Success often breed success and you can quickly find you have a radically simpler program and consequentially far fewer defects.

It is clear that there are common defects that all developers make. They tend to be simple things, such as not checking a pointer for null before dereferencing it, or using an out of range index on an array. This implies that defect check-lists are a good way to keeping common defects out. It also implies that there may be ways to automate detection of these defects at check-in.

If high cyclometric complexity indicates the presence of bugs, perhaps you should ban checkins of routines that score highly.

So if the purpose of testing is not to find defects then what is it for? It's for confirming that the specification has been implemented correctly.

Test-Driven-Development (TTD) doesn't produce high-quality code because it finds bugs; although that is a nice side effect. It increases the quality of the code because in order to unit-test code you have to make sure your code has low coupling, high cohesion and low cyclometric complexity. You have to have a reasonably good design.

When you're doing TTD what you're actually doing is writing an executable version of the requirements document. You're specifying what the behaviour of the program is intended to be before you actually write the program.

This is the purpose of testing; to ensure the requirements have been fully implemented.

Is Object Orientation Working?

Sat, 12 Jul 2008 15:47:29 GMT

Does object orientation really work?

The vast majority of software that people write inside a business talks to a database. Yet object orientated programming does not gel well with databases. The code that looks clean and simple in a modern object orientated program looks positively terrible when you look at the traffic that code causes at the database.

It's called the object-relational impedence mismatch. The often aired solution is to use some sort of persistence framework to map your classes to a collection of database tables. There are a variety of packages that do this: NHibenate, Gentle.Net, JPersist etc.

The problem is that none of them work too well. The writers of these packages are much like the inventor of this machine:

Solving the impedance mismatch problem is the computer science equivalent of the perpetual motion machine. You can't get it to work because it's impossible; they are simply incompatible ways of working.

A database is essentially a truth reasoning engine. You take a collection of facts, a series of inference rules and you use these to ask it questions. A object orientated language is essentially about setting up classes of object; templates, if you like, from which instances are created. These instances may be related to each other in a UML diagram but this is not enforced strictly in the code. You certainly can't do reasoning on them in the same way that you might a collection of tables. Moreover, what does inheritance really mean to a database?

When you try and make the square peg match the round hole, what you end up getting is a really leaky abstraction. The persistence framework writers would love you to believe that you can "forget" about the database and just write lovely object orientated code free of any worry about the vagaries of how a database works. For a while, they might be right but you can bet your bottom dollar that one day that database is going to bitch slap you in some way. Whether it's a particular query running very slow, a bone-headed auto-generated schema or trying to understand why databases don't understand inheritance - it's going to bitch slap you and when it does it'll ask you for interest you owe it for not treating it with the proper respect.

There are people who acknowledge this problem and propose that the solution is an object database. The problem with this solution is that while object databases are fast for only a very narrow selection of queries. This is because the way they work is via pointers from one object to another. Traversing these pointers can become a bottleneck under more general use-cases.

My solution is a bit more radical. My view is that the database, not the programming language, is the primary tool in solving business problems. The way to solve the impedance mismatch is to ditch object orientation and follow the conventions of the database.

A case in point, no impedance mismatch exists in C because it is not object orientated. We programmed with procedural languages for decades before object orientation turned up - it's not like programming is suddenly vastly harder when we remove object-orientation.

Actually, I'd go further. My admittedly heretical view is that object orientated design is a profound waste of time. I don't believe that object databases are the right answer because I don't think object orientation is the right way to design a program that talks to a database. It takes a colossal amount of object orientated code to achieve anything useful and when it does so, it usually punishes the very thing that's the hardest, most expensive, to scale: the database.

Think about this for a second, you have to work out how you're going to decompose your program in to classes then you're going to have to type out those classes (or if you're bit smarter you have a code generation tool to do this). Then you need to write tests the exercise that object model; large parts of which you probably won't really re-use in other scenarios.

Honestly, what's so wrong with just manipulating the data directly? What's wrong with taking the data from a form and doing a straight insert?

I think it comes from the fact that people think the object model is more maintainable and more reusable.

Is it really more maintainable? If you change the database you're going to have to update your object model in all but the most trivial of changes. Ahh, you say but a change is business logic is easier in the object model! Is that really true? A change in business logic implies a change in a database transition. It has to in order for that change to be recorded anywhere. Who's to say that updating a direct access call is going to be any more difficult than updating the object model?

Secondly, you should only make reusable components you actually do reuse. An object model may be flexible but does every problem need that flexibility? Is it worth spending all that time to create that flexibility where no flexibility is required?

This has a real cost too, in Robert Glass' "Facts and Fallacies of Software Engineering", Fact 18 says this:

There are two "rules of three" in reuse: (a) It is three times more difficult to build a reusable component as a single use component. and (b) a reusable component should be tried out in three different applications before it will be sufficiently generate to accept into a reuse library.

Building a flexible object model will probably cost you more money per unit of functionality. Think about that!

That aside, there's a deeper point here. For a computation to have any impact on the world, it must mutate the state of the machine it's running on. The goal of computation, therefore, is to mutate the state of the machine in just the right way under just the right conditions to record useful information. In real world business programming the goal of a program is ultimately to mutate the database in just the right way under just the right conditions. The program does not exist to be a self-serving statue to object orientated perfection - it exists to manage state transitions in the database.

So why not use the tools the database gives us natively to do this job? Not only does it provide answers to really complicated queries quickly but it provide transaction support, views of the data, the ability to set complex rules to enforce constraints on entries. Taken together, these are features you'd be simply crazy to ignore. They're there to help us write solid applications. They're there to make our work easier!

And yet people do ignore it! I honestly think it's because most developers hate SQL. They've got this pathological desire to expunge every last SQL statement from their program. To me, it's absolutely necessary to understand this language. Just like you need to understand C to understand how high level languages talk to the low level hardware, you need a deep understanding of SQL to understand how databases represent information. You just do, it's not negotiable.

It's a problem domain specific language like XPath or Regular Expressions. No-one would dream of writing their own XML navigation language or their own pattern matching language. We already have well developed, ubiquitous mini-languages that solve these problems extremely well. If you try and solve this problem independently, it is guaranteed to be less flexible and less useful. Yet when it comes to the problem of how to get data in to and out of your application, we invest a huge amount of time in solving problems already solved completely by using the database in the way it's intended to be used.

If object orientation is the casualty of giving proper respect to the database, then so be it.

Why Exceptions Suck

Mon, 30 Jun 2008 21:26:36 GMT

Why Exceptions Suck

Introduction

In previous posts I've focused on the importance of removing cyclometric complexity from your software and how that can dramatically reduce the defect rate.

This post is related to this theme, but rather than talking solely about the cyclometric complexity I want to tackle a particularly egregious type of complexity that exists in most modern programming languages: structured exception handling.

What is structured exception handling?

Python, C# and Java all have structured exception handling. In C# and Java you find code that looks like this:

    try
    {
        // Some IO operation.
    }
    catch(IOException e)
    {
        // Do something
    }

The idea is that if something exceptional occurs within the body of the try block, we can catch one or more exceptions and handle them. If there is no try/catch block in the routine you're currently in, the exception will "bubble" up the call stack, with the hope that it will be handled by some catch block. If it isn't, the program crashes.

The idea behind this is that we can clearly delineate error handling code from normal unexceptional code.

Why do they suck? - The Theory

Exceptions considered harmful

In 1968 Edagar W. Dijkstra had a letter published called "Go-to statement considered harmful." In this letter he gave a detailed account of why he thought that the GOTO statement in high-level languages should be abolished.

The central premise of his letter was that when designing a program, it should be possible to uniquely define a "meaningful" position in a program by looking at a few basic co-ordinates. Such as the current line number, the current call stack, the value of any loop counters.

The GOTO statement allows code that completely destroys any meaningful co-ordinate system. For example, it is legal to jump to code within a loop construct. In this case, how do you know whether the entry point was the natural start of the loop or whether you jumped in to it?

You could add another co-ordinate for each GOTO statement in your program, but then your program's control flow becomes dependent on a vast array of GOTO branch conditions.

Exceptions are a super GOTO. Rather than allowing you to jump anywhere in the position of the procedure, they allow you to teleport up the call stack to any number of handlers.

At first, it seems like we have adhered to the advice of Dijkstra because the exception code will only be executed when an exception has been thrown. There is no other way to have that code be executed. We could add the "CurrentException" item to the laundry list of co-ordinates and be on our way.

I don't think this is a valid defence of structured exception handling. While we have probably adhered to the recommendation of Dijkstra technically, we have certainly violated the spirit of it. The whole reason he wanted a meaningful co-ordinate system for tracing program execution was so that we can reason about the code. I would say that exceptions are much more difficult to reason about than return codes.

Exceptions make reasoning about a program hard

Let's say I throw an exception. How do I know as a programmer where that error is going to get handled? In most languages, the answer is "you don't know." The exception might not be handled at all, which will result in a program crash. If this is not the case, the exception might be handled in any of the routines up the call stack.

This is where the real pain begins, since the exception may not be handled in the routines that directly call your routine. They may be handled much higher up in routines that are functionally unrelated to the purpose of your code. So it is not enough to search all the code for uses of your routine, you have to search for every routine that might, on one code path, result in the execution of your routine! How on earth do you reason about such a construct? It's also worth bearing in mind that different code paths may have vastly different error handling logic, so you'd have to develop correctness proofs for each handler.

Checked exceptions the answer?

It's worth pointing out that the writers of Java knew about this and developed something called checked exceptions. Checked exceptions are exceptions that you are forced to provide a handler for. This allows you to trivially show that some types of exception are always handled, however, this does not tell you where they will be handled and whether the handling procedure is correct for that type of problem. Checked exceptions are certainly better than unchecked exceptions, since you know they must be handled but even so they're just as hard to reason about as their unchecked counter-parts.

But even checked exceptions are evil..

The problem with checked exceptions is not such much the theory but the practice of how they get used.

When you force a lazy developer to do something, what happens is they reach for the quickest way to make the compiler happy. In this case, that means indiscriminately catching all exceptions!

Now you have code that'll never fail but then it'll never work properly either.

Exception throwing code is invisible

How do I know whether a routine is going to throw an exception? Resolving this question requires inspecting my routine for any thrown exceptions, then inspecting the routines my routine calls for any exception and so on recursively until we have a definitive list. This is true even for Java since handlers are not forced for unchecked exceptions.

This is a huge amount of work to perform when writing a routine because even if the library is well documented, you usually don't have complete access to the source code. This means that you have no idea what pre-conditions you need to fulfil to avoid the exception. How on earth can you reason about your code when you don't have access to that information?

Throwing exceptions breaks encapsulation

To some degree error reporting always break encapsulation. It's important that a routine is able to signal that it has failed and doing so will obviously communicate something about how the routines work to the outside world. The problem is that exceptions are promiscuous in who they will communicate this information to. They will communicate it anybody on the call stack and they will communicate it to modules totally unrelated to the routine that threw the exception in the first place.

Consider an example where you've decided to use flat files as a store for database tables. For some reason, one of the table files is missing so a SQL query throws a "FileNotFoundException", this exception communicates the fact that files are used as the underlying data store to parts of the system that have no business knowing how the database works.

Most people designing a library would say that throwing a FileNotFoundException is not what you want to do. Instead, you want to throw a TableNotFoundException, with FileNotFoundException as the inner exception. Okay, fair enough. But what if some tables are accessed over a network connection and others are stored locally and I want to be able to distinguish between these two types of failure?

There is no way to distinguish between these errors without leaking information about the implementation.

Being able to distinguish between them is important. If the table doesn't exist on disk, then it probably never will, so the correct action is to gracefully exit and report an error. However, you might decide that if the network is unavailable then it is may be a temporary connection issue so it is worth retrying whatever we were doing. These are two different failure modes.

This might seem a bit of an contrived example but I see this pattern fairly regularly myself, particularly in code that manages a long running transaction.

It's true that if I was using return codes, I'd have to break encapsulation as well however only callers to that routine would be able to notice the broken encapsulation. With structured exception handling, all the routines on the call stack can potentially see that broken encapsulation. With structured exception handling, the break leaks much further.

Why do they suck? - In Practice

The theory of why exceptions suck is just the start of the problems with exceptions. What happens in real world programs is significantly more nasty. The problem with exceptions is they encourage developers to do some really bad things. In this section I'm going to look at some of the patterns I've seen.

Paranoid coding

Most developers are under serious time pressure to deliver a working product. Given the difficultly of establishing which routines will throw what exceptions, they often don't bother finding out at all.

This leads to bizarre "just in case" code and this leads to programs that are difficult to debug.

For example:

public static decimal ComputeSmokerPercentage(PersonCollection personCollection)
{

	try
	{
		int numberOfPeople = personCollection.Count;
		int numberOfSmokers = personCollection.NumOfSmokers;
		return (Convert.ToDecimal(numberOfSmokers)/Convert.ToDecimal(numberOfPeople)) * 100M;
	}
	catch(Exception)
	{

		// Exception might be thrown, caught so that it doesn't leak out.
		return 0M;
	}

}

Because they have absolutely no idea what exceptions might be thrown, they just wrap every non-trivial piece of code in a try/catch and try and attempt to pick a good default for when the routine fails. It's a sign the developer doesn't really know what's going on in their program; they're asking the computer to just blindly handle any errors in their logic. That's never a good policy because if you don't understand why each failure occurs, you can't be certain that your default case is really the right thing to do in all of the cases.

Consider this bug, which is paraphrased from a bug I actually found at work:

public static Collection<Widget> RetrieveList(int companyId)
{
	try
	{
	
		return Broker.RetrieveList<Widget>(new Key(COMPANY_ID, companyId));
	
	}
	catch(Exception)
	{
		return new Collection<Widget>();
        }
}

In this case, the choice of default obscured a problem with the configuration-store for the persistence framework. Since the exception wasn't logged anywhere or wasn't even checked to see what type of error it was, it frustrated the debugging process for longer than it should have.

If the developer had thought about what exceptions could be thrown, he might have made the configuration exception result in a program exit with a clear message. In some cases, you might even have a body of code that never produces an exception but is in a try/catch block none-the-less. This just makes it confusing and hard to read the code.

People use this pattern because they believe that catching the exception means their program will "never fail." I've said it once and I'll say it again: Yes your program probably won't crash, but it will probably never work correctly either.

Failure to avoid exceptions

An exception avoided is worth fifty caught. Handling exceptions is expensive in any language. As soon as you throw an exception, the program has to unwind the stack in an attempt to find a handler. Moreover, code in a try/catch block can't be optimised by just-in-time compilers very easily.

One of the key failure modes of structured exception handling is the fact the people do not attempt to avoid exceptions. In fact, they'll march in to them without barely a second thought.

Consider the following code snippet:

public int CalculateDamage(Player player, int hitSize)
{
	int returnedDamage = hitSize;

	try
	{
		if (player.PlayerClass == PlayerClass.Orc)
		{
			returnedDamage = hitSize * 0.5;
		}
	}
	catch(Exception)
	{
	}

       	return returnedDamage;

}

The exception the programmer is guarding against is the null reference exception. It would occur when player is referenced in the comparison against PlayerClass enum.

The first problem with this code is that it is hard to work out what it does. For example, consider this question: under what circumstances will the catch block be hit? You have to do mental gymnastics to get at the underlying reason for the catch block's existence.

Isn't this a lot simpler?

public int CalculateDamage(Player player, int hitSize)
{
       	int returnedDamage = hitSize;
       
	if (player != null && player.Class == Class.Orc)
	{
    		returnedDamage = hitSize * 0.5;
	}

	return returnedDamage;
}

Not only is the routine shorter, but it's also easier to understand.

I still don't like the routine. It's clear that if a null player is passed to this routine then there is clearly a programming defect that the user needs to know about. There is no sense in hiding the bad behaviour behind a default.

Yet, in the vast majority of cases where I see this sort of SEH, this is exactly what the programmer does.

This leads to buggy programs where the root cause of a crash is difficult to determine. The program will wander on for some time, strolling from one exception handler to the next, until eventually an exception leaks out and crashes the entire program. Quite often, the place where the program finally dies is totally unrelated to the root cause of the problem.

And SEH was meant to make writing reliable programs easier?

Can't this be fixed by hiring better developers?

Many notable people in the software industry think that all our problems can be solved by hiring practices. I think this is a fallacy.

For a start, unless you're Google, attracting talent is hard.

Even Joel Spolsky of Joel on Software fame has trouble attracting talent. He's arguably one of the best communicators in our industry. Joel's opinions on hiring are well known. He's written entire books about the importance of hiring the right people.

However, even Joel is starting to learn the hard way that the best and brightest minds do not want to work on a bug-tracker. Joel isn't writing cool software. Just like 95% of the products in the world his product is pretty boring.

I pick on Joel not because I have a grudge to bear with him; quite the opposite in fact since I enjoy his writing and his podcasts. I pick on Joel because he is loudist advocate of the "best developers solve all philosophy." For all his talent both in programming and in communication he can't change the force of gravity. Most of he products in the world are not sexy and it's going to be hard for him to attract talent.

Given that all the really outstanding developers are at places like Google and Microsoft, a small company like Fogcreek or the one I work for can't really can't fix the problem by hiring the best. Rather than saying "The best developer can use Structured Exception Handling properly, so we'll hire them.." What are we going to say about the 75% of average developers who mis-use the technology on a daily basis? What tools are we going to give the much maligned mediocre developer to make sure they don't screw up their programs?

To me, that is an absolutely essential question. This field doesn't advance much by making the super-star hotter but by improving the median developer. Super-stars develop 1% of the world's software. The median developers develop the rest.

So yes, structured exception handling can be used effectively by the best but that is entirely beside the point. The real question is whether a given language feature is ergonomic enough to be used by the rest without compromising the programs they write?

It's in this sense that I think return-codes are better than structured exception handling. It's much harder to use a return code badly.

Reducing Complexity Using Higher-Order Functions

Sat, 03 May 2008 16:43:17 GMT

Reducing complexity using higher-order functions

Introduction

In my last post, I talked about the correlation between the overall cyclometric complexity and the defect count in your software.

I listed a few ways that you can reduce the overall complexity. One of those ways was to use the Map/Reduce/Filter patterns extensively in your software. In this article, I'm going to show you how to use these routines to dramatically lower your overall cyclometric complexity.

Map, Reduce and Filter are just routines that describe common behavioural patterns in programs. They all share a crucial similarity; they take as one of their arguments a function which is executed within the routine itself.

As we shall see, this innovation will allow you to reduce the cyclometric complexity of your application. Sometimes the reduction can be very dramatic.

This concept of passing routines as arguments to other routines is not new, it's been around in languages such as Haskell, Lisp and Erlang for decades. The lack of side-effects in these programing languages force you to think about functions in a more enlightened way. This sort of thinking is now only just becoming mainstream in common programming languages such as Python, Java and C#.

Anonymous Routines

When we write a routine during our normal programming lives, we usually explicitly name them, much like this snippet below:

	def add(a,b):
		return a+b

	print add(6, 10)

The vast majority of program code written in modern software is of this form; variables are passed into routines, these routine do computation on this data and return a result. These routines have a specific name and access rules, so they can be reused within the framework of your program.

But what if rather than merely having data stored in the arguments of a function, we could have routines themselves. Take a look at the following snippet:

	def construct_adder(valueToAdd):
		return lambda y: y + valueToAdd
  
	adder = construct_adder(10)
	print adder(6)

What's going on here? Well, the routine called "construct_adder" is a special sort of routine. It is a routine that returns another routine. We call any routine that takes another function as a parameter or returns a function a "higher-order function." In this case, construct_adder returns a routine that adds the number specified in the "valueToAdd" argument to whatever is passed to it. This routine is assigned to the variable "adder", which is executed in the print statement on the next line.

The key point here is that routine is dynamically constructed at run-time. The routine that we've created doesn't have a name in the traditional sense. The only way it can be referenced is through the variable it assigned to, in this case "adder." Once "adder" goes out of scope, the routine no longer exists and can no longer be executed.

The magic occurs in the "lambda" expression in the construct_adder routine. I don't want to get too hung up on the syntax, but essentially the lambda says, "construct an in-line function here, with y as an argument." The result of the function is the evaluation of the expression to the right of the colon.

While the example given above is a pretty superficial, the ability to construct custom routines at run-time using user supplied data is extremely powerful. It is a massive cyclometric complexity slayer. In this rest of this article, I'm going to try and give you a flavour how you can use anonymous functions to reduce the cyclometric complexity of your programs.

Reducing complexity by using anonymous routines with higher-order functions

Using the Filter routine

One of the first things you learn when you start writing software is how to do a loop. The vast majority of places where you see a loop, the loop is usually iterating over a collection of things and performing some operation. These routine often have a many decision points. They are decision points that can be culled, if higher-order functions are properly deployed.

Consider this Python snippet:

	def get_matching_words(startswith, wordlist):
    		result=[]
    		for word in wordlist:
		      if word.startswith(startswith):
			      result.append(word)
    		return result

The aim of this routine is to return a list of all the words in "wordlist" that start with the value passed in the "startswith" argument.

This routine has a cyclometric complexity of three. There's a code path for the case where wordList is empty. There's a code path for where there are items in the word list but there are no matches, and finally there is a code path for when there are items in the word list that do match, and the result gets appended.

You can do the same job in a single line using the Filter routine:

	def get_matching_words(startswith, wordlist):
    		return filter(lambda word: word.startswith(startswith), wordlist)

The filter routine takes two arguments.

The first is a routine that takes a single argument. That routine must return "True" or "False." Filter than executes that routine, passing each member of "wordlist" to the routine. If the routine returns true for that entry, the item is added to the list returned by filter. If the routine returns false, the item is omitted from the list.

The second argument is the collection of items to run the routine on.

In the case above, I defined a single anonymous routine that checks whether the argument passed to it starts with the value specified by "startswith."

It's pretty clear that this program is equivalent to the longer snippet above, except that this routine has a cyclometric complexity of one. Your immediate thought here might be that I've just moved the cyclometric complexity in to the inbuilt "filter" routine. Yes, you're correct but bear in mind that there are a whole host of problems that can be solved using the filter routine. The complexity increase of your program when using filter is constant, regardless of how many times you use it. Each time you use a for loop, you're going to increase the complexity by at least two. Once for the code path represented by the body of the loop and one for case where there are no items to loop through.

Another objection that might be raised is that loops are easy and that removing complexity in this way is not focusing on the real cause of bugs. I disagree, people do screw up loops and they're easier to get right if you use higher-order functions.

Using Map and Reduce

When I was writing this article, I decided to go and hunt through the code base at work and find some real-life loops that we can apply these ideas to. For copyright reasons, I can't post the exact code. However, I did come across a loop that is easy to screw up and looks much neater using Map and Reduce. I've sort of translated it in to python:

Take a look at this code snippet:

      def construct_list(list):
	      result = ""

	      for counter in range(0,len(list)):
		      result += str(list[counter])
		      if counter<len(list)-1:
			      result+=","
	      return result

The purpose of this routine is to concatenate a list of integers in to a string separated by commas. In our C# code, this routine was used to build a string for use inside a "in" clause in a SQL select statement.

It's very easy to screw up this routine. You can easily get an off by one error on both the "range" routine at the top of the loop and the condition that decides whether to add a comma or not in the middle of the loop.

The snippet above has a cyclometric complexity of three.

It might seem hard to get the complexity below this value, but let's compare this with what we can do with map and reduce:

      def construct_list(list):
	      if len(list) == 0:
		      return ""
      
	      string_list = map(lambda item: str(item), list)
	      return reduce(lambda firstItem,secondItem: firstItem + "," + 
secondItem, string_list)

This routine has two phases, first map is called. What this does is for each element in the list, it runs the function passed to it and maps its result to a duplicate list. This list is returned when the map routine has completed and is assigned to string_list, in my program. I've simply used map to convert all the items in the list to strings. In the "reduce" phase, we reduce the list down to a single value.

This routine has a cyclometric complexity of two. So we've made a saving of one over the first snippet. We could remove the if check at the top of the snippet but doing so would mean that if we passed an empty list to the routine, it will cause an error because reduce can't handle empty lists. One possible approach around this is to write a wrapper routine around reduce t handle empty lists. You could then handle all list loops using this new routine, with no further complexity cost.

The clearest way to write this routine is probably to use recursion, but even this has a cyclometric complexity of three:

      def construct_list_recursive(list):
	      if len(list) 
== 0:
		      return ""

	      if len(list) == 1:
		      return str(list[0])

	      return construct_list_recursive(list[:-1]) + "," + str(list[-1])

It's clear that when the goal is to reduce cyclometric complexity, it's hard to beat map/reduce.

Passing routines as arguments to other routines

So far, we've looked at reducing cyclometric complexity by focusing on Map, Reduce and Filter. However, these routines are just higher-order functions provided by the Python standard library. Even bigger savings can be made if you write your own higher-order functions.

Take for example, code that looks like this:

	def insert_meta_data(data):
		title_field = "Title"
		if data.has_key( title_field) and len(data[artist_field]) <= 30:
			insert_rowdata(title_field, data[title_field])
		
		artist_field="Artist"
		if data.has_key(artist_field) and len(data[artist_field]) <= 
50:
			insert_rowdata(artist_field, data[artist_field])
		
		release_date_field = "Release Date"       
		if data.has_key(release_date_field) and valid_release_data(release_date_field):
			insert_rowdata(release_date, data[release_date_field])
	
	..... The code continues for twenty such fields .....

The point I want to illustrate with the snippet above is that there may be different validation rules required for each field. When you have code like this, it precludes looping through some list of parameters and validating each one in turn, because each one has slightly different constraints.

Now imagine that we want to implement an update routine. There are two ways to do this traditionally. The most obvious way is to simply duplicate the checks in both routines. This clearly has a massive cyclometric complexity penalty and duplicates code so that approach should be avoided at all cost.

The next is to rewrite the routine so that validation occurs before insertion. The validation step would produce a list of fields that are valid for insert/update and both routines could share a call to this validation logic. Each body of code would loop through the fields, adding or updating them as you go. Each of these routines must have a minimum cyclometric complexity of two, so you have a minimum complexity spend of four to implement it this way. It's also quite a bit more complicated, not in the cyclometric sense, but in the mental sense. There's more to fit in your head with this design.

However, there is a dramatically more powerful way to solve this problem. You could do this:

	
	def change_meta_data(data, update_routine):
		title_field = "Title"
		if data.has_key(title_field) and len(data[artist_field]) <= 30:
			update_routine(title_field, data[title_field])
	
		artist_field = "Artist"
		if data.has_key(artist_field) and len(data[artist_field]) <= 50:
			update_routine(artist_field, data[artist_field]
	
		release_date_field = "Release Date"
		if data.has_key(release_date_field) and valid_release_data(release_date_field):
			update_routine(release_date, data[release_date_field])

The second argument contains a routine that will be executed during the body of the code. You now pass in which routine you want to run within change_meta_data and it will either update or insert a row. Notice how the cyclometric complexity of the change_meta_data routine has not gone up, even though the routine can now operate in two (or more!) distinct modes. The most obvious choice of update_routine would be something that inserts or updates a table row, but this doesn't have to be case. For example, the update_routine could be inserting or updating an in-memory cache.

The point is, we can decide at run time what gets executed within this routine and we can get this ability to choose without having to introduce any additional complexity.

Conclusion

In this article, I wanted give you a taste of how higher-order functions can help you control complexity. With a small amount of careful thought, it is possible to chop out a considerable amount of cyclometric complexity contained within your application.

I hope I've convinced you that higher-order functions are worth investigating in your project. They're a powerful tool that helps you to write less code and the less code you write the less you have to debug and maintain.

Cyclometric Complexity and Defects

Mon, 31 Mar 2008 22:13:23 GMT

Cyclometric Complexity and defects

When we say a program is complex, what exactly do we mean by complex? In future posts I'm going to keep returning to this issue, but today I'm going to deal with one type of complexity: cyclometric complexity.

The cyclometric complexity of a routine is the number of independent code-paths through the procedure. Examine the following Python routine:

				def f(x):
				if x==0:
				print "Zero"
				if x>0:
				print "Greater than Zero"
				if x<0:
				print "Less than zero"

Assuming that we pass in an integer to this routine, there are three distinct code paths. One for when x is zero, one for when x is greater than zero and one for when x is less than zero. The cyclometric complexity of this routine is therefore three. The larger the cyclometric complexity of a routine the more complex it is.

You can then add up the total cyclometric complexity over all the routines in a program to get a feel for the complexity of the entire program. For even the smallest of programs the cyclometric complexity is likely to be in well in to the thousands. In large projects, it will probably run in to tens of a millions.

This seems to tie in nicely of what our intuition tells us complexity should be. The more decision points your program has the more complex your piece of software is.

Does lowering cyclometric complexity reduce defects?

We know that complexity is the source of bugs. A natural question to ask is whether working to reduce the cyclometric complexity of a program also reduces the defect count in that program?

In Steve McConell's pivotal book, Code Complete, he has this to say about Cyclometric complexity and defects:

Control-flow complexity is important because it has been correlated with low reliability and frequent errors (McCabe 1976, Shen et al. 1985). William T. Ward reported a significant gain in software reliability resulting from using McCabe's complexity metric at Hewlett Packard (1986b). McCabe's metric was used on one 77,000-line program to identify problem areas. The program had a post-release defect rate of 0.31 defects per thousand lines of code. A 127,000 line program had a post-release defect rate of 0.02 defects per thousand lines of code. Ward reported that because of their lower complexity, both programs had substantially fewer defects than other programs at Hewlett-Packard. My own company, Construx Software has experienced similar results using complexity measures to identify problematic routines in the 2000s."

I think the verdict here is pretty resounding. Lowering the over-all cyclometric complexity of a program is going to reduce bugs considerably.

How do I go about lowering complexity?

The number of decision paths in your typical program is going to be huge. But reducing overall decision complexity can be achieved by aggressively adopting the following principles:

Reuse code really aggressively. Anywhere you see decisions repeated, you want to wrap that up in a separate routine even if it's only a few lines long.
Don't use short-circuit evaluation where you don't have to. This simply adds decision paths that aren't neccessary.
Try and reduce the number of return points in a routine. Ideally, there should only be one return point.
Take long, complicated routines and break them up in to smaller routines. Although this won't lower the overall complexity, it will make the code easier to reason about and therefore less likely to contain errors.
Try and use set operations such as map and reduce to reduce boiler plate decision logic (such as loops). You will be surpised how many problems fit in to the map/reduce model.
Use try/catch clauses sparingly. It's very easy to produce exceedingly complex control flow with exception handling.
Don't implement "speculative" features.

The primary message is that lowering complexity is about writing less code. The less code you write the less decision points there are. At every opportunity you should try to write less code.

How can I identify problematic routines?

There are a number of tools in a variety of languages that allow you to flag routines that have a high cylcometric complexity. The rule of thumb is to rewrite any routine where you have a cyclometric complexity of greater than ten.

You can use the following tools in your organisation to measure the cyclometric complexity of your routines:

Is refactoring worth the money?

Mon, 24 Mar 2008 18:47:00 GMT

Introduction

It is generally appreciated at an abstract level that the idea of improving the design of code that is already written is a good idea. I have the heart of a purist and I like to see nice, well structured code but I also work in a small business. I am acutely aware that refactoring costs time and time is most definitely money. So I ask, is refactoring worth the money?

The idea behind refactoring

Martin Folwer, in his famous book on refactoring, defined refactoring as:

"Refactoring is the process of changing a software system in such a way that is does not alter the external behavior of the code yet improves its internal structure. It is a displined way to clean up code that minimizes the changes of introducing bugs. In essence, when you refactor you are improving the design of code after it has been written."

Apart from getting a warm fuzzy feeling for code, why would I do this? Well, the principle argument for refactoring is that it speeds up the development of new features. The easier it is to develop a feature, the less money it costs.

The Hard Numbers - The optimistic

Okay, so if refactoring saves us time then we might want to do a major refactoring project so that we can increase productivity amongst our developers. Let's say, that for the sake of argument, that refactoring saves 10% on development time. Let's say and I plan to spend a single developer's time on the effort for three months to achieve this 10% efficiency gain. Let's call the money made on this venture "Account A".

Further, let's take the same time we were going to spend on refactoring and invest it in a low risk savings account. There is nothing special about this account; any bank's saving rate on this will do. I'm going to choose Northern Rock's saving rate which is roughly 6% in the UK. Let's call this "Account B."

At first, this might seem a bit of a weird account to have, but this is meant to represent the minimum "opportunity cost" of doing refactoring. The money you spent on refactoring could have been allocated elsewhere to make you money. The most basic way you could allocate the funds is by simply investing it at market rates. If refactoring can't beat basic investment, it isn't work doing.

Because days of man time cost a fixed amount of money, we can just calculate interest on the development days directly and save the additional effort of having to manipulate values in any given currency.

Here is the table of data comparing both accounts:

Year	Account A	Account B
0	0.00	90
1	36.53	95.40
2	73.05	101.12
3	109.58	107.19
4	146.10	113.62

There are a couple of things to note here. After two and a bit years, you will have paid back your capital expenditure. After nearly three years you will beat what you would have earned if you'd just invested that development time in the bank.

It's worth noting that by staying the course on Account B, the exponential gains of interest will eventually out pace Account A. This occurs fifty or so years after the refactoring was done. That data point is of little practical relveance since the product will probably not exist by that time.

After just under three years we're getting more money from our refactoring exercise than the trivial case of investing the money in a savings account. That looks like a solid medium term investment.

The Hard Numbers - The Pessimistic

I don't know of any studies that give the real productivity improvement of refactoring but I feel that 10% may be a little too generous. Let's say that refactoring improves developer productivity by 5% instead of 10%.

Our table of accounts table now looks like this:

Year	Account A	Account B
0	0.00	90.00
1	18.26	95.40
2	36.25	101.12
3	54.79	107.19
4	73.05	113.62
5	91.31	120.44
6	109.57	127.67
7	127.84	135.33
8	146.10	143.45

At this rate it takes a shade under five years to repay the initial cost of the refactoring. It takes about eight years for the refactoring to come out ahead of straight-forward investment.

In software, eight years is an age. If refactoring gives anything less than a 10% productivity improvement it probably isn't worth doing.

The Hard Numbers - Doing Refactoring every five years

Except, that when you're done refactoring, are you really done? As time goes on, code gets more hairy and every so often you boss might factor in time to refactor your code. Let's say your boss wants to do three months of refactoring every five years. Let's be say that refactoring improves your productivity by 8%. This is somewhere between the two figures given in this article and represent my gut feel for how much time a well factored program can save.

Here is how our accounts look now:

Year	Account A	Account B
0	0.00	90.00
1	29.22	95.40
2	58.44	101.12
3	87.66	107.19
4	116.88	113.62
5	56.10	120.44
6	85.32	127.67
7	114.54	135.33
8	143.76	143.45
9	172.98	152.05
10	112.20	161.18
11	141.42	170.85
12	170.64	181.10
12	73.05	113.62
13	199.86	191.96
14	229.08	203.48
15	168.30	215.69

Every five years, 90 is subtracted from Account A to cover the cost of refactoring. It is clear that over each cycle, you will cover the cost of the refactoring but in the long term, simply investing the cash will give you a better return on investment.

Conclusion

This article is a mixed bag. For refactoring to be a better investment than just banking the cash, you have to improve productivity by 5-8% or more. This is quite a big ask and I have big doubts this can be achieved in practice. If your opportunity cost is much higher than the figures represented by "Account B" then you're probably best spending the money on something else, like new features or defect reduction technqiues.

This analysis also assumes that refactoring was done perfectly and didn't introduce errors. If refactoring is done even slightly imperfectly, it is very easy to wipe out the productivity benefits of refactoring through the increased debugging cost.

So where is refactoring useful? Well, I would say that refactoring has utility where a code base, as a black box, works but is unmaintainable. The more unmaintainable the code base, the more refactoring is likely to give you. If a code-base is horribly buggy then a large amount of logic is going to have to be reworked in any event. Refactoring may help in spotting problems but the real solution is to rewrite the buggy modules - which is really new development rather than refactoring.

Personally, I think refactoring solves the wrong problem. If code is badly designed then some process resulted in that bad design. Fixing the process is going to be a lot more effective than continually fixing the code.

If you just refactor periodically without learning anything about why you needed to refactor in the first place, then you have missed an important opportunity to fix things.

It should be obvious that fixing mistakes before they even occur is the cheapest way to fix a mistake. Refactoring seeks to fix a preventable mistake that should have been prevented.

Coverage analysis and what it means to you

Sat, 08 Mar 2008 17:29:35 GMT

Coverage Analysis and what it means for you

Introduction

In my last post, I talked about how much bugs cost. In this post, I'm going to talk about how we can remove them. Removing bugs has always been a multi-faceted problem. Any bug removal strategy has to contain a number of separate attacks on the problem of bugginess.

Formal inspections have been shown to be the most effective way of removing bugs, both in terms of bugs discovered per hour of testing and the cost of finding those bugs, but formal inspections are only part of the solution.

In this post, I'm going to talk about another technique for rooting out bugs; namely code coverage.

What is code coverage testing?

When you write code you should always have a plan of how you're going to test that code. One question you can ask about your test suite is "How many code paths do my test cases cover?" Or perhaps, more precisely "What percentage of my code is exercised by the test suite?"

http://

Coverage analysis allows you to answer those questions. Most well used languages have at least one coverage analyser.

These coverage tools work by injecting instrumentation code in to your code which records when each statement in your program is executed. When the test suite is executed against this augmented code, a report is produced of all the statements executed when your test suite is run.

Here's a screen-shot of the NCover tool I use at work:

The blue bits show the statements that were covered by the test suite. The red bits shows the code that was not covered by the test suite.

If I get 100% coverage does that mean my program is bug free?

No, this is the poisonous side of coverage testing. It's very easy to turn it in to a game where the goal is to get 100% coverage. This is generally not a productive use of your time. The whole point of doing coverage analysis is to try and find untested areas of your code with the goal of finding bugs.

It's a well supported fact that 80% of the bugs reside in 20% of the code. If you go for 100% coverage of your code base you're going to spend a lot of money testing the modules that are not very buggy. You could take the same resources you're using to achieve 100% coverage and put them in to properly testing your buggy modules and you would remove a lot more bugs.

Another problem is that coverage analysis has is that it can only test code that is there. There are a great number of errors that are caused by code that is missing; code that doesn't exist.

For example, consider the following function written in Python. It tries to compute the roots of a quadratic equation of the form ax² + bx + c:

				def roots(a, b, c):
				root = (-b + math.sqrt(b**2 + 4*a*c)) / a
				return root

This code contains a few bugs. First, the inputs a, b, c are not checked to see if they're numbers. Second, even if 'a' is a number the program will crash if the value of 'a' is zero. These are both precondition errors and in a language like Python, with its duck-typing, we might let the first of these slide. However, not checking whether 'a' is zero is definitely a bug. There is an even more serious error: there are two roots to a quadratic equation and the code only gives you one of them.

All the defects I've pointed out are defects of omission. There is code missing from the program that means an error can occur in some circumstances. If you were blindly trying to get 100% code coverage, you'd probably miss all these defects. You can get a 100% coverage of this routine by just passing a=1, b=2, c=3 to the algorithm and you're done.

So what does coverage analysis actually tell me?

Coverage analysis tells you the areas of code that didn't get executed by your test suite. This means that your test suite is deficient in someway as it doesn't cover all the functionality.

It is important to use your coverage analyser as a tool to inform the development of further tests in your suite and not as an end to itself.

My approach to writing tests is to not use the coverage analyser on my first attempt at writing a test suite. After composing my first lot of tests, I then remove any bugs I've found and then run the coverage analyser.

This will normally throw up a few more features that need testing. I then code up these tests and check how I did again. Assuming I covered all the important scenarios, I then stop and move on to other types of testing.

The skill to correctly using a coverage analyser is knowing when to quit. If you quit too early, you'll leave too much code untested, if you quit too late then you're just wasting your time that could be spent on more focused, effective testing.