I’ve already talked about Patrick’s measures of code quality. His approach is to take a very direct mathematical and analytical approach to it. I appreciate the sophistication of the tools he’s using, but I think that the results of this kind of analysis need to be treated with the same level of suspicion as an analyst treats a set of accounts. e.g. Cash flow is hard to fake, but why are the accruals so low?
Ayende, on the other hand, argues for a very different approach. He lists as his measures of code quality:
- Peer code review (especially for things that are either tricky or complex)
- Reported bugs
- Ease of change over time
- Quality of deliverable product
Let’s be frank, only one of these is actually a metric: reported bugs. The others are a bit confused.
Peer Code Review
Let’s talk about peer code review. Peer code review is absolutely best practice, it’s a cracking and important way of improving code quality. But it’s a measure, it’s a practice. Now, there are measurable things that we can ask about the reviews:
- How often are code reviews carried out with developers in the same room?
- How often do code reviews identify a bug?
- How often do code reviews result in a change being rethought?
Even after that, you’d need to be very cautious about what you do with the metrics: a 0% defect rate and a 100% rate would probably both indicate code quality problems.
Code Quality Outputs
Two of the other “measures” that Oren refers to are actually the goals we’re trying to achieve: ease of change over time and the quality of the delivered product. Now, ultimately these are all we actually care about. Code quality itself is merely a means to an end: getting things done. But again, we have two problems: there’s no actual measures here, and we’re not actually looking at the code quality itself. Let’s remind ourselves of Martin Fowler’s dictum: you can’t measure productivity. Now, with an open source project, we have some advantages over the classic corporate development scenario: mostly that people are less likely to game the system. However, there are ways of achieving these aims that don’t involve improving the quality of the code base: I’ve improve the project velocity and project quality of legacy systems by spending a lot of time developing test suites for them.
Both of them suffer from the serious question “relative to what?”. It’s taken NHibernate 2.1 two years to hit GA. Does that mean the code quality is bad? No, probably not. There are any number of possible reasons for this: the changes were large and complex, the contributors couldn’t devote much time to it, sticking with 2.0 slowed things up, the project admininstrators didn’t regard going GA as a priority. I’m sure you could think of more without much trouble.
You can’t measure project velocity “in general”, because it depends what you want to do to the codebase. The release notes testify to a phenomenal amount of number of improvements, but the criteria API still isn’t LINQ-complete. Why is this? Well, it’s because the criteria API was developed well before LINQ was on anyone’s radar. With the best will in the world, sometimes you end up with the wrong abstraction.
Reported Bugs
Now here, we have something we can measure. I’ve already spoken about the dangers of doing so, so take that as read. The problem is, a quick look at JIRA doesn’t really seem to tell a good story. More than 10% of all logged issues are open. The vast majority of those are unassigned.
Some of these are really arguable. The two most popular open issues are a problem with Oracle stored procedures and a desire for the lazy loading of columns. Both can be filed under the category “working with legacy databases”. Do I care? No. Do the people voting on the issues care? I’d say yes. Have they submitted a patch? No. But then, submitting a patch would require you to be up to speed with the code base, which is where this whole discussion began.
So what is NHibernate’s Code Quality, then?
Frankly, I don’t know. A naive scan of the code reveals a code base structured according to good engineering practice, but that won’t reveal indirect and subtle dependencies. Ayende isn’t worried about the direct dependencies that do appear in the system, but it’s quite rare to meet a maintainer worried about this before he’s advocating a wholesale rewrite.
NHibernate, as a live port of a Java project, is special. Code quality, as with everything else, is context sensitive. However, whereas I’m not convinced Patrick’s metrics necessarily tell the story it initially appears, I’m still not convinced NHibernate has any measures of quality. That’s not the end of the world: you don’t necessarily need a process for something that isn’t broken. I’d keep an eye on JIRA, though.
Next, I want to talk about what readability and maintainability actually are. And I don’t want to talk about metrics.
Colour Coding 