One of my favourite old films is “Mr. Blandings Builds His Dream House” starring Cary Grant and Myrna Loy as a couple that decide to buy an old house and renovate it to create their dream house. It’s based on a true story. Of course nothing goes as hoped, they encounter all sorts of problems, and it ends up costing much more than they originally planned. They get their dream house in the end, though maybe not the one they started out hoping for.

I was reminded of this film recently when we decided to buy an old house in the Balkans mountains region, which we are now in the process of renovation. Doing this house renovation project made me think about the similarities and differences to my experiences in software development over the last forty odd years.

I think that a most real software development projects are more like Mr. Blanding’s experience than the nice clean straightforward development scenarios that software development and project management methodologies and tools are designed for. We often start with some existing software that after years of poor maintenance is in need of renovation and often encounter similar difficulties to those that Mr. Blandings encountered.

Great expectations

One of the first difficulties is unrealistic expectations. We start out as Mr. Blandings did, with a dream of turning our tumble-down old software into a perfect end-result, with a minimal expenditure of cost and time. It is not just the customer that has unrealistic expectations. In the early stages of software projects, in my experience, we software developers are equally liable to the wearing of “rose coloured spectacles”. We are optimistic that although the previous projects didn’t work out as well as we had hoped, the next project will not be like the previous ones, because we have learned from our mistakes, or quite often because we believe that some new language, tool, methodology, or magical pointed hat will really do what its promoters claim.

I believe a big factor in whether any project, software or otherwise, succeeds is whether it was ever realistic in the first place. Unfortunately most of our methodologies focus on how to satisfy customer needs, but don’t provide anything to help us to establish whether those needs can realistically be satisfied, and to negotiate realistic project goals before we start. Software developers are focused on technological aspects. Sales people are focused on getting a sale. Project or product managers may try to address this, but they are reliant on what they are told about what is needed by the sales side, and what they are told can be achieved by the technical side.

In order to know whether a project is realistic we need an accurate estimate of what the real cost of the software development will be. One truth that is rarely admitted, in my experience, is that not everything can be estimated. For example, when renovating an old house in the country, one thing you need is water. Mr. Blandings had to drill a bore-hole. The cost of this cannot be estimated, it depends on how far you have to drill, which you cannot know for certain in advance. Even if you do a geo-physical survey, it is not guaranteed that you will strike water. In the film they drilled down to where the water was expected to be found, but hit nothing, then had to decide whether to keep on drilling in the hope of hitting water lower down. We are hoping for better luck with our own house renovation. So far, we have just had a man with a couple of sticks walk about the garden!

In software projects, it is also the case that not everything can be estimated. However, usually everything is required to be estimated. Obviously whoever is providing the money wants an estimate of costs in advance. So, we are usually forced to estimates that are full of guesses. Software estimates are hopelessly inaccurate in general. Usually we under-estimate. Experienced managers will know this, and adjust accordingly. When I was working as a junior software developer a couple of years after graduation, I was asked to estimate the time needed for a software change. I gave my estimate to my manager who, being a wise old fellow, doubled it. He gave the estimate to his manager, who being equally wise, doubled it again. We didn’t win the work! Sadly managers are rarely wise, in my experience. They are more likely to put pressure on the software developers to reduce their estimates, than to increase them. So we usually start software projects with unrealistic estimates, and hence unrealistic expectations.

Knock it down and build a new one

A second difficulty that we encounter is the decision whether to update the old code base, or to re-build from scratch. This is a critical decision, but in my experience we rarely make it based on a systematic analysis or defined criteria.

Early on in the development of his dream house, Mr. Blandings received a shock. The builder told him the old house was structurally unsound and needed to be replaced with a newly built one. Whether that was true is an interesting question. In the real-life experience which the film is based on, the old house was not knocked down, and stood next to the new house for another 30 years – not bad for a structurally unsound building!

Something we have learned from our experience of planning the renovation of our house in the Balkans is that more often than not when asked about renovation of an old building builders and architects will say that it is better to knock it down and build a new one. I think that this is because that is preferable for them. They can build it using the methods that they are familiar with and understand, and don’t have to find craftsmen that can work with the building materials and techniques used in earlier times. They would rather you had the house that they would like to build, than the house that you would like to have.

Software developers do the same as builders, in my experience. When faced with an existing code base we will prefer to re-write it, using the languages and tools we are familiar with. Sometimes it is even the case that the re-write is with a language or tool that the developers would like to become familiar with. I was once involved in a project (as a safety engineer watching from the sidelines) in which the developers changed the programming language twice, in the space of a couple of years. I tend to believe that such decisions are driven more by what will look good on the programmer’s CV (resume) than by any technical consideration.

Although we often may think the best option is to re-write, it is I think not always the case. Existing code-bases have been refined over many years to fit the particular needs of an application and end-user. Even in a well documented system these small incremental refinements are not all documented. The code has also been proven in use over many years. It is likely to be quite efficient, being written in a time when processing time and memory space were not so readily available.

In a non-trivial application, to develop and refine an efficient and effective architecture is a time-consuming and difficult task. Too often we compare on the one hand the difficulties of understanding and improving old code, and on the other hand the effort needed to make “working software” that has the same basic features. This is not a like-for-like comparison. Our initial estimate is that the re-build will be quicker, but by the time we achieve an actually equivalent product, it isn’t. Mr. Blandings got a house, but it was a boxy design, lacking the attractive features of the original old house. By the time it had all the features that the Blandings wanted, it cost many times what was estimated.

We can see the same with our renovation of an old house in the Balkans. Often old houses have a very effective architecture, based on centuries of incremental improvement in the design, according to the specific needs of the location. For example, our old house has a design that keeps it cool inside in the summer, with a big overhanging tiled roof and a basement dug into the hillside. In summer when the outside temperature is 37 Celcius, inside it does not hit 30, with no use of air-conditioning. If we let builders knock it down and build the concrete box that they prefer, it would not be the case that they would include in the specification for their new build that it can achieve the same passive temperature management as the old one. Even such a primitive building material as wattle and daub (sticks and mud) has a benefit of moderating humidity levels in the building, creating more comfortable living environment and reducing the risk of condensation and mould. This benefit is unlikely to be replicated using most modern building materials. So, we are keen to keep as much of the benefits of the original architecture as we can, and have rejected the idea of a re-build.

The flower arranging room

A third difficulty is the natural tendency of the scope of any project to increase, and the complexity of introducing changes during development. At one point in the film Mr. Blandings is looking at the builder’s bill, and is shocked by the additional cost of including a “flower arranging room” that he asked to be added. He doesn’t understand how adding one small room can be so expensive. It was a small room, but it had sink in, so the plumbing had to be re-done, the floor reinforced, etc. Each small change had knock-on effects requiring further changes. This is a very familiar scenario in software engineering. A small new feature needs a small change to part of the architecture, which affects another part, and there is a chain of changes needed, so it takes much longer to complete than anyone had originally imagined. Often we don’t do impact analysis properly, but even if we do I think we don’t have good methods and tools for working out the overall impact of a change before we start. Sometimes we have a process telling us that we should do an impact analysis before deciding to make a change, but rarely any systematic way to do it.

Even if we did, by luck or judgement make a perfect analysis of the change impact, and our estimate of time and costs is good, that is not the end of it. We still have work out how to add a feature that will involve architectural changes in the middle of an ongoing project. This inevitably will affect the ongoing work, so we must manage the risk of introducing defects in that ongoing work. There are a few approaches we might adopt. We could do no work on the change until we have finished the ongoing work and released a build of software. We could branch the code-base, make the change on the branch and merge back in at a later time. We can pause work on all areas that could be affected by the architectural change until that change is complete. Or we could simply go ahead and make the changes while continuing with the ongoing work and rely on good communication between the software developers.

We often need to make a decision as to how to introduce a change to an ongoing software development. Getting this decision right is critical, but I think we don’t have good methods to support making the decision, that ensure we do a thorough analysis on the base of well-established criteria before deciding how to proceed. Bearing in mind the frequency with which changes occur during software development this is a major contribution to the failure of software projects and products. Configuration management and change control, although present in the process standards, and nearly always largely ignored both by management and software developers.

The Cindarellas of software

I’ll try to draw a conclusion from comparison of Mr. Blandings experience of building a dream house and my own experiences in software development.

We often have difficulty in achieving successful software projects and products because most of our software development methods and tools are for creating the code, and we have poor methods for making vital decisions that have a huge impact on the success of our software development. We put a lot of money and effort into improving the methods and tools we use for software development, but get little improvement, because of a lack of effective methods for more important activities such as: establishing realistic objectives and estimates; making informed decisions about dealing with existing code-bases; and managing the complexity arising from introducing changes in the course of software development.

It is true that some companies’ processes include these activities but they are usually little more than a re-phrasing of an ideal process from a standard or process model. Even if these activities are part of a company processes, in practice they do not get a fraction of the effort put into them as is expended in the processes for code creation. Not only do they get little effort spent on them, there is also little methodological and tool support available. These processes are the Cindarellas of software development. Only when we let Cindarella go to the ball will we be able to achieve better outcomes for our software projects and products. We don’t need cleverer programmers we need a fairy godmother!