Apples vs Oranges

A lot of the time when reading about software development, I used to end up pretty confused. It just seemed as if the world of software development that I was reading about was a totally different world to what I was experiencing day-to-day at work as a software developer, and when trying to use ideas that I read in my work, I was observing different results to what I should expect. Over time, I have come to understand why this is. It is because the jobs I end up in tend to differ in various ways to what gets typically written about.

Image courtesy of “e g g” on Flickr

In particular I have identified four “axes” along which my experience differs from the typical situation.

• Projects vs Products

A lot of what I learned at uni, the books I read, people I talk to and websites etc, assume that software is developed mostly as projects. That is, they have their own organisational structure (project teams existing alongside functional departments), their own managers and budgets, clear notions of what counts as success and failure, and most importantly a semi-fixed timespan or some date in the future at which the project finishes, or at least enters some phase (often the expensive maintenance phase) at which software development stops.

Apart from a couple of exceptions, almost all of my software development jobs have involved ongoing products rather than projects, no separate teams apart from the software development department, no separate budget, no clear notions of success or failure and no end date. The software will usually be continually updated, and expected to be so permanently. In project terms, such work resembles a permanent maintenance phase.

• In House or Outsourced vs Mass Market

In my course of study, we often compare two types of software development depending on customer involvement. The first is the classic in-house IT development, where the customer is someone, usually another department, within the same organisation, and the second is where a specialist software development or consultancy firm develops software for an external customer. I was never able to make much sense of this, until I realised that except for a couple of exceptions, the software I am typically involved in represents a third main type. Software that is sold on the open market, to potentially many customers rather than a single customer.

When discussing issues involving customer involvement like business and requirements analysis, market oriented software is typically very different to what gets discussed in most books on software development. One major difference is that instead of the customers telling the developers what they want, marketing attempts to convince customers what they need. Often rather than developing software to improve an existing business process, market oriented software attempts to provide whole new processes that often need to be sold as part of the package alongside the software itself.

• Domain Complexity vs Technical Complexity

For a long time I was annoyed at myself for not “getting” domain driven design, or at least not appreciating the power of the ubiquitous language. Until I realised that the DDD books are coming from the classical enterprise IT angle. In such environments it really is possible to talk to the customer in their own language about all levels of the software, not just the problem but the solution as well, from the database to the gui, and to a high level of detail within the business logic itself. This has however not been possible with the types of software I have worked on, where the most we can do with the customer language, or even the customer knowledge, is form a thin wrapper over the true technical complexity.

For example if the customer, say the owner of a security company, wants a software program to match faces on a security camera video feed with a database of stored photographs, then there is not much point getting into too much detail with the customer on exactly how we can solve his problem with a neural network, we should rather talk to other developers who may not know the domain language required to specify the problem, but understand the technical complexity required to implement the solution.

• Cowboy Coding vs Traditionally Managed Development

I would like to think that now-days lightweight and agile methods are making inroads into both “Ad Hoc” (a polite term for unmanaged, or at least non-professionally-managed) and traditionally managed (e.g. waterfall) development efforts, so this distinction is becoming less relevant, but over the last few years as I have tried to improve the way I (and in some cases others) develop software, I have found the advice available less realistic or relevant to the cowboy teams I tend to work in.

Especially earlier on in my career, by bosses were typically not trained to manage software development. They simply knew what they wanted, and requested it to be implemented. Any decision making beyond that, such as whether something should be done or not, how it should be done, or how to improve quality, was simply driven by a mixture of blind hope, guesswork and wishful thinking. Real seat of the pants stuff, I guess it would have been exciting had it not been so frustrating and wasteful.

The difference is most apparent when talking about migrating to agile methods for managing software development. Pretty much everything written about migrating to agile, is talking about migrating from “waterfall” to agile. Trying to use advice that works for waterfall teams, while helping to introduce agile to teams that are not used to a disciplined approach to managing their software development efforts is very likely to fail. It should be obvious really. A team migrating from a traditional to an agile approach needs to scale a heavy process down to size. A team of cowboys needs to build up a “process” (or culture is probably a more appropriate word) where none existed. Following inappropriate advice is likely to have the opposite effect to that intended!

These four axes are not totally independent. I have noticed that projects, for a single customer, than have solutions that can be fairly deeply specified side by side with the customer and have professional management tend to form one “clump”, and the mass-market, cowboy coded, technically complex products that I have worked on form another. The former are basically your classical enterprise business projects, focused on using data to achieve a fairly specific objective, and the way it moves between the database, processing, and user, while the latter is pretty much everything else, including consumer software, device drivers, software sold as part of a hardware package, or to be used to enable a certain profession to work in new ways and solve a wide variety of objectives.

The latter category seems like it should be a much larger, richer, more varied environment, so I wonder why so much that is written seems to assume the perspective of the classical enterprise IT project? I still don’t really know.

Because so much of what is written comes from the perspective of single-customer-driven, managed, projects, yet so much of my experience comes from significantly different environments, I guess it is important to state where I come from so any differences in perspective can be explained. I also hope that future posts will help fill a void, and provide information for developers who work outside of the classical IT environment.

Quotes

Here are some quotes that I used to have on my wall in my old job. I don’t necessarily agree with all of them, some of them are interesting because they summarize points of view that were once widely held, but now serve to remind us how things change. However some of the oldest quotes are also the ones most valid today.

“Simplicity is prerequisite for reliability “
- Edsger W. Dijkstra

“The process of preparing programs for a digital computer is especially attractive, not only because it can be economically and scientifically rewarding, but also because it can be an aesthetic experience much like composing poetry or music.”
- Donald Knuth

“The perfect project plan is possible if one first documents a list of all the unknowns.”
- Bill Langley

“Complexity kills. It sucks the life out of developers, it makes products difficult to plan, build and test, it introduces security challenges and it causes end-user and administrator frustration.”
- Ray Ozzie

“Copy and paste is a design error”
- David Parnas

“There are two ways to write error-free programs; only the third works.”
- Alan J. Perlis

“Technology is dominated by two types of people: Those who understand what they do not manage. Those who manage what they do not understand.”
- Putt’s Law

“Ugly programs are like ugly suspension bridges: they’re much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code.”
- Eric S. Raymond

“You know you’ve achieved perfection in design, not when you have nothing more to add, but when you have nothing more to take away.”
- Antoine de Saint-Exupery, Wind, Sand and Stars

“Without requirements or design, programming is the art of adding bugs to an empty text file.”
- Louis Srygley

“Computer programming is tremendous fun. Like music, it is a skill that derives from an unknown blend of innate talent and constant practice. Like drawing, it can be shaped to a variety of ends – commercial, artistic, and pure entertainment. Programmers have a well-deserved reputation for working long hours but are rarely credited with being driven by creative fevers. Programmers talk about software development on weekends, vacations, and over meals not because they lack imagination, but because their imagination reveals worlds that others cannot see.”
- Larry O’Brien and Bruce Eckel in Thinking in C#

“It is not necessary to change. Survival is not mandatory.”
- Edwards Deming

“Quality is not a project based variable, but a corporate asset.”
- Ken Schwaber

“Adequacy is sufficient.”
- Adam Osborne

“Brilliant process management is our strategy. We get brilliant results from average people managing brilliant processes. We observe that our competitors often get average (or worse) results from brilliant people managing broken processes.”
- A senior Toyota executive

“He who will not apply new remedies must expect old evils.”
- Francis Bacon

Agile Quality

I just set up a new google group/mailing list called agile-quality.

Check it out here: http://groups.google.com/group/agile-quality

My initial welcome email says:

I intend it to be a place to discuss all aspects of software quality,
particularly as relevant to or involving agile principles, values or
practices.
This includes things like:

• What is quality?
• Quality Control
• Quality Assurance
• Relevance of quality to project managers/product owners/scrum masters etc
• Quality Management
• Testing (even though there is already a great agile-testing group)
• ISO 9001 Certification
• ISO 9126 Software quality model
• Other standards and models for software quality
• Technical debt and the management thereof
• Metrics for software quality
• Customer involvement in quality issues
• Discussing other “movements” or groups with a focus on software quality
• and agile
• Craftsmanship as it relates to software quality
• Software design; how it relates to software quality
• The Law as relevant to software quality.
• Discussing books, magazines, websites, blogs, quotes etc about software
• quality
• Which agile methods and practices specifically address quality and how?

I expect it to cover technical aspects of interest mainly to developers,
such as clean code design, as well as more managery type topics, such as
QA, certification and legal aspects, and everything in between. Ideally
this could be one mailing list where the two groups can mix, and form
consensus etc.
About me: I am just a software developer (and project management
student) interested in software quality and agile. I am not an author or
consultant (yet/at the moment). I don’t really have much of an agenda to
push except to discuss and learn more about software quality in agile
environments. I do have opinions and they will pop up in later emails.

Currently the list is unmoderated, and I will only change that if
spammers attack. Personal disagreements, teasing, slightly off topic
messages and banter will probably not be moderated away (at least for
now), I think such things help build community. The moderation policy
will be pretty loose but I reserve the right to make changes if it seems
necessary.
Thanks for checking out the mailing list, and enjoy!

Circling the drain?

I was just reading the XP mailing list and followed a link to a debate in the Agile blogosphere that seems to be a sort of XP vs Scrum type thing. Basically the popularity of Scrum means that people are adopting Scrum and considering themselves Agile without engaging in the appropriate development practices that XP provides. Some claim the direction of Agile needs to change, to put more emphasis on development practices and less on Scrum, or its the end of Agile.

I don’t intend to join in this debate, as I think both sides are wrong, but I will take the opportunity to write how I see it.

The word “direction” is interesting. As if there is something in the distance called Agile thats pulling us towards it. I guess thats how most business processes work. Someone has an idea, and prescribes it in the form of a concrete model that is supposed to be emulated. In such cases this concrete model looks indeed like an endpoint, towards which one finds the direction to be followed.

Agile is, to me at least, a bit different. Its more of a decidedly non-concrete set of principles and values. They don’t define an end point at all, hence no fixed, single direction. They define a starting point, from which more concrete practices and process are pushed out, or emerged. Once a team or an individual develops those principles and values as part of their culture, they exhibit certain behaviors that are influenced by those principles and values. They may or may not end up looking more or less like XP or Scrum. Different interpretations of those Agile principles and values, as well as principles and values outside of Agile, as well as subjective circumstances will mean that whatever emerges, is likely to emerge in as many different directions as individuals and teams that start from those principles and values. XP and Scrum are just head-starts along pretty good directions. I would suggest that a team that adopts Agile practices and processes only for planning while neglecting the development aspects or vice versa has made the mistake of neglecting the principles and values and seeing (some particular sellable concrete package of) Agile practices as a destination to move towards rather than a set of principles and values to guide the emerging of behavior. Hey at least they are a step above those who think Agile and Scrum are TFS templates though!

So rather than watching Agile circle the drains of concrete Scrum (and XP) practices, let it pour out from the values tap of the manifesto and the principles.

Forget about selling Scrum practices (or XP practices) as “directions” for Agile. Sell the principles and values and let the practices emerge, in all sorts of directions!

What do Team Foundation Server and Pink Lycra Pants have in common?

When I was in school, I chose cycling as my school sport, mainly because it was unsupervised and we could simply cycle straight home, but it turns out I was actually good at it and enjoyed it, mainly because I was riding up hills every morning delivering newspapers anyway so had the leg muscles for it. But I only had an old fashioned single speed bike from the 40s that I used for both my paper round and “racing”.

So I asked my parents for a decent racing bike for Christmas. I ended up getting an old 10 speed which was better than nothing, I was not too disappointed as it was reasonably good in its day and we weren’t especially rich. But then I opened the next present. It was a lycra cycling suit, pink no less. I am pretty sure my parents spent more on that than they did the bike, and I secretly thought it would make more sense to have not gotten me the cycling suit, and spent more on the bike.

Now I live in Germany now, and people wear cycling suits all the time even when just riding their bike to work, just as they have a special get-up for walking along trails in the hills (red bandanna around neck, knickerbockers and a ski pole), and no one wears trackpants outside the gym (except Russians.)

But this was New Zealand, and we wear the same things for everything. Trackpants if you are a sporty type, jeans otherwise, or shorts in the summer, fairly independently of what activity one is performing.

If you want to wear lycra you better be a pro cyclist, and have an expensive bike to match, and even then only the shorts will be worn (with a normal t-shirt), and you will get teased anyway.

I wore the full suit anyway, with my old ten speed, and it was an entertaining spectacle. (It was Pink! What were my parents thinking!)

What does this have to do with team foundation server? Well we use team foundation server at work. Started off with version 2005, and we only used a fraction of it. We used its source control feature, and we used the default agile template to store some of the activities that we were supposed to do. The other features like the automated builds, automated testing, the web based interface, the reporting and the project and excel integration, we don’t use. It was administered by the support department.

This version was notoriously buggy and so we started talking about alternatives. I would have preferred a simpler solution to suit our small, fairly informal team, such as mercurial (or subversion) for source control, and a lower tech solution for planning our activities, perhaps a pinboard, or one of the many open source (or otherwise free) web based tools available, and to run them internally within the software development department, so we can experiment ourselves with different development processes and techniques like continuous integration and automated acceptance testing.

But no, support was tasked with upgrading to TFS 2008, which took around a month. Now this isn’t a single package. It requires specific versions of sql server and sharepoint to be set up too. A lot of overhead considering we weren’t even using it properly.

When I talk with other developers about our using TFS, their first reaction is something like “wow that’s some heavyweight stuff, we just use subversion, are you guys using the Agile template? You must have a pretty professional process”. In fact we don’t even have a development process, nor project management, nor technical leadership, nor any quality assurance procedures, just the boss and two developers. The boss asks for a feature, and we type it in. (And yes our quality is what you think it is)

I now understand what people thought when they saw me riding my old 10 speed wearing my tight pink lycra cycle suit. “That mismatch looks rather comical, is he doing it for a joke or does he really think he is getting value out of that cycling suit, surely he should have rather invested his money into a proper racing bike”? Of course what they actually said as I strutted across the school in my tight pink lycra was a little different.

I feel a similar sort of embarrassment now, being part of a team, and I use the term loosely, that thinks they are bleeding edge by (mis)using a specific tool. Why oh why didn’t we invest all that effort into adopting a development process instead? Or introducing unit tests, or refactoring away some of our technical debt? Or implementing a continuous integration system? I suppose some things just require a bit more thought than double clicking a setup.exe

Perhaps I should ask my parents why they chose to buy me a pink lycra cycling suit and I might get a little closer to understanding the types of decision making that go on here at work.

.NET Open Space 2008

Last weekend I drove up to Leipzig for the first .NET Open Space in Germany. It was great. I originally heard about the thing on the alt.net mailing list, and nearly decided not to go when I found out it was just a .net thing (without the alt). I am glad I did go. It wasn’t as technology or tool focussed as I thought, and the discussion oriented nature of the event helped ensure that.

On the first day I attended two fishbowls; a TDD one and a DDD one. I even jumped into a chair and rambled on about technical debt at one of them. In the afternoon I attended two presentations, “Presentation Zen” and “Getting Things Done”. The second day I attended another fishbowl about functional programming (it was supposed to be about F# but no one there actually knew much about F#). I was kind of surprised that not many people had much experience with functional programming. What do they teach at uni these days? It turned out I ended up doing a fair bit of talking at that one thanks to a LISP project I did one time. I hope I didn’t come across as a functional fanboy, I am actually more of an object bigot that has seen first hand that functional programming does have it’s uses. (I must admit it did inspire me to look into functional programming a bit more) I followed that session by attending a live demo of TDD and pair programming.

I had to leave at midday as I had a long drive ahead of me.

I didn’t leave empy handed though. The sponsors provided everyone with coffee cups, and a table was laid out with books and a few software packages that we could put named slips inside if we wanted to take them home. I put all five of my slips in the handful of English books that were there. I ended up getting a couple that no one else wanted: Behind Closed Doors and Ship It! They both look pretty interesting so I am pleased with that.

There are some flickr photos here, and other reports about it here.

Oh and it also inspired me to start blogging again. Lets see if it sticks.