Kurt's Blog

May 20, 2013

Pricing, and a little bit about estimation.

Filed under: agile, kanban, lean, scrum, Uncategorized — Tags: , , , , , , — Kurt Häusler @ 12:54 pm

So I keep getting involved in discussions on twitter related to the different ways of basing prices for software projects, and it is hard to get some of the subtle details across in 140 character snippets so I thought it might be a good idea to put some ideas together in the longer form of a blog post. The whole thing is somewhat related to the #NoEstimates debate.

I also believe the base of my ideas is fairly mainstream. Most of the agile community (dominated by the Scrum community) talks about estimation in terms of estimating PBIs so you can plan how many to take on in each sprint. A few people still do it in ideal person days like Mike Cohn recommended back in the dark ages, which I am opposed to (but not to the point of getting into arguments about it) as I think there are some serious downsides to that approach. That is not a radical opinion, and it is the reason why most people use more abstract, relative units of size like story points, which I personally don’t advocate, but I don’t oppose either. Even most of the defenders of story point estimation for sprint planning recognise that the true advantage is in getting the team to talk about what needs to be done, and the actual estimate is a secondary benefit. A few of the more advanced teams, and even the inventor of story points Ron Jeffries himself, prefer to simply use a count of PBIs. This is the approach I find most sensible, when combined with backlog grooming and story splitting. Also after using Kanban for several years I have realised that estimating is pretty much wasteful. I don’t think that is a radical opinion either as it seems shared by much of the Kanban community, who make use of a metric called throughput which is essentially the same as velocity and is typically presented as a number of items done per unit of time.

But as long as all we are talking about is how many items to plan for each sprint, who cares? The consequences of over or underestimating are minimal. If we plan too few ask the PO for more, and the velocity goes up to plan more for the next sprint. If we over plan then we drop the velocity and plan less for the next sprint. As long as this is all we are using estimates for then who cares how we do it.

For me estimates are only really a problem when they are taken more seriously than intended. For example when calculating prices, or when forming an offer or contract that a customer may rightfully interpret to be a commitment to deliver, and expect to be based on something more solid than a guess. Sadly the agile community talks so much about estimation in conjunction with sprint planning, and very little about contracts and pricing. I have even heard people say that the way that contracts and prices are done is completely orthogonal to how the software “project” is carried out. I disagree completely. From my experience the way the contract is written and the way the work is priced are the fundamental factors that indicate to what degree any software development endeavour will be “agile” or not. Or should I say to what degree it is even appropriate for knowledge work and product development, or anything in the complex domain for that matter.

If some manager comes to the team and says “these requirements have to be done by the end of July, and we have a budget of 20 architect “person-days” and 60 developer person-days and 30 tester person days and 10 manager person days, now use Scrum to make it a reference agile project”, then your options are pretty limited.

I think the disagreements in the NoEstimates debate are based on a misunderstanding. The ProEstimates crowd are saying “What is wrong with estimates? It is just a simple tool to help teams plan how much work to take on each sprint”. The NoEstimates crowd are saying “if you are making promises to customers, and trying to price your work, there are much more professional ways to do it than guessing”. The ProEstimates crowd just don’t see things like offers, contracts and pricing to be relevant. Heck the Scrum Guide doesn’t even mention them. All that businessy stuff happens way upstream anyway so we just have to accept whatever deal management outside the Scrum team has made, and as long as we carry out the work according to Scrum we are all agile right?

I agree with both sides, which is possible because they are talking about completely different things. I just find the ProEstimate argument to be boring, obvious, non-controversial and not addressing any particular problem. The NoEstimate argument on the other hand is advocating one of the critical next steps that the software development community needs to take in order to restore a culture suitable for software development and good faith with our customers.

The other misunderstanding I see in the NoEstimates debate is that the ProEstimates crowd thinks the NoEstimates advocates are somehow refusing to answer questions like “How much does it cost?” and “How long will it take?”. This is quite false. As far as I am aware the NoEstimates crowd are attempting to answer exactly these questions using techniques with a higher level of professionalism and more reliability than guessing.

Anyway the opinions I have on pricing are based on experience, and they are also more relevant to some contexts than others. I believe this blog post is highly relevant to the following types of software development, which is where I have most experience:

  • Companies doing custom software development work for other companies.
  • Product companies where the customers pay for the features they want and even the bugs they want fixed.

I suspect this blog post might be less relevant for the following types of software development:

  • Internal IT. I have little experience in this area, and even less in how it is priced and funded, but I suspect things are a lot simpler than when dealing with an external customer
  • Product companies where the product is funded in another way separate to the development of features. E.g. via a subscription model, or advertising, or purchases of new versions. I do have a bit of experience in this area, and things are a lot simpler.

I think a lot of people on the ProEstimates side of the fence are engaged in the latter areas and simply don’t see the issues that the NoEstimates guys are having to deal with.

There is also a grey area in-between, such as where are large company has outsourced its IT and while the company doing the software development is technically doing custom development for a customer things are also much simpler as there is only one customer. In effect it is basically the same as internal IT.

So what is the problem?

I have worked for a number of companies in the former category that used similar methods for pricing software and suffered similar disadvantages because of it. Generally these companies rely on a horrendous mix of time tracking, rate cards, up front estimation in person-days, time & materials billing, bookable time and “slack” divided up according to percentages, and the need to explain to customers why the actual amount billed differed slightly from its up-front estimate. For me this is the worst. And so many of these customers are using agile approaches like Scrum to develop the software too. You would think they would be aware of the dangers of things like time tracking. The community (with the exception of Scott Ambler) seem to be unanimously against it. But unfortunately those with an understanding of knowledge work and product development are not those making the deals. They are usually a very different type of person unlikely to read books like Don Reinersten’s Flow, or engage in the subtleties of the economics of software development at conferences or online. They are more like the software equivalent of the used car salesman. 

So here is how it works: the customer has an idea of what they want. Sometimes this is just a vision (which is good), sometimes it is a more detailed list of requirements (which is ok, but a little bit wasteful as it is highly likely to change). The salesman will then take some of the team capacity and if the customer only had a vision those team members will then have to come up with some business and technical concept and estimate how many person-hours of each role is required to develop each bit of the solution. If the customer presented a more detailed list of requirements then they can skip the business concept, and just come up with the technical concept and an estimate. The salesman has to choose carefully how many and which team members get to be a part of the conception and estimation activity. You don’t want the whole team as this is all non-bookable time, which is perceived by such cultures as expensive, so usually the salesman picks the architect, the analyst and the best senior developer, or something like that.

This approach is already non-agile, or worse than that, it is fundamentally against the realities of knowledge work and product development, regardless of “agile”:

  • The idea of having a complete set of requirements at the beginning is long known to be unrealistic, the the downsides to this way of thinking are well known. Many of the requirements will, according to the Pareto Principle, be superfluous in the sense that their effort in being developed will be out of proportion to their use in the finished product. It is also a myth that anyone can know in advance what is required, This always changes. The customer never gets exactly what they want, and even before they see the first version they change their mind about what they think they require.
  • It is actually preferred, and more realistic to start with a mere vision. This allows the true scope to be discovered, adjust and grow as the development endeavour proceeds, and starts off with the knowledge that scope is and should be negotiable. The idea of having a complete set of requirements works against this. For the actual software company to start with something good (a vision) and discard all the advantages by turning that into a detailed list of requirements up front is bizarre and ridiculous. The chances of it being what is actually best for the customer are even less than if the customer themselves came up with a list.
  • It is also pretty disadvantageous to select a sub-set of the team to create the conceptions and estimates. This requires crossing the problem-solution line, and breaks one of the rules of estimation that it has to be done by the whole team. In software development, the team should be problem solvers. They should get an idea of something small that the customer desires, and work with the customer in solving the problem, implementing the solution as software and realising it as true customer value. With the used-car-salesman approach, the small sub-team of elites are trusted with the task of solving the problem, and determining the solution up-front, and the actual development team are reduced to mere source code typists, which is highly demotivating.
  • Additionally the estimates are usually much more optimistic than if they had been given by the team, and those in the sub-team are not available to the development team for assisting with the actual development of actual software.
  • It is also a waste of time, all these discussion that take place during conception and estimation still need to be held again so the actual whole team can understand them. (At least this second discussion might be bookable to the customer).
  • If you are using something like Scrum, combined with estimation in story points which is fairly typical, you have to go through and estimate everything again anyway.
  • Also these estimates are usually “ideal”. They only reflect the actual minimal time an expert might spend on typing in the solution once and that is all. Anything like research, or learning, or pair programming, or communicating with each other, or improving the system, or tests, or refactoring is extra. That is why project managers add a buffer to to the estimates or the price that the customer sees, so they have some wiggle room later, but the actual team still sees the tighter estimates. After all if you let them see the actual buffered values they will simply chew the buffer up with non-value-adding activities like refactoring or writing tests, and there won’t be any buffer left at the end to make up for the shortfall which for some reason always occurs at the end.

So what happens next? The developers start counting hours present in the workplace and recording them in some electronic system. This is called time-tracking, and suffers the following problems:

  • Some people say the main or only problem is the time that time-tracking takes. This might be the most obvious issue but it is the least of it. Sure it can chew up around 15 minutes per day, but time tracking is much more evil than that.
  • The developers have a budget per feature, and there is a general hectic vibe with everyone primarily concerned with which booking position every minute of the day belongs to. No one want’s to be the guy that has less bookable hours than anyone else. In fact everyone tries to keep their bookable hours as close to 100 percent as possible, as the general perception is that any hour not booked is wasted. Typically project managers and even product owners insist that anything like knowledge transfer, or writing tests, or refactoring or even one person in a programming pair shouldn’t be at the customer’s cost, and has to be booked to an internal booking position. As a result, these activities simply do not get done, or at least no one wants to spend much time on them.
  • The teams more expensive roles end up siting there idle. Since the project manager was involved in the technical conception and estimation, and was concerned with presenting as low a cost as possible to the customer, he preferred to see more (cheaper) junior developer hours being offered as more expensive architect or senior developer hours. As a result, even when the team is struggling to meet its sprint goals, the architects sit there doing nothing because if he did the work the customer would get a more expensive bill than if the junior developer did it. The customer would then come back and demand to know why he was charged architect costs for something that could have been done by a junior developer. As a result project managers insist on architects NOT helping the team to meet their sprint goals, which damages the feeling of trust in the team.
  • A related issue is when cheaper developers get assigned the tasks where there is a high risk of requiring more time than that estimated. I have seen a team suffer a huge drop in motivation when a student, with a cheap per hour rate, was given all the most interesting tasks (that were sold to the customer as junior or senior developer hours), because he essentially had more than twice as much time as estimated to do it, and you automatically have some extra buffer for the risky, more innovative and interesting parts of the “project”.

I hear a lot of managers claim that time & materials works well for them, and that estimates correspond very well with actual development hours. That is because developers are painfully aware of their budget, and compromise on quality in order not to “overspend” their time budget. Without fail these managers also wonder why they run into technical debt issues a year later. Actually they don’t wonder. They know it is down to poor commitment and discipline on the part of the developers. On the other side of that coin is that it discourages innovation, and encourages maximising expended effort. Assume a task was estimated at 5 days. If someone can find a clever way to do it in 2 days he probably should right? Well with T&M there is no incentive. If he does it in 2, then he might not have anything else to do for the week and his bookable hours will drop. Also the company will  get 3 days worth of money less. Usually project managers solve this by shuffling hours around and lying to the customer about what took how long. They will simply say it took 5 days, and use the extra 3 days to reduce something that took longer than expected.

Particularly sad are the companies that think they are cool and generous like google for allocating a 20% “slack” time that doesn’t have to be bookable. This implies that everyone is under extreme pressure to book the other 80% of their time. Guess what ends up being done in the 20%? Meetings, bug fixes that the product owner thinks should not be billed to the customer, writing tests, learning from other developers, pair programming and refactoring. It is much better NOT to have such a percentage goal of bookable hours. The team needs the freedom to decide for themselves how to best to allocate their time in order to deliver, and besides, the customer pays for everything anyway. It is not like the government pays for the 20% time. This trick might look good at first glance, but it is total bullshit that has no pros, and only cons.

There is one general issue that lies at the root cause of all the parts of this nasty approach to pricing software development efforts: The idea that it is appropriate to measure software in terms of person-hours. Once we get beyond that dangerous myth, we instantly solve a number of problems and we open things up to the next level of professionalism in managing software development.

So what is the alternative?

So let’s look at better models for pricing software in order from best to worst, according to my opinion:

  1. Value based pricing: The best way to price software is according to the value it brings the customer. We can negotiate with the customer and help him discover what sort of value the delivered software might bring, and come up with a price that correlates with the expected level of investment required to bring such a return. However I find this suffers from a couple of problems that might make this approach difficult:
    • Business value is hard to determine. Software development, when worthwhile, is a high risk endeavour, featuring an asymmetric payoff function similar to the development of medications. We can expect to start a large number of experiments with the hope that they will be highly successful, and expect that most of them will be canceled as soon as we discover that they are not going to be a big winner. The occasional big winner results in so much profit that it more than covers the losses associated with the “failed” experiments. (Actually, the information we learn from these experiments usually makes the investment worthwhile, so failure is probably mot the right word). The fact is, we often have to at least start the development process and test a feature or two on the market before we even know what its value might be. This is hard enough at the project level, and even harder at the “story” level that agile development prefers, so I suspect for this reason value based pricing to be very difficult in practice. One model that could work would be to measure actual value afterwards, but this requires a high level of trust, and exposes extra risks for all participants. It essentially makes the software development company a full stakeholder in the product, which may or may not make business sense.
    • The other issue is that customers may not be willing to share their expected value, even if they know it.  They want to get the lowest price possible, and not give the software vendor a hint as to the maximum they might pay. There are also a lot of things that can be done cheaply and easily, but result in huge wins for the customer. The vendor might think it is fair to take the usual percentage cut of the wins, but the customer most likely does not.
    • Even if we know the customer value, before we decide if we should do it or not we might like to know to what degree, if at all, it covers our costs. This means that even when using value based pricing, we might still have to use one of the following more cost-based methods to determine if we should do it or not.
  2. Price per feature (determined at time of commitment): My current favourite model is very simple and based on measuring the teams price per (average) groomed feature. It works especially well with a system where Kanban plays a role. You should have a rough idea of your costs. Ideally they are mostly a function of team salaries and fairly fixed, otherwise in more dynamic environments you can determine separate components, perhaps basing it partially on an average of previous months, and known costs for the next month. You simply divide these costs by the throughput and add an appropriate margin. Before the customer and team commit together to developing a feature, some backlog grooming should occur and the team should perform the usual steps of story splitting. Queue replenishment consists of presenting the split stories to the customer, together with the current price per feature. They can select as many stories as there are free places in the input queue. When working with multiple customers it is good to authorise the product owners to do this on behalf of the customer, or to establish some idea of how much team capacity is to be allocated to each customer (ideally in terms of total system WIP). This shouldn’t be too big a deal, the PO can communicate with the customer up until queue replenishment about what the price could be and how many free slots the customer will have. The price is fixed at the time of moving the item from the backlog to the input queue, and payment is due a number of days after the item is done. If you have a two stage commitment system, or end up cancelling more than a handful of items in the middle of the process and have multiple customers then it could get a little more complicated, but I might blog separately about that. I don’t currently see any disadvantages, but I know someone will have some questions about it so please challenge me in the comments, and allow me the opportunity to make this concept even more solid. I know a lot of people seem to think it will only work with items that are “roughly the same size”. This is not true, and in fact the concept of size is a bit bogus when considering the whole value stream. We groom and split in order to preserve some liquidity and flow in the process, to prevent large items from blocking the system, and to inhibit the lead time distribution from blowing out too far to the right. I fully expect significant differences in the measured lead times of different items, that is the reality of knowledge work and product development, but there is no real need why they all need different prices. If the customer has 3 features in the backlog, but only feels 2 are offer enough value to justify the price per feature, then he should only choose those 2.
  3. Renting team capacity for a period of time (in terms of system WIP, not in available person-hours): This might be an ok option for those doing a longer term software development effort, and wan’t to save the communication overheads of accepting a small offer every week or so. It is also based on a Kanban approach. Assuming costs (plus margin) of 100,000 per month, and a total system WIP limit of 100, then we could have one customer paying 30,000 per month for exclusive rights to 30 of that WIP, and so on. If the throughput is 100 items per month, we might expect he gets to see 30 items per month. Or he might not. We could provide both options and allow him to choose which is best for him.
  4. Fixed price per team-day or team-sprint (aka. T&M light): This is probably good for Scrum teams, or teams when you only have a single customer and don’t have too many issues with balancing capacity between different customers or classes of service. It is also based on costs plus margin. One discussion I got into recently on twitter was with a defender of T&M. It turns out they don’t do up-front estimates of entire projects, they don’t have rate cards, they don’t have a target number of billable hours and most importantly they don’t do time tracking. Basically for every day a team works, they get a bill for a team day with a discount for team members on holiday. To me this is a lot closer to a fixed price per team-day than the true T&M monster as typically practiced. However they reported it requires a large amount of trust, and only works when the team has a single customer (which some consider ideal, but I personally consider it a risk of the “all eggs in one basket” variety.) The team just developers iteration after iteration until either the product is good enough, or the customer decides they have spent enough money.
  5. Fixed price variable scope: This is the classic agile fixed price scenario where there is some budget set up in advance, and the team simply develops iteration after iteration until the money runs out. It is essentially equivalent to the fixed-price per sprint, except presumably the option to quit before the fixed budget runs out is excluded. I don’t know what is supposed to happen if the product is good enough before the budget runs out. I guess the team is supposed to keep developing unwanted features until the budget runs out. There is also the question of how is this budget determined. I guess the customer has some idea of what he wants to spend. I could also imagine it being based on a rough release plan, which is in turn based on an initial but negotiable backlog and a known velocity. I am not sure what advantages this model brings over the previously mentioned ones.
  6. Fixed price, fixed scope, but the price is obviously sufficient to cover costs and the scope is easily achievable by the due date: Sometimes we get lucky even with dysfunctional traditional methods. As long as we ask for more than enough money to cover things even if things go wrong, and we leave ourselves plenty of time to get things done, even with the inevitable changes in scope, then why not? It is still better than the full-on T&M model with all its disadvantages. But not much better.
  7. Fixed price, fixed scope, fixed due date, and the team thinks it will be tight: This might even be worse than full-on T&M. I don’t know which one is worse. They are both extremely damaging. Sadly they are the 2 most common approaches. In fact the typical Scrum-using but otherwise not very agile company thinks they are the only two options. They are convinced that one of them is not “agile” so the other one must be. Usually a company has a bad experience with number 7 so declares it un-agile, and then feels super smug and ultra-agile for using full-on T&M together with Scrum.

So there you have it. My current views on the various approaches to the pricing of software development. I have to mention again that my favourite is 2. I don’t really have a problem with any of 1-4. I don’t see much point in 5. 6, 7, and the full-on T&M I described at the beginning are the used-car salesman approaches. They are lazy, damaging, unprofessional and betray a poor understanding of how knowledge work and product development works. Instead of violating our customers and teams with such approaches we need to think about things a bit deeper, and consider the damage we are doing, and start behaving more like professionals. The relationship between the software development and the customer is too important to be left to sales-people. It requires product developers to be involved. It needs to be more like the way a doctor and their patient communicate, than the way we currently operate, as if it were possible to fix scope and sell person-hours like the kids at McDonalds sell fries.

I recently read a blog post from someone, it isn’t that great, it exhibits one of the misunderstandings about NoEstimates (he somehow thinks NoEstimates is all about refusing to answer the “how much” or the “how long” questions), but one quote was interesting:

If you don’t know where you are going, if you don’t have some notion of what done looks like, if you can’t tell me approximately how much and how long, please don’t start.

At first I thought, woah wait a minute, this pretty much excludes anything in the complex domain, aka anything worth while. Even beginner agilists know that software development is an experimental process, and it is quite typical to begin something before we know how much it will end up costing, or how long it will take. But on second thought I agree with him. There is no way to know what done looks like when we are talking about a large project, no way of knowing how much it should cost, and no way to know how long it will take. They can only ever be discovered once we have invested some effort in the information generation aka software development process. So I agree. We should not do large projects. However we do know what done looks like at the story level. We can say how much it will cost, and we can provide a certain degree of confidence about how long a story will take based on concrete measurements and backed up by statistics. So I agree. We should only ever do story sized things. Actually why not take his quote even more seriously and use it to base our definition of what we should attempt to develop? If we don’t know what done looks like, we can’t put a price on it, and we can’t say how long it will take then we shouldn’t start it. It is obvious what he means. We shouldn’t do anything larger than stories, we have to get away from projects being the main batch size of relevance to pricing, offers and customer communication in general.

So here are the main points again:

  • Don’t sell person-hours, it is not appropriate for things like software
  • Don’t pretend scope can be fixed in advanced
  • Don’t use time tracking
  • Don’t use T&M billing
  • Only estimate where and how it is appropriate, the whole team may use story points to plan how much work to take on per sprint, and that is pretty much the only place for estimation in software development.
  • Never estimate in hours, only relative units like points
  • Never base prices on estimates, use measurements instead
  • If you have customer contact, write offers, work out prices or negotiate contracts, please go to a couple of conferences, read a book or two, and engage with the community
  • Don’t think you are cool by gifting your employees 20% slack time (by insisting 80% of hours must be bookable), all you end up doing is pressuring the employees not to spend time doing what they should. And it really messes with the idea that a team should self-organise
  • If the offer and pricing is fundamentally not “agile” or at least makes no sense from a knowledge work and product development perspective, then planning the development using Scrum won’t help you much
  • Forget about “projects”. When it comes to prices and offers, we don’t want to deal with anything larger than a story

I have a feeling this might be a blog post I keep coming back to and refining. I already suspect I have left some interesting details out.

Any questions or comments? 

Addendum 1: Two additional models that I have been made aware of but forgot to include to this list are the “price per story point” model. I don’t prefer it, as it suffers from many of the same disadvantages that other approaches based on up-front estimation suffer, but if you are estimating in story points anyway, it might work for you. I would probably put it somewhere between 4 and 5 maybe. The other model is the incremental funding method. This is a really great one if you can get it to work. In this method we minimise initial investment and risk, and focus on developing a product that makes sufficient money to fund further development. This concept isn’t really an alternative pricing model as such, and can thus be combined with any of the above listed pricing models as an additional technique. You could even use it to filter out projects that should not be invested in past their initial MVP. Any projects that don’t self fund after a calculable investment in money or time should be cancelled, so that that investment can be channeled in more worthwhile activities. I believe Don Reinertsen’s book Flow even presents some useful numbers that can be used to help support such decisions in the section on asymmetric payoff functions.


May 11, 2013

The Limited WIP Society Rhein-Main

Filed under: agile, kanban, lean — Tags: , , — Kurt Häusler @ 4:33 pm

I just wanted to write a short note to inform any interested readers that I am in the process of setting up a Limited WIP Society in the Rhein-Main region (the region around Frankfurt). A Limited WIP Society is a user group mostly for those interested in David Anderson’s Kanban method for supporting evolutionary change in your knowledge work and product development processes and systems.

Go Lean Limit WIP

Some of the other topics relevant for a Limited WIP Society include:

  • Flow-based product development
  • The Theory of Constraints
  • Lean Systems
  • Systems and Complexity Thinking
  • Modern management methods in general

I have set up a Xing group that as of writing already has 21 members, and I have planned the first meetup which will be an informal gathering at a restaurant later this month. If you are interested in any of the topics mentioned above then please come along. I hope to see enough interest in organizing a regular meetup. Ideally in a location more suited to other formats like the lean coffee format or talks.

Yes We Kanban

The first meetup will be somewhat casual, focussing on getting to know each other, sharing our experiences and tips, and making plans for future events.

You join the Limited WIP Society Rhein-Main here: Limited WIP Society Rhein-Main Xing group

You can register for the first meetup here: First meetup of the Limited Society Rhein-Main

Vote Ltd WIP

Here are some other interesting links:

Advocate Ltd WIP

Thanks for reading, and I hope to see you at the next meetup!

December 6, 2012

Lean Kanban Netherlands 2012

Filed under: conferences, kanban, lean — Tags: , , — Kurt Häusler @ 5:59 am

Over a month later, I am finally getting around to blogging about this great conference I not only attended, but spoke at. Apart from open spaces, and leading semi-prepared informal discussions at user groups and stampedes, it was my first time presenting at a conference.

David Anderson‘s keynote was of course very interesting. I found his idea of a second, or multiple, committal points interesting, and the idea that the lead time should stop at the first non-WIP-limited queue or buffer (and presumably start again from the next WIP-limited queue or buffer). I am not convinced on that last point. I think even if a buffer should be non-WIP-limited because it is under the control of an external stakeholder, it could still make sense to include time spent there in the lead time, if you are using it to predict how long things might take. Why not just take both lead times I guess. 

He also mentioned that the cost of delay was perhaps more useful at the portfolio level than the user story level, expressed some interesting opinions about calculating business value, and combining that with strange cost estimates, which is how a lot of product owners etc currently spend a lot of time. An alternative could be to focus on risk. Capacity-constrained people should never work on optional things like estimation as it brings the whole team capacity down, only those with slack capacity should do such things. The bulk of his talk was about liquidity which is something I need to spend a lot more time looking into. He presented some interesting metrics such as “pull transactions” per person, or per unit of money.

Another interesting session on the first day was Ralf Kruse‘s Kanban Pizza Game. It was good to do something a little bit physical after sitting in a chair watching others talk.

Laurens Bonnema gave an interesting talk that appealed to me, because talked a lot about the more subversive aspects of culture hacking. There were a lot of good tips for being an effective change agent, and not the usual types of tips one reads either.

The evening meal was pretty classy. Lots of little courses, very delicious. It was a great location too by the way: De Fabrique

The next morning I realised I drank too much at dinner the night before and was worried I wouldn’t survive my talk. I found Don Reinertsen‘s keynote a little heavy going. Lots of graphs and maths and I had trouble relating it back to the concrete experience of helping a team develop software. Someone else mentioned he should try story-telling. Then I gave my talk. I think it went well, but there was a minor technical hitch. The screen-saver kept kicking in,and the main presentation display kept working but my secondary display with the timer went dark. So I lost track of time and had to really rush through the last bit. I still went a few minutes over time. My hangover wasn’t an issue. Once I was standing up and moving around and talking it kind of went away.

You can see the slides here:

Deep Kanban – a biased and opinionated participant in the path towards synergy from Kurt Häusler

After that Jasper Sonnevelt and Sarah Reeder made us dance. I saw Arne Roock‘s nice looking presentation about creating a Kaizen culture. The last session I attended was probably my favorite. My “classmate” from my MSc program, Steve Tendon, presented a very informative presentation about the Theory of Constraints, with really great looking slides. Steve should really present at conferences more often!

On the last evening a few of us joined up with the local Theory of Constraints community and went out to dinner. We went to the Oudaen where we had the surprise menu, but it was also a very classy dinner with lots of little courses.

Over all I really enjoyed this conference. I am starting to get to know a few people now, and actually look forward to catching up with them as much as the actual content. 2012 was a fairly quiet year for conferences, but I will definitely have to look at not only attending but talking at more conferences in 2013. I already have one unconference on my plan. Play4Agile in February.

May 16, 2012

Scrum, Kanban, The Marshall Model, Rightshifting, The Synergistic Mindset & Stoos

Over 10 years as a software developer has taught me that investing in technical skill can only get you so far. In fact once I got to the “average agile developer” stage, further investment in acquiring advanced technical skills was generally waste. Despite agile leaders emphasizing the importance of technical excellence, despite the software craftsmanship movement desiring to see the bar raised rather than lowered, most companies are still making decisions based on the assumption that it is too hard to hire technical excellence, or that you can get more done with a larger number of lesser developers, with a lowered bar. Essentially they are quite openly and explicitly rejecting the premises of the Agile Manifesto, and Software Craftsmanship, in favor of the old Taylorist split between thinking and doing. Even places partially embracing certain Agile practices! It got to the point where management had no interest allowing the use of idea A, or technology B, or practice C, because “no one else knows how to do it” or “yeah but our tool vendor doesn’t support that”, or “I have never heard of that”, or “its hard enough hiring as it is, without restricting ourselves to applicants who know that as well”.

It has been very clear to me, that the issues impeding effective software development are mostly related to the management culture. Even at the technical level, decisions are made that prevent, or make difficult, the adoption of advanced technical practices.

Scrum goes a long way towards improving this situation. It is based on some ideas that really challenge the, what I call, Command & Control, or Taylorist mindset. To a certain extent at least. The concept of having a ScrumMaster as servant leader, facilitating self-organisation, and encouraging inspect and adapt, can really go a long way towards liberating team members from a demotivating work-life, and permitting them a larger degree of freedom to make at least technical decisions themselves. Unfortunately it is usually restricted to the team. The same wall that Scrum builds to protect the team from external influences, also prevents such radical ideas from escaping into the wider organization. The vision of ScrumMasters as organizational change agents, stepping outside the development departments, waving the Agile Manifesto around, explaining to sales and marketing experts that there is a better way, doesn’t seem to have had much effect. Particularly in organizations where software is not the main product, or only part of the product. Scrum works by making problems visible. However it is a fairly focused magnifying glass, only providing visibility to one little link in the value chain. The solutions too, are usually fairly specific, with limits on who can participate in the improvement process, and the optimizations can only be local. Typically a problem causes a decrease in velocity. Someone might get less of a feature on a certain date, or might get their whole feature a month or so later than expected. In the overall scheme of things, Scrum does a great job of shielding team-level problems from the wider organization, and shielding organizational problems from the team.

Kanban, when applied properly, across an entire value stream, does a better job of making team-level problems and organizational problems the same thing. Scrum is fairly mild when a problem occurs. Velocity goes down a bit, big deal. When a WIP limit is hit in Kanban, all of a sudden you can have an entire team either standing around doing nothing, or knocking on the doors of a completely different department offering to help. When problems occur in a (proper) Kanban system, the effects are much more radical than in Scrum. (Of course it is possible to do Kanban inside a walled garden, in a Scrum-like fashion, and not see such radical effects.) This is a good thing. All of a sudden you have a lot more people interested in the problem, and participating in finding an urgent solution. All of a sudden you have people talking with each other, that might not have previously communicated much. It sends shockwaves up and down the value stream, ideally affecting even the customer. In Scrum, it is too easy to say yes to customers. Write another story, and stick in the backlog for later. With Kanban your capability is staring you in the face. You are forced to negotiate with the customers. You are compelled to insist that they review, accept or deploy finished work (and free up WIP) before they can request any more changes. In the best case, your sales people have to face the exciting opportunity of helping to shorten lead times, by saying a simple “no” to customers! “No, sorry, our system is full, we can’t do that for you, but lets look at what we can do, and when”.

But Kanban is still not enough. If you start off with the typical Taylorist, Command & Control, low trust culture, perhaps with a bit of Scrum in the development departments, and simply throw Kanban in the mix,  you will very likely make things worse. You will have a system for evolutionary change all right, but in what direction? Chances are the dysfunctional management culture will use the Kanban change mechanism to extinguish what little spark of agility did exist in the Scrum teams. Kanban, like Scrum, is great at identifying problems, but leaves the solution space fairly open. An unprepared management system, combined with the radical nature of the Kanban change management tool, will result in certain types of decisions being reached that will have far reaching, permanent negative consequences for motivation, lead time, costs, quality, ability to react to urgent situations, relationships with customers, overall level of learning for individuals and the organization as a whole, and relationships within the organization. The Kanban community intended Kanban to be used to make things better, but the typical organization will end up using it as a weapon against itself!

In fact I have observed the following occur:

  • Cards spend much longer in the “development” column than in the “analysis” and “test” columns, so management assumes “development” is too slow, and requires more detailed time tracking, and imposes fixed budgets per item adding pressure to developers to work faster
  • In a retrospective normal workers identify the potential communications benefit from cross functional teams (perhaps oriented on subdomains), asks management (who were too busy to attend) to implement it, management decides to implement teams specializing in technical layers (DB team, GUI team etc)
  • Actual measured lead times look unattractive to customers, so management decides to ignore them and continue to promise delivery dates based on wishful thinking
  • Management felt the transaction costs associated with user stories were too great in almost all aspects of the system (but great in development), so standardized on mini-projects as the main entity “flowing” through the system, lengthening the time it takes for developers to get feedback, and increasing the potential for waste, significantly
  • Defining slack as “preparing for (i.e. starting without pulling) the next item to be worked on”. Stating quite openly “we don’t trust consultants to write tests” and basically laughing at the idea of allowing developers to be involved in customer-facing aspects of the process. “They might say something that conflicts with what is written in the contract, or make it sound cheaper than what we charged!”

and much more… The pattern was clear. Normal employees didn’t trust themselves to help improve aspects of the process that lie partially outside their area. Management didn’t trust normal employees to make decisions, and didn’t accept allowing them to be involved in the implementation of improvements.

Management were just making the same decisions, based on the same mindset that they always did. Kanban forced these decisions to be made quicker, and under the pressure of more urgency. Motivation and relationships deteriorated significantly. Management openly rejected the idea of moving to a high-trust culture, claiming it was never discussed as part of Kanban, or a goal of Kanban. Kanban was supposed to make the software get developed faster and better, reduced lead times right? No one mentioned anything about we managers having to change anything, or anything about culture at all, so we will continue to work how we please, and “you” will use Kanban to get things done quicker.

But this is all fair enough. They were correct. You can’t blame them for thinking like that, as that is how Kanban was “sold” to them. We had 2 initial workshops with well known consultants at the beginning. We discussed our goals. I was the only who mentioned cultural advantages, like:

  • Having our development department and project department working closer together according to the same (ideally agile-like) mindset, rather than having one department working according to a waterfall like process, and another one working (at least to a certain level) according to Agile values
  • Increasing motivation by allowing people too not only see how their work fits into a wider value stream, but allowing them to actually get involved in other aspects of it
  • Enabling and encouraging employees to take on more of the collaborative decision making process, at the level of actual work (e.g. standing together at the board discussing what to pull based on business value etc), and in improving the process itself.

Everyone looked at me like I was from outer space. I think we settled on things like shorter lead times, reduced costs, and better quality as the goals.

I seem to remember going over the first 4 core properties of Kanban, and skipping over the 5th. I also remember being concerned about focussing on the use of Kanban as a static tool, moving cards around the board etc, and I actually asked if we were also going to discuss about what sorts of actual changes we could expect, how they might come about, and whether Kanban can truly stand alone as a useful tool, or would we be better off looking at it as if it were part of some more fundamental effort to improve the management system. I was told the focus would be just on Kanban, and we would not be addressing those points. Even then I had concerns about what this would do to our organization. But you can’t blame the external consultants either. They didn’t know us, they didn’t know what challenges we might face, or how our culture would react with Kanban. They were also under commercial pressure to deliver something that would be accepted by those making the decisions and paying the bills. If they turned up to the boss and said “yeah Kanban is great, it will completely turn the organization upside down. Problems will smack you in the face and force you to confront them and come up with solutions over night. Normal employees will have to be trusted with decision making, management will have to step back and remove impediments, and focus on improving the system according to the ideas of Deming, Ohno, Goldratt and others, forget about detailed project plans, that wont work anymore. Forget about your strongly batch oriented process that you push work into without considering the system capability, Kanban will exert strong pressure to change to a flow based process, and you will have to start saying no to customers to avoid stressing the system, compromising quality and demotivating employees. Oh and you will have to trust the employees more too”. The boss would have said, “I thought Kanban was a cooler type of Scrum, and change was optional and evolutionary. There is no way I want to pay you to deliver that kind of disruption”. Nope. The consultants shrewdly identified what our culture would accept. I think I remember the old “Kanban can be introduced in any culture, and a high-trust culture will simply emerge” line.

What was missing? Kanban can’t be applied to an entire value stream in a low-trust culture on its own. Maybe the consultants didn’t know that, maybe they did. Without paying some attention to issues of culture, and the management system, in addition to applying the mechanical aspects of Kanban, you are severely limiting your chances of success.

Rightshifting is one approach that should always be mentioned as the context in which Kanban can be used to improve the effectiveness of an organization. There are other ideas. Some are fairly old ideas, a little abstract, like Deming’s System of Profound Knowledge. Or more detailed and concrete approaches like Beyond Budgeting. I feel like Rightshifting, and the Marshall Model of Organizational Evolution hit that sweet spot as ideal companions to the tool of Kanban. I won’t try and interpret or regurgitate things here, instead I will urge you to go look at the original material on the topic. In fact I am a beginner myself when it comes to understanding and appreciating the power and utility of these ideas. I have heard about them a while ago, and been only loosely following them since. But recently I have come to think about things and realize I need to pay more attention to what is happening in this area. I think I heard once that you can’t change culture, all you can do is change the management system, and the culture will follow. I interpreted this to mean, it doesn’t make sense to pay attention to things like culture directly, and I lumped mindset into this too. Instead we should simply pay attention to the management system, e.g. things like Kanban, and culture will hopefully follow. There was one thing that didn’t occur to me though. You can change the way you think! The mindset is not part of the not-directly-changeable culture, but it is a basic part of the very-much-changeable management system. Seems obvious, but it really freed me up to pay more attention to a wider set of dials that can be adjusted in the organizational context. The other thing was, and I kind of hinted at this above regarding the futility of a ScrumMaster waving around the Agile Manifesto in the sales and marketing department, and I also experienced it when talking to managers about the direction we wanted to let Kanban take us in (I said we wanted to go in an “Agile” direction, but the managers saw that as being a very software development thing of little relevance to them, who were managers involved in creating products in a domain where software was just a part). I desperately needed a way to communicate where I thought Kanban should take an entire organization, independently of software development. I needed something that addressed the mindset that I guess “Agile” is a software specific realization of. The Synergistic Mindset was exactly what I was looking for, and it can be directly addressed as a new way of thinking. From now on I don’t think I can responsibly talk to teams or organizations about Kanban, without first framing it in the larger context of improving effectiveness by addressing necessary changes in the way we think with the help of the Marshal Model. I think it is exactly the type of model they had in mind when formulating the 5th core property of the Kanban method: “Improve Collaboratively (using models & the scientific method)

So. How do we learn more about this, and other similar ideas? By getting involved with the Stoos community and attending Stoos conferences. They won’t just be discussing rightshifting, but also things like Beyond Budgeting, Radical Management, Complexity Thinking, Servant Leadership, Management 3.0, you name it. Pure gold for anyone expecting to actually take on Kanban the way it is intended, and expect significant positive results, particularly in an “unprepared” cultural environment.

The main central point for the Stoos community is the LinkedIn group: http://www.linkedin.com/groups/Stoos-Network-4243114

There is also a main site that explains what it is about reasonably well: http://www.stoosnetwork.org/

There is a small get together here in Germany soon, the StoosXChange (I will not be attending this one though): http://stoosxchange.org/tiki-index.php

I will attend the Stoos Stampede in Amsterdam: http://www.stoosnetwork.org/stoos-stampede-amsterdam/

In fact as a manager one of the main parts of your role in the future, as well as designing systems and removing impediments, will be attending events like these. There won’t be too much in the way of concrete “best practices”, as everything is context dependent. You wont be concerned with detailed project plans, or allocating “resources”, or controlling costs. You will be engaging in a community of like-minded professionals, gathering a toolkit (or more accurately a valuekit, or principlekit) of diverse types of ideas, and learning how to experiment yourself, in collaboration with other participants of the system you work in, and coming up with your own tools.

And the world will be a better place for it.

January 12, 2010

Retrospective 2009

Filed under: Uncategorized — Tags: , , , , — Kurt Häusler @ 9:00 am

It is a bit late, but I have busy with assignments since before Christmas.

I guess 2009 was a fairly eventful year. Had a baby, moved to another part of the country, started a new job. Passed my first year of uni.

Started a mailing list, agile-quality, it didn’t really catch on, but that is fine. I wasn’t really prepared to promote it by starting a lot of the discussions myself. I thought about it but that would have made it a bit like a more pervasive blog.

Went to three open space conferences, a software architecture one, and two .NET ones.

Did several presentations at work. One for developers on the Dependency Injection Principle, one for project managers on Scrum, Lean and Kanban, and another one for developers on Domain Driven Design. Unfortunately they have stopped the regular presentations for developers now. Supposed to be replaced with workshops that are presumably more relevant for our work, which to me defeats the purpose a bit. It was nice to be able to explore ideas from outside our day to day work.

Went to the local .NET usergroups fairly regularly in the middle of the year, but I stopped for two reasons, too busy with uni and less interest because my job was mostly MFC C++. I also try to keep my programming for work life and my programming for fun life as separate as possible. I don’t even have Windows at home now, just Linux, although for most of 2009 I did have a Windows 7 Beta VM set up. I know a lot of people are devoted to .NET and the Microsoft ecosystem, but it is fine for .NET to be just my work platform. It turns out that by the end of the year my job involved more .NET, so I am reevaluating attending .NET usergroups, as the .NET community here is still a pretty good community. Depends though, my wife and baby will be going swimming on Tuesdays nights which is when the .NET usergroups meet, so I will first have to see how the schedule plays out. In the meantime, I will start looking out for other software developer communities, perhaps more general ones, nothing too platform or language specific.

I mentioned in a previous blog post that I was thinking about a programming language for 2010. It is a bit of a cop out, but because I don’t feel like I invested enough effort into learning Haskell last year, I will give it a proper go this year. I have set a goal to have completed going through Real World Haskell by the middle of the year, and will try and use it properly for something in the second half of the year, perhaps starting a, or contributing to an open source project. The two runners up were Erlang and Factor. I look forward to learning those in subsequent years.

I started off being subscribed to a number of blogs, mailing lists and twitterers, and the number steadily grew over the year. I couldn’t keep up with that much information overload. Especially when I got my new Nokia N97 phone, and started getting emails on my phone, I ended up creating a special mailing lists folder and redirecting all mailing lists there so they my phone doesn’t vibrate every couple of minutes, and non-mailing-list emails are easier to find in my inbox.The result of that is that I hardly ever read mailing lists anymore, and by the time I try and catch up on them, the conversation has moved on. It got to a point where I did a mass unsubscribe on Twitter too, cut down the people I follow from around 800 to 200. I did the same with the blogs I read in Google Reader. In the next few days I will have to think about which mailing lists are the best for me, and probably unsubscribe from the others. Another option would be to have the good ones and ok ones going into separate folders. I am still undecided if I will allow any to go directly to my inbox or not. I will let you know which mailing lists I have found the most interesting and will try and take part in.

For a few years I had been a user of the Dvorak keyboard layout, and I found it superior for when typing in larger blocks of text. However it is unsuitable for programming with shortcuts. If you use the mouse a lot, and only type in the source code, Dvorak is ok, but if you try and use the mouse as little as possible, and want to use shortcuts a lot, Dvorak doesn’t help. Also by pair programming it is a nuisance having to switch all the time. For these reasons I have switched back to Qwerty, using my terrible three finger approach. One thing I miss about Dvorak is that it forced me to properly touch type, and it was faster. However my typing speed is not the bottleneck no matter what I do, so my three finger qwerty pecking is fine for me. Another bonus of Qwerty is that it works one-handed while Dvorak needs both hands, so I can hold the baby and still use the computer.

I experimented with using personal, or household Kanban as an organizational tool. I found it didn’t work so well for the different types of things that need to be done. Sometimes you need to put something aside when something with a different schedule comes along. My Kanban board told me there were smells in my value stream, but I knew that. The solution was to look upstream, and limit what is coming in, but if you don’t control or own the entire stream, there isn’t much you can do. As an example, my backlog might be filled in with things like “Write a blog post”, “Go shopping for shirt”, and “Read Clean Code”. Notice how they all have very different natures. Writing a blog post can be done in one session, but can usually be done at any time. Going shopping for a shirt has constraints on when it can be done, I can’t do it between 5 and 7am on any arbitrary day. I have to wait until a sale is on, go into town, which costs extra travel time, and so it is best combined with other things that need to be done in town. Reading a book is something that doesn’t happen in one session, you start it, and do bits here and there over time, while you do other things.

My In Progress column ended up getting filled with things like “Buy a shirt” “Read Clean Code”, and “Develop CQRS example program”. When a university assignment came along, with a deadline of course, there would be no slot for it until I finished one of those three items. And you can’t tell your professor “sorry, I don’t have any slots  free for that, hopefully my wife has time to help me buy a shirt next weekend and I can make a start then”.

I have seen people solve exactly these issues by complicating Kanban with elaborate swimlanes, and special categories but the best thing about Kanban is its simplicity. We have started using it at work for our bug-fixing sprints, and it seems to work well, and I could see how it would work well for some people as a personal organization tool, but I decided that complicating Kanban to accommodate my needs was too much of a compromise, and decided to try something else. Now I know that every day I have time to get things done that don’t have constraints on when they have to be done. Things like writing blog posts, doing assignments, developing software and reading books. Of those that can be done in one session, like blog posts, I simply pick a day, (usually a day when things with deadlines are out of the way is a good choice, for example, weeks ago I had two assignments to do, and I wanted to write this blog post, so I scheduled it for today, the day after the assignments were due). For the other things, like assignments, and reading books, that have to be done over time, I have set up a spreadsheet that tracks my progress in words, or progress or whatever, and knows when the deadline it. It tells me for the current point in time, where I should be, and how far ahead or behind I am as a percentage. It is sorted according to this percentage and I work on (in my regular morning timeslot that I have set aside) whatever thing I am the furthest behind on, or the least ahead. I will try and keep the number of things on this spreadsheet limited though. I am thinking somewhere between 3 and 5 should be the limit. For other things, that can only be done at certain times, or have to be coordinated with other people, you just have to pick a time to get it done and do it then. General todo lists are evil though, anything that can’t be managed by the approach described above will either be done immediately, or not done. I find with my diary, and my spreadsheet there is enough to do without a todo list hanging over my head.

The second year of uni is about half way through, it is very interesting, we only have 4 students in the course. The assignments are getting longer. It will be interesting to see what is in store for us in the next half of the year, or indeed the next half of the course.

In November I attended the XP Days in Germany which I blogged about. Hopefully there is the opportunity to attend further such conferences in  the future. OOP in Munich would have been good, I should have asked to go to that.

So that is about it for the year. I don’t have any goals as such, because I find the things that people usually list as goals are at the opposite end of the cause and effect chain to the things that people can actually control. I am going to think a lot more about values and principles, and try and make decisions according to those values and principles and assume that where that takes me is the right place for me to be. In writing this blog post, I have thought about two more blog posts I want to write. One about which mailing lists I have decided to take part in, and another about my values and principles which could take a bit more thought. I won’t schedule them now though, just keep them in the back of my mind for now.

Blog at WordPress.com.

%d bloggers like this: