Surviving the Transistion to an Agile Methodology, Part 2

So your organization has decided to go Agile, and everyone is onboard, and you've started holding Scrum meetings, and coding in sprints, and everything's going to be just fine, right?

For Sale:  One bridge in Brooklyn

One of the biggest problems is the tendency from members of management to decide that "just this once" you need to circumvent the Agile rules, and (add work to the sprint in the middle|work without a Product Owner|pull your one expert on Technology X "for just a few days")

There is usually one manager who continually subverts the plan.  S/he will never say they are not doing Agile, but they will never really get with the program.

On the flip side is the manager who is perfectly fine with doing Agile development, but wants the metrics to keep improving, since they will be measured by the metrics, even if the metric is explicitly not a performance metric (like team velocity).  So this manager will be gaming your estimates so that their team looks like it is always improving.

Another pitfall is the gradual falling away of the sprint planning rules - that the team should make the estimates, that the stories need to be broken up into tasks, and that tasks should be self-assigned.  You gradually see managers trying to re-assert their traditional roles and hand out work based on their own ideas of effort.

Finally, the big Agile-slayer might show up - field support.  Nobody knows how log it will take to handle field issues, because the level of information can vary greatly.  Often you have customers that have political agendas keeping them from giving you good info, or sometime it's just really clueless support people.

Surviving a transition to Agile methodology - Part One, Defining the Problem

Greetings all,

There are tons of books, courses, blog posts, and probably YouTube videos about how to convert your organization to Agile, but I don't recall seeing many (any?) that talk about how to deal with the pitfalls of such a transition and how to survive them.

The most common issue is that many conversions to Agile are driven from below, as the developers decide that they need to stop doing things the Bad Old Way and get on this Agile train, and this means that they have to convince Upper Management to convert.

This typically fails.

If your Upper Management does not buy into Agile, they will drag their wingtip-shod feet at every point, question every Agile tenet, and resist.

However, this is not my topic - I'm assuming that your Upper Management has either agreed to convert the organisation, or perhaps is insisting that the organization convert - one of the C-levels probably read about Agile in the trade press and wants to get ahead of this new thing.

While this might seem good, it is still dangerous.  The parts of the organization that deal with customers will have to sell the concept to the customers, and that leads into a swamp of problems - the customer doesn't want to allocate resources to work daily/weekly on the scrums; the customer does not want to agree to anything less than a full solution in X months, bound up in a bulletproof contract with penalties for late delivery; and so on.

Then you find that most of the non-development part of the organization doesn't want to spend all of its time thinking about the stuff currently being made, and would rather move on to the next thing, so you can't get the answers you need, or they want to keep using the old waterfall process documentation ("Just write the whole thing out, and you can modify it if you run into anything that changes")

Next in line is typically the old chesnut that Agile will let the organization develop code faster.  Management sees the promise of a faster acceptable product as meaning faster coding, ignoring the parts that mention writing test code, spike solutions, refactoring, and all the rest of the things that mean you will not be writing anything faster from a code perspective.

Moving along, we come to the Misuse of Metrics, in which Upper Management starts to use the team metrics (velocity, number of user stories completed, etc) as part of the performance review process for the developers.  This is usually fatally toxic to Agile adoption, in that the developers naturally become more concerned with saving their jobs by tweaking their teams' numbers to look good.

And tagging along at the caboose of this trainwreck is the expectation that it will go smoothly from the start.  Upper Management, after a sprint or two will realize that often the forward progress is slower than before, so they will begin to "help" by changing things up - reorganizing teams, taking personal interest in the prioritizing of User Stories and tasks, etc.

Now, it's fairly common knowledge in the Agile world that organizations can have pockets of resistance to the conversion, but much of the remedies are on the order of "Give your CTO a stern look and tell him that's not how Agile works", which is hardly helpful at either changing the organization or staying gainfully employed.

In part two, I'll explain some of the things a good Software Ninja can do to survive the whole mess.

Faster, Prototype! Kill! Kill!

Recently at The Job, the upper management types have been talking about Agile as a way for us to get to market faster.  The problem is that they keep addressing it towards the developers, implying that they will be finishing development faster under Agile.

However, one of the big things about Agile is that developers are often writing more test code, and spending time writing spike solutions, which means that there is more code overall to write.

Sure, the focus in Agile on automated testing removes the later debugging hassles, but those are not the "Concept to Deployment" bugs that are getting pre-emptively squashed, but the field issues.  We developers still need to write the fully functional code, and refactor it.  Agile limits the amount of code needed per release, but requires more releases to reach the desired feature set of a full product

So we're facing an executive team that thinks Agile will cut development time in some grand way, while we face all the usual headaches of transitioning to Agile, as well as the actual development time.

The Paradox of the Human Condition

I'm going to wax a bit philosophical here, in contrast to my usual technocentric rants, so bear with me.

I was struck with a realization yesterday that the two traits that allow us humans to do such great things are the same traits that allow us to do such horrible things.

The first trait is the ability to sublimate our personal needs to that of a grater cause.  This leads us to do such things as give up a comfortable life to travel to an undeveloped land and help the local people improve their condition; to fight against injustice in the face of personal attacks; to jump onto a grenade to save our comrades.  We can identify with a cause larger than our own lives, and work towards furthering that cause at the expense of our own well-being.
But is also leads us to harm those who do not share our history; to pilot planes into buildings; to attempt to eliminate entire peoples because they are different.  It is the adoption of an greater goal that removes our sense of kinship to the rest of humanity.

Conversely, the second trait is that selfishness that drives us to excel at collecting wealth that can be used to do great good.  The billionaires that fund disease-eradication efforts, or publicize tragedies in far-off countries; the driven individuals that want to build rockets, or non-polluting cars - they expect to get rich from it, along the way.

But this leads to the seduction of power.  Once one has enough money, one becomes accustomed to using that money to influence the rest of the world, and then that distorts the political process in the local governments - either directly through bribery and outright force, or via indirect means - funding 'policy groups' and industry councils that present an individual's view as a public opinion or social good.  Soon any deviation of society from the personal preferences is decried as a bad thing.

So we must learn to walk the middle way, somehow, avoiding the extremes of selflessness and of selfishness, and tempering our personal view with that of society's.

The Red Queen Problem in Software Development

I was listening to one of The Job's internal meetings the other day, and the CEO was talking about how we needed to do things better and faster than ever.  I got angry, because this is exactly the thing that is screwing up everything in the tech industry - nobody wants to do the right thing, because that would take longer and might allow some other company to actually be the first with a crappy product in that niche.

Then I read Drew Crawford's "Of Wizards and Magical Machines", and damned if he didn't capture some of the things I wanted to say here.

Software has come to be seen as something easily changed, taking no time to alter and configure.  Management thinks that a bar napkin counts as requirements, and there is no need for clarification or details - we can pull those out of thin air and it's the developers' fault if the guesses are wrong.

(I have a paranoid theory that software company management hates the developers because development is the only phase of the software life-cycle that cannot be skipped, and we developers continually whine that we need more time to create the castles in the sky that they think up)

But we can't always be faster - the problem takes a finite amount of time to understand, and another finite amount of time to analyze, and only then can we start to develop it correctly.  No amount of Agile process will remove that time.

Bug-Ception

It amazes me that people writing bug-tracking systems never seem to find all the right points to track.

At The Job, the one we have was written under the (apparent) assumption that nobody ever has more than 1 release stream, where a bug might be present in both streams, and need to be tracked as the same bug, but with different fixes (either due to schedules for the releases, or for differing code in the streams)

So we get a mess of duplicated bugs because each tester thinks of a different way to describe the bug, and can't search for the kindred bug, since it will have a different ID in the other streams

Then the process geeks came along and decided that we could keep all the releases under one bug.  Of course, that leads to total confusion about if it's been fixed in any specific stream.

Here's a tip

When you are writing a set of methods for something where you want to get the stuff compiling, but you are not looking to do full TDD, because you are worried you will forget something, then just put assert(False) in the bodies of the methods, to make sure that they will stop the system if they are run before you have the code in them.

Languages that don't have assert() can use something else - a macro that calls exit().

All you want is something that will stop things dead if you forget to fill it in.  This is better than log message or output, because you can't forget to make it work