Surviving a transition to Agile methodology - Part One, Defining the Problem

Greetings all,

There are tons of books, courses, blog posts, and probably YouTube videos about how to convert your organization to Agile, but I don't recall seeing many (any?) that talk about how to deal with the pitfalls of such a transition and how to survive them.

The most common issue is that many conversions to Agile are driven from below, as the developers decide that they need to stop doing things the Bad Old Way and get on this Agile train, and this means that they have to convince Upper Management to convert.

This typically fails.

If your Upper Management does not buy into Agile, they will drag their wingtip-shod feet at every point, question every Agile tenet, and resist.

However, this is not my topic - I'm assuming that your Upper Management has either agreed to convert the organisation, or perhaps is insisting that the organization convert - one of the C-levels probably read about Agile in the trade press and wants to get ahead of this new thing.

While this might seem good, it is still dangerous.  The parts of the organization that deal with customers will have to sell the concept to the customers, and that leads into a swamp of problems - the customer doesn't want to allocate resources to work daily/weekly on the scrums; the customer does not want to agree to anything less than a full solution in X months, bound up in a bulletproof contract with penalties for late delivery; and so on.

Then you find that most of the non-development part of the organization doesn't want to spend all of its time thinking about the stuff currently being made, and would rather move on to the next thing, so you can't get the answers you need, or they want to keep using the old waterfall process documentation ("Just write the whole thing out, and you can modify it if you run into anything that changes")

Next in line is typically the old chesnut that Agile will let the organization develop code faster.  Management sees the promise of a faster acceptable product as meaning faster coding, ignoring the parts that mention writing test code, spike solutions, refactoring, and all the rest of the things that mean you will not be writing anything faster from a code perspective.

Moving along, we come to the Misuse of Metrics, in which Upper Management starts to use the team metrics (velocity, number of user stories completed, etc) as part of the performance review process for the developers.  This is usually fatally toxic to Agile adoption, in that the developers naturally become more concerned with saving their jobs by tweaking their teams' numbers to look good.

And tagging along at the caboose of this trainwreck is the expectation that it will go smoothly from the start.  Upper Management, after a sprint or two will realize that often the forward progress is slower than before, so they will begin to "help" by changing things up - reorganizing teams, taking personal interest in the prioritizing of User Stories and tasks, etc.

Now, it's fairly common knowledge in the Agile world that organizations can have pockets of resistance to the conversion, but much of the remedies are on the order of "Give your CTO a stern look and tell him that's not how Agile works", which is hardly helpful at either changing the organization or staying gainfully employed.

In part two, I'll explain some of the things a good Software Ninja can do to survive the whole mess.

Faster, Prototype! Kill! Kill!

Recently at The Job, the upper management types have been talking about Agile as a way for us to get to market faster.  The problem is that they keep addressing it towards the developers, implying that they will be finishing development faster under Agile.

However, one of the big things about Agile is that developers are often writing more test code, and spending time writing spike solutions, which means that there is more code overall to write.

Sure, the focus in Agile on automated testing removes the later debugging hassles, but those are not the "Concept to Deployment" bugs that are getting pre-emptively squashed, but the field issues.  We developers still need to write the fully functional code, and refactor it.  Agile limits the amount of code needed per release, but requires more releases to reach the desired feature set of a full product

So we're facing an executive team that thinks Agile will cut development time in some grand way, while we face all the usual headaches of transitioning to Agile, as well as the actual development time.

The Paradox of the Human Condition

I'm going to wax a bit philosophical here, in contrast to my usual technocentric rants, so bear with me.

I was struck with a realization yesterday that the two traits that allow us humans to do such great things are the same traits that allow us to do such horrible things.

The first trait is the ability to sublimate our personal needs to that of a grater cause.  This leads us to do such things as give up a comfortable life to travel to an undeveloped land and help the local people improve their condition; to fight against injustice in the face of personal attacks; to jump onto a grenade to save our comrades.  We can identify with a cause larger than our own lives, and work towards furthering that cause at the expense of our own well-being.
But is also leads us to harm those who do not share our history; to pilot planes into buildings; to attempt to eliminate entire peoples because they are different.  It is the adoption of an greater goal that removes our sense of kinship to the rest of humanity.

Conversely, the second trait is that selfishness that drives us to excel at collecting wealth that can be used to do great good.  The billionaires that fund disease-eradication efforts, or publicize tragedies in far-off countries; the driven individuals that want to build rockets, or non-polluting cars - they expect to get rich from it, along the way.

But this leads to the seduction of power.  Once one has enough money, one becomes accustomed to using that money to influence the rest of the world, and then that distorts the political process in the local governments - either directly through bribery and outright force, or via indirect means - funding 'policy groups' and industry councils that present an individual's view as a public opinion or social good.  Soon any deviation of society from the personal preferences is decried as a bad thing.

So we must learn to walk the middle way, somehow, avoiding the extremes of selflessness and of selfishness, and tempering our personal view with that of society's.

The Red Queen Problem in Software Development

I was listening to one of The Job's internal meetings the other day, and the CEO was talking about how we needed to do things better and faster than ever.  I got angry, because this is exactly the thing that is screwing up everything in the tech industry - nobody wants to do the right thing, because that would take longer and might allow some other company to actually be the first with a crappy product in that niche.

Then I read Drew Crawford's "Of Wizards and Magical Machines", and damned if he didn't capture some of the things I wanted to say here.

Software has come to be seen as something easily changed, taking no time to alter and configure.  Management thinks that a bar napkin counts as requirements, and there is no need for clarification or details - we can pull those out of thin air and it's the developers' fault if the guesses are wrong.

(I have a paranoid theory that software company management hates the developers because development is the only phase of the software life-cycle that cannot be skipped, and we developers continually whine that we need more time to create the castles in the sky that they think up)

But we can't always be faster - the problem takes a finite amount of time to understand, and another finite amount of time to analyze, and only then can we start to develop it correctly.  No amount of Agile process will remove that time.

Bug-Ception

It amazes me that people writing bug-tracking systems never seem to find all the right points to track.

At The Job, the one we have was written under the (apparent) assumption that nobody ever has more than 1 release stream, where a bug might be present in both streams, and need to be tracked as the same bug, but with different fixes (either due to schedules for the releases, or for differing code in the streams)

So we get a mess of duplicated bugs because each tester thinks of a different way to describe the bug, and can't search for the kindred bug, since it will have a different ID in the other streams

Then the process geeks came along and decided that we could keep all the releases under one bug.  Of course, that leads to total confusion about if it's been fixed in any specific stream.

Here's a tip

When you are writing a set of methods for something where you want to get the stuff compiling, but you are not looking to do full TDD, because you are worried you will forget something, then just put assert(False) in the bodies of the methods, to make sure that they will stop the system if they are run before you have the code in them.

Languages that don't have assert() can use something else - a macro that calls exit().

All you want is something that will stop things dead if you forget to fill it in.  This is better than log message or output, because you can't forget to make it work

The Long Dark Night of the (Professional) Soul

I had a somewhat upsetting epiphany today.

Over the past couple of years, I've had the chance to take a number of job-provided training.  Some of it was mandated by The Job, and some was my choice.  In that time, I've noticed that even though I don't mind the classes in some cases and in others actually really *like* the class, I spend the days before and after in something akin to dread.  It creeps up on me and lingers around the time I take the class.

Today I realized what it was.

It's the awareness that I can't use the stuff I'm learning to improve my code, because The Job won't let me.

The Job has tight restrictions on the number, size, and type of code changes that can be made to released products.  Almost all of the things that need to be done to the code are non-trivial changes of infrastructure, medium to large refactorings, additions of many unit tests, etc.  All of which require Change Notices, which need approval for altering code that gets released to the customers.  And all of which will not pass the current standards, which basically are "Is some customer aware of it, and complaining about it?"

So I sit in class learning many new and wonderful things, and never get to use them in my main coding space.  And it makes me unhappy.

It's Hard to Soar With The Eagles When You Are Surrounded By Turkeys

Some days you just can't win.

After a hectic week, I can definitely say that one of the big demotivators I've run across is the dismissive attitude you sometimes find in quasi-competing groups.  I found out that one such group has been making interface changes and not passing the info back to my group because it was their fervent belief that we would be deprecated Real Soon Now, and our product would never need to connect over the modified interface.  This despite having at least 2 prior cases where that belief was proven false, and I had to make sudden adjustments to my product to cover for their omissions.  

It is poisonous to morale, this empire building mentality.

Don't let that behavior go on at your company

Optional, Expected, Mandatory

Hello again!

Today I got reminded that not matter what anyone tells you, there is no such thing as a temporary feature.  Once you've added it to your system, even as a hidden setting, your support staff will talk about it, and soon customers will ask about it, and it's not long until someone depends on it, and after that, you're getting enhancement requests for it.

So when you are writing the code for that kludge that one collaborator needs just to get things past integration testing, you need to think like this will be a fully-supported, documented feature, because it soon will be.

Don't use an environment variable when you have a database.  If you can't add it to your database immediately, write the code to make it easy to convert when you do get a chance to change the schema.

Don't expect it to be ok to set this once at process start and never re-read the value until the next restart.

Don't slip the code change into a side branch of execution that suits just that one customer/collaborator/installation.

Do write a Change Request to fix the kludgy nature of this in the immediate next release

Invisible Pains

This past week, I ran into a situation that I have seen in the past, but never quite grokked fully.

At The Job, the tech support organization is fairly savvy in the ways of computers, so they have a good number of programs and scripts they use to gather data on issues, fix known problems, etc.  What happened was that there was a customer experiencing a problem, and Tech Support had written a script that fixed that problem, running nightly.  The customer was happy.

What caused the problem was that Tech Support did not come back to development and let us know that there was a problem ongoing that they were covering for.  So the problem festered, and now we have a deadline and a problem that we do not have time to fix and run through our QA process in time.

So when you are talking to your Tech Support people, be sure to ask them if there are any ongoing issues that they are handling that might be fixable in the product code, so tha you can solve them before they become big issues.  The Tech Support people usually love to be the fixers, but they sometimes let that override their duty to report things up the line to get real fixes.

Something Sure Is Rank

Recently I ran across a few articles discussing the not-uncommon corporate HR practice of stack ranking.  Frankly, my jaw dropped when I read that that practice has a lot of traction in the corporate world.  The premise is so flawed that I cannot believe so many people swear by it.

For those not familiar, stack ranking is the practice of ranking a set of employees in a linear fashion, from worst to best, and then marking the top 20% as excellent, the middle 70% as adequate, and te bottom 10% as failures, and suitable for dismissal.  It assumes that the population follows the familiar bell curve.

So let's take a look at the problems with this.  First the technical ones

1) Sample Size
The bell curve ( or normal curve) is the expected distribution is valid only for sufficiently large populations.  Most workgroups are not nearly large enough that they would match the normal curve in the distribution of ability.  Use of stack ranking on small groups can incorrectly mark people of lesser ability as part of the bottom 10%.

2) Random Selection
The normal curve also assumes that there is no skewing influence on the population.  Most companies proclaim that they are selective in their hiring, so they should not have any poor performers.  Moreover, if the company has run any rounds of performance-based layoffs, the population of employees is no longer normally distributed.  Once you remove the lower 10% of the population, there is no longer a "lowest 10%" to choose from.

And now the social reasons against stack rank:
3) Arbitrariness
Since most people are aware of the basic fact that statistics requires large groups of people, they know that in a small group there can't be a bottom 10%.  So when one of them has to be sacrificed as the goat, they begin to see all the rankings are either arbitrary or bogus, so they become cynical as to the merit of the evaluations

4) Gaming the system
Once employees know that they are ranked against each other and avoiding being in the "bottom 10%" is vital, they will start to alter their behavior to try and avoid being the low man.  This leads to anti-social activities such as information hoarding, sabotage, and grandstanding.  They stop trying to further the group goals in favor of their own.

Given all this, I cannot understand why this is favored by corporate management except as a means to instill fear in the employees and enable the rewarding of those liked by the managers.

Getting Input, But Not The Wrong Kind

One of the things that I noticed a while back is that quite often it seems impossible to get others to provide their input to a design or document.  Nobody wants to respond to your questions, or doesn't have the time, etc.

So I started presenting every document or design as a strawman - I made sure to put in things that I thought good but controversial into every section.  Then I sent it out for review as version 0.1.

The controversial ideas would galvanize the reviewers to respond with their better ideas.  It seems that while few will stick their necks out to propose solutions, almost everyone is comfortable with criticizing someone else's solution.  It seems to allow everyone to assume their preferred role in the organization - wise sage, gadfly, elder statesman, or just plain old curmudgeon - without risk.  It's not their idea that's being looked at, and if their input is ignored, they are free from blame, but if the idea is a success, they contributed to the discussion, and therefore helped bring it to market!

Curiously, there is a counterpoint to this situation - sometimes you have a manager/PM/customer who cannot allow a document to pass by without making some changes.  I'm not the only one to notice this, not by far.  And the solution is exactly as described - make some very specific things either wrong or superfluous, and let the manager/PM/customer tell you to change them. 

A corollary to this happened to me at my first job.  We had a group of senior customer executives visiting to see our initial demo of the product.  We showed them what we had - a mockup running on a terminal, and then they were shown the various other bit s of technology that were being assembled into the final product.  Now, this product had some very specific color-coding of icons on the UI, and we were already planning on making the color specifications loadable from a configuration, to allow for the differences in the color monitors of that time, yet we had the whole group of executives staring at a screensaver trying to point out some specific colors that specific icons needed to be.  It was something that was best dealt with in a document, but they felt the need to impress us (or each other) with their acumen at determining between close shades of magenta.

False Urgencies

Recently, The Job has offered me a number of occasions where the usual software development process of asking for an estimate has been skipped entirely,  Several times, the Manager is sending out an email announcing the next code freeze in 2 days, with a list of unfinished bug reports.  So there is a sudden scramble to finish things that night well be too much to finish in the remaining time.

This rant about bogus deadlines addresses this issue.  One of the tenets of good software is that is takes some time, and it is at least somewhat a creative endeavor.  So rushing your developers just to meet an arbitrary deadline is counterproductive.

Give them room.  Provide the list of desired changes, get estimates, and then juggle the schedule or the list to fit.

Oh, and never think that twice-daily, 2+hour progress meetings with VPs, EVPs, and the rest of the C-level wannabees are going to help.  Nothing ruins concentration more than knowing that in 3 hours you are going to have to explain why you haven't made "satisfactory" progress solving a problem, where "satisfactory" means "saving my division customer satisfaction rating so I get my 35% bonus" for the EVP

The Tyranny of the Best

I ran across this post about different skill types among programmers and it struck a chord.  I've seen a great many places described as looking for "The Best Programmers", implying that the rest of the programmers need not apply.

As that post notes, those places may be missing out on some other Best Programmers due to their different approaches to things.  However, I think that it is also a bad thing to expect all your developers to be the best at their craft.

If every company is gunning for the best, the salary budget will be stressed - if they are the best, won't they expect the best pay, or at least more than the average/market rates?  So there will be smaller teams, or fewer teams of a given size, which means your company will not be able to work on as many things.  This might be ok in a startup, where the goal is often a single product with relatively few sections, but in a large established company, there can be hundreds of projects.

And then what happens when your Bests decide to jump ship, because your projects no longer excite them, or the next company offers a better deal?  You will have to hire some lesser lights, and then either you will hit a slowdown in your development rate, or you will see problems.

Of course, if you let slip that the current team is not regarded as highly as their predecessors, you'll have a morale issue.  Nobody likes to be looked down on, especially if they are working hard.

So think long and carefully about the phrases you put in your job ad, or you might end up driving off some people that would do a great job.

A Little Python Partisanship

Although most of my time at The Job is spent in compiled languages, I have been trying to develop some skill in Python to enliven my technical skills.

This week, I got a chance to try something that I know would have been a real pain in a comp[iled language like C++.

The gist of the issue is that I want to automate some integration tests I'm writing for the system, so that I can just run them all with one command.  I am trying to make them as modular as possible so that I can add new ones easily, since I know from past experience that if it takes any significant time to add tests, I will not have the time to add them.  So I come up with a skeleton program that will run the tests, dividing them into preconditions that must pass, the test itself, and postconditions that must pass.  Now I run into the issue that at the moment I do not want to package my tests, so I need to import the modules directly and individually.  However, that makes it hard to add new tests, since the import list will grow with each test.

A quick search of www.python.org/documentation and I find the import statement is a shortcut for __import__(), and __import__() can take runtime strings as arguments.  This means I can pass the test modules' names into the test skeleton.  A little help from exec/eval and I can call an arbitrary function from my test module based on command line input.

Here's the code I have so far:

d = {}
with open(sys.argv[1]) as f:
for line in f:
(key, val) = line.split()
d[key] = val

execStmt1 = d['modulename'] + " = __import__('" + d['modulename'] + "', globals(), locals(), [], -1)"
execStmt2 = d['modulename'] + '.' + d['functionname'] + '(d)'

print execStmt1
print execStmt2
exec execStmt1
exec execStmt2
targetFunction = eval(d['modulename'] + '.' + d['functionname'])

And the data file sent in looks like this:

modulename dummy
functionname dummy2
arg1 1
arg2 2


And the dummy module dummy.py looks like this:

def dummy(arg1, arg2):
print "dummy(", arg1, ",",arg2,")"

def dummy2(d):
print "dummy2(",d['arg1'], ",", d['arg2'], ")"

if __name__ == '__main__':
dummy(0,0)

So this gets me most of the way there.  I still need to rework how the preconditions and postconditions are run and their return values checked, and a few other things, but this was just so amazingly EASY to do, I thought it was worth a post

Taking Out A Contract

Earlier this week, I was involved in a discussion about the relative merits of Design By Contract vs Defensive Programming.  While both are good, I was struck by the thought that in my experience, it's hard to get other developers to keep up their end of the bargain for DBC.

For example, one protocol I programmed to had a field in one message that specified the maximum time it would take for a file to be transferred across a link,  When we wrote the code, the client received that message, took that time value, added a small fudge-factor to account for some specific conditions on their end, and all was well.  However, at some later point, a new group of programmers for that project decided that "No file will ever take longer than 30 seconds to be transferred", so they would wait a maximum of 30 seconds before timing out, even though we routinely had files that took 2 minutes to transfer.

Basically, any program that is built in a work environment where the original programmers will be replaced at some point is almost guaranteed to have one or more of the new programmers ignore the constraints at some point, rendering the contract useless.  And unless you're programming in Eiffel, you get bad results.

Defensive programming, however, differs in that the receiving side will check the inputs and can complain when its expectations are violated.  The big problem with Defensive programming is that it tends to clutter up the code, and foster paranoia.  The compromise I have come up with is to but string defensive checks on things that cross object boundaries - the public methods, API calls, etc.  It's generally easy to at least document the expectations within a class so that later developers will see them and follow them, but once you have to call something defined in another file/directory/subsystem, you risk not being able to find the details.

Hey, some programming content, at last! Finite State Machines!

So I decided that I've been using this blog as a ranting place instead of an informing place.  That;s not really what I was intending when I got started, so I'm going to try and focus on informational content for a while.  Not quite a New Year's Resolution, but it's worth a try

Today's Topic:  Finite State Machines
Finite State Machines (FSMs) are a powerful tool for handling program behavior in a flexible manner without losing track of the details.  If you are not familiar with FSMs, here's the Wikipedia article on them, which is a bit obtuse, but gives some of the details.

Typically, you will use an FSM in a situation where you have a set of behaviors you need to model, and the actions that get taken on certain inputs depend on the state of the object.  Most examples use mechanical devices, like turnstiles, automatic doors, traffic lights, and vending machines.

The real power of FSMs comes from when you combine a generator utility (easily found online) to generate the skeleton code for the state transitions, and the Command pattern to encapsulate the events/inputs.  This leads to compact code for the transistions, and lets you concentrate on the meat of your functionality.

The GoF State pattern might be appear to be useful here, but that is more helpful when you have something that does multiple things depending on its internal state, as opposed to the FSM itself.  The example in the link shows that this is less about the transitions and more about the actions, with the transitions not doing useful work, or being almost dataless signals.  The typical FSM even is more diverse, being a value or a signal whose meaning will differ.

One of the drawbacks to FSMs is that the logic of the transitions can be absent from the code, making it difficult to follow.  This is of course solvable with good documentation, but you need to keep it clear.  Most of the FSM generating programs have a data file that drives the process, so keep the descriptions there, and link to the in the generated/skeleton code, or if you generate skeletons and then modify them manually, in that code.

One other common use of FSMs is in regular expression recognition, but that's fairly tightly tied into the fields of compilers and interpreters, which is a really complex subject to cover.

Get Outta My Way!

You know what I really hate about software development process?

When the blasted parts of the process fail to work and make more work for developers!

Case in point - The Job has a number of scripts that are run when the product is near a final build for a release, that check for all the expected fixes being complete, all the process steps are done, etc.

So today I get the nasty email from the PM that I have some unfinished fixes.  I look, and I've done everything that I am supposed to do, all parts are present, but the scripts are not finding them.  And this was noted a week ago and nothing was done when I noted that part of it was clearly wrong (as in I showed the supposed missing parts by ID)

But the dysfunctional process favors PMs bothering developers instead of following through on responses.

I repeat the gathering of part IDs, and then have to re-save some things to trigger the scripts again.  Why they do not run automatically on the production build, I'll never know.

---

The Horizon Effect

Today I got started on one of the more fun things in software development - removal of old code.  Not a refactoring of functionality, but outright removal.  It's been a while since I had the chance to do this.

It's cool, getting to excise code and make the codebase smaller and more efficient.  The feature unfortunately was not wrapped in a simple method or methods, but the functions and data are all well-named, so it's a simple search and destroy replace.

But I was musing that this was something that gets ignored by many shops - getting rid of old features that are unused.  Old hardware, obsoleted features, invalidated combinations, all of these lurk in many (most?) applications, waiting to spring on the unwary.  But I rarely see mention of this pruning in the literature - refactoring gets mentioned as a response to changing requirements, but the requirements-gathering always gets described from the viewpoint of a new project, not a mature product.

One of the hard parts is that it's often impossible to determine which features are no  longer valid, because the reasons for them in the first place are lost in the mists of time (or Bob, the original architect, retired 2 years ago, and is sipping Corona on the beach at Cozumel).  So I was thinking that it would be a cool feature of an advanced source code management system to tag a set of changes with an expiration/re-examination date, so that the developers would be reminded of the need to do this, and contain the explanation of the feature so that this later validity could be checked.

---

New Year Hopes, Plans, and Ideas

Well, it's now January 1st, and it's time for the obligatory New Year's post about resolutions, plans, and so on.

What, you thought I was going to do something different?

So here are my plans:

Professionally - improve my coding skills; work more on getting testing infrastructures built around my work code; start learning more about the other areas of The Job's new products; build something useful to the public in Python; learn more about VMs and use of the cloud.

Personally - sustain an exercise program for more than 2 months; work on my photography project; declutter the house (at least one item per day disposed of by trash, sale, recycling, or reuse); blog at least once a week; write some fiction every month.

Socially - well, let's just say I need to see how much time I have left after doing all of the other things.

I'll post later with the list of specific things of interest to technical types.

---