Software Infrastructure and You

In a recent rant, Rockford Lhotka tears into developers as spending too much time on the software infrastructure and not enough on delivering business value. He uses a Visual Basic application written some time ago as an example of something that worked just fine then, and is even better now that PCs have increased speed by at least one order of magnitude.

As a developer, I quite naturally take offense to this. We are not, by and large, avoiding improving the product by playing with the background stuff. Here are a few counterpoints:

1. Business app are rather pedestrian to write, but the bigger question is why are we still writing them? Because someone wants something a little bit different from anything else out there, and instead of living with the existing applications, they want someone to build an application that fits their precise needs. IBM had a small business suite back in 1985 that covered everything, and I mean everything, so all businesses should be using that app now, right? But instead, a business-type wants to be creative, and requests new software.
2. He says that the customer doesn't care about infrastructure. That doesn't mean infrastructure is not important. The customer may not care about the condition of the roads when he asks for his supplies to be delivered to him and his products to be delivered to the buyers, but bad roads will damage trucks, old narrow roads will delay trucks, and low overpasses will stop trucks, and the customer will lose money. The infrastructure of the dialup days will not handle today's DSL/cable modem traffic levels; today's high-perfomance network no longer needs the heavy error-correction protocols required by the slow and noisy networks of old. A prime example of this is email - remember when an email address was a series of machine names separated by "!"s, with other odd characters in the last part? It was no worse an email system then that it is now (as far as getting a message from one computer to another), but the infrastructure changes (DNS, STMP/IMAP/POP everywhere, and generally universal TCP/IP connectivity) have made email much better for the end-user. [And due to some original decisions about this infrastructure, worse, from SPAM, because the infrastructure was not developed with expectations of anti-social use. To fix this, there is ongoing infrastructure work in a number of projects]
   
3. He bemoans the trends to web services and similar things away from the older methods. In my experience, one of the biggest forces in driving changes in infrastructure is the business press. I recall one boss asking I investigate moving our application to a CORBA interface, because he had read about CORBA in the business press and thought to jump on the bandwagon. There was no reason to move to CORBA at that point - our application was only talking to other applications we wrote, and the socket-based protocol was adequate for our expected needs. Other times, the move to new technology is mandated by the government, like the demise of analog TV signals.
4. Another case is that sometimes the customer drives these changes. The big draw of web-based services is that the customer does not need to install something on every PC in the organization in order to use the application - each user has a PC with a web browser already on it. Or the customer favors something else. I know of one vendor whose application was chosen over another vendor's because it had drop shadows on the buttons, while the application not chosen used Motif widgets that "only" had bevels on its buttons.
   

Sure, given the choice, I'd rather develop something cool, instead of Yet Another Accounts Payable system. But plenty of customer-value things are cool - Firefox/Thunderbird, flickr, deli.cio.us, blogging, P2P software, VOIP, and so on - we developers hardly need to play with infrastructure to be doing cool stuff.

---

Technorati Tags --
Software
SoftwareDevelopment
Computers
Programming

Offshoring Software - a Developer's Opinion

A brief scan of Google's Blog Search looking for other blogs on software development and I happened upon a post by a Rockford Lhotka about how software is too hard, and that the developers have brought offshoring down on themselves because they are always re-inventing things that already work well enough for the business purposes.

Well, I have an opinion about that (qu'elle suprise)

Offshoring is driven almost totally by costs. Companies look at what it costs to develop software, and it's significantly cheaper to do it offshore in India (or sometimes China), because of the much lower cost of living over there. The additional expenses of getting a reliable development center set up, sending managers over there to hire and spin up the offshore team, and communicate are smaller than the saving on salaries and benefits. So the company have a lower total cost to develop, and that looks good to them.

There are 2 flies in this profitable ointment, however: the offshoring boom is driving Indian developer salaries up fast; and dealing with a remote team from a different culture, with significant communication barriers, makes it difficult to develop good software.

Let's take a quick look at the first point - salaries. Indian cities like Bangalore, Hyderabad, and Chennai (formerly Madras) all compete for the local developers. The boom has brought many US companies to the country, where they have lots of jobs to fill. So they offer better salaries to attract people. Then the next big company arrives, and has to offer bigger salaries to draw people their way. Sure, there's a long way to go before they cost as much as Americans, but they are cutting into the "big savings" with every raise. A corollary to this is turnover - as salaries rise in Bangalore, the Hyderabad developers see a chance to better their lot by finding a job over there; 2 months later, Chennai salaries create envy in Bangalore developers.

Now for the second point - developing with a remote team. I often find it ironic that American managers who won't let their staff telecommute "because if I can't see them, how do I know they're working?" are willing to create an entire department half a world away where they will meet face-to-face no more then once a month at best. Indians in software all speak English fairly well, in theory. But quite often the combination of accent and speech patterns conspire to make them difficult for Americans to understand, particularly over the phone. This make collaboration less efficient, and truly harder to resolve misunderstandings.

The time difference is another factor. If the finishing touches to a product must be made in the US, the Indian team will need to have finished them the previous workday, meaning 36 hours ago, to allow for their changes to percolate through most CMS systems. Even if your CMS system can take changes rapidly, the 10.5 hour time difference means that the morning is spent pulling in the India work and rebuild the application, which for local development could have been done in the overnight hours. And if there are bugs, the Indian team has gone home for the night, making it harder to get their input to fix things. So the Indian team has a deadline that may be 2 days earlier than the local team.

Another issue is the development culture. American developers, especially after the dot-com boom, are wont to question everything about a design. While this can be very frustrating for the designers, it does tend to flush out any problems with the design, the design documents, and sometimes even with the project itself. I have found a fair number of errors in the documentation of a design by questionsing the intent. From anecdotal evidence, significant numbers of Indian developers do not question such things, for whatever reason, leading to incorrect software. It may be that they see the inconsistency, but do not feel comfortable bringing it to the attention of the designer, or that they assume that the designer intends it for a reason, but does not explicitly state the reason. Whatever the cause, it does not further the project. Add to this the reluctance the remote team may have for reporting difficulties they are having back to the home office - few people like to report bad news, and the cultural divide makes this reluctance greater.

A large proportion of Indian developers have come out of longer academic careers than typical for American developers. This is problem in a production environment, regardless of the location of the developer, because the rules of software development in academia are quite different than in business. Many former academic programmers have difficulty transitioning to an environment where the projects are usually quite detailed on how it looks, and less detailed on how it works, yet requiring high-perfomance output. I've seen fresh-out developers struggle over what to name database columns, because their boss did not give them a detailed naming plan. Other common academically-induced quirks are oddball variable naming schemes (sports teams, etc), and a really rigid view of commenting style.

[I hasten to add here, that I'm not in the least saying that Indian developers are not smart, or capable of developing killer software, but rather that in the context of developing software as part of an American company, they have some hurdles to reaching peak performance, most of which are purely part of the distance and differences in language and culture. Such issues arise between American and European development teams, as well, but the greater commonality of history makes the cultural issues a little less complex.]

And another thing - one of the key issues with a remote team is being able to provide them with good documentation - requirements, architecture, design, and so forth. Many (most?) American development shops, frankly, suck at development documentation. So, no matter how great the Indian team may be, they may not have any chance of developing the desired product because they don't know features to develop!

Part 2 of this will address the other side of Rockford's rant - that developers spend too much time fiddling with the internals of software and not enough time working on business value.

---

Technorati Tags --
Software
SoftwareDevelopment
Computers
Programming

Meeting your Schedule - A Few Tricks

So your boss has just walked in and informed you that the project has 4 new requirements from the customer, one of which forces a redesign of some basic part of the project. The deadline doesn't change, of course, because that was set in a contract a few months ago, and the customer would be shocked at the suggestion that these newly-discovered requirements should make the project take longer!

I can't tell you how to redesign your project to meet the new requirements, but I can give you some tricks that, taken at the start of the project, can make your life easier when this situation occurs - and believe me, it will occur.

1. Buy and read "The Pragmatic Programmer". I cannot stress enough how utterly cool this book is for developers. This book codifies a great deal of wonderfully useful information.
   
2. Put your program configuration into a file or a database. Resist the urge to use environment variables. If you make your programs read from a database or file from the start, and you have a trigger to re-read the configuration when desired, you will be able to adjust configurations on the fly.
3. Put your configuration in a Singleton object, and have methods that return the various values you need from this object. The re-read function can be done either as an explicit method, or part of the constructor, where the signal to update the configuration will delete the configuration and the next use will resurrect the object from the modified file/database.
4. (added later) If you have a configuration option that is intended for a old feature/new feature type of thing, write the code to default to the new feature version in the absence of any configuration, and only do the old feature stuff if the configuration is set to 'use old'. This will allow you to phase out the configuration flags as the old features drop of the supported list, and your code will be cleaner as you delete code the references the unused configuration flags.
   

The point of all this is to make it easy to introduce flags and values for new features into your system, so that when the boss tells you that there are new requirements, you at least can have your programs detect these variations easily and in a consistent manner. You need to build this in from the start, so that the code will have only one method of getting configuration information and you will not have to spend much time tracking down oddball data accesses.

---

Technorati Tags --
Software
SoftwareDevelopment
Computers
Programming

Development in the Corporate World, Part 2

Schedules are the trickest part of software development, because it is difficult to predict the future, and predict software complexity from a distance. Predicting just one would be a feat to rival the Amazing Kreskin, but the two together....

From a business standpoint, schedules are vital; businesses must coordinate their activities, and the schedule is the major junction of alignment. So the use of a schedule is seen as a binding agreement, the Holy Grail of the endeavor. So it is set in stone, early in the process, and rarely negotiated.

From a development standpoint, schedules are vital, too. But they are fluid - things change as design uncovers hidden assumptions, and new requirements are uncovered. As new work is added, the sane assumption is that the schedule will expand. But because of the business importance of the schedule, it cannot expand. So in the grand tradition of "Good, fast, cheap; pick any 2", the time dimension is fixed, and the only variables are cost and quality.

Now, compunding this is the problem that attempting to maintain quality by spending more money doesn't work. If the bosses intend to give bonuses to the existing team to reward them for overtime (a rare and noble group of bosses, indeed), it will work to a certain extent, until the team hits the wall from overwork. If the plan is to increase the team, they run into Brook's Law - adding manpower to a late project makes it later. The effort of bringing the new people up to speed, and keeping the larger team in communication will swamp any benefits of added manpower.

So it appears that the only likely outcome is for quality to go down.

Now that I've painted a doom and gloom scenario, I'll give some suggestions on how to better survive when you're working in a shop where schedules are set in stone.

1. Learn how to estimate your work accurately. If you don't have a good feel for how long it will take you to develop a piece of software, you can make accurate estimates
2. Learn your company's overhead load. Some companies have a process that's very streamlined - developers are allowed to spend most of their time developing. Others have lots of meetings, documentation, etc., that consume time that would otherwise be spent developing. Knowing what percentage of your work day will really be free to use for develop the software is critical to your estimation process. Your original estimate should be adjusted by this factor to get a 'real day' estimate.
   
3. Get a feel for your boss's estimate slop - how much does s/he adjust your estimate when adding it to the overall schedule? Some managers take your words as gospel. Others, more cautious, add a few days/weeks/months to your numbers to get a 'safe' schedule. Knowing this will allow you to pad your estimate a bit to account for a brave manager's lack of adjustment.
4. Learn how often developers are called on to support code in the field. Field support always trumps development work, because paying customers trump potential customers. This, of course, introduces a whole new set of issues with schedules, because at any moment your development time may be pre-empted by a support call that may take the rest of the day and into the night, as well as blossom into a string of meetings, conference calls, and possibly site visits. So the percentage of time you might spend dealing with the field will be a factor in your estimations.

Next: Tricks of the wily developer, or how to prevent mental breakdowns when they move your dates.

---

Technorati Tags --
Software
SoftwareDevelopment
Computers
Programming

Development in the Corporate World, Part 1

"The difference between theory and practice is smaller in theory than in practice" - Unknown

When I was in school, learning software development, there was only one accepted methodology in the industry - what is now called the Waterfall model, ironically proposed as a flawed approach. Despite that, even the iterative variation of the Waterfall model was influential in supporting the rigid separation of the phases of development - Requirements was to be finished before Design started; all Design was done before Development was started, etc. As the referenced Wikipedia article notes, many forces work against such a system being successful.

The first and foremost of these is that, except in government contracts, requirements gathering is never finished until the product goes obsolete. Few customers have a complete idea of what they want at the start of a project, especially one that is a new product for them. The initial idea they will have will typically be a computerized version of their existing workflow. As the storyboards are presented, they will usually OK them without much thought, spending most of the time debating niceties of field placement, drop shadows, and background colors. The Requirements Document is blessed by both sides, and Design begins. Once the Design Document is produced, reviews will start.

Document reviews can be a Hell all their own, but in most cases the customer does not have the expertise to critique a design in detail. Once they see the storyboard for workflow, however, they start to feel comfortable to assert themselves. They have some new ideas, or perhaps their workflow has changed in the meantime; sometimes government regulations have changed, or a new technology has emerged to catch the fancy of the VP. [I once lost a job when the Director in charge of my project was distracted by the new Apple Newton, and withdrew his attention from the politics of my project. But more about that another time.]

So new requirements roll in from the customer, after the Design is supposed to be done. But by now, the schedule clock is ticking, and so some development needs to be started, or the project will be late. So the team starts development, and the analysts respin the Requirements Document, and the designers respin the Design Document, and another review ensues. With luck, the new design does not seriously impact the work already done, but sometimes it will invalidate much of it, requiring a fresh start by the developers, and wasting more time.

And this brings us to the second big problem with Waterfall, as commonly seen in the wild: the Schedule never gets longer, even if one of the phases gets redone. I'll cover this in the next entry....

---

Technorati Tags --
Software
SoftwareDevelopment
Computers
Programming

Watts Humphrey's TSP/PSP

I can't say much about the TSP/PSP because I have not been through their training, and their official web site has minimal details about the methodology - odd, considering that they charge money for the training (heh).

What little I do know makes it appear to be a very heavy methodology, requiring significant investment by a whole organization. This makes it a difficult methodology to introduce into a company for just a small group, and almost impossible bring in from the bottom up, since the training requires managerial approval, and costs $1000 per head for the initial course (as of January 2006).

The one anecdote I have about this is that one developer of my acquaintance was on a team scheduled to take the PSP training. Management was told that they were expected to attend, offsite, and turn off their cellphones during the training, to show that they were committed to the new process, and to make sure they understood the training.

The management agreed to the offsite, but not to turn off their cellphones. Less than halfway through the first day, the upper mamangers were out of the classroom, taking calls. So much for commitment.

---

Technorati Tags --
Software
SoftwareDevelopment
Computers
Programming