Logging advice

Ok, an actual post about doing things right, instead of me venting my spleen.

So, let's talk about logging messages.

When writing logging messages for technical use - monitoring, statistics, etc, pick a separating character that is not part of the various fields you have interest in.

For example, most log messages will include a timestamp, so use of ':' is out.  Likewise, if the timestamps include the date, the '/' or '-' is right out.

A good choice for the field divider is the '|', and then the use of spaces as separation of the subfields.

It's the goddamn 21st century....

Why is someone declaring a message string with only 256 characters on a server-side program?  I mean, we are running on systems with hundreds of gigs of RAM, and it's for a freaking LOG MESSAGE, local variable, on the fucking stack.

But 256 characters it was, and  we decided we wanted to log a command line for a Java program, so between the log4j path argument and the Java binary path, it was 270 chars long, and it stomped on the stack, leading to a premature exit.

The Mismeasurement of Code

Christ on a crutch.

My PM is suddenly gung-ho on getting measurements for "KPIs", and beaing able to "get ahead of things" when event volume changes.

Never mind that speaking up gets everyone and his brother pointing fingers at you for their problems, even when it's someone else in the middle who has screwed up.  The corporate mindset is such that whoever says they have a problem is suddenly the cause of the problem.

Never mind that we have a cient to who *not* in production, but acts like any small time out of service is like the Fires of Hell are burning their ass and it's all your fault.

Been sayin that for years

Just read an article in the PyCoders weekly list about refactoring functions to have multiple return points.

The article is well-written, but I was floored by the need for it.  I've been an advocate of multiple return points for 2 decades.

The only reason to hold to a single return point in a function is to guarantee release of resources, and every OO language has had a means to do that with a local variable and destructor for over 25 years.

Multiple return points make the logic clear, handle special cases better, and allow better refactoring later for all purposes.

Do as I say, not as I do

I am beyond tired of being the one who has to follow all process and procedures, while being subject to other systems whose owners do not.

We had 2 major system failures this week, that essentially stopped our application from running or getting its data out.  In both cases, we were contacted by the downstream application teams, demanding to knwo th reason for the outage, and when it would be over.  Curiously, neither of the mission-critical systems that failed contacted us in any way, despite our system being well-documented as being on those machines and using that system.

Yet somehow, there seems to be no fallout to those systems for this utter lack of communication to the end users about this.

I may have complained previously about a manager who once told me that once I was told about a problem I had to "own it".  I responded with the obvious question - why didn't the person who told be about the problem have to own it?  I did not receive a good answer, except that the manager wanted to seem in control.

Readiness

So one team that wants data we will provide is pressing for us to turn the spigot on.  And this afternoon, they inform us that they can't decrypt the encrypted data, and the error code is that they are not accessing the encryption system correctly.
So we had to get the data to them right away, so that they could find out they weren't ready for the data.

Vanishing staffers

So The Job, like many megacorporations, is trying to cut staff by having contractors do the work.

My team consists of 4 permanent employees, and  5 contractors, and half the temporary folks will be gone in a year.

So I am stuck with trying to figure out how to transfer knowledge, and juggle workload, and transistion support.

Sheesh

Ignoring the obvious

Y'know, sometimes The Job just fucking wears on you.

Ok, so I have one cow-orker who is not the sharped knife in the drawer.  We are running a system that needs certain portions of the data encrypted, so he was tasked with writing an encrypting filter.  He does this, and is testing it on our data stream.

Performance results show that the time from our sender pushing the records into the system to his filter reading them increasing.  Our test reader is showing a time delta of nearly nothing, and it just reads.  And we have another process that is reading and storing records from the stream, and it backs up pretty steadily as the DB load increases

So the obvious conclusion is that his reader is backing up because of the processor load.

But no, that's not what he thinks.  He is adamant that there might be another reason.

Why the fuck am I having to explain to a nominally experienced programmer that if the passthru does not show delayy and the proessor-loaded filter shows delay, ,it's due to processor load?

Lying Liars

Ok, so The Job has me working for a manager, and he's hired a Project Manager contractor (some time back, I'm just setting the stage)

Recent corporate bullshit has led to a requirement that everyone be working on funded projects.

So my manager has decided what percentage of "work' everyone is doing on his funded projects, for the next few months.  To be fair, he's keeping everyone on paid-for work, so it's a good thought.  But b/c of this, the PM is asking me what tasks can be given to the 2 devs who are not really on our project to cover the percentage of time they are supposed to be working on our funded project.

So to assuage his conscience and not lie, he wants be to invent work for them, even if they really have no business working on our project (b/c they have no context, not for any skills issue)

I hate this bullshit

Really Stupid Corporate Policies

Ok, The Job has really pissed me off this past couple of weeks.

They have a number of policies intended to protect company assets that are just plain useless.

#1 - only staff with "developer" in their title are allowed admin access to code repositories, despite a number of projects where the entire development staff has been on a revolving door, and only the manager is a continuous presence.  But we can't have one of the non-developers controlling access, can we?

#2 - despite a growing number of contract staff, several tasks can only be done be a full-time employee, which means that *I* get stuck with setting up those tasks for our projects and a couple of other groups'

#3 - Several tasks need approval by some specific level of management, and by 'specific' I mean exactly a certain level in the food chain.  Not "at least a 2nd-level manager", but "only 2nd-level manager, not 3rd, not 4th, etc"

SSDD, Agile Misunderstandings

So the latest job is moving to Agile.  They've got a nifty all-in-one Agile tool set up, with the usual problems such things involve.

But the real fun is that people are writing user stories who have no idea what the technical people on the team are doing, and without asking them at all.

This is driven by the mandate from On High that everyone will be doing Agile ASAP.

So we stumble along, as the uninformed are filling our backlog with items that cannot be broken down.

And the best - the tool training uses a team's per-iteration completion of story points as one of the metrics.  IOW, they expect every team to maintain a high rate of finishing business value. 

That won't have any effect on the estimation process, nosireebob.

Like Neo in "The Matrix"

...I dodged a bullet with panache yesterday.

A couple of weeks ago, I got an email from Blizzard, asking if I was interested in a position there.  I said I was at least interested in listening, so a phone screen was set up for yesterday.  And on Friday, I read in the trade press that Blizzard is rumored to be planning a significant layoff by Feb 12.  So I take the call on the 11th, and the recruiter was totally silent about the layoff.  But since they are not at all remote-work-friendly, the interview was pretty short.

It's like working for Jello

So, about 2 years ago, I was asked to spin up a web interface for some data our project produces, to interface with another project.  They had a lot of hidden requirements, and turned out to not have a viable business case, so they went away. 
In the interim, I had to maintain this interface, modify it every time our intermediate communications paths changed, handle the massive increase in data volume, etc.
When another tool dropped export support, my boss looked for a replacement that was found in another project he manages, and the lead of that project suddenly was really keen on developing a replacement for the web interface.  And my boss was falling over himself to say we would switch to that interface. 
Fast forward to yesterday when we were meeting to finalize some details, and it suddenly becomes a bake-off. We have to run some tests and list pros & cons.
Then today, it becomes back to using my interface, until the other guy mentions some things, and the pendulum swings back that way.

Basic Technical Competence

Most companies have a mix of skillsets needed in the employees, and so you end up with the technical folks doing technical work, and folks managing them, who are typically less technical.  But there is a basic technical competence level that needs to be maintained, or the managers end up wasting everyone's time with bad ideas.

Case in point:
Our application generates a ton of data.  So much that the clients need to be in the same datacenter to get it at the rate we produce it.  The remote datacenters do not have enough incoming bandwidth to receive the data in a timely fashion.  One day we were discussing this problem, which limits the locations of our clients, and one of our project managers suggested "Why don't we just use the cloud to get it to the other datacenters fast?"

I mean, ya gotta have the basics in your head to play the game right, y'know?

Doing the absolute minimum is A BAD FUCKING IDEA

This blog has become my vent for all the crap from The Job.

Ok, so we have a system that sends data into a central routing system, built by another group.  In order to be able to prove that any delays in the system are not due to our system being behind, we have a subscriber program that reads the data and checks transit time.  Because that project may get moved to other servers, we wrote a minimal subscriber to output just that info.

Cow-orker is tasked with writing the script to analyze that program's log files to track the time.

He write a program that expects every message output to be the one with the data he needs.  Absolutely no filtering by content at all.  So if we alter that message, or add other messages, that script will fail. 



All it would have taken is to filter for the 2 words before the data value to guarantee that only relevant messages would pass into the script.  But No!

sheesh.

I really wonder about people sometimes.

Really Stupid Training

Ok, TPTB at work decided to drop new training right away, so I was filling in some time (that I was not interested in doing real work during) with taking it.

It's the repeat of the workplace safety dreck, and diversity training.

So I got to watch a crappy slideshow of how to lift things, walk up stairs safely, and how to arrange by desk.

Then the diversity training.....

I'm all for diversity, and it's good that The Job at least follows the law in its efforts to hire/retain staff.  But it serves nobody to have 45 minutes of videos where a person with a specific disability natters on about how it does and does not affect them on the job, and what others should do to when they do/do no need help.  It really doesn't.  Particularly when it is someone with a visible condition, where only the most idiotic of people still publicly offends with comments or actions.

Still not getting through to that guy

This morning, returning to the salt mines, I noticed that one of our monitoring tools was showing an error condition that I knew was not strictly correct.  When cow-orker arrives, we discuss, and determine that it's an artifact of the significantly different rates of monitoring and reporting of that metric.

So cow-orker and I decide it would be good to make it configurable, so that we can have it only report if it's consistently bad.

Cow-orker makes code change, tests in isolation, deploys to the one machine where the problem was manifesting.

Cut to an hour later, when second cow-orker reports that he is suddenly seeing malformed messages arriving at his client.  We look, and the time coincides with cow-orker One deploying his changes.  But he's off at an early lunch, so we put it aside.

After lunch, I pass One in the hall and mention it.  He says that Two had mentioned it to him, but that they were seeing the errors from another machine, which had not been altered, so it could not possibly be his code.

So when I return to the pod, I ask Two, and he shows me the single, solitary error from another machine, and the ongoing several in a minute stream from the suspect machine.  I show this to One, and he then looks and realizes he did not pull the repo before making his changes, so his binary reverted a week's worth of fixes.  Off he goes to pull, re-implement, and redeploy.

Sheesh, you'd think that this 4th time he'd start to learn that at least he should at least just so "Ok, I'll check it out" before trying to dodge blame.