Monday, May 21, 2018

More Practical advice - Producer-Consumer threads

Two in one day!  What's the world coming to?

Ok, a common paradigm is a producer-consumer threaded application, using a queue to pass the message to be processed.  I recently coded one, and we were discussing how to manage the number of threads in the future, when we would want the application to dynamically allocate threads to match load.

Now, adding threads to increase capacity is simple - you just need to create the thread with the appropriate parameters (input queue, output sink, etc), save off the info so you can reap the thread at termination, and Bob's your uncle.

But removing threads is more complicated, because you don't want to kill a thread in the middle of processing a message.  So you need a means to tell a thread to die when it's done with the current message.

What I had come up with for shutting down all the threads at program end was pretty simple - the Producer emits a message with a specific format that when read by a thread, tells the thread to exit its  main loop and return.  What was moderately cool was that I had the thread re-emit the message into the queue, so that the next thread to read it would also stop, and would re-emit the message again, until all threads stopped.

The today I realized that if I added a simple counter onto the message format, I could use that as a way to reap a specific number of threads without any new mechanism.  The Consumer thread sees that flag message, exits its main loop, and then checks the counter value for 0.  If it's not 0, it decrements it by 1, and re-emits the message with the new counter value.  So start the message with a counter of 3, and 3 threads see it, stop, and re-emit it once each.  The last one sees the counter is 0, and does not re-emit it.

This can be used to kill all the threads but setting the counter to -1, which will never be 0, so all the threads will see, process, and re-emit the flag message (unless you have so many thread that the value wraps, but that's an entirely different problem)



Practical advice re CSV files

Morning, everyone!

I'm sure everyone has had a situation where they were generating data for another system, and needed a text format, and decided that comma-separated values (CSV) would work nicely.  Well, here's a little advice:  if you have any reason to expect a) human-driven input for fields, or b) reformatting of fields by some programatic means, then chose to use a pipe '|' as the separator instead.

The reason is that there is little use for that character in normal human text usage, so it will not get used within a field, the way commas often are in things like addresses, or names.  And as I found out recently, some encryption software is perfectly happy producing cyphertext with commas embedded

Thursday, July 20, 2017

Losr in Corporate Hell

I just got out of a meeting where I was gobsmacked at the insistence by some architects that no development could be done unless the specific IP addresses of the machines in the system were set in a design document.

Think about that.

No design or development could be done unless the specific machines were known, despite the interchangability of any machine.

IP addresses are standardized, so the only thing needed to be known is the number of machines.

As I said, gobsmacked.

Saturday, May 13, 2017

Muzzling diversity

I've got a new corporate master now, and one of the big "perks" is working in an open office plan.

There are plenty of blog posts, articles, studies, etc that show such offices are crap for software development.

However, I realized something this week.  They also serve to minimize dissent in the company.

If you are surrounded by people who can watch your every move, you are likely to self-censor your speech, because you have to get along with the cow-orkers every day, so setting yourself up as someone who disagrees with your colleagues is career-limiting behavior.

If you had an office or a halfway decent cubicle (6-ft partitions, minimum) you could have soft conversations about topics that your neighbors did not agree with.

Open offices lead to groupthink.  Just say no

Monday, November 21, 2016

Yet Another Bad Catchphrase

"...It's in our DNA"

Gaaahh.  I *HATE* when corporations use that phrase.

It's not in your DNA, even metaphorically.  If it were, you would not have to keep apologizing for security breaches, or for crappy customer service, etc., etc., etc.

It's just some uneducated marketing hack who wants it to sound like they can't fail.

Nearly as bad is "laser-focused".  'Cause the whole point of a laser is that it is not focused, it's collimated.  If you focused it, it would no longer be a laser, and no longer any use.


Sunday, November 20, 2016

Juggling a few too many things

At The New Job, the product I'm supporting is a good example of how a small skilled team can do amazing things.  It's going strong after 3 years when it was supposed to be a stop-gap lasting only 6 months.

But in the course of my learning the ropes, I've run into a situation that allowed me to finally put into words something that I've felt for a long time about systems design - simplicity of all things is a virtue.

The setup - the product is a collection of scripts and binaries that process incoming files.  There are bash scripts, Python scripts, Ruby scripts, C binaries, cron jobs, and even a Java binary.  The upshot of this is that it requires a fairly experienced developer to be able to work on the system as a whole.  Now, this can be an issue in finding suitable developers, but it's not a high hurdle.  But where it does cause a problem is that the corporate sysadmin environment is very controlled, and it is not a simple thing to get tools installed, and the more tools you use, the worse it gets.

So, I'm having to navigate the labyrinth of corporate IT to get the desired versions of Python and Ruby approved and installed on our production systems, as well as the boilerplate necessities of user accounts.

Moral of the story - consider the overall ease of use when choosing languages to build a system in

Saturday, July 30, 2016

Movin' On Up

Well, Dear Readers, you can rest easy - I am gainfully employed (as of the middle of May, actually)

I moved to a small group in a large corporation, with all the chaos that implies.  I"m also working a project that an old friend started, so I have a bit of a narrow ledge to walk, not wanting to disparage work that is provably profitable.

I will not, however, spare the company.  They can take it.

The big thing in my immediate headlights is the weird corporate labyrinth of permissions, requests, and partitioning of tasks.  It seems to take 6 different groups to get anything done.