robsyvertsen.org

There’s an old folk warning that if you throw a frog in boiling water he will quickly jump out. But if you put a frog in a pan of cold water and raise the temperature ever so slowly, the gradual warming will make the frog doze happily . . . in fact, the frog will eventually cook to death, without ever waking up.

One of my tasks this week is to gather together a sample of all the test data we store and send it to a vendor for analysis. I’ve always known that we had multiple formats of test data, in fact I created one of them. Today I just realized how bad it really is.

Since we never took control of our test data and tried to find a single home for it all throughout the company, we had little pigeon holes of data. Some comma-separated value files on this tester, some csv files on that tester in a different format, an Access database on the tester in the factory down south, some paper records… This is getting ridiculous!

The closest we’ve come to trying to harmonize this data was about 6 or 7 years ago when we hired a contractor to convert our paper system to an electronic system. His answer was a bad, bad Access database. This system worked for a number of years. It had it’s share of problems when it came to multiple users accessing it on the network, but it did keep us focused on having one source for test data.

Since then we’ve added multiple automated systems, each with a slightly different log file format. Now that we’ve expanded, merged with other companies, and partnered with multiple manufacturers in China, the number of file formats has grown to the point where it’s unmanageable!

Trying to gather these samples reminded me of the folk warning about how to boil a frog… If someone had suggested to me that we have 10 or 15 different file formats for our test data, I would have jumped out of the pot and tried to come up with a single format to handle everything… Instead they crept in one by one, year after year and now the water is starting to boil.

software utilities