Change control chaos…

Nothing like a good work related story to inspire a good blog post…

So, when dealing with any sort of programming project that is either distributed or client facing, you need some form of release version control. These tools allow multiple person to work on a single program or code, pull defective code from circulation, prevent unauthorized changes to be put in. Most people are probably familiar with Github, and we have similar tools to control what version of the code is used in our processing or available to our clients. One of the tools we use is a scheduling tool to time what code needs to do be compiled and when. This tool has apparently been discontinued and out of support for a while now and management has decided to develop an in-house solution. Why bother pay IBM for an old and outdated tool when we could make our tool?

They started working on this new tool sometime last year and in October, they showed us a preliminary version of the web based tool and it had promise. Some of the page layout were a bit rough and misnamed but we were assured that most things were subject to change as they were getting feedback from the different programming teams. There was no set release date for the tool, and all seemed well.

A couple months passes and it is now sometime in February. As part of the development cycle, the dev team started weekly training session to test and get people familiarize with the tool. I went to one of session and found that little to no changes had occurred since the early demo. That was slightly worrying but it is hard to judge how much work was done under the surface. I played around with it and reported a couple bug that I have found. Minor wrinkles but that is to be expected for a project on the scale of that one.

Another month goes by and still, it appears that little progress has been made on the new tool. More worryingly, a release date was set. Not only that but the implementation plan stated that there would no parallel deploy period when both tool would be active. The old tool would be used until a specific date, at which point it would be turned off and and the new one activated. This lack of overlap in tools means that if the new tool is not 100% functional on day one, we would be left with nothing to control our code release. As far as I am aware there is not plan to move scheduled  code changes from one tool to the next, meaning that we cannot schedule anything on the old tool for dates after the release of the new tool, and the new tool cannot be used until after its release to schedule code move! There is a planned 3 day window where no code changes will be possible. This is, for a lack of a better word, rather “unwise”. No one seem particularly enthused by this. Even my manager expressed his opinion that this new tool is the pet project of someone from upper management and that is being pushed to look good on paper.

As weeks went on, functionality were slowly being added to the new tool but were not reliable. A functionality of the current tool is to create a list of necessary code recompiles when a copybook is changed. This process is usually very quick and never takes more than a couple minutes, but with the new tool, that process took several hours. Sometime, that process does not even kick off properly and may requires several tries. The save functionality, an evidently important function, also did not work properly. This project started to look like a train wreck.

An official “pre-released” period started a couple weeks ago, and some of the teams were instructed to use the new tool. Word around the water cooler is that they are not using the tool. Several major bugs have still not being fixed either Continuing that train wreck from earlier, the train is now hurling out of control into a tight corner.

And today, a rather important announcement has been made. Due to compounding issues and defect in the new tool, the current release date has become “unreachable”, but more importantly: the lead developer has resigned! We are now completely off the rails! Everything is tumbling out of control! I can only imagine that this how the development team feels like right now… They are supposedly trying to find a new target release date but I doubt that it will be this year. I have a suspicion that the lead developper was facing a lot of pressure to get the project rolling but perceived the timeline unfeasible and bail out before it was too late.

This project seemed doom from the start and it appears to have run into the ditch that everyone saw coming. What a surprise…


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s