The disaster recovery disaster and the file that never was…

This past week was the company’s annual Disaster Recovery exercise, where we simulate our mainframe system going into complete system failure and forcing us to restore the system from back up tapes at a secondary site. It is a very important exercise that any big tech companies do on regular basis. It would start on Monday, as the tapes would be driven to the secondary site and all the back ups would be loaded in the secondary mainframe, and by Tuesday afternoon we should have everything setup to process transactions again.

From what I had been told, things usually go according to plan and end before the end of the week. Filling optimistic (and we all know how that usually ends, don’t we?), I decided to sign up for the 3 to midnight shift on Tuesday with a couple of my colleagues. One one them did the exercise a couple years ago and the other was a fairly new employee who had been working just a couple more month than I. I thought I was in good hands.

But things wouldn’t be fun if there was no challenge, now would it?

  • Monday:

I came to the office still slightly annoyed that I couldn’t login in Sunday to discover that, as I mention in the post before last, things didn’t go well on Sunday. They were hopeful that would be able to fix the issue before Tuesday’s run of the program, but we didn’t share their enthusiasm. More importantly, I was reading the status report of the Disaster Recovery (DR) stating that they were falling behind schedule when they discovered some of the tapes had seemingly expired…

A foreboding sign of what was to come…

  • Tuesday:

Being the ever punctual person that I am, I woke up as usual at 6:30. Not the best things to do if your trying to stay up late, I might say. I took the opportunity to do a couple chore before heading to Com Site two, our backup system location. The name sounds like some kind military code name, but the actual building takes the cake. It looks like a bunker. It’s on a hill, there was a ten feet tall concrete wall surrounding the place, a field large enough for a helicopter or two to land and the building has this weird airlock door you have to use. They even have one-way mirror/glass in the entrance before the airlock for security to inspect you or whatever.

Anyways, as I entered the main room, Everyone was looking busy, but not the good kind of busy. Files were missing, recovery jobs were going down and we were at least three hours behind schedule.

Now it is important to note something: the exercise assumes that the incident that would have brought the main system of line would have happened sometime on Monday morning, meaning that all the system back ups would be from Sunday. On Sunday, my ID was expired. Which meant that when I tried to log into the DR mainframe, It wouldn’t let me…

After getting my access restored, I did do much to be honest. Technically, we were supposed to be here to make sure that the batch cycles were running properly and creating Incident Reports for any jobs that went down. The obvious problem was that batch cycles were not running yet. As we were trying desperately to recover files, a pattern became to emerge. The directory tool which kept track of where each file was archived on which tape was out of whack and some of the tapes were incorrectly labeled as scratch and overwritten…

While that was going on, I was also CC’d on an email thread about on RTD-37, the procedure that I worked on for the install that has been giving us such hard time. As it turns out, the other team (the one that uploads the input file for our programs) had thought they fixed the problem from Sunday, but it turned out that they didn’t. They thought the problem was with the internal layout of the file while the actual problem was the record length. The actual data was formatted correctly but was uploaded to the wrong length, causing the record to “wrap around” as it were. Each record should take exactly one line of the file, but because the line were too short, each record spilled over unto the next line, causing everything to misalign into a jumbled mess.

As we approached midnight, things were looking gloomy…

  • Wednesday:

After a short night of sleep (maybe like 5 and a half hour), it was time to go back to the office. Not surprisingly, the situation over the DR exercise had not improved over night. They attempted to start one of the processing cycle only to discover that file were still missing. The client side of the exercise was not going well either, with many of them not being able to access our software.

Things were quiet on the RTD front however, and that got the project manager somewhat worried. As it turns out, the other team seem to be fairly tightlipped when asked how much progress they made. They still seemed optimistic they could fix the problem before the next day. Just like the three previous day, we ran an empty file.

  • Thursday:

Business as usual: Nothing was working as it should for the disaster exercise and we still were not producing the reports from our procedure. By midday thing were starting to move on the DR front. Some decided that it was not worth it to try and recover stuff from the tapes and decided instead to pull directly from the production cycles (something you could not do in a real disaster because there would be no production cycle running…). Ergo the exercise was a complete a resounding failure. Or not, depending on who you asked. According to some of the upper management, the problems with the scratch tapes would not had happened because… reasons… and voila! Everything is fine. Even my manager thought that was some serious bullsh*t excuses.

By the end of the day, we also decided on a “fix” for the file upload problems. The other team would process and format the file and, instead of uploading it automatically, they would send it to us and have us manually upload it into the mainframe until they figured out what is wrong with their system…

SSSIIIIIIIGGGGGGHHHHHH………

Seriously though, this program/procedure has been running for the better part of a decade, It can’t be that hard to fix!

When I was in college, I remembered reading this post about how being a programmer can be a thing of nightmare, and the sad truth is that it is. Working on a project for a comp-sci class in college will never prepare you for the reality of the real world. You don’t write programs from scratch, you build on someone else’s work. I can look at each line of a program and understand what it does, but it would be a hopeless task trying to figure out the complex logic, the bigger picture, that drives said program. I am starting to see why so many small tech start up are popping all over the place. You don’t have to deal with multiple compartmentalized programming team, you can start from scratch rather than maintain decade old system technology and software, and probably most importantly it would be much more satisfying work.

Aahhhh…. Things should go better next week, but I not holding my breath to much.

Advertisements

April’s Loot Crate!! (or the “loads o’ photo update”)

WP_20150421_14_32_11_Pro

Yes! It finally arrived! It actually came in the mail on Tuesday but I didn’t really got to look into it because of how things were going at work (none of it is my fault, by the way. More on that in Tomorrow’s post). This month, the theme was “fantasy” so I figured Game of Thrones, D&D stuff would be included. Let’s a look at what we got:

WP_20150421_14_32_26_Pro
OPEN SESAME!

WP_20150421_14_32_41_Pro

A Dungeon and Dragon theme bowtie, pretty cool. A Loot Crate exclusive product.

WP_20150421_14_32_50_Pro

A princess bride theme pack of card, meh. Maybe I can gift to some else.

WP_20150421_14_33_15_Pro

A game’s of throne USB drive? Nice! This one was a Loot Crate exclusive as well.

WP_20150421_14_38_12_Pro

Unfortunately, it only holds 4 Gigs so not the most useful but handy none the less. The one other issue is that it’s large and it will cover multiple USB slots.

WP_20150421_14_33_34_Pro

A badge/pin thingy. Apparently there is one in every box. Meh.

WP_20150421_14_33_55_Pro

A Dungeon and Dragon T-shirt!?!? Heck yeah!

WP_20150421_14_34_13_Pro

A Harry Potter theme bag tag. Pretty cool but not sure how useful it is. It feels like it would break easily.

WP_20150421_14_34_26_Pro

More Game of throne stuff, this time fridge magnets. Nifty.

WP_20150421_14_34_50_Pro

The monthly Loot crate magazine, which contain a short descriptive of all the content of the box among other things.

WP_20150421_14_34_59_Pro

What’s this now?

WP_20150421_14_36_30_Pro

It’s an inflatable crown… Kinda cool, I guess. It’s way too small for my head though. Also an exclusive of the Loot crate.

WP_20150421_14_37_01_Pro

Over all, pretty good! Most of it has some value and not just useless collectibles (except the crown I suppose). They could have had a custom set of role playing dice instead of the playing cards or the bag tag in my opion. Hopefully the next one will have some cooler stuff.

Going to work on Sunday…

Last week was a big week, we were going to release the new version of our software. Most of of the code changes happened on Friday but one of the projects I worked on had to be installed on Sunday instead. something to do with it running on the previous day’s output of another program and we don’t process anything on Saturdays. Being the guy who programmed the changes, I am supposed to verify that the program runs and all that. Ergo: I had to come to the office on Sunday afternoon.

Oh man, I am not sure where I parked my car! There are just so many of them.... not...
Oh man, I am not sure where I parked my car! There are just so many of them…. not…

Now, i would normally be just fine with except of thing: I got a warning message Friday morning saying that my Mainframe access would expire on that same Sunday . I ask my manager to have my access renewed but she was sure I would get on time. In my logic, if it expire Sunday, It would still work Sunday and stop on Monday, So I was optimistic about thing going well.

It didn’t…

I got to the office, got into the building no problem, turned on the computer, tried to log onto the mainframe and… Denied! GUHHH… I called my project analyst to let her know that I was not able to get on the mainframe and I also told the person on call that I would not be able to help. I ended leaving the office shortly after that. But the question remained: did my program run or did it crash miserably?

It crashed.

Remember how I wrote a couple post about the grueling experience of trying to get a test file from that other team? And how they failed several times? Well, I guess you could say that lightning struck for the third time in a row. As it turns out, they had trouble implementing their side of the project, did not tell anyone and decided to send the input file in the old format. That made our program crash we had to manually run the old version of the the procedure. Even by Monday evening, they had not solve the problem and we ran empty file instead.

Trials and tribulation of working in a corporate environment, where you can’t assume anyone has their stuff in order…

on a lighter note, I said I would write about that bacon flavored chocolate bar I found so here is a short review:WP_20150416_17_11_40_Pro

Predictably, there was no trace of bacon in the taste, even though the back of the packaging advertised that the bar had bacon bits. The sea salt has a pretty powerful taste so it is possible that it undermined the bacon, but I am somewhat doubtful of that. Surprisingly however, it smelled like bacon. If you closed your eyes and just smelled it, you could swear there was a plate of bacon in front of you. Overall, it’s a decent sea salt milk chocolate bar that smelled like bacon. It’s worth a try but would not recommend.

Fitbit Flex: a follow up review…

Alright, so it has been a couple day and I think It is time to do a review of sorts of the Fitbit Flex. So here we go:

WP_20150413_17_09_50_Pro
Dun dun DUUUUNNNN….

After the nightmare inducing rage of trying to synchronize my Fitbit at least one of my many devices, It eventually decided to sync to my phone. Neither my mac nor my pc would work for some reason. Also the program for the computer doesn’t actually show you anything, you need to go on the website after the sync to see how you are doing and stuff. So here is tip number 1: don’t bother trying synchronizing via computer, the app works better and has many useful features which we will talk about in bit.

The overall quality of the product is a bit of a mixed bag. The rubber use for the bracelet doesn’t feel very nice and there is noticeable flash left over from the molding process. More interestingly, the bracelet that came with the Fitbit has a surprising but mostly negligible defect: there is a small groove that follows the left edge of the bracelet and a lip protruding on the right side of the bracelet, suggesting that the two halves of the mold used did not align perfectly… It is barely noticeable and nit picky but noteworthy.

The Fitbit flex comes with two bracelets, one large and one small. The two give enough range for it to fit on any one’s wrist. For some reason, when I wear mine, it feels almost too tight on my wrist despite having some wiggle room. If I fasten it one hole looser, than it’s too loose. My wrist must have an unconventional size…

The actual tracker is removable and has to be pulled out of the bracelet for charging. That is a bit annoying as the tracker fits pretty snugly in the bracelet and can be hard to grab. The battery life is something like 6 to 7 day so it is not that much of an issue.

The tracker half removed
The tracker half removed.

There are two modes: normal/walking mode for the day and a sleep mode for tracking sleep patterns. How do change modes? By wildly taping the device for 1 to 2 second until it vibrates and display a light pattern. Brilliant. Also, there are no instruction contain with the box, and the only way I knew about was by stumbling through the FAQ when my Fitbit was not cooperating. Thankfully, you can retroactively log your sleep through the app in case you forgot or could not be bother to turn the sleep mode on.

It seems pretty accurate most of the times. You could shake violently and not see the step count go up for example. It does seem to have trouble with more subtle movement tough. I have seen the step count increase by 150 after I had typed an email while at the office and I suspect that some steps were “counted” while I was driving to and from the office.

The phone app has some pretty cool functionality that makes the fit bit worth it. It allows you to see your current step count and, if you slept with the Fitbit, check your sleep pattern. It also feature a burnt calorie tracker and food intake tracker. That way you can track how much calories you spent and how many you ate. This is a nice feature if you want to follow a specific diet for example. The application can also use the GPS tracker inside your phone to give stats while you work out. It will trace on a map where have been, tell you how long you have been working out and what your pace is. It is a really cool tool for runners and you don’t actually need a Fitbit to use the app. So here is tip number two: the app is worth it by itself, even without the tracker.

I went for a run around the lake/pond of the Apartment complex. Pretty cool!
I went for a run around the lake/pond of the apartment complex. Pretty cool! The two breaks in the path were when I took a short break.

In conclusion: The bracelet it self feels like something out a mildly successful Kickstarter but the app makes up for it in its functionality. Would I recommend this product to a friend? Not really. The Fitbit Flex feels like it was design to cater to the upper class New York socialite who need to feel good about themselves and to impress their fellow socialite. If you are looking for a good activity tracker, you are probably better off with one of the more expensive one, which have more features and are probably more accurate. It is functional enough to give the impression of a high tech gadget but really is more like a overpriced pedometer. It works for what I need it to do but it is a bit of let down.

Fitbit Flex: first impressions

Having a office job where you seat all day in front of a computer is hardly conducive to an active lifestyle. Since i have such a job, I have been trying to stay active. I go to the gym every other day and try to stay outside the apartment on the weekend. I figured I could use one of those health tracker thingies to help me keep track on how I am doing. I did not really need anything fancy so I figured I would just go and buy the Fitbit Flex. It cheap and keep tracks of a few things for you. So I went and bought one of Amazon. Here is my experience so far:

WHY WON’T YOU WORK!?!?!?! FOR THE LOVE GOD, SYNC YOU SON OF A *****!!!!!!

This thing is terrible!! How could they have messed it up so bad? How is this thing even sold?

I have two computers, a Mac and a PC, and a smartphone, all of which are supposed to be compatible with the Fitbit. After 45 minutes, none of them would connect to the damn thing. The Fitbit was as close to the wireless USB receiver as the laws of physics would allow and still nothing! Eventually, after restarting the Fitbit half a dozen times (which, annoyingly, is done through the charging cable thing), I got to sync with my PC… It took me an entire hour to set the stupid thing up!

Things were starting to look up. The phone seems to connect to the Fitbit just fine but doesn’t always sync. The mac is the biggest offender and never worked in the slightest.

The Fitbit website is pretty bad as well. It went offline twice while I was trying to figure out my synchronizing issues. Oh and now my pc won’t find and sync my Fitbit anymore….

I cannot believe how terrible this whole thing is. I did not necessarily expect this thing to last forever but at least to work straight out the box! Short of having just exploded or just not been functional at all, this is the worst product I have seen in a while. I have seen Chinese knock-offs work better than this! Chinese knock-offs!

I am going to persevere and see if things work themselves out over time, otherwise I am asking for my money back because it is not worth $99 for such an unreliable piece of junk….

The food update…

WP_20150411_14_51_10_ProOH MY GOD!!! A Starbucks Coffee!!! We are saved! I found civilization!</sarcasm> I am actually happy to know where to get my Frappuccino latte fix or whatever. And, it is walking distance from the office, which is always nice. Also, note the “drive Thru” sign. That’s definitely something that you would not see in down town New York. Pretty much everything around here can be done in a drive thru. Drive thru banking, drive thru fast food, drive thru coffee stores. No wonder the pedestrian infrastructure is so bad around here.

On a more interesting note, I discovered something cool this week. There used to be a gas station right across the street which literally closed the week I moved into my apartment. There’s another one down the road so whatever but they started building a new thing there called “Runza”. It is a local chain of restaurant and their speciality is called, as you could guess, Runza sandwich. According to Wikipedia, Runza (or Bierock) is an eastern european recipe that originated from Russia, and it was brought along by germanic settlers. That’s pretty cool, right? Traditional Russian sandwich in Nebraska. Perfectly logical.

Nebraskan fast food!
Nebraskan fast food!

Of course, you still have to do some frogger style action to cross the road. If only they had proper cross walk and stuff….

On the subject of food, the new Coca Cola found its way here. It’s green and called Coca cola life. instead of being sweetened by sugar or artificial sweeteners, it uses Stevia, which is essentially and plant extract. The whole “gimmick” is that it is less caloric than the standard Coca Cola and has a natural sweetener as oppose to aspartame in Coke Zero and Diet Coke.WP_20150411_16_37_57_Pro

Unfortunately, the Stevia does add a bit a flavor to the original syrup and it really doesn’t taste like the “real” thing. It taste more like a store brand coke actually. It’s probably more a fad thing, organic/natural products are trendy these days but I doubt this will stay around for too long. Also, it is not sold at the whole foods, so obviously it is still pretty much full of mysterious chemicals…

oh and brownies… Brownies are tasty…

WP_20150411_19_32_09_Pro

WP_20150411_21_20_12_Pro
Brownies for weeks!!!!

P.S.: Oh, talking about chocolate, I found some Bacon flavored chocolate bars while i was doing grocery shopping at Wholes foods earlier today. I haven’t tried it yet but from my experience, Bacon flavored things that are not bacon are pretty terrible. Most notably, I tried Bacon flavored vodka while in college (the only time I ever had any vodka by the way) and it tasted like mouthwash, it was terrible! Hopefully this bacon chocolate is better. I’ll try and remember to write a post about it sometime this week…

The yet to be named update…

Woo! Warm temperatures! Tax refunds! Good times…

Yup, it’s finally April, and the weather is still as erratic as ever. one day it’s almost 90 degrees, today was only in the mid 50s. Oh tornado season is starting, and Wednesday night we got our first thunderstorm. Exciting. There is one thing that I found surprising was how “quiet” it was, . There was a lot of thunder and lightning but it sounded distant, like a low, consistent rumble rather then the sporadic loud crash of thunder we get in the city. Probably something to do with the number of Lightning rods on top of building in cities like New York versus the flat plains of Nebraska.

In other news, I went and bought myself a subscription to LootCrate. It’s this cool company who send you a themed mystery box full of stuff random every month, though it seems you expect to find a T-shirt and/or a comic book in every box. I remember first hearing about it when it first started a couple years and not thinking a lot about it. As I recall the first few month there were no theme to each box and the boxes ended containing a random assortment of gimmicky stuff like you would find on ThinkGeek. They seem to have gone up in quality since then, giving away LootCrate exclusive stuff, and they even give away a special “mega crate” worth almost $2000 of stuff to a lucky subscriber every month. The theme for March was “covert” and featured some cool spy related stuff. This month, the theme is “fantasy” and I am pretty excited to see what I’ll get. I’ll probably make an unboxing post with pictures when I the thing.

Fast and furious 7 just came out. I can’t believe they made that many! The first one came out in 2001! that’s one movie every two years! Holy crap! Surprisingly, I have only seen the odd numbered ones, the first, third, and fifth. The first one was pretty good, the plot made sense and the action was within the realm of reality. The third one was Tokyo drift, which people seem to like the least. I thought it was great. I was living in japan at the time, so my opinion might have been biased. Finally the fifth was way over the top! Ripping a vault out of building and dragging it around? Rubbish! Fun but thing really started to get over the top. This one seem in the same line of crazy stunts like the fifth one. dropping cars out of Airplanes! driving cars out of buildings to crash into another! The law of physics might have something to say about that. I’ll probably go see it because I don’t really have anything better to do, but I don’t expect anything exceptional.