Reproducing a Race Condition

We have a job at work that runs every Wednesday night. All of a sudden, it aborted the last 2 weeks. This caused some critical data to be late. The main question was what changed? Initial analysis indicated nothing relevant changed. However things did change. An architect identified the most likely candidate. Some new job was trying to improve performance. This new job ran 10 to 11 minutes before the Wednesday job aborted each time.

I babysat the job this week. We ran it one day early to give us time in case we encountered errors. Of course the job ran fine when I ran it during the day. So I declared the next order of business was to replicate this problem in development. My old architect buddy thought it could be done. Maybe not. But I was going to give it the old college try.

I worked with a developer from the team that introduced the new performance job at night. We both ran our scripts at the same time. After a couple tries, I had still not replicated the problem. I told the dude to give me a copy of his code. Then I went to work. I spent 30 minutes reading up on the UNIX Korn shell. Then I wrote a Korn shell script to (1) generate new data, (2) run my job in the background, (3) wait a bit, and (4) run the new job.

Now I varied the amount of new data added each time from 1 record to 300 records. I also varied the amount of time waiting from 1 second to 5 minutes. Made sure to log everything. And since I launched this thing from an interactive shell, I prepended it with a "nohup" and ran my main script in the background too. I estimated best case this thing would finish in 12 hours. However it might run for as long as 24 hours. Luckily it is Friday. I imagine nobody is working in the development environment on a weekend.

Virtual Desktop Corrupted

I had a good sized task to complete at work. Told my boss it could be done in around two weeks. He thought that estimate was too small. I did not argue with him. We settled in at three weeks. I got some initial work done and published some docs and fixed one bug. Things were looking good.

I used this opportunity to refactor some test data gen programs. Was going to merge three separate programs into one. That would decrease future maintenance a lot. The code was written in Oracle PL/SQL with hard coded tables and columns. I figured it was time to turn this code into some dynamic SQL where I could plug in appropriate tables.

Spent a week and a half refactoring the code. Did all the work on a virtual machine because that is where my best tools were. Last Friday, the machine was acting weird. I could only access my Visual Studio program. Nothing else in Windows would come up. I rebooted the machine at the end of the day.

Monday morning came around. I could not log in. Tried to contact the customer help desk. Spent all afternoon on hold. Got up early Tuesday. Was able to speak to the help desk. The technician seemed confused and could not resolve my problem. Issue was escalated to second tier help desk.

Later on I found out a couple other developers lost access to their virtual machines. Apparently they pushed some "Data Loss Prevention" software to our machines. The software install must have been super buggy. It corrupted all our virtual machines. I lost all my work for the last two weeks.

Worst part is now I don't have a virtual machine to do my work on. I need to rely on my laptop and shell access to a Solaris UNIX server. Ouch. I told my boss I am not that productive in this environment. I started out using Windows Notepad. But I recalled that I had NotePad++ from a prior project installed. A bit better. But I work best in Visual Studio.

The only upside is that I love database programming. So I just got two more weeks of heavy work in my favorite environment. Schedule is not looking too good though.

Humble Bundle Book Deals

There is a book sale going on at Humble Bundle. You can get 15 Python books for $18. Now I already own some of those books. But I did find at least 5 that I would definitely buy. The value here seems so strong. I am a sucker for deals. However these books are in digital form only. I have come to hate eBooks.

I do have another option. If I buy books for college or for certification, my company will usually reimburse me. They have a high budget per employee each year for this. This year I have already got the company to buy me 6 books. I figure I might be reading these books past the end of the calendar year.

Still. I would like to also get these books. So I am going to wait until I am ready to take a mid-level Python certification exam. Then when I do, I will ask for those 5 interesting books to be purchased by the company. That should keep me busy well into 2022. The total cost for the print versions from Amazon is $135. That is even with Amazon's discount from the MSRP.

Perl Code Broken

We were getting close to turn on some software we developed last year. The last piece we forgot about was the piece that automatically ran the jobs on a weekly basis. I negotiated the timing of the jobs with other stakeholders. Then I figured out some cron job entries to make these jobs wake up at the right time. Last piece was to test the cron job runs in development.

At first, I scheduled the simple job to run. The thing bombed because it could not find an include file. That seemed odd. The file was in the same directory as the Perl script that was running. But that directory was not in the @INC path. Why not? I thought this had worked before. It actually works when I run the job manually.

Upon further inspection, I found that the profile that was run when the cron job executes had a bug. It was not setting the PERL5LIB directory correctly, which in turn would get added to the @INC array. That was too much details for somebody who is not a Perl programmer. But hey. I guess this was a small opportunity to learn something.

Harvard Learning

I have previously heard about Harvard offering some courses online for free. Well you can do that now with CS50. This course seems to teach a ton of info. It seems to last almost as long as a normal college semester. It claims to teach you C, PHP, and JavaScript programming. You will also learn SQL.

There are a total of 9 assignments in the class. There is also 1 final project. You only get to audit the class. However, if you plunk down $199, you can get a certificate of completion. I bet I could have my company pay for this course. Not sure if it will do me any good.

I do like the idea of being able to tell people that I "went to Harvard" for computer science. LOL.

Getting Access to the Code

Right now I am working on a tiger team to research why this new reporting system has different data than the old one. Most of the time I try to figure out what exactly is the complaint. The customers speak in a code of sorts. And lots of the tickets are written with the perspective of the new data loading system.

Now I did do a stint on the new data loading system for a few years. And I spent half my career on the old system. I think that is why they have me doing the research here. There is this one small database that I don't have much experience with. All the values in one certain field in it differed between the two systems. Hmm.

I found some documentation for the old system. Something looked screwy. There was an input field from a COBOL program that produced a values in dollars. However in the old system target database, they seemed to be storing dollars and cents. The cents portion was not zero. Felt like a defect to me.

To truly declare this a bug, I needed to look at the source code. Oops. The source code is only available on this Solaris UNIX system. Okay. I got access to a common account on there. Nope. You got to have a personal account to get access to the code. I tried to log in. Wrong password. I requested a new password.

Eventually after a second password reset, I got access to the UNIX. They got some Perl shell script to create a view to the code. Ran the script successfully. But where the heck is the source code. Turns out it gets stored on some shared storage off on the network. Whew. Tough job just getting to the darn source code.

Defect Ticketing System

At the end of last year, I left my prior project. Not sure if the cause was lack of money or retirement of the system I was working on. Could have been both. Luckily, and old manager offered to bring me back onto his project. They had interesting stuff to work on. It was mostly new development or new automated testing software.

 Eventually I had to work a trouble ticket from the customer. I thought I would just look the issue up in their ticketing system. Nope. I no longer had access to it. Maybe this was because I switched projects? Turns out I lost access because I had not used the system in a few months. Seems like an easy fix. I put in a request to regain access.

Some sys admin denied my request. He said I was not authorized to use the system. LOL wut? Turns out he looked up my name in his system, and did not find the proper authorization. I got help from my rep in the customer organization. They schooled the sys admin, telling him he was using some outdated system to look me up.

All right. I just want to get access to the ticketing system. So I asked the system admin to add me back in. The dude says that my request was denied, and I have no new requests. OK. Turns out I have teleported into a scene in The Office. Yeah the request got denied by you dude. You are using the wrong procedure to vet requests.

Oh well. Slackers are going to slack. Guess I need to put in another request. Hopefully I don't get the same goon servicing my request.