Simple Solution Wins

The customer found some problems in our reporting system. Whole parts of data were missing. Counts were not matching up. I was put on the task of resolving this. Had to dig a bit to found the root cause. There were a view that depended on another view that depended on some data being loaded. Many levels of indirection going on here.


To truly fix all the problems, it would require a lot of analysis. It might also slow down the system. Then I decided to look at this from another viewpoint. What was the true minimal cause of the problem? Once I found that, I attacked it with some solutions that were outside of the box.


I had to change just 2 lines of code. I love it when I get a solution like this, even if it is sort of a hack.

Finding a Way

I got word from my manager that a new problem would be resolved imminently. I had not even looked at the problem. So I dug in. I told my manager that it would not be done on time. The customer got angry. All of a sudden, I am approved to work all kinds of overtime.

I was tired. So at first I tried to run the 10 jobs that generate the data responsible for the problem. The first job took almost 3 hours before it died due to an out of memory error. I got the second job to run. The third job never finished and the database shut down for maintenance at night.

Ouch. A new deadline was created. Again I told my boss we would not be making the new deadline. Then I got offers for help from another team. I declined. They would only come, ask question, and literally waste time. Finally time to start working smart.

Instead of running these massive jobs, I decided to inspect the code. I figured I should just study each function until I was 90% sure it wasn't a problem. Then I would skip onto the next one. I got to the 9th of 10 function. This one had some trouble. This was it.

This was the sort of the problem where the research is the majority of the work. I coded up a solution, tested it, and passed it on. It is still going to be a tight schedule with the configuration management and test people. But I got it done.

Be Brave to Get Work Done

I was woken up this morning from a call from work. Not a good sign. Apparently the customer found a potential problem in our delivery. I got on a conference call to sort the thing out. Yes. There was a problem. Our whole team went into high gear to get this problem done.

First we brought in another senior developer. The problem and our proposed solution was discussed a second time around. Our team lead asked everyone if they understood what needed to be done. Everyone concurred. We split the work up 5 ways because there were 5 of us. Then we broke to get it done.

Immediately I get a call from someone on our team. He was asking what he needed to do. WTF? I told him if he was not clear, he should have spoken up at the meeting. No need to save face and waste my time. I could not do his work and mine, because I got lions portion of the share to do.

I gave him a quick synopsis and told him to get it done. If he found that he could not do it or it was impossible to do, he would have to convene a meeting with the whole team so we could regroup. I got a couple more calls and emails from him. Eventually he completed his task. Of course he was the last one done. So he did not look good anyway in the end.

Come on people. Put your pride aside and step up. Don't try to use someone else as a crutch. You are hurting yourself and the whole team.

Quality Assurance

I work in the back end on my current project. It is a big database reporting project. My role is to provide the structure for new tables, write code to load those tables, and sometimes write the code to create database views. You would think that would be a straight forward job. Not so.

 

We have a lot of environments for our system. I work in a personal schema. We have another schema that I share with other developers. I consider that an integration schema. Then there is a separate schema for our internal testers. Plus one for our customer acceptance testers. We have a performance environment with a lot of data. Then there is production.

 

Recently there have been numerous problems deploying changes to these environments. The process is for me to check code into Clearcase source code control. Then I submit a ticket in ClearQuest to request deployment of my changes. After me there is a long chain of people involved before my changes get deployed in the assorted environments.

 

The deployment problems culminated in a specific delivery we failed three times in a row. Some managers tried to understand what in the world is going on. Their conclusion was that developers such as myself need to guarantee that the deployments are done correctly.

 

Now my life involves emails where somebody will run a massive deployment which includes my changes, produce a huge log or logs. Then they ask me if the deployment was done correctly. LOL wut? This got old the first time I had to pour through the logs. Not only that, I got to go to the target environment and actually check if my changes are in there.

 

I don’t want to spend my days doing quality assurance for a bunch of people who work on the deployments and just ask me if they got it right. I want to be doing development. Perhaps the right way to look at this is as an opportunity to automate the verification of the deployments. I figured we could use some kind of embedded versioning to identify what is deployed. Or I could buld some smarts into a script that checks for the specific changes made.

Use the Requirements Already

I am working on a release at work. Initially we were supposed to replicate some bunch of database tables that the customer had in an old system. We did a quick dive into what was involved and came up with a rough level of effort. That translated into a schedule. The customer said the schedule as too long. Okay. Then if the schedule was fixed, we can up with a subset of the task we could finish on time.

Right no we are getting towards the end of the cycle for this release. Internal testers are trying to verify the work we did. One tester is sort of a programmer. She does automated testing with some Java programs she writes. Recently she sent out an email asking what she should test. LOL wut? I responded and directed her towards the requirements. Seems simple enough.

Later the test called me up. She said she wanted to send me a document. And she wanted me to mark it up and tell her what to test. Nope. First of all that's not my job. And second of all, she should not just test what I tell her to do. She should test what is required of us for this release. I had to say that a couple times. Use the requirements document. It is there for a reason.

I ended the call with the tester by telling her if she does not have the requirements document, call up our requirements analyst. She does not bite. Very simple. I worry that our testers do not know what to test. I don't want to be part of the problem by leading them astray. For developers who encounter a software problem, I say use the source. For everyone else, use the requirements.

Check Your Subroutines

We are delivering our latest release to internal test today. Had a code review yesterday. Many issues were found. We are fixing the highest priority problems. Before shipping to test, thought we would do one final smoke test. Ran all the functions. Checked to see if data was loaded. Most looked good. One target table was not getting any records.

Very odd. Called the function in the same way. The template for the function matched the others we were using. The routine even spit out some debug messages which confirmed each of the select statements found a lot of records. Me and my buddy ran this thing a total of four times.

The clue that got me on track to a solution was that a table with the same name in another database schema kept getting more records. How was that possible? We were setting up the PostgreSQL search path explicitly at the top of the function. But by the end of the function, it seemed the search path was changed.

Initially I thought maybe some weirdness was going on because the DBAs recently dropped the tables and recreated the tables in our target schema. However that DDL should have been instantaneous. It was just suspicious because they cloned the tables from a prior schema (the one now getting all the data).

Finally I had an epiphany. Maybe our big routine was calling some helper functions that were doing something naughty. I searched for the word "perform" which is how we call subroutines. Bam. There was a helper function resetting the search path to the wrong database schema. That was a very tough bug indeed.

Struggling With the Calendar Code

Turns out we need to work all weekend on fixing bugs before the customer will even consider accepting our system for production use. I got assigned a ticket to analyze. At first I had some trouble because they shut down the databases for maintenance. Damn. I asked them to cancel the shutdown so we could get to work.

Eventually the databases come up. I verify that there are some formatting issues with the data. And there is a vast shortage of data. Next I am told to fix all the problems I found. The formatting changes were not too hard. The code is just a view on top of a table which acts as a calendar.

Next I tackle the lack of data problem. Well the underlying table is just missing a lot of rows. Can I just generate some new rows? Not easily. There are all kinds of weird values in this table. My boss asks if the problem is fixed yet. Nope. He asks if there is anyone who can help. Yeah - the person who created the underlying data.

I call up the developer. She is out and about. But she shares with me a grand secret. The method to generate the data is a function in the database. Jackpot. Of course the thing does not work. But at least I have a starting point.

So I drill down to see the table that drives her generation function. It requires only a couple key fields to make the whole process go. Figuring out those key fields is a major undertaking. I dig up a manual I have that is 15 years old. It helps me figuring out some seed data. I eventually write my own function to populate the key fields.

Eventually I got a pretty good function to generate the very low level data. Then I make some calls to the function that takes it up one level. Then I call my function to create a view to wrap the table and expose stuff in the correct format. My counts are close but not exact. I trace it down to some duplicate records. Luckily my view can filter out this duplicate nonsense.

It is past midnight when I am done. I email out the team, letting them know that I am sticking some new data in these tables. I also share that I found some dup data down in there. Get a message from my old boss. He told me the function I found is buggy, and does not work for future dates. FML.