Good Status

Every week our project sends a status report to our customer. This is a contractual requirement. The way status gets collected is that it is rolled up by team leads to managers to the status person. I am on the bottom of the totem pole. So I give my status to my team lead. It is actually quite simple. I send an email outlining what I did in the past week, and list percentage complete on long term scheduled tasks. This is simple right?
Here is what happens. Each week our team is stressed out with the emergency of the week. As a result, I don’t think anybody sends our team lead status. He then needs to make something up to send to his boss (the manager). Do this enough times and you get inaccurate status. Sooner or later you will get found out.

I make sure I am not part of this cycle. It does not matter the circumstances. I always send that status email on time every single week. Am I off for half the week? My team lead still gets my status. Is my e-mail broke down? I go home to my personal email and send off my status. It gets done. Why am I so adamant about this? I have been a team lead before who had to compile status. And it is such a simple and short task that gets a lot of bang for the buck. There is no excuse to not send status.

Now there are some tools I use to help me compile my status easily. Whenever I work on anything, I make a note in my “work done” system. Each day is a separate text file. To compile status, I look through the files from the last week. It is almost a copy and paste of the highlights. A good process makes status reporting a breeze instead of a chore. I highly recommend it.

Stick to Your Guns

Our customer’s test organization found a bug in our software. This happened during a week where the edict was to fix bugs ASAP. So a new guy shipped out a fix to the bug. Turns out the fix did not work. That usually makes us look bad. From the outside, suspicious customers think we are just trying to make it look like we are fixing stuff when we are not.

My manager asked me to step in and find out how long it would take me to recover from this situation. I gave him a date. This is where the madness began. My team lead had a lot of questions about the date. I told him it was simple. I would guarantee the fix could be in by the date I gave. My team lead kept asking questions about how I arrived at that date.

Later I was given the task to take over the work for this problem. However this was not the end of the story. My team lead had a lot of questions about the problem. It took a long time for him to understand the complexities of the problem and related problems caused by the bug. He still was trying to bring in the delivery date. He told me he was getting pressured by a VIP in our company.

In the end I got fed up. I told my team lead that he might have some insight into the problem, but getting him up to speed was slowing down the resolution. I told him to tell the VIP to trust me as I have a track record of delivering when I say I am going to. My team lead had to man up. He told the VIP that there was no way to make the delivery any earlier.

All my team lead had to do in the end was write a memo with the exact plan I had to fix the bug and make sure nothing else got broke in the process. I know it is hard to be a team leader or a middle manager. But if you have the job you need to actually lead. Don’t cave in every time somebody important tries to get you to commit to shorted delivery time frames. Fight that fight. Your whole team will benefit.

I think my goal for this year is to hold my team lead responsible for representing my interests in dealing with management from our company. Otherwise I will be working late every night, working through the weekends, and be perpetually late on my projects.

Getting Time Off

For the second week in a row, my team lead decided we would work into the weekend to fix a lot of bugs. I decided it was time to make sure I got something out of this effort. So I requested my manager approve some days off next week since I have already put in a lot of extra time this month.

My manager asked if my team lead approved. I called up my team lead. He said he would not approve this time off. I was shocked. This guy looked to me in the last couple weeks to make some big releases happen. That meant staying late every day, taking work home, and working on the weekends.

Well I decided I had enough of that nonsense. I informed my team lead that if that is the reward I get for putting in extra time, there would be an end to the extra time on my behalf. Thus I am unavailable for work this weekend. And I told him to stop calling me after hours for support. This story only gets worse. I will save some more details for next time.

Domain Expertise

A new guy on our team got a challenging bug to fix. He took a long time to get up to speed in the business of the trouble. Our team lead decided to take over the work. However there was just one problem. It was difficult for the lead to figure this stuff out as well. So what does he do? He called me up.

I told my team lead that it was a challenging problem. He would need to dig in there to figure out what was going on. Unfortunately he was directed to fix this problem ASAP. I advised him to just give up and declare the problem done. He was not happy with that idea. So I gave him some other hacks that might qualify as getting the problem masked for now.

Turns out our customer rejected the hack. This issue was starting to get a lot of visibility. Both my team lead and I got an email from the customer declaring that the fix did not fix anything. I ignored the e-mail since my team lead made the command decision to ship a hack.

I somehow got involved in a long conference call with the customer for some other business. During the meeting I saw a lot of emails about the failed fix. Then my phone was ringing off the hook. I ignored the calls since I was in a conference call. However when the lengthy meeting was over, I found my team lead in a bind.

Now I could have just told him that I told him so. But that would do nobody any good. So I gave him some ideas on how to proceed. We could have saved ourselves from the drama by just doing the hard work in the first place. Shoddy efforts like this will only make the customer mad. Let’s see how it all turns out. The only thing I do know is that I got the deep domain knowledge. That knowledge is power. I must wield the power wisely.

No Dup No Fix

I got a call from my team lead. He said another developer was having problems fixing a bug discovered by the customer. This other developer thought he found some code that was suspicious. So he made some changes hoping that would fix the problem. This scenario was easy for me to analyze. If the guy could not duplicate the problem, he sure as heck did not fix it. It is simple as that.

This problem got reassigned to me. Like everyone else who could not replicate the problem, I tried to check out the behavior. It seemed to work. But I know that is not the end of the story. I went to peek at the production data. Then I recreated records in development that matched production. Finally I followed the steps the production user was taking. I encountered the bug.

I will admit this was a tricky bug. The software seemed to work for the trivial case. But you had to have some certain properties set for the initial record, and you had to navigate to another record that was different than the initial one. We had a couple teams try to recreate the problem. They were unsuccessful. Once I determined the steps required to recreate the problem, the fix was just a couple lines of code away.

Here is the lesson learned for maintenance programmers. You must be able to recreate a problem before you can do any more work on it. If you cannot do this, you have no business making any code changes to fix the bug.

Working on a Sunday

The customer wants more bugs fixed. They informed management. Management informed team leads. Team leads informed developers. Now I am here working on a Sunday. Great. Well I better make the most of it. I caught up on some e-mail. Then I helped out a tester. However the latest build was not complete. This was the primary objective for the weekend. Verify the build. What am I going to do instead? I am going to work on writing some tools. Yeah.

We have some ambitious system administrators working for our customer. They have some scripts which detect if your source control views haven’t been used in a while. If not, they mark the views as inactive and you can’t use them. Furthermore there is something wrong with their scripts. When they run, they permanently screw up your scripts. You cannot recreate them with the assistance from an administrator.

My solution to the above problem was to write a program which simulates a developer using all of his views. It is cool. The thing checks out some files from every view, edits them, then checks the updated versions back into source control. It looks like I am working on each and every view to the system administrators. I updated this program to work with my latest views.

Our software ships with the ability to connect to our customer’s databases. However we use different databases for development and test. To point the applications to these dev/test databases requires some hacking of the Windows registry. I wrote a small program which does this for you automatically. This comes in handy because each time you install our software, it overwrites the registry and limits the applications to just our customer databases. Today I updated my program to work with the latest software for next year. All in a good days work.

Too Many Questions

The other night I got a call from a DBA on my team. He was trying to respond to some problems our users were having in a new database. This DBA had some ideas about what was causing the problem. But he could not determine the source for sure. He had a lot of questions for me.

The next day I responded to his questions. My response also went to all kinds of people in the customer organization. That's when I started getting all kinds of emails and phone calls. Then my sponsor in the customer organization called me up and started asking me whether some of the things he was talking about were the cause of the problem.

That is when I realized we were in trouble. We were not going about this in the right way. I had to be abrupt with my customer. I told him that we just needed to assign this problem to a developer. The developer could figure out for certain what the problem is. We were wasting a lot of time that development does not have chasing down theories and answering questions that have nothing to do with the problem at hand.

The bad news is that the problem got assigned to me. However once I was on a mission to solve the problem, I was able to concentrate on and zoom in on the source. I did not have to educate any users. I also did not need to explain a lot of the system to new users, or trace down false leads by smart people who just don't know the system well enough. The fix will go out in our next release. I learned a valuable lesson. Don't contribute to bad processes for problem resolution. It will only be a total waste of time.

How Much to Fix

Yesterday I fixed a bug found by our customer. However I realized that the same problem affected many other parts of our application. What was I do to? Should I just ship the specific fix and move on to other bugs? Or should I correct every instance of the problem?

I like to code. To research other bugs would be non-coding. Therefore I was inclined to fix the other instances of the first bug that I fixed. Plus it would look bad if the customer discovered the other instances. There would be a lot of questions and meetings to discuss why we did not fix all these other scenarios.

So I spent the rest of the day banging out some code to fix all the scenarios. This was really fun. In fact, when it was time to go home, I decided to keep coding. Unfortunately all good things come to an end. Now I just need to do a lot of unit testing. Got to make sure that I did not break anything.

Out of Order

I started working on a trouble ticket today. My team lead called me and wanted to know how long it would take to resolve the problem. I estimated a couple weeks since I did not know what the issue was. That was a worst case estimate.

Luckily I was able to duplicate the problem. The user interface called a Pro*C function to fill up a structure. The UI code looked like it was extracting the results correctly. So I went into the Pro*C. It was a bit confusing. A bunch of records were being summed up. Then the code tried to transfer the result of the SQL query into the structure required by the caller.

The code to copy results in the structure was convoluted. That was a sign that the error might be in there. There were multiple loops going on, coupled with a number of counters. The loops went through the query results. The counters identified where we were in filling up the output structure. That's when I spotted some disturbing logic. If the two did not match, the output got set to zeros. This was the exact behavior that was being reported in the bug.

It was then that I realized what was going on. The code assumed a specific order in the query results. But the SQL did not have an ORDER BY clause. I decided against adding the ORDER BY to the SQL. We might be querying a whole lot of code. I did not want to add extra overhead for the database to do the sort. I just wrote a quick routine to scan through the results while putting the answers in the output format. Problem solved. Now I need to fix all the other places where the code assumes a specific order coming back from the query. This was quite an exciting problem to solve. I even worked some extra hours to get this thing fixed today.

Fix Problems Faster

Today I heard two similar comments on my development team. The customer is concerned about the number of defects that are opened against our system. That is a valid concern. However the response from our management was for us to fix problems faster.

Yeah. What do they think? I have been sitting around all this time artificially slowing the pace at which I resolve bugs? LOL. From the outside I would have thought this was a joke. Knowing my project, it is no joke.

Ok. So how do you get problems to be fixed as a faster clip? Well you cannot will it into existence. And you just can't "try harder". You got to work smarter.

Here is the catch 22. There is no silver bullet for this problem. It will take some head down analysis to identify the process problems which are inhibiting bug fixing productivity. Who can tackle such a challenge? Yep. They need developers to do it. And if you spend time trying to fix the process, you bug resolution rate will be even lower.

There is only one decision I need to make. Do I just ignore the nonsense about the "try harder" speeches? Or do I step up and call them on this joke. I am going to have to meditate before making that decision.