No Volunteers

Today our customer discovered a big problem in our production system. As usual I got called into a conference call to discuss the plan of attack. Our manager had drawn up a 6 phase plan, ranked in order of importance.

The problem occurred in the back end part of our system. The guy responsible for that piece would do most of the heavy lifting on the resolution. However our manager asked who was going to do what tasks. He first asked our lead DBA what he would do. Essentially the DBA said he would do nothing as it was somebody else's problem.

Then my manager came to me. I volunteered for the lowest priority task on the list. It also seemed like one that was fun. My team lead said he would help if needed, but he committed to nothing. I tell you. These leads are smart guys. They make it sound like they are team players while they slip out the door and go home.

It seems like I got tricked into doing more work for this problem. It was for a good cause. Our back end developer is overworked as it is. So I reluctantly agreed to take another one of the six tasks required for completion of the problem. Now I need to come up with my own plan to get some of the new guys to start doing production support. Then I won't be stuck in these conference calls and taking work home all the time. Time to start planning.

Difficulty in Design

Right now my team is engaged in the design of the new changes for next year. The team consists of me and two new guys. Luckily the new guys are experienced in software development. We also have a documented set of requirements. There is just one problem. The new guys can't make heads or tails of the requirements.

Here is how the requirements were collected. The customer sends us the experts in the business. These people have made a career out of their work. They know the current system inside out. And they have a vision of the new stuff and changes they must have. Somebody from our requirements team transcribes what these experts say. The result is a document which makes a lot of sense to an expert in the system domain. To anybody else, including our business analysts, this document is gibberish.

Luckily I have been working on this project for a very long time. My job is to translate the requirements documents to the developers. I fill in all the expected knowledge that you need to understand the document. For example I show the new guys how the data flows through the system. I also decipher the secret codes that the document refers to. And I show how competing requirements have to work together, assigning a priority to them based on outside knowledge. The new guys are a little apprehensive, but I think we will hit our design document delivery.

New Guy Troubles

Last week our new developer got a laptop from the company. Unfortunately we do all our development on laptops owned by our customer. However obtaining one of those is a lengthy ordeal. We need the new guy designing software now. So I figured he could mock up some new screens in Visual Studio.

Once the new guy got the laptop, I sent him to the keep of the Visual Studio 2005 CDs. We have a license for the whole team. Unfortunately the install kept bombing on the poor dude. The install asked for disk 1. He had the first CD ROM in his computer. Must be a corrupted CD.

I had the guy try to install the fewest options. That did not work. Then I sent him to Microsoft to download an express version of Visual Studio. Too bad those versions did not have resource editors. This was a senior guy. So he figured something out. It almost got to the point where I was going to go home to get my personal copy of Visual Studio to loan to him. However like I said, he got something working on his machine. Now he is knocking out prototype screens like crazy. We are back on track.

Developer's Developer

Our team at work has too much to accomplish in too little time. Yes I know that’s always the case in software development. However the upper management has said that we must deliver on time to the customer. The response from development was that you either had to cut back on the scope of deliverables, or add massive resources to development. It seems somebody actually heard us. We have been given the green light to hire people.

Hiring good developers seems like a very difficult task. We get resumes of people who look ok on paper. But when we get them in for an interview, they seem as if they have never coded before. Maybe they were the developer who did the grunt tasks, but did not actually design or develop anything. This pattern seemed to repeat itself over many interviews. We were getting to the point where we thought we might have to accept some substandard candidates. Then we got a guy in here who seemed ideal.

The guy we interviewed had many years of experience. He seemed to work in commercial software development. Here is how he operated. He would speak to the actual users to find out what they wanted. Then he would go code those features, release quickly, and obtain feedback. This sounds like the latest agile software methodology. The guy was fluent in Windows development. Normally I ask candidates the easy questions and they fail. This guy passed with flying colors. He was even able to address some difficult questions we threw at him. I had one thing to tell my managers – hire this guy.

There was some initial reluctance as this candidate did not seem to have a track record of producing a lot of documentation. Unfortunately our customer requires that. He also did not seem to have a lot of structure in his methods. I did not sweat these details. This was a programmer’s programmer. We would love to have him on our team.

I knew the first week the new guy started that we had made the right decision. He read over the requirements that the customer had sent us. We were quickly debating the interpretation and some proposed designs for those requirements. This new guy was able to state and defend many opinions on how we should develop. He was very concerned with good Windows user interfaces for our changes. He also was adamant about the way we should design some of those user interfaces. I have a feeling we are going to benefit highly from this new hire.

Good Night's Sleep

When you support a big system that runs 24x7, you are going to get those late night support calls. But you can try to minimize their frequency. That gets done by applying best practices to your development. Let's talk about some controversial best practices.

The first is that you should not mix data from development, test, and production servers. Now I can agree that production should never mix with anything else. However on my project we do share some development data with the internal test team. This is not optimal as it does not allow test to add the most value. But sometimes they need some data from development.

Next I have heard it said that you should not roll your own caching techniques in your application. This one is debatable. I recall when users were complaining about 15 to 45 seconds of delay to do a repetitive task in our application. That was unacceptable. There were some optimizations that gave us a few seconds. However the eventual fix was for me to implement an optimized caching system that I wrote. The customer was happy. So was no. No harm done I say.

In the next post on staying away from work on night, I will go over practices like queues, staging, and unit testing. Until then, I hope you do not get the late night call from work.

Night Calls

For the most part, I love maintenance programming. However there are some bad things that come with the territory. I hate getting those late night production support calls. Don't get me wrong. When my code crashes the system, I want somebody to call me up so I can fix it. However I hate being the go to guy who gets called regardless of the problem source.

This is why I read with great interest a set of tips for avoiding that late night call. One is to teach others. You got to make sure other people know why things are implemented a certain way. That way they are in deeper than just an understanding of the code. And they can take those late night calls instead of you. So noted.

Another global activity that can prevent the midnight calls is to keep fixing bugs until the system works. Too often we hurry up and patch something. Then we move on to other tasks like writing new software. The problem is that the simple band aid will eventually come off one day. Murphy's Law states that it will come off at the most inopportune time. For me that is 3:00am.

We have a lot of back end code that runs during a nightly process. Here is a criminal design methodology we use. We do not validate the input. Yes I know. That is a recipe for disaster. The problem is that this stuff needs to run quickly due to the volume. So we assume everything will be ok. In case you have not figured this out by now, nothing is ever ok. That's it for now. I have a lot more to say about this issue. Stay tuned for some more tips.

Build Broke Again

I have been on a tight schedule recently. We are late with the next software release. I told my team lead that any interruptions will cause further delays in the schedule. Luckily I have been able to eliminate most interruptions due to people needing my help. That is the usual drain of development time. However yesterday I found another stumbling block. The version of the code checked into source code control is broken.

You should never check in code that breaks the build. Sure nobody did this on purpose. However it affects me in a big way. I get the latest code checked out to make the changes for the software release. If the code I check out already does not compile, I am not going to be able to make much progress on my task.

From a casual inspection, it looks like there are two sources to the breakage. The first is that somebody made a lot of changes. But they did not check in all their files. So the build works locally on their machine. However the missing file updates cause compile issues on the source code control version. The other problem stems from the fact that we often make changes in the application that must coincide with database changes. It looks like the database changes are missing from the database against which the code compiles.

There must be a way to minimize these problems. We used to do nightly builds from the source code control version. That way we knew immediately when the code base was broke. Now it seems we don’t find out until somebody checks out some code and it does not work. Perhaps we should analyze the code and track down the guilty parties. Then we could do some evil things to them. This may minimize further disruptions.

Premonition

Previously I had written about how I kept getting pulled off my new development task for all kinds of emergencies. The end result was that the software release is late. Very late. My team lead called me up and said the customer and upper management are not happy with the lateness.

Today I got a request to go to headquarters to discuss the state of the the current and future software release. Normally I just answer any such summons from the big managers. However my team lead had told me to not work on anything other than getting this done. I called him up immediately.

My team lead said he would go personally talk to the big boss. He later called me up, saying that the big dog required my presence. I would have to call him up to explain why I could not report to headquarters and get my work done at the same time. Well I guess we are going to lose this battle. I did leave a voice mail for the big boss. We will see if he calls me back.

Here is the reality. When I get to go to my office and write code, we make progress. We don't make progress if I do anything else. Hey. Things could be worse. I was once on this death march of a project where we spent about 4 hours a day in status meetings with the customer. That was every single day, seven days a week. But that is a story for another time.

Attack of the Pointy Haired Boss

We have a big release due to the customer in a few weeks. A large portion of the tasks have been assigned to me. The first task was to estimate how long the work would take. Luckily I have been working on this project for a long time. Therefore I was able to produce a high fidelity estimate as to how long it would take.

You would think this would be a project manager's dream. A worker can tell exactly how long a task would take. If I was able to work the amount of hours listed in the estimate, I would be able to produce the software update. That is just too simple. If there is one thing I have learned in this life is that nothing is ever simple.

When the time came in the schedule to start working on the release, I got down to business. Then the inevitable happened. My boss said I needed to work on another task. I told him I was busy on this software release. He told me something else was more important. This happened every single week that I was working on the software update. One week I had another task that took up the whole week.

This is all fine and dandy with me. I did my part and warned my manager that if you have me work other tasks, and you don't have somebody else come in to work the software release, the thing is definitely going to be late. My boss told me he would deal with it. Now we are late on the software release. The guy who leads my team said the customer is unhappy that things are late. And some big boss promised the release would get done next week. Hello? I got the feeling that I'm going to be getting a visit from a big boss that has pointy hair. I just know it. Unlucky me.

Job Perks

I have been hearing or reading about a lot of great perks that developers get. My own company is decent with the extras they provide employees. However check out some of what the good life offers at other companies:

Some people get free parking. I guess that would be good if you worked in a big city. Others get access to a gym or a paid outside gym membership. These sound like standard fare.

Others get all kinds of free drinks at work. That ranges from coffee or tea to soda or even beer. The top end guys get free Starbucks. Some companies provide you with a free phone and paid wireless service. That is a blessing and a curse. I bet they expect you to answer the phone after hours for work emergencies.

Here is something I normally don't associate with software companies. Some people say they get cable TV with all the premium channels. Who has time for TV at work? A number of the Joel on Software followers report that they are given Aeron chairs to sit on.

Although there are some rare locations that provide free food, some places provide a subsidized meal plan. That sounds good to me. I have heard about the great computer hardware available to some geeks. This includes everything from dual processor systems to large dual displays. Nice.

Some companies provide a private office for each person. This is even better if the office comes with a great view. To take it a step further, other companies allow you to work from home. Personally I don't think that's a great idea. But that's just me.

Here is the ultimate perk. It sounds almost like something only vice presidents get. One dude got an American Express card and was free to buy whatever he wanted, including trips to places. I want to share some of the great things my company provides me. But that will have to wait for another post.

Crazy Comments

I am trying to fix some problems in one of the newer applications in our suite. This code is difficult to maintain. I have to give the original coders some credit though. These guys have some of the funniest comments in the source code ever. At times I feel like they are in a confessional, and are telling me their coding sins by way of comments.

Here is just a sampling of some of the commenting gems I have found in one of the components:

//To blink or not to blink, that is the question

// We are rushing and I am hacking.
// Short term gain, long term pain

// so, you see, my inability to create a better design
// is not just a limitation, it is a security feature.

// We kind of hacked this together
// without a lot of forethought
// ok, I hacked it together (anon)

You can't make stuff up like this. I am often laughing out loud as I come across these ramblings in the code. Now I wonder where all those young programmers are now that initially made these comments. Good luck boys of the Case application.

Pattern Mania

I needed to code some changes in one of our newer applications. After tracing the code, I found all the logic in a stored procedure. This was good news because I am an Oracle certified developer. I knocked the changes out in about an hour.

The problem came during my testing. The new application would not display the new data. Here is the thing about this application. It was written by a bunch of young programmers that had just developed design patterns. I don't have anything against design patterns per se. However these guys have patterns to manage their patterns, if you know what I mean.

I could not figure out how to get this application to display the data. This is unusual for me. I pride myself on being a top notch maintenance programmer. Normally I can go into anybody's code, and figure out how to modify the application. These guys have a lot of code that throws me off. There is much code that does not really do anything. And the pattern craziness just makes things more abstracted. I decided to give myself the weekend to regroup and tackle the problem next week.

Google Chrome Operating System

The other day I was in the car with a bunch of guys going out to lunch. One of them asked me what I thought about the Google Chrome operating system. I told him I did not think Google was in the operating system business. Well I was wrong on a few accounts. Google already has the Android operating system for mobile phones. More importantly, Google just announced that they are working on the Google Chrome Operating System.

The Chrome O/S will be open source. There are rumors that it will be free. I have also heard some speculate that there will be a small cost for it. It will initially be targeted for netbooks. The goal is for the O/S to be able to start up quickly (i.e. a couple seconds).

You would think that it would take a long time to create an operating system from scratch. Google said that this is going to be different from Android. However the key is that they are going to run on top of the Linux kernel. That makes this a much smaller scope. Yeah it is a little more than just Chrome running on Linux. The point is that they may be able to pull this off by the second half of 2010, which is when they think they will release the O/S.

What is the big deal about this announcement? Well it might cause Microsoft to react in ways it has not in the past. Microsoft Windows could get a shave in price. Better yet, perhaps Microsoft will come out with its own new lightweight O/S. This announcement by Google may diminish the role or importance of Google Android.

I think I will hold my excitement over this development until I see the Google Chrome operating system code being released as open source. Then I will see what type of code we are dealing with. It will also be significant when Google actually ships this thing. I bet it may be like other Google products where it remains in Beta forever. Hopefully it will also not pan out to be like some other Google products (other than search) which are not first rate.

Little Bit of Perforce

I am in the middle of testing out a code review client from SmartBear Software. Apparently it requires a source code control system. One of the free ones it supports is Subversion, which I have hear a lot about. However I found that I needed to register and create an account to download Subversion. No thanks. Instead I chose to test out Perforce.

Perforce is a client server source code management system. There are a number of client programs that are available with Perforce. One is P4V, which is a visual client. Another is P4Win, which is a Windows client. What? Are those two the same thing. I really could not figure it out by reading up on it. Maybe P4Win is an older version?

I installed the Perforce "server components". Now I am assuming that means I got the Perforce server. It gets registered to a port, and the clients need to connect to the name Perforce assigns to that port. Then I download and install P4V (the visual client). The darn install requires a reboot of my machine before I use it. Now I am ready to give this code review client a spin. Let's see if the integration with my source code control makes it any easier than the browser based tool I previously examined from SmartBear.

Project Management Beginnings

My project has a weekly team meeting. Usually this is a conference call. However once a month we physically meet together in a big room. Today was one such physical meeting. Somebody from the project management side gave a brief overview of how the schedule (and changes to it) works.

Some interesting facts are that we track both original and projected dates. This is for both the start and completion of every task. The funny thing is that most people on the project don’t know the difference between the two. Hopefully after out training today, everybody will at least have a basic understanding.

If there is a task that is not completed on time, and it will affect the critical path, we need to make a schedule update. This update is a lengthy process with negotiation between the stakeholders. We were told that being late, in and of itself, is not something that will get you in trouble. You just need to make sure you communicate any projected lateness early.

The schedule itself has most of the tasks we shall be undertaking. That alone is a significant accomplishment. The problem with our schedule is that there is no resource loading. For example, we have the tasks flagged in the schedule that are for our team. But there is not a mapping of the task to an individual. Hopefully there is at least a straw man effort to match high level effort with team sizes. But until this is more detailed, we are still going to be surprised with how closely we can track to the published schedule.

Code Collaborator

I have previously reviewed SmartBear Software's Code Reviewer. There were some issues with the browser based user interface I used in that tool. A guy from SmartBear recommended that I use their client tool instead. It was supposed to be easier to do the code review tasks I was attempting. Therefore I installed the Code Collaborator Client.

When I launched the application after install, I was a bit underwhelmed. The browser code review tool was a huge web page that was very busy. The client tool was a dialog with a list control in it. I could not find any place to start a code review. The buttons seemed to only apply to interaction with source code control systems.

Now I was doing my tests on my home computer where I don't have source code control set up. But I am a sucker for pain. So I searched the source code control systems that Code Collaborator worked with. There were a few free ones listed like CVS, Perforce, and Subversion. I had heard good things about Subversion before. So I tried to Google it. All I came up with were customer clients for Subversion. I did not want that. Then I went to the next free choice. That was Perforce. I will give you more details about that experience in a future blog entry. And I will details how the source code control worked with Code Collaborator.

Configuration Mismanagement

Yesterday the guy that leads my team asked me to do a build. We have a separate configuration management team that does the actual builds. I just needed to label some files in source control, and tell the build guys to do their thing. This should be an easy task. But I forgot the first rule of software development. Nothing is ever easy.

I got in this morning. The build was broken. So I investigated what was happening. Then I found it. We are in the middle of making some big changes in our system. Some of those changes had been committed to source control. The problem is that we use one branch for all of development. When the latest build needed to go out with a few files changed, at least one of those files had dependencies on some other in progress file changes which are not getting promoted.

Identification of the problem is usually the hardest part of fixing bugs. I got through the hard part. But how to we proceed? Yes I can manually undo the changes which were breaking the build. That did not seem right. It would make me the new source code control. This should be done by the process or the source code control software. No such luck here.

I did reach a compromise with my manager and team lead. Here is the plan. I am to analyze the in progress files which are absolutely needed to get the build to work. Then we will ship these files with the other ones which are required for this delivery. Hey. If it works, we will have been lucky this time. However the process itself is broken. If we do not fix the underlying process, we will repeat the problem over and over again.

Review of my Code Review

Previously I had downloaded Code Reviewer from SmartBear. I got an email wondering if I tested out the software. So I decided to give it another run after having put the software to the side for a while.

In my mind, I created a typical scenario of code peer review at my work. Adline has some code changes. She creates a review, indicating the files of interest. Natalie then does a review of the changes listing 4 defects. Adline concurs with 3 of the 4 defects, but decides to override the 4th.

I fired up the browser based version of Code Reviewer. From past experience, I knew Adline had to Apply the changes creating the review, then Attach Files. When Natalie logged in to do the review, the review was nowhere to be found. It took a while for me to figure out that Adline had to also Begin the review. When it came time for Adline to accept the 3 defects but override the 4th, I found that only Natalie could do that. The author does not seem to have that capability in Code Reviewer. Also, Natalie had two options: (1) delete the 4th defect, or (2) accept the 4th defect as fixed. Neither of these seemed to match the reality.

The Code Reviewer software is an interesting package. I suspect some of its power involves integration with source code control systems. And although I did investigate that aspect, perhaps I will write about that another day. For now I think that this product has a flow which does not match the process my team has in mind for a light peer review process.