The Empty Guarantee

Today I received an e-mail requesting that I leave a block of time open in my schedule for an interview. Apparently this was to determine whether I would be placed in a position with another project. This seemed very strange since I had already interviewed with the project, done well, and received a memo from Human Resources stating that I was going to be reassigned.

So without delay I called up the person who sent me the e-mail. She was familiar with the previous interviews. But she stated firmly that I did not have a job with her project. The prior interview and transfer letter was null and void. I thanked her for the information.

Next thing I did was call up Human Resources. Left them a message. Have not heard back from them. So I called my project manager. He has a cell phone and tells us all to call him when anything comes up. I spoke with him and told him about the disturbing conversation I had. He then went to speak with the project manager on the project where I was supposed to have had a job. It seems that they got a smaller budget for the next release of their software. So they are reevaluating what developers they can pick up.

Great. So now I have to go back and interview with the project again. But this whole scenario is very unsettling. In the past, senior management made promises that they would find us all jobs when our current contract ran out. I guess those promises were only as good as the budget they could arrange for future contracts. This is after all a business. Now my mind is racing back and forth, thinking about how I may have very well been conned. Perhaps the smart employees are the ones that have already bolted. We shall see how this pans out.

Telephone Problems

I place and receive a lot of business phone calls at work. Two weeks ago I was on a teleconference call. The host of the call hung up, and the call hosed my phone. It just displayed the last number dialed on the screen. I could not get a dial tone.

So I called up the Help Desk. They patched me through to Nortel since they service our phone system. A trouble ticket was created for me. Then I went on vacation. I got back to the office a week and a half later. My phone was still broken.

I called Nortel up directly. Apparently they tried to fix my phone, then left me a "voice message" stating that they were done. Only one problem. They did not fix my phone. This time around, some guy came out and manually reset my line in the telco closet. And to ensure there were no further problems, he gave me another phone. He said this was not really necessary. But due to the circumstances he wanted to make sure I was good to go.

Everything seemed back to normal until I received my first new call on the working phone. Apparently the ringer volume was set up to max. I jumped out of my chair when I heard it. What kind of company sets the default ringer volume to loudest? And a few tests and scrolling through the menu options on the phone, I think I am back to normal. Sheesh. Come on Nortel. Keep my phone up and running.

Software Changes

I spent most of my day training the new team that will take over the maintenance of our software. The new team was especially interested in the new features we had added in the last year. We have been very busy this year. So the change list was long.

Luckily I had conducted a Customer Technical Review with the end user of our applications. So I grabbed my 20 page overview of the application changes. I went through this presentation with the new developers. It was well received.

Next I reviewed the most recent big changes that we did for our customer. These were last minute high priority requests that our customer made. I described each of the changes and how they impacted out applications. In addition, I pointed out some of the weaker parts of the design and requirements. These guys need to support the system now.

By the end of the day I had to get back to my normal work. So I escorted the new developers out of the building. When I got back to my desk, our client's headquarters had called and e-mailed me asking about some of the details of the recent software changes. I quickly scanned the new code to make sure I had my story straight. Unfortunately what I found out is that the customer wants some different business rules to be implemented. I spoke to our Project Coordinator to see how we would respond to any requests for this change in our last days on the project. The Coordinator told me that the customer had to submit a formal change request, and the new contractor would have to do it. Good luck to them. They will most definitely need it.

Training

Another company has won the maintenance contract that I work on. We have a couple weeks until our contract is up. The new team has already been assembled. They requested some time with us to help train them. This seemed like a reasonable request.

When the new team came to meet with me, I asked them what they wanted to get out of today's meeting. I assumed I would just walk them through a normal day of customer support. They agreed that this would be good. However they also wanted to know all the new features we added to our application suite this year. I told them that would take a long time.

So I went to one of our developers, and took a trouble ticket from him. This was a real life example that I would use to train the new team. I reviewed the text of the trouble ticket with them. Then I made sure we all understood the behavior in the application that the users experienced. Then I went over the behavior that the users were expecting.

This particular trouble ticket was very thorough in that it provided specific examples of the problem. One of the examples did not make sense. So I emphasized that when something like this happens, you should overcome any reservations and call up the users and/or system administrators. Right there in the conference room I called up the sys admin who submitted the trouble ticket. He clarified the details which seemed strange. We we now ready to run with the problem.

I showed the newbies how to log into the Production database and run queries to see the sample data. Then I found some data in development that was similar to the problem production data. Made a couple updates to my development data to exactly match. Then I followed the steps the users listed, and we were able to duplicate the problem. This was all done in record time. We got lucky.

Unpaid Overtime

I recently came back to work after a nice vacation. The project my team works on is about to end in a month. We are all scheduled to move to another project in our company that is similar to ours. Just about all of us had interviewed with members of this project, and received officials letters informing us of our upcoming transfer.

The first thing I heard when I got back was that there was another round of interview with this project. This seemed strange as the transfer was supposed to be a done deal. But here was the real kicker. A developer told me he was grilled about being required to work 60, 70, or 80 hours a week on the new project. And apparently they are all on call 24 hours a day.

There must be more to this than meets the eye. I am awaiting another round of interviews myself to get to the bottom of this nonsense. All of my team members are salaried employees. It makes no sense to be working on a death march. We get paid the same regardless of hour many hours we work.

I hope my first task on my new project is not to write a white paper on the drawbacks of working extra hours. However this may be the route to go. There comes a time when you need to make a stand against insanity. I think the right way to go about this is to come in with some hard data on the price the business will have to pay if the old "more hours mean more productivity" is not dispelled.

Black of Hat Blog

While on vacation from work I have done a lot of things. I visited a casino to do some gambling and lose money. Still working on the house, replacing my broke down oven with a shiny black new one. And I have been writing some software that is featured in my new blog Black of Hat.

The first free program I have posted to my Black of Hat Blog is one which launches Internet Explorer and navigates to sites of your choosing. I wrote the application using Visual Studio C++ which I am most familiar with. It is a self contained executable with no installation other than copying the EXE.

I already have an idea for my second Black of Hat program. It will be a web site crawler which extracts a unique list of URLs. Now if I can only finish up and post this program by the time I return to work, I will have had a successful vacation.

Going on Vacation

Today was my last day for a nice long vacation. The plan was to go in, wrap things up, and say goodbye to everyone. Of course as soon as I entered the building, everybody came running to me to try to talk about the emergencies going on.

Apparently our customer's configuration management team had delivered the wrong version of the application to the users. And all holy hell was breaking loose. I had to tell my whole team to leave me alone while I determined what had happened. As soon as I identified the problem being with customer CM, not our company, the project manager breathed a sigh of relief.

Many trouble tickets that got opened today were flase positives. They were due to the fact that a really old version of the application got installed everywhere. The good part about this was that they were easy tickets to close out once users got the correct version of the application. There is a saying that goes "when it rains, it pours". Well it decided to pour down hard today. My phone stopped working right in the middle of a conference call. It ceased to function for the whole day. So I had to call into my voice mail from different phones to catch all the messages people were leaving me. Nice.

The real kicker happened late in the afternoon. But some unusual bad luck, a high priority trouble ticket had come in. But the ticket did not show up in our inbox. The customer started complaining that nobody was working this high priority ticket. By this time the Help Desk had gone home. I was going to sneak out the back door, but I warned the closest manager that once again things were going to get rocky. He talked me into doing a quick initial investigation, and providing the team with some information to help them get through this tough ticket. I tried calling the user but he had left for the day. So I went back and reviewed how I handled the last trouble ticket similar to this high priority one. And I tried to explain in English what the normal operation of the software was, what the perceived problem was, and how some actions by the user could make the software appear faulty.

My deep hope is that my vacation does not get cut short or cancelled due to this latest high priority problem. My fingers are crossed.

Flying out to Customers

The development team has a difficult problem to resolve. The problem never happens in the development environment. It only happens in production. My plan to deal with this is simple: grant one developer a temporary access to the production system. We can them work with our users to duplicate and resolve the problem.

Of course our client is one that has super sensitive data. So it will require an act of god to get us the production access we desire. However I try to put this in business terms that the customer can understand. When they have the problem, it costs them a certain amount. I can project how long it will take us without the production access (a very long time). I am aware of the risks involved with allowing developers access to sensitive data. But I put it back in our customer's hands. If it is important enough to them, we will get the access we need.

The funny thing is that a consultant had an alternative plan to diagnose the problem. They want to fly out to the customer's location and see the problem. On the surface this may sound like a good thing to do. However as a developer, I can already determine that this will add no value to our resolution of the problem. We already can Netmeeting in to our customer's computer and see exactly what they are doing. Somebody standing there next to the user will be no different.

Role of Detective

Today we got a trouble ticket about some data that just would not behave in our application. The phenomenon seemed to be a scenario that was not supposed to happen. At first I assigned this to a junior developer. Then I checked up with the developer. Seems he was still stuck on the last problem I assigned him. So I took over today's problem myself.

Luckily, we audit a lot of things in our system. So I checked the audit data in the Production database. Indeed this data had a strange history. After calling a couple users, I at least figured out the final state that the data should be in. So the first order of business was to write an Oracle PL/SQL script to fix the data. No problem.

The more difficult task was to determine why this problem happened in the first place. This was not a common occurrence. Otherwise we would have heard about it by now. I asked a system administrator to retrieve the log file that our application writes to the user's workstation. I knew which user was working with the data when it went haywire (the database audit told me this). When I got the user's workstation log, it did not match up with the database audit. Something was really fishy. At this point a junior developer, or even a senior one without domain experience, would have not been able to progress any further.

I pushed this problem to the back of my mind. I did not even try to figure it out right away. And sure enough a theory popped into my head. Suppose this user was working on two different workstations on the day when the problem occurred. Sure enough, after making a couple calls, I found out that this was the case. This does not solve the problem. I need to get my hands on the workstation log from the other machine that the user worked on. But I have a better idea of what may have been the problem. This is a perfect example of how maintenance developers need strong detective skills to resolve all but the most trivial of problems.

Hungry Consultants

Our customer had a high priority short time frame requirement to fill. They asked if somebody could do the job. The leader of the development team said this could not be done. And the project manager said the customer did not follow the proper procedure in requesting the work.

Turns out a consultant took on the task. And who said consultants never actually do any real work? This consultant took on the task, worked hard to get the fix right, and stayed late until it was implemented. In fact the whole consultant team stayed late to ensure that nothing went wrong with the change. True dedication.

Now this was not some charity work. The consultants were billing the customer high priced rates to get this work done. But this translated into the utmost in customer responsiveness. The thing I wondered was how could our company transform into one that delivers service like that. If we can do it at a significantly lower price than the consultants, wouldn't it follow that we would get more work? In fact, our company has lost the contract to maintain the software for this client. Somebody else won the bid with a cheaper price. Could be the result of some bad business decisions. No reason to cry over spilled milk. But we have to learn not to make the same mistake again.

Project Coordinator Duties

We have a member on our team that was hired as a team lead. He seemed to have a background in project management. So this should have been a snap. Some other developers thought that this lead was a bit weak in the technical skills department. I withheld my judgement.

As our contract started winding down, this team lead got promoted to project coordinator. I guess he had been constantly talking about his past experience as a project manager. So our project manager added some duties to his responsibilities.

The funny thing is that this lead (now coordinator) started to feel some pain with the new duties. The coordinator has to stay on top of the status for all the trouble tickets. And the coordinator needs to keep tabs on what the dwindling staff was doing. Finally the coordinator took over cost estimation for new tasks.

Hey. I guess when you talk the talk, you need to be able to walk the walk. Here on my project I have always said that I only want to be a programmer. I don't strive to get the architect title. I run away from team leadership. Definitely refuse to become management. This attitude is not always successful. Currently I have been drafted into a team lead position. But that is mainly due to my skills, and not because anybody thought I actually wanted this role. It feels like I am rambling. So I will leave you with a thought: Be careful what you aspire to become in the development world. You wishes may come true and haunt you.

Team Lead Duties

Now that I lead the development team, I need to participate in trouble ticket meetings. Twice a week we go over every single open problem. This in and of itself may not seem like a large chore. But you need to make sure you have your facts straight before the meeting. And you can't prepare too early because the customer wants the most recent status. The real pain for a developer like me is that these meetings are held early in the morning.

I cannot complain too loudly. The last time I got drafted into team lead duties we were really in trouble. There were a whole lot of open trouble tickets. So we had meetings every day with an irate customer. If the meetings went poorly, they were held twice a day. So I have to keep looking on the bright side.

Some of the pain from our current meetings is that there seems to be a lack of coordination. Customers are looking at different trouble ticket lists than our team is. And our project coordinator is new at this. Most of the responsibility of keeping the show in order belongs to the project coordinator. Our help desk support is also new to the project. These little things add up. It is not quite pandemonium yet. But some people are worried.

The good news is that I shield our rank and file developers from this heartache. I just tell them to fix trouble tickets and post their status into the system frequently. I might not like the role of a team lead. But since I am here, I might as well do a good job. But not too good. I do not want to do this for the rest of my life.

Nice Book

I read a number of web sites and blogs about software development. Saw a post in one of them about a developer with excess cash in the budget to purchase books. He was looking for recommendations of good software development books to buy.

One such recommendation from another poster was "Working Effectively with Legacy Code" by Robert C. Martin. I looked this book up on Amazon. And on the surface it looks pretty good. It seems to cover topics like setting up a test hardness for old code.

Then I got the sticker shock. The MSRP for the book was $54.99. Ouch. That's a lot of greenbacks. But I guess this means that this is a serious subject demanding a high price. Still not ready to make an investment in this book, even with the discounted Amazon price of $46.30. But maybe I will put it on the "to buy someday" list.

The Team Lead

When I first joined my project, I was very happy in my position. We always had some subcontractors that would lead the development team. I was able to go off and solve technical problems. The "lead" would normally be stuck attending meetings, compiling status reports, doing project scheduling, etc.

Then came that horrible day when we stopped paying our subcontractors. At first they thought it was a glitch. But then it appeared to be a change in direction for our company. No more paying subcontractors to do work that company employees can do. That's when I got drafted to be team lead of development. What a crock. This resulted in me meeting with an irate customer every day explaining why there were many problems with our software. Then I had to start compiling metrics reports. And I had to do a whole lot of other managerial tasks which I hated.

Luckily with such responsibilities comes a little bit of power. I was tasked with hiring new members for the development team. So I searched high and low and hired my replacement! The new hire became team lead. And I went back to being senior developer. Ahhh this was the good life. But like real life, all good things come to an end. Recently this team lead left the company. And once again I found myself drafted into becoming the team lead. Now I found myself skipping lunch to prepare for tedious status meetings.

So you can imagine my lack of enthusiasm when one of our system administrators called to congratulate me on my promotion. The problem is that there was no corresponding promotion in pay. And even if there was, I would not want it. Leading a team is a thankless job which prevent hands-on guys like me from doing what we do best. That is, software development.

To Spec or not to Spec

I just finished reading a book that strongly recommended you do not write a functional specification for your web application. The book stated that such a spec leads to all kinds of problems. In fact, it stated that the only people who are interested in such a spec are those people who really don't have anything else to contribute to the project. On the surface, this recommendation seems to make sense.

On my own project, we used to have a mammoth functional specification document for each of our applications. These documents actually contained both the application requirements and design together. That's what added to their size. The funny thing is that, even though it was a chore to keep updated when the new requirements came in, it was a very useful document. The spec always informed us what the correct and required behavior was. Many times the customer would complain and issue trouble tickets on our software. The spec saved us a number of times by authoritatively documenting the correct behavior of the application.

Some time ago, the project manager decided to move away from this functional specification style of documentation. Instead we would separate design and requirements. Unfortunately, now the requirements are stand-alone and are hidden in some requirements tracking system. And now the newly created design documents are light and do not contain enough information for anyone to make much use of them. It is strange. Our customers and even our own developers find ourselves going back to the functional specifications to get the real details on different parts of the applications. This experience may differ, but I am wholly unconvinced that a specification, in and of itself, is evil.

Build Script Language

I was reading a book that recommended you be a Maven to your industry to gain interest in your software product. It made me think back to when we had an excellent software architect to recommended we port our build scripts to the Maven tool by Apache. We did end up porting our build scripts. However we did not use Maven. And the awesome software architect has moved to another company and has since informed me that there is an even better build tool out there.

First let's talk a little history. I have been on my current project for a very long time. Before I got here, we had a UNIX shell programmer who worked for configuration management. He build an elaborate set of korn shell scripts which produced our official builds. These scripts were complex and performed a lot of good functionality. The problem was that after the shell programmer left, it was difficult to maintain these scripts.

So we had a couple Java developers try to rewrite the build scripts. They were excited to use Apache Ant because it provided a lot of functionality that they needed. However these developers did not fully understand all of the tasks that the old build scripts did. So our new scripts are lacking in some areas. For example, if the build fails at compile time, the new build just repackages an old build as the new one. That is not the desired effect. We want builds which fail to compile to halt, informing us that we have a problem. Who knows what other bugs and missing features there are. At least the new builds are good enough to get by for now.

All of this brings up the question of what is the best language in which to write a build script for a somewhat complex product. Back when I was doing some prototypes for the rewrite, I accomplished pretty much all the functionality with Windows command files (*.bat). But the inner programmer in me wanted to write it all in plain C. The real goal however is to produce a set of build scripts which are easy to debug and maintain. The Ant scripts we have now are not too difficult. I think the jury is still out on this question.

Team Morale

Our company lost the contract for the applications we maintain for the customer. As expected a number of developers have jumped ship to work for the new contractor. This is expected. In fact, I joined our company when my previous employer lost their contract for the application suite.

The new contractor had scheduled some meetings for us to transition the information to the new maintenance team. I had one such meeting last week which was scheduled for 3 hours. In the middle of the meeting I got interrupted twice. Apparently there was an emergency all hands meeting for our project. I had to cut my transition meeting short.

The project manager and the company director above him called the emergency meeting. They wanted to reaffirm that they would try to place all our developers on other projects in our building. I guess a lot of people had been getting nervous. This is all understandable.

The company director remarked, "I am out of here". He had slipped that he himself was leaving the business area for a new opportunity. This alarmed me. Instead of reaffirming me, I now started to have my doubts. Why couldn't they just hire the Purrfect Angelz to keep our morale up like the did for the boys in Iraq (see picture above)?

P.S. I later found out that the company director has since changed his mind, and instead of switching to another project in our company, he is joining the company that has won the maintenance contract. Rumors are flying wildly.