Job Choice

I was working on a project that was a government contract. A new company won the re-compete of the contract. So I inquired with the new company about joining them. I have a lot of years experience on this project. The new company said I cost too much.

At this point I considered staying on with my own company. They had another huge project which seemed to fit my skill set. I got a letter from Human Resources confirming my switch to the next project. Then I got a message from that project stating that I needed to schedule an interview. That seemed odd. I called them up and told them that Human Resources was already reassigning me to their project. At that point I found out that I did not have a position on that project. They were only considering me.

Well I went to the interview. I showed them I knew my stuff. My only stipulation about joining their project was that I was not planning to work overtime. This project had a lot of problems. The default means of confronting those problems was to have everyone work crazy hours. I wanted to be sure from the onset that I was not going to engage in that madness.

An old coworker of mine had joined the new company that won the contract for my old project. He told me that he could really use my help on the project. I informed him that they balked at my salary requests. My old friend said he would talk to the top manager on the project. He said he would try to convince him of my value.

Eventually I met the top manager in the new company. He said he could not pay me my desired salary. However we made some concessions and I decided to join the project. My old buddy was very happy. I was also pleased since I really liked this project, and it would not require any overtime to speak of.

I got the real confirmation when I heard the customer was phasing out the big project I was thinking about joining at my previous company. Being the new guy, I would have been the first to go probably. There was no way I could have guessed this would happen. Well maybe I knew something was up when every developer was working crazy overtime. Sometimes you just get lucky with your employment choices. I know I sure did.

Evil Coding

I got assigned a difficult trouble ticket. The customer was reporting that a lot of garbage was being displayed on the screen. Initially a developer thought it was just the wrong version of some DLLs being installed. So they made the resolution be to reinstall the application. This did not correct the problem. The customer reopened the trouble ticket. We get dinged when that happens. My team lead assigned this to me.

Some other developers continued to believe this had to do with old DLLs being installed. I told them this might be the cause. However I needed to see it myself. I would not believe this unless somebody could duplicate the problem. Nobody was able to do that except the customer. I took a look at some log files. That did not help much. It seemed like the program got messed up immediately upon startup.

Another developer said he solved a similar problem before. He said he found an instance where the code takes a CString object, gets a constant pointer to the char data it holds inside, then casts away the const. Then the code would go ahead and write to that pointer. It just so happened that the CString object was empty. So the location in memory that has the empty string got corrupted. That just blew away my mind.

I went and searched through the code that gets executed early or frequently in our program. Sure enough there was another instance where the code does the exact same thing. The const was casted away. And the code started writing to the location identified by the pointer. Here is where this insanity actually turns funny. I saw a comment from the original developer who wrote this code. He admitted that he did not quite understand the CString class. That is why he was casting back to the familiar character pointer.

The good news is that the fix to this problem resolved a number of trouble ticket the customer had opened. These problems were ones that developers could not reproduce. They were instances where our application was behaving very strangely. In fact, another developer asked me to let him know how I was progressing with my trouble ticket. When I explained some of my findings, he recalled back to the similar bug he fixed before. And it turns out that this led me to finding the solution for the problems he was investigating. Teamwork does pay off.

Game Creators Dark SDK

I ran a second instance of Visual Studio 2005 to debug a problem. Normally I close the default window that comes up in the source code pane. However this time I took a quick peek at it. There was a free download called the Game Creators Dark SDK. Apparently it was to assist you in writing games quickly. That sounded good to me.

So I forwarded the link to my second work computer. It is the one that is not mission critical. First I downloaded and installed Visual Studio 2008 Express Edition. It is free from Microsoft. Then I also downloaded and installed MSDN 2008 Express as well. Finally I downloaded the Dark SDK. However when I ran the installer, it complained that I did not have DirectX installed. Luckily it prompted me to start a download of the latest Microsoft DirectX SDK.

The DirectX installer had a couple errors. They said that the XAudio2 version that came with it had expired. I was directed to get an upgraded version. It was a little hard to figure out this was holding up the installation. The error dialog came in under the main install screen. I just figured something was wrong because it was taking so long. Then I thought I was ready for the Dark SDK. However I still needed to run Visual C++ at least one time to allow it to do some setup.

I created a test Windows application. It was a little disturbing because the Express Edition only comes with limited support for Windows programming. There were not frameworks such as MFC that come with the Professional versions I am used to. The generated code actually had a WinMain there. The prime objective was met. I finally got the Dark SDK installed. I ran the demo browser. The screen shots for the demos look exciting.

The only problem I have now is finding time to dig into this SDK. I am also trying to bone up on computer security to get another gig here at work. Next month I start a Java programming class at the local community college. I thought maybe I could play with this SDK during my vacation. However the missus had put the kibosh on that idea. Oh well.

DreamSpark Advantage

Microsoft has this program where they give free software to college students. It is called DreamSpark. Recently I signed up for a community college course to help brush up on my Java coding skills. So I figured I would take advantage of this offer. I wanted to get a free copy of Visual Studio 2005 Professional. Yeah I have a licensed copy of this tool at work. But I want a copy for my home computer. This thing retails for almost $700 on Amazon. Even the upgrade version is over $400. Free sounded like too good an offer to pass.

I went to the DreamSpark web site. You have to verify that you are actually enrolled in a college program. The instructions said to choose your college from the drop down. Mine was not listed. Oh no. The fine print said that even if your college is not listed, you can use a third party to validate that you are enrolled. I was willing to jump through such hoops for a $700 software package. I went to JourneyEd, which Microsoft uses to validate non participating colleges. They had me “order” a validation even though it did not cost me anything. I emailed them a scanned copy of my acceptance letter to my college.

The next day I got an email back from JourneyEd stating that they approved my request. I return back to the Microsoft site and got verified. I chose Visual Studio 2005 Professional and started the download. There were two files to download. I assume these files correspond to the two CDs that come with the real product. The next day the download was complete. I did not want to burn these images to real CDs. So I downloaded another unsupported Microsoft utility which mounts the ISO images files and makes them look like they are actual CDs in a CD drive.

I tried a couple times to install Visual Studio 2005 Professional. Each time I chose a custom install and only chose those components I really wanted. And each time the install bombed in the middle stating an install file was missing. I don’t know whether there is a bug in their install program, or whether the images they provided were bad, or if the virtual CD utility I was using was buggy. The end result was that I could not get the software to install.

Hey. We are talking about $700 worth of software here. I was not about to give up that easily. I decided instead to choose another download. This time I chose Visual Studio 2008 Professional. Even though we are not using that version at work yet, I figured it was close enough. The install went fine. Now I got a C++, C#, and Visual Basic development environment. I also installed all MSDN documentation I would even need. Thank a million Microsoft.

Value of Time

Our project has a requirements analysis team. Everyone on that team is new. So they don’t really have any domain knowledge. That’s a bad position to be in when you are supposed to work with the customer to figure out what changes they need to the system. I have been working on this system for a very long time. Once they found this out, the requirements team started needing a lot of my time. They schedule a lot of conference calls. In order to get my work done, I decided to just skip these calls.

The requirements team figured out that I was no longer dialing in to their conference calls. They are smart. They started asking me questions about the system as it pertains to some new requirements. That is fine as it does not disrupt my schedule too much. The problem is that they are unable to fully comprehend the answers I give them no matter how hard I try. Later they try to discuss the issues with the customer. Due to the lack of full understanding, I end up getting called in again.

I am not exactly sure how to deal with all of this. Everybody is new on the project at one time or another. But you only get to use the new card for the first couple months. After half a year, you really don’t have much of an excuse if you cannot do your job. I don’t like doing other people’s jobs for them. I have my own duties. And I expect everybody else on the team to pull their own weight. Otherwise why does the team need you?
In the past, I have dealt with people on the requirements team that lacked the domain knowledge to do their jobs. This was a typical problem. I guess I got a bit spoiled when the last team of requirements analysts was actually pretty good. They did not know everything about the system. However they were able to figure things out themselves. They also did not pretend to know a lot. That attitude went a long way to good relations with the development team. Now that they have left the project, it feels like we have to start all over again with the team.

I think I can fully appreciate a comment made by one the gurus in the customer organization. Some of our key developers left the project for greener pastures. The customer was disappointed in that he had finally got some developers up to speed. And then they left.

Home Network Help

Previously I had written about how a colleague and I used Pair Debugging to solve a tricky problem our customer was encountering. It took a while and some thinking outside of the box to figure out what was going on. The software was trying to FTP some data files. The FTP failed. The software responded by deleting the files. The missing files was the problem that was reported.

We had to duplicate and debug the problem in a development environment. At some point we got to the point where we believed that if the FTP worked, the software would work correctly. However we had to prove to ourselves that this was the case and there was not anything else wrong with the software. There was only one problem with this plan. We did not have an FTP server with which we could test. This is because our customer, due to security concerns, has shut down FTP all across the enterprise. That included our development servers.

Since we were already operating "outside of the box", I reasoned that we should be able to set up our own FTP Server to do these tests. I am sure we were violating one or more security rules by doing this. However our prime directive was to solve a high priority customer problem. Sometimes you need to bend the rules a little bit to get an important job done. I told my colleague to do a Google search for "free FTP Server for Windows XP". Luckily we got a result for Home Network Help. They had a site which walked us through enabling the FTP Server that comes with Windows XP Professional. Thanks guys for helping us out when we were in a bind.

Pair Debugging

When I got in this morning, I had a number of voice mail messages from my boss. Then I found that I had a bunch on my cell phone too. That did not bode well. So I called him up. He said another team member went to the customer site to resolve a high priority problem. And he told me he wanted me there too.

I packed up my things and headed to the customer site. Luckily I got one of the last parking spaces in the lot. A coworker signed me in the front door. I found the guy I needed to help. And we got down to business. A number of our customers saw me there and wanted help with whatever they were working with. I tried to give them a little bit, but I needed to get busy on this problem. I called my boss and he talked to the top dog in the customer organization. We needed to be left alone to solve their most critical problem.

Me and my buddy traced the code to the one line in a Korn shell script that should have been created a file. But after running out program, no file was to be found. We added some debug statements and were baffled. The line should have been creating a file. There were no permissions problems. We were sure the code was getting executed. I told my buddy that although you would not expect it, perhaps the file was getting created but getting deleted later. We traced the program all the way through to completion. Wouldn't you know it? When an FTP action fails, the programs deletes the file. We recommended that the customer get their FTP server up and running. The instructed us to just remove the line of code that deletes the file so they could get on with their work immediately. Pair programming, or more correctly pair debugging, actually seemed to work well with my and my buddy.

Effective Communications

Today I got assigned two trouble tickets to work. By the time I got into work, one of them got reassigned to another developer. Early bird gets the work. Slacker gets out of some work. Before lunch I decided I had better get hot on the problem that was still assigned to me.

I immediately called the customer that had reported the problem. She gave me the additional details I needed to investigate further. When it looked like this was a real problem, I called the customer again to clarify the expected scope of the solution. Then I looked at the code and the requirements.

It became clear to me that the code was working fine per the documented requirements. It just seemed like there was a problem because there were certain situations where the code was supposed to do weird things. Once again I called the customer back and explained all this. I followed up with an e-mail showing the requirements that were being followed.

The best moral of this story is that you need to be proactive and unafraid to pick up the phone and call a customer. There have been too many times where I have seen other developers missing some crucial information from a problem that a customer was having. And these developers would, for one reason or another, e-mail somebody else on the team and say they did not have the information they needed. Don't e-mail anybody. Pick up the phone and give the customer a call.

Maintenance Trouble

Our customer has a system acceptance test team. They reported a problem they found in our system. Our internal test team could not figure out what they were talking about. Unfortunately the responsibility to test our fix went to the newest guy on our internal test team. This guy had no chance of figuring out what to do. He sent me an email and left me a voice mail asking for help. All he could figure out was that the DBA Team had given them something to test. And he was truly clueless as for what to do next.

I don’t want to spend my life on this project doing other peoples jobs. So I knew I should not just give the tester all the answers. I recommended he start at the beginning and see if he could understand and duplicate the problem that the system acceptance team found. He said he tried running the applications, but could not see where they were finding any discrepancies. I proceeded to spend the next hour or two going over how the system works, how it is supposed to work, and how he could experience the problem in his own environment.

Previously I had assigned this problem to the DBA Team. Their yearly process had deleted some data that was required to be kept around for a couple years. The DBA Team lead told me he was never informed of that requirement. I responded that this was his official notification. As he got into working the solution for the problem, he realized that the fix to restore the data was a programming nightmare. I would have loved to have written a bunch of PL/SQL code to do the work. However I was tied up with other duties.

Later the DBA Team lead came up with a solution that would make his job easier. However it would require adding some new tables to the schema, and also some changes on the application development side of the house. He asked me to negotiate the new tables with the data architect on the project. So I called her up and let her know the situation, and where we wanted to go with the solution. She had some suggestions but was quite flexible. The I came in and wrote some new code in one of our PL/SQL packages. That was really fun. I updated a couple database triggers, and gave the code to the DBA to promote.

Here is what I have learned from this problem. You really need to know the business of the system to understand the complex issues. Most of the work in resolving system problems at this level does not involve sitting down and writing code. I have some other lessons learned about database design. However I will save those for a future post.

Fixing the Release

Last week we had a big software release due. The build went to our internal test by the end of the week. The test team found one application blowing up due to an Oracle exception. I told our team lead that we had better get to the bottom of this problem. The lead thought that this was just some discrepancy between the expected and actual database version. He thought a database change would resolve the problem. With that in mind, I left for the weekend.

It turns out the database change did not fix the problem. A bunch of people on the team got together late Friday night to try to figure out the problem. They left me some emails and voice messages. But by that time I was long gone. When I got back to work on Monday, we were in a state of emergency. My team lead said our company was losing money because we were late on the software release.

Apparently my team lead had spent the weekend trying to determine the cause of the problem. He still thought it had something to do with recent database changes. That did not seem encouraging. In software development you cannot think. You have to know. Thus we were nowhere with the problem. I got assigned the task to figure this out. I tried to duplicate the problem by running equivalent SQL against the database. But I had no luck.

I started applying my normal techniques. The next thing I tried was to run the application against an old version of the database. It had the same problems. At that point I eliminated any new database changes as the issue. Finally I started reviewing the history of the files that had the code that was crashing. A developer recently tried to fix a problem in that file. I rolled back those changes and found the source of the problem. At that point we were able to continue with the software release.

It turned out we were only one day late. That is still not a good thing. I did not lose any sleep over this problem. What could we have done to avoid this in the first place? We could have eliminated rushed last minute changes to the application. Or at least we could have run sufficient regression testing on the late changes. Better yet we should have done a better peer review on those changes. Let’s see if our project learns anything from these mistakes.

Stumbling Blocks

A production problem got assigned to me. The fix was due in three weeks. It is a bug that was hard to analyze. A big problem I had was existing commitments of my time. Currently I spend about a third of my time helping our requirements team. I am also supposed to spend half of my time helping another development team. Normally I also find myself spending a third of my team dealing with the emergency of the day. These tasks alone overbook me.

So if I were to do all that I am currently asked, I would be spending well over 40 hours a week meeting these duties. Now I am tasked with an extra difficult problem to solve. Once you look at it in this light, you can see why three weeks is not enough. If this were a trivial problem, I could knock it out real quick and there would not be any fuss. But this one I got assigned is no simple case.

Here is a review of the difficulties I encountered researching this problem. I could not build a debug or release version of the application that is having the problem. After some research I found that this project assumes that I also have the source code for another project on my system. Then I found the application was throwing all kinds of assertions when I ran it. Some more research showed the test data in my development environment was not good.

Finally I got to the heart of the problem. There were two set of nearly identical code. The production release was using one version of the code which had the problem. The development release was using another version of the code that had this problem fixed. So in this case, we need to make sure our configuration management is up to snuff to avoid more problems like this. However my real beef is that I was over committed by management.

How did I resolve my problem? I told my boss that I was over committed. And I pitched some alternatives to resolve the problem. I said that they should stop loaning me out to other projects when I am too busy with my own. Another tough decision was to stop assisting our requirements team. That will cause some long term pain as the requirements will be no good. I also need to find good ways to stop being tricked into spending time on requirements. But that is a story for a future post.

Software Engineering Stats

I have about some interesting statistics related to software engineering. The exact numbers are not of extreme importance. However the trends themselves were eye opening. Some numbers are not what you would think. I am going to go over some of them here to provoke thought.

The first is one I have heard before. A good programmer is 30 times better than a mediocre one. That seems massive. But I know there is a huge gap between the great programmers and the average ones. There is an ongoing debate over how great this difference is. This is just saying that it is huge (30 times equates to 3000 percent).

In comparison to great developers, great tools only provide a 5 to 30 percent increase in developer productivity. There is a lot of hype in how much productivity gains a tool provides. The root source of this hype is the companies that are selling the tools. That would be expected. What is not normally discussed is that there is an initial decrease in productivity when developers learn how to use a tool.

Software maintenance takes up 40 to 80 percent of the cost for a project. However this maintenance is not all just fixing bugs. As much as 60 percent of this maintenance is for enhancements to the code.

There are many causes for runaway projects. Unstable requirements are one of the causes. Another cause is optimistic estimation. You are going to be in trouble if the estimates are provided by the management or marketing teams. Schedule pressure in general can spell doom for a project.

Developers do conduct unit testing. However they normally cover 60 percent of the possible paths at most. Code reviews can eliminate 90 percent of software errors. Unfortunately rigorous reviews are skipped by most developers.

Finally, a high order language can achieve 90 percent of the speed reached by pure assembly language. This assumes that you turn optimization on in the compiler. The moral is that you do not need to step down to assembly language for most applications to achieve good speed. Moreover, you can get great gains by choosing the correct design.

Code Review Tool

Previously I had watched a video about the code review tool used at Google. Recently I read a review of a number of open source code review tools. It mentioned the one used at Google. This has prompted me to want to do some more research. We used to have a lot of code reviews on my project. Right now we don’t really do them any more. Part of the problem may be that it is a highly manual process. An easy to use review tool may help us get back on track.

The Google tool is called Rietveld. It was written by Guido Van Rossum. Of course Guido chose to write this in Python. You only need a Google account to use this tool. But it is implemented to work for the Subversion code repository. The review said that this was a bare bones tool. I would not mind using the Google tool. However we don’t use Subversion here on my project.

My reservations have been nullified by another tool called Code Striker. It was created by David Sitsky. This tool was written in Perl. It is also open source, and it is a web application. The beauty of this tool is that it works with Clearcase. That is what we use on our project. It allows you to review code diffs. Like any good diff tools, it highlights differences in color.

The goal of Code Striker was to minimize paperwork done with reviews. It also has the benefit of recording comments and issues in a database. I checked out some screen shot online at the Code Striker web site. It was funny to see some sample comments on a fictitious review. Some guys were saying things like “excellent work”. Maybe the comments were directed towards the Code Striker tool.

There are some other open source alternatives in the code review market. However I think I am going to propose we start using Code Striker on my project. I will let you know how it goes.

Developer DBAs

A manager I am starting to read more and more is Redmond News. It reminds me of the days where I was trying to keep up with the rapid pace of Microsoft technologies being released. The last issue discussed how normal developers are taking on database administrator roles. This is in addition to their normal development duties.

The article did agree that DBAs are still required in the enterprise. However there is a general drive to do more with less. Agile developers are taking over traditional DBA roles. In addition, past DBAs are starting to do development work. This is because there are fewer and fewer hard core DBA jobs out there.

Microsoft is released the next version of SQL Server in the first half of 2010. It is code named Kilimanjaro. There has been some talk in the SQL Server community about the desire to integrate SQL Server management tools into Visual Studio. SQL Server 2005 had the Visual Studio shell for SQL Server management.
On our own team, we used to have a number of full time DBAs. Now we are down to just one. And he is a subcontractor. There are a couple other individuals who pitch in with DBA work on an as-needed basis.

Definitive Answer

Our team is considering an upgrade to our reports technology. We are still stuck using Oracle Reports 6i. This is a client server version of Oracle Reports that is getting really old. We want to move to a web based approach. To determine the benefit of this upgrade, our manager wanted to know how many of the 70+ reports in our system are actually being used by the end user.

So our manager called a meeting of the technical staff. One DBA said he thought the application may be logging this somewhere in the database. The reports developer said that we might be writing this information in a database event log table. Nobody could say for sure. That did not put the manager in a good position to speak intelligently with the customer.

When I got back from lunch, my manager asked me if we had the ability to see what reports the users were actually running. I told him it would take me a few minutes. But by that time I could tell him for sure. I also thought we might audit this information. So I scanned all database tables until I found one that looked promising. I checked the code but found only one application actually used that audit table. Then I ran a test to verify that we are actually using that table for this application.

Then I went into each of the other applications, and found that one other application logs the reports run to a file on the local disk. The other application does not log the reports anywhere. I got back to my manager. He was excited to know that at least some of the reports get logged in the database. He had me run a query to find out exactly which reports were being run, and how often.

It is a bit disturbing that many members of the technical team are unable to provide authoritative answers to questions about our system. These are people that have been working on the project for a long time. I know that our project is a bit complicated because it is big. However that is no excuse. Perhaps it is time for a chalk talk with the team to let them know how to fully research questions such as this. The first step will be for me to have them read this blog post. If you are one of my team reading this, it is time to gain the ability to perform due diligence.

So So Workers

I read an article in the Wall Street Journal entitled Slacker Nation. It discussed the new trend found in younger workers in Japan. They did not want promotions. Therefore they were skipping career advancement exams which were required to move ahead. This is an unusual development. Previously the Japanese were known as workaholics and wanted to get ahead.

The stereotype of a promotion in Japan was a higher salary and better title. However this often came with a requirement to work late daily. These days the promotions may not mean that much more in salary. Regardless of pay, many young workers do not want to get the promotion. As such they are not putting in a full effort at work.

Companies are trying to figure out this newer generation. They are calling this new breed of worker the “so-so folks”. These individuals want to forget about goals. They value staying true to yourself. They think that not everybody needs to be a leader.

I find this thinking quite entertaining. It seems nice that somebody has figured out that killing yourself at work is just not worth it. When I first got out of school I worked hard. Then I started burning out and got laid off. At that point, I felt just like these Japanese workers. I played a lot of video games. And I did not feel the need to jump back into the work force.

Hey. I know some people want extra responsibilities and more money. That’s fine for them. But can’t a less ambitious individual do his time at work and then go home? There is more to life than working. Perhaps the sooner we figure this out, the better off we will be in the long run.

Staying Focused

The customer had asked us to make change to our system for next year. I had a lot of questions about the details of the change. I asked our requirements team a lot of questions. They did not have the answers, so they scheduled a meeting with the customer. A lot of people chimed in with their ideas before the meeting. Some people were trying to do some database design. I ignored most of this since we had not nailed down the customer requirements yet.

I was pleased when the technical advisor that works for the customer got on the conference call. He said we don’t have a lot of time in the schedule. And so he wanted us to make the changes, but to implement them all in code. He did not want any new database columns being created to hold intermediate values. He also did not want any existing database columns to be populated any differently.

During the meeting, the customer clarified the requirements very clearly. The technical advisor essentially made the impact to my team negligible. Only one back end developer needed to do the work. That’s the way project management should be. Without this guy’s direction, there would have been all kinds of database changes. And my team would have been required to implement them all. We are way too busy for any of that.

I am not sure what we are going to do when this customer technical advisor retires next year. Maybe we can hire him to work for our company. He consistently helps cut the overhead and wasted efforts from our software development schedule. He also has the clout to make his decisions stick.

Getting it Right

This past year our customer gave us a list of new feature they needed in the applications. I was initially not a part of gathering the requirements since I had not joined the company yet. However a requirements analysis team had produced a requirements document. There was one requirement that was suspiciously vague. I asked the requirements team a bunch of questions about what was needed. They could not answer any of the questions definitively.

The requirements team set up a conference call with the customer. I voiced my concerns that we did not understand one of the requirements at all. It was a dreaded one liner that did not mean anything to development. The customer quickly explained what they needed. I then tried to confirm my understanding. They concurred this is what they wanted. I then proceeded to design and implement a solution.

After we had delivered our changes to internal test, we scheduled a design review with the customer. I walked them through most of the design for the new changes. When we got to the design for the requirement that had previously been a mystery to me, the customer said we got it wrong. I said that I was sure we followed what we discussed at the conference call where we clarified the requirement. The customer was adamant that this was nothing like what they had previously discussed before I joined the project.

Once again we had to go back to the drawing board. This time around, I told the requirements team that we needed to write everything down in detail about the new requirements. And we needed to get the customer to agree in advance before we spend a lot of time in redesign and rewriting of the application. Another senior developer and myself worked closely with the requirements team. We thought up all kinds of questions that we then proceeded to hash out with the customer. That’s what requirements gathering and analysis are all about.

Currently we have a draft set of new requirements for this piece. Development is going to wait until these detailed requirements get customer sign off. Then we need to figure out how to schedule all these changes this late in the year before we go live in production. That story is one for a subsequent post.

Tough Bug

I have been assigned a trouble ticket that has been extremely difficult to diagnose. This problem had been left over from the previous contractor that maintained the software. They were unable to resolve the problem. Now we were on the hook to fix it. Like most problems, I attacked this one head on by checking the production audits of the problem items. At first I could not make head or tails of the situation.

In general I refuse to let any problems get the better of me. So I started looking more broadly at related items that we audited in production. Then I found something that was of interest. The user seemed to make an unexpected change in some of the data right before the problem happened. This was exciting. It seemed like this was the source of the problem. I tried to duplicate this problem in the development environment. I was disappointed to find out that I could not make the problem happen. Still I thought I had been on to something.

Sometimes you need to try out a couple things to get to the bottom of the matter. I decided to install the version of the application that the users were running. However I instead pointed the application to my development database. That’s when I first made the problem happen myself. Usually this is the point where a fix comes quickly. I was still perplexed why I could not make the problem happen when I used my debug version. Oh well. I tried a release version that I built. I still could not make this problem on anything other than our official release.

This troubled me. However like I said before, I am not a quitter. So I set up my virtual machine to be a build machine. I did a build just like our configuration management team does. Now I was going crazy, because my build would not make the problem happen. This is where I broke down and asked the configuration management team if I could borrow their build machine. Wouldn’t you know it? On their machine, the application uses a second copy of our code which is ever so slightly modified. The small modification was causing the problem.

Development is certainly partially responsible for this. Why do we have two copies of the same code, but with subtle differences that cause bugs? I plan to get to the bottom of this. However I am now at the point where the fix is trivial. That is a bonus because this was causing me to lose sleep this weekend. I crush bugs. They don’t crush me.

Review Time

I read a reader entry on one of my favorite software blogs. It was entitled “Review Time Again”. The author had a performance review. And despite having a stellar year, he got a rating of meets expectations. His question is what’s a developer to do in such situations? The great part about this post was the great feedback from other readers.

There was a general sense that performance reviews are themselves bad. They pit employees to compete which each other. Sometimes a little competition can be healthy. But if I need to do better than my coworkers to get that raise, why should I help anybody else. Right?

One source of feedback from another reader was to call attention to the fact that reviews are not about how much effort you have expended. They are or should be able what kind of results you have generated. Then again, another reader recommended you just threaten to quit your position. You might magically receive an “exceeds expectations” ratings sooner than you think.

There are many times in the Dilbert world where the pointy haired boss does you wrong. However in the case of performance reviews, the boss may not have very much flexibility. The manager may have to grade on the curve. The majority of the team is expected to have a rating of meets expectations. Maybe 10% can get an exceeds expectation, while another 10% must get a poor rating.

It is clear that one thing is for sure. Receiving a meets expectations rating hurts developer morale. And it could very well be the case you are working at a place where management is just bad. Sometimes you just need to move on. You often do get a pay raise when you jump ship. At other companies there is no set pay increase other than a cost of living each year.

The last idea, which is one that is implemented in my own company, is that of a 360 degree review. You don’t just get a review from your manager. You get it from your peers as well as people under you. I just received one of these. Or at least I was supposed to. I never got the results back from my manager. Am I unhappy? No. I was already told it would not impact my pay because I am new here. So there was really no point.

WinZip Protection

I need to do some tests that involve the output of our application builds. Normally configuration management does the builds. They use their own build machine which is under their control. I want to run a modified build that goes against our branch in source code control. I have my own virtual machine. So I decided to configure it to perform my own builds. This turned out to be a lot harder than I thought it would. I am still fighting with the configuration to get the build script to run to completion.

One requirement for the build machine is to have a certain version of WinZip, along with some configuration files used by WinZip. Luckily I had all the files needed for this version of WinZip. The build scripts assume that the correct version of WinZip is located at C:\Program Files\Winzip. I thought this would be no problem. I decided to move the existing copy of WinZip from there to another place. Then I would put the version needed by the build there. Even this simple task was hard to accomplish.

The system administrators of my workstation much have set it up to ensure that the version of WinZip that they expect is always located at C:\Program Files\WinZip. I tried to delete the files there. They were locked. I was able to temporarily rename the WinZip folder. However when I tried to copy my new version of WinZip to the right location, a WinZip install process immediately kicked off, restoring the wrong version of WinZip to that location.

This was very frustrating. I did not even know what secret process was protecting the installed version of WinZip. I found that I was able to cancel the automatic reinstall process each time it ran. Then I could sneak in the version that I wanted. However at every turn I had to keep canceling this job. This was very frustrating.

My next problem is that Ant is looking for a “tools.jar” file. It is missing. I think I have successfully installed Ant. This must be some Java install from Sun Microsystems. I tried installing the latest Sun JRE. I also tried to install the JDK. Any still keeps complaining about this. I am a Windows C++ developer. The build scripts were written by Java guys who have long since left the project. It looks like I am going to have to dig into this problem. I need my build.

Security Debacle

I had previously mentioned that a security audit showed we were logging sensitive information to a plain text file. My solution to this problem was to encrypt the file. I thought it would be fun to research and implement some encryption code. However the boss said we had to use the approved method from the customer. Furthermore we had to use their routines. This quickly turned into a boring task. So I put it aside. The boss told me to get our security guy to find out what method the client requires, and to obtain the necessary code to do it.

So I gave out security guy a call. He said he would get us the information in a couple days. The boss checked on the status with me. Apparently our security guy had made some promises to the customer based on his research. I told the boss I was still waiting to hear back from this dude. We both conference called him. He said he had researched the question and provided an email with his findings to development. My boss asked who exactly in development he sent the information to. He said he emailed it to Hugh. The only problem with that is there is no Hugh working in software development. I thought to myself “poor Hugh”.

The next day I got an email containing the information that the security guy had sent to Hugh. It seems our guy queried some security folks in our company instead of checking with the client. The assorted responses were precious. I could not make this stuff up if I tried. One person stated, “In college I wrote a paper on applications for database security, including encryption.” Another guy chimed in with “Why don’t you look at Oracle 11G? I believe inherent within the RDBMS there is encryption as part of the package.”

These responses put the boss over the edge. I spoke with our security guy. And I told him we have some Windows C++ applications that wrote log files in plain text on the local hard drive. We just need to write encrypted files. It has nothing to do with Oracle or databases. Then the boss made me call him back to tell him we needed the information today. That caused security guy to recommend that we use McAfee SafeBoot. Now I am no security expert. However I think SafeBoot is something you use to encrypt the whole hard disk, or a file that the user specifies. However I think it best to keep an open mind about this. You never know. I might get a few more laughs out of this exercise. In the end, I think my boss will cave in and let me roll my own encryption algorithms. At least I could implement an industry standard like 3DES or AES.

Video Game Developer

I read an article in the Los Angeles Times about video game developers. More and more young developers are getting into the video game arena. But is this a legitimate career? And is there any money in it? This is what many parents want to know.

Initially, most video game developers were hobbyists. This was the same for early personal computer adopters. Things have changed now. Big games require lots of people to produce them. The largest often have staffs of 100 people or more.

The video game development field is still young. It involves multiple parts such as game design, artwork, and audio. There is nonetheless a high placement success for graduates. Many students go to work at Electronic Arts.

However video game development is not all fun and games. For example, it is not as fun to thoroughly test a game as it is to play a finished game. There is also a lot of stress due to tight deadlines. There is some hope. The average salary for a game developer is over $73,000 a year. This is according to Game Developer Magazine.

I remember I once worked for a defense contractor. We had some good work to do. But it was not too exciting. Almost all the people on the development team dreamed of being game developers. They went to game development conferences. And they were coding games at home in their spare time.

Me I really do not idolize the game development world. Yes it may have a product which is fun. However I don’t think many people will get rich doing game development. Sometimes the hours can be tough. And I do not think there is a lot of maintenance in the game development arena. You are most likely going to be developing the next big game all the time. I will let the younger more enthusiastic crew deal with that.

Reports Mystery

We have a number of reports in our system. They are written with an old version of Oracle Reports Builder. We store them in RDF format. Recently a customer wanted some changes to the most complicated report we have. The reports developer coded up the changes. The modified report was included in our build to the test team. However it seemed that the test team was having a lot of problems. Our reports developer said it looked like the testers always got an old copy of the report. The reports developer decided the only solution was to turn the RDF report into an executable.

That solution screamed of doubt. Our team lead also said he could not imagine that this could be the best course of action. He asked me to assist to find out what was really going on. I said that we could turn the report into an executable. However I strongly encouraged that we understand the cause of the problem first. I am really to busy to spin my wheels on fixes that are guesses. I tasked the reports developer to get to a stage where the problem could be reproduced at will. I said that the testers could duplicate the problem. So this should not have been a difficult task.

Once the reports developer got a tester login, we were ready to move on. I said we needed to find out whether the report that got modified was actually getting called. I told the reports developer to look at a database table which logs the path of each report that we run. Sure enough, the updated report was getting called. I then had the developer look at the logic inside that report. It seemed this report was spawning off another worker report. The report developer jumped at the chance, and modified this report to use another technique to spawn the second worker report. That only caused more errors.

I instructed the reports developer to roll back those changes. Then we continued on. I asked the reports developer to determine whether the second worker report was getting called. Some logging code was added to this report. And yes indeed, the second worker report was getting called. Then I had the reports developer mimic the configuration of how that second worker report was getting called. With that configuration in place, I had the reports developer run the report using the Oracle Reports Builder tool. At once the problem was evident. This second report was expecting a certain table structure in a temporary table. This new structure was supposed to be controlled by an update to a stored procedure. It looks like that stored procedure change did not get promoted to the test database.

What is the moral of the story here? There are a few. The first is that you should not fix a bug that you can not reproduce. You should also not try to code a fix if you do not fully understand the root cause of the problem. Breaking down complex processes into smaller steps helps isolate problems. Old school error logging also helps you get a handle on things gone awry. I hope these lessons get learned with experience. Everybody has to learn sooner or later.

Sun Downloads

My customer told me to enter a request in a web application that their organization hosts. I tried to launch the application. However it required a newer version of the Java runtime environment. The application navigated me to the Sun Microsystems download page. There were a lot of downloads listed there. I went through the list and found the one I needed. It happened to be the J2SE Runtime Environment 5.0 Update 16. Or in other words, I needed the latest Sun JRE.

On the Sun web page, I chose my operating system. The only button I saw was one which started up a Sun downloaded application. So I clicked the button. It was strange that I only selected the latest JRE, but this downloader was trying to download more than that. On top of this, the downloader was aborting with an error. The status said that it could not create the file on my disk. This was disappointing. I had enough disk space. What was I to do?

I looked more carefully on Sun’s web page. There was actually a link for me to perform a normal download of the JRE without using Sun’s download application. It was then that I was able to download the JRE update. This whole process was a bit disturbing. Couldn’t the initial application from my customer just configure my system automatically? That would have been much easier for me. Sun is also partially to blame here. Why are these people trying to push their download application? I also strongly recommend they ensure the darn thing actually runs the first time I try it.

Hey. I am a developer. So you know I am going to eventually be able to figure out how to get the latest JRE downloaded and installed. I pity the poor average computer user who has to deal with this. Perhaps this is an opportunity for my company to get some more work with this client. It is time to turn these lemons into lemonade.

Fun With Security

Our application suite logs debug information to a file on the local disk drive. A security audit found that we were capturing sensitive information to this file. The fix was to stop doing that. However we also got dinged for allowing free access to this log file. I thought we could address this concern by encrypting the file. This gave me all kinds of ideas of interesting security encryption code to write.

The software development manager said we had to use the approved encryption method that the client advocated. Still being excited over this project, I poured through the mounds of security requirements that our client has. This was a dry exercise. I thought I found a section on encrypting files like ours. It pointed to another standard. I read a little bit about this standard. It seemed a bit involved. There was a lot of math used. But that’s ok.

I found an example of the algorithm written in the C programming language. However I still wanted to roll my own implementation. It is fun to write code. As a precaution, I went and told the software development manager what my intentions were. He told me we could not write our own implementation of the encryption. The security guys would not go for that. Instead I had to get the source code from our client.

This task keeps getting more and more boring by the second. Now my plan is to dump this onto a subcontractor. Perhaps instead I can work on a rule based engine that parses the audit information and suppresses the sensitive information. Hey. I am a company man. And although the overall mission is to solve problems for our customers, I got to have some fun while I am at it. Life is too short to do otherwise.

Commercial Example

Our project has been having numerous problems with our install packages. I downloaded what I consider to be a commercial software application this weekend. It seemed to have a moderately complex install process that went through without a hitch. Perhaps our project can learn something from this team’s choices.

Let’s start with a review of our project’s install process. We control our release builds with scripts written in Ant. The Ant calls Visual Studio to perform the compilation. Then the results are packaged up with Installshield Professional. The results are zipped up using Ant. And finally they are turned into an executable with Winzip Self Extractor.

The commercial software I tried out this weekend did a number of things at the time of install. It seemed to install and configure a Sybase database on my machine. It also installed an application server, as well as at least one Microsoft Windows service. That is a tall order. However it did so seamlessly.

I was able to see that this commercial software used IzPack. Now I have heard about other installer software. But I have never heard of this one. Apparently it is an open source package. It seems to support features we need such as an uninstaller and an unattended installer. This looks promising. You can’t beat the price of open source.

Perhaps I will give this package a try. Right now it is not my job to fix the install problems with our project. However in the end, the problems affect the whole team. And the boss comes to me frequently when other developers on the team can’t cut it. It is best to be prepared.

Self Extractor

Install problems continue to plague my project. The development lead has taken on the task of resolving these problems. I took over some of the work he was doing. The dev lead called me up and explained what he had found out. Our install executable is unpacking the compressed files. However it is then deleting those files before installation is complete. The install program is then unable to access the files needed for installation. He asked me to research whether it was possible to turn off the delete of temporary install files.

Our build process uses the Winzip Self Extractor program. First we create a zip file which contains all of the installation files. Then we use the Self Extractor program to change the zip file into a self extracting archive which automatically launches the setup program. Unfortunately I did not have my development machine with me. However I did have a computer. I tried to research the Self Extractor options using Google. But I did not have any luck.

I decided to download Winzip Self Extractor itself to see if there was any documentation that came with it. Winzip has an evaluation download copy for free. I did clarify that we were using the self extractor for software installations. This product always deletes the temporary files when it is done. However is does have a wait option that delays the temp file deletion until your setup program is done. By default it waits until setup is complete. Out setup program is called something else. So we need to pass the name of our program to the Self Extractor as an option.

Having done the necessary research, I passed the information on to the dev lead. He commented that we already use the wait option with Self Extractor. However it seemed like we only wait until our setup program removes the old version of our application before Winzip deletes the temporary files. That was surely not correct. I hope this latest bit of information helps our lead solve this problem. It is embarrassing when we release software to the customer and the darn thing just won’t install.

Laptop Lock

I currently have two laptops at work. One of them was provided by the client. It came with its own security cable and lock. This lock works fine. I dial up the combination and unlock it whenever I need to be on the go. The other laptop I have is the company one. At first I received no security cable or lock with it. So I kept it safe at home. Then I got a lock handed down through the chain to me. I proceeded to bring my laptop in to work. There was only one problem. The lock did not seem to work.
This lock was a Kensington lock. The documentation said it was supposed to work with Kensington security devices. The instructions said to insert the lock into the laptop, turn it 90 degrees, then change the combination. I did this. However I was able to rotate the lock 90 degrees back even when the combination had been changed. Not much security there.

I emailed the big boss and said this lock was not cutting it. He replied that nobody else was reporting any trouble. So the problem must be with me. I went to my direct manager. He could not figure out how to use the lock. He said he asked the customer to supply him with an extra good lock. Another guy on the team overheard this. He said the locks the company gives out are hard to work with. He proceeded to jam the lock into my laptop hard. Now it was stuck in place. I thanked him for his help.

It came to the day when I needed to go on travel. I still had a lot of work to do. So I needed to take my company laptop with me. Wouldn’t you know it? I could not unlock the lock that was jammed into the back of it. I tried and tried. Finally I gave up and left without my laptop. There is definitely something wrong with this situation. I should not have to fight with this lock. How much could a new working lock cost?

Yeah I can probably get somebody to get the lock off my laptop. But if I cannot do it, it defeats the purpose. Sometimes my company is very generous. At now it seems sometimes it is stingy. Unfortunately the security of my company laptop is my responsibility. If somebody steals it, my head is on the chopping block. I guess it is time to go and buy a new lock myself. I have some other ideas on how to deal with this. However there are just too risky.

Security Pie

About a year ago, our system had a security audit. We failed in quite a few areas. The client directed us to plug these security holes. One of the weak areas was logging. We write a log file to the local disk on the workstation. It contains sensitive information. And the file itself has no protection.

Our boss thought it might be prohibitive to plug these gaps. He thought we might have to go through every place in all our applications where we do logging. Then after analyzing all that code, we could find the places where we log sensitive data. Finally we could change that code. This would take forever since we log stuff all over the place in the code.

There is a simpler way I thought. So I pitched an idea. We could make some modifications to the common routine that write the log information to the disk. We code add a parser at this layer. The parser could scan the text being logged for sensitive information. It could then strip out this information prior to writing to the log file.
This technique has a number of benefits. First of all, this sounds like a fun project. More practically, it cuts down on the time for analysis versus the manual method. We actually don’t need the sensitive information in the logs. We are pretty sure we know most of the patterns where the log information contains sensitive info. It should be a snap to write a parser that strips this information out.

My other idea was to encrypt the file. The security folks would love that. This helps us lock down the logs pretty easily. I envisioned another fun software project where we could invent and implement a new encryption method. However the boss said the security guys would not go for that. Ok. We can take whatever algorithm they recommend, and implement that. Not as fun. But it is still easy to do.

There are many instances where the right outlook on a problem can reel the cost estimates in significantly without increasing risk. Isn’t that why they pay us designers the big bucks? I will let you know as soon as my company starts paying me them

Install Resolved

Our team has had some lingering problems with our installations not working. We have a total of four applications which each have their own install program. Two of them work all the time for me. The other two were recently just not working on my workstation. I would run the install executable. It would do nothing. There would be no error messages. The only thing I saw were some partial temporary files left behind.

This got escalated when our test team could not install all of our applications. The problem was especially troublesome because the installs worked sometimes. It was an intermittent problem. One developer thought it would work for them if they tried to run the install program twice. However that did nothing for me. And it seemed that it did not always work for that developer either.

I took a look at the Ant script that builds the install executable. However I did not see anything unique about the scripts for the problematic install programs. Another developer concluded that this must have something to do with the VPN software we use to connect from remote locations. I noted that I would encounter the problem both on my laptop, and also on my virtual machine. This was a double whammy.

The problem became ultra high priority when the customer confirmed they also could not install the problem. I tried to do a little more research. However I had my own normal work to attend to. One of our install script developers had left the project. So the onus fell on the other install script developer to figure out what was going on.

Finally the install guy came back with the cause. Apparently the install program was trying unsuccessfully to overwrite a system DLL that was sometimes in use by another program. This was shameful. Why are we trying to put this in the system directory? We have our own directory on the local hard drive. Furthermore, analysis showed that this DLL was not even required by our application.

At least we had somebody figure out this troubling problem. We are not out of the water just yet. There are some other strange install behaviors that people are starting to complain about. Our install guy better get busy again.

Business Travel

This week I am returning to some company training. The commute to the training location is evil because I live near a big city. So I told my company I needed a hotel room close to the training facility. To my surprise they said ok.

We are always busy at work. I need to be in training a couple days this week. Therefore I need to find a way to do a whole week’s worth of work in a couple days. Part of this involves saying no to requests for help. However I have to pick up the phone when the customer calls. I also have to do the work that the boss says needs to get done.

On the day that I need to travel, I still have a lot of work to do. That’s ok. I figure I can take my computer with me and do some work on the road. There is just one problem. I can’t get the darn security lock off my computer. And the guy that knew the trick and locked it in the first place is no longer on the project. Hey. They pay me to deal with situations like this. So I get another computer and bolt on travel.

I check into what I thought would be a luxury establishment. However this hotel seems to be all show and no substance. There is no fridge in my room. And the high speed Internet is no speed. What a crock. I guess I am going to have to go into the training facility in the middle of the night, or real early tomorrow to get on the network. Life sucks. Then you die.

My company considers this local travel I think. They will pay me to drive to the site. But I don’t get any money for food. Lucky for me I made a killing in the stock market this week. So I ordered up some room service. The sky is the limit. I might even leave a good tip for the delivery guy. Life throws you curve balls. And you need to roll with the punches. I have been beaten up this week.

SharePoint Communications

The development team updated the design docs for the latest changes to the applications. I did most of this work. Then we presented the updates to all the stakeholders. This presentation spanned two meetings. There was a lot of feedback.

My team lead made a comment that I needed to update the design docs based on stakeholder feedback. I figured I could do that. I did most of the writing, and also most of the speaking.

I called up my manager and asked him how he wanted me to handle the updates. He said to make the changes, hold a peer review internally, then publish the updates to SharePoint. I did all of this.

Before sending out the big message to everybody that an updates was available, I called my manager one last time. I told him I was done and was about to broadcast that the update was available. I am glad I called him. He said I was not supposed to publish the changes until a lot more things were done. I was about to grill him as to why he did not mention this before. However he is the boss. And it would also serve no good purpose.

So I deleted the docs off the SharePoint site. Then I passed my updated copies to him. I figure I am done here. The moral of the story is that it is important to confirm communications when you are about to share information with the world. The best way to conduct such communications is face to face. If that is not possible, you should pick up the phone and make a call.

Business Case

I recently had the need to hand out some business cards. However I have not received any from my company. You apparently have to order them. I thought this would not be a problem. I went to the online order form. After filling out most of the required information, I chose a custom title of “Rock Star Programmer”.

A message was generated that stated my order was cancelled. The reason was that you can only use your official company title on business cards. Otherwise you have to submit a business case to the anonymous Program Administrator. At first I was taken aback. Why do I have to go through the motions of justifying a minimal purchase such as business cards?

I decided to go online myself and pay for my own business cards. There are some sites which will print them for you for free. You just have to pay the postage which amounts to 6 bucks. I did so, and added some extra customization, bringing the total to ten dollars. This is a crock. I made sure the business cards I paid for did not have my company name on them. Hey. If I am paying for the darn things, they are going to only have my name.

In the end, I went back and composed a business case for the company to approve a custom title on my business cards. I figured I would give the company one last chance. Besides, I determined I would expense the company anyway for the ten bucks for my own cards if they would not approve my business case. The anonymous Program Administrator eventually gave me an approval for the custom title of Rock Star Programmer.

That was a good decision. I would hate for my opening line when handing out my cool new business cards to be, “I work for a big company. They would not let me choose my own title. So I had to buy my own cards.” The company has already made the decision to hire me. They should not scrutinize me for a $10 business card order. I am going to make the best decision for the business. If not, you should not have hired me in the first place. We are not in kindergarten any more.

Training Days

My company provides mandatory training for all new employees. As soon as I joined the company, I got busy on my project. I was able to delay the mandatory training. However last week it caught up with me. Human Resources informed me that I must attend this training. So I reported to the training facility. It is only 40 miles away from my house. However due to city traffic gridlock, I had to leave very early for there to be any chance of arriving on time.

The company provides breakfast and lunch. This is to make sure you eat and get back to training quickly. The food is ok. We learned about the history of the firm. I got a nice coffee table book describing the company over the years. The training was geared around how we market ourselves as a company to clients.

There were some funny harassment videos. I saw my vice president being interviewed in another video clip. It seemed that the table I sat at lost all the competitions with other new employees. So I walked out of the 2 day training class without any SWAG. There were a lot of retired military new hires in my firm. I also met some new people that work in the same building that I do.

I ran into an interesting fellow developer at the training session. He was in another class, but I spoke with him during one of our networking sessions. He is currently doing Linux and C development. However he was positioning himself to get into Microsoft Sharepoint development. This guy seemed to have a very broad background. This is in sharp contrast with my own specialization in one or two technical areas and one particular client. I did notice that this guy was a level below me in the organization.

Even though I have been with the company for almost half a year, I never got any business cards. This was a real disadvantage at the training. An important component of our company is making contacts through networking. I felt a little funny having to jot down my contact info on blank pieces of paper to hand out. I was impressed that most of the new hires seemed like top notch people. Perhaps this is because we are a management consulting company. And that draws a certain kind of individual to the firm.