A fellow developer coded a change to fix a problem reported by our users. He scheduled a formal release of the application involved. The request went through our normal process. Configuration management actually did the build. We have set up the process so that development gets to review the build before it gets shipped out. This is where I come into the picture. The developer asked me to do a peer review on the documentation for the release. So like any good reviewer, I pretended like I was the customer and followed all instructions in the release document. Found a couple clerical problems that could be fixed real quick. I also found a show-stopper: the install program did not install the application. So I informally used my veto power and held up the release.
When I went to discuss this main issue with the developer, he said he also found that the install program did not actually install the application on his computer. But he said it worked on another machine. At this point, the install appeared to only be working 1 out of every 3 times. I don't like these odds.
Due to the fact that this was a critical release, I volunteered to dig in and find out why the install program was not working. Our build scripts are written with Apache Ant. The scripts call Visual C++ to produce the EXEs and DLLs. The scripts also call Installshield to convert these into install files that we deploy. I think the scripts also use WinZip and/or Ant to turn the final set of files into one self-extracting executable that we deploy.
So I started by manually extracting the files. Everything looked good. The I ran the install in verbose mode. No errors seemed to come up. I tried closing out all other Windows apps before running the install. No luck. I tried uninstalling a lot of other applications first. Still no clues. Finally I ran the install in verbose mode one more time and looked for anything unusual. Even though the install went by fast, I saw some of the files it was unpacking and installing. These files were not part of the application that my coworker was trying to release. These were from another application in our suite. That was it.
Turns out somebody took the install executable from one of our other applications, renamed it to look like the latest release we needed, and passed it on to development. This in and of itself was a heinous act. But the real crime would have been if we allowed this release to go out even after crucial problems were detected during peer review. Luckily the our process saved us.
