Good-fast-cheap. Pick two.

I got invited to a meeting with the customer today. There was a problem in production. And the customer wanted answers. When it came time, I explained what was going on. Our new system used the same data source as the old one. But we were not doing the exact same transform of data. Therefore they saw different values in the reporting system.

I said we could fix this by updating the function that loads the data. We could apply this transform there. We could also write a script to correct all the wrong data loaded so far. Should close the case. The customer understood that. But then they asked if we could be proactive and prevent this type of problems.

Immediately I saw through the request. I said we could take the action to find out other times when the old system does a transform, and make sure we have those transforms present. However no, I could not do anything to make sure everything was being done 100% correctly. This is a huge system with non-trivial software involved. There are bugs in there.

Essentially I responded that no I could not do anything to make the reporting results 100% correct. It is just not feasible. This is an application of the good/fast/cheap management trifecta. They only get to pick 2 of those. The third one will suffer. In our scenario, the schedule is fixed, and the cost is fixed. Therefore you get the quality that you get. No way to increase that without more money or longer schedules.

Trouble with Technical Books

I self-register as a PL/SQL Oracle database developer with my company. So it came as no surprise when I got a targeted email from my company. It highlighted the book Mastering Oracle PL/SQL: Practical Solutions.

This book is ancient. It was published back in 2004. The funny thing is that most of the information is still relevant today. I checked on Amazon. There seems to be an update to the book from 2013. But it is selling used for $800. WTF?

I went through the introduction section of the book. It had me set up auto trace in my Oracle database instance. Also read up on sql trace and using tkprof. Not bad seeing how I have not even made it to Chapter 1 yet.

The only downside is that the book is made available to me through the Books 24x7 web site. My company has some sort of bulk license with them. The web user interface is painful. I had to go through each page. But the pages do not fit on my screen. So there is a lot of scrolling with their custom controls.

I tried to print out a copy of the book. I have the ability to print to a file. Unfortunately the Books 24x7 site intercepts print requests. All I got in the output was a page with copyright information on it. How useless.

So I went through the whole book, copy and pasting the pages into Word documents, one per chapter. Now I can read in my leisure. I could buy myself a copy from Amazon. New ones go for $19. I can also pick up a used copy there for under $8. Could be a good investment.

Simple Solution Wins

The customer found some problems in our reporting system. Whole parts of data were missing. Counts were not matching up. I was put on the task of resolving this. Had to dig a bit to found the root cause. There were a view that depended on another view that depended on some data being loaded. Many levels of indirection going on here.


To truly fix all the problems, it would require a lot of analysis. It might also slow down the system. Then I decided to look at this from another viewpoint. What was the true minimal cause of the problem? Once I found that, I attacked it with some solutions that were outside of the box.


I had to change just 2 lines of code. I love it when I get a solution like this, even if it is sort of a hack.

Finding a Way

I got word from my manager that a new problem would be resolved imminently. I had not even looked at the problem. So I dug in. I told my manager that it would not be done on time. The customer got angry. All of a sudden, I am approved to work all kinds of overtime.

I was tired. So at first I tried to run the 10 jobs that generate the data responsible for the problem. The first job took almost 3 hours before it died due to an out of memory error. I got the second job to run. The third job never finished and the database shut down for maintenance at night.

Ouch. A new deadline was created. Again I told my boss we would not be making the new deadline. Then I got offers for help from another team. I declined. They would only come, ask question, and literally waste time. Finally time to start working smart.

Instead of running these massive jobs, I decided to inspect the code. I figured I should just study each function until I was 90% sure it wasn't a problem. Then I would skip onto the next one. I got to the 9th of 10 function. This one had some trouble. This was it.

This was the sort of the problem where the research is the majority of the work. I coded up a solution, tested it, and passed it on. It is still going to be a tight schedule with the configuration management and test people. But I got it done.

Be Brave to Get Work Done

I was woken up this morning from a call from work. Not a good sign. Apparently the customer found a potential problem in our delivery. I got on a conference call to sort the thing out. Yes. There was a problem. Our whole team went into high gear to get this problem done.

First we brought in another senior developer. The problem and our proposed solution was discussed a second time around. Our team lead asked everyone if they understood what needed to be done. Everyone concurred. We split the work up 5 ways because there were 5 of us. Then we broke to get it done.

Immediately I get a call from someone on our team. He was asking what he needed to do. WTF? I told him if he was not clear, he should have spoken up at the meeting. No need to save face and waste my time. I could not do his work and mine, because I got lions portion of the share to do.

I gave him a quick synopsis and told him to get it done. If he found that he could not do it or it was impossible to do, he would have to convene a meeting with the whole team so we could regroup. I got a couple more calls and emails from him. Eventually he completed his task. Of course he was the last one done. So he did not look good anyway in the end.

Come on people. Put your pride aside and step up. Don't try to use someone else as a crutch. You are hurting yourself and the whole team.

Quality Assurance

I work in the back end on my current project. It is a big database reporting project. My role is to provide the structure for new tables, write code to load those tables, and sometimes write the code to create database views. You would think that would be a straight forward job. Not so.
 
We have a lot of environments for our system. I work in a personal schema. We have another schema that I share with other developers. I consider that an integration schema. Then there is a separate schema for our internal testers. Plus one for our customer acceptance testers. We have a performance environment with a lot of data. Then there is production.
 
Recently there have been numerous problems deploying changes to these environments. The process is for me to check code into Clearcase source code control. Then I submit a ticket in ClearQuest to request deployment of my changes. After me there is a long chain of people involved before my changes get deployed in the assorted environments.
 
The deployment problems culminated in a specific delivery we failed three times in a row. Some managers tried to understand what in the world is going on. Their conclusion was that developers such as myself need to guarantee that the deployments are done correctly.
 
Now my life involves emails where somebody will run a massive deployment which includes my changes, produce a huge log or logs. Then they ask me if the deployment was done correctly. LOL wut? This got old the first time I had to pour through the logs. Not only that, I got to go to the target environment and actually check if my changes are in there.
 
I don’t want to spend my days doing quality assurance for a bunch of people who work on the deployments and just ask me if they got it right. I want to be doing development. Perhaps the right way to look at this is as an opportunity to automate the verification of the deployments. I figured we could use some kind of embedded versioning to identify what is deployed. Or I could buld some smarts into a script that checks for the specific changes made.

Use the Requirements Already

I am working on a release at work. Initially we were supposed to replicate some bunch of database tables that the customer had in an old system. We did a quick dive into what was involved and came up with a rough level of effort. That translated into a schedule. The customer said the schedule as too long. Okay. Then if the schedule was fixed, we can up with a subset of the task we could finish on time.

Right no we are getting towards the end of the cycle for this release. Internal testers are trying to verify the work we did. One tester is sort of a programmer. She does automated testing with some Java programs she writes. Recently she sent out an email asking what she should test. LOL wut? I responded and directed her towards the requirements. Seems simple enough.

Later the test called me up. She said she wanted to send me a document. And she wanted me to mark it up and tell her what to test. Nope. First of all that's not my job. And second of all, she should not just test what I tell her to do. She should test what is required of us for this release. I had to say that a couple times. Use the requirements document. It is there for a reason.

I ended the call with the tester by telling her if she does not have the requirements document, call up our requirements analyst. She does not bite. Very simple. I worry that our testers do not know what to test. I don't want to be part of the problem by leading them astray. For developers who encounter a software problem, I say use the source. For everyone else, use the requirements.