Be Brave to Get Work Done

I was woken up this morning from a call from work. Not a good sign. Apparently the customer found a potential problem in our delivery. I got on a conference call to sort the thing out. Yes. There was a problem. Our whole team went into high gear to get this problem done.

First we brought in another senior developer. The problem and our proposed solution was discussed a second time around. Our team lead asked everyone if they understood what needed to be done. Everyone concurred. We split the work up 5 ways because there were 5 of us. Then we broke to get it done.

Immediately I get a call from someone on our team. He was asking what he needed to do. WTF? I told him if he was not clear, he should have spoken up at the meeting. No need to save face and waste my time. I could not do his work and mine, because I got lions portion of the share to do.

I gave him a quick synopsis and told him to get it done. If he found that he could not do it or it was impossible to do, he would have to convene a meeting with the whole team so we could regroup. I got a couple more calls and emails from him. Eventually he completed his task. Of course he was the last one done. So he did not look good anyway in the end.

Come on people. Put your pride aside and step up. Don't try to use someone else as a crutch. You are hurting yourself and the whole team.

Quality Assurance

I work in the back end on my current project. It is a big database reporting project. My role is to provide the structure for new tables, write code to load those tables, and sometimes write the code to create database views. You would think that would be a straight forward job. Not so.
We have a lot of environments for our system. I work in a personal schema. We have another schema that I share with other developers. I consider that an integration schema. Then there is a separate schema for our internal testers. Plus one for our customer acceptance testers. We have a performance environment with a lot of data. Then there is production.
Recently there have been numerous problems deploying changes to these environments. The process is for me to check code into Clearcase source code control. Then I submit a ticket in ClearQuest to request deployment of my changes. After me there is a long chain of people involved before my changes get deployed in the assorted environments.
The deployment problems culminated in a specific delivery we failed three times in a row. Some managers tried to understand what in the world is going on. Their conclusion was that developers such as myself need to guarantee that the deployments are done correctly.
Now my life involves emails where somebody will run a massive deployment which includes my changes, produce a huge log or logs. Then they ask me if the deployment was done correctly. LOL wut? This got old the first time I had to pour through the logs. Not only that, I got to go to the target environment and actually check if my changes are in there.
I don’t want to spend my days doing quality assurance for a bunch of people who work on the deployments and just ask me if they got it right. I want to be doing development. Perhaps the right way to look at this is as an opportunity to automate the verification of the deployments. I figured we could use some kind of embedded versioning to identify what is deployed. Or I could buld some smarts into a script that checks for the specific changes made.

Use the Requirements Already

I am working on a release at work. Initially we were supposed to replicate some bunch of database tables that the customer had in an old system. We did a quick dive into what was involved and came up with a rough level of effort. That translated into a schedule. The customer said the schedule as too long. Okay. Then if the schedule was fixed, we can up with a subset of the task we could finish on time.

Right no we are getting towards the end of the cycle for this release. Internal testers are trying to verify the work we did. One tester is sort of a programmer. She does automated testing with some Java programs she writes. Recently she sent out an email asking what she should test. LOL wut? I responded and directed her towards the requirements. Seems simple enough.

Later the test called me up. She said she wanted to send me a document. And she wanted me to mark it up and tell her what to test. Nope. First of all that's not my job. And second of all, she should not just test what I tell her to do. She should test what is required of us for this release. I had to say that a couple times. Use the requirements document. It is there for a reason.

I ended the call with the tester by telling her if she does not have the requirements document, call up our requirements analyst. She does not bite. Very simple. I worry that our testers do not know what to test. I don't want to be part of the problem by leading them astray. For developers who encounter a software problem, I say use the source. For everyone else, use the requirements.

Check Your Subroutines

We are delivering our latest release to internal test today. Had a code review yesterday. Many issues were found. We are fixing the highest priority problems. Before shipping to test, thought we would do one final smoke test. Ran all the functions. Checked to see if data was loaded. Most looked good. One target table was not getting any records.

Very odd. Called the function in the same way. The template for the function matched the others we were using. The routine even spit out some debug messages which confirmed each of the select statements found a lot of records. Me and my buddy ran this thing a total of four times.

The clue that got me on track to a solution was that a table with the same name in another database schema kept getting more records. How was that possible? We were setting up the PostgreSQL search path explicitly at the top of the function. But by the end of the function, it seemed the search path was changed.

Initially I thought maybe some weirdness was going on because the DBAs recently dropped the tables and recreated the tables in our target schema. However that DDL should have been instantaneous. It was just suspicious because they cloned the tables from a prior schema (the one now getting all the data).

Finally I had an epiphany. Maybe our big routine was calling some helper functions that were doing something naughty. I searched for the word "perform" which is how we call subroutines. Bam. There was a helper function resetting the search path to the wrong database schema. That was a very tough bug indeed.

Struggling With the Calendar Code

Turns out we need to work all weekend on fixing bugs before the customer will even consider accepting our system for production use. I got assigned a ticket to analyze. At first I had some trouble because they shut down the databases for maintenance. Damn. I asked them to cancel the shutdown so we could get to work.

Eventually the databases come up. I verify that there are some formatting issues with the data. And there is a vast shortage of data. Next I am told to fix all the problems I found. The formatting changes were not too hard. The code is just a view on top of a table which acts as a calendar.

Next I tackle the lack of data problem. Well the underlying table is just missing a lot of rows. Can I just generate some new rows? Not easily. There are all kinds of weird values in this table. My boss asks if the problem is fixed yet. Nope. He asks if there is anyone who can help. Yeah - the person who created the underlying data.

I call up the developer. She is out and about. But she shares with me a grand secret. The method to generate the data is a function in the database. Jackpot. Of course the thing does not work. But at least I have a starting point.

So I drill down to see the table that drives her generation function. It requires only a couple key fields to make the whole process go. Figuring out those key fields is a major undertaking. I dig up a manual I have that is 15 years old. It helps me figuring out some seed data. I eventually write my own function to populate the key fields.

Eventually I got a pretty good function to generate the very low level data. Then I make some calls to the function that takes it up one level. Then I call my function to create a view to wrap the table and expose stuff in the correct format. My counts are close but not exact. I trace it down to some duplicate records. Luckily my view can filter out this duplicate nonsense.

It is past midnight when I am done. I email out the team, letting them know that I am sticking some new data in these tables. I also share that I found some dup data down in there. Get a message from my old boss. He told me the function I found is buggy, and does not work for future dates. FML.

High Paying Development Positions

I looked around at some job postings the other night. Saw a few that paid really well. I don't think I am ready for those specific jobs. But it gives me something to shoot for. I am going to highlight a couple of them. How much can you make? And what skills do you need for them?

Let's start with the CNO Engineering position. Pay tops out at $185k per year. There is a long list of requirements that cross over to the other jobs as well. You need to know C, C++, and Python programming languages. Should know TCP/IP networking. You must have experience with hard core programming such as kernel development and/or device drivers on Windows. You should know about reverse engineering and exploitation. And of course you need a Top Secret/SCI security clearance.

Next up is the Reverse Engineering position. Pay tops out at $190k per year. You should have the CNO Engineering position requirements. Plus you also should have experience developing for the Linux operating system. Also need to know about vulnerabilities.

The last job I want to talk about is the Software Reverse Engineering position. Pay tops out at $210k per year for this one. You need the CNO and Reverse Engineering skillsets. You should also know about architecture and be able to lead teams. You should know estimation and assessments. You should be able to work on proposals. You should be a subject matter expert. You should know assembly plus disassembling. Finally you got to have experience with firmware and embedded systems.

These all sound like jobs with excellent compensation. More than your average software development position. Got to have that security clearance. Plus there seems to be a theme of cyber security style skills needed. Time to crack the books, study up, and get a promotion.

Disparate Systems

On my current project at work, I have a team lead and a project manager. I take most of my direction from my team lead. He manages the list of trouble tickets on our system. And he hands them out to members on our team. When we fix the issues, we give him the updated source code. He then works some magic to get those changes deployed. He keeps on top of this. So it works well for our team.

Recently the project manager has been unhappy because trouble tickets are not being updated in our trouble ticket tracking system. After a while, I told our project manager that our team is really not using the tracking system at all. The team lead is managing this stuff on his own outside of the tracking system. Thus, there is conflict.

In theory, the tracking system is the way to go. But we just need everyone on board. The main guy that needs to follow the process is our team lead. Without his buy in, none of this is going to work. It will take a time to get all the older tickets sorted out in the tracking system. Unfortunately, thing are crazy at work. So who has the time for that?