Check Your Subroutines

We are delivering our latest release to internal test today. Had a code review yesterday. Many issues were found. We are fixing the highest priority problems. Before shipping to test, thought we would do one final smoke test. Ran all the functions. Checked to see if data was loaded. Most looked good. One target table was not getting any records.

Very odd. Called the function in the same way. The template for the function matched the others we were using. The routine even spit out some debug messages which confirmed each of the select statements found a lot of records. Me and my buddy ran this thing a total of four times.

The clue that got me on track to a solution was that a table with the same name in another database schema kept getting more records. How was that possible? We were setting up the PostgreSQL search path explicitly at the top of the function. But by the end of the function, it seemed the search path was changed.

Initially I thought maybe some weirdness was going on because the DBAs recently dropped the tables and recreated the tables in our target schema. However that DDL should have been instantaneous. It was just suspicious because they cloned the tables from a prior schema (the one now getting all the data).

Finally I had an epiphany. Maybe our big routine was calling some helper functions that were doing something naughty. I searched for the word "perform" which is how we call subroutines. Bam. There was a helper function resetting the search path to the wrong database schema. That was a very tough bug indeed.

Struggling With the Calendar Code

Turns out we need to work all weekend on fixing bugs before the customer will even consider accepting our system for production use. I got assigned a ticket to analyze. At first I had some trouble because they shut down the databases for maintenance. Damn. I asked them to cancel the shutdown so we could get to work.

Eventually the databases come up. I verify that there are some formatting issues with the data. And there is a vast shortage of data. Next I am told to fix all the problems I found. The formatting changes were not too hard. The code is just a view on top of a table which acts as a calendar.

Next I tackle the lack of data problem. Well the underlying table is just missing a lot of rows. Can I just generate some new rows? Not easily. There are all kinds of weird values in this table. My boss asks if the problem is fixed yet. Nope. He asks if there is anyone who can help. Yeah - the person who created the underlying data.

I call up the developer. She is out and about. But she shares with me a grand secret. The method to generate the data is a function in the database. Jackpot. Of course the thing does not work. But at least I have a starting point.

So I drill down to see the table that drives her generation function. It requires only a couple key fields to make the whole process go. Figuring out those key fields is a major undertaking. I dig up a manual I have that is 15 years old. It helps me figuring out some seed data. I eventually write my own function to populate the key fields.

Eventually I got a pretty good function to generate the very low level data. Then I make some calls to the function that takes it up one level. Then I call my function to create a view to wrap the table and expose stuff in the correct format. My counts are close but not exact. I trace it down to some duplicate records. Luckily my view can filter out this duplicate nonsense.

It is past midnight when I am done. I email out the team, letting them know that I am sticking some new data in these tables. I also share that I found some dup data down in there. Get a message from my old boss. He told me the function I found is buggy, and does not work for future dates. FML.

High Paying Development Positions

I looked around at some job postings the other night. Saw a few that paid really well. I don't think I am ready for those specific jobs. But it gives me something to shoot for. I am going to highlight a couple of them. How much can you make? And what skills do you need for them?

Let's start with the CNO Engineering position. Pay tops out at $185k per year. There is a long list of requirements that cross over to the other jobs as well. You need to know C, C++, and Python programming languages. Should know TCP/IP networking. You must have experience with hard core programming such as kernel development and/or device drivers on Windows. You should know about reverse engineering and exploitation. And of course you need a Top Secret/SCI security clearance.

Next up is the Reverse Engineering position. Pay tops out at $190k per year. You should have the CNO Engineering position requirements. Plus you also should have experience developing for the Linux operating system. Also need to know about vulnerabilities.

The last job I want to talk about is the Software Reverse Engineering position. Pay tops out at $210k per year for this one. You need the CNO and Reverse Engineering skillsets. You should also know about architecture and be able to lead teams. You should know estimation and assessments. You should be able to work on proposals. You should be a subject matter expert. You should know assembly plus disassembling. Finally you got to have experience with firmware and embedded systems.

These all sound like jobs with excellent compensation. More than your average software development position. Got to have that security clearance. Plus there seems to be a theme of cyber security style skills needed. Time to crack the books, study up, and get a promotion.

Disparate Systems

On my current project at work, I have a team lead and a project manager. I take most of my direction from my team lead. He manages the list of trouble tickets on our system. And he hands them out to members on our team. When we fix the issues, we give him the updated source code. He then works some magic to get those changes deployed. He keeps on top of this. So it works well for our team.

Recently the project manager has been unhappy because trouble tickets are not being updated in our trouble ticket tracking system. After a while, I told our project manager that our team is really not using the tracking system at all. The team lead is managing this stuff on his own outside of the tracking system. Thus, there is conflict.

In theory, the tracking system is the way to go. But we just need everyone on board. The main guy that needs to follow the process is our team lead. Without his buy in, none of this is going to work. It will take a time to get all the older tickets sorted out in the tracking system. Unfortunately, thing are crazy at work. So who has the time for that?

Teamwork Challenges

One of the newer guys on the team asked our project manager for some info. He needed to figure out how to get his scripts logging in to run unattended. The project manager asked our team lead to schedule a meeting to sort this out.

We had a meeting scheduled this morning. The new guy was not at work when it started, so we had to reschedule. When the meeting did get under way, we were not sure how to get this guy's script running. So we phoned an expert. Actually it was just some other guy on our team who has experience with it.

So we got through most of the questions. I thought we had a plan. The team lead said another new guy would help out the guy writing the script. Script writer got real salty with that setup. At first he said the newer guy could just take over the job. Then he complained loudly that he had done a lot of work, and just needed some info to get his scripts to run.

Later my team lead and I wondered what the heck went on? We usually work as a team and pair up on tasks all the time. Personally I like it when someone else can take a piece of the pie and lighten my load. There is definitely something wrong going down. Might be that the guy is insecure. I think he has some reason to be. Obviously he needs assistance a lot of the time. We will see how this plays out.

Save by the Log

I had promised to finish my current task within two days. First day I paced myself and did pretty good. As soon as I got in the next day, my boss called me up. We needed to debug some code that got sent out but had problems. It took about an hour. Okay I guess I could absorb that and still make my deadline.

Next there was a session scheduled to fix some data issues with our latest releases. It was a conference call with a lot of my teammates on it. So I tried to work on my stuff while the call was going on. Of courses a number of issues came up during the call where I was the only guy who knew how some data was being set. So I could only half-multitask. The call went on all morning.

In the afternoon I decided to cut lunch short to try to make my deadline. Things were going well. Then I realized the work I tried to squeeze in the morning was done wrong. Had to go back and fix that stuff. Deadline approaching. As I am getting near the home stretch of my task, my network connection goes down. Darn.

I reconnect but my database tool aborts. Luckily I always save all my work to a daily log. I just restarted my database tool, loaded up my log with all my commands, and picked right back up where I left off. What is the moral of this story? I think I had better pad my estimates to factor all these interruptions in there.

Non-Technical Challenges on the Job

I got my next assignment from my team lead. Needed to figure out the code to create a bunch of database tables. Then I need to write the code to put data in those tables. The good news is that we have the blueprint of the tables from another system. And somebody has written some database views that we are converting into tables. So most of the work is just the details.

Thought I would actually have some fun with this task. Then I got the catch. I had to split up the work and assign some of it to another developer on the team. This other guy has hours to work on our project. And he needs some work. The reason I figured this was a setup was because I have worked with the guy before. He needs a lot of hand holding.

Now I don't mind helping out a junior developer or even a new developer. But the guy I needed to work with is supposed to be a senior guy. And he is not new any more. So the management folks figure with his help, we should be able to get things done more quickly. On paper that is fine. In reality, not so good.

So I carved out a little over 25% of the work to give to the other guy. He had some questions early on. I tried to explain the answers to him. The he went loose. When he got to the one piece that was a bit more complicated, he was lost. I had to spend a bunch of time going over it with him in detail. It
took a few sessions. But I think it got through.

The last part of the task was to roll up both our work into one package. It all needs to go out into one release. So I did what I would do if I were developing the work myself. I slowly added the other guy's stuff into mine one small piece at a time. And let's just say there was pain as I debugged the errors and got the stuff to work. How can I get out of this type of setup? Maybe that is the point. I got to figure out how to work in this challenging situation.