Be Brave to Get Work Done

I was woken up this morning from a call from work. Not a good sign. Apparently the customer found a potential problem in our delivery. I got on a conference call to sort the thing out. Yes. There was a problem. Our whole team went into high gear to get this problem done.

First we brought in another senior developer. The problem and our proposed solution was discussed a second time around. Our team lead asked everyone if they understood what needed to be done. Everyone concurred. We split the work up 5 ways because there were 5 of us. Then we broke to get it done.

Immediately I get a call from someone on our team. He was asking what he needed to do. WTF? I told him if he was not clear, he should have spoken up at the meeting. No need to save face and waste my time. I could not do his work and mine, because I got lions portion of the share to do.

I gave him a quick synopsis and told him to get it done. If he found that he could not do it or it was impossible to do, he would have to convene a meeting with the whole team so we could regroup. I got a couple more calls and emails from him. Eventually he completed his task. Of course he was the last one done. So he did not look good anyway in the end.

Come on people. Put your pride aside and step up. Don't try to use someone else as a crutch. You are hurting yourself and the whole team.

Quality Assurance

I work in the back end on my current project. It is a big database reporting project. My role is to provide the structure for new tables, write code to load those tables, and sometimes write the code to create database views. You would think that would be a straight forward job. Not so.

 

We have a lot of environments for our system. I work in a personal schema. We have another schema that I share with other developers. I consider that an integration schema. Then there is a separate schema for our internal testers. Plus one for our customer acceptance testers. We have a performance environment with a lot of data. Then there is production.

 

Recently there have been numerous problems deploying changes to these environments. The process is for me to check code into Clearcase source code control. Then I submit a ticket in ClearQuest to request deployment of my changes. After me there is a long chain of people involved before my changes get deployed in the assorted environments.

 

The deployment problems culminated in a specific delivery we failed three times in a row. Some managers tried to understand what in the world is going on. Their conclusion was that developers such as myself need to guarantee that the deployments are done correctly.

 

Now my life involves emails where somebody will run a massive deployment which includes my changes, produce a huge log or logs. Then they ask me if the deployment was done correctly. LOL wut? This got old the first time I had to pour through the logs. Not only that, I got to go to the target environment and actually check if my changes are in there.

 

I don’t want to spend my days doing quality assurance for a bunch of people who work on the deployments and just ask me if they got it right. I want to be doing development. Perhaps the right way to look at this is as an opportunity to automate the verification of the deployments. I figured we could use some kind of embedded versioning to identify what is deployed. Or I could buld some smarts into a script that checks for the specific changes made.

Use the Requirements Already

I am working on a release at work. Initially we were supposed to replicate some bunch of database tables that the customer had in an old system. We did a quick dive into what was involved and came up with a rough level of effort. That translated into a schedule. The customer said the schedule as too long. Okay. Then if the schedule was fixed, we can up with a subset of the task we could finish on time.

Right no we are getting towards the end of the cycle for this release. Internal testers are trying to verify the work we did. One tester is sort of a programmer. She does automated testing with some Java programs she writes. Recently she sent out an email asking what she should test. LOL wut? I responded and directed her towards the requirements. Seems simple enough.

Later the test called me up. She said she wanted to send me a document. And she wanted me to mark it up and tell her what to test. Nope. First of all that's not my job. And second of all, she should not just test what I tell her to do. She should test what is required of us for this release. I had to say that a couple times. Use the requirements document. It is there for a reason.

I ended the call with the tester by telling her if she does not have the requirements document, call up our requirements analyst. She does not bite. Very simple. I worry that our testers do not know what to test. I don't want to be part of the problem by leading them astray. For developers who encounter a software problem, I say use the source. For everyone else, use the requirements.

Check Your Subroutines

We are delivering our latest release to internal test today. Had a code review yesterday. Many issues were found. We are fixing the highest priority problems. Before shipping to test, thought we would do one final smoke test. Ran all the functions. Checked to see if data was loaded. Most looked good. One target table was not getting any records.

Very odd. Called the function in the same way. The template for the function matched the others we were using. The routine even spit out some debug messages which confirmed each of the select statements found a lot of records. Me and my buddy ran this thing a total of four times.

The clue that got me on track to a solution was that a table with the same name in another database schema kept getting more records. How was that possible? We were setting up the PostgreSQL search path explicitly at the top of the function. But by the end of the function, it seemed the search path was changed.

Initially I thought maybe some weirdness was going on because the DBAs recently dropped the tables and recreated the tables in our target schema. However that DDL should have been instantaneous. It was just suspicious because they cloned the tables from a prior schema (the one now getting all the data).

Finally I had an epiphany. Maybe our big routine was calling some helper functions that were doing something naughty. I searched for the word "perform" which is how we call subroutines. Bam. There was a helper function resetting the search path to the wrong database schema. That was a very tough bug indeed.

Struggling With the Calendar Code

Turns out we need to work all weekend on fixing bugs before the customer will even consider accepting our system for production use. I got assigned a ticket to analyze. At first I had some trouble because they shut down the databases for maintenance. Damn. I asked them to cancel the shutdown so we could get to work.

Eventually the databases come up. I verify that there are some formatting issues with the data. And there is a vast shortage of data. Next I am told to fix all the problems I found. The formatting changes were not too hard. The code is just a view on top of a table which acts as a calendar.

Next I tackle the lack of data problem. Well the underlying table is just missing a lot of rows. Can I just generate some new rows? Not easily. There are all kinds of weird values in this table. My boss asks if the problem is fixed yet. Nope. He asks if there is anyone who can help. Yeah - the person who created the underlying data.

I call up the developer. She is out and about. But she shares with me a grand secret. The method to generate the data is a function in the database. Jackpot. Of course the thing does not work. But at least I have a starting point.

So I drill down to see the table that drives her generation function. It requires only a couple key fields to make the whole process go. Figuring out those key fields is a major undertaking. I dig up a manual I have that is 15 years old. It helps me figuring out some seed data. I eventually write my own function to populate the key fields.

Eventually I got a pretty good function to generate the very low level data. Then I make some calls to the function that takes it up one level. Then I call my function to create a view to wrap the table and expose stuff in the correct format. My counts are close but not exact. I trace it down to some duplicate records. Luckily my view can filter out this duplicate nonsense.

It is past midnight when I am done. I email out the team, letting them know that I am sticking some new data in these tables. I also share that I found some dup data down in there. Get a message from my old boss. He told me the function I found is buggy, and does not work for future dates. FML.

High Paying Development Positions

I looked around at some job postings the other night. Saw a few that paid really well. I don't think I am ready for those specific jobs. But it gives me something to shoot for. I am going to highlight a couple of them. How much can you make? And what skills do you need for them?

Let's start with the CNO Engineering position. Pay tops out at $185k per year. There is a long list of requirements that cross over to the other jobs as well. You need to know C, C++, and Python programming languages. Should know TCP/IP networking. You must have experience with hard core programming such as kernel development and/or device drivers on Windows. You should know about reverse engineering and exploitation. And of course you need a Top Secret/SCI security clearance.

Next up is the Reverse Engineering position. Pay tops out at $190k per year. You should have the CNO Engineering position requirements. Plus you also should have experience developing for the Linux operating system. Also need to know about vulnerabilities.

The last job I want to talk about is the Software Reverse Engineering position. Pay tops out at $210k per year for this one. You need the CNO and Reverse Engineering skillsets. You should also know about architecture and be able to lead teams. You should know estimation and assessments. You should be able to work on proposals. You should be a subject matter expert. You should know assembly plus disassembling. Finally you got to have experience with firmware and embedded systems.

These all sound like jobs with excellent compensation. More than your average software development position. Got to have that security clearance. Plus there seems to be a theme of cyber security style skills needed. Time to crack the books, study up, and get a promotion.

Disparate Systems

On my current project at work, I have a team lead and a project manager. I take most of my direction from my team lead. He manages the list of trouble tickets on our system. And he hands them out to members on our team. When we fix the issues, we give him the updated source code. He then works some magic to get those changes deployed. He keeps on top of this. So it works well for our team.


Recently the project manager has been unhappy because trouble tickets are not being updated in our trouble ticket tracking system. After a while, I told our project manager that our team is really not using the tracking system at all. The team lead is managing this stuff on his own outside of the tracking system. Thus, there is conflict.


In theory, the tracking system is the way to go. But we just need everyone on board. The main guy that needs to follow the process is our team lead. Without his buy in, none of this is going to work. It will take a time to get all the older tickets sorted out in the tracking system. Unfortunately, thing are crazy at work. So who has the time for that?

Teamwork Challenges

One of the newer guys on the team asked our project manager for some info. He needed to figure out how to get his scripts logging in to run unattended. The project manager asked our team lead to schedule a meeting to sort this out.

We had a meeting scheduled this morning. The new guy was not at work when it started, so we had to reschedule. When the meeting did get under way, we were not sure how to get this guy's script running. So we phoned an expert. Actually it was just some other guy on our team who has experience with it.

So we got through most of the questions. I thought we had a plan. The team lead said another new guy would help out the guy writing the script. Script writer got real salty with that setup. At first he said the newer guy could just take over the job. Then he complained loudly that he had done a lot of work, and just needed some info to get his scripts to run.

Later my team lead and I wondered what the heck went on? We usually work as a team and pair up on tasks all the time. Personally I like it when someone else can take a piece of the pie and lighten my load. There is definitely something wrong going down. Might be that the guy is insecure. I think he has some reason to be. Obviously he needs assistance a lot of the time. We will see how this plays out.

Save by the Log

I had promised to finish my current task within two days. First day I paced myself and did pretty good. As soon as I got in the next day, my boss called me up. We needed to debug some code that got sent out but had problems. It took about an hour. Okay I guess I could absorb that and still make my deadline.

Next there was a session scheduled to fix some data issues with our latest releases. It was a conference call with a lot of my teammates on it. So I tried to work on my stuff while the call was going on. Of courses a number of issues came up during the call where I was the only guy who knew how some data was being set. So I could only half-multitask. The call went on all morning.

In the afternoon I decided to cut lunch short to try to make my deadline. Things were going well. Then I realized the work I tried to squeeze in the morning was done wrong. Had to go back and fix that stuff. Deadline approaching. As I am getting near the home stretch of my task, my network connection goes down. Darn.

I reconnect but my database tool aborts. Luckily I always save all my work to a daily log. I just restarted my database tool, loaded up my log with all my commands, and picked right back up where I left off. What is the moral of this story? I think I had better pad my estimates to factor all these interruptions in there.

Non-Technical Challenges on the Job

I got my next assignment from my team lead. Needed to figure out the code to create a bunch of database tables. Then I need to write the code to put data in those tables. The good news is that we have the blueprint of the tables from another system. And somebody has written some database views that we are converting into tables. So most of the work is just the details.

Thought I would actually have some fun with this task. Then I got the catch. I had to split up the work and assign some of it to another developer on the team. This other guy has hours to work on our project. And he needs some work. The reason I figured this was a setup was because I have worked with the guy before. He needs a lot of hand holding.

Now I don't mind helping out a junior developer or even a new developer. But the guy I needed to work with is supposed to be a senior guy. And he is not new any more. So the management folks figure with his help, we should be able to get things done more quickly. On paper that is fine. In reality, not so good.

So I carved out a little over 25% of the work to give to the other guy. He had some questions early on. I tried to explain the answers to him. The he went loose. When he got to the one piece that was a bit more complicated, he was lost. I had to spend a bunch of time going over it with him in detail. It
took a few sessions. But I think it got through.

The last part of the task was to roll up both our work into one package. It all needs to go out into one release. So I did what I would do if I were developing the work myself. I slowly added the other guy's stuff into mine one small piece at a time. And let's just say there was pain as I debugged the errors and got the stuff to work. How can I get out of this type of setup? Maybe that is the point. I got to figure out how to work in this challenging situation.

Druva inSync for Enterprise Backup Needs

I have been backing up the data from my company laptop to CDs or DVDs for year. Initially I just used the reader/writer in my laptop. However newer versions of my work laptop did not have such a device. That's okay I thought. I could just copy the files to another computer using a USB thumb drive. Took a little longer, but it got the job done.

Now we have a new security policy at work. No home USB drives work in my computer. Damn. I requested a secure USB drive. My boss had to approve the purchase. I selected one with minimal capacity, so the cost would not be too high. The boss was apprehensive. He figured we have some backup systems available online to do the job.

Our company seems to have an enterprise version of Druva inSync available. It seems to back up your data to the cloud. The only experience I have with the product are the emails I get when the software has troubles with a backup. I tested the latest automated backup the software did. Was able to restore a file to my disk. Guess it works.

I played with the Druva client and added some more folders to get backed up every time it kicks off. It is around 2GB worth of data. Hopefully it works. I will have to make sure that I periodically check and test the backups. Cannot go along thinking everything is fine, only to discover that the thing is broke when I need to do a restore.