Reproducing a Race Condition

We have a job at work that runs every Wednesday night. All of a sudden, it aborted the last 2 weeks. This caused some critical data to be late. The main question was what changed? Initial analysis indicated nothing relevant changed. However things did change. An architect identified the most likely candidate. Some new job was trying to improve performance. This new job ran 10 to 11 minutes before the Wednesday job aborted each time.

I babysat the job this week. We ran it one day early to give us time in case we encountered errors. Of course the job ran fine when I ran it during the day. So I declared the next order of business was to replicate this problem in development. My old architect buddy thought it could be done. Maybe not. But I was going to give it the old college try.

I worked with a developer from the team that introduced the new performance job at night. We both ran our scripts at the same time. After a couple tries, I had still not replicated the problem. I told the dude to give me a copy of his code. Then I went to work. I spent 30 minutes reading up on the UNIX Korn shell. Then I wrote a Korn shell script to (1) generate new data, (2) run my job in the background, (3) wait a bit, and (4) run the new job.

Now I varied the amount of new data added each time from 1 record to 300 records. I also varied the amount of time waiting from 1 second to 5 minutes. Made sure to log everything. And since I launched this thing from an interactive shell, I prepended it with a "nohup" and ran my main script in the background too. I estimated best case this thing would finish in 12 hours. However it might run for as long as 24 hours. Luckily it is Friday. I imagine nobody is working in the development environment on a weekend.

Virtual Desktop Corrupted

I had a good sized task to complete at work. Told my boss it could be done in around two weeks. He thought that estimate was too small. I did not argue with him. We settled in at three weeks. I got some initial work done and published some docs and fixed one bug. Things were looking good.

I used this opportunity to refactor some test data gen programs. Was going to merge three separate programs into one. That would decrease future maintenance a lot. The code was written in Oracle PL/SQL with hard coded tables and columns. I figured it was time to turn this code into some dynamic SQL where I could plug in appropriate tables.

Spent a week and a half refactoring the code. Did all the work on a virtual machine because that is where my best tools were. Last Friday, the machine was acting weird. I could only access my Visual Studio program. Nothing else in Windows would come up. I rebooted the machine at the end of the day.

Monday morning came around. I could not log in. Tried to contact the customer help desk. Spent all afternoon on hold. Got up early Tuesday. Was able to speak to the help desk. The technician seemed confused and could not resolve my problem. Issue was escalated to second tier help desk.

Later on I found out a couple other developers lost access to their virtual machines. Apparently they pushed some "Data Loss Prevention" software to our machines. The software install must have been super buggy. It corrupted all our virtual machines. I lost all my work for the last two weeks.

Worst part is now I don't have a virtual machine to do my work on. I need to rely on my laptop and shell access to a Solaris UNIX server. Ouch. I told my boss I am not that productive in this environment. I started out using Windows Notepad. But I recalled that I had NotePad++ from a prior project installed. A bit better. But I work best in Visual Studio.

The only upside is that I love database programming. So I just got two more weeks of heavy work in my favorite environment. Schedule is not looking too good though.

Humble Bundle Book Deals

There is a book sale going on at Humble Bundle. You can get 15 Python books for $18. Now I already own some of those books. But I did find at least 5 that I would definitely buy. The value here seems so strong. I am a sucker for deals. However these books are in digital form only. I have come to hate eBooks.

I do have another option. If I buy books for college or for certification, my company will usually reimburse me. They have a high budget per employee each year for this. This year I have already got the company to buy me 6 books. I figure I might be reading these books past the end of the calendar year.

Still. I would like to also get these books. So I am going to wait until I am ready to take a mid-level Python certification exam. Then when I do, I will ask for those 5 interesting books to be purchased by the company. That should keep me busy well into 2022. The total cost for the print versions from Amazon is $135. That is even with Amazon's discount from the MSRP.

Perl Code Broken

We were getting close to turn on some software we developed last year. The last piece we forgot about was the piece that automatically ran the jobs on a weekly basis. I negotiated the timing of the jobs with other stakeholders. Then I figured out some cron job entries to make these jobs wake up at the right time. Last piece was to test the cron job runs in development.

At first, I scheduled the simple job to run. The thing bombed because it could not find an include file. That seemed odd. The file was in the same directory as the Perl script that was running. But that directory was not in the @INC path. Why not? I thought this had worked before. It actually works when I run the job manually.

Upon further inspection, I found that the profile that was run when the cron job executes had a bug. It was not setting the PERL5LIB directory correctly, which in turn would get added to the @INC array. That was too much details for somebody who is not a Perl programmer. But hey. I guess this was a small opportunity to learn something.

Harvard Learning

I have previously heard about Harvard offering some courses online for free. Well you can do that now with CS50. This course seems to teach a ton of info. It seems to last almost as long as a normal college semester. It claims to teach you C, PHP, and JavaScript programming. You will also learn SQL.

There are a total of 9 assignments in the class. There is also 1 final project. You only get to audit the class. However, if you plunk down $199, you can get a certificate of completion. I bet I could have my company pay for this course. Not sure if it will do me any good.

I do like the idea of being able to tell people that I "went to Harvard" for computer science. LOL.

Getting Access to the Code

Right now I am working on a tiger team to research why this new reporting system has different data than the old one. Most of the time I try to figure out what exactly is the complaint. The customers speak in a code of sorts. And lots of the tickets are written with the perspective of the new data loading system.

Now I did do a stint on the new data loading system for a few years. And I spent half my career on the old system. I think that is why they have me doing the research here. There is this one small database that I don't have much experience with. All the values in one certain field in it differed between the two systems. Hmm.

I found some documentation for the old system. Something looked screwy. There was an input field from a COBOL program that produced a values in dollars. However in the old system target database, they seemed to be storing dollars and cents. The cents portion was not zero. Felt like a defect to me.

To truly declare this a bug, I needed to look at the source code. Oops. The source code is only available on this Solaris UNIX system. Okay. I got access to a common account on there. Nope. You got to have a personal account to get access to the code. I tried to log in. Wrong password. I requested a new password.

Eventually after a second password reset, I got access to the UNIX. They got some Perl shell script to create a view to the code. Ran the script successfully. But where the heck is the source code. Turns out it gets stored on some shared storage off on the network. Whew. Tough job just getting to the darn source code.

Defect Ticketing System

At the end of last year, I left my prior project. Not sure if the cause was lack of money or retirement of the system I was working on. Could have been both. Luckily, and old manager offered to bring me back onto his project. They had interesting stuff to work on. It was mostly new development or new automated testing software.

 Eventually I had to work a trouble ticket from the customer. I thought I would just look the issue up in their ticketing system. Nope. I no longer had access to it. Maybe this was because I switched projects? Turns out I lost access because I had not used the system in a few months. Seems like an easy fix. I put in a request to regain access.

Some sys admin denied my request. He said I was not authorized to use the system. LOL wut? Turns out he looked up my name in his system, and did not find the proper authorization. I got help from my rep in the customer organization. They schooled the sys admin, telling him he was using some outdated system to look me up.

All right. I just want to get access to the ticketing system. So I asked the system admin to add me back in. The dude says that my request was denied, and I have no new requests. OK. Turns out I have teleported into a scene in The Office. Yeah the request got denied by you dude. You are using the wrong procedure to vet requests.

Oh well. Slackers are going to slack. Guess I need to put in another request. Hopefully I don't get the same goon servicing my request.

Microsoft Certification for SQL Server

At the end of last year, I got a surprise. Was the victim of a layoff from my project. Had been there 3+ years. I should have known something was up as the custom was rushing me to get a lot of stuff done before the end of the year. My takeaway is that I need to always be prepared to get another job. To do that, I need to stay sharp, especially with my technical skills.

My company did not lay me off. Just got the boot from my project. I searched around within the company. Found a couple jobs that seemed interesting and related to my experience. They all seemed to be using the Microsoft SQL Server database though. I have not dealt with SQL Server in forever. Time to brush up on my skills.

I figure I might as well gain a certification as I am learning this stuff. Plan to achieve an Associate (MCSA) certification in SQL 2016 Database Development. That should get my feet wet. It feels like a mid-level cert. I will need to pass two cert exams: 70-761 and 70-762. I have signed up for a Udemy course to prepare for the first exam.

Installing SQL Server right now. That feels like a story for another blog post.

Microsoft Azure Basics

Today I attended a one day class on the basics of Microsoft Azure. This is the cloud computing platform from Microsoft. Normally the course retails for $690. There was some agreement between Microsoft and the training company where I would not have to pay any money to attend the class. I like that price.


One subtlety that I think I picked up on was the true meaning of a private cloud. I previously thought that meant you have all the servers on your own premises. This is not necessarily true. The real meaning of private in this context is that nobody but you has access to these cloud objects. The machines they run on are 100% dedicated to you and you alone. The machines might be physically located in your building. But they could also be hosted by a cloud provider.


I was also surprised at the costs involved. Sure, Microsoft will take care of all the hardware, networking, patching, and so on. But I used their calculator to estimate the cost of a normal virtual machine. Would run me about $750 a month. Ouch. I think if I were counting costs, I might just want to buy a physical machine and handle the patches myself.


Some things about Azure were encouraging. It was really easy to spin up a virtual machine. You could also put together a server running SQL Server with a minimal amount of clicks. Just know that you will be paying a lot for these things. I hope I can get a free Azure account and play around with some of these configurations without incurring any real costs.

Low Code Development

At the recommendation of my boss, I am looking into some new technologies. Got to remain relevant for the future. My boss told me "low code" is one way to go. You create apps using a GUI. This is referred to as visual development. You draw the app like a flowchart. It reduces hand coding. It results in faster delivery.

You don't need hard core developers for low code development. The tools take care of the plumbing and infrastructure. Devops is provided by the tools. The apps generated are normally either web based or native mobile applications.

Forrester tracks the market. It says the leaders are Appian, Mendix, Outsystems, and Salesforce. Some other players are Zoho (Creator), Microsoft (PowerApps) and Google (App Maker). Most solutions have prebuilt templates you can employer. Most also have a WYSIWIG editor.

Costs are assessed on an ongoing monthly bases. Some of free to try. Others such as TrackVia cost $2k. Many are not just low code, but are no code. Some offer a preview of how the apps will look on mobile. Many let you deploy with a single click. Some generate native apps for mobile devices. Salesforce in particular has a mature market for third party components.

Hadoop

My boss has been encouraging me to learn some new technologies related to my interest in database development. He mentioned that Hadoop may be such a candidate. Hadoop is a set of open source utilities. They are used to solve problems with a massive amount of data and computation. They rely on distributed storage implemented by HDFS.

The programming model used by Hadoop is MapReduce. Data files are split into large blocks. These blacks are distributed across nodes. Code is also transferred to these nodes. Data is processed in parallel. The result is faster than conventional computing, even faster than mainframes. One of the biggest users of the tech is Yahoo for their search.

Hadoop is mainly written in Java. Some of it is in C. And there are some shell scripts involved. You access it through a Java API. However any programming language can use it via the Thift API or some other third party library. It does require the JRE. Files can be in the gigabyte or terabyte size. It does not require a RAID and data replication is built in. Usage is for immutable files. Concurrent write is not supported.

HDFS is a distributed, scalable, and portable file system. It is also written in Java. It acts like a data store. It is not POSIX compliant (which helps it achieve great speed). It also has a Java API. There are also shell commands that can interact with it.

Apache has many packages that go atop Hadoop. These include Pig, Hive, HBase, Spark, and Ooozie.

Databricks

My boss at work told me to maybe go into some newer technologies related to my data interests. One such item was Databricks. The name sounds cool. But I did not know much about them. Databricks is a company founded by the creators of Apache Spark. Recently they raised $250M in capital on a $6.2 valuation. Their yearly revenue is $200M.

Databricks produces a big data web based platform. You use it to work with Apache Spark. It provided what they call Unified Analytics. It develops pipelines across different storage systems. The pipelines are used to build ML models. The data is passed to third parties like Tableau, RStudio, and SnoFlake.

Databricks consists of a workspace, a runtime, and a cloud service. The latter is a managed service to host the tools. The runtime is built on an optimized version of Apache Spark. You choose which version of Spark to run on. There is a shared virtual notebook interface like the iPython UI.

So you can't understand Databricks without knowing what Spark is. That is a story of its own. In summary, Spark is a big data analytics engine. It is an alternative to MapReduce. Latency is reduced compared to competing products like Hadoop. It is open source. The engine runs on a cluster computing framework. It provides a distributed dataset. It also has a machine learning library.

And in case you did not know, iPython stands for Interactive Python. It is a command shell interface originally developed for Python. However it can now be used with multiple programming languages. It has tools to perform parallel computing jobs.

New Skills Needed

I spoke with my manager this past week. He said my project might not be around much longer. So I should gear up and acquire skills for the next gig. He recommended I learn some hot technologies. One was "Pega". I presume he meant Pegasystems.

Pega mainly targets the banking and financial markets. But they are moving into healthcare. They compete with SAP ERP. They also compete with Oracle. They partner up with a lot of other firms such as Accenture and Ernst and Young.

Pega deals in the world of customer engagement. They help connect a customer's interface with the back end. They do CRM and BPM. Buzzwords. Yay. They want to optimize customer interactions with the companies they assist.

The big set of offerings they have are the Pega Infinity Suite. The flagship product is the Pega Platform. It is a low code solution. It is AI-powered. And it is architected on the cloud. These all sound like hot commodities. But it does not feel like you will get to do much custom coding if you work with this technology.

Project Management Required

I have a lot of trouble ticket assigned to me. We release them to the customer in batches. The development cycle is usually about a month long for each. We have a trouble ticketing system. In that system, each ticket gets assigned to a release. Many of my tickets were not assigned to releases. There are only two future released currently planned.

The big boss said all tickets must get a release assigned in the ticketing system. Okay. I went through and just assigned everything to the next release. Check. But I wonder what they are using that release for? I bet this is used to convey expected resolution times to the customer. Oops.

Realistically there are only a few tickets that I am going to resolve in the next release. I already know that. I don’t get to spend all my time working this new release. I have to support the previous release in all the stages of testing it goes through. I am also taking a vacation next month. All this adds up to some unpleasant surprises come delivery time.

I need to make sure I am not part of the problem. Time to stand up and raise the alarm about this release issue. My team lead said I should just leave the release dates blank, forcing everyone to deal with the underlying issue. That is all good and dandy, but I am tired of getting beat up because my tickets don’t have release dates set.

Analysis Failure

My new team leader asked me to cost a fix to a bug the customer found. I looked up the ticket in our system. Somebody had already done an analysis and provided a design for the fix. I said I could code the fix in 3 days. There were a couple tricks to figure out. But hey. The hard part seemed to be done already. It could not take more than 3 days.

I did some additional analysis first. Took almost a whole day. I lined up each of the items in the report to their sources. Then I started trying to lay out how I was going to code up this mapping. I realized that half of the data was missing from the database.

I tracked down the author of the trouble ticket. She told me that the data was missing now, but would be present in a future release. Oops. I was supposed to deliver a fix now. While I could implement that missing part, that would be a big job. I called my team leader and broke the bad news to him.

My team lead said that commitments were made. Umm yeah. They were made based on someone already having done a design, and that the design was feasible. My lead started looking for ways for us to do a smaller scope of work and declare the problem fixed. This did not look like it would end well.

Continuing Education

I read a post a lot of replies to a Reddit group on computer science graduates. Met a dude there who is an old timer, looking to switch careers into computer science. He said that he is taking an online program at OSU. That is, the Oregon State University. They have a program especially for people who already have a degree in another major.

I checked out the degree. Looks interesting. They have different paces for you to complete the courses. Total cost is $29k. That might seem high at first. But if you think about it being spread over 5 years, that is under $6k per year, or $3k per semester. Still costly but doable. Who knows? I might even be interested.

Back in college, I choose engineering. I really should have done Computer Science. By the time I figured that out, I was pretty far along in the engineering route. Would have seriously extended my graduation date if I switched over to Comp Sci during my junior year. Hindsight is 20/20. However this might be an opportunity to do it all over.

The Non Scrum

I recently joined a newly formed team at work. The whole project was changing how they tracked work. We were going to have a daily scrum. Great. Sounds like agile to me. Been there done that. I got to the first meeting. Was surprised that the team lead was running the meeting and writing down status. Umm. That's not agile.

So what exactly is the daily scrum supposed to be like? Things like this:

• Developers run the meeting
• Only developers talking
• Check to see if on track to meet sprint goal
• Team members making commitments to each other
• Developers addressing each other when talking
• Sharing the big picture of what is happening

Here is what the scrum is not supposed to be about:

• Not a status update
• Not the scrum master running the meeting
• Not interruptions from non-developers
• Not for the boss to get reports from team members
• Not developers talking to the scrum master
• Not a recording meeting

Yikes. What have I got myself into? I have a few ideas. This might be partially some people who have no clue what agile is about. Might also be somebody selling a bill of agile, but not really committing to the real thing. Or it could be somebody with a different methodology, borrowing a lot of the agile terms but meaning something else. Who knows?

LeetCode Getting Downright Ugly

I continue doing LeetCode exercises. Worked on another "Medium" difficulty problem today. Seemed straightforward enough. Find the longest length of any substring in a specified string that does not have any duplicated characters.

This is my practice for C++. Therefore I am using the STL to get more experience with it. Store all substrings in a vector. Write a function to determine if a string has any dup chars using a map. This works for most of the one thousand test cases. But there are a few where LeetCode says my solution took too long.

LOL wut? Some test cases pass in a massive amount of text. I am enumerating all combinations of substrings in there. After I tighten up my code, I guess that my dup detector is just taking too long. Therefore I get rid of the STL map from it. Code is up with straight C arrays. Works fine.

LeetCode Getting Tricky

I decided I would try to LeetCode exercises for practice. Nailed the first one I tried yesterday. It was an easy problem. The one I tried today was assessed as Medium difficulty. At first I thought the problem was easy. It passed the sample test case pretty quickly. Then I saw what the challenge was. They were testing you for huge numbers.

At first I tried to just change my variables from int to long. Nope. Not good enough. Then I tried long long int. Figured they might be testing up to that. Nope. They were dealing with huge numbers. I did not want to implement a big number library. Should not have been a need to copy and paste the code for such a library too.

Then it dawned on me. They were providing the input numbers in a weird linked list format that encouraged the use of big number addition. I did not need to implement the whole big number library. Just needed to be able to add two digits, and determine if there was any carry over to the next place.

These LeetCode problems are tricky. And I have not gotten to a Hard problem yet.

Practicing on LeetCode

I have been reading the Computer Science Careers subreddit recently. There is a lot of talk about how to prepare for job interviews. Many people mention doing problems on LeetCode. I have never heard of that site. I want to interview for another job. Maybe I should be doing practice problems on there too.

So I created an account. Seems like I could proceed without paying for a developer subscription. I went to problem number one. The site was preconfigured to use C++ as the programming language. Does the site just use that because it is a popular language? Or did it figure out I know C++ from my profile?

Had some trouble getting my first entry to compile. The error messages were a bit cryptic. No trouble. I know how to debug, isolating the faults. Got my code to return the correct result for the sample inputs. Then I submitted my solution.

Was a bit disappointed that, although my solution was accepted, 95% of the other solutions executed faster than mine. Nobody told me we would be graded on speed. Next time I will have to optimize my solution.