Getting Access to the Code

Right now I am working on a tiger team to research why this new reporting system has different data than the old one. Most of the time I try to figure out what exactly is the complaint. The customers speak in a code of sorts. And lots of the tickets are written with the perspective of the new data loading system.

Now I did do a stint on the new data loading system for a few years. And I spent half my career on the old system. I think that is why they have me doing the research here. There is this one small database that I don't have much experience with. All the values in one certain field in it differed between the two systems. Hmm.

I found some documentation for the old system. Something looked screwy. There was an input field from a COBOL program that produced a values in dollars. However in the old system target database, they seemed to be storing dollars and cents. The cents portion was not zero. Felt like a defect to me.

To truly declare this a bug, I needed to look at the source code. Oops. The source code is only available on this Solaris UNIX system. Okay. I got access to a common account on there. Nope. You got to have a personal account to get access to the code. I tried to log in. Wrong password. I requested a new password.

Eventually after a second password reset, I got access to the UNIX. They got some Perl shell script to create a view to the code. Ran the script successfully. But where the heck is the source code. Turns out it gets stored on some shared storage off on the network. Whew. Tough job just getting to the darn source code.

Defect Ticketing System

At the end of last year, I left my prior project. Not sure if the cause was lack of money or retirement of the system I was working on. Could have been both. Luckily, and old manager offered to bring me back onto his project. They had interesting stuff to work on. It was mostly new development or new automated testing software.

 Eventually I had to work a trouble ticket from the customer. I thought I would just look the issue up in their ticketing system. Nope. I no longer had access to it. Maybe this was because I switched projects? Turns out I lost access because I had not used the system in a few months. Seems like an easy fix. I put in a request to regain access.

Some sys admin denied my request. He said I was not authorized to use the system. LOL wut? Turns out he looked up my name in his system, and did not find the proper authorization. I got help from my rep in the customer organization. They schooled the sys admin, telling him he was using some outdated system to look me up.

All right. I just want to get access to the ticketing system. So I asked the system admin to add me back in. The dude says that my request was denied, and I have no new requests. OK. Turns out I have teleported into a scene in The Office. Yeah the request got denied by you dude. You are using the wrong procedure to vet requests.

Oh well. Slackers are going to slack. Guess I need to put in another request. Hopefully I don't get the same goon servicing my request.

Microsoft Certification for SQL Server

At the end of last year, I got a surprise. Was the victim of a layoff from my project. Had been there 3+ years. I should have known something was up as the custom was rushing me to get a lot of stuff done before the end of the year. My takeaway is that I need to always be prepared to get another job. To do that, I need to stay sharp, especially with my technical skills.

My company did not lay me off. Just got the boot from my project. I searched around within the company. Found a couple jobs that seemed interesting and related to my experience. They all seemed to be using the Microsoft SQL Server database though. I have not dealt with SQL Server in forever. Time to brush up on my skills.

I figure I might as well gain a certification as I am learning this stuff. Plan to achieve an Associate (MCSA) certification in SQL 2016 Database Development. That should get my feet wet. It feels like a mid-level cert. I will need to pass two cert exams: 70-761 and 70-762. I have signed up for a Udemy course to prepare for the first exam.

Installing SQL Server right now. That feels like a story for another blog post.

Microsoft Azure Basics

Today I attended a one day class on the basics of Microsoft Azure. This is the cloud computing platform from Microsoft. Normally the course retails for $690. There was some agreement between Microsoft and the training company where I would not have to pay any money to attend the class. I like that price.


One subtlety that I think I picked up on was the true meaning of a private cloud. I previously thought that meant you have all the servers on your own premises. This is not necessarily true. The real meaning of private in this context is that nobody but you has access to these cloud objects. The machines they run on are 100% dedicated to you and you alone. The machines might be physically located in your building. But they could also be hosted by a cloud provider.


I was also surprised at the costs involved. Sure, Microsoft will take care of all the hardware, networking, patching, and so on. But I used their calculator to estimate the cost of a normal virtual machine. Would run me about $750 a month. Ouch. I think if I were counting costs, I might just want to buy a physical machine and handle the patches myself.


Some things about Azure were encouraging. It was really easy to spin up a virtual machine. You could also put together a server running SQL Server with a minimal amount of clicks. Just know that you will be paying a lot for these things. I hope I can get a free Azure account and play around with some of these configurations without incurring any real costs.

Low Code Development

At the recommendation of my boss, I am looking into some new technologies. Got to remain relevant for the future. My boss told me "low code" is one way to go. You create apps using a GUI. This is referred to as visual development. You draw the app like a flowchart. It reduces hand coding. It results in faster delivery.

You don't need hard core developers for low code development. The tools take care of the plumbing and infrastructure. Devops is provided by the tools. The apps generated are normally either web based or native mobile applications.

Forrester tracks the market. It says the leaders are Appian, Mendix, Outsystems, and Salesforce. Some other players are Zoho (Creator), Microsoft (PowerApps) and Google (App Maker). Most solutions have prebuilt templates you can employer. Most also have a WYSIWIG editor.

Costs are assessed on an ongoing monthly bases. Some of free to try. Others such as TrackVia cost $2k. Many are not just low code, but are no code. Some offer a preview of how the apps will look on mobile. Many let you deploy with a single click. Some generate native apps for mobile devices. Salesforce in particular has a mature market for third party components.

Hadoop

My boss has been encouraging me to learn some new technologies related to my interest in database development. He mentioned that Hadoop may be such a candidate. Hadoop is a set of open source utilities. They are used to solve problems with a massive amount of data and computation. They rely on distributed storage implemented by HDFS.

The programming model used by Hadoop is MapReduce. Data files are split into large blocks. These blacks are distributed across nodes. Code is also transferred to these nodes. Data is processed in parallel. The result is faster than conventional computing, even faster than mainframes. One of the biggest users of the tech is Yahoo for their search.

Hadoop is mainly written in Java. Some of it is in C. And there are some shell scripts involved. You access it through a Java API. However any programming language can use it via the Thift API or some other third party library. It does require the JRE. Files can be in the gigabyte or terabyte size. It does not require a RAID and data replication is built in. Usage is for immutable files. Concurrent write is not supported.

HDFS is a distributed, scalable, and portable file system. It is also written in Java. It acts like a data store. It is not POSIX compliant (which helps it achieve great speed). It also has a Java API. There are also shell commands that can interact with it.

Apache has many packages that go atop Hadoop. These include Pig, Hive, HBase, Spark, and Ooozie.

Databricks

My boss at work told me to maybe go into some newer technologies related to my data interests. One such item was Databricks. The name sounds cool. But I did not know much about them. Databricks is a company founded by the creators of Apache Spark. Recently they raised $250M in capital on a $6.2 valuation. Their yearly revenue is $200M.

Databricks produces a big data web based platform. You use it to work with Apache Spark. It provided what they call Unified Analytics. It develops pipelines across different storage systems. The pipelines are used to build ML models. The data is passed to third parties like Tableau, RStudio, and SnoFlake.

Databricks consists of a workspace, a runtime, and a cloud service. The latter is a managed service to host the tools. The runtime is built on an optimized version of Apache Spark. You choose which version of Spark to run on. There is a shared virtual notebook interface like the iPython UI.

So you can't understand Databricks without knowing what Spark is. That is a story of its own. In summary, Spark is a big data analytics engine. It is an alternative to MapReduce. Latency is reduced compared to competing products like Hadoop. It is open source. The engine runs on a cluster computing framework. It provides a distributed dataset. It also has a machine learning library.

And in case you did not know, iPython stands for Interactive Python. It is a command shell interface originally developed for Python. However it can now be used with multiple programming languages. It has tools to perform parallel computing jobs.