Low Code Development

At the recommendation of my boss, I am looking into some new technologies. Got to remain relevant for the future. My boss told me "low code" is one way to go. You create apps using a GUI. This is referred to as visual development. You draw the app like a flowchart. It reduces hand coding. It results in faster delivery.

You don't need hard core developers for low code development. The tools take care of the plumbing and infrastructure. Devops is provided by the tools. The apps generated are normally either web based or native mobile applications.

Forrester tracks the market. It says the leaders are Appian, Mendix, Outsystems, and Salesforce. Some other players are Zoho (Creator), Microsoft (PowerApps) and Google (App Maker). Most solutions have prebuilt templates you can employer. Most also have a WYSIWIG editor.

Costs are assessed on an ongoing monthly bases. Some of free to try. Others such as TrackVia cost $2k. Many are not just low code, but are no code. Some offer a preview of how the apps will look on mobile. Many let you deploy with a single click. Some generate native apps for mobile devices. Salesforce in particular has a mature market for third party components.

Hadoop

My boss has been encouraging me to learn some new technologies related to my interest in database development. He mentioned that Hadoop may be such a candidate. Hadoop is a set of open source utilities. They are used to solve problems with a massive amount of data and computation. They rely on distributed storage implemented by HDFS.

The programming model used by Hadoop is MapReduce. Data files are split into large blocks. These blacks are distributed across nodes. Code is also transferred to these nodes. Data is processed in parallel. The result is faster than conventional computing, even faster than mainframes. One of the biggest users of the tech is Yahoo for their search.

Hadoop is mainly written in Java. Some of it is in C. And there are some shell scripts involved. You access it through a Java API. However any programming language can use it via the Thift API or some other third party library. It does require the JRE. Files can be in the gigabyte or terabyte size. It does not require a RAID and data replication is built in. Usage is for immutable files. Concurrent write is not supported.

HDFS is a distributed, scalable, and portable file system. It is also written in Java. It acts like a data store. It is not POSIX compliant (which helps it achieve great speed). It also has a Java API. There are also shell commands that can interact with it.

Apache has many packages that go atop Hadoop. These include Pig, Hive, HBase, Spark, and Ooozie.

Databricks

My boss at work told me to maybe go into some newer technologies related to my data interests. One such item was Databricks. The name sounds cool. But I did not know much about them. Databricks is a company founded by the creators of Apache Spark. Recently they raised $250M in capital on a $6.2 valuation. Their yearly revenue is $200M.

Databricks produces a big data web based platform. You use it to work with Apache Spark. It provided what they call Unified Analytics. It develops pipelines across different storage systems. The pipelines are used to build ML models. The data is passed to third parties like Tableau, RStudio, and SnoFlake.

Databricks consists of a workspace, a runtime, and a cloud service. The latter is a managed service to host the tools. The runtime is built on an optimized version of Apache Spark. You choose which version of Spark to run on. There is a shared virtual notebook interface like the iPython UI.

So you can't understand Databricks without knowing what Spark is. That is a story of its own. In summary, Spark is a big data analytics engine. It is an alternative to MapReduce. Latency is reduced compared to competing products like Hadoop. It is open source. The engine runs on a cluster computing framework. It provides a distributed dataset. It also has a machine learning library.

And in case you did not know, iPython stands for Interactive Python. It is a command shell interface originally developed for Python. However it can now be used with multiple programming languages. It has tools to perform parallel computing jobs.

New Skills Needed

I spoke with my manager this past week. He said my project might not be around much longer. So I should gear up and acquire skills for the next gig. He recommended I learn some hot technologies. One was "Pega". I presume he meant Pegasystems.

Pega mainly targets the banking and financial markets. But they are moving into healthcare. They compete with SAP ERP. They also compete with Oracle. They partner up with a lot of other firms such as Accenture and Ernst and Young.

Pega deals in the world of customer engagement. They help connect a customer's interface with the back end. They do CRM and BPM. Buzzwords. Yay. They want to optimize customer interactions with the companies they assist.

The big set of offerings they have are the Pega Infinity Suite. The flagship product is the Pega Platform. It is a low code solution. It is AI-powered. And it is architected on the cloud. These all sound like hot commodities. But it does not feel like you will get to do much custom coding if you work with this technology.

Project Management Required

I have a lot of trouble ticket assigned to me. We release them to the customer in batches. The development cycle is usually about a month long for each. We have a trouble ticketing system. In that system, each ticket gets assigned to a release. Many of my tickets were not assigned to releases. There are only two future released currently planned.

The big boss said all tickets must get a release assigned in the ticketing system. Okay. I went through and just assigned everything to the next release. Check. But I wonder what they are using that release for? I bet this is used to convey expected resolution times to the customer. Oops.

Realistically there are only a few tickets that I am going to resolve in the next release. I already know that. I don’t get to spend all my time working this new release. I have to support the previous release in all the stages of testing it goes through. I am also taking a vacation next month. All this adds up to some unpleasant surprises come delivery time.

I need to make sure I am not part of the problem. Time to stand up and raise the alarm about this release issue. My team lead said I should just leave the release dates blank, forcing everyone to deal with the underlying issue. That is all good and dandy, but I am tired of getting beat up because my tickets don’t have release dates set.

Analysis Failure

My new team leader asked me to cost a fix to a bug the customer found. I looked up the ticket in our system. Somebody had already done an analysis and provided a design for the fix. I said I could code the fix in 3 days. There were a couple tricks to figure out. But hey. The hard part seemed to be done already. It could not take more than 3 days.

I did some additional analysis first. Took almost a whole day. I lined up each of the items in the report to their sources. Then I started trying to lay out how I was going to code up this mapping. I realized that half of the data was missing from the database.

I tracked down the author of the trouble ticket. She told me that the data was missing now, but would be present in a future release. Oops. I was supposed to deliver a fix now. While I could implement that missing part, that would be a big job. I called my team leader and broke the bad news to him.

My team lead said that commitments were made. Umm yeah. They were made based on someone already having done a design, and that the design was feasible. My lead started looking for ways for us to do a smaller scope of work and declare the problem fixed. This did not look like it would end well.