Zuul

Zuul

###Jekyll

Jekyll is a markdown parser that allows creation of lightweight blogs in markdown without the need of databases.

##Quickstart with Jekyll:

• Fork https://github.com/poole/hyde
• Rename forked repository to .github.io
• Copy Rakefile from https://github.com/plusjade/jekyll-bootstrap.git which allows tasks like:
  • rake post title=”new post”
  • rake page name=”new_page.md”
• Change url in _config.yml to http://.github.io
• Change CNAME entry to .github.io
• Push changes to git repo and view blog at http://.github.io

##Folder Layout:

Time to start looking

The layoffs season is upon us or has it been a layoff year? Hard to keep track of. TechCrunch seems to have tried to come up with a comprehensive list but of course this is not even close to a complete list as it doesn’t contain the innumerable startups that sprung up during the covid era and disappeared soon after leaving scores of jobless engineers in their wake. Layoffs can of course be excruciatingly hard irrespective of whether you are a novice engineer or a seasoned veteran. Being laid off can have significant financial impact as well as cause a fair amount of emotional damage and loss of self confidence as having been part of a failed venture.

Of course this instability makes engineers very hesitant to work for startups which is bad for progress in society in general. We all need talented engineers working on new ventures to move the needle on technological progress. I regularly get asked by novice engineers on how to decide to work for a startup and how to hedge against the risks that come with working at a startup. Two cardinal rules that I follow while picking/working for a startup are:

Make sure you really want to work there

One of the main questions facing engineers while picking a startup is: Is this the right place? Engineers get drawn to startups for a myriad of reasons

• Competitive pay: Startups(of course funded ones) tend to pay a lot to hire the best talent
• Equity: If they can’t afford the cash, the main incentive to work at a startup is the equity which comes with the dream of hitting it big. Silicon Valley lore is rich with stories of early employees turned multi-millionaires
• Cutting Edge Tech: Generally startups are trying to solve problems with unique solutions that are only possible with the latest technologies for e.g: LLMs, GenAI etc or they are trying to revamp existing solutions built on old slow Technology. Both of these offer engineers the opportunity to work on the latest and greatest tech which is definitely alluring

Given there are a plethora of startups out there offering talented engineers some combination of the above, how do you go about choosing the right belle to the ball? Having worked for a few startups of varying degree of success, there are a few criteria I stick to while deciding to work for one:

• Mission statement: Make sure you really believe in the company’s mission statement, I mean really believe it. One easy way to tell if you really understand it is try to explain it to a friend and make them understand why the business is useful. This doesn’t necessarily mean the company will be successful and profitable, but you will be committed to the mission and startups with limited headcount need their employees to be completely committed to have any shot at succeeding. Weed away all the fluff about disrupting a billion dollar industry that all of them claim to do and boil it down to a very basic question: What do they do and why are they useful?
• Revenue: Dig into how profitable the company is or how close to profitability they are or if they even have a solid path to profitability. Companies might not be completely honest about this during the interview but you will get a sense of if there is a realistic chance of the business being profitable and if the current employees understand the path to it. Profitability is of course critical to long term employment.
• Overhead: This is the amount of non payroll expenditure the company spends in order to run their business. Higher the overhead, harder it is for a business to scale and become profitable and inability to scale is a death knell for a tech startup. To understand better what overheads are, take a look at Uber’s expense breakdown here. Insurance expenses, credit card processing fees, data center expenses etc. are large overheads for Uber. Of course a company is not going to divulge detailed revenue statements during an interview, but its still illuminating to ask the question: What is your largest non payroll overhead?
• Funding: Funding is a complicated topic. A lot of companies raise multiple rounds (Series A, B, C, D E etc) to be able to grow and expand their business. While this is necessary in a lot of cases, every additional funding round dilutes the value of equity held by an employee. And if the company is sold, the shareholders are paid off first before the employees which often results in the employees getting very little payout in the end. Of course a lot of companies go an alternate route of funding which can be a boon for employees:
• IPO: Working for a publicly traded company is one of the best possible solutions for an employee of a startup. Usually this means the company is valued more than when you joined it and so now your equity is worth more. Not only that, now you have a public platform to sell the equity and profit. However getting to an IPO is a long and twisted path and trying to IPO too early can be disastrous. So a good question to ask is, whether the company will be looking to go public or is the best result a sale to a larger organization which can also be a good result, but keep in mind the point about funding above.

Know when to leave

This is the million dollar question. Say you have joined a startup but you reach a point where you start questioning the viability of said startup, but you have put in a lot of effort into the work there, have made some deep connections and are vested both emotionally and … in options so understandably do not want to leave but yet don’t want to be blindsided by layoffs. Due to the vicissitudes that nearly every startup goes through, this is a pretty normal situation that most engineers find themselves in. Of course in almost every case if after spending some time at a startup an engineer is ambivalent about it’s chances of succeeding, it probably won’t happen. They have already subconsciously picked up on indicators of this throughout their time there and the next step is to consciously acknowledge it. There are a few signs of things not working out that I have observed over the years that would be useful for younger engineers to be aware of:

• Not having enough work: Of all the things in an employee’s control, this is one of the main indicators of the coming doom. This can take many forms including teams not having enough work and just doing startup-y things like playing ping pong all the time(or darts in a place I worked at), teams building things with no clear direction but never actually delivering it to customers, teams doing a lot of busy work to existing products without anything new being built etc. If this goes on for a while, it’s time to get out. Engineering is usually the most expensive part of any Tech startup and if engineers do not have enough to do, they would be the first to get laid off.
• Lack of updates: Most tech companies love doing all hands where the CEO or department heads provide updates around customers, sales, projects, upcoming quarter targets etc. If you as an engineer receiving these updates start feeling like everyone’s going through the motion that is a definitive sign of slowdown. When things are slow or stagnant the CEO/Head of sales etc either repeat the same updates using different verbiage or you will start hearing a lot of updates to the “pipeline”. There will be a lot of “we are in early stages of discussions” etc. which is meaningless till a contract is signed. There is no magic threshold of when these kind of updates become problematic, but if you as an employee start feeling the doldrums, start looking.
• Lack of transparency: Not every startup will open their books up to their employees like Netflix famously does, but lack of transparency around runway or revenue is definitely a red flag. You as an employee don’t need to be able to read the P&L disclosures but if you have no idea about what your company’s ARR is and roughly what runway it has, then you have no control over your destiny. If you see perks being cut and organizational heads not being transparent about layoffs, its time to start putting feelers out. Now transparency can cause issues in an organization where engineers can quit when faced with slowing growth, but transparency helps breed loyalty and the ones that remain are truly committed to the cause.

This is not meant to be a definitive guide on how to approach startups, but hopefully it provides some overarching rules that will help engineers have control over their careers and not get caught unawares.

Java 8-10 migration

Below are the steps I took to upgrade an existing legacy application(Voyager) to Java 10.

1) Setting up the environment: Download the jdk and jre for Java 10 and open up powershell and run the below commands to allow you to run with Java 10 without affecting your entire machine:

$Env:JAVA_HOME="C:\Program Files\Java\jdk-10.0.2" $Env:PATH="C:\Program Files\Java\jdk-10.0.2\bin;"+$Env:PATH

2) One of the major changes in java 9/10 is the module system. The modules define the public API for any library and prevent access to internal classes. One of the main reasons for doing this was that a lot of legacy systems were built using internal jdk classes which made it very hard for the jdk to evolve and deprecate old APIs. The same concept can be used for products built with java 9 onwards which can be modularized where boundaries are clearly defined and it becomes easier to change internal APIs to handle changing needs.

The problems that arose for legacy applications with this change is obviously the need to change the code that uses internal jdk APIs. To anticipate the changes required, Java8 introduced a tool called jdeps(in JAVA_HOME/bin). Use this as below:

jdeps.exe --jdk-internals .\VoyagerJava-8.0.0.jar

which will produce an output similar to the below:

	VoyagerJava-8.0.0.jar -> java.base
	com.dhs.shared.util.EmailImpl                      -> com.sun.net.ssl.internal.ssl.Provider              JDK internal API (java.base)
	com.dhs.shared.util.MimeWQIMessagePreparatorImpl   -> com.sun.net.ssl.internal.ssl.Provider              JDK internal API (java.base)

	Warning: JDK internal APIs are unsupported and private to JDK implementation that are
	subject to be removed or changed incompatibly and could break your application.
	Please modify your code to eliminate dependence on any JDK internal APIs.
	For the most recent update on JDK internal API replacements, please check:
	https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+Analysis+Tool

	JDK Internal API                         Suggested Replacement
	----------------                         ---------------------
	com.sun.net.ssl.internal.ssl.Provider    Use java.security.Security.getProvider(provider-name) @since 1.3

The above tells us that two classes use the com.sun.net.ssl.internal.ssl.Provider internal API. The solution for the same is also provided. This was fixed as below:

Security.addProvider(Security.getProvider("SunJSSE"));

jdeps allows to obtain a lot of other information about the projects dependencies, but for the purpose of migration, this is the most important aspect.

3) Another tool provided by the JDK in Java 9 onwards is jdeprscan. This allows you to scan for deprecated API and plan for API that can get removed in future versions. To make this work accurately, the dependent jars of the project need to be provided in a separate folder accessible to the tool:

jdeprscan.exe --class-path ".\lib/*" .\VoyagerJava-8.0.0.jar

This produces output similar to the below:

	Jar file .\VoyagerJava-8.0.0.jar:
	class com/dhs/billing/delinquency/mapper/DelinquencyMapper uses deprecated method java/lang/Long::<init>(J)V
	class com/dhs/billing/exports/directdebit/util/Formatter uses deprecated method java/lang/Float::<init>(F)V
	class com/dhs/billing/exports/prenote/util/Formatter uses deprecated method java/lang/Float::<init>(F)V
	class com/dhs/billing/premcalc/service/PremiumCalculationServiceImpl uses deprecated method java/math/BigDecimal::setScale(II)Ljava/math/BigDecimal;
	class com/dhs/enroll/imports/trr/service/MmpTrrServiceImpl uses deprecated method java/lang/Long::<init>(J)V
	class com/dhs/jdbc/shared/CrossReferenceDAOImpl uses deprecated method java/lang/Integer::<init>(Ljava/lang/String;)V
	class com/dhs/jdbc/shared/MemberDAOImpl$MemberHistoryLicsLevelRowMapper uses deprecated method java/lang/Double::<init>(D)V
	class com/dhs/jdbc/shared/MemberGroupMemberDAOImpl uses deprecated method java/lang/Integer::<init>(I)V
	class com/dhs/jdbc/shared/report/ReportDAOImpl uses deprecated method java/lang/Integer::<init>(I)V
	class com/dhs/jdbc/shared/WorkQueueDAOImpl uses deprecated method java/lang/Integer::<init>(I)V

If we want to determine what might be removed in future releases: jdeprscan.exe --class-path ".\lib/*" --for-removal .\VoyagerJava-8.0.0.jar

4) Now we are ready to start the migration. The first step is to upgrade maven compiler plugins to work with Java 10:

<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-compiler-plugin</artifactId>
			<version>3.7.0</version>
			<configuration>
				<release>10</release>
			</configuration>
			<dependencies>
				<dependency>
					<groupId>org.ow2.asm</groupId>
					<artifactId>asm</artifactId>
					<version>6.2</version> 
				</dependency>
			</dependencies>
</plugin>

Let’s try to compile now:

D:\VoyagerGitLab\VoyagerFramework> mvn clean install "-Dmaven.test.skip=true"

This will produce errors like below:

 [ERROR] /D:/VoyagerGitLab/VoyagerFramework/src/com/dhs/framework/messaging/MessageHandlerRepository.java:[6,24] cannot find symbol
  symbol:   class Resource
  location: package javax.annotation
[ERROR] /D:/VoyagerGitLab/VoyagerFramework/src/com/dynamichealthsys/common/error/_1/VoyagerSoapFault.java:[4,17] package javax.xml.ws is not visible
  (package javax.xml.ws is declared in module java.xml.ws, which is not in the module graph)

This points to modules in the jdk that are not accessible directly and need to be included. The jdk provides command line argument –add-modules for the same:

  <plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-compiler-plugin</artifactId>
				<version>3.7.0</version>
				<configuration>
					<release>10</release>
					<compilerArgs>
						<arg>--add-modules</arg>
						<arg>java.xml.ws</arg>
						<arg>--add-modules</arg>
						<arg>java.xml.ws.annotation</arg>
					</compilerArgs>
				</configuration>

Now when we try to compile the jar it compiles successfully.

5) Though compilation might succeed, some errors are discovered during startup. If we build Voyager and start it now, we see the below during startup:

Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected exception parsing XML document from URL [jar:file:/C:/Users/kkrishnan/Software/apache-tomcat-9.0.10-windows-x64/ap
ache-tomcat-9.0.10/webapps/Voyager-8.0.0/WEB-INF/lib/VoyagerJava-8.0.0.jar!/applicationContext-webservice.xml]; nested exception is java.lang.NoClassDefFoundError: javax/xml/ws/WebServiceException

This is because certain classes need to be included through 3rd party jars and are not available in the jdk modules:

	<dependency>
    		<groupId>org.wso2.orbit.sun.xml.ws</groupId>
    		<artifactId>jaxws-ri</artifactId>
    		<version>2.3.0</version>
	</dependency>

All Voyager jars need to be upgraded similarly and tested after startup. There might be other runtime issues which will have to be resolved similarly.

Based on Spring documentation, the minimum supported version for Java 9/10 is Spring 5.0. However, based on stackoverflow and my testing, Spring 4.3 seems to be work on it. Spring 5.0 will be a major upgrade and will have to be deferred to a later time

Java 11 introduces more depcrecated APIs and the Spring supported version is 5.1. It also removed the command line option and all the modules have to be included using third party jars. This will also be deferred to a later release

JVM Young generation tuning

Diagnosed an interesting performance issue recently with Garbage collection.

A process in production was running very slow and when I looked at the JVisualVM monitor, the CPU and memory graphs looked like this:

If you look at the CPU activity, the spikes in blue are actually Garbage collection cycles. It was almost consuming more processing time than the actual process. If you look at the memory map on the right, we are not even close to reaching the heap max. It was confusing as to why this was happening and so I read up on GC in Java 8 and remembered young generation vs old generation:

Garbage Collection

In short young generation is where the objects get created and then get transitioned to the old generation when the young generation reaches a capacity threshold. There are various stages within the young generation itself which cause minor GC cycles to run, but when objects move from the young generation to old, a major GC cycle runs(blue spike above). Clearly there were a lot of blue spikes above which indicated that objects were being transitioned very frequently. This implied that the young generation was not large enough(default is 1/4th of the heap). To tune this, I read further:

GC tuning

Using the setting of XX:NewRatio=2(YG is 1/3rd and OG is 2/3rd of the heap. YG should never be larger than OG), I got the below Monitor map:

You will notice that there is a lot of activity, but its mostly orange(good) and the blue spikes are small and the big ones are limited. This means that GC is not occupying a large portion of the processor cycles and allowing the process to use as many resources as it needs to. This doubled the performance(300 messages per second to over 600 per second) and the process remained stable. What is also interesting is that the memory graph has a lot of jitter. This is because the heap is allowed to grow in the young generation and then cleaned. Another curious effect was that the memory consumption remained very stable where as with the previous configuration, heap consumption kept growing rapidly.