Hot, black, sugar-free

  • All sorts of things We first started work with Java in 2001, using JDK 1.3, which was upgraded to JDK 1.4 by the time the project was deployed. This was a browser-based application where the code was written as servlets, with an Oracle database backend, all running on Linux. From about 2003, we engaged in a long-term engagement with a large customer for Java applications, and some of that engagement continues till today.

    We built a payment gateway integration layer for this customer, to allow him to connect to any of a variety of payment gateways. After we wrote the code individually to handle multiple payment gateways, we decided to abstract this code into a service (more details here).

    We also started working with Web services. Initially, we implemented simple calls from one servlet to another over HTTP, without conforming to any protocol or standard. In later projects we implemented SOAP Web services.

  • Tomcat travails In 2004, we encountered serious memory leaks in Tomcat, which were triggered after a certain number of hits or after Tomcat had been allowed to keep running for a certain number of days and nights. When this condition was reached, Tomcat's memory consumption would rise sharply, and the system would stop responding to requests. This was with an older version of Tomcat which did not have much instrumentation to help us monitor its guts. So we built external programs which would call a special-purpose servlet. If this servlet did not respond within a specific duration, the program would restart Tomcat. This appeared to work quite well, and was retained in production servers for many years till systems were upgraded to Tomcat v6 after v6 was released.

    We later discovered another problem with Tomcat's memory management. We discovered that Tomcat would freeze at random moments for many seconds at a time, refusing to respond to any requests, and then wake up again afterwards without us doing anything. As is often the case with customers, our customer blamed our application code. Our investigations took us to the Tomcat GC (garbage collection). Tomcat's JVM usually does incremental GC, but sometimes needs to do a full GC. During full GC, all user code execution is suspended till GC is complete. This GC happened at random moments, and exactly correlated with the system freeze reports. We began to correlate them with very large quantities of object creation/release in some servlets. We modified that code to reduce the object creation quantity, and frequency of full GC dropped. We also changed Tomcat's configuration parameters to stabilise its GC behaviour.

  • Database disturbances When we began using database connection pooling provided by Tomcat, we began to notice resource depletion at odd moments, leading to database connection failure. We built a wrapper around the connection pool class and forced all code to access the connection pool through the wrapper. Using this wrapper, we logged each connection request and release. Each call to any method in the wrapper was passed a unique ID corresponding to the servlet. The logs of request and release calls therefore allowed us to track the average time taken by each servlet to perform its database I/O and allowed us to pin-point which servlets occasionally showed unbalanced request-release patterns (i.e. a request without a corresponding release). This allowed us to rapidly fix problems in code related to connection pool handling and also allowed us to identify slow servlets. These slow servlets were the ones causing serious system throughput problems when under heavy load, and we modified their code to make their database interactions faster.

    We also realised that all our old servlet code would grab a database connection at the start, do their work, and then release the connection at the end. So we implemented a configuration layer in the connection pool wrapper to allow the administrator to shut down certain servlets by configuring the wrapper to deny database connections to it. When a servlet failed to connect to the database, it would display a standard error message informing the user that the system was under maintenance. Therefore, using this configurable switch-on-switch-off of servlets one at a time, we could control when to take certain servlets off-line and keep others active. This flexibility was highly appreciated by our customer, who operated one of the country's most critical online business services.

  • Killer crons One of our teams working with Nortel/Avaya IVR technologies was maintaining large quantities of undocumented and confusing Java code linking an IVR front-end with a Java Web-services back-end. They discovered that this the IVR front-end would suddenly begin to report connection loss with the back-end at odd times, without any obvious pattern or reason. The legacy code's logs did not give any indications to the root cause. After a few days of intense traces and experiments, our team discovered that someone had written a cron job to kill and restart a JSB (Java Services Bridge), a module which connected the IVR front-end to the rest of the back-end. No one in the current team from the customer's side or our team knew who had written this code and why. The cron job would run every 10 minutes from 3 AM to 5 AM and would look for some possible error condition. If it found this error condition, it would restart the JSB. We discovered that this error-detection code was buggy, and the cronjob would sometimes detect spurious errors and kill the JSB. This would happen at any odd time between 3 AM and 5 AM and would therefore be very hard to put into a pattern. Removing this cron job solved the service interruption problems. Technically, this cron job was not part of the scope of our team's work, but in the absence of a clear root cause analysis, the customer blamed our maintenance team for system instability. It all ended well and sanity was restored.

  • Going forward We have been watching the growth of Java in the enterprise software space. Java is no longer "cool", the way, say, Go or even Python is today. Google popularises Python, Facebook works with PHP, but none of the cool sites seem to be visibly evangelising Java. It also has a verbose syntax which can become irritating to expert programmers. However, it has incredible richness of libraries, integration interfaces to almost everything, an excellent development interface with a rich set of tools, and is a relatively safe language for slightly less proficient programmers. Therefore its future is assured, unless Oracle, the current IP owner, does something which destabilises its popularity.

    We initially were impressed with the richness of Java class libraries which allowed Java programmers to call (i.e. method invocations) anything anywhere. We have now begun to see that Java has reached a different dimension -- anything and everything can call Java code. In other words, if we need to add our own custom modules within large existing systems, Java is often the language of choice. At the turn of the century, Oracle announced that stored procedures in its database would not be restricted to PL/SQL -- they could now be written in Java. We now see that document management systems (e.g. Documentum and Alfresco) allow the base system to be extended if you plug in your Java code. Portal frameworks allow customisations if you plug in your own Java modules. Therefore, Java's reach has extended both "inside" and "outside" enterprise software to a degree which few other languages have. We continue to be committed to Java programming and we expect to see interesting developments in the years ahead.