Resin vs Tomcat Benchmark

A company that I know have been using resin for years so far. Now that resin changed its license so that it will be charged per CPU core, they are looking in migrating to tomcat.

I've been told ( by someone in this company) that resin was chosen because a benchmark done years ago showed that it was much much faster.

Ok, then, I've been doing a bit of tests and benchmark to find out if that statement was still true.

Another statement found in the caucho's blog was that:

"Resin Open Source is about 45% faster than Tomcat, Resin Professional is about 80% faster than Tomcat."

Quite a bold statement, that hasn't proved to be true.

Let's go to the benchmark, if you allow me.

• Tools

I used Jmeter to simulate concurrent users, and got a couple of URL's from a live log to simulate production conditions. I then used excel to plot the charts and a in-house tool to show CPU usage. The application uses heavily network access to a separate backend.

I used resin 3.0.14 and tomcat 5.5.26.

• Tests

I run three types of tests:

• Long run (or overnight) test: running over night . I run this only with tomcat to prove that it would be reliable over a greater period of time
  

• Progressive load: I made separated test runs, increasing the number of users, from 5-170
  
• Ramp-up comparison: In the same test I used the jmeter ramp-up mode to simulate an increment of users.

Results:

• Long run.

No. of samples: 205.289
Average response time: 9106 ms
Minimum response time: 3.150 ms
Maximum response time: 22 seconds
Percentage of error: 0%
Final throughput 6.2 per second

This test only shows that tomcat is indeed reliable.

• Progressive Load

The chart below shows the final chart with the different tests executed.



The chart shows that the response time is almost the same, however tomcat is slightly superior in all of the tests.

The main thing I noticed in this test is the CPU consumption. Tomcat is visibly lighter in this regard. The graph below shows a run using tomcat, following a run with resin:




This chart was done when I was testing 100 concurrent threads sending requests to one server at a time.

• Ramp-up comparison

On this test I used the rampup optin in Jmeter to simulate an increment of users during the same test. I then got the final report ("View Results in Table" listener in Jmeter) and plotted a graph.

Tomcat Results:

• Samples: 1067
• Average: 10427
• Min: 3216
• Max: 27350
• Throughput: 4.4/sec
• Kb/Sec: 66.34KB/sec

Resin Results:

• Samples: 1090
• Average: 10450
• Min: 3434
• Max: 32360
• Throughput: 4.5/sec
• Kb/Sec: 66.6/sec

Not much difference here (a little advantage for tomcat though), although I would say that the resin is a little bit more erratic on its response time.

Again, the CPU monitoring shows that the resin consumed a lot more CPU than tomcat (the graph shows tomcat and ):

• Conclusion

From the different tests I've made, we can say that tomcat is slighly superior in terms of performance and visibly superior in terms of memory consumption. I want to clarify here that I used an specific version of resin, not the latest one available (3.1.6 been the latest). I also found several references stating that tomcat 6 uses has several improvements regarding performance, but we have decided to stick to 5.5.

Besides the performance, during the port of the application to tomcat, I also found out some other issues with resin:

- Resin xml validation is less strict: this means that a web.xml that work in resin might fail in tomcat. Resin also allows to use a non-compliant web.xml, based on a resin's schema.
- Resin allow mounting external path-mappings. This "feature" is used, for example to locate JSP's outside the war. Tomcat, in its version 6 had received a patch that allowed to do the same, but the patch was vetoed as this is against the specification.

- Shutdown and Start up: Contrary to my expectation, resin had a fast startup, which I'm counting until it comes back with the request response. It took 20 seconds both. From my experience I would say resin start was quite slow. Shutting resin down also is a bit problematic.