Resin vs Tomcat Benchmark

A company that I know have been using resin for years so far. Now that resin changed its license so that it will be charged per CPU core, they are looking in migrating to tomcat.

I've been told ( by someone in this company) that resin was chosen because a benchmark done years ago showed that it was much much faster.

Ok, then, I've been doing a bit of tests and benchmark to find out if that statement was still true.

Another statement found in the caucho's blog was that:

"Resin Open Source is about 45% faster than Tomcat, Resin Professional is about 80% faster than Tomcat."

Quite a bold statement, that hasn't proved to be true.

Let's go to the benchmark, if you allow me.

• Tools

I used Jmeter to simulate concurrent users, and got a couple of URL's from a live log to simulate production conditions. I then used excel to plot the charts and a in-house tool to show CPU usage. The application uses heavily network access to a separate backend.

I used resin 3.0.14 and tomcat 5.5.26.

• Tests

I run three types of tests:

• Long run (or overnight) test: running over night . I run this only with tomcat to prove that it would be reliable over a greater period of time
  

• Progressive load: I made separated test runs, increasing the number of users, from 5-170
  
• Ramp-up comparison: In the same test I used the jmeter ramp-up mode to simulate an increment of users.

Results:

• Long run.

No. of samples: 205.289
Average response time: 9106 ms
Minimum response time: 3.150 ms
Maximum response time: 22 seconds
Percentage of error: 0%
Final throughput 6.2 per second

This test only shows that tomcat is indeed reliable.

• Progressive Load

The chart below shows the final chart with the different tests executed.



The chart shows that the response time is almost the same, however tomcat is slightly superior in all of the tests.

The main thing I noticed in this test is the CPU consumption. Tomcat is visibly lighter in this regard. The graph below shows a run using tomcat, following a run with resin:




This chart was done when I was testing 100 concurrent threads sending requests to one server at a time.

• Ramp-up comparison

On this test I used the rampup optin in Jmeter to simulate an increment of users during the same test. I then got the final report ("View Results in Table" listener in Jmeter) and plotted a graph.

Tomcat Results:

• Samples: 1067
• Average: 10427
• Min: 3216
• Max: 27350
• Throughput: 4.4/sec
• Kb/Sec: 66.34KB/sec

Resin Results:

• Samples: 1090
• Average: 10450
• Min: 3434
• Max: 32360
• Throughput: 4.5/sec
• Kb/Sec: 66.6/sec

Not much difference here (a little advantage for tomcat though), although I would say that the resin is a little bit more erratic on its response time.

Again, the CPU monitoring shows that the resin consumed a lot more CPU than tomcat (the graph shows tomcat and ):

• Conclusion

From the different tests I've made, we can say that tomcat is slighly superior in terms of performance and visibly superior in terms of memory consumption. I want to clarify here that I used an specific version of resin, not the latest one available (3.1.6 been the latest). I also found several references stating that tomcat 6 uses has several improvements regarding performance, but we have decided to stick to 5.5.

Besides the performance, during the port of the application to tomcat, I also found out some other issues with resin:

- Resin xml validation is less strict: this means that a web.xml that work in resin might fail in tomcat. Resin also allows to use a non-compliant web.xml, based on a resin's schema.
- Resin allow mounting external path-mappings. This "feature" is used, for example to locate JSP's outside the war. Tomcat, in its version 6 had received a patch that allowed to do the same, but the patch was vetoed as this is against the specification.

- Shutdown and Start up: Contrary to my expectation, resin had a fast startup, which I'm counting until it comes back with the request response. It took 20 seconds both. From my experience I would say resin start was quite slow. Shutting resin down also is a bit problematic.

Security interface in Anthill - Adding permissions to N resources at a time

If I was to point one of the most annoying weaknesses in Anthill I would say that it is it's web interface when setting security.

There is no way to apply the security for several resources at once, or to apply several permission at once.

I had to add execution permission to our server operations team, but I didn't want to spend 1 hour opening all of the 20 projects and 60 workflows and add one by one.

So I decided to use remote scripting to achieved what I wanted. Remote scripting enables you to use Beanshell language with the AnthillPro API and add the permissions programatically.

The script that follows add a read permission to all projects and execution permission to all workflows that have deploy in its name. You can change it as you want to tailor it to your needs.

To execute it, you need to :
1) save it in a text file ("script.bsh", for example)
2) Get the remote scripting API at your local AHP server: http://subversion:18080/tools/remoting/anthill3-remoting.zip
3) Expand it and run your script using:
ah3client.cmd [file_name]

import com.urbancode.anthill3.main.client.AnthillClient;
import com.urbancode.anthill3.persistence.UnitOfWork;
import com.urbancode.anthill3.domain.project.*;
import com.urbancode.anthill3.domain.workflow.*;
import com.urbancode.anthill3.domain.security.*;

String serverHost = "server";
int serverPort = 4567;
String userName = "admin";
String password = "adminpasswd";
// obtain connection to the Anthill server
AnthillClient anthill = AnthillClient.connect(serverHost, serverPort,
userName, password);
// create a Unit of Work
UnitOfWork uow = anthill.createUnitOfWork();


String roleName = "Tech Lead"; // change to the name of the role you want to grant permission to
Role role = RoleFactory.getInstance().restoreForName(roleName);


//adding role permission to all deployment workflows
Project[] projects = ProjectFactory.getInstance().restoreAll();
for (int i = 0; i < projects.length; i++) {
Project project=projects[i];
//giving permission for the project
Resource resource =ResourceFactory.getInstance().restoreForPersistent(project);
Permission perm = new Permission(true, resource, "read",role);
perm.store();

Workflow[] workflowArray = project.getWorkflowArray();
for (int j = 0; j < workflowArray.length; j++) {
Workflow workflow = workflowArray[j];
if (workflow.getName().indexOf("deploy")>0) {
System.out.print("Adding server support permission to: "+project.getName()+" - "+workflow.getName()+"--"+workflow.getDescription());
System.out.println();
Resource resource =ResourceFactory.getInstance().restoreForPersistent(workflow);
Permission perm = new Permission(true, resource, Workflow.SEC_PERM_EXEC,role);
perm.store();
}
}

}
uow.commitAndClose();

Some more info here in the docs:
http://www.anthillpro.com/anthill3-help-3.5/html/ch46.html