Home  /  RSS  /  RSS Comments  /  Enter

CppCMS benchmarks vs Java, C#, PHP

10/16/10, by artyom ; Posted in: Benchmarks, Cache; 25 comments

Long time ago I had posted a benchmarks comparing CppCMS based blog and PHP based one.

I wanted to compare real life applications with each other. For a long time I had been searching for similar applications in several technologies doing very similar jobs in leading technologies: PHP, Asp.Net and Java/JSP. The last two were particularly important as they use static type system and "compiled" languages as C# and Java that are known to be faster then other dynamic typed languages like PHP, Python, Ruby and Perl popular in web development.

Setup

Unfortunately I had failed to find such application, so finally I decided to write something representative and small on my own an application with following requirements:

  1. Uses simple time-out based page caching
  2. Uses MySQL and the database and keeps open connections in pool.
  3. For each request access to database (if page is not cached), fetches the page content and comments for "sample article" in blog.
  4. Converts text to HTML using a markdown filter and displays it on page.

I used following technologies:

I tested following parameters:

For each test the application was "warmed up" with 100 requests to fill the cache, and then 1000 request with max concurrency of 5 request are done, while certain percent of them is new pages and the other are taken from "warmed up" once.

Notes:

I used the fastest Markdown implementation I had found.

C# implementation is the same one that http://stackoverflow.com uses - it is actually heavily optimized implementation based on older C# implementation

The Java implementation is based on the above C# and the fastest one I had found.

Discount is the fastest C implementation of markdown that I had found.

Results:

Benchmarks Markdown

Memory

Summary

  1. C#, Java and PHP implementation behave very similarly and without significant differences.
  2. The memory usage of Java/Tomcat and Mono/Asp.Net was significantly higher - up one or two orders of magnitude in comparison to CppCMS and PHP
  3. Surprisingly PHP behaves very well, in comparison to "compiled" languages like Java and C#.

Revisiting

After doing some profiling it was clear that C implementation of Markdown was significantly faster then all other implementations. So I decided to create my own mini-markdown that make some basic handing of titles, lists, paragraphs and quotes at one level only. That is very simple syntax but implemented similarly in all 4 languages using same algorithm.

The results were following:

Benchmarks Markdown

The difference between CppCMS and other implementations was still significant but still much smaller then the difference between real markdown implementation. So the performance difference was less dramatic.

2nd Revision

And in the last revision I decided not to use any text filters by fetch ready HTML formatted content from DB and display it on the web as is.

Such comparison actually profile the most basic stuff:

  1. Caching
  2. SQL Connection
  3. Request/Response handling

And would ignore hundreds lines of code used in any web applications responsible for the actual business logic.

Benchmarks HTML

Conclusions

  1. Using C++ with CppCMS provides significant performance gains in developing web applications even in very basic case.
  2. The performance is effected not only by the framework itself but also by many other libraries that are in use. Using highly optimized C and C++ libraries may give significant performance gain in many cases.
  3. Such called "jit-compiled" languages as C# and Java and the frameworks based on the use significant amount of memory and still provide much lower performance then the one that can be achieved using real compiled languages like C++.
  4. It is good to remember that these benchmarks are still quite synthetic ones and in real life the actual performance depend on many factors - but using high quality and high performance libraries available for C++ have significant impact on performance.

Results Data

Markdown
---------
Miss %  CppCMS      Mono        PHP         JSP/Tomcat

0       3200.73     747.164     974.142     821.887
1       2891.2      427.727     724.173     337.736
2       2734.69     300.017     544.162     257.44
5       2285.95     162.686     301.507     130.023
10      1749.14     89.4447     174.724     68.5387
20      1247.86     47.7347     93.7919     25.7081
50      642.769     19.8311     38.979      15.1298
100     356.968     9.77116     20.1892     7.96328

Mini-Markdown
---------
Miss %  CppCMS      Mono        PHP         JSP/Tomcat

0       3103.14     763.222     1152.63     744.72
1       2933.97     728.971     1076.38     765.599
2       2944.42     726.338     1016.42     724.869
5       2804.44     661.613     866.32      822.927
10      2592.99     584.725     705.465     753.218
20      2239.03     471.576     507.021     674.488
50      1625.5      309.443     274.962     374.26
100     1156.09     197.123     159.974     164.515

HTML
-----
Miss %  CppCMS      Mono        PHP         JSP/Tomcat

0       3286.51     849.849     1147.21     808.038
1       3055.53     776.305     1137.35     748.829
2       2991.02     691.502     1122.88     693.439
5       2687.84     693.257     1074.22     756.618
10      2390.12     615.311     1016.27     604.452
20      1886.69     521.467     917.225     668.23
50      1947.93     346.672     669.693     289.656

System and Hardware

Related:

Code

The Code can be downloaded from there. note, to run it you will need to have some libraries installed and configure some hardcoded paths to make it run.

Comments

shlomil, at 10/16/10, 11:25 PM

Pretty cool results.

You know where I think this framework has great potential? In the cloud. Using this framework on Amazon EC2 (for example) would be great to boost performance and save money since you have to pay for memory/CPU resources and CppCms definitely needs less of those.

Sam, at 10/17/10, 12:38 AM

Possibly consider making your graph labels a little bit bigger and explaining the results mean; I was a little confused at what I was looking at on the graph.

UncleOxidant, at 10/17/10, 2:56 AM

Wonder how it would compare with this: http://www.sairyx.org/2010/10/running-ocaml-directly-on-top-of-xen-for-massive-performance-gains-in-the-cloud/?utm_source=twitterfeed&utm_medium=twitter

Jonathan Allrn, at 10/17/10, 2:59 AM

Why didn't you use the same markdown library for ASP.NET and CppCMS? The whole point behind the Common Language Runtime is that you are supposed to be able to use libraries from any other language.

Filip, at 10/17/10, 4:28 AM

It would be interesting to see how the new Mono 2.8 stacks up. It has a better garbage collector and LLVM backend. It also claims to have better ASP.NET performance.

anonymous, at 10/17/10, 8:49 AM

Hey kid,

your article is full of spelling and benchmarking mistakes. It is completely useless and only shows results that we already know. By the way does this comment system have spam protection?

anonymous, at 10/17/10, 8:51 AM

Seems like it doesn't have spam protection. Benchmarks won't help you if you just can't code :D

artyom, at 10/17/10, 9:52 AM

Wonder how it would compare with this ...

I hadn't try to compare to every existing technology, because it just to hard.

It would be interesting to see how the new Mono 2.8 stacks up.

Mono 2.8 was released only few weeks ago. I didn't noticed that, but I don't think that the difference will be huge. Also it is quite predictable that Java and C# implementations are quite similar.

your article is full of spelling mistakes.

I'm not English speaker, also I had a small problem with Chrome's spell checker that didn't show all errors, now it fixed, if you see more, you are welcome to tell me.

your article is full of benchmarking mistakes.

What exactly, in your opinion the mistakes in this article? I'd suggest to be constructive.

It ... shows results that we already know.

Interesting... What results you already know?

Seems like it doesn’t have spam protection.

In fact it has, but against robots and not human beings. I found it much more effecting then 1000 other spam protecting systems that WP and other has and fail on false positive.

artyom, at 10/17/10, 10:18 AM

The whole point behind the Common Language Runtime is that you are supposed to be able to use libraries from any other language.

Not exactly. The code is written in C, I can use "native" interface but what is the point? In same way I can use JNI and integrate C library in PHP, this is not the point.

The idea is to take ready components and combine it. Also note C++ is not C++/CLR ;-)

dmon, at 10/17/10, 12:10 PM

You should have used ASP.NET on IIS and not Mono, because Mono is known to be very slow compared to native microsoft implementation.

artyom, at 10/17/10, 12:35 PM

You should have used ASP.NET on IIS and not Mono

It is problematic for several reason:

  1. .Net EULA is very strict about publishing benchmarks involving it. So basically it makes almost impossible to run such benchmark.
  2. I need to run Asp.Net on Windows - that means I need to also run MySQL on windows, and it is slower there.
  3. I need same hardware with Windows and Linux preinstalled - which I don't.

, because Mono is known to be very slow compared to native microsoft implementation.

Not exactly:

  1. Mono on Windows is slower then .Net, but on Linux is much faster as it is optimized for it.
  2. Latest versions of Mono are fast and very good and run in close performance - or in some cases Mono is faster in other .Net faster.

This wasn't the case 2 years ago but now it is fine.

And actually you can see that Java and Mono have similar performance and basically Java and C# are basically same languages ;-) so I'm not surprised that they perform similarly.

Filip, at 10/17/10, 9:53 PM

I'd still argue you should try Mono 2.8 (remember to use LLVM and SGen garbage collector). While the start up time will suffer a bit, LLVM backend is known to produce code that's better optimized than Mono's fast JIT compiler. SGen GC is also better at cleaning up (comparable to the one from MS), so I'd expect to see improved performance.

Brian, at 10/18/10, 4:11 AM

I wonder what the results for php would look like compiled with hip hop. Mind giving it a shot?

ryan, at 11/2/10, 2:51 AM

thanks for posting this work.

shlomil, at 11/11/10, 4:04 PM

CPPCMS vs. node.js would be interesting as well...

Brian, at 12/6/10, 11:39 PM

ANSI C scripts run much faster than ASP.Net C#, Java or PHP, see:

http://gwan.ch/

http://gwan.ch/imgs/loan100_lan.png

And the footprint (less than 200 KB) includes the scripting engine AND the Web server.

.Net (200 MB), Java (120 MB) and PHP (80 MB) are far away in terms of memory usage and performances.

A new mechanism is in the works to make all existing socket code asynchronous even if it was programmed as blocking (great for all client libraries, like for database engines).

Daniel, at 12/13/10, 8:39 PM

Why you did the cppcms test with "Internal HTTP Server" connection instead of "Lighttpd + FastCGI" connection?

Are there significant differences between the two types of connections?

artyom, at 12/13/10, 8:45 PM

Why you did the cppcms test with "Internal HTTP Server" connection instead of "Lighttpd + FastCGI" connection?

Because Tomcat supports only HTTP connectivity and Asp.Net/Mono runs much faster using HTTP then using FastCGI connector (which is really slow in case of mono)

So I wanted to compare eggs with eggs, the only tool that had not direct HTTP support was PHP that was run behind lighttpd.

Are there significant differences between the two types of connections?

There are non negligible difference - as there IPC and additional layer is involved but still FastCGI connectivity is very fast and good.

puchuu, at 1/9/11, 1:28 PM

Now I am starting learning this great framework!

artyom, at 1/9/11, 1:58 PM

puchuu...

I'd suggest wiki from SVN "/wikipp/branches/for_cppcms_v100" branch... That uses CppCMS 1.x.x and not the old one.

stan corner, at 1/6/12, 7:34 PM

You might want to include other parameters like CPU times and the type of memory usage to have the full picture, like this article did:

http://www.codeproject.com/KB/dotnet/RuntimePerformance.aspx

rjoshi, at 4/7/12, 1:17 PM

I did quick performance test to see how CppCMS compares with Play framework or RestExpress (Netty).

http://joshitech.blogspot.com/2012/04/performance-nginx-netty-cppcms.html

artyom, at 4/7/12, 1:25 PM

What exactly did it test? It is not described.

What application? What kind of page?

Hello World? (according to the link)

Is least interesting thing. Also it is not really correct to compare static page and dynamic application.

Basically it says that CppCMS is as fast as high performance web server.

But for real tests you need real applications, not something trivial.

asava samuel, at 3/31/13, 2:27 PM

Here is another database that is compatible with Mono: http://www.kellermansoftware.com/p-43-ninja-net-database-pro.aspx

ivan, at 12/19/13, 9:20 PM

What about trying to take part at http://www.techempower.com/benchmarks/

Add Comment:

 
 the email would not displayed
 

You can write your messages using Markdown syntax.

You must enable JavaScript in order to post comments.

Pages

Categories