CppCMS benchmarks vs Java, C#, PHP
Long time ago I had posted a benchmarks comparing CppCMS based blog and PHP based one.
I wanted to compare real life applications with each other. For a long time I had been searching for similar applications in several technologies doing very similar jobs in leading technologies: PHP, Asp.Net and Java/JSP. The last two were particularly important as they use static type system and "compiled" languages as C# and Java that are known to be faster then other dynamic typed languages like PHP, Python, Ruby and Perl popular in web development.
Setup
Unfortunately I had failed to find such application, so finally I decided to write something representative and small on my own an application with following requirements:
- Uses simple time-out based page caching
- Uses MySQL and the database and keeps open connections in pool.
- For each request access to database (if page is not cached), fetches the page content and comments for "sample article" in blog.
- Converts text to HTML using a markdown filter and displays it on page.
I used following technologies:
CppCMS:
Version: 0.99.3
MySQL Connection: dbixx/libdbi library using libmysqlclient
Markdown library: discount
Connection: internal HTTP serverPHP
Version: 5.2.6
MySQL Connection: internal driver
Markdown library: PHP-Makrdown
Connection: Lighttpd 1.4.19 + FastCGI
Bytecode Cache: XCacheAsp.Net/Mono
Version: 2.6.7
MySQL Connection: Connector/Net
Markdown library: MarkdownSharp
Connection: internal HTTP server XSP (found to be much faster than fastcgi server)JSP/Tomcat
Version - Tomcat: 6.0.18
Version - Java: Sun Java 1.6.0_12
MySQL Connection: Connector/J
Markdown library: jmd-0.8.1
Caching: oscache 2.4.1
Connection: HTTP
I tested following parameters:
- Pages per seconds generation for different cache hit/miss ratio: stating from 0% miss ratio up to 100% miss ratio.
- Memory usage
For each test the application was "warmed up" with 100 requests to fill the cache, and then 1000 request with max concurrency of 5 request are done, while certain percent of them is new pages and the other are taken from "warmed up" once.
Notes:
I used the fastest Markdown implementation I had found.
C# implementation is the same one that http://stackoverflow.com uses - it is actually heavily optimized implementation based on older C# implementation
The Java implementation is based on the above C# and the fastest one I had found.
Discount is the fastest C implementation of markdown that I had found.
Results:
Summary
- C#, Java and PHP implementation behave very similarly and without significant differences.
- The memory usage of Java/Tomcat and Mono/Asp.Net was significantly higher - up one or two orders of magnitude in comparison to CppCMS and PHP
- Surprisingly PHP behaves very well, in comparison to "compiled" languages like Java and C#.
Revisiting
After doing some profiling it was clear that C implementation of Markdown was significantly faster then all other implementations. So I decided to create my own mini-markdown that make some basic handing of titles, lists, paragraphs and quotes at one level only. That is very simple syntax but implemented similarly in all 4 languages using same algorithm.
The results were following:
The difference between CppCMS and other implementations was still significant but still much smaller then the difference between real markdown implementation. So the performance difference was less dramatic.
2nd Revision
And in the last revision I decided not to use any text filters by fetch ready HTML formatted content from DB and display it on the web as is.
Such comparison actually profile the most basic stuff:
- Caching
- SQL Connection
- Request/Response handling
And would ignore hundreds lines of code used in any web applications responsible for the actual business logic.
Conclusions
- Using C++ with CppCMS provides significant performance gains in developing web applications even in very basic case.
- The performance is effected not only by the framework itself but also by many other libraries that are in use. Using highly optimized C and C++ libraries may give significant performance gain in many cases.
- Such called "jit-compiled" languages as C# and Java and the frameworks based on the use significant amount of memory and still provide much lower performance then the one that can be achieved using real compiled languages like C++.
- It is good to remember that these benchmarks are still quite synthetic ones and in real life the actual performance depend on many factors - but using high quality and high performance libraries available for C++ have significant impact on performance.
Results Data
Markdown
---------
Miss % CppCMS Mono PHP JSP/Tomcat
0 3200.73 747.164 974.142 821.887
1 2891.2 427.727 724.173 337.736
2 2734.69 300.017 544.162 257.44
5 2285.95 162.686 301.507 130.023
10 1749.14 89.4447 174.724 68.5387
20 1247.86 47.7347 93.7919 25.7081
50 642.769 19.8311 38.979 15.1298
100 356.968 9.77116 20.1892 7.96328
Mini-Markdown
---------
Miss % CppCMS Mono PHP JSP/Tomcat
0 3103.14 763.222 1152.63 744.72
1 2933.97 728.971 1076.38 765.599
2 2944.42 726.338 1016.42 724.869
5 2804.44 661.613 866.32 822.927
10 2592.99 584.725 705.465 753.218
20 2239.03 471.576 507.021 674.488
50 1625.5 309.443 274.962 374.26
100 1156.09 197.123 159.974 164.515
HTML
-----
Miss % CppCMS Mono PHP JSP/Tomcat
0 3286.51 849.849 1147.21 808.038
1 3055.53 776.305 1137.35 748.829
2 2991.02 691.502 1122.88 693.439
5 2687.84 693.257 1074.22 756.618
10 2390.12 615.311 1016.27 604.452
20 1886.69 521.467 917.225 668.23
50 1947.93 346.672 669.693 289.656
System and Hardware
- OS: Linux, Debian Lenny, 64 bit
- Hardware: AMD Athlon XP 3000, 64 bit, 1GB memory
Related:
Code
The Code can be downloaded from there. note, to run it you will need to have some libraries installed and configure some hardcoded paths to make it run.
more...CppCMS 0.0.7 and 0.99.3-beta3 released
This release is security fix release for stable branch of CppCMS and both security and feature release for CppCMS 1.x.x branch.
All users are encouraged to update to latest version.
If it is not possible to upgrade don't use "hmac" session backend, switch to "aes" or server side session storage backend.
Changedlog 0.0.7
- Bugfix of hmac backend: generation of signature with too small block size
Changedlog 0.99.3
Security:
- Bugfix of hmac backend: generation of signature with too small block size
Features:
- New version of Boost.Locale
Added support of multiple hmac cookie signatures:
Built in: hmac-md5, hmac-sha1
With libgcrypt: hmac-sha224, hmac-sha256, hmac-sha384, hmac-sha512
By default hmac now uses sha1 instead of less secure md5
Bugs:
- Fixed memory leak in aes session encryptor
- Fixed incorrect validation of UTF-8 encoding that could cause some illegal sequences to pass through.
- Fixed missing attributes of some form widgets
- Fixed incorrect code generation in templates in
foreachloop - Fixed race condition when dispatch and context assignment may happen not simultaneously
Boost.Locale v3 preview version is released
I want to announce a preview of the third version of Boost.Locale:
- Tutorial: http://cppcms.sourceforge.net/boost_locale/html/tutorial.html
- Reference: http://cppcms.sourceforge.net/boost_locale/html/index.html
- Downloads: https://sourceforge.net/projects/cppcms/files/boost_locale/
There are following significant changes:
- Implemented multiple localization backends:
- icu - the default and recommended backend, based on ICU library
- std - based on C++ standard library localizations support,
- posix - based on POSIX 2008 API (newlocale, strftime_l,...)
- winapi - based on Windows API functions
- Significantly simplified locale generation.
- Improvements in UTF-8 handling by ICU where possible
- Thread safety fixes when using ICU library
- Fixed std::codecvt facet support to handle UTF-16 instead of UCS-2 only.
- Removed support of compilers missing wide character support, gcc-3.4 on windows is not supported any more, latest gcc-4.x required with support of wide streams and strings, for example gcc-4.5
Tested Platforms:
- Compilers: GCC (3.4, 4.2, 4.3, 4.5, 4.5/c++0x), Intel 11.0, MSVC 2008, SunStudio/stlport
- Operating Systems: Linux, FreeBSD, OpenSolaris, Windows XP, Cygwin 1.7, (TODO Mac OS X)
- ICU version: 3.6 to 4.4
It would be soon integrated into CppCMS 1.x.x.
CppCMS 0.99.2 - Beta 2 is released
Update: Please take version 0.99.2.1
The second beta is availible for download:
Changes from Beta 1.
- Significant performance improvements in Booster.Aio
- Significant performance improvment in FastCGI backend.
- Important bug fixes in file upload handling.
- Fixed response/status handling in synchronous and asynchronous API.
- Fixed incorrect numbers handling in JSON.
- Added more examples
- Various platform related fixes.
First beta version of CppCMS 1.x.x is officially out!
Hello all CppCMS users.
The first beta version of CppCMS 1.x.x is available for download from the Sourceforge. The build instructions can be found here.
This version very different from CppCMS 0.0.x branch - it fixes many design flaws that had been done the previous version, it is almost 90% rewrite of the original code according to new design.
It also includes many important features
Most significant ones:
Full CppCMS core rewrite that introduced:
- Asynchronous programming support
- Removal of 3rd part libraries from the core api.
- Stable API and ABI through all major releases.
- Improved Ajax support with introduction of JSON-RPC
- Powerful i18n and l10n
- Native Windows support including support of MSVC.
- And much more...
So now CppCMS beta is ready.
Hopefully first release candidate version will be ready withing about a 3 month. And the first stable release is expected at the end of 2010 beginning of 2011.
