CppCMS Blog :: Benchmarks http://blog.cppcms.com/ A blog on CppCMS - C++ Web Development Framework Serving All Israeli News Web Sites from a Single EC2 instance... http://blog.cppcms.com/post/114 http://blog.cppcms.com/post/114 <div style="direction:ltr"> <p>For the last year the development of the CppCMS project was less active. The vast majority of the work hours were spent on a customer's project that used CppCMS to create an outstanding advertisement system.</p> <p>The project is called <a href="http://linicom.co.il">Linicom</a>.</p> <p>Today, when Linicom is up, running and maintained by a larger team, so I can resume the activity on the core CppCMS project itself.</p> <p>Few words about Linicom:</p> <p>Linicom is an engine that provides content and visitor sensitive advertisements for almost all large Israeli news web sites: including Ynet, Haaretz, Jerusalem Post, Mako, Walla and other significant web sites in Israel and abroad.</p> <p>Here some interesting facts:</p> <ul> <li>Linicom is based on CppCMS technology.</li> <li>The system serves around 10,000,000 <em>custom</em> requests a day, i.e ~115 req./s.</li> <li>During peak hours, it servers around 160 requests per second.</li> <li>Its typical network output is around 11 megabit per second.</li> <li>Its total memory consumption (web server, database, applications, OS) is only around 350Mb.</li> <li>The server's average CPU load is around 5%</li> <li>The server runs on a <code>c1.medium</code> Amazon EC2 instance.</li> </ul> <p>The system runs behind Lighttpd and uses PostgreSQL for persistent data storage. Also PostgreSQL is used extensively, all real time data is stored in memory.</p> <p>Almost every request needs data processing in order to provide highly customized advertisements. In technical terms it means that almost no request can be "outsourced" to a static files - every request for every customer should be processed explicitly.</p> <p>This system is probably one of the classic applications of CppCMS technology - web based system that required to be fast and efficient, being able to handle outstanding and sometimes unexpectedly changing loads without problems and provide high QoS.</p> <p>Use of in-memory data storage, caching and efficient handing of the data that can't be cached is were CppCMS shines. Having a big growth potential with a minimal required maintenance and high reliability allows the to handle the business-end safely without worrying about performance issues.</p> </div> When the memory allocation matters http://blog.cppcms.com/post/84 http://blog.cppcms.com/post/84 <div style="direction:ltr"> <p>There are many different bottle necks in C++ networking applications, many of them related to the architecture, the way system calls are used, the way application generates the data.</p> <p>However at certain point applications reach the point when memory allocation becomes the bottle neck, this is especially relevant for the cases where text strings are widely used. And this is usually the case for all networking and web applications that finally need to deal with strings.</p> <p>The old and good <code>std::string</code> has one limitation - it requires memory allocation for the chunk it represents. This is actually correct for almost any kind of string around whether they reference counted, immutable - all they allocate chunks of memory.</p> <p>Consider situation, where I need to search against path <code>/foo/bar/123</code> in some hierarchical structure like <code>std::map</code> that uses <code>std::string</code> as the key. So I need to find a path using after fetching two keys <code>foo</code> and <code>bar</code> from the path.</p> <p>So basically for the function:</p> <pre><code>void find_path(std::string const &amp;str); </code></pre> <p>The call:</p> <pre><code>find_path("/foo/bar/123"); </code></pre> <p>Would need to:</p> <ul> <li>Create a string <code>/foo/bar/123</code></li> <li>Create a sub-string <code>foo</code></li> <li>Create a sub-string <code>bar</code></li> </ul> <p>So I've got three memory allocations and this is not matter whether these strings are mutable or immutable, reference counted or not.</p> <p>CppCMS should handle such code in multiple places so how it solves this problem?</p> <ol> <li><p>Create a special object <code>cppcms::string_key</code> it holds inside a full <code>std::string</code> but also can be created explicitly from a pair of two <code>char const *</code> pointers that define a text in range <code>[begin,end)</code>.</p> <p>What is important that in in the second case object does not copy the data but only references it and I'm as a user responsible to ensure that the text range remains valid as long as I use this object.</p></li> <li><p>Now we add an overload for the function above:</p> <pre><code>void find_path(char const *str); void find_path(std::string const &amp;str); </code></pre> <p>And when we split the text into parts we use only "unowned-strings" such that creation of a strings <code>/foo/bar/123</code>, <code>foo</code>, <code>bar</code> would not require memory allocation at all.</p></li> </ol> <p>It is a general line, but there are much more interesting tricks to deal with 0 or few allocation strings and streams, like creation of a small memory pools that release all strings in once, like using on stack storage for small text chunks and much more.</p> <p>This technique is widely deployed in CppCMS 0.99.9 code and had allowed to improve performance of page generation by up to twice in some cases.</p> <p>Now these tricks should be done careful as they rely on manual memory management so, unless you do something many-many times or you detect some bottle-neck in the application still stick with <code>std::string</code> as it is usually good enough. Memory allocation today is very fast, just don't abuse it.</p> </div> Version 0.99.9 Released http://blog.cppcms.com/post/83 http://blog.cppcms.com/post/83 <div style="direction:ltr"> <h2>New Features:</h2> <ul> <li><p>Clang is support provided, CppCMS was tested against Clang 2.8.</p> <p>Now CppCMS supports 5 families of C++ compilers:</p> <ul> <li>GCC 3.4.x to 4.6.1</li> <li>Visual Studio 2005 - 2010</li> <li>Clang 2.8</li> <li>Intel 11</li> <li>Sun Studio 5.10</li> </ul> </li> <li><p>Significant performance improvements in XSS filtering by rewriting URI validation using a C++ parser rather then using complex regular expression.</p> <p>Added support of fully custom validation for HTML attributes using callback functions in the XSS filter.</p></li> <li><p>Significant performance improvements over multiple places in code by eliminating multiple memory allocations:</p> <ul> <li>HTTP, SCGI and FastCGI backends - improved memory allocation for CGI variables.</li> <li>Fetching values from JSON objects using get(...), find(...) APIs is now done with 0 memory allocation.</li> <li>URL mapping is now done with 0 or very low memory allocation.</li> <li>Various filters like <code>escape</code>, <code>urlencode</code> and some others now work with no or few memory allocations.</li> </ul> </li> <li><p>Performance improvements in caching by replacing the balanced binary tree by hash table in the primary cache key index.</p></li> </ul> <h2>Breaking Changes:</h2> <ul> <li><code>json::object</code> had changed from <code>std::map&lt;std::string,value&gt;</code> to <code>std::map&lt;string_key,value&gt;</code>. It should be fully transparent for almost all users.</li> </ul> <h2>Bugs:</h2> <ul> <li>Fixed a crash in http::response when writing HTTP headers throws due for example to incorrect file permissions.</li> <li>Fixed a bug in <code>booster::regex</code> that prevented some valid patterns to be matched against some regular expressions.</li> <li>Fixed a bug that may prevent from <code>booster::regex</code> to work on big endian 64 bit platforms</li> <li>Added initial support of Python3 for templates compiler.</li> <li>Added a workaround for systems that use python3 by default.</li> </ul> </div> CppCMS benchmarks vs Java, C#, PHP http://blog.cppcms.com/post/67 http://blog.cppcms.com/post/67 <div style="direction:ltr"> <p>Long time ago I had posted a <a href="http://art-blog.no-ip.info/cppcms/blog/post/22">benchmarks</a> comparing CppCMS based blog and PHP based one.</p> <p>I wanted to compare real life applications with each other. For a long time I had been searching for similar applications in several technologies doing very similar jobs in leading technologies: PHP, Asp.Net and Java/JSP. The last two were particularly important as they use static type system and "compiled" languages as C# and Java that are known to be faster then other dynamic typed languages like PHP, Python, Ruby and Perl popular in web development.</p> <h2>Setup</h2> <p>Unfortunately I had failed to find such application, so finally I decided to write something representative and small on my own an application with following requirements:</p> <ol> <li>Uses simple time-out based page caching</li> <li>Uses MySQL and the database and keeps open connections in pool.</li> <li>For each request access to database (if page is not cached), fetches the page content and comments for "sample article" in blog.</li> <li>Converts text to HTML using a markdown filter and displays it on page.</li> </ol> <p>I used following technologies:</p> <ul> <li><p>CppCMS:</p> <p>Version: 0.99.3<br/> MySQL Connection: dbixx/libdbi library using libmysqlclient<br/> Markdown library: <a href="http://www.pell.portland.or.us/~orc/Code/discount/">discount</a><br/> Connection: internal HTTP server</p></li> <li><p>PHP</p> <p>Version: 5.2.6<br/> MySQL Connection: internal driver<br/> Markdown library: <a href="http://michelf.com/projects/php-markdown/">PHP-Makrdown</a><br/> Connection: Lighttpd 1.4.19 + FastCGI<br/> Bytecode Cache: XCache</p></li> <li><p>Asp.Net/Mono</p> <p>Version: 2.6.7<br/> MySQL Connection: Connector/Net<br/> Markdown library: <a href="http://code.google.com/p/markdownsharp">MarkdownSharp</a><br/> Connection: internal HTTP server XSP (found to be much faster than fastcgi server)</p></li> <li><p>JSP/Tomcat</p> <p>Version - Tomcat: 6.0.18<br/> Version - Java: Sun Java 1.6.0_12<br/> MySQL Connection: Connector/J<br/> Markdown library: jmd-0.8.1<br/> Caching: oscache 2.4.1<br/> Connection: HTTP</p></li> </ul> <p>I tested following parameters:</p> <ul> <li>Pages per seconds generation for different cache hit/miss ratio: stating from 0% miss ratio up to 100% miss ratio.</li> <li>Memory usage</li> </ul> <p>For each test the application was "warmed up" with 100 requests to fill the cache, and then 1000 request with max concurrency of 5 request are done, while certain percent of them is new pages and the other are taken from "warmed up" once.</p> <h2>Notes:</h2> <p>I used the fastest Markdown implementation I had found.</p> <p>C# implementation is the same one that <a href="http://stackoverflow.com">http://stackoverflow.com</a> uses - it is actually heavily optimized implementation based on older C# implementation</p> <p>The Java implementation is based on the above C# and the fastest one I had found.</p> <p>Discount is the fastest C implementation of markdown that I had found.</p> <h2>Results:</h2> <p><img src="http://cppcms.com/files/test1-small.png" alt="Benchmarks Markdown" /></p> <p><img src="http://cppcms.com/files/mem-use.png" alt="Memory" /></p> <h2>Summary</h2> <ol> <li>C#, Java and PHP implementation behave very similarly and without significant differences.</li> <li>The memory usage of Java/Tomcat and Mono/Asp.Net was significantly higher - up one or two orders of magnitude in comparison to CppCMS and PHP</li> <li>Surprisingly PHP behaves very well, in comparison to "compiled" languages like Java and C#.</li> </ol> <h2>Revisiting</h2> <p>After doing some profiling it was clear that C implementation of Markdown was significantly faster then all other implementations. So I decided to create my own mini-markdown that make some basic handing of titles, lists, paragraphs and quotes at one level only. That is very simple syntax but implemented similarly in all 4 languages using same algorithm.</p> <p>The results were following:</p> <p><img src="http://cppcms.com/files/test3-small.png" alt="Benchmarks Markdown" /></p> <p>The difference between CppCMS and other implementations was still significant but still much smaller then the difference between real markdown implementation. So the performance difference was less dramatic.</p> <h2>2nd Revision</h2> <p>And in the last revision I decided not to use any text filters by fetch ready HTML formatted content from DB and display it on the web as is.</p> <p>Such comparison actually profile the most basic stuff:</p> <ol> <li>Caching</li> <li>SQL Connection</li> <li>Request/Response handling</li> </ol> <p>And would ignore hundreds lines of code used in any web applications responsible for the actual business logic.</p> <p><img src="http://cppcms.com/files/sql-only.png" alt="Benchmarks HTML" /></p> <h2>Conclusions</h2> <ol> <li>Using C++ with CppCMS provides significant performance gains in developing web applications even in <strong>very</strong> basic case.</li> <li>The performance is effected not only by the framework itself but also by many other libraries that are in use. Using highly optimized C and C++ libraries may give significant performance gain in many cases.</li> <li>Such called "jit-compiled" languages as C# and Java and the frameworks based on the use significant amount of memory and still provide much lower performance then the one that can be achieved using real compiled languages like C++.</li> <li>It is good to remember that these benchmarks are still quite synthetic ones and in real life the actual performance depend on many factors - but using high quality and high performance libraries available for C++ have significant impact on performance.</li> </ol> <h2>Results Data</h2> <pre><code>Markdown --------- Miss % CppCMS Mono PHP JSP/Tomcat 0 3200.73 747.164 974.142 821.887 1 2891.2 427.727 724.173 337.736 2 2734.69 300.017 544.162 257.44 5 2285.95 162.686 301.507 130.023 10 1749.14 89.4447 174.724 68.5387 20 1247.86 47.7347 93.7919 25.7081 50 642.769 19.8311 38.979 15.1298 100 356.968 9.77116 20.1892 7.96328 Mini-Markdown --------- Miss % CppCMS Mono PHP JSP/Tomcat 0 3103.14 763.222 1152.63 744.72 1 2933.97 728.971 1076.38 765.599 2 2944.42 726.338 1016.42 724.869 5 2804.44 661.613 866.32 822.927 10 2592.99 584.725 705.465 753.218 20 2239.03 471.576 507.021 674.488 50 1625.5 309.443 274.962 374.26 100 1156.09 197.123 159.974 164.515 HTML ----- Miss % CppCMS Mono PHP JSP/Tomcat 0 3286.51 849.849 1147.21 808.038 1 3055.53 776.305 1137.35 748.829 2 2991.02 691.502 1122.88 693.439 5 2687.84 693.257 1074.22 756.618 10 2390.12 615.311 1016.27 604.452 20 1886.69 521.467 917.225 668.23 50 1947.93 346.672 669.693 289.656 </code></pre> <h2>System and Hardware</h2> <ul> <li>OS: Linux, Debian Lenny, 64 bit</li> <li>Hardware: AMD Athlon XP 3000, 64 bit, 1GB memory</li> </ul> <h2>Related:</h2> <ul> <li><a href="http://www.reddit.com/r/programming/comments/ds69u/web_development_benchmarks_of_ccppcms_vs_php/">Comments on Reddit</a></li> <li><a href="http://art-blog.no-ip.info/cppcms/blog/post/22">CppCMS Based Blog vs WordPress Benchmarks</a></li> </ul> <h2>Code</h2> <p>The <a href="http://cppcms.com/files/benchmarks-code.tar.gz">Code</a> can be downloaded from there. note, to run it you will need to have some libraries installed and configure some hardcoded paths to make it run.</p> <p> <a href="/post/67">more...</a> </p> </div> Asp.Mono in Linux? Not Yet... http://blog.cppcms.com/post/27 http://blog.cppcms.com/post/27 <div style="direction:ltr"> <p>During development of CppCMS I have always wanted to do some benchmarks against one of the most popular web development technologies: Asp.Net. I had found a blogging system: <a href="http://www.dotnetblogengine.net/post/BlogEngineNET-145-released-today.aspx">BlogEngine.Net</a> that in its latest version had full support of mono under Linux, support of MySQL --- all I need to run benchmarks against CppCMS.</p> <p>The beginning was promising. The instructions were simple, there was an actual blog running BE.Net under Apache mod_mono on Ubuntu. There were <a href="http://blog.ruski.co.za/page/Install-BlogEngineNET-on-Ubuntu.aspx">Linux specific</a> instructions as well.</p> <p>So, first of all I had installed mono 1.9.1 from Etch backports. The first problem I had to deal with was an installation problem --- the version of C# compiler and mono environment were different. This was solved quite simply. So, I could finally see BE.Net running under Mono using XML as data storage backend.</p> <p>Then, I wanted to add MySQL storage backend. The problems had come very soon. The MySQL database script was written under assumption that table names are case insensitive. That was not true for MySQL under <em>Linux</em>. This problem was fixed.</p> <p> <a href="/post/27">more...</a> </p> </div> CppCMS vs WordPress http://blog.cppcms.com/post/22 http://blog.cppcms.com/post/22 <div style="direction:ltr"> <h3>Setup</h3> <p>I had compared two blog systems: this one and WordPress 2.5 with a patched WP-Cache-2 addon. I used following configuration:</p> <ol> <li>Web Server lighttpd 1.4.13</li> <li>Interface FastCGI</li> <li>PHP 5.2</li> <li>Bytecode cacher: XCache 1.2.1</li> <li>Database MySQL 5.0</li> <li>Caching for WP: WP-Cache-2 with an additional <a href="http://art-blog.no-ip.info/cppcms/blog/post/20">performance patch</a></li> <li>Hardware: AMD Athlon XP 64bit, 1G RAM</li> <li>OS: Linux, Debian Etch 64bit.</li> </ol> <p>I prepared two blogs that were filled up with 1000 articles each. Each article had 10 comments, all the articles were organized in 10 categories in each blog.</p> <p> <a href="/post/22">more...</a> </p> </div> Patch For WP-Cache-2 plugin http://blog.cppcms.com/post/20 http://blog.cppcms.com/post/20 <div style="direction:ltr"> <p>I'm going to run a heavy benchmarks comparing WordPress -- the blog system I know very well, with CppCMS based blog -- the system I had written.</p> <p>The new caching system that was developed for CppCMS is quite smart, it stores the entry pages twice: original and gzip compressed. On heavy loads, this allows serving pages significantly faster because only thing that should be done is to push html or compressed html page directly from the cache. Otherwise, gzip compression (even fastest) would take lots of resources and reduces a preformace of the system.</p> <p>When it comes to benchmarks, I had discovered that WP-Cache-2 plugin does the job well, but it caches only html version of the file, thus, even if the page is cached it still must pass a compression by Apache's mod_deflate or by PHP engine itself.</p> <p>I had patched this plugin and now it stores two versions of same page: an original and compressed. and was able to get 60% performace improvement.</p> <ul> <li>WordPress native plugin: 450 requests per second</li> <li>WordPress patched plugin: 720 requests per second</li> </ul> <p>So after this patch I can feel that the benchmarks would be proper, because without it this would be incorrect to compare time required for fetching a cache with the time required for compressing entry page.</p> <p>Links:</p> <ul> <li><a href="http://mnm.uib.es/gallir/wp-cache-2/">WP Cache 2</a></li> <li><a href="http://cppcms.com/files/wp-cache.patch">Patch</a></li> </ul> <p><strong>N.B.:</strong> The full benchmarks coming soon</p> </div> New Templates System http://blog.cppcms.com/post/14 http://blog.cppcms.com/post/14 <div style="direction:ltr"> <p>New templates system was introduces to the CppCMS framework. It is based on ideas of dynamic typed languages inside static typed C++.</p> <p>The original template system had several problems:</p> <ol> <li>The each template variable was referenced by and integer key that was generated during compilation of templates.</li> <li>The rendering process required from the developer some kind of activity -- update content values according to requests from rendering engine.</li> <li>The values of the entries where limited to string, integer and boolean values.</li> </ol> <p>In any case, the design of the first template system was just unacceptable, thus new template system was build.</p> <p>It introduced following features:</p> <ul> <li>Dynamic typed variable values using boost::any. They allow assigning of any kind of objects to the variables and rendering them to the templates using custom engines.</li> <li>All the variables are references by their names.</li> <li>Content now has hierarchical structure when each variable can include list of items or callbacks that allow one step template rendering.</li> <li>The design of the engine is now much more modular.</li> </ul> <p>Additional features I'm still working on them are:</p> <ul> <li>Support of different filters like "html escaping", "urlizing" etc.</li> <li>Support of custom filters, including filter chains.</li> <li>Support of localization and translation.</li> </ul> <p> <a href="/post/14">more...</a> </p> </div> SOCI or DBI, stage 2 http://blog.cppcms.com/post/13 http://blog.cppcms.com/post/13 <div style="direction:ltr"> <p>Discussing soci <a href="http://sourceforge.net/mailarchive/forum.php?forum_name=soci-users">performance</a> with its developers I had found that soci is compiled without any optimizations by default.</p> <p>So, after recompiling soci with -O2 option I've got much better results. Simple comparison of dbixx and soci had given very close result. I had run my tests once more and got following results -- pages per second (no gzip): DB soci dbixx -------------------- MySQL 710 800 Sqlite3 550 410 PgSQL 385 430</p> <p>We can clearly see that for MySQL and PostgreSQL dbixx is still faster in about 10% however in case of Sqlite3 dbixx is significantly slower (25%).</p> <p>So it seems to me that both solutions are quite reasonable to use without clear advantage of one over another.</p> </div> SOCI or DBI http://blog.cppcms.com/post/12 http://blog.cppcms.com/post/12 <div style="direction:ltr"> <p>One of the problematic issues in writing cross-SQL code is an API that differs from one SQL to another.</p> <p>There are two open source libraries that provide unified API</p> <ol> <li><a href="http://soci.sourceforge.net">soci</a> -- C++ library.</li> <li><a href="http://libdbi.sourceforge.net">dbi</a> -- C library.</li> </ol> <p>At the first point I had chosen soci as native solution of C++ programmer. After running some benchmarks on the new version of this blog I had found <a href="http://art-blog.no-ip.info/cppcms/blog/post/11">20% performance</a> reduction for MySQL database. But I also remembered that there should be <a href="http://art-blog.no-ip.info/cppcms/blog/post/8">negligible difference</a> between MySQL and Berkeley DB. This was mostly due to incorrect design of my BDB database layout I had done.</p> <p>That had seem to be strange and I stared benchmarking the system more and more.</p> <p> <a href="/post/12">more...</a> </p> </div>