CppCMS Blog :: Unicode and Localization

How not to do Unicode...

All started from a small problem, how to print Unicode text to the Windows Console with option redirect to a file.

Let's say we have a program Hello that prints few words in several languages to the screen..

#include <stdio.h>

int main()
{
    printf("Мир Peace Ειρήνη\n");
    return 0;   
}

The program above is trivial and works fine under Windows if current console codepage is set to UTF-8. Also this can be fixed from the program by calling SetConsolseOutputCP(CP_UTF8).

Now simple tweak... Instead of that standard C printf we would use standard C++ std::cout... It works fine for GCC. But under Visual C++ it prints squares...

If I try redirection test.exe >test.txt - I get perfectly fine UTF-8 text...

I had started researching the issue and found the post of one of the Windows Unicode Gurus Michael Kaplan's.

I've tried to run _setmode(_fileno(stdout), _O_U8TEXT) as recommended by the Microsoft's Unicode guru and... By program crashed on attempt to write to the output stream.

Keeping searching for an answer I've got to this bug report...

Short summary:

User: Can't print UTF-8 to console with std::cout
MS: Closing - this is by design, see Michael Kaplan's article about writing to console
User: But if I do what suggested program crashes, and I still can't write Unicode to console
MS: Reactivate the ticket if it does not works
User: it does not!
MS: Use wide output...
User: I'd rather use fprintf in first place!?

To the summary...

If you use Visual C++ you can't use UTF-8 to print text to std::cout.

If you still want to, please read this amazingly long article about how to make wcout and cout working, but it does not really give a simple solution - finally falling to redefinition of the stream buffers...

So please, if you design API or Operating System, do not use kind of "Wide" API... This is is the wrong way to do Unicode.

Which reminds me... Spread around:

http://www.utf8everywhere.org/

Related Posts: http://blog.cppcms.com/post/62

The lecture slides and the poster from August Penguin 2011 conference.

I had given a lecture at August Penguin conference about Boost.Locale and presented a poster about CppCMS project.

The slides for the lecture about Boost.Locale:
- odp
- pdf
The poster pdf

CppCMS 0.99.8 and Boost.Locale 4.0.0 Rleased

New Versions of CppCMS and Boost.Locale were released.

New Features:

Boost.Locale is updated to the latest version that is going to be merged into Boost svn tree.

It includes some breaking changes:
- Redesigned boundary analysis interface:
  
  Instead of using mapping, token_iterator and break_iterator new classes that provide same functionality introduced:
  
  segment_index, boundary_point_index and the elements that can be iterated segment and boundary_point.
  
  See: http://cppcms.sourceforge.net/boost_locale/html/boundary_analysys.html
- Updated messages interface, now messages use same type of character for key and output message, i.e.
```
  std::wstring wh = translate(L"hello").str();
  std::string   h = translate( "hello").str();
```
  Instead of
```
  std::wstring wh = translate("hello").str<wchar_t>();
  std::string   h = translate("hello").str<char>();
```
  It allows to use non-US-ASCII keys transparently.
- Update date_time interface to be more consistent with Boost.DateTime and Boost.Chrono. Operations are more type safe now.
Introduced support of SunStudio Compiler on OpenSolaris.
New nightly tests: Linux Armel and Solaris/SunStudio.

Bug Fixes:

Fixed bug that virtually disabled gzip compression in CppCMS 0.99.7
Some compilation and testing fixes for older versions of Mac OS X/Darwin 8.

Note Darwin 8 is not supported due to bugs in the standard C library, but there should be no problems with newer Mac OS X versions.
Fixes to support ICU 4.8
Fixes to support gcc-4.6 and gcc-4.0
Fixes to support Python 2.3.5

Note to SVN-trunk users

Do not forget to untar the updated cppcms_boost.tar.bz2 file.

Boost.Locale was accepted into Boost

Now it is official. Boost.Locale was accepted into Boost.

So the Localization part of CppCMS would be spread all over the C++ world.

Formal Review of Boost.Locale starts today

The formal review of the Boost.Locale library starts today.

I hope it would pass the review and would be accepted as official Boost library.

It is official, Boost.Locale on its way to formal review

Now it is official, Boost.Locale is scheduled for a formal review at April 7-16.

Boost.Locale is important part of CppCMS as was developed for its needs, however I had found that this library was so important and useful that I decided to "Boostify" and make it ready for a formal review for Boost.

What does it provides:

Message formatting based on gettext dictionaries
Localized numbers, dates, currency formatting and parsing (and more)
Collation
Text manipulations like case handing and Unicode normalization
Text Boundary analysis
Support of non-Gregorian calendars like Hebrew calendar.
And much more

Most of these features are based on the state-of-the-art Unicode library ICU but it also allows to handle many of them using only standard operating system API significantly reducing its size and requirements of external components.

Most important is that is provides platform independent and uniform interface for C++ localization and internationalization tightly integrated to C++ iostreams and existing std::locale framework.

The most up-to-date version of the library and documentation would be released soon.

CppCMS 0.0.7 and 0.99.3-beta3 released

This release is security fix release for stable branch of CppCMS and both security and feature release for CppCMS 1.x.x branch.

All users are encouraged to update to latest version.

If it is not possible to upgrade don't use "hmac" session backend, switch to "aes" or server side session storage backend.

Changedlog 0.0.7

Bugfix of hmac backend: generation of signature with too small block size

Changedlog 0.99.3

Security:

Bugfix of hmac backend: generation of signature with too small block size

Features:

New version of Boost.Locale
Added support of multiple hmac cookie signatures:

Built in: hmac-md5, hmac-sha1
With libgcrypt: hmac-sha224, hmac-sha256, hmac-sha384, hmac-sha512
By default hmac now uses sha1 instead of less secure md5

Bugs:

Fixed memory leak in aes session encryptor
Fixed incorrect validation of UTF-8 encoding that could cause some illegal sequences to pass through.
Fixed missing attributes of some form widgets
Fixed incorrect code generation in templates in foreach loop
Fixed race condition when dispatch and context assignment may happen not simultaneously

Boost.Locale v3 preview version is released

I want to announce a preview of the third version of Boost.Locale:

Tutorial: http://cppcms.sourceforge.net/boost_locale/html/tutorial.html
Reference: http://cppcms.sourceforge.net/boost_locale/html/index.html
Downloads: https://sourceforge.net/projects/cppcms/files/boost_locale/

There are following significant changes:

Implemented multiple localization backends:
- icu - the default and recommended backend, based on ICU library
- std - based on C++ standard library localizations support,
- posix - based on POSIX 2008 API (newlocale, strftime_l,...)
- winapi - based on Windows API functions
Significantly simplified locale generation.
Improvements in UTF-8 handling by ICU where possible
Thread safety fixes when using ICU library
Fixed std::codecvt facet support to handle UTF-16 instead of UCS-2 only.
Removed support of compilers missing wide character support, gcc-3.4 on windows is not supported any more, latest gcc-4.x required with support of wide streams and strings, for example gcc-4.5

Tested Platforms:

Compilers: GCC (3.4, 4.2, 4.3, 4.5, 4.5/c++0x), Intel 11.0, MSVC 2008, SunStudio/stlport
Operating Systems: Linux, FreeBSD, OpenSolaris, Windows XP, Cygwin 1.7, (TODO Mac OS X)
ICU version: 3.6 to 4.4

It would be soon integrated into CppCMS 1.x.x.

First beta version of CppCMS 1.x.x is officially out!

Hello all CppCMS users.

The first beta version of CppCMS 1.x.x is available for download from the Sourceforge. The build instructions can be found here.

This version very different from CppCMS 0.0.x branch - it fixes many design flaws that had been done the previous version, it is almost 90% rewrite of the original code according to new design.

It also includes many important features

Most significant ones:

Full CppCMS core rewrite that introduced:
- Asynchronous programming support
- Removal of 3rd part libraries from the core api.
- Stable API and ABI through all major releases.
Improved Ajax support with introduction of JSON-RPC
Powerful i18n and l10n
Native Windows support including support of MSVC.
And much more...

So now CppCMS beta is ready.

Hopefully first release candidate version will be ready withing about a 3 month. And the first stable release is expected at the end of 2010 beginning of 2011.

Surviving Windows Development

One of the issues that had bothered my in CppCMS on Windows, was actually absence of full support of Unicode file-names.

It is known, that standard library functions like fopen or std::fstream are not capable of using UTF-8 encoded file names, like they are on all "normal" operating systems.

So in order to deal with such issues in transparent way, I added booster::nowide library that makes this conversion transparent. All functions in booster::nowide use UTF-8 encoded strings and convert then internally to wide strings for brain-damaged Win32 Wide API.

I also had created an implementation of std::fstream over stdio, especially for thous windows compilers that do not "extend" their standard libraries with non-standard wide-crap.

So, now CppCMS in fully Unicode capable over Win32. So if you use CppCMS with Windows, please note that all pathes are represented as UTF-8 strings and not "ANSI" one.

When Windows will finally enable UTF-8 locales? Maybe in Windows 15 (if it survive till then).

If you interested why do I hate "wide" API so much, read this.

You can download independent version of "nowide" library there:
http://art-blog.no-ip.info/files/nowide.zip