Implementing xlocale APIs Project Update

The following project update was written by David Chisnall who received a grant from us to implement xlocale APIs to enable porting libc++. We're pleased that the project is almost completed! It's traditional to start this sort of thing by telling you who I am. I started using FreeBSD around 2001. At the time, I'd used Linux but switched to FreeBSD because it sounded like it worked correctly - I could have xmms playing music, my IM and email clients notifying me of new messages, and BZFLag making gunshot noises all at the same time. Apparently, ten years later, this still doesn't work reliably on Linux...
I got involved with clang via a somewhat indirect route. I'm a member of the core team of the Étoilé project, which aims to build a (BSD licensed) desktop environment on top of GNUstep. I grew increasingly frustrated with the level of Objective-C support in GCC, which included shipping one release with Objective-C completely broken and displaying no progress towards supporting the Objective-C 2 extensions that were about 5 years old at the time. I looked at the code, but it was an incomprehensible mess of spaghetti.

Apple had just released a new compiler front end (clang) that had Objective-C parsing mostly finished, but code generation missing. I started poking the clang code to try to support the GCC Objective-C runtime, and a few weeks later had a working Objective-C 2 compiler. I then grew frustrated with the limitations (including the license) of the GCC Objective-C runtime and wrote a more modern (MIT licensed) replacement. Clang now supports both and with the new runtime is at feature parity with Apple's implementation of the language.

While hacking on clang - which I do on FreeBSD - I fixed various FreeBSD-related bugs. This put me in contact with Roman Divacky, who had been working on importing clang into the base system. This is an important task, because FreeBSD currently uses the last GPLv2 version of GCC as the system compiler. Although this release seems less buggy than subsequent ones, it is now over 4 years old and is no longer supported upstream. It won't, for example, get any of the features of C1x or C++11.

The compiler is only half of the problem. The other half is the standard library. For C, this isn't an issue: FreeBSD has had its own C standard library implementation since before it was FreeBSD. For C++, it's a bigger problem. FreeBSD currently ships with GNU libstdc++, which has undergone the same sort of license change as GCC, leaving FreeBSD stuck with an old version.

A good candidate for replacement is libc++, developed as part of the LLVM project and available under UIUC and MIT licenses. This has a few dependencies. One is a low-level C++ ABI library, which implements the dynamic parts of C++ such as exception handling and run-time type information. I'd written an implementation of this for PathScale, and the FreeBSD and NetBSD Foundations jointly paid for it to be open sourced. I've since extended it with some additional features required to support C++11, which includes an std::exception_ptr object that allows exceptions to be passed between threads.

The other dependency is the C standard library. Libc++ was written by Apple developers (Apple is in the same situation as FreeBSD with regards to the GPLv3) and so uses some features of Darwin libc that are not portable. Specifically, Darwin libc has a convenient set of extensions to localisation: xlocale. This extends the POSIX 2008 per-thread locale APIs (missing in FreeBSD) to provide a set of _l variants of locale-aware libc functions that use a specific locale, rather than the global one.

My recent work, sponsored by the FreeBSD Foundation, has been to implement the missing xlocale APIs. This is now mostly done and pending code review. With this and the new tweaks to libcxxrt, it's now possible to build libc++ on FreeBSD and most of the tests pass.

Most of the remaining test failures are in the header. This defines a lot of complex atomic operations and requires a lot of compiler support. Eli Friedman has been working on adding this support in clang, and with his latest patch applied 25 of the 52 atomic tests pass. There are still a few remaining failures:

- 27 caused by clang not fully supporting the atomic operations yet
- 3 caused by clang not fully supporting the C++11 type-trait intrinsics
- 20 that I don't think are real failures - they're caused by the VM where I'm running the tests not having sufficiently fine-grained time reporting for the thread operation timeout tests to work properly
- 1 is caused by FreeBSD lacking the C1x quick_exit() APIs.
- 2 caused by FreeBSD lacking the uchar.h header

For comparison, Howard Hinnant, the libc++ lead developer, just sent me a list of the failures on OS X. On FreeBSD, 4271 tests pass, 53 fail. On OS X 4253 pass, 71 fail. This is looking very promising for an entirely GNU-free C++ stack.

DIFFUSE for FreeBSD

The FreeBSD Foundation is pleased to announce that Swinburne University of Technology's Centre for Advanced Internet Architectures has been awarded a grant to implement DIFFUSE for FreeBSD.

DIFFUSE (Distributed Firewall and Flow-shaper Using Statistical Evidence) is an extension to the FreeBSD IPFW firewall subsystem developed by CAIA. It allows IPFW to classify traffic based on statistical properties of flows being observed in realtime, and instantiate network actions across a distributed set of "action nodes" for particular flows if required.

This project will tidy up and integrate the existing DIFFUSE prototype into FreeBSD, and incorporate a number of important new features. Integration of DIFFUSE into FreeBSD will increase FreeBSD's utility to designers and implementers of FreeBSD-based networking infrastructure.

Network architects frequently require the ability to classify different traffic types flowing across a network, typically using packet inspection capabilities of base system tools such as ipfw and pf. Traffic classification then enables the provision of customized service levels to different traffic types (such as priority packet queuing and forwarding, or allocation of specific bandwidth guarantees).

DIFFUSE uses machine learning techniques to enable robust and efficient classification of IP traffic flows based on their unique statistical properties in addition to traditional inspection of packet header or payload contents. DIFFUSE also allows traffic classification to occur in one place (e.g. in the core of a network) and trigger traffic shaping and differentiation elsewhere (e.g. at the edges of a network). DIFFUSE has applications in ISP, residential broadband and large corporate network scenarios to name a few.

The project will conclude the end of October 2011.

Implementing xlocale APIs

The FreeBSD Foundation is pleased to announce that David Chisnall has been awarded a grant to implement xlocale APIs to enable porting libc++.

The C standard library (libc) is one of the most important parts of a UNIX system as most programs interact with the kernel through interfaces written in C. Porting code between platforms with similar libc implementations is trivial and if something is supported by libc, higher-level languages can use it without being reimplemented.

Over time, the C language has slowly evolved to modern multicore systems, but there are still some places that are problematic. One of these is localization as C began originally had no localization support. FreeBSD libc and Darwin libc (used by Mac OS X) are similar, making it much easier to port code from OS X to FreeBSD than from OS X to Linux. The libc used by OS X supports a set of extended locale functions (xlocale) that allow locale to be set on a per-thread basis.

Additionally, libc++, from the LLVM project, was originally developed on Darwin, so it uses xlocale for most of the C++ locale support. The lack of this support is the primary obstacle to porting it to FreeBSD.

Once xlocale is supported in FreeBSD libc, we can port libc++ to FreeBSD, giving us an MIT-licensed C++11 standard library implementation. This, in conjunction with Clang and libcxxrt, means that the entire C++ stack in FreeBSD will be free of any GNU code. This leaves the linker as the only significant obstacle to a GPL-free FreeBSD 10.

The project will conclude the end of September 2011.

Participate in Software Freedom Day

This Saturday, September 17, is Software Freedom Day (SFD). SFD is an annual global event that encourages open source software users to reach out to their local community to educate others about the benefits of using open source.

Frederic Muller, President of SFI, the non-profit organization behind Software Freedom Day, has been very helpful in encouraging FreeBSD users to participate in SFD. FreeBSD is listed as a partner on the SFD website. In addition, the FreeBSD logo is included on the cover letter and a copy of PC-BSD was included with the 210 packages that were shipped to the pre-registered teams. He also added the FreeBSD news RSS feed to planet SFD so that other SFD participants will get FreeBSD updates.

Julian H. Stacey has also been helpful in spreading the word and will be participating with the Munich team.

BSD user groups are encouraged to put together a team and to register it on the SFD wiki. If you have not participated in SFD before, take a moment to read through the Start Guide for instructions on how to promote your team. If you do register a team, leave a comment so that we can mention it on the @bsdevents twitter feed.