Trip Report: Marius Strobl

The EuroBSDCon trip reports are starting to arrive and will be posted here. The first report is from Marius Strobl:

The generous sponsorship of the FreeBSD Foundation for the first time allowed me to attend EuroBSDCon'11 as well as the associated Developer Summit in its entirety (so far I could only attend one day of last year's EuroBSDCon Developer Summit). Having the chance to take part in the busdma(9), for which Marcel Moolenaar and myself had a preparatory email conversation, and the bmake working groups on the first day was really helpful. Having done work for the sparc64 port and machine independent network and storage controller drivers, I came across several limitations in the current implementation of busdma(9). Other architectures will benefit from the proposed busdma(9) overhaul. Switching to a bmake based build infrastructure will provide better and optimized cross-compilation of FreeBSD on other operating systems. While this isn't something that I have a direct need for, it will come in handy should the rest of the FreeBSD developers decide to remove the GPL'ed toolchain in the base for all platforms in favor of LLVM/clang, which so far doesn't support 64-bit SPARC v9. The bmake based build infrastructure will provide a backup plan by allowing the use of an external GCC and binutils via its external toolchain support. Surprisingly, there was next to no opposition to the plans of these two working groups. Collaboration on the bmake build infrastructure and busdma(9) overhaul working groups will take place via the FreeBSD Wiki.

The working group presentations and the brainstorming sessions on the second day of the Developer Summit provided a good overview of what's going on and is planned in the FreeBSD project beyond the src base, including the infrastructure and the ports collection. Both the first and the second day of the Developer Summit as well as "day zero" were concluded by nice dinners which allowed one to make contacts with fellow FreeBSD developers, especially doc and ports committers, which I as a src committer otherwise don't have that much contact with, even online.

On the third day of the Developer Summit, which also was the first official day of EuroBSDCon'11, I gave a short talk about the status of the FreeBSD/sparc64 port. Based on the feedback that I received, the presentation was well received. The rest of that conference day I spent mainly attending the rest of the Developer Summit track, which was an official part of EuroBSDCon'11 and also attracted some non-FreeBSD committers. This again provided me with a good overview of what else is going in FreeBSD. The pkgng project seems like an especially worthy addition. Slides of the talks given in that track are available here. That day was closed by a nice social event at the Utrecht Railway Museum, again giving good opportunities to make contacts with fellow developers.

The second day of EuroBSDCon'11, which no longer hosted Developer Summit events,had some interesting talks, of which the "OpenBSD/sun4v: Porting OpenBSD to Sun's UltraSPARC T1 and T2 processors" by Mark Kettenis turned out to be especially useful as it gave me 2-3 new ideas about how to address integration of sun4v support into FreeBSD/sparc64. A short chat with Mark afterwards yielded helpful advice for adding support for the line of Fujitsu/Oracle Mx000 Enterprise servers to FreeBSD/sparc64.

I'm especially thankful that the Developer Summit and EuroBSDCon'11 for the first time allowed me to meet my former mentor Marcel Moolenaar in person. Other interesting contacts I've made and topics relevant to the FreeBSD project I've talked about with them were:

• Jonathan Anderson
• Gavin Atkinson (Solaris binary-compatibility for FreeBSD/sparc64)
• Baptiste Daroussin (pkgng)
• Pawel Dawidek (ZFS)
• Wilko Bulte (donate a dual Sun SCSI-HBA and will try to get me disks for a machine used for package building and for my development machine)
• Beat Gaetzi (will try to get me UltraSPARC T1/T2 hardware)
• Justin Gibbs (fixing issues revealed by a recent commit of Kenneth Merry regarding sense residual handling in Justin's ahc(4) and ahd(4) drivers)
• Alexander Motin (NCQ timeouts with ahci(4))
• Gabor Pali (seems like I've even managed to catch his cold ...)
• Daniel Seuffert (will try to get me UltraSPARC T1/T2 and SPARC64 VI/VII hardware)
• Ed Shouten (thankfully agreed to adapt/fix rp(4) to his TTY-rework if I send him the respective card)
• Shteryana Shopova
• Florian Smeets
• Bjoern Zeeb

While at the conference and during the trips to and from Maarssen I managed to finally commit support for SCHED_ULE to sparc64 and vice versa, which should also provide a tiny performance improvement on other architectures, commit fixes for two PRs, clean up some PHY device driver stuff Warner Losh left behind, fix fallout like the ahc(4)/ahd(4) issue mentioned above in other HBA drivers, port the NetBSD MII bitbang'ing code and convert most of the network device drivers that can take advantage of it to use it, and fix a real bug that is present in 153 device drivers.

All in all, I think it was fruitful for me to attend EuroBSDCon'11 and the associated Developer Summit as a member of the FreeBSD project and community.

Implementing xlocale APIs Project Update

The following project update was written by David Chisnall who received a grant from us to implement xlocale APIs to enable porting libc++. We're pleased that the project is almost completed! It's traditional to start this sort of thing by telling you who I am. I started using FreeBSD around 2001. At the time, I'd used Linux but switched to FreeBSD because it sounded like it worked correctly - I could have xmms playing music, my IM and email clients notifying me of new messages, and BZFLag making gunshot noises all at the same time. Apparently, ten years later, this still doesn't work reliably on Linux...
I got involved with clang via a somewhat indirect route. I'm a member of the core team of the Étoilé project, which aims to build a (BSD licensed) desktop environment on top of GNUstep. I grew increasingly frustrated with the level of Objective-C support in GCC, which included shipping one release with Objective-C completely broken and displaying no progress towards supporting the Objective-C 2 extensions that were about 5 years old at the time. I looked at the code, but it was an incomprehensible mess of spaghetti.

Apple had just released a new compiler front end (clang) that had Objective-C parsing mostly finished, but code generation missing. I started poking the clang code to try to support the GCC Objective-C runtime, and a few weeks later had a working Objective-C 2 compiler. I then grew frustrated with the limitations (including the license) of the GCC Objective-C runtime and wrote a more modern (MIT licensed) replacement. Clang now supports both and with the new runtime is at feature parity with Apple's implementation of the language.

While hacking on clang - which I do on FreeBSD - I fixed various FreeBSD-related bugs. This put me in contact with Roman Divacky, who had been working on importing clang into the base system. This is an important task, because FreeBSD currently uses the last GPLv2 version of GCC as the system compiler. Although this release seems less buggy than subsequent ones, it is now over 4 years old and is no longer supported upstream. It won't, for example, get any of the features of C1x or C++11.

The compiler is only half of the problem. The other half is the standard library. For C, this isn't an issue: FreeBSD has had its own C standard library implementation since before it was FreeBSD. For C++, it's a bigger problem. FreeBSD currently ships with GNU libstdc++, which has undergone the same sort of license change as GCC, leaving FreeBSD stuck with an old version.

A good candidate for replacement is libc++, developed as part of the LLVM project and available under UIUC and MIT licenses. This has a few dependencies. One is a low-level C++ ABI library, which implements the dynamic parts of C++ such as exception handling and run-time type information. I'd written an implementation of this for PathScale, and the FreeBSD and NetBSD Foundations jointly paid for it to be open sourced. I've since extended it with some additional features required to support C++11, which includes an std::exception_ptr object that allows exceptions to be passed between threads.

The other dependency is the C standard library. Libc++ was written by Apple developers (Apple is in the same situation as FreeBSD with regards to the GPLv3) and so uses some features of Darwin libc that are not portable. Specifically, Darwin libc has a convenient set of extensions to localisation: xlocale. This extends the POSIX 2008 per-thread locale APIs (missing in FreeBSD) to provide a set of _l variants of locale-aware libc functions that use a specific locale, rather than the global one.

My recent work, sponsored by the FreeBSD Foundation, has been to implement the missing xlocale APIs. This is now mostly done and pending code review. With this and the new tweaks to libcxxrt, it's now possible to build libc++ on FreeBSD and most of the tests pass.

Most of the remaining test failures are in the header. This defines a lot of complex atomic operations and requires a lot of compiler support. Eli Friedman has been working on adding this support in clang, and with his latest patch applied 25 of the 52 atomic tests pass. There are still a few remaining failures:

- 27 caused by clang not fully supporting the atomic operations yet
- 3 caused by clang not fully supporting the C++11 type-trait intrinsics
- 20 that I don't think are real failures - they're caused by the VM where I'm running the tests not having sufficiently fine-grained time reporting for the thread operation timeout tests to work properly
- 1 is caused by FreeBSD lacking the C1x quick_exit() APIs.
- 2 caused by FreeBSD lacking the uchar.h header

For comparison, Howard Hinnant, the libc++ lead developer, just sent me a list of the failures on OS X. On FreeBSD, 4271 tests pass, 53 fail. On OS X 4253 pass, 71 fail. This is looking very promising for an entirely GNU-free C++ stack.

DIFFUSE for FreeBSD

The FreeBSD Foundation is pleased to announce that Swinburne University of Technology's Centre for Advanced Internet Architectures has been awarded a grant to implement DIFFUSE for FreeBSD.

DIFFUSE (Distributed Firewall and Flow-shaper Using Statistical Evidence) is an extension to the FreeBSD IPFW firewall subsystem developed by CAIA. It allows IPFW to classify traffic based on statistical properties of flows being observed in realtime, and instantiate network actions across a distributed set of "action nodes" for particular flows if required.

This project will tidy up and integrate the existing DIFFUSE prototype into FreeBSD, and incorporate a number of important new features. Integration of DIFFUSE into FreeBSD will increase FreeBSD's utility to designers and implementers of FreeBSD-based networking infrastructure.

Network architects frequently require the ability to classify different traffic types flowing across a network, typically using packet inspection capabilities of base system tools such as ipfw and pf. Traffic classification then enables the provision of customized service levels to different traffic types (such as priority packet queuing and forwarding, or allocation of specific bandwidth guarantees).

DIFFUSE uses machine learning techniques to enable robust and efficient classification of IP traffic flows based on their unique statistical properties in addition to traditional inspection of packet header or payload contents. DIFFUSE also allows traffic classification to occur in one place (e.g. in the core of a network) and trigger traffic shaping and differentiation elsewhere (e.g. at the edges of a network). DIFFUSE has applications in ISP, residential broadband and large corporate network scenarios to name a few.

The project will conclude the end of October 2011.

Implementing xlocale APIs

The FreeBSD Foundation is pleased to announce that David Chisnall has been awarded a grant to implement xlocale APIs to enable porting libc++.

The C standard library (libc) is one of the most important parts of a UNIX system as most programs interact with the kernel through interfaces written in C. Porting code between platforms with similar libc implementations is trivial and if something is supported by libc, higher-level languages can use it without being reimplemented.

Over time, the C language has slowly evolved to modern multicore systems, but there are still some places that are problematic. One of these is localization as C began originally had no localization support. FreeBSD libc and Darwin libc (used by Mac OS X) are similar, making it much easier to port code from OS X to FreeBSD than from OS X to Linux. The libc used by OS X supports a set of extended locale functions (xlocale) that allow locale to be set on a per-thread basis.

Additionally, libc++, from the LLVM project, was originally developed on Darwin, so it uses xlocale for most of the C++ locale support. The lack of this support is the primary obstacle to porting it to FreeBSD.

Once xlocale is supported in FreeBSD libc, we can port libc++ to FreeBSD, giving us an MIT-licensed C++11 standard library implementation. This, in conjunction with Clang and libcxxrt, means that the entire C++ stack in FreeBSD will be free of any GNU code. This leaves the linker as the only significant obstacle to a GPL-free FreeBSD 10.

The project will conclude the end of September 2011.

Participate in Software Freedom Day

This Saturday, September 17, is Software Freedom Day (SFD). SFD is an annual global event that encourages open source software users to reach out to their local community to educate others about the benefits of using open source.

Frederic Muller, President of SFI, the non-profit organization behind Software Freedom Day, has been very helpful in encouraging FreeBSD users to participate in SFD. FreeBSD is listed as a partner on the SFD website. In addition, the FreeBSD logo is included on the cover letter and a copy of PC-BSD was included with the 210 packages that were shipped to the pre-registered teams. He also added the FreeBSD news RSS feed to planet SFD so that other SFD participants will get FreeBSD updates.

Julian H. Stacey has also been helpful in spreading the word and will be participating with the Munich team.

BSD user groups are encouraged to put together a team and to register it on the SFD wiki. If you have not participated in SFD before, take a moment to read through the Start Guide for instructions on how to promote your team. If you do register a team, leave a comment so that we can mention it on the @bsdevents twitter feed.

Accepting Travel Grant Applications for EuroBSDCon 2011

Calling all FreeBSD developers needing assistance with travel expenses to EuroBSDCon 2011.

The FreeBSD Foundation will be providing a limited number of travel grants to individuals requesting assistance. Please fill out and submit the Travel Grant Request Application by September 5, 2011 to apply for this grant.

How it works:

This program is open to FreeBSD developers of all sorts (kernel hackers, documentation authors, bugbusters, system administrators, etc). In some cases we are also able to fund non-developers, such as active community members and FreeBSD advocates.

(1) You request funding based on a realistic and economical estimate of travel costs (economy airfare, trainfare, ...), accommodations (conference hotel and sharing a room), and registration or tutorial fees. If there are other sponsors willing to cover costs, such as your employer or the conference, we prefer you talk to them first, as our budget is limited. We are happy to split costs with you or another sponsor, such as just covering airfare or board.

If you are a speaker at the conference, we expect the conference to cover your travel costs, and will most likely not approve your direct request to us.

(2) We review your application and if approved, authorize you to seek reimbursement up to a limit. We consider several factors, including our overall and per-event budgets, and (quite importantly) the benefit to the community by funding your travel.
Most rejected applications are rejected because of an over-all limit on travel budget for the event or year, due to unrealistic or uneconomical costing, or because there is an unclear or unconvincing argument that funding the applicant will directly benefit the FreeBSD Project. Please take these points into consideration when writing your application.

(3) We reimburse costs based on actuals (receipts), and by check or bank transfer. And, we do not cover your costs if you end up having to cancel your trip. We require you to submit a report on your trip, which we may show to current or potential sponsors, and may include in our semi-annual newsletter.

There's some flexibility in the mechanism, so talk to us if something about the model doesn't quite work for you or if you have any questions. The travel grant program is one of the most effective ways we can spend money to help support the FreeBSD Project, as it helps developers get together in the same place at the same time, and helps advertise and advocate FreeBSD in the larger community.

Semi-Annual Newsletter Available

We are pleased to announce the publication of The FreeBSD Foundation's 2011 Semi-Annual Newsletter. In this edition:

• Letter From the President
• Fundraising Update
• Development Project Updates
• IPv6 support in FreeBSD and PC-BSD
• Implementing support of GEM, KMS, and DRI for Intel Drivers
• Resource Containers Project
• Feed-Forward Clock Synchronization Algorithms Project
• Five New TCP Project
• libcxxrt C++ Runtime Available Under BSD License
• Conference Updates
• AsiaBSDCon 2011
• BSDCan 2011
• 2011 Grant and Travel Grant Recipients
• NYI Testimonial
• Foundation Update
• Financials

You can help to support the FreeBSD Project by making a donation to the Foundation.