Booth at MeetBSD Poland

There will be a FreeBSD Foundation booth during MeetBSD Poland this Friday and Saturday. Feel free to drop by the booth to learn more about the Foundation's efforts, check out the cool Foundation swag (baseball caps and toques), pick up our latest brochure, and/or make a donation to the Foundation.

Dtrace Userland Project

Rui Paulo has been awarded a grant to add DTrace userland support to FreeBSD.

DTrace is a general purpose and lightweight tracing framework that allows administrators, developers and users to investigate causes of system failure or performance bottlenecks. The FreeBSD operating system has had support for kernel-only DTrace since FreeBSD 8.0, but DTrace userland support was missing. Having userland support in DTrace allows inspection of userland software itself and its correlation with the kernel, thus allowing a much better picture of what exactly is going on behind the scenes.

This project will first concentrate on adding libproc support for symbol to address mapping, address to symbol mapping, breakpoint setup and the rtld interactions with DTrace. Next it will focus on DTrace process control, importing the pid provider and adapting it to FreeBSD and porting the userland statically defined probe provider (usdt). Finally it will bring in the plockstat provider.

"By having userland DTrace support, companies can make their products perform much better on FreeBSD due to the fact that they now have access to this amazing tool," said FreeBSD developer Rui Paulo. He also said, "When we mix the userland support with the kernel side DTrace support, we can also make FreeBSD a better operating system because we can investigate performance bottlenecks much easier."

The project should be completed by September 2010.

BSNMP Improvements Project

The FreeBSD Foundation is pleased to announce that Shteryana Shopova has been awarded a grant to make improvements to BSNMP.

This project includes several enhancements to the existing FreeBSD SNMP framework, including SNMPv3-compliant user authentication, packet encryption and view-based access control. In addition, the project also includes a new module that will allow full SNMP management and monitoring of the FreeBSD wireless networking stack. When the project is completed, FreeBSD should be the OS of choice when building open source-based embedded wireless appliances due to the advanced capabilities of its wireless network stack, and the light-weight, secure and complete management solution that bsnmpd(1) will provide out of the box. Existing FreeBSD installations that use bsnmpd(1) for monitoring will also benefit from the added security and finer-grained access-control to SNMP data.

"SNMP is the defacto standard for network monitoring," said Shteryana Shopova, FreeBSD developer. She also added, "SNMP is used everywhere - in network servers, switches, routers, firewalls, workstations, ip phones, printers, UPSs, all sorts of embedded appliances. I am happy to have the opportunity to work on several additions to bsnmpd(1) that have been requested by the FreeBSD community."

This project will be completed in October 2010.

FreeBSD and PostgreSQL

Last month the FreeBSD DevSummit was once again held the two days prior to BSDCan. While the DevSummit is aimed primarily at FreeBSD Developers, some invitees were from other organizations that use, contribute to, or are otherwise interested in the development of FreeBSD. Such a mix offers opportunities to discuss pain points and ways to collaborate.

One of the invited speakers was Greg Smith from 2ndQuadrant, a company that provides professional services and support for PostgreSQL. Greg wrote about his experience at the DevSummit on the 2ndQuadrant blog and has given permission to repost that entry here. It should be noted that the FreeBSD Foundation is currently funding a project for userspace DTrace.

This week I did something I'd prefer to never repeat: I left the country, did something useful, and made it back again in the same day. The occasion was the FreeBSD Developer Summit, held just before BSDCan--the convention that happens in Ottawa the week before PGCon every year. So I get to head right back again next week, but stay a while that time.

The FreeBSD developers were nice enough to sponsor my trip so that we could talk about both the business and technical hurdles that I felt were keeping the sort of companies I work with from deploying their databases on FreeBSD more often than they do. My slightly updated slides are available on our talks page, I cleaned up a couple of things from what was presented (the most important rewording I'll talk about below).

I was very pleased at how friendly and receptive the developers were even to some of my critical comments. FreeBSD and PostgreSQL have very like minded communities: open for any purpose BSD license, academic roots, developers focused on stability, and even a strong documentation culture. There's been plenty of cross-over too.

Much of the PostgreSQL infrastructure has been run using FreeBSD jails for quite some time (although plans are moving to use more Debian in its place, details on why at Inside the PostgreSQL Project Infrastructure). My running joke during the talk was that if PostgreSQL developers are eating their own dog food by deploying critical infrastructure that depends on the database, much of that has been served in a FreeBSD bowl. (The lunch at the conference session was pizza, much better choice)

And there's been plenty of FreeBSD development that's used PostgreSQL benchmarking as a measuring stick for the success of their advances. The very popular Introducing FreeBSD 7.0 slides that not only showed their achieving performance parity against Linux during that release, it doubled as a document showing how PostgreSQL outscales MySQL. Cheers all around for community driven, BSD licensed code.

One bit of audience contention during my talk came from my assertion that not having support for Emulex fiber channel cards in FreeBSD was preventing a significant amount of "big iron" adoption for databases, due to their perception as the market leader for connecting up expensive hardware like SANs. The guys from FreeBSD hardware and support vendor iXsystems called me out on that, suggesting that the alternative vendor here--QLogic--is both completely trusted by the big boys and has top notch FreeBSD driver quality.

I did a bit more research into whether I was suffering from sampling bias from the set of people I'd talked to about this, and it looks like that was the case. While Emulex claims they've been named Sun's "Best-in-Class Supplier for OEM products", and all the Sun FC cards I've personally run into came from them, there are tons of Sun rebrands of both Emulex and QLogic cards. Same thing is true at all the other vendors I mentioned in my talk; you can get FC cards from both manufacturers via HP and Dell too. I think my general point, that not supporting both Emulex and QLogic hurts the perception of FreeBSD as a serious choice for large businesses, still stands; it's just not quite as bad as I'd feared. Accordingly, I tweaked the wording in the slides I'm publishing, to better match reality here than the ones I presented.

In additional to the solid core they've been growing for years, FreeBSD's license has allowed them to incorporate two very valuable features Sun released as open-source, ZFS and DTrace, into their operating system, both of which are incompatible with Linux's license and are extremely valuable for PostgreSQL deployments. It's still not ideal yet; FreeBSD DTrace can currently be used only by root for example. Limitations such as these have in the past kept me from being particularly motivated to work with FreeBSD. The existence of a free commercial Solaris that ran on generic hardware, combined with the steady progress and open enough community around OpenSolaris, satisfied my needs better. While not many of my PostgreSQL installations have been on Solaris, its has a monopoly share for hosting the terabyte scale databases I've worked with. High quality filesystem snapshots via ZFS and the additional piece of mind you get from disk block checksums alone justified those platform decisions.

The problem today is that hating everything about how Oracle does business is what got me working with PostgreSQL in the first place, and now that they own Sun they're doing the same things to Solaris. No more Solaris on non-Sun hardware, serious cutbacks on the open-source version (OpenSolaris looks like a walking corpse to me), cutting off even basic OS patches unless you have a support contract--that's what we've seen just in the first round from Oracle here. Solaris isn't free in any sense of the word again, we're right back to the same dynamics that pushed me away from them and toward Linux fifteen years ago.

But I continue to be dissapointed at how little focus there is on quality control in Linux. How poorly the filesystem mechanics work for the sorts of database work I do doesn't help either. The Linux OOM killer might as well be named the Linux PostgreSQL Hater for how it acts on my servers. And those sexy Solaris features I know work so well for databases, still not there (even if SystemTap is getting better at DTrace emulation).

Meanwhile, FreeBSD has the whole "free" thing sorted out right in their name, and their quality control paranoia is similar to that of your typical good DBA. It looks to me like they're very close to fully assimilating ZFS and DTrace to the point where they can start improving them, rather than just working on getting the original feature set Solaris already had complete and the matching code stable. I think all of us who work on business critical PostgreSQL deployments and who value free software should do a sanity check on just what dog food we're chewing on, and start making sure there's a FreeBSD bowl there at least sometimes. From what I heard this week, the FreeBSD developers are gearing for another round of chewing on ours too. They're looking into database oriented performance improvements as part of future development, and they're not any happier about using MySQL for that than I am about running PostgreSQL on Solaris. Looks like it might be bowls of dog food all around. Nobody said that leading the software industry was going to be tasty.

MeetBSD Poland

The Foundation is pleased to be a sponsor for MeetBSD, to be held in Krakow, Poland July 2-3. Tomasz Dudzisz, one of the organizers of MeetBSD, recently sent a thank you to the FreeBSD Foundation Board of Directors:


We would like to thank you for your generous donation which we received a few days ago. Your contribution makes it possible for us to organize the best BSD event in our part of Europe. With the help of donations from supporters such as you, we can give a great chance for participants to talk to the FreeBSD project's contributors in person.

We believe that high-calibre technical content of the offered lectures and unforgettable atmosphere of the side talks make meetBSD one of the greatest UNIX-oriented conference not only in Poland but world wide.

We hope for your continued support in the future. Once again thank you for your generous donation.


Thanks! We agree that sponsoring conferences provides many benefits to the FreeBSD community.

Update on Jail Based Virtualization Project

One of the proposals selected for funding earlier this year was for jail based virtualization. Bjoern Zeeb, the developer being funded, recently provided an update on the progress of this project:

Bjoern A. Zeeb has been awarded a grant to improve FreeBSD's jail based virtualization infrastructure and to continue to work on the virtual network stack. His employer, CK Software GmbH is matching the Foundation's funding with hours.

FreeBSD has been well known for its jail based virtualization during the last decade. With the import of the virtual network stack, FreeBSD's operating system level virtualization has reached a new level.

This project includes cleanup of two years of import work and development and, more notably, brings the infrastructure for a network stack teardown. Cleanly shutting down a network stack in FreeBSD will be the major challenge in the virtualization area to get the new feature to production ready quality for the 9.x release
lifecycle.

Further, the project includes generalization of the virtual network stack framework, factoring out common code. This will provide an infrastructure and will ease virtualization of further subsystems like SYSV/Posix IPC with minimal overhead. All further virtualized subsystems will immediately benefit from shared debugging facilities, an essential feature for early adopters of the new technology.

Improved jail based virtualization support, that continues to be very lightweight and as easily manageable as classic jails, will be a killer feature for the next few years. It will allow people to partition their FreeBSD server, run simulations
without racks of hardware, or provide thousands of virtual instances in hosting environments fairly easy and efficiently. While this follows the trend of green computing, it also adds to FreeBSD's virtualization portfolio with Xen or other more heavyweight hypervisor support, which can be mixed with jails as needed.

While work in this area will have to continue, the funding for this project will end mid-July 2010.

BSDCan Trip Report: Barry Steyn

Barry Steyn, Software Development Manager at RedButton, writes the following in his BSDCan trip report:

Let me start by saying that I am from Africa, South Africa to be exact. Free software is very advantageous as South Africa is not a rich country. I am using FreeBSD as a server due to its incredible stability in such an environment. Whilst I have been on the FreeBSD mailing lists for sometime, I have always wanted to meet the community. South Africa is quite far away (geographically) from the first world countries; this compounded with the price of airfare makes it very infrequently that South Africans travel. I was very fortunate to be able to attend BSDCAN 2010 with the support of the FreeBSD foundation.

There were several very interesting talks. For me, the four most interesting talks were the talks on Clang BSD, Security implications of Ipv6, Colin Percival's cryptography overview and wireless meshing under FreeBSD. The ClangBSD talk introduced me to LLVM – something that I was unfamiliar with, and am now very excited about. Ipv6 is the way of the future; after this talk, I realise that it may not happen all that smoothly! The highlight for me was Doctor Percival's talk. It was informative and humorous – the perfect combination. My business in South Africa has to do with WiFi hotspots, so I felt that the wireless meshing talk was put there just for me!

All in all, a fantastic conference. I would like to thank the coordinators of the conference (Dan Langille comes to mind, not to forget about all the volunteers) for organising such a wonderful event.

BSDCan Trip Report: Florent Thoumie

Florent Thoumie recently blogged about his trip to BSDCan:

First I’d like to start this post by thanking the FreeBSD Foundation for funding my trip. I’ve been contemplating attending BSDCan for years and without their financial support I would have missed it this year again.

I’ve been a FreeBSD ports committer since 2006. In 2007, my commit privileges were extended to the src tree. In 2008, Pav approached me to become part of the Ports Management Team.

I’ve had the chance to meet up with a few people (Ed Maste, Garrett Cooper, Tim Kientzle) and discuss the coordination of the work that is being done and will be done on package tools as part of Google Summer of Code. During the developer summit, Mark Linimon, Erwin Lansing and myself held a discussion about the current state of packages and how to improve the user experience. A few people offered suggestions and
portmgr took good note of them. I did take some time to go through the problem reports assigned to portmgr. I also attended a chat about FreeBSD mirrors along with some members of core, admins and portmgr.

There were a lot of interesting talks during the conference and obviously choices had to be made on which ones I would go see. I really enjoyed Will Backman’s keynote. The talk about the PCBSD installer was very interesting and it looked like there could be a drop-in replacement for sysinstall in the very near future. Lawrence Stewart’s talk was a good summary of what tools to use when doing FreeBSD developement work.

BSDCan 2010 was a great time, I really enjoyed it and I feel it was time spent in a productive fashion. I would like to thank the following people: Dan Langille and his volunteers for the brilliant conference they put together, Sam Leffler / Philip Paeps / Gavin Atkinson / Jonathan Anderson for sharing a room with me, Jordan Hubbard for a memorable meal in the Works Burger in Glebe and Kevin Van Vechten for the invaluable insight on American Sports and the FreeBSD Foundation, once again, for sponsoring my trip.

Attending conferences makes the difference between being a contributor and being part of a community. It is the perfect opportunity to meet new people with similar interests, meet people you’ve been exchanging emails with (putting a face on a name) and make sure you stay updated with the works in progress.