Letter from the President

From the December Foundation Newsletter:

In 2009, the FreeBSD project had the misfortune of losing two long time contributors: John Birrell and Jean-Marc Zucconi. I chatted with John recently, during this year's BSDCAN, so his death was all the more shocking. It forced me to recognize my own mortality and to consider what contributions from our lives remain after we pass away. Reviewing the heritage of FreeBSD it becomes clear that our work on this project takes on a life of its own. John and Jean-Marc's efforts live on in FreeBSD.

The value of the FreeBSD legacy has become even more apparent to me with my return, after an almost 10 year break, to working on FreeBSD during my day job. The kernel has the familiarity of an old friend, even though it has been improved in countless ways by dozens of new contributors. Although the principles and best practices engendered by the FreeBSD project have changed little in the last decade, they are no less relevant today. The faces and challenges may have changed, but the qualities of FreeBSD - solid design, high performance, stability - have not.

Trying to communicate this real value of FreeBSD can be difficult. Valuations typically take current features into account, but neglect to consider the community that created and will support those features. And what value can you place on future, community driven, improvements to FreeBSD? There is no guarantee that every feature you need will be added to the system in the time frame you need it, but the collaborative environment created by the FreeBSD project makes it very likely. The greatest asset of FreeBSD is our community. If we continue to invest in the FreeBSD community, I have every confidence that the FreeBSD legacy will endure.

The FreeBSD Foundation is doing its part to continue FreeBSD's legacy. Through conservative management and careful planning we have improved our financial position even when faced with the most severe recession in 50 years. Our growing strength and capabilities, all made possible by your generous donations, are proof that the FreeBSD Foundation will be a faithful supporter of FreeBSD for many years to come. Together, building on the legacy left by John, Jean-Marc, and the countless contributors before them, we will ensure the future of FreeBSD.

FreeBSD Project Receives Bad Code Offset

The Bad Code Offsets project of The Alliance for Code Excellence is "a way to undo the bad code other people have written without actually replacing the bad code. Much like carbon offsets, money used to buy Bad Code Offsets goes towards open-source projects which not only produce good code, but produce software that helps developers build good software".

The FreeBSD Project was recently added as a supported project and the FreeBSD Foundation will be receiving a check for $500. Thanks to those who suggested the FreeBSD project be added and to the Alliance for supporting good code!

Google Summer of Code Mentor Summit 2009

Brooks Davis recently reported on his trip to the Google Summer of Code Mentor Summit 2009. The Foundation assisted in some of his travel costs to this event. Brook writes:

The Google Summer of Code Mentor Summit was held at the Google campus in Mountain View October 24th and 25th. I represented the FreeBSD project at the event along with Tim Kientzle who was one of our other program admins.

Google runs the Mentor Summit as an un-conference which means that attendees pick topics they want to discuss, others indicate interest in them, and then rooms are allocated based on demand. Sessions were about things including general open source process issues, education, technical collaborations, and individual project meetings.

One of the highlights of Saturday was a session on multi-core and other acceleration technologies like GPUs. The session didn't come to a strong consensus other than incorporating these technologies is difficult. The most concrete thing that came up was the idea of putting the technologies behind widely used APIs so they automatically provide benefit. Another topic of discussion was debugging tools and
techniques. I think that is an area where FreeBSD is sometimes ahead with technologies like witness and now DTrace.

The other session of interest was an impromptu session of non-Linux OSes including DragonFlyBSD, FreeBSD, Haiku, and NetBSD. It was mostly people talking about current status. One thing other projects seemed to want was a way to take advantage of FreeBSD network drivers by providing more common interfaces. In concept this was interesting, but probably isn't some thing that would make a whole lot of sense for FreeBSD given that we're rethinking and redesigning the interfaces we have to meet modern performance requirements.

The most useful session on Sunday was when Tim and I grabbed a small conference room and started reviewing our GSoC admin materials for next year. Based on that information, we plan to start engaging with FreeBSD developers in January in anticipation of a 2010 program.

Over all the conference was interesting and fun. It was good to talk to
people from other projects that don't often attend the BSD conference.
I'm not sure anything concrete came of those interactions, but it was
probably useful nonetheless.

Thank You!

We would like to thank everyone who has donated to the FreeBSD Foundation this year. We have raised $182,778 towards our 2009 goal of $300,000! We are almost 2/3 of the way to reaching our goal! Oh, and BTW, we have had 663 donors this year. This is compared to just over 300 this time last year. This is important not only to help us keep our Public Charity Status, but it shows there are many users who are passionate about FreeBSD and want to show their support.

With the weakened economy we have been very conservative with our spending this year. But, like each previous year we have increased the amount we have spent on the FreeBSD Project and community. We were blown away with the number of project proposals we received this year. We were able to fund 7 projects this year. Unfortunately we didn't have the budget to fund all the proposals we received.

This coming year we want to double the amount we spend on project development. In order to accomplish this, we need to meet our fund-raising goal.

Why do we need donations?

The goal of the FreeBSD Project is to provide software that may be used for any purpose -- and without strings attached. Our mission is to support the FreeBSD Project and community. Our funding comes from people like you – those who are determined to keep FreeBSD free!

How have we spent the money this year?

• Sponsored FreeBSD related conferences like BSDCan, EuroBSDCon, AsiaBSDCon, KyivBSD, and DCBSDCon.

• Provided 15 travel grants and funding to individuals to attend these and other conferences this year.

• Provided grants for projects that improve FreeBSD, like wireless mesh support, FreeBSD TCP stack improvements, new console driver, safe removal of disk devices, flattened device tree, and high available storage projects.

• Provided equipment for developers working to improve FreeBSD and projects like the NetPerf cluster. We purchased servers, USB analyzers, power controllers, and parts for computer repairs for the Project. We also paid for shipping of equipment to various projects.

• Provided legal support for the project on issues like understanding the GPLv3 impact on FreeBSD, providing a privacy policy, trademark ownership and permission, and other legal issues that come up.

How can you help?

Your financial support is critical for the FreeBSD Project. Please help us keep FreeBSD free. Any amount you can donate will help. And thank you for your continued support of the FreeBSD Foundation.

Update on HAST Project

Pawel Jakub Dawidek has completed the first milestone for the High Available Storage Project. He writes:

I want to report that first milestone of the HAST project is complete.

Summary of the work that have been done:

• Implementation of hastd daemon.
• Implementation of hastctl utility to manage hastd daemons.
• GEOM_GATE class was extended so that the caller can specify the name of GEOM provider. Before only /dev/ggateX names were supported. HAST will use /dev/hast/<resource_name> names.
• Implementation of communication protocol. There is abstraction layer on top and below there are three protocols implemented currently:
  

1. proto_tcp4 - It is used for communication between primary and secondary nodes.
2. proto_uds - (UDS - UNIX Domain Socket) It is used for communication between hastctl and hastd.
   
3. proto_socketpair - It is used for communication between main hastd daemon and worker processes forked from it.

• Implementation of nv (name-value) API, which allows to easy create packets containing name-value pairs. It is used for entire communication through the protocols above. It is also responsible for managing correct byte-order.
• Implementation of ebuf (extendable buffer) API, which provides a way to extend given buffer by adding data at the back, but also at the front without reallocating it and copying the data very often (or never).
• Implementation of logging API (pjdlog). The API decides if messages should be logged on standard output/error (before going into background) or to syslog (when we daemonize). It also provides some shortcuts for logging a message and exiting, etc. It supports notion of debug level and can skip messages intended for higher debug level than requested.
• Implementation of configuration file parser in lex/yacc. Configuration file is designed in a way that it can be kept identical on both nodes.
• Checksumming and compression for the data is not one of the project's goal, but the stubs are there, so this can be added easly.
• A lot of care was taken to be able to handle more nodes in the future. This is not implemented and in not project goal, but I wanted to make it ready for future improvements.
  

HAST can be used by starting hastd daemons on both nodes:

# hastd -h
hastd: [-dh] [-c config]

Then on the secondary node we mark all resources as secondary:

# hastctl -h
hastctl: [-d] [-c config]
hastctl: [-d] [-c config] status [all | name ...]

# hastctl secondary all

On the primary node we mark all resources as primary:

# hastctl primary all

The hastd daemon running on primary node will connect to the secondary node and fork a child to handle communication. There is a socketpair between parent and child so that they can communicate. Primary node creates two connections: one for incoming data and one for outgoing data. There are seven threads in total for each working resource:

1. ggate_recv

Thread receives ggate I/O requests from the kernel and passes them to appropriate threads:
WRITE - always goes to both local_send and remote_send threads
READ (when the block is up-to-date on local component) -only local_send thread
READ (when the block isn't up-to-date on local component) -only remote_send thread
DELETE - always goes to both local_send and remote_send threads
FLUSH - always goes to both local_send and remote_send threads

2. local_send

Thread reads from or writes to local component.
If local read fails, it redirects it to remote_send thread.

3. remote_send

Thread sends request to secondary node.

4. remote_recv

Thread receives answer from secondary node and passes it to ggate_send thread.

5. ggate_send

Thread sends answer to the kernel.

6. ctrl

Thread handles control requests from the parent.

7. guard

Thread guards remote connections and reconnects when needed, handles signals, etc.

On the secondary node when both connections are successfully established it forks a worker process, which operates using four threads:

1. recv

Thread receives requests from the primary node.

2. disk

Thread reads from or writes to local component and also handles DELETE and FLUSH requests.

3. send

Thread sends requests back to primary node.

4. ctrl

Thread handles control requests from the parent.

At this point primary and secondary nodes can communicate and requests are properly replicated. IO errors on local read failure are handled by redirecting read request to remote node. Replicated storage can be accessed through /dev/hast/<resource_name> GEOM provider.

I'm confident that the first milestone is complete. If you have any questions, I'll be happy to answer them. If you have any suggestions or comments, I'll also be happy to hear them.

Colin Percival on Supporting the Foundation

Developers not only benefit from the work done by the Foundation, they also help to financially support it. Colin Percival writes about donating to the Foundation on his blog. His blog post begins:

As a FreeBSD user and developer, I obviously care about the success of FreeBSD. I make a small contribution towards this success via my role as Security Officer; but the time I spend working on my Tarsnap online backup service prevents me from making as much of a direct contribution as I would like. Fortunately the FreeBSD Foundation does an excellent job of supporting FreeBSD development; but like most such organizations, they are funded entirely by donations and are always in need of more. In light of this, I am pleased to announce that I will be donating all of the profits made by Tarsnap for the month of December to the FreeBSD Foundation.

Much of the work done by the FreeBSD Foundation is done behind the scenes, but the importance of the work they do is undeniable. They sponsor a range of important FreeBSD development which would likely not get done otherwise; in the past year this has included...

In Memory of Donations

It is with sadness that we note the passing of long time FreeBSD committer, John Birrell. Many in the FreeBSD community had the opportunity to interact with John online and perhaps meet him in person at a BSDCan. He will be missed.

It is now possible to make a donation to the FreeBSD Foundation in memory of someone. Select the "in memory of" button on the donation page and type in the name of the person the donation is being made in memory of.

AsiaBSDCon 2010

The FreeBSD Foundation is pleased to once again be a sponsor of AsiaBSDCon 2010. This conference will be held in Tokyo, March 11-14, 2010.

If you are considering speaking at this conference, the call for papers ends on December 20.

End-of-Year Fund Raising Drive

We're kicking off our End-of-Year Fund Raising Drive! Our goal this year is to raise $300,000. So far, we have raised just over $159,000 so we are little more than half way towards our goal. Please consider making a donation and help us to spread the word by tweeting and blogging about the drive.

Click here to find out how to make a donation.

Another New Funded Project

Rafal Jaworowski and Semihalf have been awarded a grant to provide FreeBSD with support for the flattened device tree (FDT) technology. This project allows for describing hardware resources of a computer system and their dependencies in a platform-neutral and portable way.

The main consumers of this functionality are embedded systems whose hardware resources assignment cannot be probed or self-discovered.

The FDT idea is inherited from Open Firmware IEEE 1275 device-tree notion (part of the regular Open Firmware implementation), and among other deployments is used as a basis for Power.org's embedded platform reference specification (ePAPR).

"Thanks to this project, embedded FreeBSD platforms will grow in a uniform and extensible way of representing hardware devices, compliant with industry standards (ePAPR, Open Firmware), independent of architecture and platform (portable across ARM, MIPS, PowerPC etc.)," said Rafal Jaworowski, FreeBSD Developer.

Semihalf is a privately owned company, based in Krakow, Poland. They specialize in embedded systems design and development, with expertise in both software and hardware. Among their portfolio are FreeBSD ports to high-end embedded processors (including multi-core) with a wide range of peripheral drivers (storage, networking, pattern matching, security engines etc.); most of this work is publicly available from the FreeBSD repository.

This project will complete by February 2010.

New Funded Project

The FreeBSD Foundation is pleased to announce a new funded project!

Pawel Jakub Dawidek has been awarded a grant to implement storage replication software that will enable users to use the FreeBSD operating system for highly available configurations where data has to be shared across the cluster nodes. The project is partly being funded by OMCnet Internet Service GmbH and TransIP BV.

The software will allow for synchronous block-level replication of any storage media (GEOM providers, using FreeBSD nomenclature) over the TCP/IP network and for fast failure recovery. HAST will provide storage using GEOM infrastructure, which means it will be file system and application independent and could be combined with any existing GEOM class. In case of a master node failure, the cluster will be able to switch to the slave node, check and mount UFS file system or import ZFS pool and continue to work without missing a single bit of data.

"High-availability is the number one requirement for any serious use of any operating system," said Pawel Jakub Dawidek, FreeBSD Developer. "Highly available storage is one of the key components in such environments. I strongly believe there are many FreeBSD users that have been waiting a long time for this functionality. I'll do my best to deliver software that matches FreeBSD quality and that will satisfy the needs of our users."

Pawel has been an active FreeBSD committer since 2003. During this period, he has touched almost every part of the kernel. But, his main interest in FreeBSD is storage and security related topics. Pawel is the author of various GEOM classes (eli, mirror, gate, label, journal, hsec, etc.), geom(8) utility, various open crypto improvements as well as port of the ZFS file system from OpenSolaris to FreeBSD.

The project will complete by February 2010.

EuroBSDCon Feedback

Reports are starting to trickle in from those who attended last week's EuroBSDCon. Robert Watson notes that the FreeBSD Developer's Summit was the largest yet in the EU with around 70 attendees. And, despite the poor economy, attendance for the conference was up at about 180 attendees, and the dinner sponsored by iXsystems sold out at 150 with a waiting list.

Slides, papers, and audio recordings should be available soon on the UKUUG website. We'll post summaries of the trip reports as they become available.

Rodrigo Osorio has posted pictures here. If you have EuroBSDcon pics, leave a comment containing a link.

EuroBSDCon

The FreeBSD Foundation is pleased to once again be a sponsor of EuroBSDCon. This year, Robert Watson of the Foundation is a co-organizer of the conference with UKUUG. The Foundation is hosting a FreeBSD Developer Summit which will bring together 50-60 developers for an intense two-day workshop to discuss all aspects of FreeBSD development, including features for the now-in-the-works FreeBSD 9.0. The Foundation will sponsor 9 developers and users to attend the conference and/or Summit, and Robert, Hiroki Sato, and Sam Leffler will be in attendance.

EuroBSDCon has lots of great talks this year, including several by Foundation members and developers whose work has been sponsored by the Foundation. These include:

Sam Leffler will give a presentation entitled 100km@36Mb/s, or Long Distance Wireless.

Rui Paulo will present the results from his Wireless mesh networks under FreeBSD project.

Ion-Mihai Tetcu, who recently described the importance of speaking at conferences, will give a presentation entitled 20,000 Ports Under The Hood.

Take a look at the full list of talks and consider registering if you haven't already. The Foundation would love to see you there!

The Importance of Speaking at Conferences

In Why We Send Developers to Conferences, Thomas Abthorpe discussed the value in face-to-face networking with both committers and BSD users.

In this post, Ion-Mihai Tetcu discusses the importance of BSD developers speaking at non-BSD specific and international conferences. His report also shows some of the lessons that can be learned from meeting with users and learning first-hand how a global project is meeting their local needs.

I had the opportunity, with the Foundation's help, to participate as a speaker at FISL 10 in Porto Alegre, RS, Brazil. This year's anniversary edition had 8,232 paying attendees, speakers from 28 countries, a lot of vendors and user groups, and a powerful media presence. FISL was sponsored, among others, by the Brazilian Federal Government and Brazil's President Mr. Luis Ignacio Lula da Silva gave a nice speech .

Open source in Brazil has a powerful momentum, being promoted by the federal and various states and local governments. It is seen as a way to reduce the country's dependency on foreign companies, a way to reduce unemployment, and generate local revenues and expertise. It is used by the national bank, federal government institutions and even by local subsidiaries of big multinational companies. Open Source in Brazil pretty much equals Linux, especially RedHat, Suse and Debian (who all have powerful user-groups) and a few local distributions. (Free)BSD is also used, especially by telcos and in the embedded market. More that 50% of the attendees didn't understand English and the situation is even worse in the general public. This practically implies that, without localization, a software product can not have any significant market share in Brazil.

Apart from my DSPAM talk, I gave a general talk about FreeBSD ports and packages and PCBSD's PBIs, was one of the hosts of the BSD-Meeting and assisted at the FUG-BR stand. Unfortunately, the other BSDs had zero presence. The 6 hours of the BSD-Meeting were a micro-conference attended by 65-70 people. Of the 5 talks, 3 could have easily found a place on the main schedule and I repeatedly kicked those speakers for not submitting their talks to FISL organizers. From the Brazilian user's perspective, the biggest problem faced is the lack of a localized version of FreeBSD. For example, I was asked if we could provide a framework for localizing the OPTIONs and pkg-message of our ports. When faced with a new operating system, many users will choose a localized Linux variant over the effort of learning both a new operating system and a new language. As a first step, I am pursuing with the PC-BSD folks the idea of doing a custom-built PC-BSD variant localized for Brazil. Marcelo Araujo presented what the translation process implies and one of the results of the BSD-Meeting is a restart ofthe Brazilian Documentation Project.

Lacking any promotional materials except a few posters, the FUB-BR booth didn't attract as many people as the other booths. However, it was a place where people could meet some FreeBSD committers and we had many interesting discussions with both FreeBSD and Linux users. One of the things practically everybody I spoke with during the conference told me was that they desire international speakers. At least 30 people attended FISL because there was a FreeBSD speaker from abroad. I think this is an important idea and that we should also encourage developers to give talks at general F/OSS conferences.

Safe Removal of Active Disk Devices

Earlier this year, the Foundation sponsored Edward Tomasz Napierala to fix FreeBSD's #1 reported bug: a USB disk causing a panic when detached before unmounting. Edward describes the project as follows:

One of the long-standing problems encountered by FreeBSD users was the fact that the system could often crash after a mounted disk device - for example, a USB flash drive - was removed. This behavior was not only annoying, but also made a bad impression about the overall stability and robustness of the operating system.

The project was not about fixing one buggy driver, as it could seem at first glance. Fixing the problem involved changes in CAM (Common Access Method, FreeBSD SCSI subsystem), GEOM framework, Virtual Filesystem layer, and finally the UFS filesystem. (Ironically, there were no problems with the USB itself.) There were no big design changes of any sort; just an iterative process of finding a way to crash the system, tracking down the bug that was causing it, fixing it, and proceeding to the next one. Most of the fixes were backported to FreeBSD 7-STABLE and will appear in FreeBSD 7.2.

It is now possible to remove mounted devices - and to unmount them afterwards - without any user-unfriendly behavior, such as crashes. Also, the system became more robust in the presence of non-USB disk removal, such as SCSI or SATA drive detachment or failure.

New Board Member

We are pleased to announce the addition of Dru Lavigne to our Board of Directors. Dru brings with her years of experience as a writer, FreeBSD advocate, and FreeBSD administrator/teacher.

Be Counted Campaign

Millions of systems run FreeBSD. Hundreds of volunteers contribute to FreeBSD's success. But what is the size of FreeBSD's user base? This simple question is very hard to answer, but its answer is vital to the cause of promoting FreeBSD. It is extremely difficult to convince businesses to invest time and money to add FreeBSD support to their products based solely on vague estimates of the size of our community. We should know - working to make FreeBSD a more widely supported platform is a task the FreeBSD Foundation has worked on since its inception.

Please help us in our fight to promote FreeBSD. A donation to the FreeBSD Foundation helps fund our work, but it also gives us strength in numbers. Our count of unique donors is a vital indication of the size and buying power of our community. However, we have never broken even one thousand donors in any year. We know in our hearts that this is a small fraction of our user base and of those who want to help expand FreeBSD's presence.

So stand up and be counted! Make a donation. Encourage other FreeBSD users to donate as well. No donation amount is too large or too small. Just by becoming a donor you are making a powerful statement about the strength of FreeBSD!

Hot off the Press: FreeBSD Foundation Newsletter

In this Edition:

* Letter From the President
* 2009 Fundraising Drive
* Dru Lavigne Helping Foundation
* Safe Removal of Active Disk Devices
* Wireless Mesh Support
* Improvements to the FreeBSD TCP Stack
* AVR32 Support
* Problem Reporting Prototype
* FreeBSD Powers Long Distance Wireless Link
* DCBSDCon 2009
* AsiaBSDCon 2009
* Foundation at BSDCan and Developer Recognition
* 2009 Grant and Travel Grant Recipients
* BSDCan Spotlight
* Financials

Why We Send Developers to Conferences

You probably know that the FreeBSD Foundation provides travel assistance for developers to attend conferences. If you've ever attended a BSD conference yourself, you have experienced first hand the value in networking with both committers and BSD users.

We'll be asking developers we've sponsored to share their experiences and will start with Thomas Abthorpe, a FreeBSD ports committer who attended this year's BSDCan. In Thomas' words:

It is an over used and abused saying, and I will invoke it, "Been there, done that, got the t-shirt". On the back of the shirt I received at registration it said "FreeBSD it's all about the people, from all around the world". For me, attending BSDCan was an opportunity to meet the people behind FreeBSD face to face. Email and IRC are great ways to collaborate with other developers, ideas can be shared, and projects brought to fruition, but in the end, the opportunity to get together with like minded people and just brainstorm in person is still the best way to get the job done.

I live in Thunder Bay, Ontario, a small city in central Canada, just north of the Minnesota border. My day job is as a Systems & Networks technician for the Canadian Grain Commission. FreeBSD is what I do for "fun" on my own time. Where I live, there are no local/user groups for any form of open source software. I have to rely on Internet technologies to reach out to others interested in FreeBSD. My interests in FreeBSD ports are quite varied; I maintain approximately 40 ports of various descriptions. Before I became a ports committer, I participated regularly in ports related bug busting weekends. Since becoming a committer,I worked with the FreeBSD KDE team that was instrumental in introducing KDE 4.x to the ports tree. I have also worked actively with the donations@ team, and have mentored other ports committers up through the ranks...

You can read the rest of Thomas' writeup in this PDF.

New Console Driver

Ed Schouten has been awarded a grant to write a new console driver for the FreeBSD project. The FreeBSD Foundation is excited to support Ed in providing a more efficient and user-friendly console driver.

This project will allow Ed to add an additional abstraction layer to the kernel. This new terminal layer will sit between the TTY layer, the kernel console and the actual console driver. The existing terminal emulator will be moved into the new terminal layer.

The advantage of the new layer is that the console driver itself will not have to handle any TTY semantics and will just receive a set of character drawing, filling and copying actions. This should make it easier to implement Unicode. It will also be much easier to make the boot process look nice on desktop systems such as PC-BSD).

This project will be completed by the end of December, 2009.

Travel Grants for EuroBSDCon

The FreeBSD Foundation is now accepting travel grant applications for EuroBSDCon. If you are a FreeBSD developer and need assistance with travel costs for this conferece, please submit a completed travel grant application. Applications need to be in by August 20.

Welcome!

Welcome to the FreeBSD Foundation blog!

This is a great place to learn more about what is happening within the FreeBSD Project, how the Foundation supports the Project, and what happens to the dollars you donate to the Foundation.

We have a lot of information to disseminate and many success stories to be told. It will take some time, so stay tuned.

You can be notified of new posts using this feed or twitter.