Bandwidth Limiting with pf and ALTQ

Eric Hodel | Tue, 28 Jul 2009 05:34:00 GMT

Every now and then my neighbors end up chewing up too much of my bandwidth. In order to give my computers a more-reliable connection and keep the number of access points in my neighborhood low, I started limiting their traffic using ALTQ in the pf packet filter.

My router runs FreeBSD with a connection to the internet on one side and an Airport Extreme on the other end. My router handles all the network configuration (DHCP, DNS, firewall) and the Extreme is just a bridge. I've classified my hosts on the wifi network into hosts I know about that get assigned fixed addresses via DHCP and everybody else that gets assigned an address out of a DHCP pool.

ALTQ allows you to create a hierarchy of queues each with a particular amount of bandwidth. If a queue fills up width bandwidth you can either throw out the extra packets or borrow from one of the other queues. You can get some ideas of how to set up a hierarchy of queues from the examples on the ALTQ page.

For my network I chose to start with two queues one for my hosts and one for everybody else: 

altq on $wifi_if cbq bandwidth 100Mb queue { mine, other }

queue mine  bandwidth 98Mb priority 1 cbq(borrow)
queue other bandwidth 2Mb  priority 7 cbq(default ecn)

This is much simpler than the ALTQ examples which allow prioritization of ACK packets and interactive sessions like SSH or IM connections, but I really don't care about the service of everybody else.

Since the queues are hierarchical, you end up with a 100Mb queue for all traffic on my wifi interface split into a 98Mb queue for myself and a 2Mb queue for everybody else. My queue gets top priority and can borrow bandwidth from everybody else. The other queue picks up the rest of the traffic and uses Explicit Congestion Notification as it fills up (which includes dropping packets).

Now we have to assign packets to the queues: 

my_hosts = "192.0.2.1/27"

pass  out quick on $wifi_if from any to $my_hosts queue mine
pass  out       on $wifi_if from any to any       queue other

After reloading pf and letting traffic go by for a while you can see the queue in action: 

$ sudo pfctl -v -s queue
queue root_vr2 bandwidth 100Mb priority 0 cbq( wrr root ) {mine, other}
  [ pkts:     134533  bytes:  151561160  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50  borrows:      0  suspends:      0 ]
queue  mine bandwidth 98Mb cbq( borrow ) 
  [ pkts:      40780  bytes:   56881335  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50  borrows:      0  suspends:      0 ]
queue  other bandwidth 2Mb priority 7 cbq( red ecn default ) 
  [ pkts:      93753  bytes:   94679825  dropped pkts:   1022 bytes: 1237280 ]
  [ qlength:   0/ 50  borrows:      0  suspends:   6933 ]

You can get more information from the ALTQ page and from the QUEUING/ALTQ section of the pf.conf man page.

Posted in  | no comments

Mail Hosting

Eric Hodel | Sat, 09 May 2009 04:28:43 GMT

I host my own mail using the following software:

OpenBSD’s spamd sits in front of Postfix and greylists and blacklists for me (it runs on all my MXs to prevent spammers from sneaking around). spamd runs via a firewall rule that redirects unknown connections to the spamd daemon during the greylist period then later whitelists them for direct connection to Postfix. I’ve also added a few spam-collecting addresses to the spamtrap list to help with automatic blacklisting.

Postfix directs email through amavisd for spam control which allows me to bounce spam (from amavisd’s Postfix README). I have amavisd configured to use Postgresql to enable its pen pals feature which lowers spam scores for frequent correspondents (from amavisd’s SQL README and Postgresql README).

Postfix hands mail off to procmail via my .forward file (not mailbox_command) which is just "|/usr/local/bin/procmail -tf-". (It seems that other values you see around, like setting IFS, are to work around bugs in ancient versions of sendmail.)

In procmail, I use dovecot’s deliver to keep its indexes updated. A sample from my .procmailrc:

DELIVER = /usr/local/libexec/dovecot/deliver

# ...

:0 w
* List-Id:.*<rubygems-developers.rubyforge.org>
| ${DELIVER} -m Lists/Ruby/Rubygems

# ...

# last rule, delivers to INBOX
:0 w
| ${DELIVER}

My outbound mail goes through the submission port using dovecot for SASL authentication and gets filtered by amavisd for the pen pals feature and DKIM signing (set up per amavisd’s DKIM documentation).

I’ve set up DKIM and SPF records for my domain in order to be a good internet citizen. You can get an SPF record pretty quick from the SPF record wizard. I have SpamAssassin using DKIM verification for improved filtering.

To ensure delivery of my mail, one of my backup MXs is at my home for connection redundancy and the second is Ryan Davis’ primary MX (we have mutual backups). A backup MXs out of my control prevents me from losing mail by screwing up both machines under my control.

My IMAP clients include Apple’s Mail for day-to-day mail reading, IMAPCleanse for cleaning out my lists and flagging threads I should follow up on. Soon I’ll be adding an IMAP to RSS tool for mail with lots of unimportant stuff (like Amazon, bank transfers, etc.).

Posted in ,  | no comments

ZFS for FreeBSD

Eric Hodel | Mon, 28 Aug 2006 01:07:02 GMT

Hi.

I started porting the ZFS file system to the FreeBSD operating system.

There is a lot to do, but I’m making good progress, I think.

[...]

You can find all those on FreeBSD perforce server:

http://perforce.freebsd.org/depotTreeBrowser.cgi?FSPC=//depot/user/pjd/zfs&HIDEDEL=NO

Ok, so where am I?

I ported the userland bits (libzfs, zfs and zpool). I had ztest and libzpool compiling and working as well, but I left them behind for now to focus on kernel bits.

I’m building in all (except 2) files into zfs.ko (kernel module).

I created new VDEV – vdev_geom, which fits to FreeBSD’s GEOM infrastructure, so basically you can use any GEOM provider to build your ZFS pool. VDEV_GEOM is implemented as consumers-only GEOM class.

I reimplemented ZVOL to also export storage as GEOM provider. This time it is providers-only GEOM class.

This way one can create for example RAID-Z on top of GELI encrypted disks or encrypt ZFS volume. The order is free. Basically you can put UFS on ZFS volumes already and it behaves really stable even under heavy load.

Currently I’m working on file system bits (ZPL), which is the most hard part of the entire ZFS port, because it talks to one of the most complex part of the FreeBSD kernel – VFS.

I can already mount ZFS-created file systems (with ‘zfs create’ command), create files/directories, change permissions/owner/etc., list directories content, and perform few other minor operation.

— Porting ZFS file system to FreeBSD via freebsd-current

Eventually this will make its way into -current, then -stable. Yay!

Posted in  | 4 comments

FreeBSD 6 vs. Broadcom NetXtreme II

Eric Hodel | Sun, 20 Aug 2006 22:20:41 GMT

When we first got our new Dell PowerEdge development box the Broadcom NetXtreme II would occasionally report Error mapping mbuf into TX chain followed by the NIC no longer responding to ethernet. If you’ve got a new Dell, here’s how to update if_bce on FreeBSD 6 to fix the error.

Posted in  | no comments

socket_sendfile and socket_accept_filter

Eric Hodel | Fri, 24 Mar 2006 01:19:00 GMT

These are two packages that I use to speed up WEBrick image serving now freshly released. Unfortunately I haven’t tested them on any platform other than FreeBSD so please file bugs if they don’t work for you.

$ sudo gem install socket_sendfile
$ sudo gem install socket_accept_filter

socket_sendfile adds sendfile(2) to Socket and forms the cornerstone of our WEBrick image serving.

socket_accept_filter makes it easy to set the SO_ACCEPTFILTER socket option so you can enable the accf_http(9) and accf_data(9) accept filters. Accept filters delay the return from accept(2) until enough data has arrived on the socket for processing.

Shortly I’ll have enough software released to do a full write-up of high-volume image serving with WEBrick.

Posted in , ,  | 2 comments

MogileFS + Ruby take one

Eric Hodel | Sun, 30 Oct 2005 04:21:00 GMT

Right now I just support file operations, and only over NFS since FreeBSD + HTTP mode don’t like each other. I’ll get the admin operations finished next, then I’ll have something shippable.

Posted in ,

MogileFS + NFS = hard

Eric Hodel | Sat, 29 Oct 2005 11:26:00 GMT

I couldn’t get HTTP mode working, but I didn’t try very hard. It looks like the components only conditionally include Linux::AIO, but I’m not sure.

I submitted a documentation patch for MogileFS + NFS to the mailing list to help out the next people who try this. Its really only two extra sentences that you need.

Posted in ,

Safety Belts

Eric Hodel | Fri, 28 Oct 2005 22:53:00 GMT

Yes, I know Linux runs faster, but they can do that because they have thrown out the weight of the airbag, collision frame and safety belt.

—Poul-Henning Kamp

Posted in

FreeBSD, Linux and Timekeeping

Eric Hodel | Fri, 28 Oct 2005 18:12:00 GMT

I’ve been reading a fascinating thread on the freebsd-current mailing list about why MySQL on FreeBSD performs so much less well than Linux.

Apparently MySQL makes many, many calls to the kernel to figure out what time it is so that it can keep track of various things. On Linux this is no problem because they’ve got sloppy timekeeping in their kernel, but FreeBSD is much more accurate.

It eventually lead to a discussion of timers and timing in FreeBSD and how to get the current time faster.

Part of discussion reveals that on the x86 architecture it is incredibly difficult to get the current time in a fast and reliable way.

Posted in