With IPv6 on Linux, things aren’t so straightforward. Duplicate Address Detection (DAD), which basically does what it says on the tin, introduces a short delay before addresses are fully configured, the address has been added to the network interface, but not really.

I recently came across this whilst attempting to get BIND to listen on some secondary service addresses on a particular machine. BIND would not er, bind, to the IPv6 addresses at boot, failing with messages like this:

bind9 could not listen on UDP socket: address not available

Modifying /etc/init.d/bind9 to print the output of “ip addr show” to a file at the time BIND attempted to start up showed the tell-tale “tentative” flag on each IPv6 address being added to eth0. Since the addresses are in this state, BIND or other daemons will refuse to listen on them.

The problem has become very noticeable since parallel boot systems such as Upstart have become the default in quite a few Linux distros. Daemons will often fire up before the network is fully ready and in some extreme cases network filesystems that reference hostnames may fail to mount if you are using an IPv6 DNS resolver. Of course this isn’t the case across the board, some daemons and services appear to handle the unavailability of an IPv6 address somewhat gracefully, backing off and trying again a short time later rather than simply giving up on the first go.

Anyway, a simple “hairy hack” to get over this problem is to add something like the following to your startup script:

sleep 5

Yes, a one line sleep command to make the daemon wait a short while before actually starting. This seems to ensure that the IPv6 address has moved out of the tentative state, but it’s still somewhat silly..

AIB helpfully publish SPF records for the aib.ie domain, so the first port of call was SAs SPF lookups. Which I noticed weren’t happening, despite being enabled some time ago. Reloading SA resulting in the following log message in spamd.log:

Tue Aug 11 11:26:52 2009 [2221] warn: Error creating a DNS resolver
socket: at /usr/share/perl5/Mail/SpamAssassin/DnsResolver.pm line 233.

Aw snap, SA can’t do any DNS lookups, at all. Which means all of the DNS-based tests will be silently skipped. Going back through the spamd.log, I noticed these messages first started occurring when we enabled IPv6 DNS resolvers a number of months ago. Hmm, perl is obviously missing something fundamental.

So, it turns out a perl library required to create IPv6 sockets wasn’t installed, IO::Socket::INET6. This is conveniently packaged in Ubuntu:

aptitude install libio-socket-inet6-perl

A quick reload of spamassassin and we can say goodbye to all those “URGENT NOTIFICATION”s about our AIB online banking accounts. Although, the spammers have now copped on and aren’t even bothering to send from aib.ie addresses anymore..

Anyway, after reading through some forum posts, here is a sample haresources file that will give you IPv4 and IPv6 address failover. We are using eth1 as our primary network interface:

my.failover.address 193.1.219.93
my.failover.address IPv6addr::2001:770:18:2:0:0:c101:db5d/64/eth1

The important thing is to specify the IPv6 address fully, ie: pad it out with 0′s, no :: shortcuts!

With v6 support recently added to IPVS, it should now be possible to do full IPv6 failover and load balancing.

Deadly!

For some background; we used the ng_fec (4) device on FreeBSD 6.2-RELEASE to make a logical network interface out of two physical interfaces. This is referred to as bonding on Linux, IP Multipathing on Solaris and I think Cisco call it Ether Channel. Whatever way you go about it, the end result is that you get a network interface that is reachable via a multiple different paths. If each physical interface is plugged into a different switch, you get a nice amount of resiliency.

This works great in FreeBSD with ng_fec, however if you try to use an IPv6 address on the logical interface thats where things start to get interesting. Without getting into too much detail – the problems relate to how IPv6 maps IP addresses to MAC addresses, this is done by way of “neighbour advertisement” messages sent via multicast. However the ng_fec driver was not telling the physical network interfaces to join the correct multicast groups so these neighbour advertisement messages could not be processed correctly, which led to very flaky and mostly unavailable IPv6 connectivity. From what I can gather, you can use other drivers in -CURRENT that don’t exhibit this problem.

But with some diagnoses and a patch to ng_fec.c from Dave Malone we managed to correct the issue. This may make it up into the FreeBSD source tree at some point.

Hopefully things will stay stable from now on

Edit: Ah, there is a bug report raised for this already.