IPv6 router solicitation, Link-Local gateway, Ubuntu and a bit of WTF…

I have been using RHEL/CentOS lineage Linuces for a good while  now, and I believe the strength and distro mindset is their dedication to package dependencies.  Some packages or software that require bleeding edge or even not so recently updated/released libraries/CPAN perl modules etc don’t behave well or won’t install/compile without breaking those dependencies without installing source packages.  This paradoxally is the weakness of RHEL/CentOS, their dedication to dependencies runaway  They occasionally are a bit behind when it comes to the latest and greatest, and you cat get handcuffed if you want to remain within the confines of the dependancy sandbox.

 

In any case I wanted to toy with some software that clearly seems written for Ubuntu (first tell-tale sign is that you need to add a bunch of user-contributed repositories so you can install it….)  This software requires IPv6 connectivity so I decided to trash a BIND virtual server I was using and replace it with Ubuntu 14.04 from the .ISO.  I am not totally foreign to Ubuntu as I use Ubuntu desktop on a couple laptops, however in a server environment I have never bothered to consider it out of laziness, mostly.  Here are a few observations, some of which I am still trying to wrap my head around.

 

BIND startup

My first challenge was getting BIND to start correctly.  I usually use “listen-on-v6” to specify what interfaces I allow BIND to bind to, and was seeing this incredibly annoying message on bootup and BIND was not binding to any IPv6 addresses, defeating the purpose of running BIND on this server in the first place:


yet 2001:db8:9::37/64 is statically configured in my interface config (!!):


 

apparently, Ubuntu is in such a hurry to boot that it starts daemons before the interface init is even finished (!)  Major-wtf.  So it would seem that it is prudent (and necessary?) to put the IPv6 definition of eth0 inet6 *before* IPv4.  Guess that is good to know doh  Once thats done, BIND starts correctly and we can move on.


IPv6 default gateway gong-show

My next hurdle was not far away.  Even if I manually set the gateway, for some reason Ubuntu seems to feel it necessary to send a router solicitation… My network has regular router advertisements disabled, just because I don’t want SLAAC to work in that particular test environment.  So even if I statically configure Ubuntu with a prefix, mask and gateway, it still seems to feel a need to go exploring for routers and sends out an ICMPv6 router solicitation…so when I check the routing table:


now wait a minute!  My network does not send periodic RAs, I have statically defined my gateway, and Ubuntu overrides whatever I defined with auto-learned crap??  surprised  After 500 seconds, the RS-learned gateways disappear and whatever I defined as default then remains, and sometimes no gateway at all…..

Link-Local Gateway

Remember I mentioned above the bit about putting the IPv6 config before IPv4?  My network uses HSRP as first-hop redundancy, and IOS didn’t allow for global addressing (and some supported IOS trains do not have this feature) until “recently”.  In any case, I want to use a link-local address as a gateway.  For some reason, if the IPv6 interface in Ubuntu is defined *after* the IPv4 address, the LL gateway sometimes is ignored (what?runaway)  So not only do I get possibly bogus gateways via ICMPv6 router solicitation, I might end up with no gateway at all once those RAs lifetime is up.  not good.

The solution is to disable router-advertisement learning in /etc/sysctl.conf by adding the following:


and contrary to RHEL/CentOS (and pretty much any environment where a link-local is used to route something that I have seen), it is not necessary to specify the interface – I suppose Ubuntu is doing some logic there:

 


 


So we note the absence of %device.  I would have to test if with multiple NICs this still works.  Hopefully it does pc

 

Network restart broken

 

My last adventure was trying to restart networking in some controlled way after making changes.  I believed I had broken my install when I got the following as a result:


After some googling, apparently this is known.   So shucks, another thing that will hopefully get fixed wassat   At the rate Ubuntu seems to update packages, I suppose it might not be a long wait pc

 

 

 

Samsung Galaxy S4 – IPv6 borked….

I have been running around trying to figure out why my Samsung Galaxy S4 (Android 4.3) seems to lose its IPv6 prefix when it has been sleeping for a while. At first I blamed my edge CPE device, as I had rebuilt a OpenWRT recently and switched from WIDE-DHCPv6+RADVD to 6relayd.  And I had never noticed if the previous implementation had this issue or not.

Symptoms:  If I leave the device idle for a certain amount of time (which apparently would be longer than the preferred life of the prefix), when I open the browser I note that v6-sites are not reachable until my router sends an unsolicited RA.   I installed some free apps such as MyIPv6 and Network Info ii and although all seemed normal during phone use, after a certain time the prefixes would be non-functional, and would even not appear in these apps as existing, until the next RA sent by the router and all would resume correctly.

As well, if for whatever reason the prefix received by prefix delegation changes, an idle device (read: screen off….) will only learn of the new prefix once the route sends an RA containing that new prefix.

 

Out of desperation I started googling and apparently I am not alone:

http://developer.samsung.com/forum/board/thread/view.do?boardName=General&messageId=239890

http://code.google.com/p/android/issues/detail?id=32662

I especially liked the comment from the Samsung developers:

OK, well I tested this, as did the reporter of this issue.  Simply locking the phone (turning off the screen), with an ICMP ping to the IPv4 and IPv6 addresses respectively.  The IPv6 address of my phone ceases to reply to ICMP echo requests when I turn off the screen (as expected) –  YET IPv4 REQUESTS ARE NOT IGNORED and replies are sent by the GS4!   So what gives, Samsung devops?  This is with or without the power saving features enabled.  I don’t believe this cockamamie notion about  running down batteries – its either laziness, incompetence or an outright dismissal of a bug.

And frankly, I wouldn’t really care if ICMP echo/echo-reply data was discarded, however ND, RA and other critical v6 traffic shouldn’t be.  Or, at the very least, send a solicited RA or reset the IPv6 stack entirely, because when the phone wakes from sleep or is unlocked, the IPv6 stack is broken up until a router sends an RA.  And in my case, 6relayd seems to send one every 7 minutes or so, with a randomized +/- interval of a few seconds.  And some networks I have connected to send RAs every hour, not every few seconds, since IPv6 protocols allow for address assignment to occur without RAs….So it could be a long wait!  And even if a network sends an unsolicited-RA every 30 seconds, that is 30 seconds where your connectivity is v4-only when you benefit from a dual-stacked environment.  Not very modern-ish behaviour by a so-called industry leader!

I hate to admit it, but I tested iPhone4, 4S, 5 and iPad mini as well as Windows XP/7, and they all behave correctly when woken from whatever state they were in post-prefix-expiry.  I will be borrowing some other Android devices as I can get my hands on them to compare results.  But as of now I don’t have a very high regard for Samsung’s IPv6 implementation!

 

Samsung coders need to get a clue!  “End users can connect to networks continually by IPv4” is not an acceptable answer.  As IPv6 grows in popularity and necessity, that is a pretty bone-headed opinion.  Need I point out that at the time of this writing, that forum post is less than 6 months old!

 

BIND DNS64 and views

I am starting to experiment with IPv6-only networks and require the use of a DNS64 service to be used in conjunction with NAT64.  Luckily, BIND 9.8.x offers this capability.  However, I don’t want my DNS server to reply with a quad-A record for every query now that I have enabled DNS64, since this DNS server is used by non-v6 clients, as well as dual-stack clients that I do not want to send them searching for 64:ff9b::/96. :S  I therefore added a secondary IPv6 address on the DNS server for my v6-only clients to use specifically for DNS64, and put it in its own view.  In the config below, the BIND DNS64 service will utilize 2001:db8::64 🙂

 

acl rfc1918 { 10/8; 192.168/16; 172.16/12; };
options {
        listen-on port 53 {
                192.168.1.1;
                127.0.0.1;
        };
        listen-on-v6 port 53 {
                2001:db8::10/64;
                2001:db8::64/64;
                ::1;
        };
view "external" {
match-destinations { 2001:db8::10; ::1; 192.168.1.1; 127.0.0.1; };
[...]

}; //end view external
view "dns64" {
    match-destinations { 2001:db8::64; };
    dns64       64:ff9b::/96 {
        clients { any; };
        mapped { !rfc1918; any; };
        exclude { 64:ff9b::/96; ::ffff:0000:0000/96; };
        suffix ::;
}; //end view dns64
};

 

the corresponding digs show the difference between

dig @2001:db8::64 +short -t AAAA www.cnn.com
www.cnn.com.vgtf.net.
cnn-atl.gslb.vgtf.net.
64:ff9b::9da6:e21a
64:ff9b::9da6:e219
dig @2001:2b8::10 +short -t AAAA www.cnn.com
www.cnn.com.vgtf.net.

Looks like I am ready to start working on learning how to use TAYGA

Facebook starts to publish AAAA ahead of June 6th 2012 – Like!

I don’t know if this is a late night test, however apparently Facebook is publishing (I am guessing deliberately) a AAAA via DNS, even for “non-whitelisted” resolvers…

 

They had announced in April that they would enable AAAA for “beta.facebook.com” for May 18th, yet the plan at that time was to enable AAAA on the www. on June 6th like everyone else.  However dig currently has results that push forward that deadline:

 

dig -t AAAA @8.8.8.8 www.facebook.com

; <<>> DiG 9.3.2 <<>> -t AAAA @8.8.8.8 www.facebook.com
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13463
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.facebook.com.              IN      AAAA

;; ANSWER SECTION:
www.facebook.com.       115     IN      AAAA    2a03:2880:10:1f03:face:b00c:0:25

;; Query time: 18 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon May 21 22:53:38 2012
;; MSG SIZE  rcvd: 62

Their initial IP on June 8th 2011 was  [2620:0:1cfe:face:b00c::3] (aka www.v6.facebook.com, and www.facebook.com for the 24 hours of World IPv6 Day 2011) , which is within a /40 out of ARIN space.   beta.facebook.com and www.facebook.com do not have the same IP so I am guessing they are not necessarily the same source.

It also seems they got their own /32, from RIPE?!  Facebook Ireland LTD?  What the….

 whois 2a03:2880:10:8f02:face:b00c:0:25@whois.ripe.net
[Querying whois.ripe.net]
[whois.ripe.net]
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '2a03:2880::/32'

inet6num:        2a03:2880::/32
netname:         IE-FACEBOOK-201100822
descr:           Facebook Ireland Ltd
country:         IE
org:             ORG-FIL7-RIPE
admin-c:         RD4299-RIPE
tech-c:          RD4299-RIPE
status:          ALLOCATED-BY-RIR
mnt-by:          RIPE-NCC-HM-MNT
mnt-lower:       fb-neteng
mnt-routes:      fb-neteng
source:          RIPE # Filtered

organisation:    ORG-FIL7-RIPE
org-name:        Facebook Ireland Ltd
org-type:        LIR
address:         Facebook Ireland Ltd Hanover Reach, 5-7 Hanover Quay 2 Dublin Ireland
phone:           +0016505434800
fax-no:          +0016505435325
mnt-ref:         RIPE-NCC-HM-MNT
mnt-ref:         fb-neteng
mnt-by:          RIPE-NCC-HM-MNT
admin-c:         PH4972-RIPE
source:          RIPE # Filtered

role:            RIPE DBM
address:         1601 Willow Rd.
address:         Menlo Park, CA, 94025
admin-c:         PH4972-RIPE
tech-c:          PH4972-RIPE
nic-hdl:         RD4299-RIPE
mnt-by:          fb-neteng
source:          RIPE # Filtered

Maybe they are looking for ways to make news now that their IPO fell flat after the first full day of public trading? Nevertheless, good to see them venture forward!

 

More to read about June 6th 2012 – click the image below

6to4 – whats in it for me??

We are a month away from world IPv6-day(bis), which was arguably a yawnfest last year.   It was a huge success in the fact that it happened, and other than a small handful of people, nobody noticed 😛    Essentially, many heavy hitters on the Internet such as Google, Facebook, and a bunch of others, turned on their quad-As for 24 hours, as an “experiment”.  The Internet didn’t blow up, everybody pretty much went on business as usual – and for some reason, that tiny fraction of people with broken IPv6 stacks was enough to have those AAAAs removed from DNS.

June 6th 2012 will be the second coming of “World IPv6 Day”, and this time, the theory is that AAAAs will be permanently added to DNS……ZMFG!

And as network operators, we are striving to dual-stack our networks end to end, so that all our users can benefit from having IPv6 connectivity on their networks before that day, right?!  Well unfortunately not all of us have managed that 🙁  My company included.  I guess I have nobody to blame but myself.  We have some progress made, but not as much as I would have liked.  Considering many companies still have zero deployment, by north-American standards I am doing ok 😉

However, this does not mean that nobody will be, or is currently using, some form of IPv6, be it over a Hurricane Electric tunnel broker, 6RD, or even 6to4.   6to4 is particularily relevant since Windows7 (Vista as well?), Windows 2003 and probably other OSs have 6to4 enabled interfaces on by default.  I personally don’t like 6to4 very much, and I hope that it dies very, very soon; however if some folks are necessarily using it, and you haven’t gotten around to dual-stacking everyone, might as well make 6to4 slightly more useable, or at the very least, help it limp along.  One of the inherent operational issues with 6to4 is that it depends on finding a relay server, in order to go from the IPv4 world to IPv6, and vice-versa.  It would be a good idea to set up, as a network operator, your own 6to4 relay, in order to at least know where in the network your users are connecting, since 6to4 traffic will “find” the nearest one, and it can be anywhere; best it be on your network.

There is an RFC which deals specifically with 6to4 from the network operator’s perspective.

I won’t try and explain all of 6to4 here, however in essence, IPv6 prefix 2002::/16 is a reserved prefix that IPv4 hosts can derive their 6to4 address from.  As an example, 192.168.1.1 would derive its 6to4 IP to be:

192 (dec) == C0 (hex)
168 (dec) == A8 (hex)
1 (dec) == 1 (hex)
1 (dec) -- 1 (hex)

This becomes, in 6to4 – 2002:C0A8:101::/48    This unique prefix can then be routed back and forth from v4 to v6 and back over 6to4 relays.  It is the 2002:V4ADDR::/48 prefix.   A reserved anycast address, 192.88.99.1 (which microsoft seems to discover by querying 6to4.ipv6.microsoft.com), is used as a relay to the IPv6 network.  (note: RFC 1918 addressing and 6to4 prefixes cannot exist or work on the Internet, I only use them here for example addresses….)

To illustrate the point, if I disable my 6to4 relay, I see that Hurricane Electric is nicely anycasting 192.88.99.1 and would be the nearest 6to4 relay I can use.  A traceroute to ARIN over 6to4:

C:\Users\Administrator>tracert -6 www.arin.net

Tracing route to www.arin.net [2001:500:4:13::80]
over a maximum of 30 hops:

  1     *        *        *     Request timed out.
  2   116 ms   116 ms   116 ms  2001:16d8:1:6240::1
  3   110 ms   110 ms   111 ms  10gigabitethernet1-2.core1.sto1.he.net [2001:7f8:d:ff::187]
  4   110 ms   120 ms   110 ms  10gigabitethernet3-2.core1.ams1.he.net [2001:470:0:22f::1]
  5   111 ms   122 ms   111 ms  10gigabitethernet1-4.core1.lon1.he.net [2001:470:0:3f::1]
  6   114 ms   111 ms   112 ms  10gigabitethernet7-4.core1.nyc4.he.net [2001:470:0:128::1]
  7   118 ms   121 ms   124 ms  10gigabitethernet2-3.core1.ash1.he.net [2001:470:0:36::1]
  8   118 ms   126 ms   118 ms  arin.10gigabitethernet14.switch3.ash1.he.net [2001:470:1:20f::2]
  9   118 ms   118 ms   118 ms  cr2.arin.net [2001:500:4:10::12]
 10   121 ms   121 ms   121 ms  cr3-ptp.arin.net [2001:500:4:11::2]
 11   122 ms   163 ms   122 ms  2001:500:4:12::3
 12   121 ms   125 ms   122 ms  www.arin.net [2001:500:4:13::80]

Trace complete.

100+ms round trip.  Not terrible, not great.

What if I was to run a 6to4 relay for my users?  Here is the result:

C:\Users\Administrator>tracert -6 www.arin.net

Tracing route to www.arin.net [2001:500:4:13::80]
over a maximum of 30 hops:

  1     2 ms     1 ms     1 ms  2002:aaaa:5ef1::
  2    19 ms    14 ms    19 ms  2001:aaaa:2:16::ff
  3    18 ms    17 ms    18 ms  te0-3-0-0.ccr22.ymq02.atlas.cogentco.com [::ffff:154.54.0.13]
  4    19 ms    18 ms    19 ms  te0-6-0-6.ccr22.jfk02.atlas.cogentco.com [::ffff:154.54.46.46]
  5    18 ms    19 ms    18 ms  te0-3-0-3.ccr22.dca01.atlas.cogentco.com [::ffff:154.54.26.181]
  6    19 ms    18 ms    18 ms  te0-0-0-2.ccr22.iad02.atlas.cogentco.com [::ffff:154.54.1.178]
  7    18 ms    18 ms    17 ms  2001:550:3::24a
  8    31 ms    23 ms    49 ms  2001:578:1:0:172:17:249:18
  9    19 ms    19 ms    20 ms  2001:578:2800:5::27
 10    19 ms    20 ms    19 ms  2001:578:2800:5::90
 11    26 ms    20 ms    20 ms  2001:578:2803:1100::2
 12    20 ms    21 ms    21 ms  cr3.arin.net [2001:500:4:12::1]
 13    21 ms    21 ms    21 ms  2001:500:4:12::3
 14    23 ms    23 ms    21 ms  www.arin.net [2001:500:4:13::80]

Trace complete.

Neat! 25ms or so, a fraction of the round trip time the “nearest” 6to4 relay could offer me.  Success!

Of course, it must be noted that as a network operator, you can only influence the relay used towards the IPv6 network – the nearest relay from the destination node back to the IPv4 network will be used.  That is one of the drawbacks of 6to4 – although there can be relays peppered all over the Internet, the one used to get to IPv6 will in most likelyhood not be the one used to get back to IPv4.  But the point is to improve 6to4, so hosting your own relay is favourable for your customers.

 

The setup is insanely and trivially easy.  Here is an example Cisco config:

!
ipv6 unicast-routing
ipv6 cef
!
interface Tunnel0
 description 6to4 Tunnel
 no ip address
 ipv6 address 2002:C0A8:101:::/128
 ipv6 enable
 tunnel source Loopback6
 tunnel mode ipv6ip 6to4
!
!
interface Loopback6
 ip address 192.88.99.1 255.255.255.0 secondary
 ip address 192.168.1.1 255.255.255.0
 ipv6 address 2001:db8:FFFF:FFFF::1/64
 ipv6 enable
!
ipv6 route 2002::/16 Tunnel0
!

 

voila!  In the above examples, substitute 2002:C0A8:101:: and 192.168.1.1 for the IP address of your router used for 6to4 and you are off to the races.   Make sure you distribute 192.88.99.0/24 and 2002::/16 in your IGP.