Time Flies – BIND, CentOS4 and Spammers

Its been over a month since I have been able to take a moment and say hello to my blog.  After my own time off, my colleagues have dared to take their own time off and I am stuck with extra work while they are gone.  What goes around, comes around, I guess.


In the meantime, the only work I have been able to do around here is moderate spammy comments to the trashbin.  Viagra, Online Loans, Escort services, they all seem to come over here with their bots and spam my admin panel with bogus comments with links back to their crap…



One thing I guess they are trying to exploit are the WordPress pingbacks.  Much to my annoyance since I really don’t mind speaking out loud to nobody on this blog, its crappy to have bots polluting my space with their spam.   I might try a WordPress CAPTCHA plugin of some sort, was too lazy to look for one since I didn’t figure I needed one….little did I know!


In any case, as time goes on, people still (*shiver*) running RHEL4/CentOS4 systems are getting more and more vulnerabilities.  CVE-2012-1823 being the first reasonably big bug that will go unpatched with RHEL4/CentOS4, and now BIND has announced CVE-2012-1667, a rather nasty one that apparently can in some cases expose system memory (!!!).  RHEL4 and clones shiiped with BIND 9.2, and it is of course vulnerable.  Thankfully systems I manage have been moved past EL4, however I do have a server or two that is still yet to be replaced.  I came across a blog that had shown promise of a rebuild of BIND 9.7.3 (which ships with CentOS 6.2 I believe), yet I was never able to get the .src.rpm for that.  I ended up building my own.  Its based on Fedora Linux’s FC14 9.7.4 RPM, with the 9.7.6_P1 source from ISC.ORG.  You can download my .src.rpm here.  I have tested it happily for a few weeks now, your mileage may vary.  Rebuild at your own risk! 🙂  I have it running on a mail server for DNS caching as well as a reasonably busy authoritative server and so far, no issues at all.    At least now RHEL4/CentOS4 can have allow-recursion { acl; } in named.conf!!! (yay)

Anyway, if you do download the source and compile it, let me know your results.


Facebook starts to publish AAAA ahead of June 6th 2012 – Like!

I don’t know if this is a late night test, however apparently Facebook is publishing (I am guessing deliberately) a AAAA via DNS, even for “non-whitelisted” resolvers…


They had announced in April that they would enable AAAA for “beta.facebook.com” for May 18th, yet the plan at that time was to enable AAAA on the www. on June 6th like everyone else.  However dig currently has results that push forward that deadline:


dig -t AAAA @ www.facebook.com

; <<>> DiG 9.3.2 <<>> -t AAAA @ www.facebook.com
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13463
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;www.facebook.com.              IN      AAAA

www.facebook.com.       115     IN      AAAA    2a03:2880:10:1f03:face:b00c:0:25

;; Query time: 18 msec
;; WHEN: Mon May 21 22:53:38 2012
;; MSG SIZE  rcvd: 62

Their initial IP on June 8th 2011 was  [2620:0:1cfe:face:b00c::3] (aka www.v6.facebook.com, and www.facebook.com for the 24 hours of World IPv6 Day 2011) , which is within a /40 out of ARIN space.   beta.facebook.com and www.facebook.com do not have the same IP so I am guessing they are not necessarily the same source.

It also seems they got their own /32, from RIPE?!  Facebook Ireland LTD?  What the….

 whois 2a03:2880:10:8f02:face:b00c:0:25@whois.ripe.net
[Querying whois.ripe.net]
% This is the RIPE Database query service.
% The objects are in RPSL format.
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '2a03:2880::/32'

inet6num:        2a03:2880::/32
netname:         IE-FACEBOOK-201100822
descr:           Facebook Ireland Ltd
country:         IE
org:             ORG-FIL7-RIPE
admin-c:         RD4299-RIPE
tech-c:          RD4299-RIPE
status:          ALLOCATED-BY-RIR
mnt-by:          RIPE-NCC-HM-MNT
mnt-lower:       fb-neteng
mnt-routes:      fb-neteng
source:          RIPE # Filtered

organisation:    ORG-FIL7-RIPE
org-name:        Facebook Ireland Ltd
org-type:        LIR
address:         Facebook Ireland Ltd Hanover Reach, 5-7 Hanover Quay 2 Dublin Ireland
phone:           +0016505434800
fax-no:          +0016505435325
mnt-ref:         RIPE-NCC-HM-MNT
mnt-ref:         fb-neteng
mnt-by:          RIPE-NCC-HM-MNT
admin-c:         PH4972-RIPE
source:          RIPE # Filtered

role:            RIPE DBM
address:         1601 Willow Rd.
address:         Menlo Park, CA, 94025
admin-c:         PH4972-RIPE
tech-c:          PH4972-RIPE
nic-hdl:         RD4299-RIPE
mnt-by:          fb-neteng
source:          RIPE # Filtered

Maybe they are looking for ways to make news now that their IPO fell flat after the first full day of public trading? Nevertheless, good to see them venture forward!


More to read about June 6th 2012 – click the image below

6to4 – whats in it for me??

We are a month away from world IPv6-day(bis), which was arguably a yawnfest last year.   It was a huge success in the fact that it happened, and other than a small handful of people, nobody noticed 😛    Essentially, many heavy hitters on the Internet such as Google, Facebook, and a bunch of others, turned on their quad-As for 24 hours, as an “experiment”.  The Internet didn’t blow up, everybody pretty much went on business as usual – and for some reason, that tiny fraction of people with broken IPv6 stacks was enough to have those AAAAs removed from DNS.

June 6th 2012 will be the second coming of “World IPv6 Day”, and this time, the theory is that AAAAs will be permanently added to DNS……ZMFG!

And as network operators, we are striving to dual-stack our networks end to end, so that all our users can benefit from having IPv6 connectivity on their networks before that day, right?!  Well unfortunately not all of us have managed that 🙁  My company included.  I guess I have nobody to blame but myself.  We have some progress made, but not as much as I would have liked.  Considering many companies still have zero deployment, by north-American standards I am doing ok 😉

However, this does not mean that nobody will be, or is currently using, some form of IPv6, be it over a Hurricane Electric tunnel broker, 6RD, or even 6to4.   6to4 is particularily relevant since Windows7 (Vista as well?), Windows 2003 and probably other OSs have 6to4 enabled interfaces on by default.  I personally don’t like 6to4 very much, and I hope that it dies very, very soon; however if some folks are necessarily using it, and you haven’t gotten around to dual-stacking everyone, might as well make 6to4 slightly more useable, or at the very least, help it limp along.  One of the inherent operational issues with 6to4 is that it depends on finding a relay server, in order to go from the IPv4 world to IPv6, and vice-versa.  It would be a good idea to set up, as a network operator, your own 6to4 relay, in order to at least know where in the network your users are connecting, since 6to4 traffic will “find” the nearest one, and it can be anywhere; best it be on your network.

There is an RFC which deals specifically with 6to4 from the network operator’s perspective.

I won’t try and explain all of 6to4 here, however in essence, IPv6 prefix 2002::/16 is a reserved prefix that IPv4 hosts can derive their 6to4 address from.  As an example, would derive its 6to4 IP to be:

192 (dec) == C0 (hex)
168 (dec) == A8 (hex)
1 (dec) == 1 (hex)
1 (dec) -- 1 (hex)

This becomes, in 6to4 – 2002:C0A8:101::/48    This unique prefix can then be routed back and forth from v4 to v6 and back over 6to4 relays.  It is the 2002:V4ADDR::/48 prefix.   A reserved anycast address, (which microsoft seems to discover by querying 6to4.ipv6.microsoft.com), is used as a relay to the IPv6 network.  (note: RFC 1918 addressing and 6to4 prefixes cannot exist or work on the Internet, I only use them here for example addresses….)

To illustrate the point, if I disable my 6to4 relay, I see that Hurricane Electric is nicely anycasting and would be the nearest 6to4 relay I can use.  A traceroute to ARIN over 6to4:

C:\Users\Administrator>tracert -6 www.arin.net

Tracing route to www.arin.net [2001:500:4:13::80]
over a maximum of 30 hops:

  1     *        *        *     Request timed out.
  2   116 ms   116 ms   116 ms  2001:16d8:1:6240::1
  3   110 ms   110 ms   111 ms  10gigabitethernet1-2.core1.sto1.he.net [2001:7f8:d:ff::187]
  4   110 ms   120 ms   110 ms  10gigabitethernet3-2.core1.ams1.he.net [2001:470:0:22f::1]
  5   111 ms   122 ms   111 ms  10gigabitethernet1-4.core1.lon1.he.net [2001:470:0:3f::1]
  6   114 ms   111 ms   112 ms  10gigabitethernet7-4.core1.nyc4.he.net [2001:470:0:128::1]
  7   118 ms   121 ms   124 ms  10gigabitethernet2-3.core1.ash1.he.net [2001:470:0:36::1]
  8   118 ms   126 ms   118 ms  arin.10gigabitethernet14.switch3.ash1.he.net [2001:470:1:20f::2]
  9   118 ms   118 ms   118 ms  cr2.arin.net [2001:500:4:10::12]
 10   121 ms   121 ms   121 ms  cr3-ptp.arin.net [2001:500:4:11::2]
 11   122 ms   163 ms   122 ms  2001:500:4:12::3
 12   121 ms   125 ms   122 ms  www.arin.net [2001:500:4:13::80]

Trace complete.

100+ms round trip.  Not terrible, not great.

What if I was to run a 6to4 relay for my users?  Here is the result:

C:\Users\Administrator>tracert -6 www.arin.net

Tracing route to www.arin.net [2001:500:4:13::80]
over a maximum of 30 hops:

  1     2 ms     1 ms     1 ms  2002:aaaa:5ef1::
  2    19 ms    14 ms    19 ms  2001:aaaa:2:16::ff
  3    18 ms    17 ms    18 ms  te0-3-0-0.ccr22.ymq02.atlas.cogentco.com [::ffff:]
  4    19 ms    18 ms    19 ms  te0-6-0-6.ccr22.jfk02.atlas.cogentco.com [::ffff:]
  5    18 ms    19 ms    18 ms  te0-3-0-3.ccr22.dca01.atlas.cogentco.com [::ffff:]
  6    19 ms    18 ms    18 ms  te0-0-0-2.ccr22.iad02.atlas.cogentco.com [::ffff:]
  7    18 ms    18 ms    17 ms  2001:550:3::24a
  8    31 ms    23 ms    49 ms  2001:578:1:0:172:17:249:18
  9    19 ms    19 ms    20 ms  2001:578:2800:5::27
 10    19 ms    20 ms    19 ms  2001:578:2800:5::90
 11    26 ms    20 ms    20 ms  2001:578:2803:1100::2
 12    20 ms    21 ms    21 ms  cr3.arin.net [2001:500:4:12::1]
 13    21 ms    21 ms    21 ms  2001:500:4:12::3
 14    23 ms    23 ms    21 ms  www.arin.net [2001:500:4:13::80]

Trace complete.

Neat! 25ms or so, a fraction of the round trip time the “nearest” 6to4 relay could offer me.  Success!

Of course, it must be noted that as a network operator, you can only influence the relay used towards the IPv6 network – the nearest relay from the destination node back to the IPv4 network will be used.  That is one of the drawbacks of 6to4 – although there can be relays peppered all over the Internet, the one used to get to IPv6 will in most likelyhood not be the one used to get back to IPv4.  But the point is to improve 6to4, so hosting your own relay is favourable for your customers.


The setup is insanely and trivially easy.  Here is an example Cisco config:

ipv6 unicast-routing
ipv6 cef
interface Tunnel0
 description 6to4 Tunnel
 no ip address
 ipv6 address 2002:C0A8:101:::/128
 ipv6 enable
 tunnel source Loopback6
 tunnel mode ipv6ip 6to4
interface Loopback6
 ip address secondary
 ip address
 ipv6 address 2001:db8:FFFF:FFFF::1/64
 ipv6 enable
ipv6 route 2002::/16 Tunnel0


voila!  In the above examples, substitute 2002:C0A8:101:: and for the IP address of your router used for 6to4 and you are off to the races.   Make sure you distribute and 2002::/16 in your IGP.

Truth in advertising?

So I happen to glance at a local paper’s full-page advert for discount electronics at “Distribution Price Buster”, and my eye catches the ads for 7 inch and 10 inch android tablets.   I am always curious what Chinese company has made low-cost tablets for the masses at ridiculously low prices.


79$ for 7 inch tablet???    !!!129$ for a 10 inch android tablet!!!!  WOW indeed!  I wonder how much storage they have?   But wait a second – what version of Android is that?  Honeycomb?  Ice Cream Sandwich?  Gingerbread?!  Looking closely….


Its apparently iOS!   As seen on TV!


Hopefully it runs Angry Birds.


🙂 🙂


PHP updated – CVE-2012-1823 / CVE-2012-2311


A bug in PHP 5 was released, somewhat accidentally (apparently somebody made a reddit post public inadvertently…why people would use these sites for sensitive info is beyond me…anyhoo), and is finally patched by RedHat and derivatives, CentOS, ScientificLinux –


Ubuntu has also released a patch –


RHEL5&6 would be no longer vulerable to either CVE-2012-1823 or its re-incarnation CVE-2012-2311, as the bug was not correctly patched the first time around.   RedHat claims their fix is complete, I cannot vouch for Ubuntu so don’t blame me if you have to patch it again later.


The vulnerability itself was quite old, sneaking into the code 8 years ago and lying undiscovered until recently.  It relies on the use of php-cgi (running PHP as a cgi forked process, not in the more mainstream mod_php mechanism).  One of the many consequences of this bug was source code exposure (via ?-s), and many PHP sites having database username/password information contained in the PHP code, this vulnerability could and will compromise sites where this is the case that remain unpatched.  This is one of the many reasons it is a better idea to use PHP include functionality to provide database/user/password/security connnection info to PHP, and have that included file outside of the http webroot in the first place.  Other possible exploits would be to execute code/upload files on the remote filesystem….scary, nasty stuff!!!!

Even the allmighty Facebook, set for an IPO this week, was vulnerable to this (they were running as a cgi, apparently!!)

The IPO could have been something of a bust had Facebook been hacked a week before it hopes to raise 95$ billion USD.  Who would have seen that one coming?!?!?!?!


/bin/false, /sbin/nologin and SSH

It is common knowledge that if on a linux system, a local account is set to /bin/false or /sbin/nologin, that user cannot use SSH, right?




As one of my clients learned the hard way.  Alerted to the fact his server was blacklisted on a (large!) number of RBLs, we started investigating how and why spam was getting sent from his server.  What we found was that a local user account (listed in /etc/passwd) had been compromised, due to an easy to guess password.

On all the recent Linux distributions I attempted this on, it is possible to connect to the SSH daemon and initiate not a shell, but a TCP port forwarding session. Using /sbin/nologin as a shell only causes the SSH daemon to spawn a shell that closes immediately. So if you tell SSH not to spawn a shell, and use TCP port forwardings instead, you can then exploit the remote host.

Here is how it works.


if I create a user with [/bin/false|/sbin/nologin] as a shell, and give him a password. I can then SSH to the machine and forward a local port.

Like this:

[user@panel~]ssh -N testacct@linuxserver.cconn.info -L 2525:

this forwards my local port of 2525 to the remote port 25. Allowing me to spam via the remote server, via my local TCP 2525 socket I just created:

[user@panel ~] telnet localhost 2525
 Trying ::1...
 Connected to localhost.
 Escape character is '^]'.
 220 linuxserver.cconn.info ESMTP Sendmail 8.14.4/8.14.4; Sun, 6 May 2012 18:06:57 -0400

From “panel” I can connect to “linuxserver” over my ssh tunnel, and spam…my email appears to come from when you look at the headers, causing a certain level of difficulty in tracing the spam back to that account after the fact;

Received: from foo (localhost [])
 by linuxserver.cconn.info (8.14.4/8.14.4) with SMTP id q36MAbio019457
 for spambait@sucker.com; Sun, 6 May 2012 18:10:48 -0400

Why does this work?  Because sshd by default is configured, on my distros anyway, to use PAM for authentication, and has TCP port forwardings enabled by default.  If a shell listed in /etc/shells is configured for the account (/sbin/nologin is listed in /etc/shells), the password is valid and TCP port forwarding is enabled, this allows an account to establish said port forwardings and utilize services on the remote host.  Spam is an example, but there could be more.

How can this be fixed?


Well, there are a number of ways.  The simplest and most drastic, if possible in your scenario, is to simply disable TCP port forwarding in your sshd configuration file (/etc/ssh/sshd_config on many systems):

AllowAgentForwarding no
AllowTcpForwarding no
X11Forwarding no

However, note that this will not deny the actual ssh connection to be established; it will simply deny the ability to exploit the forwarding (see below);

ssh -N testacct@linuxserver.cconn.info -L 2525:
testacct@tohs.cconn.info's password:
channel 2: open failed: administratively prohibited: open failed
channel 2: open failed: administratively prohibited: open failed

The SSH session is allowed, however when I try to connect to my local 2525 TCP port, I note a “prohibited” error.  Therefore there is still room for exploitation possibly via DoS.  As far as I know however it is not possible to spam using this method 😉

Another way appears to be by not have the user shell listed in /etc/shells (/bin/false is not in every distro I checked).  Adding users with an invalid shell will invalidate this exact issue since sshd, via PAM, checks if the shell exists.  However, this will break any other system daemon that queries PAM for this very information, for example vsftpd/proftpd.  Therefore this “solution” is just a broken way of limiting ssh, however could have other, nasty consequences.

The other, smarter ways, would be to use AllowUser/AllowGroup directives in the sshd configuration, to specify exactly which users can and cannot connect.  And in all cases, it would be ideal to deny the ability to connect via SSH in iptables/ip6tables since OpenSSH has in the past (and likely will again) had security vulnerabilities that could be exploited to gain root access an pnwn the server :-\  As well, having your ssh socket open to the Internet attracts people to bruteforce scan your server, which is ugly in itself and causes a massive amout of syslog spam.



Tracepath6 – MTU and IPv6

IPv6, fragmentation and MTU

Unlike IPv4 where routers can (and do) fragment packets when a larger MTU packet must be forwarded over a path that does not support it.  Assuming the packet doesn’t have the df-bit set, the IPv4 router will fragment a packet and forward two or more packets downstream for reassembly by the end host.

In the IPv6 world, routers or intermediate nodes do not fragment packets.  A packet received by a router that is too large to forward will cause the router to drop the packet.  This practice was likely designed into IPv6 for certain security vulnerabilities, where sending fragments could hide protocol level header information and cause reassembly surprises, DoS  a host or router when trying to process these fragments or reassemble them, and a slew of various issues that made fragmentation become a IPv4 four-letter word.

In IPv6, although the packet is dropped, the router or node that drops the packet sends a ICMPv6 message “packet too big” back to the source host (Minimum IPv6 MTU has been defined as 1280).  The source host then must perform the fragmentation and re-transmit the ensuing flow at a smaller MTU.  The end node performs packet reassembly.    This behaviour clearly indicates that arbitrarily dropping ICMP, which was a common practice by clueless IPv4 system administrators of yesteryear, is now a deprecated practice 😉

Ensuring ICMPv6 messaging delivery is particularly important in today’s reality, as a significant proportion of IPv6-capable networks are utilizing a tunneling mechanism of some sort, be it Hurricane Electric’s excellent tunnel broker service, 6RD, or static IPv6-over-IPv4 tunnels; a 1500 MTU cannot be assumed.  The use of path mtu discovery (PMTUD), is critical.


The use of tracepath6, becomes particularily interesting – my distro (RedHat et al) seems to include it in the iputils package:

Here is an example of non-tunnelled, native dual-stack tracepath from a node to www.ripe.net:


tracepath6 www.ripe.net -n
 1?: [LOCALHOST]                      pmtu 1500
 1:  2001:db1::3                       1.785ms
 2:  2001:db1::2                     asymm  1   1.944ms
 3:  2001:db1:2:16::9                 asymm  2   2.937ms
 4:  ::ffff:               asymm 11  16.193ms
 5:  ::ffff:              asymm 11  16. 60ms
 6:  ::ffff:               asymm 11  16. 69ms
 7:  ::ffff:             asymm 11  16.279ms
 8:  2001:550:3::8a                   asymm  9  15.860ms
 9:  2001:5a0:600:500::a              asymm  8  16.746ms
10:  2001:5a0:f00:400::1               19.212ms
11:  2001:5a0:2000:400::19            asymm  9  90.538ms
12:  2001:5a0:2000:500::1             asymm  8  86.440ms
13:  2001:5a0:2000:500::a             asymm  9 103.741ms
14:  2001:5a0:200:100::5              asymm 11 103.324ms
15:  2001:5a0:200:200::16             asymm  9  97.851ms
16:  no reply
17:  2001:67c:2e8:27::1               asymm 13  98.410ms !A
     Resume: pmtu 1500


Apologies for the ::ffff:154.0.0.xx IPv4 mapped addresses, AS174 (Cogent…) seems to return them in traceroutes…Yuck!  And the asymm warning, which normally can indicate an asymmetrical path, is apparently caused by my local network’s use of HSRP.  It can also appear in certain tracepaths when an intermediate router is a Juniper, which I am told reduces the TTL by one before transmitting ICMP, and tracepath(6) uses TTL to determine path symmetry.  I don’t have a Juniper to test this <:-)

Anyway, here is an example of a tracepath through a IPv6-over-IPv4 tunnel:

tracepath6 -n www.ripe.net
 1?: [LOCALHOST]                      pmtu 1500
 1:  2001:db1:ffff:fffd::1             5.301ms
 1:  2001:db1:ffff:fffd::1             5.135ms
 2:  2001:db1:ffff:fffd::1             5.144ms pmtu 1480
 2:  2001:db1:ffff:ffff::1             7.608ms
 2:  2001:db1:ffff:ffff::1             8.031ms
 3:  2001:db1:2:16::9                   8.369ms
 4:  ::ffff:                20.434ms asymm 13
 5:  ::ffff:              20.127ms asymm 13
 6:  ::ffff:                23.160ms asymm 13
 7:  ::ffff:               23.119ms asymm 13
 8:  2001:550:3::8a                    29.943ms asymm 11
 9:  2001:5a0:600:500::a               25.208ms asymm 10
10:  2001:5a0:f00:400::1               25.141ms asymm 12
11:  2001:5a0:2000:400::19             95.158ms
12:  2001:5a0:2000:500::1              95.819ms asymm 10
13:  2001:5a0:2000:500::a             109.372ms asymm 11
14:  2001:5a0:200:100::5              111.653ms asymm 12
15:  2001:5a0:200:200::16             107.308ms asymm 11
16:  2001:7f8:1::a500:3333:2          181.523ms asymm 15
17:  2001:67c:2e8:27::1               105.202ms !A
     Resume: pmtu 1480


tracepath6 is a useful tool, especially in today’s Internet where IPv6 connectivity is not necessarily the usual 1500-byte MTU standard of the past and many hosts use tunneling transition mechanisms to access IPv6 content.  It will be useful to identify broken ICMPv6 paths, which for v6 traffic is essential for reliable communication.   Until dual-stack or even IPv6-only nodes are the norm, PMTUD and ICMPv6 will be critically important.  Here is hoping sysadmins will be wise enough to follow best practices and allow ICMPv6 messages, especially “packet too big” and “destination unreachable” messages.