Damn you, ICMP redirect!!! (or rather, how to flush a cached ICMP redirect under CentOS7/Linux)

Been a while since I added anything new here.  Been busy trying to keep my head above water I guess.  In any case, I came across a situation during $dayjob$ where I had to seperate two networks that were sharing the same VLAN to two distinct VLANS since they were actually always supposed to be seperate, and are geographically distinct as well.  The router configuration was as follows:

!
interface Vlan6
 ip address 10.0.11.1 255.255.255.0 secondary
 ip address 10.0.1.250 255.255.255.0
 no ip redirects
 no ip proxy-arp
end
!

Networks 10.0.1.0/24 and 10.0.11.0/24 were essentially on a common VLAN.

Anyway, the task seemed simple, leave 10.0.11.1/24 on VLAN6 and create VLAN5 on this router to add a routed interface between 10.0.11.0/24 and 10.0.1.0/24 which was being moved to another device.

!
 interface Vlan5
 ip address 10.0.249.6 255.255.255.248
 end
 !

Now, traffic from 10.0.1.0/24 would have to transit via 10.0.249.0/29, being routed by two different routers.   Hosts in 10.0.1.0/24 would have to transit via 10.0.249.1(which is another interface on the new 10.0.1.1/24 router) and 10.0.249.6 to get to 10.0.11.0/24.  Simple, everyday stuff.  Right?

 

Well for a reason that at first escaped me, the one and only host in 10.0.11.0/24 that had communications with 10.0.1.0/24 could ping every other host in the network *except* the two I needed it to communicate with, and until I added the routed interfaces, was working perfectly.  I was confounded until I tried a traceroute from one of the hosts in 10.0.1.0/24 back to 10.0.11.220:

[root@host1 ~]# traceroute 10.0.11.220
traceroute to 10.0.11.220 (10.0.11.220), 30 hops max, 60 byte packets
 1  (10.0.1.210)  849.491 ms !H  849.454 ms !H  849.426 ms !H

Now why would I get a HOST_UNREACHABLE?!?!?  From MYSELF!!!!(10.0.1.210 is host1) Here is my routing table

[root@host1 ~]#  ip -4 route
default via 10.0.1.251 dev eth0
10.0.1.0/24 dev eth0  proto kernel  scope link  src 10.0.1.210

Seems normal (?)

Other traceroutes towards 10.0.11.0/24 hosts were working:

[root@host1 ~]# traceroute 10.0.11.100
 traceroute to 10.0.11.100 (10.0.11.100), 30 hops max, 60 byte packets
 1  10.0.1.251 (10.0.1.251)  0.225 ms  0.192 ms  0.180 ms
 2  10.0.249.6 (10.0.249.6)  5.591 ms
 3  10.0.11.100 (10.0.11.100) 6.524 ms

My gateway for 10.0.1.0/24 is 10.0.1.251, and the above makes sense.  It is only then I realized the host had cached an ICMP_REDIRECT for 10.0.11.220, and I checked this with the ip route command:

[root@host1 ~]# ip route get 10.0.11.220
10.0.11.220 via 10.0.1.250 dev eth0  src 10.0.1.210
    cache <redirected>

bingo.  Cached, ICMP_REDIRECT from 10.0.1.250, which no longer exists.  I didn’t bother to check how long the timeout is for these cached entries, however it was longer than I would have expected, especially since I troubleshot this for more than 15 minutes (*cough* *cough*).

In any case, I learned a new command to zap these, and thankfully, stuff started working again:

root@host1 ~]# ip route flush cache

And with that, my troubles were gone

[root@host1 ~]# ip route get 10.0.11.220
10.0.11.220 via 10.0.1.251 dev eth0  src 10.0.1.210
    cache

from my reading of various Googles, the gc_timeout is what defines the actual timeout:

[root@host1~ proc]# cat ./sys/net/ipv4/route/gc_timeout
300

I can safely say my troubleshooting went way past this timeout, assuming it is seconds and not minutes, and therefore there is some wonkyness beyond I need to check.  In any case, knowing how to clear the cache is evidently useful as well!

IPv6 router solicitation, Link-Local gateway, Ubuntu and a bit of WTF…

I have been using RHEL/CentOS lineage Linuces for a good while  now, and I believe the strength and distro mindset is their dedication to package dependencies.  Some packages or software that require bleeding edge or even not so recently updated/released libraries/CPAN perl modules etc don’t behave well or won’t install/compile without breaking those dependencies without installing source packages.  This paradoxally is the weakness of RHEL/CentOS, their dedication to dependencies runaway  They occasionally are a bit behind when it comes to the latest and greatest, and you cat get handcuffed if you want to remain within the confines of the dependancy sandbox.

 

In any case I wanted to toy with some software that clearly seems written for Ubuntu (first tell-tale sign is that you need to add a bunch of user-contributed repositories so you can install it….)  This software requires IPv6 connectivity so I decided to trash a BIND virtual server I was using and replace it with Ubuntu 14.04 from the .ISO.  I am not totally foreign to Ubuntu as I use Ubuntu desktop on a couple laptops, however in a server environment I have never bothered to consider it out of laziness, mostly.  Here are a few observations, some of which I am still trying to wrap my head around.

 

BIND startup

My first challenge was getting BIND to start correctly.  I usually use “listen-on-v6” to specify what interfaces I allow BIND to bind to, and was seeing this incredibly annoying message on bootup and BIND was not binding to any IPv6 addresses, defeating the purpose of running BIND on this server in the first place:


yet 2001:db8:9::37/64 is statically configured in my interface config (!!):


 

apparently, Ubuntu is in such a hurry to boot that it starts daemons before the interface init is even finished (!)  Major-wtf.  So it would seem that it is prudent (and necessary?) to put the IPv6 definition of eth0 inet6 *before* IPv4.  Guess that is good to know doh  Once thats done, BIND starts correctly and we can move on.


IPv6 default gateway gong-show

My next hurdle was not far away.  Even if I manually set the gateway, for some reason Ubuntu seems to feel it necessary to send a router solicitation… My network has regular router advertisements disabled, just because I don’t want SLAAC to work in that particular test environment.  So even if I statically configure Ubuntu with a prefix, mask and gateway, it still seems to feel a need to go exploring for routers and sends out an ICMPv6 router solicitation…so when I check the routing table:


now wait a minute!  My network does not send periodic RAs, I have statically defined my gateway, and Ubuntu overrides whatever I defined with auto-learned crap??  surprised  After 500 seconds, the RS-learned gateways disappear and whatever I defined as default then remains, and sometimes no gateway at all…..

Link-Local Gateway

Remember I mentioned above the bit about putting the IPv6 config before IPv4?  My network uses HSRP as first-hop redundancy, and IOS didn’t allow for global addressing (and some supported IOS trains do not have this feature) until “recently”.  In any case, I want to use a link-local address as a gateway.  For some reason, if the IPv6 interface in Ubuntu is defined *after* the IPv4 address, the LL gateway sometimes is ignored (what?runaway)  So not only do I get possibly bogus gateways via ICMPv6 router solicitation, I might end up with no gateway at all once those RAs lifetime is up.  not good.

The solution is to disable router-advertisement learning in /etc/sysctl.conf by adding the following:


and contrary to RHEL/CentOS (and pretty much any environment where a link-local is used to route something that I have seen), it is not necessary to specify the interface – I suppose Ubuntu is doing some logic there:

 


 


So we note the absence of %device.  I would have to test if with multiple NICs this still works.  Hopefully it does pc

 

Network restart broken

 

My last adventure was trying to restart networking in some controlled way after making changes.  I believed I had broken my install when I got the following as a result:


After some googling, apparently this is known.   So shucks, another thing that will hopefully get fixed wassat   At the rate Ubuntu seems to update packages, I suppose it might not be a long wait pc

 

 

 

CentOS policy routing – why yes, it can be done!

Over the years working in LAN networking there are several situations that dictate a host/server have multiple IP addresses on the same or different, physical or logical, devices.  For instance, connecting to a private management-only network/vlan, offering connectivity to a inside network on a private NIC, etc etc.

 

This scenario often causes two somewhat annoying behaviours:

 

1) the return traffic often is sourced from the “primary” IP address of the host/server, most often the one that is on the subnet associated with the default gateway

2) a surprising number of alleged “network administrators” seem to think having multiple gateways (one for each IP address of course )  is a good idea.  Well, over the years I have come across this situation and, in every case, this has obviously  NEVER WORKED.  

 

Situation #2 can only be fixed by not mind numbingly entering multiple gateways without restraint.  As for situation #1, RedHat/CentOS and derivatives support via iproute2 the ability to make traffic rules, ensuring that IP traffic is sourced by a particular IP in cases you can define.  Great!  Multiple NIC or logical interface routing on Linux is possible! (and yes, it involves having multiple gateways, but not stupidly and blindly adding them in the routing table….)

It is very simple to implement and involves the steps below.  As an example, lets assume we created a management VLAN (VLAN4) and want to add an logical interface on a server in that VLAN to access it internally.  We will be using 10.0.10.0/28 as an inside network.

 

Step 1: Create a VLAN interface

This creates the necessary interface on VLAN4 from primary physical interface eth0:

vi /etc/sysconfig/network-scripts/ifcfg-eth0.4

DEVICE=eth0.4
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet
IPV6INIT=no
IPADDR=10.0.10.2
NETMASK=255.255.255.240
NETWORK=10.0.10.0
VLAN=yes

Step 2: Create a iproute2 table for that management network

Edit /etc/iproute2/rt_tables to add a new entry and give it a arbitrary (unused 😉 ) name:

vi /etc/iproute2/rt_tables

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
200     MGMT

Note that between 200 and MGMT is a tab character.

 

Step 3: Create a default route for that network

vi /etc/sysconfig/network-scripts/route-eth0.4

default table MGMT via 10.0.10.1

this creates a default route for the MGMT/10.0.10.0/28 network to 10.0.10.1 which is your inside routing intelligence.

Step 4: Create routing rule for 10.0.10.2

To ensure that traffic received on 10.0.10.2 is utilizing the MGMT network only as a source address, a rule must be defined to enable this:

vi /etc/sysconfig/network-scripts/rule-eth0.4

from 10.0.10.2 table MGMT

 

and thats it!  restart your network:

/etc/rc.d/init.d/network restart

 

Using iproute2 commands, we can check that what we did works (as well as using wireshark 😉 )

 

[root@server network-scripts]# ip rule show
0:      from all lookup local
32765:  from 10.0.10.2 lookup MGMT
32766:  from all lookup main
32767:  from all lookup default
 
[root@server network-scripts]# ip route show
10.0.10.0/28 dev eth0.4  proto kernel  scope link  src 10.0.10.2
66.1.1.0/25 dev eth0  proto kernel  scope link  src 66.1.1.12
169.254.0.0/16 dev eth0  scope link  metric 1002
169.254.0.0/16 dev eth0.4  scope link  metric 1006
default via 66.1.1.125 dev eth0

 

Note: this also would work with a second physical interface, for instance to utilize a second NIC card instead of a VLAN logical interface, substitute all use of eth0.4 for eth1.

 

BIND DNS64 and views

I am starting to experiment with IPv6-only networks and require the use of a DNS64 service to be used in conjunction with NAT64.  Luckily, BIND 9.8.x offers this capability.  However, I don’t want my DNS server to reply with a quad-A record for every query now that I have enabled DNS64, since this DNS server is used by non-v6 clients, as well as dual-stack clients that I do not want to send them searching for 64:ff9b::/96. :S  I therefore added a secondary IPv6 address on the DNS server for my v6-only clients to use specifically for DNS64, and put it in its own view.  In the config below, the BIND DNS64 service will utilize 2001:db8::64 🙂

 

acl rfc1918 { 10/8; 192.168/16; 172.16/12; };
options {
        listen-on port 53 {
                192.168.1.1;
                127.0.0.1;
        };
        listen-on-v6 port 53 {
                2001:db8::10/64;
                2001:db8::64/64;
                ::1;
        };
view "external" {
match-destinations { 2001:db8::10; ::1; 192.168.1.1; 127.0.0.1; };
[...]

}; //end view external
view "dns64" {
    match-destinations { 2001:db8::64; };
    dns64       64:ff9b::/96 {
        clients { any; };
        mapped { !rfc1918; any; };
        exclude { 64:ff9b::/96; ::ffff:0000:0000/96; };
        suffix ::;
}; //end view dns64
};

 

the corresponding digs show the difference between

dig @2001:db8::64 +short -t AAAA www.cnn.com
www.cnn.com.vgtf.net.
cnn-atl.gslb.vgtf.net.
64:ff9b::9da6:e21a
64:ff9b::9da6:e219
dig @2001:2b8::10 +short -t AAAA www.cnn.com
www.cnn.com.vgtf.net.

Looks like I am ready to start working on learning how to use TAYGA

CentOS 6.3 released – features….and bugs :)

CentOS logo

 

Well folks its finally here ; http://lists.centos.org/pipermail/centos-announce/2012-July/018706.html

Among the changes, imitating its parent RedHat6.3, OpenOffice is replaced by libreoffice, I have yet to try it out but apparently has a bit more community support and can read .docx files.   They are also deprecating matahari, a management API I know little about, in favor of another suite (CIM).  RedHat even recommends matahari removal, which is fine by me since I never bothered to learn it 🙂

Bind9.7 is also replaced by Bind9.8 (yea!), a rare move by RedHat in my memory since they dislike changing version numbers in the lifecycle of each distribution.

Another notable addition to CentOS6.3 is the tools to convert physical to virtual machines for use with KVM, virt-p2v and virt-v2v (for migrationg a virtual-to-virtual installation).  Unlike RedHat6.3 that ships an .ISO image seperately to boot from in order to use these tools, CentOS includes that .ISO in the .rpm.   I look forward to trying it out.

I have already upgraded a number of guests (my KVM host however is still running 6.2, I have not worked up the courage to give it a go yet, it should go well but I can’t be bothered with glitches right about now), and all seems to work smoothly; I did come across one issue however, if you utilize IPv6 resolvers, a new(?) bug in libresolv causes a segmentation fault crash in applications that make a particular call to it;  sendmail, freshclam/clamav, emacs, openvpn, chrony, postfix.  Here is the Redhat bugzilla entry, as well as the CentOS entry.

Even if only one of your resolvers is an IPv6 address, affected software will crash;

 

freshclam[6598]: segfault at 1 ip 00007f9be8b37596 sp 00007fff9ffac0b0
 error 6 in libresolv-2.12.so[7f9be8b2b000+16000]

sendmail[7374]: segfault at 1 ip 00007f95d7b73596 sp 00007fffa93295a0 
error 6 in libresolv-2.12.so[7f95d7b67000+16000]

 

I look forward to the fix.  In the meantime I will get to tinker with P2V/V2V tools.

 

 

 

 

 

Time Flies – BIND, CentOS4 and Spammers

Its been over a month since I have been able to take a moment and say hello to my blog.  After my own time off, my colleagues have dared to take their own time off and I am stuck with extra work while they are gone.  What goes around, comes around, I guess.

 

In the meantime, the only work I have been able to do around here is moderate spammy comments to the trashbin.  Viagra, Online Loans, Escort services, they all seem to come over here with their bots and spam my admin panel with bogus comments with links back to their crap…

 

 

One thing I guess they are trying to exploit are the WordPress pingbacks.  Much to my annoyance since I really don’t mind speaking out loud to nobody on this blog, its crappy to have bots polluting my space with their spam.   I might try a WordPress CAPTCHA plugin of some sort, was too lazy to look for one since I didn’t figure I needed one….little did I know!

 

In any case, as time goes on, people still (*shiver*) running RHEL4/CentOS4 systems are getting more and more vulnerabilities.  CVE-2012-1823 being the first reasonably big bug that will go unpatched with RHEL4/CentOS4, and now BIND has announced CVE-2012-1667, a rather nasty one that apparently can in some cases expose system memory (!!!).  RHEL4 and clones shiiped with BIND 9.2, and it is of course vulnerable.  Thankfully systems I manage have been moved past EL4, however I do have a server or two that is still yet to be replaced.  I came across a blog that had shown promise of a rebuild of BIND 9.7.3 (which ships with CentOS 6.2 I believe), yet I was never able to get the .src.rpm for that.  I ended up building my own.  Its based on Fedora Linux’s FC14 9.7.4 RPM, with the 9.7.6_P1 source from ISC.ORG.  You can download my .src.rpm here.  I have tested it happily for a few weeks now, your mileage may vary.  Rebuild at your own risk! 🙂  I have it running on a mail server for DNS caching as well as a reasonably busy authoritative server and so far, no issues at all.    At least now RHEL4/CentOS4 can have allow-recursion { acl; } in named.conf!!! (yay)

Anyway, if you do download the source and compile it, let me know your results.

 

PHP updated – CVE-2012-1823 / CVE-2012-2311

 

A bug in PHP 5 was released, somewhat accidentally (apparently somebody made a reddit post public inadvertently…why people would use these sites for sensitive info is beyond me…anyhoo), and is finally patched by RedHat and derivatives, CentOS, ScientificLinux –

https://rhn.redhat.com/errata/RHSA-2012-0546.html

Ubuntu has also released a patch –

http://www.ubuntu.com/usn/usn-1437-1/

RHEL5&6 would be no longer vulerable to either CVE-2012-1823 or its re-incarnation CVE-2012-2311, as the bug was not correctly patched the first time around.   RedHat claims their fix is complete, I cannot vouch for Ubuntu so don’t blame me if you have to patch it again later.

 

The vulnerability itself was quite old, sneaking into the code 8 years ago and lying undiscovered until recently.  It relies on the use of php-cgi (running PHP as a cgi forked process, not in the more mainstream mod_php mechanism).  One of the many consequences of this bug was source code exposure (via ?-s), and many PHP sites having database username/password information contained in the PHP code, this vulnerability could and will compromise sites where this is the case that remain unpatched.  This is one of the many reasons it is a better idea to use PHP include functionality to provide database/user/password/security connnection info to PHP, and have that included file outside of the http webroot in the first place.  Other possible exploits would be to execute code/upload files on the remote filesystem….scary, nasty stuff!!!!

Even the allmighty Facebook, set for an IPO this week, was vulnerable to this (they were running as a cgi, apparently!!)

The IPO could have been something of a bust had Facebook been hacked a week before it hopes to raise 95$ billion USD.  Who would have seen that one coming?!?!?!?!


 

/bin/false, /sbin/nologin and SSH

It is common knowledge that if on a linux system, a local account is set to /bin/false or /sbin/nologin, that user cannot use SSH, right?

 

Wrong.

 

As one of my clients learned the hard way.  Alerted to the fact his server was blacklisted on a (large!) number of RBLs, we started investigating how and why spam was getting sent from his server.  What we found was that a local user account (listed in /etc/passwd) had been compromised, due to an easy to guess password.

On all the recent Linux distributions I attempted this on, it is possible to connect to the SSH daemon and initiate not a shell, but a TCP port forwarding session. Using /sbin/nologin as a shell only causes the SSH daemon to spawn a shell that closes immediately. So if you tell SSH not to spawn a shell, and use TCP port forwardings instead, you can then exploit the remote host.

Here is how it works.

 

if I create a user with [/bin/false|/sbin/nologin] as a shell, and give him a password. I can then SSH to the machine and forward a local port.

Like this:

[user@panel~]ssh -N testacct@linuxserver.cconn.info -L 2525:127.0.0.1:25

this forwards my local port of 2525 to the remote port 25. Allowing me to spam via the remote server, via my local TCP 2525 socket I just created:

[user@panel ~] telnet localhost 2525
 Trying ::1...
 Connected to localhost.
 Escape character is '^]'.
 220 linuxserver.cconn.info ESMTP Sendmail 8.14.4/8.14.4; Sun, 6 May 2012 18:06:57 -0400

From “panel” I can connect to “linuxserver” over my ssh tunnel, and spam…my email appears to come from 127.0.0.1 when you look at the headers, causing a certain level of difficulty in tracing the spam back to that account after the fact;

Received: from foo (localhost [127.0.0.1])
 by linuxserver.cconn.info (8.14.4/8.14.4) with SMTP id q36MAbio019457
 for spambait@sucker.com; Sun, 6 May 2012 18:10:48 -0400

Why does this work?  Because sshd by default is configured, on my distros anyway, to use PAM for authentication, and has TCP port forwardings enabled by default.  If a shell listed in /etc/shells is configured for the account (/sbin/nologin is listed in /etc/shells), the password is valid and TCP port forwarding is enabled, this allows an account to establish said port forwardings and utilize services on the remote host.  Spam is an example, but there could be more.

How can this be fixed?

 

Well, there are a number of ways.  The simplest and most drastic, if possible in your scenario, is to simply disable TCP port forwarding in your sshd configuration file (/etc/ssh/sshd_config on many systems):

AllowAgentForwarding no
AllowTcpForwarding no
X11Forwarding no

However, note that this will not deny the actual ssh connection to be established; it will simply deny the ability to exploit the forwarding (see below);

ssh -N testacct@linuxserver.cconn.info -L 2525:127.0.0.1:25
testacct@tohs.cconn.info's password:
channel 2: open failed: administratively prohibited: open failed
channel 2: open failed: administratively prohibited: open failed

The SSH session is allowed, however when I try to connect to my local 2525 TCP port, I note a “prohibited” error.  Therefore there is still room for exploitation possibly via DoS.  As far as I know however it is not possible to spam using this method 😉

Another way appears to be by not have the user shell listed in /etc/shells (/bin/false is not in every distro I checked).  Adding users with an invalid shell will invalidate this exact issue since sshd, via PAM, checks if the shell exists.  However, this will break any other system daemon that queries PAM for this very information, for example vsftpd/proftpd.  Therefore this “solution” is just a broken way of limiting ssh, however could have other, nasty consequences.

The other, smarter ways, would be to use AllowUser/AllowGroup directives in the sshd configuration, to specify exactly which users can and cannot connect.  And in all cases, it would be ideal to deny the ability to connect via SSH in iptables/ip6tables since OpenSSH has in the past (and likely will again) had security vulnerabilities that could be exploited to gain root access an pnwn the server :-\  As well, having your ssh socket open to the Internet attracts people to bruteforce scan your server, which is ugly in itself and causes a massive amout of syslog spam.