Damn you, ICMP redirect!!! (or rather, how to flush a cached ICMP redirect under CentOS7/Linux)

Been a while since I added anything new here.  Been busy trying to keep my head above water I guess.  In any case, I came across a situation during $dayjob$ where I had to seperate two networks that were sharing the same VLAN to two distinct VLANS since they were actually always supposed to be seperate, and are geographically distinct as well.  The router configuration was as follows:

interface Vlan6
 ip address secondary
 ip address
 no ip redirects
 no ip proxy-arp

Networks and were essentially on a common VLAN.

Anyway, the task seemed simple, leave on VLAN6 and create VLAN5 on this router to add a routed interface between and which was being moved to another device.

 interface Vlan5
 ip address

Now, traffic from would have to transit via, being routed by two different routers.   Hosts in would have to transit via is another interface on the new router) and to get to  Simple, everyday stuff.  Right?


Well for a reason that at first escaped me, the one and only host in that had communications with could ping every other host in the network *except* the two I needed it to communicate with, and until I added the routed interfaces, was working perfectly.  I was confounded until I tried a traceroute from one of the hosts in back to

[root@host1 ~]# traceroute
traceroute to (, 30 hops max, 60 byte packets
 1  (  849.491 ms !H  849.454 ms !H  849.426 ms !H

Now why would I get a HOST_UNREACHABLE?!?!?  From MYSELF!!!!( is host1) Here is my routing table

[root@host1 ~]#  ip -4 route
default via dev eth0 dev eth0  proto kernel  scope link  src

Seems normal (?)

Other traceroutes towards hosts were working:

[root@host1 ~]# traceroute
 traceroute to (, 30 hops max, 60 byte packets
 1 (  0.225 ms  0.192 ms  0.180 ms
 2 (  5.591 ms
 3 ( 6.524 ms

My gateway for is, and the above makes sense.  It is only then I realized the host had cached an ICMP_REDIRECT for, and I checked this with the ip route command:

[root@host1 ~]# ip route get via dev eth0  src
    cache <redirected>

bingo.  Cached, ICMP_REDIRECT from, which no longer exists.  I didn’t bother to check how long the timeout is for these cached entries, however it was longer than I would have expected, especially since I troubleshot this for more than 15 minutes (*cough* *cough*).

In any case, I learned a new command to zap these, and thankfully, stuff started working again:

root@host1 ~]# ip route flush cache

And with that, my troubles were gone

[root@host1 ~]# ip route get via dev eth0  src

from my reading of various Googles, the gc_timeout is what defines the actual timeout:

[root@host1~ proc]# cat ./sys/net/ipv4/route/gc_timeout

I can safely say my troubleshooting went way past this timeout, assuming it is seconds and not minutes, and therefore there is some wonkyness beyond I need to check.  In any case, knowing how to clear the cache is evidently useful as well!

CentOS policy routing – why yes, it can be done!

Over the years working in LAN networking there are several situations that dictate a host/server have multiple IP addresses on the same or different, physical or logical, devices.  For instance, connecting to a private management-only network/vlan, offering connectivity to a inside network on a private NIC, etc etc.

This scenario often causes two somewhat annoying behaviours:

1) the return traffic often is sourced from the “primary” IP address of the host/server, most often the one that is on the subnet associated with the default gateway

2) a surprising number of alleged “network administrators” seem to think having multiple gateways (one for each IP address of course )  is a good idea.  Well, over the years I have come across this situation and, in every case, this has obviously  NEVER WORKED.

Situation #2 can only be fixed by not mind numbingly entering multiple gateways without restraint.  As for situation #1, RedHat/CentOS and derivatives support via iproute2 the ability to make traffic rules, ensuring that IP traffic is sourced by a particular IP in cases you can define.  Great!  Multiple NIC or logical interface routing on Linux is possible! (and yes, it involves having multiple gateways, but not stupidly and blindly adding them in the routing table….)

It is very simple to implement and involves the steps below.  As an example, lets assume we created a management VLAN (VLAN4) and want to add an logical interface on a server in that VLAN to access it internally.  We will be using as an inside network.

Step 1: Create a VLAN interface

This creates the necessary interface on VLAN4 from primary physical interface eth0:

vi /etc/sysconfig/network-scripts/ifcfg-eth0.4


Step 2: Create a iproute2 table for that management network

Edit /etc/iproute2/rt_tables to add a new entry and give it a arbitrary (unused 😉 ) name:

vi /etc/iproute2/rt_tables

# reserved values
255     local
254     main
253     default
0       unspec
# local
#1      inr.ruhep
200     MGMT

Note that between 200 and MGMT is a tab character.

Step 3: Create a default route for that network

vi /etc/sysconfig/network-scripts/route-eth0.4

default table MGMT via

this creates a default route for the MGMT/ network to which is your inside routing intelligence.

Step 4: Create routing rule for

To ensure that traffic received on is utilizing the MGMT network only as a source address, a rule must be defined to enable this:

vi /etc/sysconfig/network-scripts/rule-eth0.4

from table MGMT

and thats it!  restart your network:

/etc/rc.d/init.d/network restart

Using iproute2 commands, we can check that what we did works (as well as using wireshark 😉 )

[root@server network-scripts]# ip rule show
0:      from all lookup local
32765:  from lookup MGMT
32766:  from all lookup main
32767:  from all lookup default
[root@server network-scripts]# ip route show dev eth0.4  proto kernel  scope link  src dev eth0  proto kernel  scope link  src dev eth0  scope link  metric 1002 dev eth0.4  scope link  metric 1006
default via dev eth0

Note: this also would work with a second physical interface, for instance to utilize a second NIC card instead of a VLAN logical interface, substitute all use of eth0.4 for eth1.