CentOS policy routing – why yes, it can be done!

Over the years working in LAN networking there are several situations that dictate a host/server have multiple IP addresses on the same or different, physical or logical, devices.  For instance, connecting to a private management-only network/vlan, offering connectivity to a inside network on a private NIC, etc etc.

 

This scenario often causes two somewhat annoying behaviours:

 

1) the return traffic often is sourced from the “primary” IP address of the host/server, most often the one that is on the subnet associated with the default gateway

2) a surprising number of alleged “network administrators” seem to think having multiple gateways (one for each IP address of course )  is a good idea.  Well, over the years I have come across this situation and, in every case, this has obviously  NEVER WORKED.  

 

Situation #2 can only be fixed by not mind numbingly entering multiple gateways without restraint.  As for situation #1, RedHat/CentOS and derivatives support via iproute2 the ability to make traffic rules, ensuring that IP traffic is sourced by a particular IP in cases you can define.  Great!  Multiple NIC or logical interface routing on Linux is possible! (and yes, it involves having multiple gateways, but not stupidly and blindly adding them in the routing table….)

It is very simple to implement and involves the steps below.  As an example, lets assume we created a management VLAN (VLAN4) and want to add an logical interface on a server in that VLAN to access it internally.  We will be using 10.0.10.0/28 as an inside network.

 

Step 1: Create a VLAN interface

This creates the necessary interface on VLAN4 from primary physical interface eth0:

vi /etc/sysconfig/network-scripts/ifcfg-eth0.4

DEVICE=eth0.4
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet
IPV6INIT=no
IPADDR=10.0.10.2
NETMASK=255.255.255.240
NETWORK=10.0.10.0
VLAN=yes

Step 2: Create a iproute2 table for that management network

Edit /etc/iproute2/rt_tables to add a new entry and give it a arbitrary (unused 😉 ) name:

vi /etc/iproute2/rt_tables

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
200     MGMT

Note that between 200 and MGMT is a tab character.

 

Step 3: Create a default route for that network

vi /etc/sysconfig/network-scripts/route-eth0.4

default table MGMT via 10.0.10.1

this creates a default route for the MGMT/10.0.10.0/28 network to 10.0.10.1 which is your inside routing intelligence.

Step 4: Create routing rule for 10.0.10.2

To ensure that traffic received on 10.0.10.2 is utilizing the MGMT network only as a source address, a rule must be defined to enable this:

vi /etc/sysconfig/network-scripts/rule-eth0.4

from 10.0.10.2 table MGMT

 

and thats it!  restart your network:

/etc/rc.d/init.d/network restart

 

Using iproute2 commands, we can check that what we did works (as well as using wireshark 😉 )

 

[root@server network-scripts]# ip rule show
0:      from all lookup local
32765:  from 10.0.10.2 lookup MGMT
32766:  from all lookup main
32767:  from all lookup default
 
[root@server network-scripts]# ip route show
10.0.10.0/28 dev eth0.4  proto kernel  scope link  src 10.0.10.2
66.1.1.0/25 dev eth0  proto kernel  scope link  src 66.1.1.12
169.254.0.0/16 dev eth0  scope link  metric 1002
169.254.0.0/16 dev eth0.4  scope link  metric 1006
default via 66.1.1.125 dev eth0

 

Note: this also would work with a second physical interface, for instance to utilize a second NIC card instead of a VLAN logical interface, substitute all use of eth0.4 for eth1.

 

23 thoughts on “CentOS policy routing – why yes, it can be done!

  1. Thank you for posting. It kind of helps but I am still getting my ass kicked!

    I have a Centos 6.4 machine with two NICs.

    eth0 = 192.168.100.0/24 (IP=.10 / GW=.1)

    eth1 = 10.100.0.0/24 (IP=.252 / GW=.254)

    I would like to run the dhcp server on the 10. network and everything else on the 192 network.

  2. Pingback: Configurar VLANs y múltiples alias de red en Red Hat 6 | Hello, IT.

  3. Awesome post! It helped me configure my CentOS 7 LVS machines running keepalived, that are each on 4 different vlans.

    Rob.

  4. After working with this for a bit, we discovered that, if one wishes to communicate with devices on the 10.0.10.x network, in addition to routing packets to the Internet at large, it’s necessary to add a specific route for the local net, as well. For example, while you’d have:

    default via 10.0.10.1 table MGMT

    which handily routes packets to and from Internet sources, it appears that comms with other 10.0.10.x hosts requires the following, as well:

    10.0.10.0/24 via 10.0.10.2 table MGMT

    Until we added a similar line (using our own net numbers, of course), we were unable to communicate on the local network at all. Just wanted to add that in the event someone else was experiencing similar problems.

    Cheers!

    • thats strange I don’t have to do that on either Centos5 or Centos6 (?)

      are you sure about the format of your route-ethX.XXX file since I believe it is

      [target] table [tablename] via [gateway]
      

      I will have to re-test with your format, however I never had to do that. And that being said, I don’t have to make a special route for the local network, I only make routes for remote networks I wish to reach via that interface. What version/distro are you using?

      • Not sure why we had to do that. We’re running CentOS 6.5, and the route to the local net appeared in the standard routing table. However, except for the default gateway (using your example, 10.0.10.1, we could not exchange any packets on the local PB-routed network. Adding the additional route statement caused it all to work.

  5. Ive ventured here because of tptb’s terrible choice in not adopting netfilters route target. Let’s say u have the following

    eth0 – 46.29.40.215 – pub and def gateway
    eth0:1 – 10.0.0.0/24 – router @ 254, 3 vsphere stretch farms, and dozens of fwd facing 1918 nets
    eth0:10 – 46.29.255.211/27 – rmon probe
    as0t0 – 5.5.0.1/28 – eng remote access vpn
    eth1 – 10.1.1.254/24 – router @ 1, 47 vsphere farms n hundreds of rwd facing 1918

    Everything’s great and been working like a charm…in fact, so well that magicaly one day 2 new asa5585-ssp60-powerfire show up. U stage one on pub vlan10 .216 and u want to sR 5.5.0.3 (ur ovpnas address) thru 46.29.255.216?

    This wouldn’t have been an issue for me 2 weeks ago with nfilter, but I caved and moved this system from my nux dist to centos…and ipr2 has proven difficult for me.

  6. Hi I have an internal network (192.168.0.0 range). and a external net (10.0.0.0 range).

    I have to internet routers 10.0.0.2 and 10.0.0.3 and I’m looking for a way to set up the system to fail over form 10.0.0.2 to 10.0.0.3 when 10.0.0.2 is down.

    fedora core and centos 7
    Regards

    • well that type of failover is a bit beyond this policy routing as it is using static intelligence, not a routing protocol.

      What routers are you using? If they have some sort of link layer redundancy like VRRP or HSRP you might want to look at that from the router perspective, and have only one gateway IP (ex: 10.0.0.254) and have the redundancy protocol deal with providing failover.

      If LLR is not available, another solution is to run OSPF or some other routing protocol between your CentOS/Fedora hosts and have them learn a default route from the 10.0.0.2 and 10.0.0.3 provided you can have a higher metric on one of the two routes, when that route fails, the backup route would be utilized.

  7. My Machine(Cent OS) IP Address details below,

    eth0 : WAN
    IP : 182.74.31.138
    Netmask : 255.255.255.252
    GW : 182.74.31.137
    Network Address : 182.74.31.136

    eth1 : LAN
    IP : 192.168.25.2
    Netmask : 255.255.255.0
    GW : 192.168.25.1
    Network Address : 192.168.25.0

    I have done IP Forwarding and NAT to access internet from WAN to LAN and its working fine.

    I have configured 3 VLAN in Swutch.

    1) 192.168.25.0
    2) 192.168.24.0
    3) 192.168.20.0

    Now am able access internet from 192.168.25.0 but i am not able access internet from other 2 network( 192.168.24.0 & 192.168.20.0)

    What is the configuration i need to do in Cent OS machine to access internet on other 2 network ?

    Please help me on this…..

    • you need to make sure your IP MASQUERADE rules include other 192.168.0.0/16 networks, so you need to either MASQ for 192.168.24.0/24, 192.168.20,0/24 and 192.168.40.0/24, or as a whole 192.168.0.0/16. What is your IPtables command to MASQUERADE?

  8. Hi Chris,

    I have a problem that I cannot seem to find an answer to with relation to policy routing.

    The server is running Fedora22 (had same issue with Fedora 21), with two interfaces, making a LACP team (using teamd), named team0.
    On this team0 there are six VLAN’s, team0.30, team0.31, etc, etc.
    I have configured the rt_tables according to your instructions, and created the appropriate rules.
    The FIB_TRIE file (/proc/net/fib_trie) when viewed shows the routes as defined in the config files (ifcfg-vlan30, route-vlan30, rule-vlan30, etc, etc), but these routes appear to be ignored, with the output of “iproute” giving the routes as shown the “MAIN” routing table of FIB_TRIE.

    Any idea of where to look or how to identify where the problem is?

    All settings are at default for the Fedora distro.

    Cheers,
    Garth

      • Hi Chris,

        Posted below is the output of ip -4 route”, as well as the /proc/net/fib_trie.

        A brief description of the network connection:
        Two NIC’s, creating Team0 as the base (default route in ip route output is on team0 base), with six VLAN’s defined on Team0 (team0.30, .31, .32, .36, .37 & .38).

        If any of the VLAN’s has the default route assigned to it, with the address at the routing logic (ZyXel 110 Firewall) given as the “next-hop”, they all work fine – from that I assume that there is no issue with the configuration of the VLAN’s or routing logic.

        The problem appears only when the default route is given in the custom routing table, i.e. in “route-team0.30”, showing “default 192.168.xxx.65 table SERVINT via 192.168.xxx.66 dev team0.30” on the first line.

        it appears that for some reason the custom routing tables (200-205 inclusive) are seen and read into the /proc/net/fib_trie file, but are ignored in favour of routing table “main”.

        If you want the full set of config files, please send me an e-mail at my registration mail address.

        OUTPUT of “ip -4 route”

        default via 192.168.xxx.89 dev team0
        192.168.xxx.64/29 dev team0.30 proto kernel scope link src 192.168.xxx.66
        192.168.xxx.72/29 dev team0.31 proto kernel scope link src 192.168.xxx.74
        192.168.xxx.80/29 dev team0.32 proto kernel scope link src 192.168.xxx.82
        192.168.xxx.88/29 dev team0 proto kernel scope link src 192.168.xxx.90
        192.168.xxx.96/29 dev team0.36 proto kernel scope link src 192.168.xxx.98
        192.168.xxx.104/29 dev team0.37 proto kernel scope link src 192.168.xxx.106
        192.168.xxx.112/28 dev team0.38 proto kernel scope link src 192.168.xxx.114

        FIB_TRIE file

        Id 200:
        +– 0.0.0.0/0 2 0 2
        |– 0.0.0.0
        /0 universe UNICAST
        |– 192.168.xxx.112
        /28 universe UNICAST
        Id 201:
        +– 0.0.0.0/0 2 0 2
        |– 0.0.0.0
        /0 universe UNICAST
        |– 192.168.xxx.104
        /29 universe UNICAST
        Id 202:
        +– 0.0.0.0/0 2 0 2
        |– 0.0.0.0
        /0 universe UNICAST
        |– 192.168.xxx.96
        /29 universe UNICAST
        Id 203:
        +– 0.0.0.0/0 2 0 2
        |– 0.0.0.0
        /0 universe UNICAST
        |– 192.168.xxx.80
        /29 universe UNICAST
        Id 204:
        +– 0.0.0.0/0 2 0 2
        |– 0.0.0.0
        /0 universe UNICAST
        |– 192.168.xxx.72
        /29 universe UNICAST
        Id 205:
        +– 0.0.0.0/0 2 0 2
        |– 0.0.0.0
        /0 universe UNICAST
        |– 192.168.xxx.64
        /29 universe UNICAST
        Main:
        +– 0.0.0.0/0 2 0 2
        |– 0.0.0.0
        /0 universe UNICAST
        +– 192.168.xxx.64/26 3 0 1
        |– 192.168.xxx.64
        /29 link UNICAST
        |– 192.168.xxx.72
        /29 link UNICAST
        |– 192.168.xxx.80
        /29 link UNICAST
        |– 192.168.xxx.88
        /29 link UNICAST
        |– 192.168.xxx.96
        /29 link UNICAST
        |– 192.168.xxx.104
        /29 link UNICAST
        |– 192.168.xxx.112
        /28 link UNICAST
        Local:
        +– 0.0.0.0/0 2 0 2
        +– 127.0.0.0/8 2 0 2
        +– 127.0.0.0/31 1 0 0
        |– 127.0.0.0
        /32 link BROADCAST
        /8 host LOCAL

        (Snipped for brevity)

  9. Hello Chris,

    In desperation (since no changes I have done seems to make things work correctly), I have totally wiped the HDD, installed a newly downloaded version of Fedora 22 (Kernel 4.0.4) .

    After the initial install I disabled firewalld (am using an upstream hardware firewall) and also NetworkManager, and ran “modprobe –first-time 8021q” (bare-bones install does NOT have 8021q loaded, contrary to Fedora docs). I confirmed that the Kernel is set for policy routing (in the Kernel config), by checking that: CONFIG_IP_MULTIPLE_TABLES=y

    I set up a single /29 subnet on ifcfg-enp3s0, with only one IP address, and configured a single VLAN (ifcfg-enp3s0.30), configured as you have done above. The route and rule entries were created, and rt_tables file was modified with the new entry appended (ensuring a “TAB” between Nº and name) to the end.

    I had NO success with ping to an IP address in the router logic (not the VLAN Gateway, another centrally accessable IP address, but outside of VLAN sub-net) through the VLAN interface (works great when setting it as the DEFROUTE in ifcfg file). The reply is:
    Connect: Network is unreachable

    I then modified the /usr/lib/sysctl.d/50-default.conf file to set the following parameters:
    net.ipv4.ip_forward = 1,
    net.ipv4.conf.default.rp_filter = 0,
    net.ipv4.conf.all.rp_filter = 0
    net.ipv4.conf.default.accept_source_route = 1
    net.ipv4.conf.all.accept_source_route = 1
    (all settings are confirmed after reboot in /proc/sys/net/ipv4/……., all are as set in “50-default.conf”)

    Still no success with these changes, and on a simple network 🙁

    Any clues or ideas if there are other parameters which should be changed or set?

    After 6 months of hair-tearing, I have nearly exhausted all options.

    Cheers,
    Garth.

  10. Hi Chris,

    An update on the problem I raised, all fixed now.

    Problem stemmed from the lack of some of the packages required for policy routing, particularly for fib_rules.c

    It requires (on Fedora 22) to have kernel-headers, kernel-devel and libnl3-devel packages added in to get get the source routing to work.

    Cheers,
    Garth.

    • Hi Garth!

      Glad it finally works, sorry I was away for a bit and didn’t get to you sooner. Happy to hear and thanks for adding the info about FC22. How did you figure out these packages were required?

      • Hi Chris,

        I ended up dredging through the book on Linux Kernel Networking by Rami Rosen, there he mentions some of the more important packages required for fib_rules.c to work.

        Once I tracked down which Fedora packages they were in, and loaded them, all works fine.

        Also there was no requirement for other routings to be entered for local networks are mentioned by one of the earlier posters.
        I could not ping my Windows 7 desktop, but could ping the F22 server from it – that was Bitdefender firewall in “stealth” mode, despite setting the general firwall rules to Incoming ICMP ” ALLOWED”….. (roll:eyes)

  11. > to utilize a second NIC card instead of a VLAN logical interface, substitute all use of eth0.4 for eth1. <

    Thanks for the article. First time having to use routes. Server has two NIC's. One is on a private network, the other public. The private NIC (eth0) is working with a standard config in ifcfg-eth0 (GATEWAY=x.x.x.x is included here).

    When adding a second route using your example above, should I leave ifcfg-eth0 alone, while configuring the routing table as above?

    • I am trying to understand your question. if you are trying to segregate routing for a public and a private network, you could create a route-eth1 and rule-eth1 file and run a second default route (be careful…)

      something like

      in /etc/iproute2/rt_tables

      200 PRIVATE

      in /etc/sysconfig/network-scripts/route-eth1

      default table PRIVATE via xxx.xxx.xxx.xxx (whatever your private network GW is)

      in /etc/sysconfig/network-scripts/rule-eth1

      from xxx.xxx.xxx.xxx table PRIVATE (where xxx.xxx.xxx.xxx is the IP address of your eth1 private network IP)

      ???

Leave a Reply

Your email address will not be published. Required fields are marked *

r u a bot? * Time limit is exhausted. Please reload CAPTCHA.