RANCID and IOS 15.2 – blank config and how to work around newer file privileges

In or around IOS 15.2 apparently a privilege structural change was made that breaks non-priv-15 users from being able to copy the running config, useful for RANCID and other tools.  And either the RANCID/Oxidized community simply uses privilege 15 users in their configs, which I refuse to do on priciple, or my google-fu is poor because I have not found this info in any explicit form.

In any case, the symptom of the typical config allowing to download a running config without having level-15 privileges on Cisco IOS has always been documented as:

username rancidbackup privilege 10 secret [md5-pass]
privilege exec level 10 show running-config view full
privilege exec level 10 show running-config view
privilege exec level 10 show running-config
privilege exec all level 1 show

 

the above in IOS <= 15.1 was always enough to allow user “rancidbackup” to issue a “show running-config view full” and the CLI would output the current configuration.  Under IOS 15.2, behaviour appears to have changed.  Some folks receive a “permission denied” error while others, such as my experience, a simple empty config would be output.  As an example:

ssh -l rancidbackup 10.0.255.1

Password: 

Router#show running-config view full
[...nothing...]
Router#show running-config
[...nothing...]          
Router#

This annoying behaviour can be worked around in two different ways.

The first, is to cave and allow the rancid backup user to obtain level-15 privileges upon login.  The academic risk of compromise by this admin-level user can be mitigated by limiting the IP addresses used for login via ACL:  example:

username rancidbackup access-class 99 privilege 15 secret [md5-pass]
!
access-list 99 remark RANCID-ONLY
access-list 99 permit host 10.0.1.25 any
!

By adding access-class to the username definition, it is possible to avoid this user being used from other sources.  This might even be recommendable for any scenario where scripting/tools are used for access to the device.  However, this method still requires that user to have admin level-15 privileges.

 

I was pointed to https://supportforums.cisco.com/discussion/11691446/error-opening-nvramstartup-config-permission-denied as a possible second solution.  Apparently there is a mecanism to allow file access (which includes running-config, apparently…), with the command “file privilege [level]”

In our case, by issuing “file privilege 10”, we are able to see via “show running-config view full” (and not just show running-config) the info we seek to backup with our tool, in this case RANCID (as you can see, “show running config” is useless but “show running-config view full” outputs what we need):

 

Router#show running-config 

Building configuration...
Current configuration : 261 bytes
!
! Last configuration change at 22:04:36 EDT Mon May 30 2016 by user
! NVRAM config last updated at 13:15:46 EDT Thu May 26 2016 by user
! NVRAM config last updated at 13:15:46 EDT Thu May 26 2016 by user
boot-start-marker
boot-end-marker
!
!
!
!
!
!
!
end

Router#show running-config view full 
Building configuration...
Current configuration : 90474 bytes
!
! Last configuration change at 22:04:36 EDT Mon May 30 2016 by user
! NVRAM config last updated at 13:15:46 EDT Thu May 26 2016 by user
! NVRAM config last updated at 13:15:46 EDT Thu May 26 2016 by user
upgrade fpd auto
version 15.2
.... etc etc
!

huzzah!

I am still researching secondary implications from setting “file privilege” to anything other than level-15.  I haven’t found a way, using a privilege 10 user, to modify/delete the files so use at your own risk (!).  But combining the two above mecanisms I believe provides a sufficient means to allow non-privileged, read-only access to the configuration for use with config management and other scripting tools.

 

Damn you, ICMP redirect!!! (or rather, how to flush a cached ICMP redirect under CentOS7/Linux)

Been a while since I added anything new here.  Been busy trying to keep my head above water I guess.  In any case, I came across a situation during $dayjob$ where I had to seperate two networks that were sharing the same VLAN to two distinct VLANS since they were actually always supposed to be seperate, and are geographically distinct as well.  The router configuration was as follows:

!
interface Vlan6
 ip address 10.0.11.1 255.255.255.0 secondary
 ip address 10.0.1.250 255.255.255.0
 no ip redirects
 no ip proxy-arp
end
!

Networks 10.0.1.0/24 and 10.0.11.0/24 were essentially on a common VLAN.

Anyway, the task seemed simple, leave 10.0.11.1/24 on VLAN6 and create VLAN5 on this router to add a routed interface between 10.0.11.0/24 and 10.0.1.0/24 which was being moved to another device.

!
 interface Vlan5
 ip address 10.0.249.6 255.255.255.248
 end
 !

Now, traffic from 10.0.1.0/24 would have to transit via 10.0.249.0/29, being routed by two different routers.   Hosts in 10.0.1.0/24 would have to transit via 10.0.249.1(which is another interface on the new 10.0.1.1/24 router) and 10.0.249.6 to get to 10.0.11.0/24.  Simple, everyday stuff.  Right?

 

Well for a reason that at first escaped me, the one and only host in 10.0.11.0/24 that had communications with 10.0.1.0/24 could ping every other host in the network *except* the two I needed it to communicate with, and until I added the routed interfaces, was working perfectly.  I was confounded until I tried a traceroute from one of the hosts in 10.0.1.0/24 back to 10.0.11.220:

[root@host1 ~]# traceroute 10.0.11.220
traceroute to 10.0.11.220 (10.0.11.220), 30 hops max, 60 byte packets
 1  (10.0.1.210)  849.491 ms !H  849.454 ms !H  849.426 ms !H

Now why would I get a HOST_UNREACHABLE?!?!?  From MYSELF!!!!(10.0.1.210 is host1) Here is my routing table

[root@host1 ~]#  ip -4 route
default via 10.0.1.251 dev eth0
10.0.1.0/24 dev eth0  proto kernel  scope link  src 10.0.1.210

Seems normal (?)

Other traceroutes towards 10.0.11.0/24 hosts were working:

[root@host1 ~]# traceroute 10.0.11.100
 traceroute to 10.0.11.100 (10.0.11.100), 30 hops max, 60 byte packets
 1  10.0.1.251 (10.0.1.251)  0.225 ms  0.192 ms  0.180 ms
 2  10.0.249.6 (10.0.249.6)  5.591 ms
 3  10.0.11.100 (10.0.11.100) 6.524 ms

My gateway for 10.0.1.0/24 is 10.0.1.251, and the above makes sense.  It is only then I realized the host had cached an ICMP_REDIRECT for 10.0.11.220, and I checked this with the ip route command:

[root@host1 ~]# ip route get 10.0.11.220
10.0.11.220 via 10.0.1.250 dev eth0  src 10.0.1.210
    cache <redirected>

bingo.  Cached, ICMP_REDIRECT from 10.0.1.250, which no longer exists.  I didn’t bother to check how long the timeout is for these cached entries, however it was longer than I would have expected, especially since I troubleshot this for more than 15 minutes (*cough* *cough*).

In any case, I learned a new command to zap these, and thankfully, stuff started working again:

root@host1 ~]# ip route flush cache

And with that, my troubles were gone

[root@host1 ~]# ip route get 10.0.11.220
10.0.11.220 via 10.0.1.251 dev eth0  src 10.0.1.210
    cache

from my reading of various Googles, the gc_timeout is what defines the actual timeout:

[root@host1~ proc]# cat ./sys/net/ipv4/route/gc_timeout
300

I can safely say my troubleshooting went way past this timeout, assuming it is seconds and not minutes, and therefore there is some wonkyness beyond I need to check.  In any case, knowing how to clear the cache is evidently useful as well!

Cisco IPv6 flow export with “Flexible Netflow”

Sometime between IOS 12.4 and 15.x, IPv6 flow export configuration changed.  It used to be quite simple:

Pre-flexible-netflow configuration:


voilà….IPv4 and IPv6 flows exported to your favorite collector (mine being the wonderful and always useful  NFDUMP/NFSEN).

Somebody at Cisco obviously found this too easy, so it is now required to re-engineer this functionality into “Flexible Netflow”.  It does allow you, via the creation of a flow record, customize your own format, which is beyond this simple blog post and likely my current understanding as well ;).

For my purposes, I simply want to export IPv6 flows to my collector for business as usual.  This is done by defining a flow exporter and flow monitor, attaching the exporter to the monitor, in the gobal configuration, and applying it to the interface somewhat as before:

Flexible-netflow configuration:


Presto. The original-output defined in the flow monitor specifies using the predefined (?legacy?) record instead of specifying a flow record with user-defined parameters.

And as a side note, if you haven’t played with NFDUMP/NFSEN, you really should give it a try, its very useful as a tool for traffic analysis/DDoS/post-mortems.

 

CentOS policy routing – why yes, it can be done!

Over the years working in LAN networking there are several situations that dictate a host/server have multiple IP addresses on the same or different, physical or logical, devices.  For instance, connecting to a private management-only network/vlan, offering connectivity to a inside network on a private NIC, etc etc.

This scenario often causes two somewhat annoying behaviours:

1) the return traffic often is sourced from the “primary” IP address of the host/server, most often the one that is on the subnet associated with the default gateway

2) a surprising number of alleged “network administrators” seem to think having multiple gateways (one for each IP address of course )  is a good idea.  Well, over the years I have come across this situation and, in every case, this has obviously  NEVER WORKED.

Situation #2 can only be fixed by not mind numbingly entering multiple gateways without restraint.  As for situation #1, RedHat/CentOS and derivatives support via iproute2 the ability to make traffic rules, ensuring that IP traffic is sourced by a particular IP in cases you can define.  Great!  Multiple NIC or logical interface routing on Linux is possible! (and yes, it involves having multiple gateways, but not stupidly and blindly adding them in the routing table….)

It is very simple to implement and involves the steps below.  As an example, lets assume we created a management VLAN (VLAN4) and want to add an logical interface on a server in that VLAN to access it internally.  We will be using 10.0.10.0/28 as an inside network.

Step 1: Create a VLAN interface

This creates the necessary interface on VLAN4 from primary physical interface eth0:

vi /etc/sysconfig/network-scripts/ifcfg-eth0.4

DEVICE=eth0.4
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet
IPV6INIT=no
IPADDR=10.0.10.2
NETMASK=255.255.255.240
NETWORK=10.0.10.0
VLAN=yes

Step 2: Create a iproute2 table for that management network

Edit /etc/iproute2/rt_tables to add a new entry and give it a arbitrary (unused 😉 ) name:

vi /etc/iproute2/rt_tables

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
200     MGMT

Note that between 200 and MGMT is a tab character.

Step 3: Create a default route for that network

vi /etc/sysconfig/network-scripts/route-eth0.4

default table MGMT via 10.0.10.1

this creates a default route for the MGMT/10.0.10.0/28 network to 10.0.10.1 which is your inside routing intelligence.

Step 4: Create routing rule for 10.0.10.2

To ensure that traffic received on 10.0.10.2 is utilizing the MGMT network only as a source address, a rule must be defined to enable this:

vi /etc/sysconfig/network-scripts/rule-eth0.4

from 10.0.10.2 table MGMT

and thats it!  restart your network:

/etc/rc.d/init.d/network restart

Using iproute2 commands, we can check that what we did works (as well as using wireshark 😉 )

[root@server network-scripts]# ip rule show
0:      from all lookup local
32765:  from 10.0.10.2 lookup MGMT
32766:  from all lookup main
32767:  from all lookup default
 
[root@server network-scripts]# ip route show
10.0.10.0/28 dev eth0.4  proto kernel  scope link  src 10.0.10.2
66.1.1.0/25 dev eth0  proto kernel  scope link  src 66.1.1.12
169.254.0.0/16 dev eth0  scope link  metric 1002
169.254.0.0/16 dev eth0.4  scope link  metric 1006
default via 66.1.1.125 dev eth0

Note: this also would work with a second physical interface, for instance to utilize a second NIC card instead of a VLAN logical interface, substitute all use of eth0.4 for eth1.