RANCID and IOS 15.2 – blank config and how to work around newer file privileges

In or around IOS 15.2 apparently a privilege structural change was made that breaks non-priv-15 users from being able to copy the running config, useful for RANCID and other tools.  And either the RANCID/Oxidized community simply uses privilege 15 users in their configs, which I refuse to do on priciple, or my google-fu is poor because I have not found this info in any explicit form.

In any case, the symptom of the typical config allowing to download a running config without having level-15 privileges on Cisco IOS has always been documented as:

username rancidbackup privilege 10 secret [md5-pass]
privilege exec level 10 show running-config view full
privilege exec level 10 show running-config view
privilege exec level 10 show running-config
privilege exec all level 1 show

 

the above in IOS <= 15.1 was always enough to allow user “rancidbackup” to issue a “show running-config view full” and the CLI would output the current configuration.  Under IOS 15.2, behaviour appears to have changed.  Some folks receive a “permission denied” error while others, such as my experience, a simple empty config would be output.  As an example:

ssh -l rancidbackup 10.0.255.1

Password: 

Router#show running-config view full
[...nothing...]
Router#show running-config
[...nothing...]          
Router#

This annoying behaviour can be worked around in two different ways.

The first, is to cave and allow the rancid backup user to obtain level-15 privileges upon login.  The academic risk of compromise by this admin-level user can be mitigated by limiting the IP addresses used for login via ACL:  example:

username rancidbackup access-class 99 privilege 15 secret [md5-pass]
!
access-list 99 remark RANCID-ONLY
access-list 99 permit host 10.0.1.25 any
!

By adding access-class to the username definition, it is possible to avoid this user being used from other sources.  This might even be recommendable for any scenario where scripting/tools are used for access to the device.  However, this method still requires that user to have admin level-15 privileges.

 

I was pointed to https://supportforums.cisco.com/discussion/11691446/error-opening-nvramstartup-config-permission-denied as a possible second solution.  Apparently there is a mecanism to allow file access (which includes running-config, apparently…), with the command “file privilege [level]”

In our case, by issuing “file privilege 10”, we are able to see via “show running-config view full” (and not just show running-config) the info we seek to backup with our tool, in this case RANCID (as you can see, “show running config” is useless but “show running-config view full” outputs what we need):

 

Router#show running-config 

Building configuration...
Current configuration : 261 bytes
!
! Last configuration change at 22:04:36 EDT Mon May 30 2016 by user
! NVRAM config last updated at 13:15:46 EDT Thu May 26 2016 by user
! NVRAM config last updated at 13:15:46 EDT Thu May 26 2016 by user
boot-start-marker
boot-end-marker
!
!
!
!
!
!
!
end

Router#show running-config view full 
Building configuration...
Current configuration : 90474 bytes
!
! Last configuration change at 22:04:36 EDT Mon May 30 2016 by user
! NVRAM config last updated at 13:15:46 EDT Thu May 26 2016 by user
! NVRAM config last updated at 13:15:46 EDT Thu May 26 2016 by user
upgrade fpd auto
version 15.2
.... etc etc
!

huzzah!

I am still researching secondary implications from setting “file privilege” to anything other than level-15.  I haven’t found a way, using a privilege 10 user, to modify/delete the files so use at your own risk (!).  But combining the two above mecanisms I believe provides a sufficient means to allow non-privileged, read-only access to the configuration for use with config management and other scripting tools.

 

Cisco IPv6 flow export with “Flexible Netflow”

Sometime between IOS 12.4 and 15.x, IPv6 flow export configuration changed.  It used to be quite simple:

Pre-flexible-netflow configuration:


voilà….IPv4 and IPv6 flows exported to your favorite collector (mine being the wonderful and always useful  NFDUMP/NFSEN).

Somebody at Cisco obviously found this too easy, so it is now required to re-engineer this functionality into “Flexible Netflow”.  It does allow you, via the creation of a flow record, customize your own format, which is beyond this simple blog post and likely my current understanding as well ;).

For my purposes, I simply want to export IPv6 flows to my collector for business as usual.  This is done by defining a flow exporter and flow monitor, attaching the exporter to the monitor, in the gobal configuration, and applying it to the interface somewhat as before:

Flexible-netflow configuration:


Presto. The original-output defined in the flow monitor specifies using the predefined (?legacy?) record instead of specifying a flow record with user-defined parameters.

And as a side note, if you haven’t played with NFDUMP/NFSEN, you really should give it a try, its very useful as a tool for traffic analysis/DDoS/post-mortems.

 

6to4 – whats in it for me??

We are a month away from world IPv6-day(bis), which was arguably a yawnfest last year.   It was a huge success in the fact that it happened, and other than a small handful of people, nobody noticed 😛    Essentially, many heavy hitters on the Internet such as Google, Facebook, and a bunch of others, turned on their quad-As for 24 hours, as an “experiment”.  The Internet didn’t blow up, everybody pretty much went on business as usual – and for some reason, that tiny fraction of people with broken IPv6 stacks was enough to have those AAAAs removed from DNS.

June 6th 2012 will be the second coming of “World IPv6 Day”, and this time, the theory is that AAAAs will be permanently added to DNS……ZMFG!

And as network operators, we are striving to dual-stack our networks end to end, so that all our users can benefit from having IPv6 connectivity on their networks before that day, right?!  Well unfortunately not all of us have managed that 🙁  My company included.  I guess I have nobody to blame but myself.  We have some progress made, but not as much as I would have liked.  Considering many companies still have zero deployment, by north-American standards I am doing ok 😉

However, this does not mean that nobody will be, or is currently using, some form of IPv6, be it over a Hurricane Electric tunnel broker, 6RD, or even 6to4.   6to4 is particularily relevant since Windows7 (Vista as well?), Windows 2003 and probably other OSs have 6to4 enabled interfaces on by default.  I personally don’t like 6to4 very much, and I hope that it dies very, very soon; however if some folks are necessarily using it, and you haven’t gotten around to dual-stacking everyone, might as well make 6to4 slightly more useable, or at the very least, help it limp along.  One of the inherent operational issues with 6to4 is that it depends on finding a relay server, in order to go from the IPv4 world to IPv6, and vice-versa.  It would be a good idea to set up, as a network operator, your own 6to4 relay, in order to at least know where in the network your users are connecting, since 6to4 traffic will “find” the nearest one, and it can be anywhere; best it be on your network.

There is an RFC which deals specifically with 6to4 from the network operator’s perspective.

I won’t try and explain all of 6to4 here, however in essence, IPv6 prefix 2002::/16 is a reserved prefix that IPv4 hosts can derive their 6to4 address from.  As an example, 192.168.1.1 would derive its 6to4 IP to be:

192 (dec) == C0 (hex)
168 (dec) == A8 (hex)
1 (dec) == 1 (hex)
1 (dec) -- 1 (hex)

This becomes, in 6to4 – 2002:C0A8:101::/48    This unique prefix can then be routed back and forth from v4 to v6 and back over 6to4 relays.  It is the 2002:V4ADDR::/48 prefix.   A reserved anycast address, 192.88.99.1 (which microsoft seems to discover by querying 6to4.ipv6.microsoft.com), is used as a relay to the IPv6 network.  (note: RFC 1918 addressing and 6to4 prefixes cannot exist or work on the Internet, I only use them here for example addresses….)

To illustrate the point, if I disable my 6to4 relay, I see that Hurricane Electric is nicely anycasting 192.88.99.1 and would be the nearest 6to4 relay I can use.  A traceroute to ARIN over 6to4:

C:\Users\Administrator>tracert -6 www.arin.net

Tracing route to www.arin.net [2001:500:4:13::80]
over a maximum of 30 hops:

  1     *        *        *     Request timed out.
  2   116 ms   116 ms   116 ms  2001:16d8:1:6240::1
  3   110 ms   110 ms   111 ms  10gigabitethernet1-2.core1.sto1.he.net [2001:7f8:d:ff::187]
  4   110 ms   120 ms   110 ms  10gigabitethernet3-2.core1.ams1.he.net [2001:470:0:22f::1]
  5   111 ms   122 ms   111 ms  10gigabitethernet1-4.core1.lon1.he.net [2001:470:0:3f::1]
  6   114 ms   111 ms   112 ms  10gigabitethernet7-4.core1.nyc4.he.net [2001:470:0:128::1]
  7   118 ms   121 ms   124 ms  10gigabitethernet2-3.core1.ash1.he.net [2001:470:0:36::1]
  8   118 ms   126 ms   118 ms  arin.10gigabitethernet14.switch3.ash1.he.net [2001:470:1:20f::2]
  9   118 ms   118 ms   118 ms  cr2.arin.net [2001:500:4:10::12]
 10   121 ms   121 ms   121 ms  cr3-ptp.arin.net [2001:500:4:11::2]
 11   122 ms   163 ms   122 ms  2001:500:4:12::3
 12   121 ms   125 ms   122 ms  www.arin.net [2001:500:4:13::80]

Trace complete.

100+ms round trip.  Not terrible, not great.

What if I was to run a 6to4 relay for my users?  Here is the result:

C:\Users\Administrator>tracert -6 www.arin.net

Tracing route to www.arin.net [2001:500:4:13::80]
over a maximum of 30 hops:

  1     2 ms     1 ms     1 ms  2002:aaaa:5ef1::
  2    19 ms    14 ms    19 ms  2001:aaaa:2:16::ff
  3    18 ms    17 ms    18 ms  te0-3-0-0.ccr22.ymq02.atlas.cogentco.com [::ffff:154.54.0.13]
  4    19 ms    18 ms    19 ms  te0-6-0-6.ccr22.jfk02.atlas.cogentco.com [::ffff:154.54.46.46]
  5    18 ms    19 ms    18 ms  te0-3-0-3.ccr22.dca01.atlas.cogentco.com [::ffff:154.54.26.181]
  6    19 ms    18 ms    18 ms  te0-0-0-2.ccr22.iad02.atlas.cogentco.com [::ffff:154.54.1.178]
  7    18 ms    18 ms    17 ms  2001:550:3::24a
  8    31 ms    23 ms    49 ms  2001:578:1:0:172:17:249:18
  9    19 ms    19 ms    20 ms  2001:578:2800:5::27
 10    19 ms    20 ms    19 ms  2001:578:2800:5::90
 11    26 ms    20 ms    20 ms  2001:578:2803:1100::2
 12    20 ms    21 ms    21 ms  cr3.arin.net [2001:500:4:12::1]
 13    21 ms    21 ms    21 ms  2001:500:4:12::3
 14    23 ms    23 ms    21 ms  www.arin.net [2001:500:4:13::80]

Trace complete.

Neat! 25ms or so, a fraction of the round trip time the “nearest” 6to4 relay could offer me.  Success!

Of course, it must be noted that as a network operator, you can only influence the relay used towards the IPv6 network – the nearest relay from the destination node back to the IPv4 network will be used.  That is one of the drawbacks of 6to4 – although there can be relays peppered all over the Internet, the one used to get to IPv6 will in most likelyhood not be the one used to get back to IPv4.  But the point is to improve 6to4, so hosting your own relay is favourable for your customers.

 

The setup is insanely and trivially easy.  Here is an example Cisco config:

!
ipv6 unicast-routing
ipv6 cef
!
interface Tunnel0
 description 6to4 Tunnel
 no ip address
 ipv6 address 2002:C0A8:101:::/128
 ipv6 enable
 tunnel source Loopback6
 tunnel mode ipv6ip 6to4
!
!
interface Loopback6
 ip address 192.88.99.1 255.255.255.0 secondary
 ip address 192.168.1.1 255.255.255.0
 ipv6 address 2001:db8:FFFF:FFFF::1/64
 ipv6 enable
!
ipv6 route 2002::/16 Tunnel0
!

 

voila!  In the above examples, substitute 2002:C0A8:101:: and 192.168.1.1 for the IP address of your router used for 6to4 and you are off to the races.   Make sure you distribute 192.88.99.0/24 and 2002::/16 in your IGP.