Have Fun with OpenStack and IPv6 Prefix Delegation

In this post I will talk about IPv6 Prefix Delegation (PD) in OpenStack.  IPv6 is a super important feature, especially in OpenStack. Unlike traditional IPv4 deployments with OpenStack where the cloud admin does not have to worry much about overlapping IPs and colliding address scopes, this is a real issue with IPv6 in OpenStack.  Other than Provider Network scenarios where there is a real exposure of the addressing to each tenant,  in most other scenarios (like using the Neutron L3 agent and having external/private networks per-tenant) overlapping IP support and NAT are used. In these scenarios, tenant A can use 10.0.0.0/24 and tenant B can use 10.0.0.0/24 and you have no fear of address collisions.  With IPv6, there is no NAT and there is no overlapping IP function so IPv6 prefixes can collide and you need a way to prevent the tenant from just willy nilly picking prefixes out of thin are.

There are three common ways to provide IPv6 prefixes to tenant subnets:

Take a look at RFC3633 and 3769 that are referenced above to get a very good look at how IPv6 PD works.  A quick summary of IPv6 PD:

  • IPv6 PD was originally designed to allow a downstream router (Requesting router) to request a prefix from an upstream router (Delegating router) and use the assigned IPv6 prefix for the subscriber-side networks
  • Basically, it is for routers asking other routers/relays/servers for a whole prefix that can be used to ‘seed’ downstream networks with an IPv6 prefix of their own

The following figure is a very basic example of how an IPv6 PD exchange may happen.

  1. A requesting router sends a solicitation upstream asking for an IPv6 prefix
  2. The delegating router/server provides one out of its pool of prefixes
  3. A host comes online and needs an IPv6 address (by sending a solicitation [SLAAC])
  4. The requesting router advertises the IPv6 PD prefix for the host to use

ipv6-pd-flow

The figure below is an example topology that is used for this post.  There is an OpenStack All-in-One (AIO) node that is connected to an L2 switch.  A Dibbler server is deployed and also attached to the L2 switch.  In OpenStack, a public network with an IPv6 prefix is already configured. The goal is to assign an IPv6 prefix to the tenant interface of the Neutron router.

ipv6-pd-topo

Dibbler Server Configuration

The Dibbler server configuration is straightforward. After you have installed Dibber on your test Linux machine, edit the /etc/dibbler/server.conf file and update the pd-class. In the following example, the pd-pool that prefixes will come out of is 2001:db8:face::/48 and the default prefix-length (pd-length) that is assigned is a /64:

pd-class {
     pd-pool 2001:db8:face::/48
     pd-length 64
 }

On the OpenStack side, you need to follow the steps outlined in the OpenStack Neutron IPv6 documentation for installation/setup. Once that is done, the /etc/neutron/neutron.conf file needs to be updated to enable IPv6 PD:

ipv6_pd_enabled = True

Create the Neutron public/external network (In this example I am using a basic flat network type):

neutron net-create public --provider:network_type flat --provider:physical_network public --router:external

Create the Neutron subnet for the public network. Note: This network is a transient network that is used to reach the upstream Dibbler server. Don’t assign a prefix to this network from the pd-pool you defined in the Dibbler configuration:

neutron subnet-create public --ip-version 6 --name public-v6-subnet  2001:db8:bad:cafe::/64

Create a Neutron router:

neutron router-create pd-rtr

Connect the public Neutron network to the Neutron router:

neutron router-gateway-set pd-rtr public

Now, the more interesting part comes in. Within the tenant, create the Neutron network (no dependency on the actual name):

neutron net-create ipv6-pd

Create the Neutron subnet using the special flag –-use-default-subnetpool (this is super critical to do or it won’t work):

neutron subnet-create ipv6-pd --name ipv6-pd-1 --ip_version 6 --ipv6_ra_mode slaac --ipv6_address_mode slaac --use-default-subnetpool

In the output from the subnet-create step, look for”

subnetpool_id | prefix_delegation

Magic happens and unicorns and rainbows appear right after this next step. This is where the Dibbler client running within OpenStack triggers the whole process.

Attach the Neutron router to the new Neutron subnet that was just created:

neutron router-interface-add pd-rtr ipv6-pd-1

Have the Neutron L3 Log running so you can see a thing of beauty:

2016-10-17 15:03:46.822 DEBUG neutron.agent.linux.dibbler [-] Enable IPv6 PD for router 561ed48c-182c-4073-b157-77130280d5b9 subnet 3bc82226-816f-4d71-983e-7429d3d5475a ri_ifname qr-98120bdd-d1 from (pid=56056) enable /opt/stack/neutron/neutron/agent/linux/dibbler.py:123

2016-10-17 15:03:46.824 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-561ed48c-182c-4073-b157-77130280d5b9', 'dibbler-client', 'start', '-w', '/opt/stack/data/neutron/pd/561ed48c-182c-4073-b157-77130280d5b9:3bc82226-816f-4d71-983e-7429d3d5475a:qr-98120bdd-d1'] from (pid=56056) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100

2016-10-17 15:03:46.847 DEBUG neutron.agent.linux.dibbler [-] dibbler client enabled for router 561ed48c-182c-4073-b157-77130280d5b9 subnet 3bc82226-816f-4d71-983e-7429d3d5475a ri_ifname qr-98120bdd-d1 from (pid=56056) enable /opt/stack/neutron/neutron/agent/linux/dibbler.py:129

Have a tcpdump running on the bridge interface that is associated with the public network/physical NIC. The actual messages may change based on this being a brand new assignment vs a renew but the basics are:

  • Client: “Bro (server), gimme a prefix (solicit)”
  • Server: “Sure, bonehead take this one” (advertises out of its pool (2001:db8:face:2ff2::/64 – Remember our pd-pool prefix from the Dibbler config?)
  • Client: “Sweet! I got this prefix advertised and it seems fine so I will officially request it”
  • Server replies and it is a success
15:03:46.852214 IP6 (flowlabel 0x7bf3b, hlim 1, next-header UDP (17) payload length: 60) fe80::f816:3eff:feff:ccb0.dhcpv6-client > ff02::1:2.dhcpv6-server: [udp sum ok] dhcp6 solicit (xid=800a54 (client-ID vid 000022b83bc82226) (IA_PD IAID:1 T1:4294967295 T2:4294967295) (elapsed-time 0))

15:03:46.853654 IP6 (flowlabel 0x80c10, hlim 64, next-header UDP (17) payload length: 134) fe80::20c:29ff:fe87:2f6b.dhcpv6-server > fe80::f816:3eff:feff:ccb0.dhcpv6-client: [udp sum ok] dhcp6 advertise (xid=800a54 (IA_PD IAID:1 T1:2000 T2:3000 (IA_PD-prefix 2001:db8:face:2ff2::/64 pltime:3600 vltime:7200) (status-code success)) (server-ID hwaddr/time type 1 time 529454623 000c29872f6b) (client-ID vid 000022b83bc82226) (preference 0))

15:03:47.955793 IP6 (flowlabel 0x7bf3b, hlim 1, next-header UDP (17) payload length: 107) fe80::f816:3eff:feff:ccb0.dhcpv6-client > ff02::1:2.dhcpv6-server: [udp sum ok] dhcp6 request (xid=561e28 (client-ID vid 000022b83bc82226) (IA_PD IAID:1 T1:4294967295 T2:4294967295 (IA_PD-prefix 2001:db8:face:2ff2::/64 pltime:3600 vltime:7200)) (server-ID hwaddr/time type 1 time 529454623 000c29872f6b) (elapsed-time 0))

15:03:47.956239 IP6 (flowlabel 0x80c10, hlim 64, next-header UDP (17) payload length: 134) fe80::20c:29ff:fe87:2f6b.dhcpv6-server > fe80::f816:3eff:feff:ccb0.dhcpv6-client: [udp sum ok] dhcp6 reply (xid=561e28 (IA_PD IAID:1 T1:2000 T2:3000 (IA_PD-prefix 2001:db8:face:2ff2::/64 pltime:3600 vltime:7200) (status-code success)) (server-ID hwaddr/time type 1 time 529454623 000c29872f6b) (client-ID vid 000022b83bc82226) (preference 0))

On the Dibbler server log (output truncated for clarity):

03:46 Server Notice    Received SOLICIT on br-ex/8, trans-id=0x800a54, 3 opts: 1 25 8 (non-relayed)
. . .
03:46 Server Debug     Adding client (DUID=00:02:00:00:22:b8:3b:c8:22:26:81:6f:4d:71:98:3e:74:29:d3:d5:47:5a) to addrDB.
03:46 Server Debug     PD: Adding PD (iaid=1) to addrDB.
03:46 Server Debug     PD: Adding 2001:db8:face:2ff2:: prefix to PD (iaid=1) to addrDB.
. . .
03:46 Server Notice    Sending ADVERTISE on br-ex/8,transID=0x800a54, opts: 25 2 1 7, 0 relay(s).
. . .
03:47 Server Notice    Received REQUEST on br-ex/8, trans-id=0x561e28, 4 opts: 1 25 2 8 (non-relayed)
. . .
03:47 Server Debug     Checking prefix 2001:db8:face:2ff2:: against reservations ...
03:47 Server Debug     PD: Requested prefix (2001:db8:face:2ff2::) is free, great!
03:47 Server Debug     Adding client (DUID=00:02:00:00:22:b8:3b:c8:22:26:81:6f:4d:71:98:3e:74:29:d3:d5:47:5a) to addrDB.
03:47 Server Debug     PD: Adding PD (iaid=1) to addrDB.
03:47 Server Debug     PD: Adding 2001:db8:face:2ff2:: prefix to PD (iaid=1) to addrDB.
03:47 Server Debug     PD: Prefix usage for class 0 increased to 2.
03:47 Server Info      PD: assigned prefix(es):2001:db8:face:2ff2::/64

That is IPv6 Prefix Delegation for OpenStack. There is a LOT that I did not cover in this post so please read about IPv6 PD in reference links I provided in the beginning.

Deploying IPv6-Only Tenants with OpenStack

In this post, I will discuss the deployment of IPv6-only Neutron networks for tenant use.  A few things to note about IPv6-only instances:

  • IPv6-only works out-of-the-box for basic IP connectivity with OpenStack but you are hosed on the metadata service
  • Many people deploy dual stack in order to overcome the metadata issue
  • The metadata service only supports IPv4
  • An old wishlist bug has long expired: https://bugs.launchpad.net/neutron/+bug/1460177
  • Workarounds:
    • Build all/most of what you want inside the image itself
    • Use config-drive

In this post I am going to use a very basic config-drive example to show how important stuff (metadata) can be injected into an IPv6-only instance. If you don’t know much about config-drive, Red Hat has a reasonable write-up on it. There is tons of info on it, just search. Also, I am working from the topology of previous blog posts such as my last one on provider networks. I won’t be going over the topology and OpenStack setup here.

In this example, I have a file called “user_data.yaml” (no dependency on the actual file name). There is not much in it; I have a FQDN defined and a key. The only thing I am testing here is whether or not the defined FQDN (v6onlyinstance.example.com) appears in the instance at boot. Here is the output:

[root@c7-os-1]# cat user_data.yaml
#cloud-config
fqdn: v6onlyinstance.example.com
users:
  - name: cloud-user
    ssh-authorized-keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4W4RPlOBiY14iJwW9kd3Chys5bUBjy2VKJkFa5az8JHcVvOh3LO5BHdhc6WryT+blmx9LKGyVSc0rfzSEAfQ91dXJCHuhl5BNk9pLibs3oe8s/1r/vjtxQopKIIGN3PYuisvpZVLeP1kRhddIdLvuZcQm82L4VPUAOzLqbFdhsu/Y2lU5WyiTiI5VNJwwbzzc67BFHz2ov2bdBgCfFWyUQMikiyIrAv5hVcqADv7XAqY4P5sJaOaHAcNcCfMtY8RbtEMSIyw8fey1erY4ZiknTAn/eU52mc18l9xR4CwI9wYqYdpVyiNULRWH9opK30dqhhthgElzCax+WqmxMXGP root@c7-os-1.example.com

Now, I boot an instance, enable config-drive and point to the user_data.yaml file. The instance is set to use a Neutron network that I previously added that has an IPv6 prefix setup (no IPv4):

nova boot --flavor m1.small --image rh7-stateless --key-name new-aio-key --security-groups default --nic net-name=external-net rhv6-only-drive --config-drive true --user-data user_data.yaml

Notice that in the ‘nova list’ output below that the name of the instance is “rhv6-only-drive” (based on me naming it that in the ‘nova boot’ command):

[root@c7-os-1 ~]# nova list
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------------------------------
| ID                                   | Name            | Status | Task State | Power State | Networks                                          
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------------------------------
| 2244a346-a34b-4ab6-905f-71dc207a92e6 | rhv6-only-drive | ACTIVE | -          | Running     | external-net=2001:db8:cafe:16:f816:3eff:feec:3c59 
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------------------------------

Login to the instance (notice that I don’t have to mess with no stinking floating IPs or ‘ip netns’ non-sense! No NAT FTW! 🙂 ):

ssh cloud-user@2001:db8:cafe:16:f816:3eff:feec:3c59

Notice that the hostname is what I set in the FQDN line of the user_data.yaml file and that the instance only has an IPv6 address. Also, the instance has the domain name “example.com” from the FQDN in the user_data.yaml file and a nameserver is set (the instance is using Stateless DHCPv6):

[cloud-user@v6onlyinstance ~]$ ip a
. . .
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:ec:3c:59 brd ff:ff:ff:ff:ff:ff
    inet6 2001:db8:cafe:16:f816:3eff:feec:3c59/64 scope global dynamic
       valid_lft 2591952sec preferred_lft 604752sec
    inet6 fe80::f816:3eff:feec:3c59/64 scope link
       valid_lft forever preferred_lft forever

[cloud-user@v6onlyinstance ~]$ cat /etc/resolv.conf
# Generated by NetworkManager
search openstacklocal. example.com
nameserver 2001:db8:cafe:a::e

[cloud-user@v6onlyinstance ~]$ cat /etc/hostname
v6onlyinstance.example.com

Go ahead, give it a try. You might like IPv6-only. 🙂

Deploying IPv6 With OpenStack Provider Networks

Last year I posted on “Tenant IPv6 Deployment in OpenStack Kilo Release” and I also posted on “Using OpenStack Heat to Deploy an IPv6-Enabled Instance”.  Both of those posts were using the Neutron L3 Agent (Tenant routers) for my examples.  In this post I will show the deployment of IPv6 using Provider Networks. There is no specific OpenStack release that I am dictating for this setup. I have used this config on Kilo-to-Newton.

OpenStack Provider Networks with VLANs  allows for the use of VLAN trunks from the upstream Data Center access layer/leaf/ToR switches to the Neutron networks within the OpenStack cloud.  In the use case that I am discussing here, I want to use my Data Center aggregation layer switches as my first-hop layer 3 boundary. I have no use for NAT and I have no use for Neutron L3 agents (specific to running a tenant router).

The following diagram shows the topology that I am using. In this example I have a single All-in-One (AIO) OpenStack node. That node is running on a Cisco UCS C-series with a Cisco VIC which has a VPC configuration to the access layer ToR switches. There are VLAN trunks configured between the ToRs and the Data Center aggregation layer switches (only one shown for simplicity). VLAN 22 (2001:db8:cafe:16::/64) is the VLAN that is used in my examples.  The text box in the diagram shows the NIC layout (ethX<>bonds):

ipv6-provider

I explained a lot about SLAAC, Stateless DHCPv6 and Stateful DHCPv6 Stateful stuff in the blog from May 2015 and I am not rehashing all of that here. Check that post for more details if you are unfamiliar with IPv6 address assignment options. Also, if you want to know more about how Managed (M) and Other (O) flags are used with various IPv6 assignment methods, consult RFC5175.

We are going to jump right into configuration:

Assuming you have a running OpenStack deployment and have followed the guidelines for setting up Neutron to support Provider Networks with VLANs (OVS example, Linux Bridge example), all you have to do is create the provider network and subnet using the IPv6 address assignment method you want (SLAAC, Stateless DHCPv6, Stateful DHCPv6).

Create the Neutron Provider Network with VLAN

In the example below, I am indicating that the router is external (aggregation layer switches), the provider network is of the type VLAN and the VLAN (segmentation_id) associated with this network is VLAN 22:

neutron net-create --router:external --provider:physical_network provider --provider:network_type vlan --provider:segmentation_id=22 --shared external-net

Create the Neutron Subnet using SLAAC

In the example below, I am using SLAAC as the IPv6 address assignment method.  Note: It is very important to indicate the “–allocation-pool” range with provider networks with VLANs because if you don’t then the beginning IPv6 address range will likely cause a DAD (Duplicate Address Detection) failure with IPv6 address already assigned on your upstream VLAN interfaces on the aggregation layer switches.  In this example I am starting the allocation pool range at ::5 so that I do not conflict with addresses on my switches (i.e. ::1 – ::4):

neutron subnet-create external-net --ip-version=6 --ipv6-address-mode=slaac --ipv6-ra-mode=slaac --name=external-subnet-v6 --allocation-pool start=2001:db8:cafe:16::5,end=2001:db8:cafe:16:ffff:ffff:ffff:fffe 2001:db8:cafe:16::/64

Create the Neutron Subnet using Stateless DHCPv6

In the example below, I am using Stateless DHCPv6 as the IPv6 address assignment method.  With Stateless and Stateful DHCPv6 you have the option to add the “–dns-nameserver” flag (since the O-bit [Other configuration] can be set). In this example I am setting 2001:db8:cafe:a::e as the DNS entry which points to my DNS server referenced in the previous diagram.  Again, it is important to setup the “–allocation-pool” range:

neutron subnet-create external-net --ip-version=6 --ipv6-address-mode=dhcpv6-stateless --ipv6-ra-mode=dhcpv6-stateless --name=external-subnet-v6 --allocation-pool start=2001:db8:cafe:16::5,end=2001:db8:cafe:16:ffff:ffff:ffff:fffe 2001:db8:cafe:16::/64 --dns-nameserver 2001:db8:cafe:a::e

Create the Neutron Subnet using Stateful DHCPv6

In the example below, I am using Stateful DHCPv6 as the IPv6 address assignment method.  As was the case with Stateless DHCPv6, Stateful DHCPv6 allows  for the option to add the “–dns-nameserver” flag (since the O-bit can be set):

neutron subnet-create external-net --ip-version=6 --ipv6-address-mode=dhcpv6-stateful --ipv6-ra-mode=dhcpv6-stateful --name=external-subnet-v6 --allocation-pool start=2001:db8:cafe:16::5,end=2001:db8:cafe:16:ffff:ffff:ffff:fffe 2001:db8:cafe:16::/64 --dns-nameserver 2001:db8:cafe:a::e

Example Configuration for the upstream Data Center aggregation layer switch (VLAN interfaces shown):

SLAAC:

This example shows VLAN22 with an IPv6 address of 2001:db8:cafe:16::1/64. HSRPv2 is used as the First-Hop Redundancy Protocol.

interface Vlan22
 description Provider Network trunked for C7-os-1
 ip address 172.16.22.2 255.255.255.0
 ipv6 address 2001:DB8:CAFE:16::1/64
 standby version 2
 standby 2 ipv6 autoconfig
 standby 2 timers msec 250 msec 750
 standby 2 priority 110
 standby 2 preempt
 standby 2 authentication OPEN

Stateless DHCPv6:

This example is the same as the previous one with the exception of the “ipv6 nd other-config-flag” being set. This flat sets the O-bit which allows for the DNS option (or other options) to be sent to the VM in the Router Advertisement (RA).

interface Vlan22
 description Provider Network trunked for C7-os-1
 ip address 172.16.22.2 255.255.255.0
 ipv6 address 2001:DB8:CAFE:16::1/64
 ipv6 nd other-config-flag
 standby version 2
 standby 2 ipv6 autoconfig
 standby 2 timers msec 250 msec 750
 standby 2 priority 110
 standby 2 preempt
 standby 2 authentication OPEN

Stateful DHCPv6:

This example is also the same as the first one with the exception of the “ipv6 nd managed-config-flag” being set. This sets the M (Managed) and O (other) bits. The M-bit indicates that the addressing comes from DHCPv6 (Not SLAAC) and that the host wants options (DNS):

interface Vlan22
 description Provider Network trunked for C7-os-1
 ip address 172.16.22.2 255.255.255.0
 ipv6 address 2001:DB8:CAFE:16::1/64
 ipv6 nd managed-config-flag
 standby version 2
 standby 2 ipv6 autoconfig
 standby 2 timers msec 250 msec 750
 standby 2 priority 110
 standby 2 preempt
 standby 2 authentication OPEN

Consult my previous blog post on the stuff that you need to be aware of on the Linux VM side of things when you enable Stateless DHCPv6 and Stateful DHCPv6.

Have fun!