Tag Archives: OpenStack

Have Fun with OpenStack and IPv6 Prefix Delegation

In this post I will talk about IPv6 Prefix Delegation (PD) in OpenStack.  IPv6 is a super important feature, especially in OpenStack. Unlike traditional IPv4 deployments with OpenStack where the cloud admin does not have to worry much about overlapping IPs and colliding address scopes, this is a real issue with IPv6 in OpenStack.  Other than Provider Network scenarios where there is a real exposure of the addressing to each tenant,  in most other scenarios (like using the Neutron L3 agent and having external/private networks per-tenant) overlapping IP support and NAT are used. In these scenarios, tenant A can use 10.0.0.0/24 and tenant B can use 10.0.0.0/24 and you have no fear of address collisions.  With IPv6, there is no NAT and there is no overlapping IP function so IPv6 prefixes can collide and you need a way to prevent the tenant from just willy nilly picking prefixes out of thin are.

There are three common ways to provide IPv6 prefixes to tenant subnets:

Take a look at RFC3633 and 3769 that are referenced above to get a very good look at how IPv6 PD works.  A quick summary of IPv6 PD:

  • IPv6 PD was originally designed to allow a downstream router (Requesting router) to request a prefix from an upstream router (Delegating router) and use the assigned IPv6 prefix for the subscriber-side networks
  • Basically, it is for routers asking other routers/relays/servers for a whole prefix that can be used to ‘seed’ downstream networks with an IPv6 prefix of their own

The following figure is a very basic example of how an IPv6 PD exchange may happen.

  1. A requesting router sends a solicitation upstream asking for an IPv6 prefix
  2. The delegating router/server provides one out of its pool of prefixes
  3. A host comes online and needs an IPv6 address (by sending a solicitation [SLAAC])
  4. The requesting router advertises the IPv6 PD prefix for the host to use

ipv6-pd-flow

The figure below is an example topology that is used for this post.  There is an OpenStack All-in-One (AIO) node that is connected to an L2 switch.  A Dibbler server is deployed and also attached to the L2 switch.  In OpenStack, a public network with an IPv6 prefix is already configured. The goal is to assign an IPv6 prefix to the tenant interface of the Neutron router.

ipv6-pd-topo

Dibbler Server Configuration

The Dibbler server configuration is straightforward. After you have installed Dibber on your test Linux machine, edit the /etc/dibbler/server.conf file and update the pd-class. In the following example, the pd-pool that prefixes will come out of is 2001:db8:face::/48 and the default prefix-length (pd-length) that is assigned is a /64:

pd-class {
     pd-pool 2001:db8:face::/48
     pd-length 64
 }

On the OpenStack side, you need to follow the steps outlined in the OpenStack Neutron IPv6 documentation for installation/setup. Once that is done, the /etc/neutron/neutron.conf file needs to be updated to enable IPv6 PD:

ipv6_pd_enabled = True

Create the Neutron public/external network (In this example I am using a basic flat network type):

neutron net-create public --provider:network_type flat --provider:physical_network public --router:external

Create the Neutron subnet for the public network. Note: This network is a transient network that is used to reach the upstream Dibbler server. Don’t assign a prefix to this network from the pd-pool you defined in the Dibbler configuration:

neutron subnet-create public --ip-version 6 --name public-v6-subnet  2001:db8:bad:cafe::/64

Create a Neutron router:

neutron router-create pd-rtr

Connect the public Neutron network to the Neutron router:

neutron router-gateway-set pd-rtr public

Now, the more interesting part comes in. Within the tenant, create the Neutron network (no dependency on the actual name):

neutron net-create ipv6-pd

Create the Neutron subnet using the special flag –-use-default-subnetpool (this is super critical to do or it won’t work):

neutron subnet-create ipv6-pd --name ipv6-pd-1 --ip_version 6 --ipv6_ra_mode slaac --ipv6_address_mode slaac --use-default-subnetpool

In the output from the subnet-create step, look for”

subnetpool_id | prefix_delegation

Magic happens and unicorns and rainbows appear right after this next step. This is where the Dibbler client running within OpenStack triggers the whole process.

Attach the Neutron router to the new Neutron subnet that was just created:

neutron router-interface-add pd-rtr ipv6-pd-1

Have the Neutron L3 Log running so you can see a thing of beauty:

2016-10-17 15:03:46.822 DEBUG neutron.agent.linux.dibbler [-] Enable IPv6 PD for router 561ed48c-182c-4073-b157-77130280d5b9 subnet 3bc82226-816f-4d71-983e-7429d3d5475a ri_ifname qr-98120bdd-d1 from (pid=56056) enable /opt/stack/neutron/neutron/agent/linux/dibbler.py:123

2016-10-17 15:03:46.824 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-561ed48c-182c-4073-b157-77130280d5b9', 'dibbler-client', 'start', '-w', '/opt/stack/data/neutron/pd/561ed48c-182c-4073-b157-77130280d5b9:3bc82226-816f-4d71-983e-7429d3d5475a:qr-98120bdd-d1'] from (pid=56056) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100

2016-10-17 15:03:46.847 DEBUG neutron.agent.linux.dibbler [-] dibbler client enabled for router 561ed48c-182c-4073-b157-77130280d5b9 subnet 3bc82226-816f-4d71-983e-7429d3d5475a ri_ifname qr-98120bdd-d1 from (pid=56056) enable /opt/stack/neutron/neutron/agent/linux/dibbler.py:129

Have a tcpdump running on the bridge interface that is associated with the public network/physical NIC. The actual messages may change based on this being a brand new assignment vs a renew but the basics are:

  • Client: “Bro (server), gimme a prefix (solicit)”
  • Server: “Sure, bonehead take this one” (advertises out of its pool (2001:db8:face:2ff2::/64 – Remember our pd-pool prefix from the Dibbler config?)
  • Client: “Sweet! I got this prefix advertised and it seems fine so I will officially request it”
  • Server replies and it is a success
15:03:46.852214 IP6 (flowlabel 0x7bf3b, hlim 1, next-header UDP (17) payload length: 60) fe80::f816:3eff:feff:ccb0.dhcpv6-client > ff02::1:2.dhcpv6-server: [udp sum ok] dhcp6 solicit (xid=800a54 (client-ID vid 000022b83bc82226) (IA_PD IAID:1 T1:4294967295 T2:4294967295) (elapsed-time 0))

15:03:46.853654 IP6 (flowlabel 0x80c10, hlim 64, next-header UDP (17) payload length: 134) fe80::20c:29ff:fe87:2f6b.dhcpv6-server > fe80::f816:3eff:feff:ccb0.dhcpv6-client: [udp sum ok] dhcp6 advertise (xid=800a54 (IA_PD IAID:1 T1:2000 T2:3000 (IA_PD-prefix 2001:db8:face:2ff2::/64 pltime:3600 vltime:7200) (status-code success)) (server-ID hwaddr/time type 1 time 529454623 000c29872f6b) (client-ID vid 000022b83bc82226) (preference 0))

15:03:47.955793 IP6 (flowlabel 0x7bf3b, hlim 1, next-header UDP (17) payload length: 107) fe80::f816:3eff:feff:ccb0.dhcpv6-client > ff02::1:2.dhcpv6-server: [udp sum ok] dhcp6 request (xid=561e28 (client-ID vid 000022b83bc82226) (IA_PD IAID:1 T1:4294967295 T2:4294967295 (IA_PD-prefix 2001:db8:face:2ff2::/64 pltime:3600 vltime:7200)) (server-ID hwaddr/time type 1 time 529454623 000c29872f6b) (elapsed-time 0))

15:03:47.956239 IP6 (flowlabel 0x80c10, hlim 64, next-header UDP (17) payload length: 134) fe80::20c:29ff:fe87:2f6b.dhcpv6-server > fe80::f816:3eff:feff:ccb0.dhcpv6-client: [udp sum ok] dhcp6 reply (xid=561e28 (IA_PD IAID:1 T1:2000 T2:3000 (IA_PD-prefix 2001:db8:face:2ff2::/64 pltime:3600 vltime:7200) (status-code success)) (server-ID hwaddr/time type 1 time 529454623 000c29872f6b) (client-ID vid 000022b83bc82226) (preference 0))

On the Dibbler server log (output truncated for clarity):

03:46 Server Notice    Received SOLICIT on br-ex/8, trans-id=0x800a54, 3 opts: 1 25 8 (non-relayed)
. . .
03:46 Server Debug     Adding client (DUID=00:02:00:00:22:b8:3b:c8:22:26:81:6f:4d:71:98:3e:74:29:d3:d5:47:5a) to addrDB.
03:46 Server Debug     PD: Adding PD (iaid=1) to addrDB.
03:46 Server Debug     PD: Adding 2001:db8:face:2ff2:: prefix to PD (iaid=1) to addrDB.
. . .
03:46 Server Notice    Sending ADVERTISE on br-ex/8,transID=0x800a54, opts: 25 2 1 7, 0 relay(s).
. . .
03:47 Server Notice    Received REQUEST on br-ex/8, trans-id=0x561e28, 4 opts: 1 25 2 8 (non-relayed)
. . .
03:47 Server Debug     Checking prefix 2001:db8:face:2ff2:: against reservations ...
03:47 Server Debug     PD: Requested prefix (2001:db8:face:2ff2::) is free, great!
03:47 Server Debug     Adding client (DUID=00:02:00:00:22:b8:3b:c8:22:26:81:6f:4d:71:98:3e:74:29:d3:d5:47:5a) to addrDB.
03:47 Server Debug     PD: Adding PD (iaid=1) to addrDB.
03:47 Server Debug     PD: Adding 2001:db8:face:2ff2:: prefix to PD (iaid=1) to addrDB.
03:47 Server Debug     PD: Prefix usage for class 0 increased to 2.
03:47 Server Info      PD: assigned prefix(es):2001:db8:face:2ff2::/64

That is IPv6 Prefix Delegation for OpenStack. There is a LOT that I did not cover in this post so please read about IPv6 PD in reference links I provided in the beginning.

Deploying IPv6-Only Tenants with OpenStack

In this post, I will discuss the deployment of IPv6-only Neutron networks for tenant use.  A few things to note about IPv6-only instances:

  • IPv6-only works out-of-the-box for basic IP connectivity with OpenStack but you are hosed on the metadata service
  • Many people deploy dual stack in order to overcome the metadata issue
  • The metadata service only supports IPv4
  • An old wishlist bug has long expired: https://bugs.launchpad.net/neutron/+bug/1460177
  • Workarounds:
    • Build all/most of what you want inside the image itself
    • Use config-drive

In this post I am going to use a very basic config-drive example to show how important stuff (metadata) can be injected into an IPv6-only instance. If you don’t know much about config-drive, Red Hat has a reasonable write-up on it. There is tons of info on it, just search. Also, I am working from the topology of previous blog posts such as my last one on provider networks. I won’t be going over the topology and OpenStack setup here.

In this example, I have a file called “user_data.yaml” (no dependency on the actual file name). There is not much in it; I have a FQDN defined and a key. The only thing I am testing here is whether or not the defined FQDN (v6onlyinstance.example.com) appears in the instance at boot. Here is the output:

[root@c7-os-1]# cat user_data.yaml
#cloud-config
fqdn: v6onlyinstance.example.com
users:
  - name: cloud-user
    ssh-authorized-keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4W4RPlOBiY14iJwW9kd3Chys5bUBjy2VKJkFa5az8JHcVvOh3LO5BHdhc6WryT+blmx9LKGyVSc0rfzSEAfQ91dXJCHuhl5BNk9pLibs3oe8s/1r/vjtxQopKIIGN3PYuisvpZVLeP1kRhddIdLvuZcQm82L4VPUAOzLqbFdhsu/Y2lU5WyiTiI5VNJwwbzzc67BFHz2ov2bdBgCfFWyUQMikiyIrAv5hVcqADv7XAqY4P5sJaOaHAcNcCfMtY8RbtEMSIyw8fey1erY4ZiknTAn/eU52mc18l9xR4CwI9wYqYdpVyiNULRWH9opK30dqhhthgElzCax+WqmxMXGP root@c7-os-1.example.com

Now, I boot an instance, enable config-drive and point to the user_data.yaml file. The instance is set to use a Neutron network that I previously added that has an IPv6 prefix setup (no IPv4):

nova boot --flavor m1.small --image rh7-stateless --key-name new-aio-key --security-groups default --nic net-name=external-net rhv6-only-drive --config-drive true --user-data user_data.yaml

Notice that in the ‘nova list’ output below that the name of the instance is “rhv6-only-drive” (based on me naming it that in the ‘nova boot’ command):

[root@c7-os-1 ~]# nova list
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------------------------------
| ID                                   | Name            | Status | Task State | Power State | Networks                                          
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------------------------------
| 2244a346-a34b-4ab6-905f-71dc207a92e6 | rhv6-only-drive | ACTIVE | -          | Running     | external-net=2001:db8:cafe:16:f816:3eff:feec:3c59 
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------------------------------

Login to the instance (notice that I don’t have to mess with no stinking floating IPs or ‘ip netns’ non-sense! No NAT FTW! 🙂 ):

ssh cloud-user@2001:db8:cafe:16:f816:3eff:feec:3c59

Notice that the hostname is what I set in the FQDN line of the user_data.yaml file and that the instance only has an IPv6 address. Also, the instance has the domain name “example.com” from the FQDN in the user_data.yaml file and a nameserver is set (the instance is using Stateless DHCPv6):

[cloud-user@v6onlyinstance ~]$ ip a
. . .
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:ec:3c:59 brd ff:ff:ff:ff:ff:ff
    inet6 2001:db8:cafe:16:f816:3eff:feec:3c59/64 scope global dynamic
       valid_lft 2591952sec preferred_lft 604752sec
    inet6 fe80::f816:3eff:feec:3c59/64 scope link
       valid_lft forever preferred_lft forever

[cloud-user@v6onlyinstance ~]$ cat /etc/resolv.conf
# Generated by NetworkManager
search openstacklocal. example.com
nameserver 2001:db8:cafe:a::e

[cloud-user@v6onlyinstance ~]$ cat /etc/hostname
v6onlyinstance.example.com

Go ahead, give it a try. You might like IPv6-only. 🙂

Deploying IPv6 With OpenStack Provider Networks

Last year I posted on “Tenant IPv6 Deployment in OpenStack Kilo Release” and I also posted on “Using OpenStack Heat to Deploy an IPv6-Enabled Instance”.  Both of those posts were using the Neutron L3 Agent (Tenant routers) for my examples.  In this post I will show the deployment of IPv6 using Provider Networks. There is no specific OpenStack release that I am dictating for this setup. I have used this config on Kilo-to-Newton.

OpenStack Provider Networks with VLANs  allows for the use of VLAN trunks from the upstream Data Center access layer/leaf/ToR switches to the Neutron networks within the OpenStack cloud.  In the use case that I am discussing here, I want to use my Data Center aggregation layer switches as my first-hop layer 3 boundary. I have no use for NAT and I have no use for Neutron L3 agents (specific to running a tenant router).

The following diagram shows the topology that I am using. In this example I have a single All-in-One (AIO) OpenStack node. That node is running on a Cisco UCS C-series with a Cisco VIC which has a VPC configuration to the access layer ToR switches. There are VLAN trunks configured between the ToRs and the Data Center aggregation layer switches (only one shown for simplicity). VLAN 22 (2001:db8:cafe:16::/64) is the VLAN that is used in my examples.  The text box in the diagram shows the NIC layout (ethX<>bonds):

ipv6-provider

I explained a lot about SLAAC, Stateless DHCPv6 and Stateful DHCPv6 Stateful stuff in the blog from May 2015 and I am not rehashing all of that here. Check that post for more details if you are unfamiliar with IPv6 address assignment options. Also, if you want to know more about how Managed (M) and Other (O) flags are used with various IPv6 assignment methods, consult RFC5175.

We are going to jump right into configuration:

Assuming you have a running OpenStack deployment and have followed the guidelines for setting up Neutron to support Provider Networks with VLANs (OVS example, Linux Bridge example), all you have to do is create the provider network and subnet using the IPv6 address assignment method you want (SLAAC, Stateless DHCPv6, Stateful DHCPv6).

Create the Neutron Provider Network with VLAN

In the example below, I am indicating that the router is external (aggregation layer switches), the provider network is of the type VLAN and the VLAN (segmentation_id) associated with this network is VLAN 22:

neutron net-create --router:external --provider:physical_network provider --provider:network_type vlan --provider:segmentation_id=22 --shared external-net

Create the Neutron Subnet using SLAAC

In the example below, I am using SLAAC as the IPv6 address assignment method.  Note: It is very important to indicate the “–allocation-pool” range with provider networks with VLANs because if you don’t then the beginning IPv6 address range will likely cause a DAD (Duplicate Address Detection) failure with IPv6 address already assigned on your upstream VLAN interfaces on the aggregation layer switches.  In this example I am starting the allocation pool range at ::5 so that I do not conflict with addresses on my switches (i.e. ::1 – ::4):

neutron subnet-create external-net --ip-version=6 --ipv6-address-mode=slaac --ipv6-ra-mode=slaac --name=external-subnet-v6 --allocation-pool start=2001:db8:cafe:16::5,end=2001:db8:cafe:16:ffff:ffff:ffff:fffe 2001:db8:cafe:16::/64

Create the Neutron Subnet using Stateless DHCPv6

In the example below, I am using Stateless DHCPv6 as the IPv6 address assignment method.  With Stateless and Stateful DHCPv6 you have the option to add the “–dns-nameserver” flag (since the O-bit [Other configuration] can be set). In this example I am setting 2001:db8:cafe:a::e as the DNS entry which points to my DNS server referenced in the previous diagram.  Again, it is important to setup the “–allocation-pool” range:

neutron subnet-create external-net --ip-version=6 --ipv6-address-mode=dhcpv6-stateless --ipv6-ra-mode=dhcpv6-stateless --name=external-subnet-v6 --allocation-pool start=2001:db8:cafe:16::5,end=2001:db8:cafe:16:ffff:ffff:ffff:fffe 2001:db8:cafe:16::/64 --dns-nameserver 2001:db8:cafe:a::e

Create the Neutron Subnet using Stateful DHCPv6

In the example below, I am using Stateful DHCPv6 as the IPv6 address assignment method.  As was the case with Stateless DHCPv6, Stateful DHCPv6 allows  for the option to add the “–dns-nameserver” flag (since the O-bit can be set):

neutron subnet-create external-net --ip-version=6 --ipv6-address-mode=dhcpv6-stateful --ipv6-ra-mode=dhcpv6-stateful --name=external-subnet-v6 --allocation-pool start=2001:db8:cafe:16::5,end=2001:db8:cafe:16:ffff:ffff:ffff:fffe 2001:db8:cafe:16::/64 --dns-nameserver 2001:db8:cafe:a::e

Example Configuration for the upstream Data Center aggregation layer switch (VLAN interfaces shown):

SLAAC:

This example shows VLAN22 with an IPv6 address of 2001:db8:cafe:16::1/64. HSRPv2 is used as the First-Hop Redundancy Protocol.

interface Vlan22
 description Provider Network trunked for C7-os-1
 ip address 172.16.22.2 255.255.255.0
 ipv6 address 2001:DB8:CAFE:16::1/64
 standby version 2
 standby 2 ipv6 autoconfig
 standby 2 timers msec 250 msec 750
 standby 2 priority 110
 standby 2 preempt
 standby 2 authentication OPEN

Stateless DHCPv6:

This example is the same as the previous one with the exception of the “ipv6 nd other-config-flag” being set. This flat sets the O-bit which allows for the DNS option (or other options) to be sent to the VM in the Router Advertisement (RA).

interface Vlan22
 description Provider Network trunked for C7-os-1
 ip address 172.16.22.2 255.255.255.0
 ipv6 address 2001:DB8:CAFE:16::1/64
 ipv6 nd other-config-flag
 standby version 2
 standby 2 ipv6 autoconfig
 standby 2 timers msec 250 msec 750
 standby 2 priority 110
 standby 2 preempt
 standby 2 authentication OPEN

Stateful DHCPv6:

This example is also the same as the first one with the exception of the “ipv6 nd managed-config-flag” being set. This sets the M (Managed) and O (other) bits. The M-bit indicates that the addressing comes from DHCPv6 (Not SLAAC) and that the host wants options (DNS):

interface Vlan22
 description Provider Network trunked for C7-os-1
 ip address 172.16.22.2 255.255.255.0
 ipv6 address 2001:DB8:CAFE:16::1/64
 ipv6 nd managed-config-flag
 standby version 2
 standby 2 ipv6 autoconfig
 standby 2 timers msec 250 msec 750
 standby 2 priority 110
 standby 2 preempt
 standby 2 authentication OPEN

Consult my previous blog post on the stuff that you need to be aware of on the Linux VM side of things when you enable Stateless DHCPv6 and Stateful DHCPv6.

Have fun!

 

Using OpenStack Heat to Deploy an IPv6-enabled Instance

In this post I will talk about how to use a basic OpenStack Heat template to build a dual-stack (IPv4 and IPv6) Neutron network, router and launch an instance that will use StateLess Address AutoConfiguration (SLAAC) for IPv6 address assignment.

In the May, 2015 post I discussed, in detail, how to build a dual-stack tenant and use a variety of IPv6 address assignment methods (SLAAC, Stateless DHCPv6, Stateful DHCPv6) for OpenStack instances.

To build on the previous post, I wanted to show a basic Heat template for building an IPv4 and IPv6 network with the basic parameters such as CIDR, gateway and pools.  I want Heat to also launch an OpenStack instance (what Heat calls a Server) that attaches to those networks.  Finally, the template will create a new security group that will create security group rules for both IPv4 and IPv6.

The Heat template that I am referring to in this post can be found here: https://github.com/shmcfarl/my-heat-templates/blob/master/single-v6-test.yaml. That specific template is using SLAAC.  You can also take a look at this template which uses Stateless DHCPv6: https://github.com/shmcfarl/my-heat-templates/blob/master/stateless-demo-slb-trusty.yaml. You can modify the template from there to play around with DHCPv6 Stateful. Hint, it’s all in the resource properties of:

ipv6_address_mode: <slaac/dhcpv6-stateless/dhcpv6-stateful>
ipv6_ra_mode: <slaac/dhcpv6-stateless/dhcpv6-stateful>)

Heat Template

I am not going to teach you Heat. There are countless resources out there that do a much better job than I ever could on teaching Heat.  A couple of places to start are:

The Heat Orchestration Template (HOT) Guide is a great resource for finding the various parameters, resources and properties that can be used in Heat.

The primary place to dig into IPv6 capabilities within Heat is in the Heat template guide under OS::Neutron::Subnet.  You can jump to it here: http://docs.openstack.org/hot-reference/content/OS__Neutron__Subnet.html.  I am not going to walk through all of what is in the guide but I will point out specific properties that I have used in the example Heat template I referenced before.

Let’s take a look at the IPv6-specific parts of the example template.  In the example template file I have created a parameter section that includes various items such as key, image, flavor and so on.  The IPv6 section includes:

  • The private IPv6 network (2001:db8:cafe:1e::/64)
  • The private IPv6 gateway (2001:db8:cafe:1e::1)
  • The beginning and ending range of the IPv6 address allocation pool (2001:db8:cafe:1e::2 – 2001:db8:cafe:1e:ffff:ffff:ffff:fffe)
private_net_v6:
    type: string
    description: Private IPv6 subnet address
    default: 2001:db8:cafe:1e::/64
private_net_v6_gateway:
    type: string
    description: Private IPv6 network gateway address
    default: 2001:db8:cafe:1e::1
private_net_v6_pool_start:
    type: string
    description: Start of private network IPv6 address allocation pool
    default: 2001:db8:cafe:1e::2
private_net_v6_pool_end:
    type: string
    description: End of private network IPv6 address allocation pool
    default: 2001:db8:cafe:1e:ffff:ffff:ffff:fffe

The next section to look at is in the “resources” section and this is where things go into action. The “private_v6_subnet” has various resource types and properties to include:

  • Version is IPv6
  • IPv6 address and RA modes are SLAAC
  • The network property (set in the parameter section)
  • The CIDR property which is the “private_net_v6” from the parameter section
  • The gateway IPv6 address is defined in the “private_net_v6_gateway” parameter
  • The allocation pool is defined in the “private_net_v6_pool_start/end” parameters
  private_v6_subnet:
    type: OS::Neutron::Subnet
    properties:
      ip_version: 6
      ipv6_address_mode: slaac
      ipv6_ra_mode: slaac
      network: { get_resource: private_net }
      cidr: { get_param: private_net_v6 }
      gateway_ip: { get_param: private_net_v6_gateway }
      allocation_pools:
        - start: { get_param: private_net_v6_pool_start }
          end: { get_param: private_net_v6_pool_end }

The next IPv6-relevant area of the resource section is “router_interface_v6”. In the “router_interface_v6” resource, there is a reference to the previously created “router” resource (see template file for full resource list) and the “private_v6_subnet”. This entry is simply attaching a new router interface to the Private IPv6 subnet.

  router_interface_v6:
    type: OS::Neutron::RouterInterface
    properties:
      router: { get_resource: router }
      subnet: { get_resource: private_v6_subnet }

Next, there is the Server (AKA “instance” or “VM”) creation section. There is nothing IPv6 specific here. On the network property line, Heat is pointing to “get_resource: private_net” which is the private network that both IPv4 and IPv6 subnets are associated with. That line, basically, attaches the server to a dual-stack network.

server1:
    type: OS::Nova::Server
    properties:
      name: Server1
      image: { get_param: image }
      flavor: { get_param: flavor }
      key_name: { get_param: key_name }
      networks:
        - network: { get_resource: private_net }
      config_drive: "true"
      user_data_format: RAW
      user_data: |
        #!/bin/bash
      security_groups: [{ get_resource: server_security_group }]

Finally, there is the security group section which enables rules for both IPv4 and IPv6. In this example ports 22, 80 and ICMP are open for IPv4 and IPv6.

server_security_group:
    type: OS::Neutron::SecurityGroup
    properties:
      description: Heat-deployed security group.
      name: heat-security-group
      rules: [
        {remote_ip_prefix: 0.0.0.0/0,
        protocol: tcp,
        port_range_min: 22,
        port_range_max: 22},
        {remote_ip_prefix: 0.0.0.0/0,
        protocol: icmp},
        {remote_ip_prefix: 0.0.0.0/0,
        protocol: tcp,
        port_range_min: 80,
        port_range_max: 80},
        {remote_ip_prefix: "::/0",
        ethertype: IPv6,
        protocol: tcp,
        port_range_min: 22,
        port_range_max: 22},
        {remote_ip_prefix: "::/0",
        ethertype: IPv6,
        protocol: icmp},
        {remote_ip_prefix: "::/0",
        ethertype: IPv6,
        protocol: tcp,
        port_range_min: 80,
        port_range_max: 80}]

Now, let’s deploy this template and see how it all looks. I am deploying the Heat “stack” using the Heat “stack-create” command (alternatively you can deploy it using the ‘Orchestration > Stacks > Launch Stack’ interface in the OpenStack Dashboard). In this example I am running “stack-create” using the “-r” argument to indicate ‘rollback’ (in the event something goes wrong, I don’t want the whole stack to build out). Then I am using the “-f” argument to indicate that I am using a file to build the Heat stack. The stack is named “demo-v6”:

root@c71-kilo-aio:~$ heat stack-create -r -f Heat-Templates/single-v6-test.yaml demo-v6
+--------------------------------------+------------+--------------------+----------------------+
| id                                   | stack_name | stack_status       | creation_time        |
+--------------------------------------+------------+--------------------+----------------------+
| 688388f5-4ae1-4d39-bf85-6f9a591a4420 | demo-v6    | CREATE_IN_PROGRESS | 2015-06-29T15:44:18Z |
+--------------------------------------+------------+--------------------+----------------------+

After a few minutes, the Heat stack is built:

root@c71-kilo-aio:~$ heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        |
+--------------------------------------+------------+-----------------+----------------------+
| 688388f5-4ae1-4d39-bf85-6f9a591a4420 | demo-v6    | CREATE_COMPLETE | 2015-06-29T15:44:18Z |
+--------------------------------------+------------+-----------------+----------------------+

Here is a messy view of the obligatory OpenStack Dashboard Network Topology view (Note: Some Horizon guru needs to line break the IPv4 and IPv6 addresses for the instances so they are readable ;-)):

net-topo

 

Here’s a cleaner view of things:

  • Network list – You can see the new Heat-built “test_net” with the two subnets (IPv4/IPv6) as well as the previously built (by the admin) “Public-Network”:
root@c71-kilo-aio:~$ neutron net-list
+--------------------------------------+----------------+------------------------------------------------------------+
| id                                   | name           | subnets                                                    |
+--------------------------------------+----------------+------------------------------------------------------------+
| 2e03a628-e85e-4519-b1bb-a579880be0ae | test_net       | 93764f36-c56b-4c65-b7d7-cb78a694353b 10.10.30.0/24         |
|                                      |                | cafb610a-2aaa-4640-b0f0-8bb4b60cbaf2 2001:db8:cafe:1e::/64 |
| f6a55029-d875-48a8-aab9-1a5a5399592b | Public-Network | dda7d8f1-89a6-40bb-b11b-64a62c103828 192.168.81.0/24       |
|                                      |                | f2107125-c98e-4375-a81f-d0f4d34bdae3 2001:db8:cafe:51::/64 |
+--------------------------------------+----------------+------------------------------------------------------------+
  • Subnet list:
root@c71-kilo-aio:~$ neutron subnet-list
+--------------------------------------+----------------------------------------+-----------------------+---------------------------------------------------------------------------------+
| id                                   | name                                   | cidr                  | allocation_pools                                                                |
+--------------------------------------+----------------------------------------+-----------------------+---------------------------------------------------------------------------------+
| 93764f36-c56b-4c65-b7d7-cb78a694353b | demo-v6-private_subnet-6evpyylqyux7    | 10.10.30.0/24         | {"start": "10.10.30.2", "end": "10.10.30.254"}                                  |
| dda7d8f1-89a6-40bb-b11b-64a62c103828 | Public-Subnet-v4                       | 192.168.81.0/24       | {"start": "192.168.81.5", "end": "192.168.81.254"}                              |
| f2107125-c98e-4375-a81f-d0f4d34bdae3 | Public-Subnet-v6                       | 2001:db8:cafe:51::/64 | {"start": "2001:db8:cafe:51::3", "end": "2001:db8:cafe:51:ffff:ffff:ffff:fffe"} |
| cafb610a-2aaa-4640-b0f0-8bb4b60cbaf2 | demo-v6-private_v6_subnet-vvsmlbkc6sds | 2001:db8:cafe:1e::/64 | {"start": "2001:db8:cafe:1e::2", "end": "2001:db8:cafe:1e:ffff:ffff:ffff:fffe"} |
+--------------------------------------+----------------------------------------+-----------------------+---------------------------------------------------------------------------------+
  • Here is the “subnet-show” of the Heat-built subnet for the Private IPv6 subnet. The allocation pool range, gateway, IPv6 version, IPv6 address mode and IPv6 RA modes are all defined as we wanted (based on the Heat template):
root@c71-kilo-aio:~$ neutron subnet-show demo-v6-private_v6_subnet-vvsmlbkc6sds
+-------------------+---------------------------------------------------------------------------------+
| Field             | Value                                                                           |
+-------------------+---------------------------------------------------------------------------------+
| allocation_pools  | {"start": "2001:db8:cafe:1e::2", "end": "2001:db8:cafe:1e:ffff:ffff:ffff:fffe"} |
| cidr              | 2001:db8:cafe:1e::/64                                                           |
| dns_nameservers   |                                                                                 |
| enable_dhcp       | True                                                                            |
| gateway_ip        | 2001:db8:cafe:1e::1                                                             |
| host_routes       |                                                                                 |
| id                | cafb610a-2aaa-4640-b0f0-8bb4b60cbaf2                                            |
| ip_version        | 6                                                                               |
| ipv6_address_mode | slaac                                                                           |
| ipv6_ra_mode      | slaac                                                                           |
| name              | demo-v6-private_v6_subnet-vvsmlbkc6sds                                          |
| network_id        | 2e03a628-e85e-4519-b1bb-a579880be0ae                                            |
| subnetpool_id     |                                                                                 |
| tenant_id         | dc52b50429f74aeabb3935eb3e2bcb04                                                |
+-------------------+---------------------------------------------------------------------------------+
  • Router port list – You can see that the router has IPv4/IPv6 addresses on the tenant and public network interfaces:
root@c71-kilo-aio:~$ neutron router-port-list demo-v6-router-txy5s5bcixqd | grep ip_address | sed -e 's#.*ip_address": "\([^"]\+\).*#\1#'
10.10.30.1
2001:db8:cafe:1e::1
192.168.81.75
2001:db8:cafe:51::3f
  • Security Group list:
root@c71-kilo-aio:~$ neutron security-group-list
+-----------------------------+---------------------+---------------------------------------------------+
| id                          | name                | security_group_rules                              |
+-----------------------------+---------------------+---------------------------------------------------+
| 69f81e8e-5059-4a...         | heat-security-group | egress, IPv4                                      |                                                       |                             |                     | egress, IPv6                                      |
|                             |                     | ingress, IPv4, 22/tcp, remote_ip_prefix: 0.0.0.0/0|                                                       |                             |                     | ingress, IPv4, 80/tcp, remote_ip_prefix: 0.0.0.0/0|
|                             |                     | ingress, IPv4, icmp, remote_ip_prefix: 0.0.0.0/0  |
|                             |                     | ingress, IPv6, 22/tcp, remote_ip_prefix: ::/0     |
|                             |                     | ingress, IPv6, 80/tcp, remote_ip_prefix: ::/0     |
|                             |                     | ingress, IPv6, icmp, remote_ip_prefix: ::/0       |
+--------------------------------------+---------------------+------------------------------------------+
  • Server/Instance list:
root@c71-kilo-aio:~$ nova list
+--------------------------------------+---------+--------+------------+-------------+-----------------------------------------------------------+
| ID                                   | Name    | Status | Task State | Power State | Networks                                                  |
+--------------------------------------+---------+--------+------------+-------------+-----------------------------------------------------------+
| d7bfc606-f9da-4be5-b3e8-2219882c3da6 | Server1 | ACTIVE | -          | Running     | test_net=10.10.30.3, 2001:db8:cafe:1e:f816:3eff:fea8:7d2c |
+--------------------------------------+---------+--------+------------+-------------+-----------------------------------------------------------+

Finally, inside the instance, you can see that both IPv4 and IPv6 addresses are assigned:

root@c71-kilo-aio:~$ ip netns exec qrouter-d2ff159a-b603-4a3b-b5f7-481bff40613e ssh fedora@2001:db8:cafe:1e:f816:3eff:fea8:7d2c
The authenticity of host '2001:db8:cafe:1e:f816:3eff:fea8:7d2c (2001:db8:cafe:1e:f816:3eff:fea8:7d2c)' can't be established.
ECDSA key fingerprint is 41:e2:ea:28:e5:6d:ae:50:24:81:ad:5e:db:d7:a0:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '2001:db8:cafe:1e:f816:3eff:fea8:7d2c' (ECDSA) to the list of known hosts.
[fedora@server1 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:a8:7d:2c brd ff:ff:ff:ff:ff:ff
    inet 10.10.30.3/24 brd 10.10.30.255 scope global dynamic eth0
       valid_lft 79179sec preferred_lft 79179sec
    inet6 2001:db8:cafe:1e:f816:3eff:fea8:7d2c/64 scope global mngtmpaddr dynamic
       valid_lft 86400sec preferred_lft 14400sec
    inet6 fe80::f816:3eff:fea8:7d2c/64 scope link
       valid_lft forever preferred_lft forever

I hope this gives you a starting point for adding IPv6 to your Heat template collection.

Thanks,
Shannon

Tenant IPv6 Deployment in OpenStack Kilo Release

You folks know that I have been dancing to the IPv6 music for the last 12 years or so. IPv6 is so popular that I still earn about 75 cents a month on my book royalties. 🙂

I have been fairly disappointed in the deployment of IPv6 in an OpenStack environment.  Many individuals and companies have beat on IPv6 over the past couple of years to get it to where it is today in OpenStack and we should all be super greatful.  In this post I will go over some design and deployment considerations for IPv6 in the OpenStack Kilo release.

Note: I refer to the term “tenant” vs. what OpenStack calls a “project”.

Cloud and IPv6

IPv6 has had a rough go of it over the many years it has been around. It is usually a rough go when you try to inject it into any cloud deployment. And I do mean ANY cloud deployment.  The hard stuff with cloud and IPv6 includes (not an exhaustive list):

  • API Endpoints – Enabling IPv6 on the non-Tenant-facing portion of the cloud stack
  • Provisioning, Orchestration and Management – This includes CI/CD, bare-metal provisioning (i.e. Cobbler, Ironic), automation tools such as Ansible, Puppet, etc.. and the overall Data Center management tools
  • Management/communication protocol interoperability and support of IPv6
  • Support for IPv6 in both virtual and physical networking
  • Oh, let’s not forget – expertise

In this post I am not going to address all of the above as none of us have that kind of time.  What I will talk about is making a choice to enable ALL of OpenStack for IPv6 or settling for the really important part, which is the tenant-facing part and how that looks when configured.

There are two common ways of embracing IPv6 within OpenStack (I am not discussing IPv6-only options here):

  • Dual Stack everything
  • Conditional Dual Stack

Figure 1 shows an example of a “dual stack everything” approach. It is what is sounds like. Everything in the entire stack is both IPv4 and IPv6 enabled.  This includes the API endpoints, the DBs, the tools and systems that surround OpenStack (i.e. provisioning and automation).  This, kids, can be a real pain in the butt, especially if this is a brownfield deployment where you have to go in an muck around with database entries for IPv6 and so on.

Figure 1. Dual Stack Everything

ds-everything

This is where “conditional dual stack” comes in.  It allows you to control when and where IPv6 is enabled.  In many cases, it is just in the tenant space, at least initially.

Figure 2 shows an example of where we have an IPv4-only OpenStack control plane (API/DB/etc) but dual stack for anything that faces something a tenant would see/control/interact with.  Optionally, this is where you can begin experimenting (I stress that word) with IPv6-only tenant networks.

Figure 2. Conditional Dual Stack

conditional

 

No matter which path you go down (and there are variances to the two I have pointed out here), you will end up dealing with IPv6 in the tenant space.  So, let’s talk about that part.

Tenant IPv6 Address Options

There are many combinations to IPv6 address assignment and even the types of addresses you can use.  For tenant address types you can use:

You can use those independently or together.  As far as planning out the use of GUA, ULA or combination of the two you have to think about the requirements of the tenant.  Most of the time the tenant has no clue what ‘routable’ means or NAT and they shouldn’t.  In the context of IPv6, we don’t want to do NAT, ever. Well, with one exception – we may need, for a very temporary use case, NAT64 where we translating incoming IPv6 connections into back-end IPv4-only nodes.  This is an entirely different use case than wanting to translate between two IPv6 address spaces.  Don’t do that. 🙂

With all of that said, I tend to see two of MANY address options in production deployments:

  • Cloud provider assigned addressing – Provider owns and assigns IPv6 addressing
  • Tenant-provided addressing – Tenant brings their own IPv6 addressing and the cloud provider has to route it

What I mean by “cloud provider assigned addressing” is that the owner of the system (the cloud provider – which could be a public provider or just an Enterprise IT shop) will obtain a Provider Independent (PI) or Provider Assigned (PA) IPv6 address block, design an address plan and then assign blocks out of that plan to each tenant. This works well, is easy to do and is the way you should do it.

Tenant-provided addressing is possible, but messy.  In the Enterprise space, this is not something that usually happens unless you are dealing with a merger & acquisition (M&A) situation where an acquired company wants their existing IPv6 addressing to be used in the acquiring company’s cloud service.  This is something that is requested when an Enterprise or another SP is using another public cloud provider.  Again, it is totally doable but it requires a lot of planning with BGP, address space design, etc..

From an address assignment perspective, you can use:

  • Static/manual configuration
  • StateLess Address AutoConfiguration (SLAAC) – IPv6 prefix (i.e /64) is assigned to the end node via a router advertisement (RA) and the node self-constructs the interface ID (IID) portion of the address (i.e. the last /64 bits)
  • Stateful DHCPv6 – Just like IPv4 DHCP, a DHCPv6 server hands out full IPv6 addressing and any configured options
  • Stateless DHCPv6 – A combination of SLAAC (for address assignment) and DHCPv6 (for option assignment – DNS name, domain name)

In the OpenStack Kilo release we have functional support for SLAAC, Stateful DHCPv6 and Stateless DHCPv6.  I will cover the configuration of all three in this post.

Example Topology

The lab setup I am using for this post includes:

  • Kilo
  • Devstack – only because I did this testing before packages were available for RDO/Packstack, Ubuntu/Juju/MaaS or any other intaller
  • All-in-one node (although in another lab I have I did this same setup on 7 nodes and it worked as expected)
  • The lab I did this test in did not have external IPv6 connectivity to the Internet, but I do have IPv6 throughout that lab so I could verify proper routing beyond OpenStack

Figure 3 shows a basic topology of the example setup.  The all-in-one (AIO) has a management network interface (eth0) and an OpenStack public network interface (eth1).  Both interfaces are connected to a Top-of-Rack (ToR) switch which has an uplink to the Data Center Aggregation Layer switches (via trunked VLANs).  The Aggregation Layer switches have VLAN interfaces for each network and are providing IPv4 and IPv6 routing for the rest of the lab. There are IPv6 routes configured on the Aggregation Layer switches for each IPv6 prefix within the OpenStack tenant space.  Off of the ToR is a DNS server that is IPv4 and IPv6 enabled.

Figure 3. Example Topology

topo

Before I walk through each address assignment type, I will create the public network and subnets as the cloud admin. This can be done in the dashboard or via the Neutron client CLI. In this example a “public” network is created and the upstream Data Center L3 switches (–router:external) will be used as the gateway. An IPv4 subnet is created with an allocation range so that the addresses assigned do not collide with IPv4 addresses assigned on the Data Center L3 switches. An IPv6 subnet is created which also has an allocation range defined. If you have an upstream gateway address that is not .1 (v4) or ::1 (v6) then you need to identify the real gateway address using the “–gateway” option:

neutron net-create public --router:external

neutron subnet-create --name public-subnet --allocation-pool start=172.16.12.5,end=172.16.12.254 public 172.16.12.0/24

neutron subnet-create --ip-version=6 --name=public-v6-subnet --allocation-pool start=2001:db8:cafe:d::5,end=2001:db8:cafe:d:ffff:ffff:ffff:fffe --disable-dhcp public 2001:db8:cafe:d::/64

Let’s walk through SLAAC, Stateful DHCPv6 and Stateless DHCPv6.

SLAAC

Figure 4 gives a topology view of the SLAAC example I am referencing.  The Neutron router has a private network with IPv4 and IPv6 subnets and a public network with gateway connections (Also, dual stack) to the Data Center. The instance will have a dual stack connection and receive DHCPv4 address assigned from DNSMASQ and IPv6 address assignment via radvd in SLAAC mode:

Figure 4. SLAAC Topology Example

slaac-topo

When you use SLAAC mode in the tenant space, the Router Advertisement Daemon (radvd) is used to send router advertisements (RAs) in response to Router Solicitations (RS) as well as at a regular interval.  When you create a Neutron network and subnet for IPv6 you have to select the “IPv6 Address Configuration Mode” from the “Subnet Details” screen in the dashboard. In this case you would select “SLAAC: Address discovered from OpenStack Router”.  If you were doing this via CLI, you would use the option arguments of “–ipv6-address-mode=slaac –ipv6-ra-mode=slaac” when you create the subnet.  In the example below, a new Neutron network (“private”) is created along with the IPv6 subnet creation (IPv4 subnet creation is not shown). The option arguments are added to identify SLAAC mode. Also, no DNS servers are added here as that information for IPv6 would not get injected into the instance and the instance (since it is set for SLAAC-only) would not ask for DNS and other options:

neutron net-create private
 
neutron subnet-create --ip-version=6 --name=private_v6_subnet --ipv6-address-mode=slaac --ipv6-ra-mode=slaac private 2001:db8:cafe::/64

Created a new subnet:
+-------------------+----------------------------------------------------------------+
| Field             | Value                                                          |
+-------------------+----------------------------------------------------------------+
| allocation_pools  | {"start": "2001:db8:cafe::2", "end": "2001:db8:cafe:0:ffff:ffff:ffff:fffe"}|
| cidr              | 2001:db8:cafe::/64                                             |
| dns_nameservers   |                                                                |
| enable_dhcp       | True                                                           |
| gateway_ip        | 2001:db8:cafe::1                                               |
| host_routes       |                                                                |
| id                | 42cc3dbc-938b-4ad6-b12e-59aef7618477                           |
| ip_version        | 6                                                              |
| ipv6_address_mode | slaac                                                          |
| ipv6_ra_mode      | slaac                                                          |
| name              | private_v6_subnet                                              |
| network_id        | 7166ce15-c581-4195-9479-ad2283193d06                           |
| subnetpool_id     |                                                                |
| tenant_id         | f057804eb39b4618b40e06196e16265b                               |
+-------------------+----------------------------------------------------------------+

Inside the tenant (as a tenant member or admin) a Neutron router needs to be created along with attaching the router to the public network and the two subnets:

neutron router-create router

neutron router-gateway-set router public

neutron router-interface-add router private_v4_subnet

neutron router-interface-add router private_v6_subnet

When an instance boots inside the tenant and attaches to the “private” network it will have access to both the IPv4 and the IPv6 subnets that were defined previously. Note: Both Windows (since Windows 7/Server 2008) and Linux (from a long time ago) support SLAAC out of the box just so basic IPv6 protocol support is enabled.  The instance will receive and IPv4 address via DNSMASQ and an IPv6 address via radvd. Note: The host will NOT receive any kind of IPv6 DNS entry from OpenStack. There are a couple of considerations to understand with this. An instance can get both IPv4 and IPv6 DNS information for a host over IPv4 transport. You don’t need to access a DNS server natively over IPv6 in order to receive IPv6 host information in a lookup. But, if you do want to have an IPv6 entry in the /etc/resolv.conf file then, in SLAAC mode, you will need to configure it manually, setup cloud-init to inject it or just bake that entry into the image that is booted.

A highly summarized view of a tcpdump capture reveals the basic flow of a SLAAC exchange.  The ICMPv6 Flags field states “none” and the Prefix Information Flag field has “auto” enabled. The combination of “none” (Managed and Other bits = 0) and “auto” (Auto bit = 1) indicates SLAAC is used and DHCPv6 is not used. You can read more on this in RFC 4861:

IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::f816:3eff:fe79:5acc > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
	  source link-address option (1), length 8 (1): fa:16:3e:79:5a:cc
	    0x0000:  fa16 3e79 5acc

IP6 (hlim 255, next-header ICMPv6 (58) payload length: 56) fe80::f816:3eff:fec3:17b4 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 56
	hop limit 64, Flags [none], pref medium, router lifetime 30s, reachable time 0s, retrans time 0s
	  prefix info option (3), length 32 (4): 2001:db8:cafe::/64, Flags [onlink, auto], valid time 86400s, pref. time 14400s
	    0x0000:  40c0 0001 5180 0000 3840 0000 0000 2001
	    0x0010:  0db8 cafe 0000 0000 0000 0000 0000
	  source link-address option (1), length 8 (1): fa:16:3e:c3:17:b4
	    0x0000:  fa16 3ec3 17b4

The instance will send out a router solicitation and the router will reply with a router advertisement. In that RA, the router will send various bits of information (various lifetimes and other stuff) and, in the case of SLAAC, it will send out a “Flags [none]”, indicating that it is not giving out anything other than a prefix. In that RA is the prefix from the subnet that was configured earlier (2001:db8:cafe::/64). The instance will use that prefix, along with its own IID (discussed earlier) to come up with a valid 128-bit IPv6 address. The instance will use the router’s link-local address as its gateway.  After the host has the IPv6 address it will go through a series of events such as Duplicate Address Detection (DAD) and so on. I won’t be explaining all of that stuff here. 😉

Stateful DHCPv6

Figure 5 is a topology view of the Stateful DHCPv6 layout. It is pretty much the same as with SLAAC only using a different IPv4 and IPv6 prefix.

Figure 5. Stateful DHCPv6 Topology Example

stateful-topo

Just like with SLAAC mode, radvd is used with Stateful DHCPv6 mode.  Instead of using the prefix announcement (Auto bit set to 1) in the RA to assist with IPv6 address assignment (SLAAC mode), Stateful DHCPv6 uses the Managed bit (set to 1) and the Auto bit (set to 0) to instruct the instance to perform a  DHCPv6 solicit for both address assignment and other options.

Configuring OpenStack to use Stateful DHCPv6 is done using the same process as was shown in the SLAAC example only with different options. In the dashboard, “DHCPv6 stateful: Address discovered from OpenStack DHCP” is selected in the “subnet details” screen.  Unlike with SLAAC, a DNS name server entry can be added and it will be sent to the instance.  If the Neutron client CLI is used then the option arguments would be: “–ipv6-address-mode=dhcpv6-stateful –ipv6-ra-mode=dhcpv6-stateful”.  Here’s an example:

neutron net-create private-dhcpv6
 
neutron subnet-create --ip-version=6 --name=private_dhcpv6_subnet --ipv6-address-mode=dhcpv6-stateful --ipv6-ra-mode=dhcpv6-stateful private-dhcpv6 2001:db8:cafe:1::/64 --dns-nameserver 2001:db8:cafe:a::e
Created a new subnet:
+-------------------+----------------------------------------------------------------+
| Field             | Value                                                          |
+-------------------+----------------------------------------------------------------+
| allocation_pools  | {"start": "2001:db8:cafe:1::2", "end": "2001:db8:cafe:1:ffff:ffff:ffff:fffe"} |
| cidr              | 2001:db8:cafe:1::/64                                           |
| dns_nameservers   | 2001:db8:cafe:a::e                                             |
| enable_dhcp       | True                                                           |
| gateway_ip        | 2001:db8:cafe:1::1                                             |
| host_routes       |                                                                |
| id                | 545ea206-9d14-4dca-8bae-7940719bdab5                           |
| ip_version        | 6                                                              |
| ipv6_address_mode | dhcpv6-stateful                                                |
| ipv6_ra_mode      | dhcpv6-stateful                                                |
| name              | private_dhcpv6_subnet                                          |
| network_id        | 55ed8333-2876-400a-92c1-ef49bc10aa2b                           |
| subnetpool_id     |                                                                |
| tenant_id         | f057804eb39b4618b40e06196e16265b                               |
+-------------------+----------------------------------------------------------------+

After performing “neutron router-interface-add” for each subnet (see SLAAC example), it’s time to setup the instance operating system for DHCPv6 operation. To learn about how Microsoft Windows supports IPv6 address assignment check out this blog.  No Linux version that I have ever worked with is configured, by default, to use DHCPv6.  For Ubuntu you have to add the “inet6 dhcp” line in the /etc/network/interfaces file:

auto eth0
iface eth0 inet dhcp
iface eth0 inet6 dhcp

I have tested CentOS 7 and Fedora 21 in this test bed and the following configuration works on those (add to the appropriate interface in /etc/sysconfig/network-scripts/):

IPV6INIT="yes"
DHCPV6C="yes"

Included below is a highly summarized output from a tcpdump capture of a Stateful DHCPv6 exchange.  There is a lot more going on here than in the SLAAC example.  Just like the SLAAC example, there is a RS/RA exchange but that is where the similarity ends.  In the RA below, the ICMPv6 Flags field is set to “managed” (M bit = 1). Note: that when the M bit is set to 1, the O bit, per RFC 4861 is redundant and can be ignored.  The instance will receive the RA and see that the “managed” flag is set and will then begin the DHCPv6 client process.  The instance will send a DHCPv6 Solicit to the ‘all DHCP servers’ well-known IPv6 multicast address (ff02::1:2).  Radvd will do a DHCPv6 advertise with all of the good stuff (address and options). The instance will do a DHCPv6 request for options (it processed the address info from the advertise) and the server will finish the sequence with a DHCPv6 reply.  You can read a summary view of this process here:

IP6 (hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::f816:3eff:fe77:e5a0 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 24
	hop limit 64, Flags [managed], pref medium, router lifetime 30s, reachable time 0s, retrans time 0s
	  source link-address option (1), length 8 (1): fa:16:3e:77:e5:a0
	    0x0000:  fa16 3e77 e5a0

IP6 (hlim 1, next-header UDP (17) payload length: 64) fe80::f816:3eff:fe22:386b.546 > ff02::1:2.547: [udp sum ok] dhcp6 solicit (xid=85680b (client-ID hwaddr/time type 1 time 482446373 fa163e22386b) (option-request DNS-server DNS-search-list Client-FQDN SNTP-servers) (elapsed-time 101) (IA_NA IAID:1042430059 T1:3600 T2:5400))

IP6 (class 0xc0, hlim 64, next-header UDP (17) payload length: 175) fe80::f816:3eff:fe06:176f.547 > fe80::f816:3eff:fe22:386b.546: [udp sum ok] dhcp6 advertise (xid=85680b (client-ID hwaddr/time type 1 time 482446373 fa163e22386b) (server-ID hwaddr type 1 fa163e06176f) (IA_NA IAID:1042430059 T1:43200 T2:75600 (IA_ADDR 2001:db8:cafe:1::4 pltime:86400 vltime:86400)) (status-code success) (preference 255) (DNS-search-list openstacklocal.) (DNS-server 2001:db8:cafe:a::e) (Client-FQDN))

IP6 (hlim 1, next-header UDP (17) payload length: 106) fe80::f816:3eff:fe22:386b.546 > ff02::1:2.547: [udp sum ok] dhcp6 request (xid=9cb172 (client-ID hwaddr/time type 1 time 482446373 fa163e22386b) (server-ID hwaddr type 1 fa163e06176f) (option-request DNS-server DNS-search-list Client-FQDN SNTP-servers) (elapsed-time 0) (IA_NA IAID:1042430059 T1:3600 T2:5400 (IA_ADDR 2001:db8:cafe:1::4 pltime:7200 vltime:7500)))

IP6 (class 0xc0, hlim 64, next-header UDP (17) payload length: 186) fe80::f816:3eff:fe06:176f.547 > fe80::f816:3eff:fe22:386b.546: [udp sum ok] dhcp6 reply (xid=9cb172 (client-ID hwaddr/time type 1 time 482446373 fa163e22386b) (server-ID hwaddr type 1 fa163e06176f) (IA_NA IAID:1042430059 T1:3600 T2:6300 (IA_ADDR 2001:db8:cafe:1::4 pltime:7200 vltime:7500)) (status-code success) (DNS-search-list openstacklocal.) (DNS-server 2001:db8:cafe:a::e) (Client-FQDN))

Stateless DHCPv6

Stateless DHCPv6 is a combo of SLAAC (for address assignment) and DHCPv6 (for options).  For the sake of consistency with the last two sections, here is a topology sample for Stateless DHCPv6:

Figure 6. Stateless DHCPv6 Sample Topology

stateless-topo

The configuration of Stateless DHCPv6 is similar to that of Stateful DHCPv6.  In the dashboard, select “DHCPv6 stateless: Address discovered from OpenStack Router and additional information from OpenStack DHCP” from the “subnet details” screen.  The options argument changes if using the Neutron client CLI: “–ipv6-address-mode=dhcpv6-stateless –ipv6-ra-mode=dhcpv6-stateless private-dhcpv6-stateless”.  Here is an example:

neutron net-create private-dhcpv6-stateless

neutron subnet-create --ip-version=6 --name=private_dhcpv6_stateless_subnet --ipv6-address-mode=dhcpv6-stateless --ipv6-ra-mode=dhcpv6-stateless private-dhcpv6-stateless 2001:db8:cafe:2::/64 --dns-nameserver 2001:db8:cafe:a::e
Created a new subnet:
+-------------------+--------------------------------------------------------------+
| Field             | Value                                                        |
+-------------------+--------------------------------------------------------------+
| allocation_pools  | {"start": "2001:db8:cafe:2::2", "end": "2001:db8:cafe:2:ffff:ffff:ffff:fffe"} |
| cidr              | 2001:db8:cafe:2::/64                                         |
| dns_nameservers   | 2001:db8:cafe:a::e                                           |
| enable_dhcp       | True                                                         |
| gateway_ip        | 2001:db8:cafe:2::1                                           |
| host_routes       |                                                              |
| id                | edd1d404-e949-4cdf-9812-334bbf0e5cec                         |
| ip_version        | 6                                                            |
| ipv6_address_mode | dhcpv6-stateless                                             |
| ipv6_ra_mode      | dhcpv6-stateless                                             |
| name              | private_dhcpv6_stateless_subnet                              |
| network_id        | f65c6e60-d31e-4a1c-8136-599c3855b86a                         |
| subnetpool_id     |                                                              |
| tenant_id         | f057804eb39b4618b40e06196e16265b                             |
+-------------------+--------------------------------------------------------------+

After performing “neutron router-interface-add” for each subnet (again, see the SLAAC example), edits need to be made to the instance operating system for Stateless DHCPv6 operation. In the Ubuntu /etc/network/interfaces file, ensure the “inet6” line is set to “auto” and enable the stateless flag:

iface eth0 inet6 auto
dhcp 1

Again, I only tested CentOS7 and Fedora 21 with this setup so your mileage may vary but for those OSes, set the following for the appropriate interface in /etc/sysconfig/network-scripts/:

IPV6INIT="yes"
DHCPV6C="yes"
DHCPV6C_OPTIONS="-S"

Below is the output from tcpdump for the Stateless DHCPv6 exchange.  Radvd sends out the RS with the ICMPv6 Flags field set with the O bit set to 1 and the Prefix Information Flag field has “auto” enabled. This combination basically says “I, the router, will provide you, the instance,  a prefix via SLAAC and you come back asking for options”.  The instance will issue a “dhcp6 inf-req” to get the options and radvd will reply:

IP6 (hlim 255, next-header ICMPv6 (58) payload length: 56) fe80::f816:3eff:fec1:bc52 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 56
	hop limit 64, Flags [other stateful], pref medium, router lifetime 30s, reachable time 0s, retrans time 0s
	  prefix info option (3), length 32 (4): 2001:db8:cafe:2::/64, Flags [onlink, auto], valid time 86400s, pref. time 14400s
	    0x0000:  40c0 0001 5180 0000 3840 0000 0000 2001
	    0x0010:  0db8 cafe 0002 0000 0000 0000 0000
	  source link-address option (1), length 8 (1): fa:16:3e:c1:bc:52
	    0x0000:  fa16 3ec1 bc52

IP6 (hlim 1, next-header UDP (17) payload length: 44) fe80::f816:3eff:fefe:d157.546 > ff02::1:2.547: [udp sum ok] dhcp6 inf-req (xid=d2dbc8 (client-ID hwaddr type 1 fa163efed157) (option-request DNS-server DNS-search-list Client-FQDN SNTP-servers) (elapsed-time 94))

IP6 (class 0xc0, hlim 64, next-header UDP (17) payload length: 88) fe80::f816:3eff:fe2d:a6de.547 > fe80::f816:3eff:fefe:d157.546: [udp sum ok] dhcp6 reply (xid=d2dbc8 (client-ID hwaddr type 1 fa163efed157) (server-ID hwaddr type 1 fa163e2da6de) (DNS-search-list openstacklocal.) (DNS-server 2001:db8:cafe:a::e) (lifetime 86400))

I am sorry for the legnth of the blog post. In retrospect I should have broken this post into multiple posts but oh well.

I hope this was helpful.

Cheers,
Shannon

OpenStack – Juno Release – Layer 3 High-Availability

In the Juno release of OpenStack we were blessed with many new features and functionality. One of them is the Layer 3 (L3) High-Availability (HA) functionality. The L3 HA support that came in Juno uses Keepalived with Virtual Routing Redundancy Protocol (VRRP).

This blog is not meant to give you a full dump of Keepalived or VRRP. It’s not even meant to give you a deep dive on L3 HA. I will provide some explanation of my pictures and configs but you really should familiarize yourself with the original info on L3 HA:

Why do we need L3 HA in OpenStack?

Historically, there was not much in the way of providing a tenant a First Hop Redundancy Protocol (FHRP) for the tenant Neutron router/L3 agent.  If a tenant used a router for connectivity to other tenants or external access (Internet) and the node that housed the L3 agent died or the agent itself puked then the tenant was isolated with no connectivity in or out.  The one exception to this is if you did not use a tenant router and instead used the provider network model with VLANs and the first L3 hop for the tenant instance was a physical L3 device (like an aggregation layer L3 switch). In that case the FHRP (i.e. VRRP, HSRP, GLBP) being used between the redundant aggregation layer switches would provide your L3 HA capabilities.

So, we needed an answer for this L3 HA issue. In Juno, the L3 HA functionality was released so that we now had redundancy for the neutron router (L3 agent).

High-Level Considerations

There are a few things to keep in mind when using the L3 HA functionality:

      • L3 HA can be configured manually via the Neutron client by an admin:
        • neutron router-create --ha True|False
      • L3 HA functionality can be set as a system default within the /etc/neutron/neutron.conf and l3_agent.ini files (see examples later in the post)
      • Existing non-HA enabled router can be updated to HA:
        • neutron router-update <router-name> --ha=True
    • Requires a minimum of two network nodes (or controllers – each running the L3 agent)

What does the tenant see?

The tenant sees one router with a single gateway IP address. (Note:Non-Admin users cannot control if the router is HA or non-HA).  From the tenant’s perspective, the router behaves the same in HA or non-HA mode. In Figure 1 below, the tenant sees the instances, private network and a single router (the example below is from an admin user).

Figure 1. Tenant View of L3 HA-enabled Router

tenant-view

Routing View

In Figure 2 below, a basic view of the routing and VRRP setup is shown.  In this example the tenant network is assigned 10.10.30.x/24. VRRP is using 169.254.0.x over a dedicated HA-only network that traverses the same tenant network type (i.e VXLAN).  The router (L3 agent) on the left is the VRRP master and is the tenant gateway (10.10.30.1).

Tenant instances will use the 10.10.30.1 as their gateway and traffic northbound will pass through the master (the left router in this example). Return traffic will pass back through the router acting as a master.

Figure 2. Routing View of L3 HA

routing-view

Host View of L3 HA

In Figure 3 below,  there are three OpenStack nodes: A compute node, control node and a network node. (Note: In this example the control node is acting as a network node as well).  The L3 HA-enabled router has been created and there is a neutron router (L3 agent) running on both the control and network nodes. Keepalived/VRRP is enabled by the L3 HA code. In this example, br-int has a variety of ports connecting:

  • Port to br-tun (used for VXLAN)
  • Port to br-eth1 (used for eth1 connection)
  • Port to qrouter (qr-xxxx)
  • Port to keepalived (ha-xxxx)

The Neutron router (qrouter) in this example has the ports listed above for keepalived and br-int but also has a port to br-eth1 (via qg-xxxx).

The compute node has instances attached to br-int (via tap interfaces, veth pairs and linux bridge). br-int has a port connecting it to br-tun, again, used in this example for VXLAN.

Figure 3. Host View of L3 HA

host-view

Figure 4 shows a basic traffic flow example.  L3 HA uses keepalived/VRRPv2 to manage the master/backup relationship between the Neutron routers (L3 agents). VRRPv2 control traffic is using the 169.254.192.x network (configurable) and advertisements are sent using the well-known IPv4 multicast group of 224.0.0.18 (not configurable).

In this example, traffic leaving  an instance on the compute node will follow the path to its default gateway (via the VXLAN tunnels between br-tun on each node). The traffic flows through whichever L3 agent is acting as ‘master’. In this example the L3 agent is running on the control node.  The L3 agent will then route traffic towards the destination.

Figure 4. Traffic Flow Example for L3 HA

traffic-flow

Enabling L3 HA in Juno

On the node running Neutron server (controller), edit the /etc/neutron/neutron.conf file and uncomment/edit the following lines:

router_distributed = False
# =========== items for l3 extension ==============
# Enable high availability for virtual routers.
l3_ha = True
#
# Maximum number of l3 agents which a HA router will be scheduled on. If it
# is set to 0 the router will be scheduled on every agent.
max_l3_agents_per_router = 3
#
# Minimum number of l3 agents which a HA router will be scheduled on. The
# default value is 2.
min_l3_agents_per_router = 2
#
# CIDR of the administrative network if HA mode is enabled
l3_ha_net_cidr = 169.254.192.0/18

On the nodes running the L3 agent (in my example the control and network nodes), edit the /etc/neutron/l3_agent.ini file and uncomment/edit the following lines (Note: Set a better password than the one I’ve included ;-)):

# Location to store keepalived and all HA configurations
ha_confs_path = $state_path/ha_confs

# VRRP authentication type AH/PASS
ha_vrrp_auth_type = PASS

# VRRP authentication password
ha_vrrp_auth_password = cisco123

# The advertisement interval in seconds
ha_vrrp_advert_int = 2

Restart the L3 agent service on each node:

systemctl restart neutron-l3-agent.service

If you completed the above steps then you can create the neutron router without any ‘–ha’ flags set and the router will be created as an HA-enabled router. You can also create an HA-enabled router using the ‘–ha=True’ flag as shown in the following example (Note: Only admins have permissions to run with the –ha flag set):

[root@net1 ~]# neutron router-create --ha True test1
Created a new router:
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| admin_state_up        | True                                 |
| distributed           | False                                |
| external_gateway_info |                                      |
| ha                    | True                                 |
| id                    | 1fe9e406-2bb5-42c4-af62-3daef314e181 |
| name                  | test1                                |
| routes                |                                      |
| status                | ACTIVE                               |
| tenant_id             | 45e1c2a0b3a244a3a9fad48f67e28ef4     |
+-----------------------+--------------------------------------+

Once the router is created via the dashboard or via CLI (or via Heat) and your networks are all attached, you can look at the keepalived settings in the /var/lib/neutron/ha_confs/<id>/keepalived.conf file. In the example below, the “interface ha-0d655b16-c6” is the L3 HA interface. VRRP will track that inteface. The virtual IP (VIP) address is for the HA-xxx interface is 169.254.0.1 (only the master holds this address). The tenant-facing router IP (10.10.30.1) and the public-facing router IP (192.168.81.13) are the VIPs for each respective networks. The external default gateway is 192.168.81.2.

[root@net1 ~]# cat /var/lib/neutron/ha_confs/719b853f-539e-420b-a76b-0440146f05de/keepalived.conf
. . . output abbreviated 
vrrp_instance VR_1 {
    state BACKUP
    interface ha-0d655b16-c6
    virtual_router_id 1
    priority 50
    nopreempt
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass cisco123
    }
    track_interface {
        ha-0d655b16-c6
    }
    virtual_ipaddress {
        169.254.0.1/24 dev ha-0d655b16-c6
    }
    virtual_ipaddress_excluded {
        10.10.30.1/24 dev qr-c3090bd6-1b
        192.168.81.13/24 dev qg-4f163e63-c4
}
    virtual_routes {
        0.0.0.0/0 via 192.168.81.2 dev qg-4f163e63-c4
    }

Run a tcpdump on the L3 HA interface (inside the router namespace) to watch the VRRPv2 advertisements:

[root@net1 ~]# ip netns exec qrouter-719b853f-539e-420b-a76b-0440146f05de tcpdump -n -i ha-0d655b16-c6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ha-0d655b16-c6, link-type EN10MB (Ethernet), capture size 65535 bytes
14:00:03.123895 IP 169.254.192.33 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype simple, intvl 2s, length 20
14:00:05.125386 IP 169.254.192.33 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype simple, intvl 2s, length 20
14:00:07.128133 IP 169.254.192.33 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype simple, intvl 2s, length 20
14:00:09.129421 IP 169.254.192.33 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype simple, intvl 2s, length 20
14:00:11.130814 IP 169.254.192.33 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype simple, intvl 2s, length 20
14:00:13.131529 IP 169.254.192.33 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype simple, intvl 2s, length 20

Testing a failure
First, check who is master:

[root@net1 ~]# cat /var/lib/neutron/ha_confs/719b853f-539e-420b-a76b-0440146f05de/state
master
[root@net2 ~]# cat /var/lib/neutron/ha_confs/719b853f-539e-420b-a76b-0440146f05de/state
backup

Simulate a failure by shutting down the HA interface (remember that it was in the tracked interface list). Have a ping running on the instance to verify connectivity through the failure.:

[root@net1 ~]# ip netns exec qrouter-719b853f-539e-420b-a76b-0440146f05de ifconfig ha-0d655b16-c6 down

Check to see that the master role changed after failure:

[root@net1 ~]# cat /var/lib/neutron/ha_confs/719b853f-539e-420b-a76b-0440146f05de/state
fault
[root@net2 ~]# cat /var/lib/neutron/ha_confs/719b853f-539e-420b-a76b-0440146f05de/state
master

Checking in on the ping shows a delay but no loss (In a loaded system you will likely see a brief loss of traffic):

ubuntu@server1:~$ ping 8.8.8.8
64 bytes from 8.8.8.8: icmp_seq=20 ttl=127 time=65.4 ms
64 bytes from 8.8.8.8: icmp_seq=21 ttl=127 time=107 ms
64 bytes from 8.8.8.8: icmp_seq=22 ttl=127 time=64.5 ms
64 bytes from 8.8.8.8: icmp_seq=23 ttl=127 time=67.6 ms

Happy testing!