Category Archives: Uncategorized

We Don’t Need No Stinking NAT – OpenStack Address Scopes

I have heard from many OpenStack operators over the last few years and they are looking for a way to get rid of the mandatory implementation of IPv4 NAT with OpenStack. Historically, the way to achieve this was to deploy provider networks with VLANs, attach instances directly to these provider networks and just route out through the Data Center without any need/requirement of NAT until the traffic reached the Internet Edge.

In the Mitaka release of OpenStack, Address Scopes and Subnet Pools were introduced.  One method of disabling IPv4 NAT is to match address scopes on the external and internal sides of the Neutron router (in a future post I will talk about disabling SNAT as another option).  This is done by creating an address scope, creating subnet pools that use that address scope and then creating networks/subnets from those subnet pools.  If the external and internal (tenant) subnets use the same ‘scope’ then NAT is disabled in the Neutron L3 component.  More can be found here: OpenStack Docs: Address scopes

A quick test reveals some issues with this as a mainstream method of removing IPv4 NAT from a cloud deployment:

  • Address scope and subnet pool use are mandatory
  • You cannot manually assign a subnet allocation pool range OR a custom gateway (“Bad subnets request: allocation_pools allowed only for specific subnet requests..”). This is a major issue when you need to define a custom upstream (provider) gateway address. Note: I resolved this by editing the subnet by hand (before creating any routers) and changing the allocation range and changing the default gateway IP address.
  • You have to sub-allocate prefixes/CIDR ranges out of the subnet pool which is typically something the cloud/network operator would do upstream at the DC Agg layer, routed access or Border Leaf

Test scenario:

  • All-in-One RDO VM (Newton) on VMware Fusion
  • Eth1<>br-ex
  • Public Network: 192.168.81.0/24
  • One Address Scope
  • Two Subnet Pools out of the 192.168.81.0/24 range, divided as /26s

Create an Address Scope:

[root@c7-n-aio ~]# neutron address-scope-create --shared test-no-nat 4
Created a new address_scope:
+------------+--------------------------------------+
| Field      | Value                                |
+------------+--------------------------------------+
| id         | 6f055310-6225-403c-a14d-acd723757869 |
| ip_version | 4                                    |
| name       | test-no-nat                          |
| project_id | 67ddc0b4822f4b67b3e4a449dbbe65bc     |
| shared     | True                                 |
| tenant_id  | 67ddc0b4822f4b67b3e4a449dbbe65bc     |
+------------+--------------------------------------+

Create a new subnet pool using the new address scope:

[root@c7-n-aio ~]# neutron subnetpool-create --address-scope test-no-nat --shared --pool-prefix 192.168.81.0/24 --default-prefixlen 26 nonat-pool-ip4
Created a new subnetpool:
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| address_scope_id  | 6f055310-6225-403c-a14d-acd723757869 |
| created_at        | 2016-11-04T19:59:10Z                 |
| default_prefixlen | 26                                   |
| default_quota     |                                      |
| description       |                                      |
| id                | 964ae834-9391-4aa2-bc7b-b1c197d4d4c5 |
| ip_version        | 4                                    |
| is_default        | False                                |
| max_prefixlen     | 32                                   |
| min_prefixlen     | 8                                    |
| name              | nonat-pool-ip4                       |
| prefixes          | 192.168.81.0/24                      |
| project_id        | 67ddc0b4822f4b67b3e4a449dbbe65bc     |
| revision_number   | 1                                    |
| shared            | True                                 |
| tenant_id         | 67ddc0b4822f4b67b3e4a449dbbe65bc     |
| updated_at        | 2016-11-04T19:59:10Z                 |
+-------------------+--------------------------------------+

Create a new network for the Public/External side:

[root@c7-n-aio ~]# neutron net-create Public-Net --provider:network_type vxlan --shared --router:external
Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2016-11-04T19:59:21Z                 |
| description               |                                      |
| id                        | 4d68a85a-eac9-44a3-84b7-eec3b731d97a |
| ipv4_address_scope        |                                      |
| ipv6_address_scope        |                                      |
| is_default                | False                                |
| mtu                       | 1450                                 |
| name                      | Public-Net                           |
| project_id                | 67ddc0b4822f4b67b3e4a449dbbe65bc     |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 71                                   |
| revision_number           | 2                                    |
| router:external           | True                                 |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| tenant_id                 | 67ddc0b4822f4b67b3e4a449dbbe65bc     |
| updated_at                | 2016-11-04T19:59:21Z                 |
+---------------------------+--------------------------------------+

Create a new subnet for the public/external network and reference the new subnet pool:
#Note: This is where you cannot dictate the ‘gateway_ip’ value manually. If you attempt to you either get an error (see the output in my list of issues in the beginning of the doc) or it ignores it (not consistent).

[root@c7-n-aio ~]# neutron subnet-create Public-Net --name Public-Subnet --subnetpool nonat-pool-ip4 --dns-nameserver 8.8.8.8
Created a new subnet:
+-------------------+---------------------------------------------------+
| Field             | Value                                             |
+-------------------+---------------------------------------------------+
| allocation_pools  | {"start": "192.168.81.2", "end": "192.168.81.62"} |
| cidr              | 192.168.81.0/26                                   |
| created_at        | 2016-11-04T20:00:44Z                              |
| description       |                                                   |
| dns_nameservers   | 8.8.8.8                                           |
| enable_dhcp       | True                                              |
| gateway_ip        | 192.168.81.1                                      |
| host_routes       |                                                   |
| id                | 30c3f341-ad77-4f27-ae55-d469d8004266              |
| ip_version        | 4                                                 |
| ipv6_address_mode |                                                   |
| ipv6_ra_mode      |                                                   |
| name              | Public-Subnet                                     |
| network_id        | 4d68a85a-eac9-44a3-84b7-eec3b731d97a              |
| project_id        | 67ddc0b4822f4b67b3e4a449dbbe65bc                  |
| revision_number   | 2                                                 |
| service_types     |                                                   |
| subnetpool_id     | 964ae834-9391-4aa2-bc7b-b1c197d4d4c5              |
| tenant_id         | 67ddc0b4822f4b67b3e4a449dbbe65bc                  |
| updated_at        | 2016-11-04T20:00:44Z                              |
+-------------------+---------------------------------------------------+

Create a Tenant network:

[root@c7-n-aio ~]# neutron net-create nonat-net-1
Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2016-11-04T20:02:24Z                 |
| description               |                                      |
| id                        | e5ae8309-beb8-4bcd-a9ee-87be39004eff |
| ipv4_address_scope        |                                      |
| ipv6_address_scope        |                                      |
| mtu                       | 1450                                 |
| name                      | nonat-net-1                          |
| project_id                | 67ddc0b4822f4b67b3e4a449dbbe65bc     |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 32                                   |
| revision_number           | 2                                    |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| tenant_id                 | 67ddc0b4822f4b67b3e4a449dbbe65bc     |
| updated_at                | 2016-11-04T20:02:24Z                 |
+---------------------------+--------------------------------------+

Create a new subnet for the new Tenant network and use the new subnet pool:

[root@c7-n-aio ~]# neutron subnet-create --name nonat-subnet-1 --subnetpool nonat-pool-ip4 nonat-net-1
Created a new subnet:
+-------------------+-----------------------------------------------------+
| Field             | Value                                               |
+-------------------+-----------------------------------------------------+
| allocation_pools  | {"start": "192.168.81.66", "end": "192.168.81.126"} |
| cidr              | 192.168.81.64/26                                    |
| created_at        | 2016-11-04T20:02:43Z                                |
| description       |                                                     |
| dns_nameservers   |                                                     |
| enable_dhcp       | True                                                |
| gateway_ip        | 192.168.81.65                                       |
| host_routes       |                                                     |
| id                | f24de73f-8ff7-48df-a5e9-c2d5d43bb154                |
| ip_version        | 4                                                   |
| ipv6_address_mode |                                                     |
| ipv6_ra_mode      |                                                     |
| name              | nonat-subnet-1                                      |
| network_id        | e5ae8309-beb8-4bcd-a9ee-87be39004eff                |
| project_id        | 67ddc0b4822f4b67b3e4a449dbbe65bc                    |
| revision_number   | 2                                                   |
| service_types     |                                                     |
| subnetpool_id     | 964ae834-9391-4aa2-bc7b-b1c197d4d4c5                |
| tenant_id         | 67ddc0b4822f4b67b3e4a449dbbe65bc                    |
| updated_at        | 2016-11-04T20:02:43Z                                |
+-------------------+-----------------------------------------------------+

Create a new router:

[root@c7-n-aio ~]# neutron router-create no-nat-router
Created a new router:
+-------------------------+--------------------------------------+
| Field                   | Value                                |
+-------------------------+--------------------------------------+
| admin_state_up          | True                                 |
| availability_zone_hints |                                      |
| availability_zones      |                                      |
| created_at              | 2016-11-04T20:02:54Z                 |
| description             |                                      |
| distributed             | False                                |
| external_gateway_info   |                                      |
| flavor_id               |                                      |
| ha                      | True                                 |
| id                      | 31a09399-fd79-4d0f-8354-f700393a8138 |
| name                    | no-nat-router                        |
| project_id              | 67ddc0b4822f4b67b3e4a449dbbe65bc     |
| revision_number         | 2                                    |
| routes                  |                                      |
| status                  | ACTIVE                               |
| tenant_id               | 67ddc0b4822f4b67b3e4a449dbbe65bc     |
| updated_at              | 2016-11-04T20:02:54Z                 |
+-------------------------+--------------------------------------+

Link the router interfaces and gateway to the new networks:

[root@c7-n-aio ~]# neutron router-interface-add no-nat-router nonat-subnet-1
Added interface b27f6296-9353-4b4a-b96e-55c88ba97177 to router no-nat-router.

[root@c7-n-aio ~]# neutron router-gateway-set no-nat-router Public-Net
Set gateway for router no-nat-router

Boot an instance and attach it to the new “nonat-net-1” network which is using subnet pool “nonet-pool-ip4”:

[root@c7-n-aio ~]# nova boot --flavor m1.tiny --image cirros --key-name aio-key --security-groups default --nic net-name=nonat-net-1 nonat-vm
+--------------------------------------+-----------------------------------------------+
| Property                             | Value                                         |
+--------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                        |
| OS-EXT-AZ:availability_zone          |                                               |
| OS-EXT-SRV-ATTR:host                 | -                                             |
| OS-EXT-SRV-ATTR:hostname             | nonat-vm                                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                             |
| OS-EXT-SRV-ATTR:instance_name        |                                               |
| OS-EXT-SRV-ATTR:kernel_id            |                                               |
| OS-EXT-SRV-ATTR:launch_index         | 0                                             |
| OS-EXT-SRV-ATTR:ramdisk_id           |                                               |
| OS-EXT-SRV-ATTR:reservation_id       | r-rwlgkfoi                                    |
| OS-EXT-SRV-ATTR:root_device_name     | -                                             |
| OS-EXT-SRV-ATTR:user_data            | -                                             |
| OS-EXT-STS:power_state               | 0                                             |
| OS-EXT-STS:task_state                | scheduling                                    |
| OS-EXT-STS:vm_state                  | building                                      |
| OS-SRV-USG:launched_at               | -                                             |
| OS-SRV-USG:terminated_at             | -                                             |
| accessIPv4                           |                                               |
| accessIPv6                           |                                               |
| adminPass                            | 5efCnNeo7P5a                                  |
| config_drive                         |                                               |
| created                              | 2016-11-04T20:06:50Z                          |
| description                          | -                                             |
| flavor                               | m1.tiny (1)                                   |
| hostId                               |                                               |
| host_status                          |                                               |
| id                                   | 6f788d79-6b4b-4d58-a0be-84856de5207e          |
| image                                | cirros (403de225-2186-4fb0-8b19-020c57300fb7) |
| key_name                             | aio-key                                       |
| locked                               | False                                         |
| metadata                             | {}                                            |
| name                                 | nonat-vm                                      |
| os-extended-volumes:volumes_attached | []                                            |
| progress                             | 0                                             |
| security_groups                      | default                                       |
| status                               | BUILD                                         |
| tags                                 | []                                            |
| tenant_id                            | 67ddc0b4822f4b67b3e4a449dbbe65bc              |
| updated                              | 2016-11-04T20:06:50Z                          |
| user_id                              | cab9fe684b0a4ddc9ba0f57acafeb2bb              |
+--------------------------------------+-----------------------------------------------+

Grab reference information for the instance and the router:

[root@c7-n-aio ~]# nova list
+--------------------------------------+----------+--------+------------+-------------+---------------------------+
| ID                                   | Name     | Status | Task State | Power State | Networks                  |
+--------------------------------------+----------+--------+------------+-------------+---------------------------+
| 6f788d79-6b4b-4d58-a0be-84856de5207e | nonat-vm | ACTIVE | -          | Running     | nonat-net-1=192.168.81.74 |
+--------------------------------------+----------+--------+------------+-------------+---------------------------+
[root@c7-n-aio ~]# neutron router-list
+--------------------------------------+----------------+-----------------------------------------------+-------------+------+
| id                                   | name           | external_gateway_info                         | distributed | ha   |
+--------------------------------------+----------------+-----------------------------------------------+-------------+------+
| 31a09399-fd79-4d0f-8354-f700393a8138 | no-nat-router  | {"network_id": "4d68a85a-                     | False       | True |
|                                      |                | eac9-44a3-84b7-eec3b731d97a", "enable_snat":  |             |      |
|                                      |                | true, "external_fixed_ips": [{"subnet_id":    |             |      |
|                                      |                | "30c3f341-ad77-4f27-ae55-d469d8004266",       |             |      |
|                                      |                | "ip_address": "192.168.81.12"}]}              |             |      |
| da5fc542-9028-4315-b66f-13a11d8ef87d | Private-Router | null                                          | False       | True |
+--------------------------------------+----------------+-----------------------------------------------+-------------+------+

From an outside (external to OpenStack) host, ping the IPv4 address of the router interface (it works):

# ping 192.168.81.12
PING 192.168.81.12 (192.168.81.12): 56 data bytes
64 bytes from 192.168.81.12: icmp_seq=0 ttl=64 time=1.523 ms
64 bytes from 192.168.81.12: icmp_seq=1 ttl=64 time=0.232 ms

From an outside (external to OpenStack host, ping the IPv4 address of the instance (it doesn’t work):

# ping 192.168.81.74
PING 192.168.81.74 (192.168.81.74): 56 data bytes
Request timeout for icmp_seq 0

Add a route on the network gear or the host pointing to the Tenant subnet via the Neutron router IP (in this example it is my Mac which is hosting the VMware Fusion VMs this is running on):

# sudo route -n add 192.168.81.64/26 192.168.81.12
add net 192.168.81.64: gateway 192.168.81.12

Ping the instance from external host. It works without a floating IP and no NAT:

# ping 192.168.81.74
PING 192.168.81.74 (192.168.81.74): 56 data bytes
64 bytes from 192.168.81.74: icmp_seq=0 ttl=63 time=1.101 ms
64 bytes from 192.168.81.74: icmp_seq=1 ttl=63 time=1.058 ms

 

 

Docker Registry with IPv6

If you have been following along, you know that I started a series of posts aimed at identifying IPv6 support for the various Docker components/services.

The first blog post was focused on Docker Engine, which has pretty reasonable support for basic IPv6.

The second blog post was focused on Docker Hub, which has zero IPv6 support. This is due to it being hosted on AWS and no IPv6-enabled front-end is deployed.

This blog post will focus on Docker Registry.

As I stated in the past two blog entries, I am not here to teach you Docker (what it is, how to deploy it, etc..). I am simply showing basic functionality of various Docker components/services when used with IPv6.

For information on setting up your own Docker Registry, check out:

https://docs.docker.com/registry/

I am using Docker version 1.8.3, Docker Compose version 1.4.2 and Docker Registry version 2.

I am using the same Ubuntu 14.04.3 hosts that I have used in the last two blog posts.

My setup uses two hosts with the following configuration:

  • docker-v6-1:
    • Role: Docker Registry
    • IPv6 Address: fd15:4ba5:5a2b:1009:20c:29ff:fef3:f848/64
  • docker-v6-2:
    • Role: Docker Host/Client
    • IPv6 Address: fd15:4ba5:5a2b:1009:20c:29ff:febb:cbf8/64

My Docker Registry (running on ‘docker-v6-1’) uses self-signed cert and is started using either the ‘docker run’ syntax or Docker Compose. I show both examples below:

docker run:

docker run -d -p 5000:5000 --restart=always --name registry \
  -v `pwd`/certs:/certs \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  registry:2

Use Docker Compose to run Docker Registry
I am using a file named “docker-compose.yml” to launch my registry.

registry:
  restart: always
  image: registry:2
  ports:
    - 5000:5000
  environment:
    REGISTRY_HTTP_TLS_CERTIFICATE: /certs/domain.crt
    REGISTRY_HTTP_TLS_KEY: /certs/domain.key
  volumes:
    - /certs:/certs

Run Docker Compose:

docker-compose up -d

Verify Connectivity
On the Docker host/client (“docker-v6-2”),  verify that the Docker Registry host (“docker-v6-1”) can be reached over IPv6:

root@docker-v6-2:~# ping6 -n docker-v6-1.example.com
PING docker-v6-1.example.com(fd15:4ba5:5a2b:1009:20c:29ff:fef3:f848) 56 data bytes
64 bytes from fd15:4ba5:5a2b:1009:20c:29ff:fef3:f848: icmp_seq=1 ttl=64 time=0.402 ms
64 bytes from fd15:4ba5:5a2b:1009:20c:29ff:fef3:f848: icmp_seq=2 ttl=64 time=0.367 ms

Docker Registry Push/Pull Verification
Now that connectivity to the Docker Registry host is working, tag a local Docker image and then push it (over IPv6) to the Docker Registry:

root@docker-v6-2:~# docker tag ubuntu docker-v6-1.example.com:5000/ubuntu

root@docker-v6-2:~# docker push docker-v6-1.example.com:5000/ubuntu
The push refers to a repository [docker-v6-1.example.com:5000/ubuntu] (len: 1)
a005e6b7dd01: Image successfully pushed
002fa881df8a: Image successfully pushed
66395c31eb82: Image successfully pushed
0105f98ced6d: Image successfully pushed
latest: digest: sha256:167f1c34ead8f1779db7827a55de0d517b7f0e015d8f08cf032c7e5cd6979a84 size: 6800

A tcpdump on the Docker Registry shows traffic between docker-v6-1 and docker-v6-2 for the ‘push’ using the previously defined port 5000:

root@docker-v6-1:~# tcpdump -n -vvv ip6 -i eth0
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
19:36:09.283820 IP6 (hlim 64, next-header TCP (6) payload length: 40) fd15:4ba5:5a2b:1009:20c:29ff:febb:cbf8.56066 > fd15:4ba5:5a2b:1009:20c:29ff:fef3:f848.5000: Flags [S], cksum 0x65b1 (correct), seq 2754754540, win 28800, options [mss 1440,sackOK,TS val 645579 ecr 0,nop,wscale 7], length 0
19:36:09.283930 IP6 (hlim 64, next-header TCP (6) payload length: 40) fd15:4ba5:5a2b:1009:20c:29ff:fef3:f848.5000 > fd15:4ba5:5a2b:1009:6540:bb36:2e23:f5a2.56066: Flags [S.], cksum 0xcd92 (incorrect -> 0x50bd), seq 1577491031, ack 2754754541, win 28560, options [mss 1440,sackOK,TS val 859496 ecr 645579,nop,wscale 7], length 0

It works!

Happy Dockering,

Shannon

What you can expect from this site

I am horrible at blogging.  It takes me months to login to a blog and type anything. Much of what you will read here is drivel and nonsense. You’ve been warned.

When I do have something to say it will likely be about OpenStack, Docker, IPv6, general DC networking or whatever I happen to be interested at that moment.  I will likely be posting stuff that is not ready for primetime but yet I got it working. Don’t mistake ping testing for production-ready. I am usually too far out on the edge for anything I do to be considered production ready. 😉

Be patient, be gentle and hopefully I will have something interesting to post.