Guide for using SNAT instances in Open Telekom Cloud

The Open Telekom Cloud (OTC) allows customers to associate elastic IPs (floating IPs) with vNICs (ports) on Virtual Machines (VMs) to expose them to the internet and support in- and outgoing connections.

Many VMs only serve internal purposes and thus do not need to be directly reachable from the internet; for some of them, the public services provided by OTC (such as DNS, NTP and update repository mirrors) may not be sufficient and outgoing internet access is still needed e.g to retrieve information or download software.

Assigning a public IP to each of these VMs would consume a scarce resource (IPv4 addresses are precious), incur additional cost and may increase the attack surface; instead sharing one public IP address with many VMs is preferable. This can be done using Source-NAT (SNAT).

Since early June 2017, OTC supports an SNAT platform service; if just outgoing traffic to the internet is required, this is the easiest and recommended solution. It will be described in the new first section of this document.

Since February 2017, OTC also supports the model of a routing/SNAT instance; here one VM is configured with a public IP address and serves as an SNATting router/gateway for a set of VMs from the same subnet or VPC. Setting this up requires a lot more skills and effort and comes with some security considerations; it was the only solution until June and the original description with some security considerations highlighted even stronger are described in the second section. That section describes how to set up and secure this SNAT instance and describes the settings needed to make this work, and to have other VMs use it, for outgoing internet access.

Since early June 2017, OTC offers an opt-in platform feature to support SNAT.

While setting up a pair of SNAT instances (as described in the second section below) offers a lot of flexibility (you can set up port forwarding, reverse proxys, WAFs, load balancers, JumpHosts, ... this way), just configuring EIP-less VMs to have outgoing internet access has become very straight-forward with the platform SNAT feature.

To opt-in to platform SNAT, just change the enable_snat field in the external_gateway_info of your VPC router to true.

This can be done using

neutron router-gateway-set --enable-snat VPC-ROUTER-ID admin_external_net

The disabling works the same way, using --disable-snat.

Note that the now recommended neutron tool version (from the OpenStack Newton release, 6.0.0) needs a backport to support snat enablement (disablement curiously works). The openSUSE42 and SLES12 public images carry a neutron client tool with this patch already; if yours does not, you can use newer version of python-neutronclient tools, or revert to using otc-tools which supports enabling/disabling SNAT as well.

For your reference, here is the five-line backport for neutron in case you want to patch the neutron client yourself.

# python-neutronclient assumes that snat is enabled by default, so it
# only supports disabling, but not enabling it explicitly.
# This has been fixed in latest upstream versions.
# This is my minimal version of this feature. -KG
Index: neutronclient/neutron/v2_0/router.py
===================================================================
--- neutronclient/neutron/v2_0/router.py.orig
+++ neutronclient/neutron/v2_0/router.py
@@ -235,6 +235,9 @@ class SetGatewayRouter(neutronV20.Neutro
            'external_network', metavar='EXTERNAL-NETWORK',
            help=_('ID or name of the external network for the gateway.'))
        parser.add_argument(
+ '--enable-snat', action='store_true',
+ help=_('Enable source NAT on the router gateway.'))
+ parser.add_argument(
            '--disable-snat', action='store_true',
            help=_('Disable source NAT on the router gateway.'))
        parser.add_argument(
@@ -258,6 +261,8 @@ class SetGatewayRouter(neutronV20.Neutro
        router_dict = {'network_id': _ext_net_id}
        if parsed_args.disable_snat:
            router_dict['enable_snat'] = False
+ if parsed_args.enable_snat:
+ router_dict['enable_snat'] = True
        if parsed_args.fixed_ip:
            ips = []
            for ip_spec in parsed_args.fixed_ip:

Note that the platform SNAT feature can not yet be enabled using the OTC Web interface; you need to do this via API calls. The API interface for this setting is the same as in other OpenStack clouds; many of them however default to SNAT being enabled, so you always get internet access as soon as you configure an external gateway net in your router; instead in OTC the external router is always configured (to make floating IPs work without any further ado), but the SNAT feature is disabled by default.

If you only need SNAT, it is recommended to use the above described SNAT platform feature. Nevertheless, by using an SNAT instance, you can implement advanced routing features and policies, such as e.g. setting up openVPN tunnels and defaulting to routing packets via them. The below description covers the basic functions needed to just set up SNAT -- it was created when the platform SNAT feature was not yet available. It is still useful as a reference that you can use to base more advanced setups upon. If you do this, please keep in mind the security implications of allowing all addresses (allowed-address-pairs setting) to be forwarded via some ports -- the security group containing these ports will allow everything in ...

Many operating systems support routing packets. What we need is the operating system to rewrite the source address of the packets to its own address before forwarding them, so they are sent out with a public sender address and the return packets can find their way back. This method is called source network address translation (SNAT). The operating system needs to keep track of the SNATted packets, so when the answers are returned, the destination address can be rewritten (DNAT) and the packet be forwarded to the initial sender. We will quickly outline how this can be achieved on Linux.

Basically, you enable IP forwarding sysctl -w net.ipv4.ip_forward=1 and set up the SNAT rule iptables -t nat -A POSTROUTING -j SNAT -o eth0 -s NET/PREFLEN --to OWNIP. The process is explained in detail in chapter 3.10 of the VPC user docs of the OTC documentation.

The OpenStack networking component neutron does not normally allow a VM to send packets with a sender IP address that differs from its own address. This increases security by protecting against spoofed packets. For our SNAT scenario however, when the returning packet is forwarded from the SNAT instance to the VM originating the connection, neutron would block this, as the SNAT instance forwards a packet with a source IP address it does not own.

neutron can be instructed to disable this protection. The Web Interface ("Service Console") supports this by using the "Unbind IP from MAC" option to the virtual NIC.

neutron port-update PORTID --allowed-address-pairs type=dict list=true ip_address=0.0.0.0/1 ip_address=128.0.0.0/1

This will tell neutron to allow packets with any sender addresses being sent from this port.¹

The SNAT instance has an external IP address; thus it is fully exposed to the internet. It is advisable to protect this instance well, which includes both network protection mechanisms (firewall rules or at least closing open ports), avoiding ssh password authentication (this is switched off on all public images in OTC by default) and diligently keeping up with security updates.

So you really should make the SNAT instance carefully protect itself. As the same interface is used for the internal connection to the subnet as well as to the internet, filtering rules can not be set up based on the network interface name which make the network protection a bit trickier.

If the availability of your application depends on the outgoing internet access, you probably want to ensure it continues to work in case a SNAT instance fails. Designing for failure of a complete availability zone (AZ) allows very good availability in a cloud.

This section describes how to use a virtual IP and a pair of two VMs to provide a highly available SNAT setup.

Here we create a network and subnet that spans both AZs for the two SNAT instances and put one in each AZ. On the Web Interface you can allocate Virtual IPs in the VPC / subnet menu, Manage Private IP Address.

You would normally have to allow this address on the SNAT-instance ports (Bind private IP); however, we have already allowed all addresses before to make SNAT work, so no extra work is needed.

The only missing piece is to make your SNAT instances use the virtual IP address by logging into the VMs and issuing the command ip addr add VIP/32 dev eth0.

You can fully automate this, the custom cloud-init user_data feature offers to use the otc.addip setting for this.

# Assumptions: VPC-ROUTER exists, $SNATSG contains security-group ID
getid() { FIELD=${1:-id}; grep "^| $FIELD " | sed -e 's/^|[^|]*| \([^|]*\) |.*$/\1/' -e 's/ *$//'; }
listid() { grep $1 | tail -n1 | sed 's/^| \([0-9a-f-]*\) .*$/\1/'; }
SNATNET=$(neutron net-create SNAT-NET | getid)
SNATSUB=$(neutron subnet-create --dns-nameserver 100.125.4.25 --dns-nameserver 8.8.8.8 --name SNAT-SUBNET SNAT-NET 172.16.0.0/24 | getid)
neutron router-interface-add VPC-ROUTER $SNATSUB
neutron port-create --name SNAT-VIP --security-group SNAT-SG-DANGERZONE --fixed-ip subnet_id=$SNATSUB,ip_address=172.16.0.99
cat > user_data.yaml <<EOT
#cloud-config
otc:
   internalnet:
      - 172.16/12
   snat:
      masq:
         - INTERNALNET
   addip:
      eth0: 172.16.0.99
   autoupdate:
      frequency: daily
      categories: security recommended
EOT
IMGID=$(glance image-list | listid 'Standard_openSUSE_42_JeOS_latest')
nova boot --image $IMGID --flavor computev1-1 --key-name SSHkey-SNAT --user-data user_data.yaml --availability-zone eu-de-01 --security-groups $SNATSG --nic net-id=$SNATNET,v4-fixed-ip=172.16.0.4 SNAT-INST1
nova boot --image $IMGID --flavor computev1-1 --key-name SSHkey-SNAT --user-data user_data.yaml --availability-zone eu-de-02 --security-groups $SNATSG --nic net-id=$SNATNET,v4-fixed-ip=172.16.0.5 SNAT-INST2
SNAT_INST1_PORT=$(neutron port-list | grep 172.16.0.4 | listid $SNATSUB)
neutron port-update $SNAT_INST1_PORT --allowed-address-pairs type=dict list=true ip_address=0.0.0.0/1 ip_address=128.0.0.0/1
neutron floatingip-create --port-id $SNAT_INST1_PORT admin_external_net
SNAT_INST2_PORT=$(neutron port-list | grep 172.16.0.5 | listid $SNATSUB)
neutron port-update $SNAT_INST2_PORT --allowed-address-pairs type=dict list=true ip_address=0.0.0.0/1 ip_address=128.0.0.0/1
neutron floatingip-create --port-id $SNAT_INST2_PORT admin_external_net
neutron router-update VPC-ROUTER --routes type=dict list=true destination=0.0.0.0/0,nexthop=172.16.0.99

The above code assumes that your VPC Router carries the name VPC-ROUTER, that a security group with suitable rules for the SNAT VMs exists with the name SNAT-SG-DANGERZONE and that there is a ssh keypair SSHkey-SNAT. It will masquerade outgoing traffic from the 172.16/12 range, so this assumes all the other subnets on the VPC router, needing SNAT, use addresses in this address range. For simplicity it also just uses fixed IPs, network resources have been referenced by name (instead of their resource UUIDs), two little helper functions are used and no error handling is done. Except for these details, this is the recommended configuration.

Note the naming of SNAT-SG-DANGERZONE -- this security group does contain ports that have --allowed-address-pairs set to the internet. Thus do not allow packets from this security-group anywhere if you don't want to open up to the world.

A complete code example can be found in config_snat.sh.

Note that the otc.addip setting does not do a plain ip addr add $VIP/32 dev $DEV on both instances. Though this appears to work, it relies on the router ARP cache keeping the routing stable and not alternate between the two instances without need. While this would appear to work on our tests, this could not be guaranteed to always work in the future; the otc.addip setting thus starts a service (snat_addip.sh) that monitors availability of the internet connectivity and the virtual IP address availability and ensure that only one instance claims the virtual IP address. If one instance loses connectivity, a failover takes place.

Routing via the SNAT instance

OTC 2.0 introduces a VPC-routing feature which allows outgoing traffic originating from ports without an EIP to be routed via the SNAT instance.

This can be set in the Web Interface; see the ch. 3.9 of the VPC user manual on the OTC documentation.

We have included the command to do this via API in the above code example already:

neutron router-update VPC-ROUTER --routes type=dict list=true destination=0.0.0.0/0,nexthop=172.16.0.99 

Using this VPC routing feature, no changes to any VM routing configuration will be needed. VMs will talk to their neighbours (in the same subnet) directly as before and use the VPC router as default gateway for everything else. The router will still know all the connected subnets (connected via neutron router-interface-add or via WebInterface) as well as the peered VPCs, VPN connections and the provider network (which hosts the public services such as DNS, NTP and repository mirrors) and forward packets accordinglyly. And it will still send packets directly to the internet from ports (vNICs) that have a floating IP (EIP) associated. The VPC route configured for 0.0.0.0/0 has the lowest precendence and will only be used when none of the above conditions are met, which is exactly what you want for providing an SNAT service.

Currently, the users do not have the freedom to have more finegrained control over the routing table used on the VPC router. The setup with a default route however matches the common use cases; whether it is an SNAT instance that routes outgoing traffic to the internet or other use cases, such as an openVPN endpoint that connects remote networks

Prior to OTC 2.0, there was no way to configure the VPC router to specify routing via an SNAT gateway. So to use an SNAT instance, you had to change the route table on all VMs that needed outbound Internet access without having their own EIP.Assuming the SNAT gateway has IP 192.168.16.4 in a /22 network, the routing table could look like this:

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.16.4 0.0.0.0 UG 0 0 0 eth0
10.0.0.0 192.168.16.1 255.0.0.0 UG 0 0 0 eth0
100.64.0.0 192.168.16.1 255.192.0.0 UG 0 0 0 eth0
172.16.0.0 192.168.16.1 255.240.0.0 UG 0 0 0 eth0
192.168.0.0 192.168.16.1 255.255.0.0 UG 0 0 0 eth0
169.254.169.254 192.168.16.1 255.255.255.255 UGH 0 0 0 eth0
192.168.16.0 * 255.255.252.0 U 0 0 0 eth0

Note that packets to the local networks are all sent to the default gateway, while the rest are sent to the SNAT gateway.This is a bit complex; for hosts in other subnets, you would need to configure a host route to the SNAT gateway via the VPC router (.1) before being able to add default routes via the SNAT instance. It's not practical to manage such route tables for a large number of hosts, unless you play DHCP tricks. If you want to go down that route, you would create ports (vNICs) and tweak them with extra_dhcp_options prior to booting VMs. Setting up your own DHCP server is possible, but even more troublesome, as your IPs are then no longer managed in OpenStack which creates challenges for allowed-address-pairs, security-groups and the simple ability to find out IPs for a VM.

In the preconfigured setup, we used only one vNIC for the SNAT instance; traditional network people would have used two interfaces, one for the internal connection and one that gets the EIP attached. As the IP address unbinding only needs to be done on the internal network interface, the security group protection towards the outside world would remain effective. Also, as most Linux distribution firewall scripts assume multiple interfaces for multiple zones, it would be easier to express correct filtering policies.On the other hand, both vNIC's subnets would need to be connected to the VPC router, so the network separation remains incomplete. For this reason, the simple setup with one vNIC has been prepared using the customer otc.snat user_data settings. The script could be extended and tested to also support multi-vNIC setups, but this is not currently the case. Talk to us if you have a need ...

As NTP and DNS and package mirrors are covered by services in the OTC public service zone, the need for outgoing internet access might not be that large. Rather than having a full-blown SNAT service, you could set up an http/https or socks proxy server. Many applications do support the http_proxy and HTTPS_PROXY environment variables and would use any proxy configured this way. If sufficient interest is communicated to us, we could provide an easy way to configure squid with a few lines of user_data to provide such a service.

1 The simpler ip_address=0.0.0.0/0 setting does not work on Open Telekom Cloud.