Cloud Networking

From CSCWiki
Jump to navigation Jump to search

This page gives an overview of the network topology of the CSC cloud.

For the sections below, ".csc" is an abbreviation for "".

Hostnames and IP addresses

Some important hostnames and IP addresses to know about:

  • VLAN 529 (, fd74:6b6a:8eca:4902::/64) is the cloud management network
  • VLAN 425 (, 2620:101:f000:4903::/64) is the cloud user network
  • Due to routing decisions, everything in VLAN 529 is accessible from VLAN 425, so make sure to use iptables if there's some service running in VLAN 529 which shouldn't be accessible by regular users. See the routers section below.
  • riboflavin, ginkgo and biloba are on VLAN 529 and VLAN 134
    • Each of these hosts has a ".cloud" domain record which resolves to their VLAN 529 address, e.g.
  • Most Ceph services are explicitly placed on riboflavin and ginkgo, since they have the most storage
    • and are CNAMES for riboflavin and ginkgo
  • biloba and chamomile are the two CloudStack management servers
  • (and point to a virtual IP address (in VLAN 529) shared by biloba and chamomile via keepalived
    • CloudStack keeps cookie sessions in-memory, so it is important that only one machine is the "active" CloudStack management server at any time (otherwise, if you connect to a different machine, your cookie session will suddenly be invalid)
    • points to this virtual IP address. We are using master-master replication between two MariaDB instances on biloba and chamomile. To avoid split-brain syndrome, it is very important that only instance gets written to at any time.
  • We are authoritative for the * and * DNS zones, so do not use Infoblox for these (except for PTR records)
  • All * records point to the virtual IP
  • Most * TLS certificates are managed with

Cloud routers

The two cloud routers are and They each have 3 IP addresses: one for each of VLANs 134, 425 and 529. resolves to the VLAN 529 IP, and resolves to the VLAN 134 IP (same for router2).

The two routers are also sharing the virtual IP addresses for (VLAN 529) and (VLAN 425) via keepalived. For incoming VLAN 425 traffic, the routers basically strip off the VLAN 425 header and replace it with a VLAN 529 header, meaning that VLAN 529 is fully accessible from VLAN 425. (This is actually useful for us since we can access Ceph storage from Kubernetes this way.) Use iptables where necessary.


The routers are systemd-nspawn containers running on biloba and chamomile. Standard container setup (FQDN, DNS, etc.) applies.

For reference, here's the .nspawn file for router1:


# These are manually bridged to the host; see
# /etc/systemd/system/systemd-nspawn@router1.service.d/override.conf

Here's the aforementioned systemd drop-in:

ExecStartPost=/usr/sbin/ip link set dev ve-router1-529 master br529
ExecStartPost=/usr/sbin/ip link set dev ve-router1-529 up
ExecStartPost=/usr/sbin/ip link set dev ve-router1-425 master br425
ExecStartPost=/usr/sbin/ip link set dev ve-router1-425 up

Here's the /etc/network/interfaces in router1:

auto host0
iface host0 inet static
iface host0 inet6 static
        pre-up sysctl net.ipv6.conf.host0.accept_ra=0
        pre-up sysctl net.ipv6.conf.host0.accept_dad=0
        address 2620:101:f000:4901:c5c::18/64
        gateway 2620:101:f000:4901::1

auto veth529
iface veth529 inet static
iface veth529 inet6 static
        pre-up sysctl net.ipv6.conf.veth529.accept_ra=0
        pre-up sysctl net.ipv6.conf.veth529.accept_dad=0
        address fd74:6b6a:8eca:4902::19/64

auto veth425
iface veth425 inet manual
iface veth425 inet6 manual
        pre-up sysctl net.ipv6.conf.veth425.accept_ra=0
        pre-up sysctl net.ipv6.conf.veth425.accept_dad=0

It is very important to disable IPv6 router advertisements. Note how veth425 doesn't have a dedicated IP address because it is provided by keepalived.

Here's the /etc/keepalived/keepalived.conf in router1:

vrrp_instance VI_1 {
  state MASTER
  interface veth529
  virtual_router_id 21
  priority 255
  advert_int 2
  authentication {
    auth_type PASS
    auth_pass ********
  virtual_ipaddress {
    # VLAN 529 router
    # VLAN 425 router dev veth425
  virtual_ipaddress_excluded {
    # VLAN 529 router
    # VLAN 425 router
    2620:101:f000:4903::254/64 dev veth425

Here's a snippet from /etc/sysctl.conf in router1:


Here are the iptables rules in router1:

iptables -t nat -N CLOUD-ROUTER
iptables -t nat -A POSTROUTING -j CLOUD-ROUTER
iptables -t nat -A CLOUD-ROUTER -d -j RETURN
iptables -t nat -A CLOUD-ROUTER -d -j RETURN
iptables -t nat -A CLOUD-ROUTER -d -j RETURN
iptables -t nat -A CLOUD-ROUTER -s -j SNAT --to-source
iptables -t nat -A CLOUD-ROUTER -s -j SNAT --to-source

Note that for on-campus addresses, forwarding is used instead of NAT.