Cross-Realm traffic guidelines

From Wikitech
POLICY UPDATES: this page contains a policy, non-trivial updates to this document should be agreed upon
Following this policy doesn't automatically grant approval. Please consult with both network realm owners (SRE and WMCS) before implementation for any case.

This page contains information related to the production and cloud public network & realms relationship, policies, guidelines and best practices for planning new services and for when reworking old ones.

The production networks (production services) and WMCS networks are different realms in terms of users, scope, security, access, maintenance, etc. It is undesirable (and rejected) to add new connections or bridges between the 2 realms if not following the guidelines presented in this document. However, implementing new connections between the 2 realms should still be carefully considered. This document does not convey permission to implement any new service.

However, the situation is not that simple. As of this writing, the hardware that creates the CloudVPS service itself (running openstack components, etc) lives in the production network. These cloud hardware servers are managed like any other production service (ssh, puppet, monitoring, cumin, install, etc).

Also, there could be several currently-running services that don't follow the pattern presented here. These will be reviewed on a case-by-case basis.

Case 1: general case

A service to be offered only to CloudVPS VM instances should run inside CloudVPS.

Network flows originate and terminate inside the CloudVPS virtual network. There are no cross-realm connections.

The upstream openstack project refers to this kind of use cases as east-west network traffic.

requirements

  • Nothing special other than the usual, capacity on CloudVPS, etc.

example: toolsdb

The toolsdb service was previously running on a hardware server in the production private network. Network connections from clients to the database were crossing realms.

It was eventually moved inside CloudVPS. A VM instance was created to offer the exact same service, without any need to cross realms.

example: cloudinfra

The cloudinfra openstack project was created to host several cloud-specific services for VMs, such as shared NTP, MX and puppetmaster services.


Case 2: generic network access cloud --> prod

A CloudVPS VM instance uses the cloud edge network to contact a production service.

The network flows through both the production network backbone (core routers, etc) and the CloudVPS edge network (neutron, cloudsw, etc) and is seen like any other internet peer from the production service point of view. Therefore, the 2 realms are not actually bridged directly.

The upstream openstack project refers to this kind of use cases as north-south network traffic.

requirements

This use case must meet the following:

  • The production service uses a public IPv4 address.
  • The client CloudVPS VM instance use either the general NAT egress address or a floating IP. The cloud private VM address doesn't reach the production service.
  • The production service might be only meant for CloudVPS. But it uses a public IPv4 address, so firewalling should exists to only allow incoming connections from CloudVPS (from the general NAT egress address).
  • It has been evaluated if the service could be moved inside the cloud instead (i.e, general case)

Additional considerations:

  • the production service is likely behind LVS, as is the standard way for public production services to be offered to the internet.

example: dumps

The dumps service uses a public IPv4 address. CloudVPS client VM instances accessing the dumps service do so as any other internet client.

The dumps NFS service, however, is not offered to the whole internet, but just to CloudVPS clients (and Analytics systems). This is controlled by using firewalling policies.

The dumps service doesn't need to know the client VM private address, so VMs use the general cloud egress NAT to access the service.

example: wikis

CloudVPS client VM instances access wikis like any other internet peer.

(TODO: we hope this sentence will be true in the future, but it is not as of this writing, given they skip the general egress NAT, see News/CloudVPS_NAT_wikis).


Case 3: generic network access prod --> cloud

This case is when a CloudVPS VM is offering a service that is meant to be used from the production network.

Note that there may be other considerations (e.g. reliability, security, privacy) that may or may not make the data flow desirable. The flow should be considered on its own merits, and with the appropriate maintainers of the production service as well as the WMCS tenant. The guideline here should not be interpreted as blanket approval for all production -> WMCS data flows, but rather as a technical implementation detail if and when such flows are agreed upon.

The network connections flow through both the production network backbone (core routers, etc) and the CloudVPS edge network (neutron, cloudsw, etc) and is seen like any other internet peer from the CloudVPS VM instance point of view. Therefore, the 2 realms are not actually bridged directly.

The upstream openstack project refers to this kind of use cases as north-south network traffic.

requirements

This use case must meet the following:

  • if the service is HTTP/HTTPS, then the VM instance uses a web proxy created in horizon.
  • if the service is not HTTP/HTTPS, then the VM instance uses a floating IP address.
  • if only production is intended to consume the service, neutron security groups should be used to firewall the access.

Both the production service and CloudVPS use public IPv4 addresses, and thus each internal private addresses aren't seen in either side of the connection.

example: puppet catalog compiler

The puppet catalog compiler (PCC) is a service that runs natively in CloudVPS. It is consumed by some production services (jenkins, gerrit, etc). It uses a web proxy created using horizon (https://puppet-compiler.wmflabs.org/)

example: SAL

The Server Admin Log tool runs in Toolforge and is widely used by production. There is, however, nothing special in this tool to restrict access to production.


Case 4: cloud-dedicated hardware

This case covers an architecture in which a service is offered to/from the cloud realm using hardware servers instead of software defined (virtual) components. The hardware components are part of the cloud realm even thought they are connected to wikiland networks for ssh and management purposes.

Most of the other models in this document cover either north-south native services or extremely simple east-west native services that can easily run inside the cloud. However, we often find ourselves in a dead end when trying to adapt east-west native services that cannot run inside the cloud to a north-south approach covered by other models in this document. In general, we find it challenging to adopt others models when a given service involves a combination of:

  • heavy data replication or storage requirements (in the TB range)
  • extremely granular networking access (such as per-VM file locking or similar)
  • extremely granular policing (such as firewalling, rate-limiting or similar)
  • chicken-egg problems (running a mission-critical cloud component inside the cloud itself)
  • other complex architectures that require dedicated hardware.

So, basically, this case covers a way to have logical/physical components that are dedicated to CloudVPS, are considered to be part of the cloud realm, and therefore aren't natively part of the wikiland production realm except for ssh and management purposes.

The upstream openstack project refers to this kind of use cases with a variety of names, depending on the actual implementation details:

  • provider hardware, hardware in service of openstack software-defined virtual machine clients.
  • external network, external to the openstack software-defined realm. Also mentioned in some documents as provider networks.
  • basically, an east-west service running on hardware, without being virtual/sofware-defined inside the cloud itself.

requirements

This use case must meet the following:

  • there is no reasonable way the service can be covered by the other models described in this document.
  • the hardware server must be connected to the cloud-private subnet/VLAN, meaning their NIC should be trunked (native for ssh management in cloud-hosts-xyz and tagged for cloud-private-xyz).
  • all network traffic for cloud-realm-related services circulates using the cloud-private-xyz subnet and not cloud-hosts-xyz.
  • the management traffic that can circulate using cloud-hosts-xyz includes all usual wikiland production facilities such as puppetmasters, monitoring, LDAP, etc. Formal wikis endpoints are explicitly excluded from using this subnet.
  • in case the service needs some kind of backend data connection to a production service, such connection will use the normal network border crossing, with egress NAT on the cloud realm side and standard service mechanisms (like LVS) on production realm side.
  • CloudVPS VMs clients accessing the services may or may not use NAT to access them. The NAT is optional, and can be evaluated on a case by case basis.
  • if the service shall be accessed from the internet or from wikiland production services, such exposure will be using a public IPv4 address from the cloud-realm pool with associated DNS entries in the wikimediacloud.org domain.
  • moreover, to avoid wasting public IPv4 addresses, the new service should be behind an abstraction/load-balancing layer that is dedicated to the cloud-realm.

example: openstack native services

Openstack native services, such as the REST APIs or rabbitmq run in dedicated hardware servers with this setup, and in particular:

  • openstack REST APIs are behind HAproxy in cloudlb servers, which host the public IPv4 address for them
  • rabbitmq is contacted by both the openstack REST APIs and VMs (Openstack Trove).


See also

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Cross-Realm_traffic_guidelines&oldid=2115271"