Portal:Toolforge/Admin/Kubernetes/Networking and ingress
< Portal:Toolforge‎ | Admin‎ | Kubernetes
This page describes the design, topology and setup of Toolforge's networking and ingress, specifically those bits related to webservices in the new kubernetes cluster.
Network topology
The following diagram is used to better describe the network topology and the different components involved.
When an user visits a webservice hosted in Toolforge, the following happens:
  1. nginx+lua: The first proxys is what we know as dynamicproxy. The nginx+lua setup knows how to send requests to the legacy webservices grid. There is a fall-through route for the k8s cluster.
    This proxy provides SSL termination for toolforge.org. There is a DNS A record pointing to the floating IP associated with the active proxy VM.
  2. haproxy: Then, haproxy knows which actual ingress worker nodes are alive in the k8s cluster. Proxy for both the k8s api-server on tcp/6443 and the web ingress on tcp/30000.
    There is a DNS A record with name k8s.tools.eqiad1.wikimedia.cloud pointing to the IP of this VM.
  3. nginx-ingress-svc: There is a nginx-ingress service of type NodePort, which means every ingress worker node listens on tcp/30000 and direct request to the nginx-ingress pod.
  4. nginx-ingress pod: The nginx-ingress pod will use ingress objects to direct the request to the appropriate service, but ingress objects need to exists beforehand.
  5. api-server: Ingress objects are created using the k8s API served by the api-server. They are automatically created using the webservice command, and the k8s API allows users to create and customize them too.
  6. ingress-admission-controller​: There is a custom admission controller webhook that validates ingress config objects to enforce valid configurations.
  7. ingress object: After the webhook, the ingress objects are added to the cluster and the nginx-ingress pod can consume them.
  8. tool svc: The ingress object contained a mapping between URL/path and a service. The request is now in this tool specific service, which knows how to finally direct the query to the actual tool pod.
  9. tool pod: The request finally arrives at the actual tool pod.
Components
This section contains specific information about the different components involved in this setup.
There are mainly 2 different kinds of elements: those running outside kubernetes and those running inside.
outside kubernetes
Information about components running outside kubernetes.
dynamicproxy
See Toolforge dynamicproxy
This component is described in another wikitech page.
haproxy
This setup is fairly simple, deployed by puppet using the role role::wmcs::toolforge::k8s::haproxy​.
We have a DNS name k8s.tools.eqiad1.wikimedia.cloud with an A record pointing to the active VM. There should be a couple of VMs in a cold-standby setup (only one is actually working).
The haproxy configuration involves 2 ports, the "virtual servers":
Each "virtual server" has several backends:
inside kubernetes
Explanation of the different components inside kubernetes.
calico
We use calico as the network overlay inside kubernetes. There is not a lot to say here, since we mostly use the default calico configuration. We only specify the CIDR of the pod network. There is a single yaml file containing all the configuration, and deployed by puppet.
This file is modules/toolforge/templates/k8s/calico.yaml.erb in the puppet tree and /etc/kubernetes/calico.yaml in the final control nodes.
To load (or refresh) the configuration inside the cluster, use:
root@k8s-control-1:~# kubectl apply -f /etc/kubernetes/calico.yaml configmap/calico-config unchanged​customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org unchanged​customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org unchanged​customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org unchanged​customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org unchanged​customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org unchanged​customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org unchanged​customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org unchanged​customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org unchanged​customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org unchanged​customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org unchanged​customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org unchanged​customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org unchanged​customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org unchanged​customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org unchanged​clusterrole.rbac.authorization.k8s.io/calico-kube-controllers unchanged​clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged​clusterrole.rbac.authorization.k8s.io/calico-node unchanged​clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged​daemonset.apps/calico-node configured​serviceaccount/calico-node unchanged​deployment.apps/calico-kube-controllers unchanged​serviceaccount/calico-kube-controllers unchanged
Some things to take into account:
nginx-ingress
We use nginx to process the ingress configuration in the k8s cluster.
Worth mentioning that we are using the comunity-based one (​https://kubernetes.github.io/ingress-nginx/​) rather than the NGINX Inc. one (​https://github.com/nginxinc/kubernetes-ingress​).
Starting from ingress-nginx 0.46.0 we deploy it using Helm. Helm is installed on the control plane nodes and uses the current users kubeconfig file to authenticate.
Helm needs a values file that it uses to customize the deployment for us. This file is modules/toolforge/files/k8s/nginx-ingress-helm.yaml in the puppet tree and /etc/kubernetes/nginx-ingress-helm.yaml in the final control nodes.
On the initial deployment, create the namespace and deploy it using Helm:
root@toolsbeta-test-k8s-control-4:~# helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx "ingress-nginx" has been added to your repositories​root@toolsbeta-test-k8s-control-4:~# helm repo update Hang tight while we grab the latest from your chart repositories...​...Successfully got an update from the "ingress-nginx" chart repositoryUpdate Complete. ⎈Happy Helming!⎈​root@toolsbeta-test-k8s-control-4:~# kubectl create ns ingress-nginx-gen2 namespace/ingress-nginx-gen2 created​root@toolsbeta-test-k8s-control-4:~# helm install -n ingress-nginx-gen2 ingress-nginx-gen2 ingress-nginx/ingress-nginx --values /etc/kubernetes/nginx-ingress-helm-values.yaml coalesce.go:200: warning: cannot overwrite table with non table for extraArgs (map[])NAME: ingress-nginx-gen2​LAST DEPLOYED: Mon May 10 12:02:17 2021NAMESPACE: ingress-nginx-gen2​STATUS: deployedREVISION: 1TEST SUITE: NoneNOTES:The ingress-nginx controller has been installed.[... instructions how to create an ingress object cut ...]
To update it based on the values file, simply run:
root@toolsbeta-test-k8s-control-4:~# helm upgrade -n ingress-nginx-gen2 ingress-nginx-gen2 ingress-nginx/ingress-nginx --values /etc/kubernetes/nginx-ingress-helm-values.yaml Release "ingress-nginx-gen2" has been upgraded. Happy Helming!NAME: ingress-nginx-gen2​LAST DEPLOYED: Mon May 10 12:08:25 2021NAMESPACE: ingress-nginx-gen2​STATUS: deployedREVISION: 3TEST SUITE: NoneNOTES:The ingress-nginx controller has been installed.[... instructions how to create an ingress object cut ...]
Run helm repo update beforehand if you're updating the controller (and not just the values file).
error handling
We have a tool called fourohfour which is set as the default backend for nginx-ingress. This tool presents an user friendly 404 page.
ingress objects
Ingress objects can be created in 2 ways:
Objects have this layout. Toolforge tool name fourohfour is used as example:
apiVersion: extensions/v1beta1​kind​: Ingressmetadata: annotations: nginx.ingress.kubernetes.io/configuration-snippet​: | rewrite ^(/fourohfour)$ $1/ redirect; nginx.ingress.kubernetes.io/rewrite-target​: /fourohfour/$2 labels: name: fourohfour toolforge: tool tools.wmflabs.org/webservice​: "true" tools.wmflabs.org/webservice-version​: "1" name: fourohfour namespace: tool-fourohfourspec: rules: - host: toolsbeta.wmflabs.org http: paths: - backend: serviceName: fourohfour servicePort: 8000 path: /fourohfour(/|$)(.*)​status​: loadBalancer: {}
NOTE: the rewrite/redirect configuration is really important and is part of the behaviour users expect. See phabricator ticket T242719 for example.
This ingress object is pointing to a service which should have this layout:
apiVersion: v1kind: Servicemetadata: labels: name: fourohfour toolforge: tool tools.wmflabs.org/webservice​: "true" tools.wmflabs.org/webservice-version​: "1" name: fourohfour namespace: tool-fourohfourspec: clusterIP: x.x.x.x ports: - name: http port: 8000 protocol: TCP targetPort: 8000 selector: name: fourohfour sessionAffinity: None type: ClusterIPstatus: loadBalancer: {}
Note that both objects are namespaced to the concrete tool namespace.
ingress admission controller
This k8s API webhook checks ingress objects before they are accepted by the API itself. It enforces (or prevents) ingress configurations that may produce malfunctioning in the webservices running in kubernetes, like pointing URLs/path to tools that are not ready to handle them.
The code is written in golang, and can be found here:
How to test the setup
See logs for nginx-ingress:
root@tools-k8s-control-3:~# kubectl logs -lapp.kubernetes.io/name​=​ingress-nginx -n ingress-nginx 192.168.50.0 - [192.168.50.0] - - [18/Dec/2019:12:02:21 +0000] "GET /potd-feed/potd.php/commons/potd-400x300.rss HTTP/1.1" 503 2261 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" 402 0.003 [upstream-default-backend] [] 192.168.15.133:8000 2261 0.000 503 2d08390256b1629bc60552710e4b47e1​192.168.34.128 - [192.168.34.128] - - [18/Dec/2019:12:02:22 +0000] "GET /favicon.ico HTTP/1.1" 404 1093 "https://tools.wmflabs.org/wikisense/Gallery.php?&wikifam=commons.wikimedia.org&img_user_text=Hoanglong2807" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) coc_coc_browser/83.0.144 Chrome/77.0.3865.144 Safari/537.36" 576 0.529 [upstream-default-backend] [] 192.168.15.133:8000 2116 0.528 404 5fc83ada234e3a5a5da5e42a6583f992​192.168.15.128 - [192.168.15.128] - - [18/Dec/2019:12:02:23 +0000] "GET /mediawiki-feeds/ HTTP/1.1" 503 2236 "https://tools.wmflabs.org/mediawiki-feeds/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)" 487 0.003 [upstream-default-backend] [] 192.168.15.133:8000 2236 0.004 503 e0341d943aba393e30fc692151da0790​[..]
TODO: extend this. Perhaps refactor into its own wiki page?
See also
Last edited on 30 June 2021, at 16:10
Wikitech
Content is available under CC BY-SA 3.0 unless otherwise noted.
Privacy policy
Terms of Use
Desktop
 Home Random Log in  Settings  Donate  About Wikitech  Disclaimers
WatchEdit