Kubernetes Ingress Objects
are used to manage HTTP(S) access from the internet to inside the Kubernetes cluster.
Among other things, it lets us do the following:
Provide a HTTPS end point so users can connect to us securely
Direct traffic to various Services based on hostnames or URL paths
Allow using one public IP address for multiple domain names
We use the nginx-ingress provider to handle
our Ingress needs.
We run on Google Cloud’s Kubernetes Engine. Even though GKE comes pre-installed with
the Google Cloud Load Balancer Ingress provider,
we decided to use nginx instead for the following reasons:
GCLB has a 30s default timeout on all HTTP connections. This is counted
not just when connection is idle, but from connection start time. This
is particularly bad for websockets, since those connections usually last for
a lot longer than 30s! There is no easy way to configure this timeout from
GCLB does not guarantee you can use the same IP for multiple domains. We
want the various subdomains of mybinder.org to point to the same IP
so we can easily add / remove new services without waiting for DNS propagation
nginx-ingress is installed using the nginx-ingress helm chart.
This installs the following components:
nginx-ingress-controller - keeps the HTTPS rules in sync with Ingress
objects and serves the HTTPS requests. This also exports
metrics that are captured in prometheus.
nginx-ingress-default-backend - simply returns a 404 error & is used
by nginx-ingress-controller to serve any requests that don’t match
The specific ways these have been configured can be seen in the mybinder/values.yaml
file in this repo, under nginx-ingress.
Ingress objects are used to tell the ingress controllers which requests
should be routed to which Service objects. Usually, the rules either
check for a hostname (like mybinder.org or prometheus.mybinder.org) and/or
a URL prefix (like /metrics or /docs). You can see all the ingress objects
present with kubectl --namespace=prod get ingress.
kubectl --namespace=prod get ingress
The following ingress objects currently exist:
jupyterhub - Directs traffic to hub.mybinder.org.
The zero-to-jupyterhub guide has more documentation.
binderhub - Directs traffic to mybinder.org. You can find more details
about this in the binderhub helm chart.
redirector - Directs traffic to the HTTP redirector we run for mybinder.org.
This helps do redirects such as docs.mybinder.org or beta.mybinder.org.
The list of redirects is configured in mybinder/values.yaml. The code
for this is in mybinder/templates/redirector in this repo.
static - Directs traffic into static.mybinder.org. We serve the mybinder.org
badges from a different domain for privacy reasons.
This ingress lets us direct traffic only from static.mybinder.org/badge.svg to the
prometheus-server - Directs traffic to prometheus.mybinder.org. Configured under
prometheus in both mybinder/values.yaml and config/prod.yaml.
grafana - Directs traffic to grafana.mybinder.org. Configured under grafana in
both mybinder/values.yaml and config/prod.yaml.
kube-lego-nginx - Used by kube-lego for doing automatic
HTTPS certificate renewals.
We use Let’s Encrypt for all our HTTPS certificates.
Kube Lego is used to automatically
provision and maintain HTTPS certificates for us.
Kube-lego is deprecated, and we should move to
kube-lego is installed using the kube-lego.
kube-lego requires Ingress objects to have specific annotations and
tls values, as documented here.
We specify this for all our ingress objects, mostly by customizing various helm charts
Let’s Encrypt uses accounts
to keep track of HTTPS certificates & expiry dates.
Currently, the account is registered to email@example.com, mostly as a historical
accident. Changing it requires some amount of care to make sure we do not suffer
intermittent HTTPS failure, and should be done whenever we switch to cert-manager.