HTTPS ingress with nginx + kube-lego#
Kubernetes Ingress Objects are used to manage HTTP(S) access from the internet to inside the Kubernetes cluster. Among other things, it lets us do the following:
Provide a HTTPS end point so users can connect to us securely
Direct traffic to various Services based on hostnames or URL paths
Allow using one public IP address for multiple domain names
We use the nginx-ingress provider to handle our Ingress needs.
We run on Google Cloud’s Kubernetes Engine. Even though GKE comes pre-installed with the Google Cloud Load Balancer Ingress provider, we decided to use nginx instead for the following reasons:
GCLB has a 30s default timeout on all HTTP connections. This is counted not just when connection is idle, but from connection start time. This is particularly bad for websockets, since those connections usually last for a lot longer than 30s! There is no easy way to configure this timeout from inside Kubernetes.
GCLB does not guarantee you can use the same IP for multiple domains. We want the various subdomains of
mybinder.orgto point to the same IP so we can easily add / remove new services without waiting for DNS propagation delay.
nginx-ingress is installed using the nginx-ingress helm chart. This installs the following components:
nginx-ingress-controller- keeps the HTTPS rules in sync with
Ingressobjects and serves the HTTPS requests. This also exports metrics that are captured in prometheus.
nginx-ingress-default-backend- simply returns a 404 error & is used by
nginx-ingress-controllerto serve any requests that don’t match any rules.
The specific ways these have been configured can be seen in the
file in this repo, under
Configuration with Ingress objects#
Ingress objects are used to tell the ingress controllers which requests
should be routed to which
Service objects. Usually, the rules either
check for a hostname (like
a URL prefix (like
/docs). You can see all the ingress objects
kubectl --namespace=prod get ingress.
The following ingress objects currently exist:
jupyterhub- Directs traffic to
hub.mybinder.org. The zero-to-jupyterhub guide has more documentation.
binderhub- Directs traffic to
mybinder.org. You can find more details about this in the binderhub helm chart.
redirector- Directs traffic to the HTTP redirector we run for
mybinder.org. This helps do redirects such as
beta.mybinder.org. The list of redirects is configured in
mybinder/values.yaml. The code for this is in
mybinder/templates/redirectorin this repo.
static- Directs traffic into
static.mybinder.org. We serve the
mybinder.orgbadges from a different domain for privacy reasons. This ingress lets us direct traffic only from
static.mybinder.org/badge.svgto the binder pod.
prometheus-server- Directs traffic to
prometheus.mybinder.org. Configured under
grafana- Directs traffic to
grafana.mybinder.org. Configured under
kube-lego-nginx- Used by kube-lego for doing automatic HTTPS certificate renewals.
HTTPS certificates with kube-lego#
Kube-lego is deprecated, and we should move to
cert-manager <https://github.com/jetstack/cert-manager/>_ soon.
kube-lego is installed using the kube-lego.
kube-lego requires Ingress objects to have specific
tls values, as documented here.
We specify this for all our ingress objects, mostly by customizing various helm charts
Let’s Encrypt account#
Let’s Encrypt uses accounts
to keep track of HTTPS certificates & expiry dates.
Currently, the account is registered to
email@example.com, mostly as a historical
accident. Changing it requires some amount of care to make sure we do not suffer
intermittent HTTPS failure, and should be done whenever we switch to cert-manager.