How to deploy a change to mybinder.org?#

This document explains how to deploy a change to mybinder.org. For information on what exactly a deployment does, see what.

When a new change has landed in either BinderHub or repo2docker, they need to be explicitly deployed to mybinder.org for users to benefit from them. This is the most common kind of change deployed to mybinder.org.

Upgrades to BinderHub and repo2docker are automatically managed by the Watch Dependencies GitHub workflow.

Follow the instructions below if you need to manually update BinderHub or Repo2docker.

Deployment policy#

Deployments to mybinder.org should be:

  1. Safe. We will have good, user friendly tooling + lots of safeguards, so people can deploy without fear of accidentally breaking the site.

  2. Straightforward. We want a lot of people to be involved in maintaining mybinder.org, so we must make sure deployments are also easy to do. Most deployments should not require specific sysadmin knowledge.

  3. Timely. We deploy changes to repo2docker and BinderHub within a few days of them being merged into main.

These are all aspirational - we strive for hitting the above points, but recognize that work and life may get in the way of doing this perfectly.

Updating BinderHub#

This section explains how to upgrade the mybinder.org deployment after merging a PR in the BinderHub repo.

BinderHub, the Helm chart, is a dependency for the local mybinder Helm chart. The version of the BinderHub Helm chart is declared in mybinder/Chart.yaml. Upgrading the version of BinderHub that is used in mybinder.org corresponds to updating the BinderHub Helm chart version, which we step through below.

  1. Merge changes to BinderHub.

  2. Wait for the Publish helm chart and docker images workflow on the main branch to complete successfully.

  3. Lookup the latest BinderHub chart dev version on https://hub.jupyter.org/helm-chart/#development-releases-binderhub

  4. In your fork of the mybinder.org-deploy repository, open mybinder/Chart.yaml and change version in the binderhub section of dependencies to the latest BinderHub chart dev version.

  5. Open a pull request to merge this change into the main branch of the mybinder.org-deploy repository, following the steps in Deploying a change.

Updating repo2docker#

This section explains how to upgrade the mybinder.org deployment after merging a PR in the repo2docker repo.

BinderHub uses a docker image with repo2docker in it. When a new commit is merged in the repo2docker repository, a new version of this image is pushed. We then configure BinderHub to use the newly built image (which is identified by a tag) by editing values.yaml.

  1. Merge changes to repo2docker.

  2. Wait for the Publish helm chart and docker images workflow on the main branch to complete successfully.

  3. Lookup the latest Repo2docker tag on https://quay.io/repository/jupyterhub/repo2docker?tab=tags

  4. In your fork of the mybinder.org-deploy repository, open mybinder/values.yaml and change the tag in binderhub.config.KubernetesBuildExecutor.build_image to the latest tag.

  5. Open a pull request to merge this change into the main branch of the mybinder.org-deploy repository, following the steps in Deploying a change.

Deploying a change#

Deploying to both staging then prod#

Deploying a change involves making a PR with your desired change and merging it to main.

  1. Make the changes as described above on your fork of this repo.

  2. Keep track of the hashes that were updated. You should have both the old hash that was replaced, and the new hash that replaced it.

  3. If you haven’t already, run the list_new_commits.py script in the scripts/ folder. This will print out a URL that describes the changes made to both BinderHub and repo2docker.

  4. Make a PR to the main branch with the changes you want.

    • Name the PR like so: <TOOL-CHANGED>: <OLD-HASH>...<NEW-HASH>

    • In the description of the PR, paste the full URL that you printed out list_new_commits.py. It should have the following form:

      https://github.com/jupyterhub/<REPO-NAME>/compare/<OLD-HASH>...<NEW-HASH>
      
  5. Review, accept, and merge this PR. This will make GitHub Actions deploy the changes to staging.mybinder.org, and run tests in the tests/ directory against it. In this case, you can merge your own PR. Note that if the PR is a large change to the Kubernetes setup, this may take some time, and GitHub Actions may time-out in the process. If this happens and you expect it to happen, you can restart the build a few times.

  6. If the tests succeed, the change will be deployed to mybinder.org.

  7. If the tests fail, the change will not be deployed to mybinder.org. You must then investigate why it failed. If you can not figure out a cause in about 10 minutes, revert the change. You can revert the change with the GitHub UI and immediately merge the reversion PR that GitHub creates.

  8. Troubleshoot and make changes to your fork. Repeat the process from Step 1.

Deploying to only staging#

Sometimes you want to test out a deployment live before you make a deployment to prod. There are a few ways we can achieve this.

Testing Pull Requests from branches of mybinder.org-deploy#

This simplest way to deploy a PR to staging only is to apply the test-staging label to an open PR. This will trigger GitHub Actions to deploy the changes in the PR to the staging cluster only.

Note

If you need to re-deploy the changes in a PR to staging only, then the label will need to be removed and then re-added.

Testing Pull Requests from forks of mybinder.org-deploy#

If the PR has been made from a fork of the repo, the labelling approach discussed in the previous section will fail due to a lack of access to secrets. In this scenario, a user with OWNER, COLLABORATOR or MEMBER association with the mybinder.org-deploy repo can leave a /test-this-pr comment on the PR to trigger a deploy to staging.

Editing staging config directly#

The final option to deploy to staging only is by editing staging-only config files. To deploy to staging only, follow these steps:

  1. Make changes to config/staging.yaml on your fork. This file contains configuration for Helm that will override whatever is in mybinder/values.yaml.

  2. Make a PR to the main branch, and review, accept, and merge this PR. In this case, you can merge your own PR. This will make GitHub Actions deploy the changes to staging.mybinder.org, and run tests in the tests/ directory against it. Because we’ve only edited staging.yaml, it will not be deployed to prod.

  3. If the tests succeed, you can check out the new behavior at staging.mybinder.org.

  4. If the tests fail, the deployer must investigate why it failed. If they can not figure out a cause in about 10 minutes, revert the change. The build should not remain broken for more than ten minutes.

  5. Troubleshoot and make changes to your fork. Repeat the process from Step 1.

  6. If you are satisfied with these changes, revert the change to config/staging.yaml, and apply those same changes to mybinder/values.yaml. Now follow the steps in the section above to deploy to both staging and prod.

The what document has more details on common ways deployments can go wrong, and how to debug them.

Changing the mybinder.org infrastructure#

Sometimes we need to make changes to the mybinder.org core infrastructure. These are changes to the infrastructure that don’t directly touch binderhub or repo2docker, and often require more expertise. Examples for these include:

  1. Upgrading nginx Ingress controller

  2. Re-configuring our prometheus servers

  3. Upgrading to a new JupyterHub release

  4. Re-configuring autoscaling for the cluster

  5. Doing a kubernetes master upgrade.

These changes require a different kind of review than deploying code. In this case, ensure that you have a fellow member of the mybinder.org operations team to assist in case something goes wrong.