Skip to content

Jenkins

Jenkins is the most popular and mature open source tool for CI/CD and automation, and it's also usable for automating Data Science and Machine Learning workflows.

DagsHub has an official Jenkins plugin that you can use to automatically scan your DagsHub repos, and execute custom pipelines on each:

  • Git branch push
  • Git tag creation
  • Pull request

How to install Jenkins plugin for DagsHub?

Just go to your Jenkins' plugin management UI, and search for DagsHub. The plugin should be listed as "Available", and you can just install it.

To use it:

  1. Create a new Multibranch Pipeline Project
  2. Select DagsHub as a branch source
    DagsHub branch srouce
  3. (Optional) If the repo is private, you need to supply Jenkins with credentials to access it. We suggest using an access token.
  4. Input your repo URL, e.g.: https://dagshub.com/username/reponame
  5. Select the behaviors you want - whether to build on all branch pushes, only pull requests, etc. All behaviors supported by the DagsHub plugin start with the word (DagsHub) for your convenience.

DagsHub branch source settings

How to automatically trigger builds using webhooks?

The DagsHub plugin currently doesn't automatically install any webhook on the repo for you. This means new builds won't run after pushing branches etc. until you manually trigger a branch scan in the Jenkins project.

To solve this, you can:

  1. Install the Multibranch Scan Webhook Trigger Jenkins plugin
  2. Configure the "Scan Multibranch Pipeline Triggers" - activate the "Scan by webhook" option
  3. Choose a secret token
    Jenkins multibranch webhook trigger
  4. Copy the URL: https://YOUR-JENKINS-SERVER/multibranch-webhook-trigger/invoke?token=[YOUR-TRIGGER-TOKEN], replace the server address and trigger token.
  5. Go to your DagsHub repo's webhook settings screen, e.g. https://dagshub.com/username/reponame/settings/hooks
  6. Click on "Add Webhook" then choose the "DagsHub" hook type
    DagsHub webhook creation
  7. Define the Jenkins webhook URL from the previous steps, and chosoe the event types which should trigger it. The recommended set is marked in the following screenshot:
    DagsHub webhook definition
  8. Click on "Add Webhook".
    Now, whenever a branch or pull request is created/updated/deleted, Jenkins will receive a notification, rescan your project and trigger any required builds.

Jenkins Examples

If you're looking for examples on how to create ML automations with Jenkins, check out this article series written by one of our community members:

  • Part 1 is a high level overview of how Jenkins can be used in an ML project setting.
  • Part 2 is an in depth guide into creating an ML project with Jenkins, where models are automatically trained when new code versions are pushed to DagsHub.