Radicle CI Integrations

Radicle CI Integrations

  • Status: Open
  • Proposer: @gsaslis
  • Your Project(s): [optional]: Radicle CI Broker and CI Integrations
  • Projects you think this work could be useful for [optional]: unblock migration to Radicle, as CI is a critical missing feature.
  • Category: 3rd Party Integration

Overview :telescope:

As part of some work funded by the Radicle Core team, me and @nvourlakis have already laid out the foundations for a new component (the Radicle CI Broker) that can be deployed next to a Radicle Node, in order to integrate with external systems. We have implemented the first integration with a CI server (Concourse) and are now seeking additional funding, in order to:

  • implement integration with GitHub Actions
  • implement integrations with 2 more open source CI servers:
    • Woodpecker
    • Kraken CI
  • pave the way for additional integrations with 3rd party systems (e.g. zulip, slack, etc.) by adding the ability to Radicle to send generic Webhook Event notifications to other systems.

These integrations will aim to complement the Radicle-Native CI that the Radicle forge will soon offer, in presenting a decent range of options to maintainers, so they can migrate their projects to Radicle, together with their CI workloads! That is: the integrations to existing CI solutions allow maintainers to continue using their existing CI and break down their project hosting migration to smaller, independent, steps.

Description :page_facing_up:

This project involves work in several different areas, and so will be split into multiple Milestones.

  • Milestone 1: Required functionality on Radicle side
  • Milestone 2: Woodpecker Integration
  • Milestone 3: Kraken CI Integration
  • Milestone 4: GitHub Actions Integration

Milestone 1 will serve as the necessary building block for the remaining 3 milestones, so we will focus on that first. Work on the other 3 can then proceed in parallel.

Milestone 1: Required Functionality on Radicle side

  • Total Estimated Duration: 1 calendar month
  • Full-time equivalent (FTE): 18.75 FTE days
  • Team:
    • Nikolas Vourlakis
    • Ioannis Christodoulou
    • Yorgos Saslis
Number Deliverable Specification
1. Radicle Webhooks Event Schema Before we start shipping Radicle Events as HTTP requests to 3rd party systems (“Webhook Events”), it is important to define the API contract (i.e. the request headers, the JSON body schema, validation rules and some examples). This can help 3rd party systems start working on the integration on their side, while we work on the implementation of the Webhook Events themselves (see below deliverables). This deliverable will involve studying the GitHub, GitLab and Gitea webhook events to inform the design of Radicle’s own Webhook Events schema.
2. Long-Lived Access Tokens support We have already started a discussion with the radicle-httpd maintainers about the possibility of adding longer-lived Access Tokens as an authentication mechanism for HTTP requests to the Radicle API. This deliverable will cover the implementation we will propose, as a Radicle Patch to the Heartwood repo.
3. Extend CI Broker to send Radicle Webhook Events This deliverable will cover the necessary changes and improvements to the Radicle CI Broker, so that it can read rad node events, “translate” those to higher level Webhook Events (starting with push, patch created, patch updated and patch merged) and then forward those as HTTP requests to the preconfigured webhooks for the specific repository. As part of this deliverable, we also need to extend the broker’s configuration so that it can read which URLs to send webhook events to, for each repository, taking into account that these URLs may need to contain secret tokens (for authentication/authorization).
4. More deployment options for CI Broker As part of this Milestone, we will provide Container packaging, Docker Compose, Ansible playbook, Kubernetes manifets, a Helm chart and systemd service files for CI Broker deployments (and integration with the Radicle Node).
5. Documentation and how-to guide Provide detailed, step-by-step guides for the deployment options and for how to enable the Radicle Webhook Events. Also how to create/delete the long-lived access tokens.

Milestone 2: Woodpecker Integration

  • Total Estimated Duration: 2 calendar months
  • Full-time equivalent (FTE): 27 FTE days
  • Team:
    • Michalis Zampetakis
    • Ioannis Christodoulou
    • Yorgos Saslis

We have already began a discussion with Woodpecker maintainers about the possibility of incorporating Radicle into Woodpecker, as a new forge.

With their input and guidance, we will proceed in opening a (series of PRs) implementing the below functionality:

Number Deliverable Specification
1. Forge scaffold Create the radicle’s forge scaffold in woodpecker-ci.
2. Enable Radicle Forge in Woodpecker Allows Woodpecker users to browse Radicle projects, by connecting to a Radicle HTTPD instance, and selecting a project which they want to enable in Woodpecker. This would configure an API endpoint in Woodpecker that the Radicle CI Broker can forward Radicle Webhook Events to. (The user would manually need to configure these in the Radicle CI Broker.)
3. Parse Radicle Webhook Events in Woodpecker Adds support in Woodpecker for parsing Radicle Webhook events (at least push, patch_created, patch_updated). These can then act as triggers for Woodpecker Jobs.
4. Support updating Radicle Patches with Woodpecker Build Status This deliverable includes implementing a new Radicle plugin in Woodpecker. Plugins are pipeline steps that perform pre-defined tasks and are configured as steps in build pipelines. The first version of the plugin will simply add a comment to a Patch and, if time allows, a next version of it will allow updating the new Collaborative OBjects (COBs) that will be added to the protocol about CI Checks.
5. Woodpecker docs Update all the required docs of woodpecker-ci.
6. Radicle guide Provide a guide for setting up and running woodpecker-ci with radicle.

Milestone 3: Kraken CI Integration

  • Total Estimated Duration: 2 calendar months
  • Full-time equivalent (FTE): 29.5 FTE days
  • Team:
    • Michal Nowikowski
    • Yorgos Saslis

For the Kraken CI Integration, after discussions with the Kraken CI maintainer (Michal Nowikowski), we have agreed that he will undertake the integration directly, as long as the necessary features (described in Milestone 1) are added to Radicle to enable this integration.

Number Deliverable Specification
1. Enable Radicle Webhook Events in Kraken CI Adds support in Kraken Webhooks for parsing Radicle Webhook events (at least push, patch_created, patch_updated). These can then act as triggers for Kraken CI Jobs.
2. Support updating Radicle Patches with Kraken CI Build Status This deliverable includes an extension to the Kraken CI Workflow schema (e.g. for a radicle notification), so that Kraken CI could add a patch comment (using the Radicle HTTPD) to point back to the job results). If time allows, a next version will allow updating the new Collaborative OBjects (COBs) that will be added to the protocol about CI Checks.
3. Kraken CI docs for Radicle Update all the required docs in Kraken CI to explain how to set up the Radicle integration.
4. Radicle guide Provide a guide for deploying Kraken CI alongside Radicle and show them working together.

Milestone 4: GitHub Actions Integration

  • Total Estimated Duration: 2 calendar months
  • Full-time equivalent (FTE): 20 FTE days
  • Team:
    • Nikolas Vourlakis
    • Yorgos Saslis

The GitHub Actions Integration will perhaps be the most popular considering GitHub’s dominance in open source project hosting. It is a crucial integration to have, in terms of opening up a smoother migration path for maintainers who want to leave GitHub in favour of Radicle.

This integration is not primarily intended to provide full support for all GitHub Actions features and workflow possibilities. We assume that users who want to leave GitHub for Radicle will also want to leave GitHub Actions eventually. This is instead meant to support maintainers as they go through the different migration phases, allowing them to keep running their CI in GitHub Actions while they migrate the other parts of the project.

In a nutshell, maintainers will be able to break down their migration into smaller, easier steps:

  • Step 1: Migrate code hosting (git) and move from GitHub Pull Requests Workflow to Radicle Patches workflow
  • Step 2: Migrate Issues
  • Step 3: Migrate CI from GitHub Actions to another supported, self-hosted CI, or to Radicle-Native CI (as it becomes available).

More specifically, here is what this integration will cover.

Number Deliverable Specification
1. Extend CI Broker to track GitHub Action Workflows and Workflow Runs. This integration will be based on the assumption that the Git repository will be mirrored across Radicle and GitHub. This is necessary for GitHub Actions to work correctly. GitHub Action Workflows will be defined in the .github/workflows folder and we will only initially support workflow runs on push events (not on PR events, considering we want to replace PRs with Radicle Patches). The Broker will then query the GitHub Actions API to find the corresponding workflow run that has been triggered by the push event and track that run until the run has completed.
2. Support updating Radicle Patches with Workflow Run status This deliverable includes an extension to the Radicle CI Broker so that it would keep polling the GitHub Action Workflow Run and update the patch comment accordingly (using the Radicle HTTPD) to point back to the job results). If time allows, we will also look into updating the new Collaborative OBjects (COBs) that will be added to the protocol about CI Checks.
3. Documentation & How-to Guide Provide a detailed, step-by-step guide about how this integration can be set up, together with examples from public projects (hosted on both Radicle and GitHub) that are set up in the different stages of the migration above, to better illustrate the use case and migration path.

Total Budget :briefcase:

  • Total Duration: 95.25 FTE days
  • Total Costs: 71523 USDC

Drips-Powered Payments

We would like to continue “drinking our own champagne” with Radworks product stack and would propose to use Drips (which has worked great so far!).

Team :busts_in_silhouette:

Team members


Legal Structure

  • Registered Legal Entity: Cytech Ltd.
  • Registered Address: Science & Technology Park of Crete, Heraklion, Greece


Thanks for the work you’ve already done to get started on this. This is without a doubt one of the big missing pieces in the Radicle ecosystem - it is a complete no-brainer to fund this work. So I have absolutely no reservations on whether to fund this or not.

Non-blocking thoughts on prioritizing CI integrations:

  • GitHub Actions makes sense as so many codebases are currently on there and we’d like to entice that user base to come to Radicle
  • Open Source: I did some cursory research on the market share of CI tooling and looks like Jenkins is by far the most used with >40% of market share (source). Compare this to <1% for Concourse (source; this site also confirms Jenkins with >40%). My only point here is flag this large delta as it might make sense for us to look at future milestones from the lens of market share - integrating with the more popular CI tools may help onboard that many more users :smiley:

I’ll give others another few days to review this, but this for sure has my vote!

Thanks for the feedback and additional thoughts @bordumb !

I agree 100% about Jenkins. From my own experience, there is still a very large user base out there - and it is only slowly declining - because:

  • for all its faults, Jenkins is definitely a mature solution (perhaps the most mature in this space) and is as “proven” as they come,
  • many teams / open source maintainers just can’t justify migrating their pipeline once they have it working and they learn the CI system (and its quirks :wink: ).

I have actually already reached out to the Jenkins CI Developers mailing list some weeks ago, to see if we could attract some plugin developer to work on the Radicle plugin, however I didn’t get any interest that way.

Thanks for pointing this out though - I will keep trying to find some contributor to help us close this “gap” and will report back here if I am able to figure something out.