Radicle CI Integrations

đź’¸ Grants đź‘‹ New Grant Applications
1

Radicle CI Integrations

  • Status: Open
  • Proposer: @gsaslis
  • Your Project(s): [optional]: Radicle CI Broker and CI Integrations
  • Projects you think this work could be useful for [optional]: unblock migration to Radicle, as CI is a critical missing feature.
  • Category: 3rd Party Integration

Overview :telescope:

As part of some work funded by the Radicle Core team, me and @nvourlakis have already laid out the foundations for a new component (the Radicle CI Broker) that can be deployed next to a Radicle Node, in order to integrate with external systems. We have implemented the first integration with a CI server (Concourse) and are now seeking additional funding, in order to:

  • implement integration with GitHub Actions
  • implement integrations with 2 more open source CI servers:
    • Woodpecker
    • Kraken CI
  • pave the way for additional integrations with 3rd party systems (e.g. zulip, slack, etc.) by adding the ability to Radicle to send generic Webhook Event notifications to other systems.

These integrations will aim to complement the Radicle-Native CI that the Radicle forge will soon offer, in presenting a decent range of options to maintainers, so they can migrate their projects to Radicle, together with their CI workloads! That is: the integrations to existing CI solutions allow maintainers to continue using their existing CI and break down their project hosting migration to smaller, independent, steps.

Description :page_facing_up:

This project involves work in several different areas, and so will be split into multiple Milestones.

  • Milestone 1: Required functionality on Radicle side
  • Milestone 2: Woodpecker Integration
  • Milestone 3: Kraken CI Integration
  • Milestone 4: GitHub Actions Integration

Milestone 1 will serve as the necessary building block for the remaining 3 milestones, so we will focus on that first. Work on the other 3 can then proceed in parallel.

Milestone 1: Required Functionality on Radicle side

  • Total Estimated Duration: 1 calendar month
  • Full-time equivalent (FTE): 18.75 FTE days
  • Team:
    • Nikolas Vourlakis
    • Ioannis Christodoulou
    • Yorgos Saslis
Number Deliverable Specification
1. Radicle Webhooks Event Schema Before we start shipping Radicle Events as HTTP requests to 3rd party systems (“Webhook Events”), it is important to define the API contract (i.e. the request headers, the JSON body schema, validation rules and some examples). This can help 3rd party systems start working on the integration on their side, while we work on the implementation of the Webhook Events themselves (see below deliverables). This deliverable will involve studying the GitHub, GitLab and Gitea webhook events to inform the design of Radicle’s own Webhook Events schema.
2. Long-Lived Access Tokens support We have already started a discussion with the radicle-httpd maintainers about the possibility of adding longer-lived Access Tokens as an authentication mechanism for HTTP requests to the Radicle API. This deliverable will cover the implementation we will propose, as a Radicle Patch to the Heartwood repo.
3. Extend CI Broker to send Radicle Webhook Events This deliverable will cover the necessary changes and improvements to the Radicle CI Broker, so that it can read rad node events, “translate” those to higher level Webhook Events (starting with push, patch created, patch updated and patch merged) and then forward those as HTTP requests to the preconfigured webhooks for the specific repository. As part of this deliverable, we also need to extend the broker’s configuration so that it can read which URLs to send webhook events to, for each repository, taking into account that these URLs may need to contain secret tokens (for authentication/authorization).
4. More deployment options for CI Broker As part of this Milestone, we will provide Container packaging, Docker Compose, Ansible playbook, Kubernetes manifets, a Helm chart and systemd service files for CI Broker deployments (and integration with the Radicle Node).
5. Documentation and how-to guide Provide detailed, step-by-step guides for the deployment options and for how to enable the Radicle Webhook Events. Also how to create/delete the long-lived access tokens.

Milestone 2: Woodpecker Integration

  • Total Estimated Duration: 2 calendar months
  • Full-time equivalent (FTE): 27 FTE days
  • Team:
    • Michalis Zampetakis
    • Ioannis Christodoulou
    • Yorgos Saslis

We have already began a discussion with Woodpecker maintainers about the possibility of incorporating Radicle into Woodpecker, as a new forge.

With their input and guidance, we will proceed in opening a (series of PRs) implementing the below functionality:

Number Deliverable Specification
1. Forge scaffold Create the radicle’s forge scaffold in woodpecker-ci.
2. Enable Radicle Forge in Woodpecker Allows Woodpecker users to browse Radicle projects, by connecting to a Radicle HTTPD instance, and selecting a project which they want to enable in Woodpecker. This would configure an API endpoint in Woodpecker that the Radicle CI Broker can forward Radicle Webhook Events to. (The user would manually need to configure these in the Radicle CI Broker.)
3. Parse Radicle Webhook Events in Woodpecker Adds support in Woodpecker for parsing Radicle Webhook events (at least push, patch_created, patch_updated). These can then act as triggers for Woodpecker Jobs.
4. Support updating Radicle Patches with Woodpecker Build Status This deliverable includes implementing a new Radicle plugin in Woodpecker. Plugins are pipeline steps that perform pre-defined tasks and are configured as steps in build pipelines. The first version of the plugin will simply add a comment to a Patch and, if time allows, a next version of it will allow updating the new Collaborative OBjects (COBs) that will be added to the protocol about CI Checks.
5. Woodpecker docs Update all the required docs of woodpecker-ci.
6. Radicle guide Provide a guide for setting up and running woodpecker-ci with radicle.

Milestone 3: Kraken CI Integration

  • Total Estimated Duration: 2 calendar months
  • Full-time equivalent (FTE): 29.5 FTE days
  • Team:
    • Michal Nowikowski
    • Yorgos Saslis

For the Kraken CI Integration, after discussions with the Kraken CI maintainer (Michal Nowikowski), we have agreed that he will undertake the integration directly, as long as the necessary features (described in Milestone 1) are added to Radicle to enable this integration.

Number Deliverable Specification
1. Enable Radicle Webhook Events in Kraken CI Adds support in Kraken Webhooks for parsing Radicle Webhook events (at least push, patch_created, patch_updated). These can then act as triggers for Kraken CI Jobs.
2. Support updating Radicle Patches with Kraken CI Build Status This deliverable includes an extension to the Kraken CI Workflow schema (e.g. for a radicle notification), so that Kraken CI could add a patch comment (using the Radicle HTTPD) to point back to the job results). If time allows, a next version will allow updating the new Collaborative OBjects (COBs) that will be added to the protocol about CI Checks.
3. Kraken CI docs for Radicle Update all the required docs in Kraken CI to explain how to set up the Radicle integration.
4. Radicle guide Provide a guide for deploying Kraken CI alongside Radicle and show them working together.

Milestone 4: GitHub Actions Integration

  • Total Estimated Duration: 2 calendar months
  • Full-time equivalent (FTE): 20 FTE days
  • Team:
    • Nikolas Vourlakis
    • Yorgos Saslis

The GitHub Actions Integration will perhaps be the most popular considering GitHub’s dominance in open source project hosting. It is a crucial integration to have, in terms of opening up a smoother migration path for maintainers who want to leave GitHub in favour of Radicle.

This integration is not primarily intended to provide full support for all GitHub Actions features and workflow possibilities. We assume that users who want to leave GitHub for Radicle will also want to leave GitHub Actions eventually. This is instead meant to support maintainers as they go through the different migration phases, allowing them to keep running their CI in GitHub Actions while they migrate the other parts of the project.

In a nutshell, maintainers will be able to break down their migration into smaller, easier steps:

  • Step 1: Migrate code hosting (git) and move from GitHub Pull Requests Workflow to Radicle Patches workflow
  • Step 2: Migrate Issues
  • Step 3: Migrate CI from GitHub Actions to another supported, self-hosted CI, or to Radicle-Native CI (as it becomes available).

More specifically, here is what this integration will cover.

Number Deliverable Specification
1. Extend CI Broker to track GitHub Action Workflows and Workflow Runs. This integration will be based on the assumption that the Git repository will be mirrored across Radicle and GitHub. This is necessary for GitHub Actions to work correctly. GitHub Action Workflows will be defined in the .github/workflows folder and we will only initially support workflow runs on push events (not on PR events, considering we want to replace PRs with Radicle Patches). The Broker will then query the GitHub Actions API to find the corresponding workflow run that has been triggered by the push event and track that run until the run has completed.
2. Support updating Radicle Patches with Workflow Run status This deliverable includes an extension to the Radicle CI Broker so that it would keep polling the GitHub Action Workflow Run and update the patch comment accordingly (using the Radicle HTTPD) to point back to the job results). If time allows, we will also look into updating the new Collaborative OBjects (COBs) that will be added to the protocol about CI Checks.
3. Documentation & How-to Guide Provide a detailed, step-by-step guide about how this integration can be set up, together with examples from public projects (hosted on both Radicle and GitHub) that are set up in the different stages of the migration above, to better illustrate the use case and migration path.

Total Budget :briefcase:

  • Total Duration: 95.25 FTE days
  • Total Costs: 71523 USDC

Drips-Powered Payments

We would like to continue “drinking our own champagne” with Radworks product stack and would propose to use Drips (which has worked great so far!).

Team :busts_in_silhouette:

Team members

Contact

Legal Structure

  • Registered Legal Entity: Cytech Ltd.
  • Registered Address: Science & Technology Park of Crete, Heraklion, Greece
2

@yorgos

Thanks for the work you’ve already done to get started on this. This is without a doubt one of the big missing pieces in the Radicle ecosystem - it is a complete no-brainer to fund this work. So I have absolutely no reservations on whether to fund this or not.

Non-blocking thoughts on prioritizing CI integrations:

  • GitHub Actions makes sense as so many codebases are currently on there and we’d like to entice that user base to come to Radicle
  • Open Source: I did some cursory research on the market share of CI tooling and looks like Jenkins is by far the most used with >40% of market share (source). Compare this to <1% for Concourse (source; this site also confirms Jenkins with >40%). My only point here is flag this large delta as it might make sense for us to look at future milestones from the lens of market share - integrating with the more popular CI tools may help onboard that many more users :smiley:

I’ll give others another few days to review this, but this for sure has my vote!

3

Thanks for the feedback and additional thoughts @bordumb !

I agree 100% about Jenkins. From my own experience, there is still a very large user base out there - and it is only slowly declining - because:

  • for all its faults, Jenkins is definitely a mature solution (perhaps the most mature in this space) and is as “proven” as they come,
  • many teams / open source maintainers just can’t justify migrating their pipeline once they have it working and they learn the CI system (and its quirks :wink: ).

I have actually already reached out to the Jenkins CI Developers mailing list some weeks ago, to see if we could attract some plugin developer to work on the Radicle plugin, however I didn’t get any interest that way.

Thanks for pointing this out though - I will keep trying to find some contributor to help us close this “gap” and will report back here if I am able to figure something out.

4

UPDATE:

After close collaboration with the Heartwood team, we are happy to announce the considerable progress towards several Continuous Integration / Continuous Delivery (CI/CD) integration options that are nearing completion, ahead of the official Radicle 1.0 launch. This is an important milestone for any teams / maintainers that rely on CI pipelines for the projects that would like to migrate to Radicle.

In a nutshell, we will soon be shipping:

  • GitHub Actions,
  • Woodpecker CI,
  • Kraken CI,
  • other 3rd party CI/CD engines that support integration via webhooks.

The above integrations are still evolving and will presumably continue to do so in the future, as more users onboard to Radicle that rely on these CI engines. At the time of writing this, they are primarily dependent on an open (Work In Progress) Patch we need to merge to the Radicle Broker to enable all this.

The below sections provide additional details on the progress made on all these integrations during October 2023 - 15 January 2024. (The original funding was meant to cover until end of December, but we had some remaining budget that covered an additional 2 weeks in January).

Milestone 1: Required Functionality on Radicle side

Number Deliverable Specification
1. Radicle Webhooks Event Schema A first version of the schema has been agreed upon and published here. :white_check_mark:
2. Long-Lived Access Tokens support We have opened a Patch that has already received some approvals and is pending final merge into Heartwood. :white_check_mark:
3. Extend CI Broker to send Radicle Webhook Events In collaboration with the Heartwood team, we have created a new broker that delegates radicle node events to adapters. We have thus created the radicle-webhooks-adapter that takes care of forwarding events via HTTP. It supports multiple outgoing webhooks per project, and secure machine-to-machine communication, using an encrypted shared secret. :white_check_mark:
4. More deployment options for CI Broker There was a change of plans here, resulting to an entirely new version of the broker created and the dropping of the previous iteration of the broker. The new broker is owned by the Heartwood team, so this is now out of scope. We are, however, contributing a Patch to the Broker that enriches the information the broker sends to adapters. This is still Work In Progress. :construction:
5. Documentation and how-to guide The guide about how to persist webhook settings is available here. :white_check_mark:

Demo

You can watch a short demo of the Radicle Webhooks adapter here.

Milestone 2: Woodpecker Integration

Number Deliverable Specification
1. Forge scaffold Done. :white_check_mark:
2. Enable Radicle Forge in Woodpecker Woodpecker users who install the Radicle Forge plugin can browse Radicle projects, by connecting to a Radicle HTTPD instance, and selecting a project which they want to enable in Woodpecker. This configures an API endpoint in Woodpecker that the Radicle CI Broker (and Webhooks Adapter) can forward Radicle Webhook Events to. The user manually needs to configure these in the Radicle CI Broker. :white_check_mark:
3. Parse Radicle Webhook Events in Woodpecker Woodpecker users who install the Radicle Forge plugin can listen to Radicle Webhook events (push, patch_created, patch_updated). These trigger the respective Woodpecker Jobs. :white_check_mark:
4. Support updating Radicle Patches with Woodpecker Build Status The plugin allows users to define a Woodpecker pipeline step that adds a comment to a Patch. :white_check_mark:
5. Woodpecker docs :construction: Work In Progress.
6. Radicle guide :construction: Work In Progress.

Demo

You can watch a short demo of the Woodpecker plugin for the Radicle Forge and how you can trigger jobs in Woodpecker from Radicle Patches here.

Milestone 3: Kraken CI Integration

Number Deliverable Specification
1. Enable Radicle Webhook Events in Kraken CI :construction: Work is in progress and will be delivered in the next few weeks. No extra budget required to cover this work, it is already included in the provided budget.
2. Support updating Radicle Patches with Kraken CI Build Status :construction: Work is in progress and will be delivered in the next few weeks. No extra budget required to cover this work, it is already included in the provided budget.
3. Kraken CI docs for Radicle :construction: Work is in progress and will be delivered in the next few weeks. No extra budget required to cover this work, it is already included in the provided budget.
4. Radicle guide :construction: Work is in progress and will be delivered in the next few weeks. No extra budget required to cover this work, it is already included in the provided budget.

Milestone 4: GitHub Actions Integration

Number Deliverable Specification
1. Extend CI Broker to track GitHub Action Workflows and Workflow Runs. In collaboration with the Heartwood team, we have created a new broker that delegates radicle node events to adapters. We have thus created the radicle-github-actions-adapter that takes care of tracking GitHub Actions workflow runs for the commit that was just pushed or patch that was just updated. It responds back to the broker the results of all the workflows as soon as they finish. :white_check_mark:
2. Support updating Radicle Patches with Workflow Run status The radicle-github-actions-adapter tracks the workflow run and writes a comment on the Patch with the outcome of the workflow run. It also writes a message if the workflow run takes “too long” (there is a configurable timeout). :white_check_mark:
3. Documentation & How-to Guide Docs are currently available on the project repository and more will be made available as soon as the Broker - Adapters integration has been completely and our Patch to the Broker has been accepted.

Demo

You can watch a short demo of the Radicle GitHub Actions adapter, which is triggered using a patch event, here.

Conclusion

These deliverables conclude the work funded by this particular grant. We will seek additional funding to continue work in this area through subsequent Grants applications and/or a new Org proposal in the more distant future.

4 Likes