Incentivising Seed Nodes - The Next Challenge for Radworks?

1

Hey :seedling:,

As the community gears up for the Radicle 1.0 release – the most significant milestone to date for our community – I have started to contemplate the next frontier on the Radicle journey: gateway infrastructure.

Gateways serve as the silent backbone of decentralized apps. They manage data retrieval, indexing, and overall facilitate a (smooth) user experience. Unfortunately, they also represent a new point of centralization in many decentralized (peer-to-peer and/or blockchain) ecosystems.

A prominent example of the gateway problem is associated with Arweave: you can observe how this currently poses one of the main UX challenges and centralization concerns within the Arweave community at Fetching Transaction Data | Cooking with the Permaweb. You can also find information about some potential solutions they are exploring at Introduction | ar.io Docs and The Role of Observers | ar.io Docs.

Transitioning to Radicle, this is currently an unsolved problem for us. It has not yet become an issue as we haven’t reached 1.0, but I anticipate it will in the near future. The equivalent for a gateway network within our ecosystem would be a network of incentivised seed nodes. If we are serious about sovereignty, contemplating a network of gateways (incentivised seed nodes) is in my opinion crucial.

I know that this has been on the minds of the Radicle team for a while, and the team had already prototyped contracts for payment schemes to support what they previously described as the Radicle Cloud (GitHub - radicle-dev/radicle-cloud). I believe these contracts serve as a great starting point, but I think the time is right to invest more resources and attempt to create an incentivised gateway network using digital currencies, smart contracts, and potentially new (updated) tokenomics for RAD.

Given that we expect to release Radicle 1.0 in Q1, I believe it’s the right time to hire a team of 2 senior engineers who will start executing on the above problem and vision.

Thoughts?

5 Likes
2

I support this endeavor :+1:

1 Like
3

This sounds like a very good idea

Do you mind expanding a bit on how you imagine the tokenomics playing out?
Maybe pulling from some examples of teams who (a) do this well and (b) others who have failed/not done it very well

1 Like
4

This definitely sounds like an area worth exploring further! :+1:

Just a point of caution here: a lot of developers I talk to shut down / stop listening at the mention or mere hint of “blockchain”, crypto, etc. Considering we are targeting Free and/or Open Source Software (FOSS) developers I think we want to be very careful about how we present / communicate these incentivisation / reward mechanisms to avoid alienating some of our target audience.

From my point of view, there is a big difference in “some node operators choose to be paid in crypto” vs. “to run a radicle node you need a wallet, to get paid in RAD”.

Do you think we could target the exploration of both (fiat / crypto) incentivisation models under this initiative, or were you intending to focus on the on-chain side of things here?

4 Likes
5

It’s a great question and I don’t have the complete answer. I hope that the right team will explore this question in detail. But I will share some thoughts below:

  • I don’t think that we should consider designs where we as a community work with a company to provide this service to users. In my mind this contradicts our goal to create a sovereign network.
  • Instead our goal should be to help a market of node operators to emerge around the core Radicle protocol
  • Tokens help a lot with regards to bootstrapping a network of node operators as we’ve seen with numerous protocols (Bitcoin, Ethereum, Arweave, TheGraph etc.)
  • Tokens come with a lot of benefits in certain parts of the world and with a lot of challenges in other parts of the world :laughing:

All in all, I don’t think that we should constraint node operators to get paid in RAD, crypto or whatsoever. Practically I don’t think that we can. Instead, we should consider that a) tokens help bootstrap networks of node operators and b) that we as a community hold a lot of RAD and that’s an asset that we can use to help a market emerge.

6

Yes please! And also would be great if at least one of the engineers had experience developing good products. Like someone with product sense, so that decisions make sense and aren’t just decided purely from an engineering perspective. Basically making sure that anything user-facing is easy to use, well thought-out, and maybe user-tested.

3 Likes
7

Hey folks! Love this discussion! However, the “Proposal Discussion” category is reserved for “formal proposals in the Discussion phase of the governance process.” As this is not a draft of a governance proposal, this discussion actually belongs in the “General” category.

8

Sorry for that. I got confused and posted here. I just moved it to :wave: general.

1 Like
9

No worries! :slight_smile: The reorganization of the Discourse is still fairly new and will take some getting used to!

I tried to add detailed descriptions to each category to make finding the right place to post easier. Always happy to get feedback on if these are clear enough or not!

10

It seems like a nice and promising idea to me. It would be really valuable to validate this idea with early adopters if possible and if the feedback is positive to find an alignment with Radicle’s strategy and integration with its roadmap. I agree with @brandonhaslegs that product people or at least one such person is suitable for this kind of job.

1 Like
11

I believe that “tokens” is the most appropriate tool for implementing this idea, offering both technological advantages and adaptability. I cannot envision how we could achieve the same level of flexibility, cost-effectiveness, transparency, and speed in the traditional fiat world. I recognize the concerns raised by developers who are hesitant to engage in crypto-related projects, but Radicle does not fall into this category. While the reward mechanism may utilize crypto-related technology and practices, it is not inherently linked to blockchain technology or cryptocurrencies. Therefore, while it is still too early to make definitive statements, introducing an optional reward mechanism for those interested in receiving RAD payments for node operation seems like a sensible approach.

1 Like
12

One example that might be relevant to reference is the Hats Protocol community creating the Hats Modules Registry. It establishes an initial list of trusted “Curators” to help with the creation of a credibly neutral list of safe and usable modules for the Hats platform. They use Hats (authority/access ERC-1155 tokens) to manage and reward who has access to review and approve new modules.

1 Like
13

Hey folks! On Monday next week, @lftherios will host a call to discuss thoughts and ideas on this subject for anyone interested. Find details & links below!

:spiral_calendar: Monday, Dec. 18th @ 3pm CET / 9am ET.
:love_letter:Discord invite w/call details: Radworks

14

Thanks @lftherios - this is really interesting. And I totally agree that some kind of solution here will be critically important for the next phase of Radicle.

With problems like this that are not new, in my experience, the highest-value first step is usually to compile a summary of all existing research or solutions, to see if there is something already available, or at least to make sure that we are starting from the most up-to-date designs. So this would be my first suggestion… to compile a document like this before moving forward with building a team or hiring engineers. We may find for instance that there are existing solutions we could re-purpose, or that the initial best way forward does not primarily rely on writing new code.

Second, as you point out in your post, this problem has been around for years, even going back to P2P systems in the 90s and likely even before that. Many teams building P2P solutions have a similar need for seed nodes, including teams like Protocol Labs, Arweave and likely many others (The Graph?). With that in mind, if there are not already existing solutions, I see a lot of potential value in working closely with the projects that have the most similar needs… and maybe even teaming up with them to fund the team developing the solution, perhaps as a small standalone project, since whatever solutions are developed would likely benefit the entire ecosystem.

Finally, I find this problem absolutely fascinating and would be very interested in contributing to the effort :). Excited to see where this goes!

1 Like
15

Thank you @andrewd. Join the session today at 15:00 CET. I will present relevant work from other ecosystems (including some of the names you mentioned) and some concrete ideas about how I think that we should approach it.

1 Like
16

Thank you all for attending the session today. As promised I am posting my notes below:

Radworks Seed Node Incentivization

Problem statement

  1. Relying on a 3rd party Radicle Seed Node is not sovereign.
  2. There is currently no market for Seed Node operators.
  3. Assuming that a market of Radicle Seed Node operators has emerged that offers hosting and content delivery, switching between them based on $ rates or reputation comes with switching costs and UX complexity.

If you rephrase the above in terms of user needs then:

As an end-user:

  1. I want to have guarantees with regards to availability and performance of Radicle
  2. I want to not have to think about choosing and switching between 3rd party gateway providers

Proposal / Discussion

Users will always have the ability to self-host and professional node operators will always have the ability to create a paid offering on top of Radicle. The goal of this session is to discuss ways where Radworks tries to create a third option that addresses (some of) the problems that I mention above.

At a minimum, this initiative should incentivize a market of Seed Nodes to emerge globally and, more specifically, in areas where Radicle’s offering can have the most impact.

At best this initiative should coordinate a group of Radicle Seed node operators around the world that provide hosting and content delivery to end users, without the need for the end user to continuously monitor their performance and switch between them. In this scenario, the user pays one fixed rate and the network takes care of the rest.

Prior Work / What’s out there?

Radicle Seed Nodes serve mainly two roles within the Radicle ecosystem:

  1. they host content for end-users
  2. they serve content to end-users

Most relevant work here comes from Content Delivery Networks (CDNs) or retrieval markets. To my knowledge, there are currently no cryptographic protocols for proving a retrieval, so proof-based systems are out of the picture.

In contrast, there are a number of projects that attempt to offer a decentralized CDN without proofs. Within tokenized networks the term used to describe similar crypto-economic designs is Decentralized Physical Infrastructure Networks (DePIN).

Some of the most notable ones below:

  • Saturn that operates within the IPFS / Filecoin ecosystem.
  • Ar.io that operates within the Arweave ecosystem
  • Fleek that operates across decentralized storage ecosystems

All of them attempt to utilize economic incentives to reward node operators who perform well on various relevant metrics, such as bandwidth served, download speed, or latency.

A simplified way to think of the above designs is the following:

  1. the network provides a pool of rewards (usually in tokens) for node operators that behave as desired. This pool of rewards can come from the fees that end-users pay and / or by some form of treasury or minting process.
  2. within the network there is a group of monitoring nodes that monitor Seed Node operators and collect and compare logs of performance
  3. there is a scoring function that scores each network operator based on the logs collected and the metrics specified by the network
  4. based on that, rewards are distributed to node operators every epoch. Epochs are specified at the network level.

For a more detailed view check the Reword Distribution Module from Saturn, as it does a good job at outlining potential options and how they navigated them.

In an ideal world…

  • An end user pays a fixed price for hosting and bandwidth, so they don’t have to think about dynamic pricing and similar complex things.
  • An end user pays the network in a stable currency.
  • The end user should also have the option to pay in multiple currencies (like ERC-20s).
  • Any Seed Node operator can join the network and be considered for the pool of rewards (ie the network is permsionless).
  • There are Seed Nodes across the globe and the network can dynamically incentivize the operation of Seed Nodes in specific territories of interest.
  • The set of monitoring nodes is also permisionless. Anyone can join, contribute and be rewarded.
  • The network manages the above functionality in a private setting/environment, so it’s harder to capture.

What are our options?

  1. Do nothing. It’s not worth taking on this problem at all.
  2. Do nothing for now. It’s not worth taking on this problem right now.
  3. Let’s incentivise professional node operators with grants from our treasury, but don’t do anything else.
  4. Let’s piggy back on some of the existing decentralized CDN networks like Saturn.
  5. Let’s experiment with building something Radicle specific that tries to satisfy the above mentioned user-needs.
5 Likes
17

Thanks a bunch for writing this up @lftherios

I think it might be worth writing up a high-level, but technical RFP and possibly getting some of the initial work for this - such as research, design, or POCs - via Grants.

I’ve posted an updated 2024 Proposal. In there, you will see the concept of “Teams” within the Grants ecoystem. We may able to use a few initial projects - as noted above - to onboard a serious team to this broader work.

Let me know your thoughts.

1 Like
18

I appreciate the feedback received since my last post. During the Christmas break, I’ve had the time to reflect further on the above discussion.

In my initial post, I focused on retrieval markets for Radicle through incentivized seed nodes. I anticipated that after the release of Radicle 1.0, a global network of seeders would emerge to host and serve relevant content.

However, I thought that there may be a distinct market gap for Radicle users seeking a combination of the following attributes: high performance (minimal retrieval latency), availability, and censorship resistance.

Through one-on-one conversations with community members, it became clear that for individuals primarily concerned with censorship resistance, the unbeatable choice remains unstructured peer-to-peer networks coupled with self-hosting. Despite existing friction, this option offers the highest level of censorship resistance.

But for those seeking a blend of performance, availability, and censorship resistance, the landscape is significantly different. Let me elaborate further.

My primary question was whether there would be a market willing to pay for this combination of attributes. To explore this, I delved deeper into professional users, including traditional companies, app developers, and DAOs.

While my initial assumption was that many of these users would prefer dealing with a conventional company they can contact, I realized this might not always be the case. In the industry we operate in (crypto networks), my thoughts immediately turned to IPFS. What’s noteworthy about IPFS is the following: while not many individuals host personal websites on IPFS, IPFS has clearly achieved product-market fit with dApps, crypto protocols requiring metadata, and various crypto-related applications. On IPFS, most retrievals are handled by traditional centralized providers, though they are clearly exploring alternatives like Saturn due to the consequences of centralization and the market opportunity.

So, why do app developers use IPFS?

In our industry, most app developers highly value user data sovereignty, aim to avoid hosting user content in centralized databases to prevent their apps from being captured, and prioritize security. IPFS allows them to build applications while satisfying all three of these requirements, despite its limitations.

As users of IPFS ourselves (we extensively use IPFS for Drips and pay for a gateway for retrieval), the concern of relying on a 3rd party provider is always there. In addition, one significant drawback we encounter with IPFS is the inability to modify content once it’s published (e.g., a description on a Drip List). Users must publish a new content addressable link on IPFS and checkpoint it on the Ethereum main-net, which is prohibitively expensive. Generally, IPFS excels with static content but falls short compared to Radicle for dynamic content such as software artifacts. For example, an app developer could publish their web app as a radicle repo, and any seed node could offer to serve that repository on the web. This deserves a more detailed post, which I will share before the 1.0 launch.

Returning to my main point: assuming Radicle gains adoption as a decentralized solution for software hosting and software artefact distribution, then it’s only natural for demand for decentralized & performant retrieval to emerge from app developers. Think software artefacts, generic metadata (like the metadata we use on Drip Lists), or even more intriguing is to think about decentralized front-ends, a need that pretty much all crypto apps have/ are willing to pay for as @cloudhead pointed out to me!

I believe this presents an intriguing opportunity for Radworks to offer a decentralized network for retrieval of software artefacts. Of course, the next question is whether a decentralized retrieval service can compete with the likes of Cloudflare in terms of pricing and/or performance. But maybe that’s not even needed, as the blend of the three attributes I mention above make it unique enough. To me it feels that this is a very interesting opportunity for Radworks to explore.

5 Likes
19

Ele,
Did you have anyone in mind from Radworks that you were hoping would take this on from here? I’d be happy to throw my hat in the ring to help tackle this if it aligns with our overall priorities and potentially fill a gap.

First, a few thoughts. Your point about the need for a service that combines high performance, availability, and censorship resistance is well-taken. It’s clear that while peer-to-peer networks offer the highest level of censorship resistance, there’s a trade-off with performance and ease of use.

In some initial explorations of the landscape this morning, I came across Apron Network, which aimed to provide decentralized infrastructure services for DApp developers and users. They offered a range of services, including node services and on-chain data indexing, catering to various blockchain ecosystems. However, it seems that Apron Network’s activity has slowed down, as indicated by their lack of recent updates. Their whitepaper provides a comprehensive view of their intended services and structure, which could be insightful for us. It’s from a few years ago now so maybe is of limited value, but it helped me somewhat in getting up to speed here so figured I’d send it along.

In addition, it seems like Cloudflare does have some presence in web3 already as well, with their distributed web gateway for IPFS. I wasn’t aware of this, and was curious if you’d previously come across this?

If this is a priority, I’d be happy to delve deeper into this, perhaps by conducting an opportunity assessment that could inform some more specific next steps here. This should definitely include looking into how we might recruit others who are more knowledgeable about these specific challenges.

I think what we’d come across pretty quickly is how even this preliminary exploration would be funded… Would you be able to share more about how you envision Radworks taking on this effort more formally? Are there specific aspects of research that you think we should prioritize? If we did decide to build something here, how would you see that effort being funded and managed? Should that live wholly within Grants? What should be the role of the Strategy committee, if any?

1 Like
20

Not one person in particular, but I am working on a proposal based on the above.

Yes. I actually used IPFS + Cloudflare for the last three years for my own personal blog. It works as expected if you are simply looking for 3rd party hosting, but the experience actually has gotten slightly worse over the years, as the integration hasn’t gotten so much attention within the Cloudflare platform.

I personally think that this should be a new team or org that is funded by the DAO to pursue this vision further. But I am working on something concrete that I will share later today or tomorrow.

1 Like