Project discovery: proposal

Discovery: architecture

For the needs of the application, we are looking for a way to subscribe to a feed of new projects that surface on the network. Initially, this is a way to explore and discover a hand-curated list of projects, but the long term goal for this is that all projects on the network can be discovered this way.

From the user’s point of view, we essentially want to:

  1. Have a way to specify seed nodes to connect to
  2. Explore the projects published by these nodes
  3. Subscribe to new projects as they are published

Proposal

There are two relevant components: the seed node, which is always online, and publishes interesting events, and the application, which is a client that connects to the node(s) and fetches these events to display them to the user.

Seed node

It’s important that the seed node is able to provide a neutral timeline of the projects being created in the network. Of course, the only way it can do that is to publish the projects as it discovers them. The simplest data-structure that serves this need is an append-only log of events.

The event, in its simplest form is Event { project_urn: RadUrn }.

The end-goal is for the node to have an HTTP API that can be used to register projects to track and replicate. When this API is called, the node starts looking for the project, and when it has successfully cloned it, it publishes an event on its feed. To start though, we can feed it a list of projects to initially track.

The feed, being a radicle-link entity type, is signed by the node every time a new event is published to it.

Though the minimal event only includes a URN, the node is free to add any metadata necessary, that the user might find interesting. For example, project stats, maintainers, language, description etc. This information can be parsed from the underlying project repo and included in the event feed.

Application

The application has configuration which specifies which seed nodes to track. These are DNS hostnames which point to a (PeerId, IpAddr) pair.

When connected to a seed node, the application requests the event feed from a well-known location. Eg. /namespaces/feed.

It parses the events, displays them to the user, and allows the user to track the projects.

Future

Eventually, users will also gain the ability to publish their own event feeds, just like the seed node. These can be freely intermixed with the “aggregates” published by the seed nodes. Each feed represents the subjective perspective of the user/node publishing it.