RFC: Backup Tool for Radicle

Hello everyone!

I’ve written a prototype of a Radicle backup tool, Radicle Backup (implemented in Rust), which offers backing up Radicle projects to archives and also restoring from such archives. Shortly put, in restoration mode the Git repo itself will be restored, the linked list of Radicle state transition blocks added to IPLD and the IPNS entry made to point to the latest block. You can get more in-depth information on the tool’s workings in the reference documentation I’ve written.

Basically, I’m asking for feedback on my tool. Does it currently make sense? What is it missing? I’m pretty new to Radicle and not super familiar with how it actually works, so I could use the help of the Radicle devs to understand how to properly back up and restore a Radicle project.

Thanks in advance!

Arve

Thanks for the feedback Julian!

Hey Arve,

Hello everyone!

I’ve written a prototype of a Radicle backup tool, Radicle Backup

<https://gitlab.com/aknudsen/radicle-backup> (implemented in Rust), which

offers backing up Radicle projects to archives and also restoring from such

archives. Shortly put, in restoration mode the Git repo itself will be

restored, the linked list of Radicle state transition blocks added to IPLD

and the IPNS entry made to point to the latest block. You can get more

in-depth information on the tool’s workings in the reference documentation

<https://aknudsen.gitlab.io/radicle-tools-documentation/#_radicle_backup_wip>

I’ve written.

Basically, I’m asking for feedback on my tool. Does it currently make

sense? What is it missing? I’m pretty new to Radicle and not super familiar

with how it actually works, so I could use the help of the Radicle devs to

understand how to properly back up and restore a Radicle project.

This is really cool! It seems to make sense, though (leaving aside the

git ipns, which is not actually a machine) it seems like it’s only

backing up the specific machine mentioned. In this case, the project.

But the project is one machine, whose sole purpose is to contain

pointers to other machines – right now, 1) the issue machine, 2) the

patch machine, and 3) the git ipfs non-machine. It seems like 1 and 2

won’t automatically be backed up by this tool.

You’re correct, I wasn’t aware of this architecture. Thanks for explaining how it works!

Which I think is the right design. We’re already working on more

machines, and many won’t be listed under a project, so (assuming I’ve

understood things correctly) it’s more useful this way since we can

then also back up those machines.

But if the purpose is to back up the project, it seems like you’d need

to do a little more leg work. You’d need to get the project machines (as you

do), and back that up. And then get the other machines (

get-rsm-of-type!, from prelude/machine, should help) and again use the

tool to back those up.

You can go either way in terms of which program calls which (though see

note [0] below), but here’s an example of a Radicle program calling your

executable to backup all the machines:

Thanks Julian, I will try to make sense of the below script and get back to you after :slight_smile:

Best,

Arve

···

On Wed, Apr 24, 2019 at 5:32 PM Julian K. Arni jkarni@gmail.com wrote:

On Sun, Apr 21, 2019 at 08:59:37AM -0700, Arve Knudsen wrote:


#!/usr/bin/env radicle



(file-module! "prelude.rad"))

(file-module! "monadic/project.rad"))



(import prelude/machine :as 'machine)

(import prelude/io '[process!] :unqualified)



(def machines-to-backup

  "Get all the machine for a project, including the project itself."

  (fn [proj-machine]

    (cons proj-machine

         (filter (fn [x] (not (eq (lookup :type x) :rad-repo)))

                 (list-rsms!  proj-machine)))))



(def backup-machine

  "Backup the specified machine"

  (fn [machine]

    (process! "rad-backup" [machine] "")))



(def args (get-args!))



(match args

  ['proj] (map backup-machine (machines-to-backup proj))

  'r      (put-str! "expecting single argument: project id")

(Note - I haven’t tested this code at all. I doubt event the parens

match!)

[0] Currently we have a bug where radicle outputs to /dev/tty instead of

stdout, so running a radicle script from rust and capturing the output

is a little annoying. This is bug #1 in the rad issues for Radicle

itself. From the issue itself:

Currently things like `rad issue list | wc -l` won't work, since we're outputting directly to tty. That sucks.



(As a note, in the meantime `script -q -c 'rad issue list'` can be used

I hope this makes sense. radicle-backup is way cool - if there’s more I

can do to help let me know!

Cheers,

Julian

I tried, but ran into an issue: The add-*-rsm! functions require two arguments, machine and id. What should the two different arguments be?

Thanks,

Arve

···

On Tue, Apr 30, 2019 at 4:28 PM Arve Knudsen arve.knudsen@gmail.com wrote:

Thanks Julian!

I’m going to try your script later, hopefully by tonight.

Best,

Arve

On Tue, Apr 30, 2019 at 3:55 PM Julian K. Arni jkarni@gmail.com wrote:

Hi!

On Sun, Apr 28, 2019 at 04:23:47PM +0200, Arve Knudsen wrote:

Hello again

I’ve updated Radicle Backup <https://gitlab.com/aknudsen/radicle-backup> to

take into account all state machines for a Radicle project, i.e. also

issues and patches (in addition to the project RSM). This means that in the

new revision, the expression block lists for issues and patches also get

backed up and restored. I also take care of re-publishing the IPNS records

for issues and patches RSMs when restoring. Thanks Julian for providing me

with a script for getting RSMs!

Nice!

There’s one thing lacking though as far as I can tell, I’m not sure how to

associate issues and patches RSMs with a project when restoring. I only

know how to get them when backing up. How can I do this?

So the command to add an RSM is defined in rad/monadic/project.rad. So running something like:


(file-module! "prelude.rad")

(file-module! "monadic/project.rad")



(add-repo-rsm! <cid-of-repo>)

(add-issue-rsm! <cid-of-issues>)

(add-patch-rsm! <cid-of-patches>)

Should do it. I guess we could add command line options to rad project

so you could just run rad project add-machine patch <cid>. If that

seems helpful, please let me know here (or open an issue) and I can

implement it.

Best,

Arve

On Thu, Apr 25, 2019 at 11:46 AM Arve Knudsen arve.knudsen@gmail.com > > > > > wrote:

You received this message because you are subscribed to the Google Groups “Radicle” group.

To unsubscribe from this group and stop receiving emails from it, send an email to radicle+unsubscribe@monadic.xyz.

I just pushed another update to Radicle Backup, which I think does the right thing, but I would like for you to please provide feedback on whether the tool now backs up and restores a Radicle project correctly. I’d be very happy if the tool can now be considered functional, so I could move on to making the test suite.

Basically, when producing a backup, the tool now stores all the expression blocks for not just the project RSM, but also issue and patch RSMs. Additionally, it stores the JSON document pointed to by the repo RSM (I didn’t follow links, since I’m not sure if this content is published by the Git IPFS remote helper or what).

When restoring a backup, the tool now re-publishes the linked lists of expression blocks for the project, issue and patch RSMs. It also re-publishes the repo RSM JSON document (although, again, it doesn’t attempt to re-construct linked content for this RSM, as I’m not sure what to do).

You can read in detail how the backup tool currently works on the documentation site.

Thanks!

Arve

···

On Wed, May 1, 2019 at 10:34 AM Arve Knudsen arve.knudsen@gmail.com wrote:

I tried, but ran into an issue: The add-*-rsm! functions require two arguments, machine and id. What should the two different arguments be?

Thanks,

Arve

On Tue, Apr 30, 2019 at 4:28 PM Arve Knudsen arve.knudsen@gmail.com wrote:

Thanks Julian!

I’m going to try your script later, hopefully by tonight.

Best,

Arve

On Tue, Apr 30, 2019 at 3:55 PM Julian K. Arni jkarni@gmail.com wrote:

Hi!

On Sun, Apr 28, 2019 at 04:23:47PM +0200, Arve Knudsen wrote:

Hello again

I’ve updated Radicle Backup <https://gitlab.com/aknudsen/radicle-backup> to

take into account all state machines for a Radicle project, i.e. also

issues and patches (in addition to the project RSM). This means that in the

new revision, the expression block lists for issues and patches also get

backed up and restored. I also take care of re-publishing the IPNS records

for issues and patches RSMs when restoring. Thanks Julian for providing me

with a script for getting RSMs!

Nice!

There’s one thing lacking though as far as I can tell, I’m not sure how to

associate issues and patches RSMs with a project when restoring. I only

know how to get them when backing up. How can I do this?

So the command to add an RSM is defined in rad/monadic/project.rad. So running something like:


(file-module! "prelude.rad")

(file-module! "monadic/project.rad")



(add-repo-rsm! <cid-of-repo>)

(add-issue-rsm! <cid-of-issues>)

(add-patch-rsm! <cid-of-patches>)

Should do it. I guess we could add command line options to rad project

so you could just run rad project add-machine patch <cid>. If that

seems helpful, please let me know here (or open an issue) and I can

implement it.

Best,

Arve

On Thu, Apr 25, 2019 at 11:46 AM Arve Knudsen arve.knudsen@gmail.com > > > > > > > wrote:

You received this message because you are subscribed to the Google Groups “Radicle” group.

To unsubscribe from this group and stop receiving emails from it, send an email to radicle+unsubscribe@monadic.xyz.

Hey Arve,

Hello everyone!

I've written a prototype of a Radicle backup tool, Radicle Backup
<https://gitlab.com/aknudsen/radicle-backup> (implemented in Rust), which
offers backing up Radicle projects to archives and also restoring from such
archives. Shortly put, in restoration mode the Git repo itself will be
restored, the linked list of Radicle state transition blocks added to IPLD
and the IPNS entry made to point to the latest block. You can get more
in-depth information on the tool's workings in the reference documentation
<https://aknudsen.gitlab.io/radicle-tools-documentation/#_radicle_backup_wip>
I've written.

Basically, I'm asking for feedback on my tool. Does it currently make
sense? What is it missing? I'm pretty new to Radicle and not super familiar
with how it actually works, so I could use the help of the Radicle devs to
understand how to properly back up and restore a Radicle project.

This is really cool! It seems to make sense, though (leaving aside the
git ipns, which is not actually a machine) it seems like it's only
backing up the *specific* machine mentioned. In this case, the project.

But the project is one machine, whose sole purpose is to contain
pointers to other machines -- right now, 1) the issue machine, 2) the
patch machine, and 3) the git ipfs non-machine. It seems like 1 and 2
won't automatically be backed up by this tool.

Which I think is the *right* design. We're already working on more
machines, and many won't be listed under a project, so (assuming I've
understood things correctly) it's *more* useful this way since we can
then also back up those machines.

*But* if the purpose is to back up the project, it seems like you'd need
to do a little more leg work. You'd need to get the project machines (as you
do), and back that up. And then get the other machines (
`get-rsm-of-type!`, from prelude/machine, should help) and again use the
tool to back those up.

You can go either way in terms of which program calls which (though see
note [0] below), but here's an example of a Radicle program calling your
executable to backup all the machines:

#!/usr/bin/env radicle

(file-module! "prelude.rad"))
(file-module! "monadic/project.rad"))

(import prelude/machine :as 'machine)
(import prelude/io '[process!] :unqualified)

(def machines-to-backup
  "Get all the machine for a project, including the project itself."
  (fn [proj-machine]
    (cons proj-machine
         (filter (fn [x] (not (eq (lookup :type x) :rad-repo)))
                 (list-rsms!  proj-machine)))))

(def backup-machine
  "Backup the specified machine"
  (fn [machine]
    (process! "rad-backup" [machine] "")))

(def args (get-args!))

(match args
  ['proj] (map backup-machine (machines-to-backup proj))
  'r      (put-str! "expecting single argument: project id")

(Note - I haven't tested this code at all. I doubt event the parens
match!)

[0] Currently we have a bug where radicle outputs to /dev/tty instead of
stdout, so running a radicle script from rust and capturing the output
is a little annoying. This is bug #1 in the rad issues for Radicle
itself. From the issue itself:

    Currently things like `rad issue list | wc -l` won't work, since we're outputting directly to tty. That sucks.

    (As a note, in the meantime `script -q -c 'rad issue list'` can be used

I hope this makes sense. radicle-backup is way cool - if there's more I
can do to help let me know!

Cheers,
  Julian

···

On Sun, Apr 21, 2019 at 08:59:37AM -0700, Arve Knudsen wrote:

Thanks Julian!

I’m going to try your script later, hopefully by tonight.

Best,

Arve

···

On Tue, Apr 30, 2019 at 3:55 PM Julian K. Arni jkarni@gmail.com wrote:

Hi!

On Sun, Apr 28, 2019 at 04:23:47PM +0200, Arve Knudsen wrote:

Hello again

I’ve updated Radicle Backup <https://gitlab.com/aknudsen/radicle-backup> to

take into account all state machines for a Radicle project, i.e. also

issues and patches (in addition to the project RSM). This means that in the

new revision, the expression block lists for issues and patches also get

backed up and restored. I also take care of re-publishing the IPNS records

for issues and patches RSMs when restoring. Thanks Julian for providing me

with a script for getting RSMs!

Nice!

There’s one thing lacking though as far as I can tell, I’m not sure how to

associate issues and patches RSMs with a project when restoring. I only

know how to get them when backing up. How can I do this?

So the command to add an RSM is defined in rad/monadic/project.rad. So running something like:


(file-module! "prelude.rad")

(file-module! "monadic/project.rad")



(add-repo-rsm! <cid-of-repo>)

(add-issue-rsm! <cid-of-issues>)

(add-patch-rsm! <cid-of-patches>)

Should do it. I guess we could add command line options to rad project

so you could just run rad project add-machine patch <cid>. If that

seems helpful, please let me know here (or open an issue) and I can

implement it.

Best,

Arve

On Thu, Apr 25, 2019 at 11:46 AM Arve Knudsen arve.knudsen@gmail.com > > > wrote:

You received this message because you are subscribed to the Google Groups “Radicle” group.

To unsubscribe from this group and stop receiving emails from it, send an email to radicle+unsubscribe@monadic.xyz.

Hello again

I’ve updated Radicle Backup to take into account all state machines for a Radicle project, i.e. also issues and patches (in addition to the project RSM). This means that in the new revision, the expression block lists for issues and patches also get backed up and restored. I also take care of re-publishing the IPNS records for issues and patches RSMs when restoring. Thanks Julian for providing me with a script for getting RSMs!

There’s one thing lacking though as far as I can tell, I’m not sure how to associate issues and patches RSMs with a project when restoring. I only know how to get them when backing up. How can I do this?

Best,

Arve

···

On Thu, Apr 25, 2019 at 11:46 AM Arve Knudsen arve.knudsen@gmail.com wrote:

Thanks for the feedback Julian!

On Wed, Apr 24, 2019 at 5:32 PM Julian K. Arni jkarni@gmail.com wrote:

Hey Arve,

On Sun, Apr 21, 2019 at 08:59:37AM -0700, Arve Knudsen wrote:

Hello everyone!

I’ve written a prototype of a Radicle backup tool, Radicle Backup

<https://gitlab.com/aknudsen/radicle-backup> (implemented in Rust), which

offers backing up Radicle projects to archives and also restoring from such

archives. Shortly put, in restoration mode the Git repo itself will be

restored, the linked list of Radicle state transition blocks added to IPLD

and the IPNS entry made to point to the latest block. You can get more

in-depth information on the tool’s workings in the reference documentation

<https://aknudsen.gitlab.io/radicle-tools-documentation/#_radicle_backup_wip>

I’ve written.

Basically, I’m asking for feedback on my tool. Does it currently make

sense? What is it missing? I’m pretty new to Radicle and not super familiar

with how it actually works, so I could use the help of the Radicle devs to

understand how to properly back up and restore a Radicle project.

This is really cool! It seems to make sense, though (leaving aside the

git ipns, which is not actually a machine) it seems like it’s only

backing up the specific machine mentioned. In this case, the project.

But the project is one machine, whose sole purpose is to contain

pointers to other machines – right now, 1) the issue machine, 2) the

patch machine, and 3) the git ipfs non-machine. It seems like 1 and 2

won’t automatically be backed up by this tool.

You’re correct, I wasn’t aware of this architecture. Thanks for explaining how it works!

Which I think is the right design. We’re already working on more

machines, and many won’t be listed under a project, so (assuming I’ve

understood things correctly) it’s more useful this way since we can

then also back up those machines.

But if the purpose is to back up the project, it seems like you’d need

to do a little more leg work. You’d need to get the project machines (as you

do), and back that up. And then get the other machines (

get-rsm-of-type!, from prelude/machine, should help) and again use the

tool to back those up.

You can go either way in terms of which program calls which (though see

note [0] below), but here’s an example of a Radicle program calling your

executable to backup all the machines:

Thanks Julian, I will try to make sense of the below script and get back to you after :slight_smile:

Best,

Arve


#!/usr/bin/env radicle



(file-module! "prelude.rad"))

(file-module! "monadic/project.rad"))



(import prelude/machine :as 'machine)

(import prelude/io '[process!] :unqualified)



(def machines-to-backup

  "Get all the machine for a project, including the project itself."

  (fn [proj-machine]

    (cons proj-machine

         (filter (fn [x] (not (eq (lookup :type x) :rad-repo)))

                 (list-rsms!  proj-machine)))))



(def backup-machine

  "Backup the specified machine"

  (fn [machine]

    (process! "rad-backup" [machine] "")))



(def args (get-args!))



(match args

  ['proj] (map backup-machine (machines-to-backup proj))

  'r      (put-str! "expecting single argument: project id")

(Note - I haven’t tested this code at all. I doubt event the parens

match!)

[0] Currently we have a bug where radicle outputs to /dev/tty instead of

stdout, so running a radicle script from rust and capturing the output

is a little annoying. This is bug #1 in the rad issues for Radicle

itself. From the issue itself:

Currently things like `rad issue list | wc -l` won't work, since we're outputting directly to tty. That sucks.



(As a note, in the meantime `script -q -c 'rad issue list'` can be used

I hope this makes sense. radicle-backup is way cool - if there’s more I

can do to help let me know!

Cheers,

Julian

This is some great feedback Julian, thanks!

I’ve noted the bug you noticed about being directory dependent when calling bin/get-rsms. Will fix it soon.

About projects not using IPFS backed repo, I’m making an issue for it, but I suggest we tackle it in a second iteration. I think it’s easier to scope to IPFS backed repos in the first version and get that working (with a regression test suite).

Do you have any suggestion on what value to use for IPNS publishing timeout?

Last question - do you know how I should handle backup and restoration of the repo RSM? Should I follow links within it and backup/restore them too?

Best,

Arve

···

On Tue, May 7, 2019 at 2:08 PM Julian K. Arni jkarni@gmail.com wrote:

I successfully backed up and restored a project. It’s a really really

nice tool! Hats off on it!

A few notes:

  • Usage of “./bin/get-rsms” means the backup tool only works when run from it’s

    own dir.

  • Some projects don’t use the IPFS-backed repo (using instead gitlab

    etc.). Currently the backup fails if that’s the case. It seems to me

    like it should just not back up the repo (and backup only issues and

    patches) and continue with a warning.

  • The timeout for requests to publish IPNS might be too short - it

    timed out the first time I tried. (The variability in how long `ipfs name

    publish` takes is really annoying - one of the big UI problems we’re

    facing too).

Thanks,

Julian

On Sun, May 05, 2019 at 06:55:40PM +0200, Arve Knudsen wrote:

I just pushed another update to Radicle Backup

<https://gitlab.com/aknudsen/radicle-backup>, which I think does the right

thing, but I would like for you to please provide feedback on whether the

tool now backs up and restores a Radicle project correctly. I’d be very

happy if the tool can now be considered functional, so I could move on to

making the test suite.

Basically, when producing a backup, the tool now stores all the expression

blocks for not just the project RSM, but also issue and patch RSMs.

Additionally, it stores the JSON document pointed to by the repo RSM (I

didn’t follow links, since I’m not sure if this content is published by the

Git IPFS remote helper or what).

When restoring a backup, the tool now re-publishes the linked lists of

expression blocks for the project, issue and patch RSMs. It also

re-publishes the repo RSM JSON document (although, again, it doesn’t

attempt to re-construct linked content for this RSM, as I’m not sure what

to do).

You can read in detail how the backup tool currently works on the

documentation site

<https://aknudsen.gitlab.io/radicle-tools-documentation/#_radicle_backup_wip>

.

Thanks!

Arve

You received this message because you are subscribed to the Google Groups “Radicle” group.

To unsubscribe from this group and stop receiving emails from it, send an email to radicle+unsubscribe@monadic.xyz.

I successfully backed up and restored a project. It's a really really
nice tool! Hats off on it!

A few notes:

- Usage of "./bin/get-rsms" means the backup tool only works when run from it's
  own dir.
- Some projects don't use the IPFS-backed repo (using instead gitlab
  etc.). Currently the backup fails if that's the case. It seems to me
  like it should just not back up the repo (and backup only issues and
  patches) and continue with a warning.
- The timeout for requests to publish IPNS might be too short - it
  timed out the first time I tried. (The variability in how long `ipfs name
  publish` takes is really annoying - one of the big UI problems we're
  facing too).

Thanks,
  Julian

···

On Sun, May 05, 2019 at 06:55:40PM +0200, Arve Knudsen wrote:

I just pushed another update to Radicle Backup
<https://gitlab.com/aknudsen/radicle-backup>, which I think does the right
thing, but I would like for you to please provide feedback on whether the
tool now backs up and restores a Radicle project correctly. I'd be very
happy if the tool can now be considered functional, so I could move on to
making the test suite.

Basically, when producing a backup, the tool now stores all the expression
blocks for not just the project RSM, but also issue and patch RSMs.
Additionally, it stores the JSON document pointed to by the repo RSM (I
didn't follow links, since I'm not sure if this content is published by the
Git IPFS remote helper or what).

When restoring a backup, the tool now re-publishes the linked lists of
expression blocks for the project, issue and patch RSMs. It also
re-publishes the repo RSM JSON document (although, again, it doesn't
attempt to re-construct linked content for this RSM, as I'm not sure what
to do).

You can read in detail how the backup tool currently works on the
documentation site
<https://aknudsen.gitlab.io/radicle-tools-documentation/#_radicle_backup_wip>
.

Thanks!
Arve

Hi!

Hello again

I've updated Radicle Backup <https://gitlab.com/aknudsen/radicle-backup> to
take into account all state machines for a Radicle project, i.e. also
issues and patches (in addition to the project RSM). This means that in the
new revision, the expression block lists for issues and patches also get
backed up and restored. I also take care of re-publishing the IPNS records
for issues and patches RSMs when restoring. Thanks Julian for providing me
with a script for getting RSMs!

Nice!

There's one thing lacking though as far as I can tell, I'm not sure how to
associate issues and patches RSMs with a project when restoring. I only
know how to get them when backing up. How can I do this?

So the command to add an RSM is defined in rad/monadic/project.rad. So running something like:

(file-module! "prelude.rad")
(file-module! "monadic/project.rad")

(add-repo-rsm! <cid-of-repo>)
(add-issue-rsm! <cid-of-issues>)
(add-patch-rsm! <cid-of-patches>)

Should do it. I guess we could add command line options to `rad project`
so you could just run `rad project add-machine patch <cid>`. If that
seems helpful, please let me know here (or open an issue) and I can
implement it.

···

On Sun, Apr 28, 2019 at 04:23:47PM +0200, Arve Knudsen wrote:

Best,
Arve

On Thu, Apr 25, 2019 at 11:46 AM Arve Knudsen <arve.knudsen@gmail.com> > wrote: