Allow git as the sync backend

[NightMachinery]

Related: #120

I already manage my notes via git, and don’t like cloud services for syncing. Adding a git-backend sync would be very helpful.

Perhaps you can integrate with the app Working Copy for this. You can use that app’s beta for free here.

Some advantages:

• being REALLY tool-agnostic: There is no vendor lock-in whatsoever. One can use multiple tools simultaneously. Migrating to other apps is a breeze.
• explicit merge conflict handling (no data loss ever, no surprises, no bullshit)
• Can support streamlined sync for when no conflicts exist: ga . ; gcam “some auto generated message” ; git pull ; git push
• version control
• git’s hooks
• Supports not-realtime collaboration by default (See how awesome lists create a community list without using any special infrastructure except Github)
• if used publicly, allows automatic publishing (on GitHub pages, Github itself, etc)
• KNOWN and TRUSTED. Any other sync mechanism is either a blackbox or needs considerable time investment to be understood by users. People already know git.

[webframp]

Appreciate the detailed suggestion @NightMachinary. I definitely want to support this feature, thanks for spelling out the benefits.

I’ve had discussions with the org mode maintainer about it as well and possibly even removing the dependency on the old style org-mobile.el sync method

[webframp]

@dive any suggestions for a good git interaction library in swift? I haven't looked around much yet, but if you know of one that's in common use or has a nice api let me know.

[NightMachinery]

@webframp 😃👍
Adding support for using an external location for storage (with no sync), should be enough to use apps such as Secure Shellfish (SSH/SFTP), Working Copy (git), iSH (git, SSH, ...). I think that would be a good idea, as it allows more flexibility without (I hope) being all that hard a feature to add.

To illustrate, here is a screenshot of my working copy’s git folder in a file explorer app:

[gitonthescene]

@NightMachinary What exactly don't you like about "cloud" storage? Having a better picture of what the issue is makes it easier to assess the options available.

[webframp]

I would love to enable users to just use a private git repo, either self hosted, GitHub or elsewhere

[gitonthescene]

I still think the best approach is to understand the problem being addressed first and then assemble solutions afterwards. I believe the current staging mechanism was developed as a simple way to manage conflict resolution when syncing. Pulling from a git repository would manage conflict, but we might need to manage a git client in the app. A direct sync doesn't really seem to handle conflicts well.

[webframp]

I think I agree @gitonthescene. My interest is in simplicity where possible and enabling user choice for where they store their data.

[gitonthescene]

Maybe it would be good to walk through some scenarios/expectations.

1. User edits Org file on home computer
2. Then pushes to some git repository
3. That file is pulled (?) from git

• Is this pulling raw source?
• Is this a git pull with full merge?

4. File is edited(?) on the app.

• Is this still allowed?
• Will we need to commit this locally?
• Are we running a git client remotely and doing a raw sync there where we merge conflicts?

5. File is edited remotely and repushed
6. File is pulled from git

• Do we discard local edits?
• Do we block the sync until we’ve uploaded local changes?

We’ll need more engagement from users on these sorts of scenarios to know if we’ll be making the app better or worse.

Unfortunately more options is often at odds with simplicity. Not that we shouldn’t strive for both.

If we establish some principles it will be easier to make decisions. I propose “We must be able make edits to existing org files on the app” as a founding principle. I also propose “The app must share files with other apps” which inevitably opens up questions of synchronization. I’m completely agnostic as to how these goals are achieved, nor to I suggest this list of principles is exhaustive.

[gitonthescene]

I think there are some good ideas here FWIW: https://opensource.guide/

[NightMachinery]

@gitonthescene I have written why I want git on my first post, plus a very easy way to add support for it using other apps. But to reiterate, git is the only sync option that gives me all the power. I can self-host it (on multiple locations), I can edit files offline, I’ll never fear for a merge conflict leading to data loss, I have my files versioned and can go back in their history if needed, I can use an SSH session to edit my notes easily, and I’ll not be worried that a cloud storage will disappear from under my feet. (Once upon a time, I backuped my photos to Flickr. Then one day I saw a spammy email saying Flickr is discontinuing their free cloud storage. If I hadn’t seen that, I’d have been in for a big treat.)

I agree that simple users won’t find git support useful, but this is a FOSS app for Org mode. Simple users aren’t going to use this anyways. They have great options a dozen.

On your questions of how the git support should work.

1. Using external apps such as Working Copy: In this case MobileOrg does no syncing whatsoever. It just uses the file from that external location and edits them normally.

2. Internal git client: This should be a real git client (i.e., just using normal “git pull” and stuff), but it need not have GUI support for solving conflicts. I solve conflicts manually by vim all the time. If you really want full GUI support, option 1 is much more attractive. Pulling, committing and pushing should be explicit actions by the users, though a shortcut of “gcam AUTOGENERATED_MESSAGE && git pull && git push” will be a good idea, perhaps with an option to run this once a day or sth.

[gitonthescene]

I don’t know Working Copy but I’ll have a poke around. To be clear, you can run your own WebDav instance pretty easily too. WebDav was designed for collaborative upload as well with file locking etc, though I don’t believe that MobileOrg takes advantage of those features. WebDav is much lighter weight than git as well and can be run over ssh.

From a practical stand point, resolving merges on the phone feels like a bit of a pain. I use github a bit but have never tried doing anything substantial with it on the phone. It’s not clear to me what other choices there are for conflicts though. The current scheme shuffles these conflicts off to a staging file so as not to disturb the main files. I merge those differences on my desktop.

I’m not sure how you mean “Use multiple tools simultaneously”. Do you mean if your files are backed by github you can access those files with other tools? To me that’s sort of orthogonal to how MobileOrg chooses to sync. I picture most users of the app as doing editing on the app and their desktop and most of their editing on desktop. Syncing is about keeping these two copies from diverging too far apart. What else you do with those files is up to you. You could sync to github from your desktop working copy, though you’d want to be in sync with your phone when you do that.

Given that these files are generally also on your desktop, there really is no worry of losing the data (at least not more than losing any of the data on your desktop).

That said, I’m not against the idea, but it feels like overkill for something meant to keep your phone in sync with your desktop. If there was a simple plugin implementation, so much the better though I still worry a bit about managing the flexibility from the phone.

It would help to map out a few use cases in detail. The real rub here is conflict resolution, which makes auto pulling or pushing more suspect.

Lastly, your first link was to something called TestFlight. Is that what you meant? If so, I’m not following.

[gitonthescene]

Okay, so I had a quick look at Working Copy and I think I understand your suggestion better. The idea would be to move syncing and all the merge issues that come with it outside the app. You would point at files which are shared with some other app that MobileOrg org would freely read from and write to. The only “trick” would be making sure those files are shared with the other app.

This sounds pretty easy to do and worth doing. I wouldn’t want this to be the only mode in the short term since I don’t have any other examples of syncing apps like this, so in the short term this would be the vendor lock in you mentioned above.

I like this a lot better than having to maintain a git client in MobileOrg and is nicer than having to maintain syncing code in MobileOrg modulo that vendor lock in.

But maybe as you suggest this could be a model for other sync methods. I.e. that they be separate apps. MobileOrg could provide a trivial one for whatever sync model it settles on.

Thanks for the suggestion.

[NightMachinery]

@gitonthescene Great! There are other apps though! iSH provides a partial Linux environment that can run git and rclone. ShellFish provides SFTP and some other methods. Cloud drive apps (Dropbox, Gdrive, etc) provide cloud storage.

I’m looking forward to the update. :D

[gitonthescene]

It sounds like the other options wouldn’t as directly provide the support for merging conflicts during a sync, though.  It helps to be clear when evaluating alternatives. Sent from Yahoo Mail for iPhone On Sunday, July 5, 2020, 5:23 PM, NightMachinary <[email protected]> wrote: @gitonthescene Great! There are other apps though! iSH provides a partial Linux environment that can run git and rclone. ShellFish provides SFTP and some other methods. Cloud drive apps (Dropbox, Gdrive, etc) provide cloud storage. I’m looking forward to the update. :D — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[pashazz]

I'd like to up this feature