Add webdav.dandiarchive.org to "services" for the main instance and redirect /download/ for zarrs to webdav

[yarikoptic]

ATM trying to hit /download/ endpoint for an asset with zarr instead of blob would result in 400 Bad Request.

e.g.

• go to sample dandiset with zarr folder: https://dandiarchive.org/dandiset/000719/draft/files?location=sub-1214579789_ses-1214621812_icephys&page=1
• click on a zarr and get to https://api.dandiarchive.org/api/assets/ee648ce0-fda7-44c4-816b-6a80bbcbf850/download/?content_disposition=inline with 400 Bad Request

instead, since this asset ee648ce0-fda7-44c4-816b-6a80bbcbf850 points to zarr id (in this case fd01fe56-9c26-4da3-9c78-e57e85d34033) we already can do two alternatives

• (less preferred) as provided for the current version of dandiset (draft) from webdav: i.e. https://webdav.dandiarchive.org/dandisets/000719/draft/sub-1214579789_ses-1214621812_icephys/sub-1214579789_ses-1214621812_icephys.nwb.zarr/ solely based on the dandiset id, and path within it.
  • it does have advantage that folder name corresponds to the zarr asset path, hence possibly could be used by wget etc as the name of the folder to download under
• (IMHO preferred/future proofed) since we already know the checksum for zarr and we rely on zarr checksum for manifest names and webdav hierarchy, redirect to https://webdav.dandiarchive.org/zarrs/fd0/1fe/fd01fe56-9c26-4da3-9c78-e57e85d34033/52d161bf4654abf2b88f28c479066b6e-20609--25837557.zarr/ , which is pretty much https://webdav.dandiarchive.org/zarrs/{zarr_id[:3]}/{zarr_id[3:6]}/{zarr_id}/{zarr_checksum}.zarr/ .
  • It is more preferred since it likely to remain working without changes upon implementation of versioning support within archive (as being designed within design PR#1892)
  • cons is that path of the asset is lost.

attn @jwodder who might see the shortcomings etc.

[jwodder]

I'm not a fan of a plain https://api.dandiarchive.org/api/assets/{asset_id}/download/ URL redirecting to dandidav for Zarrs. There will surely be users who naïvely try to fetch such URLs regardless of whether the asset is a blob or a Zarr, and as downloading a Zarr with a single URL makes no sense, such requests should continue to fail. At the very least, …/download/ URLs should only redirect to dandidav if some query parameter (e.g., webview=true) is included.

However, since the primary motivation for this issue seems to be improving the UX when clicking on a Zarr in the Archive GUI, an alternative would be for the Zarr links to just go to webdav.dandiarchive.org directly without redirecting through …/download/. (Arguably, providing a browsable web interface for Zarrs should be done by the Archive itself, but that ship seems to have sailed.)

Also, if we actually want to encourage users to download Zarrs via dandidav, one idea would be for dandidav to display a message of the form "To download this Zarr, run wget ..." on root folder listings for Zarrs. File an issue with dandidav if that's desirable.

[yarikoptic]

At the very least, …/download/ URLs should only redirect to dandidav if some query parameter (e.g., webview=true) is included.

Since its is /download/ -- I would expect those to also be used with wget and other download tools, and so they would use webdav not its web view, right? so not sure why webview=true would need to be set in general for such redirects -- can only see the need for users clicking in the browser on that asset.

File an issue with dandidav if that's desirable.

dandi/dandidav#179

[jwodder]

@yarikoptic

Since its is /download/ -- I would expect those to also be used with wget and other download tools, and so they would use webdav not its web view, right? so not sure why webview=true would need to be set in general for such redirects -- can only see the need for users clicking in the browser on that asset.

First, wget doesn't support WebDAV, and the only way any other download tool would even know that it could use WebDAV for a webdav.dandiarchive.org URL would be if it bothered to make an OPTIONS request, which would be pointless 99% of the time.

Secondly, my point is that users shouldn't be running wget on a simple Zarr /download/ URL. Having such a URL redirect to dandidav would mean that naïve wget invocations would retrieve just the HTML page for the root of the Zarr; having this be the "default" behavior (i.e., what you get when you run wget .../download/) is worse than returning an error. Downloading a Zarr with wget requires specifying multiple options, i.e., it is something you have to opt into in order to do properly, and requiring a query parameter is a way for the user to say "Yes, I am intending to make a recursive download blah blah blah, and I promise I've done the necessary prerequisites."

[waxlamp]

(Arguably, providing a browsable web interface for Zarrs should be done by the Archive itself, but that ship seems to have sailed.)

Why do you think the ship has sailed? I see Dandidav as, among other things, a successful proof of concept that (1) versioned Zarr is a viable concept; and (2) that in fact the Archive can have such a thing.

We could of course simply decide to add Dandidav to the list of "external services" and be done with it. But I agree that the download URL should not simply redirect to Dandidav.