HTTP Provider Resumability

[hannahhoward]

Goals

Even with the fix in #280, we still fail fetches for cids on .storage that are >1GB. This introduces an interesting question around resumability of HTTP downloads for large requests. It seems to me we can accomplish this with entity-bytes if the remote supports it, but we also don't have full support yet for that in traversal. Also, Web3.storage doesn't support it.

[rvagg]

I reckon this might be a candidate for Trustless Gateway spec extension, maybe even using the ipld path to specify the resumption block: ?resumePath=/path/to/block/i/want/next, or just resumeFrom=bafy... and give the next CID I want to fetch. Having the ability to do this might even open up possibilities in parallelising HTTP retrievals.