Pull parsing and Stream API support

[brettz9]

Hi,

The approach I envision for a pull parser would be:

1. Detect a ref
2. Evaluate some condition
3. If the condition is true
   1. Resolve the ref
   2. Optionally repeat steps

The use cases I have in mind:

1. For https://github.com/s3u/JSONPath/ , I'm interested in being able to recursively load JSON References, but only loading (and then caching) as required. If the user specifies a JSONPath expression such as $.aProperty.anotherProperty, there may be no need for loading all of the JSON References in the document (e.g., within a $.aThirdProperty branch), so as JSONPath is traversing the JSON object to find the path, it will only resolve references (and any descendant references) if they are encountered along the way of searching for the user-specified path.
2. For the likes of https://github.com/jdorn/json-editor/ which uses JSON Schemas to build an HTML form for editing JSON, if the schema is built with JSON references, in some cases, there is no need to load the entire schema at once. For example, a JSON-referenced sub-schema file may only be required for reflection if the user indicates through the UI that they wish to create a certain kind of subobject. Lazy loading of JSON data could also come in handy in JSONEditor for the same reason, as well as for the likes of working with this use case.

Ideally an iterator (finding or resolving) could follow any of these strategies:

1. Follow iteration order
2. Follow local references before remote ones (to possibly avoid unnecessary HTTP requests).

Given the increase of availability of streams, and the potential for applications such as chat, relying on them, I'd also like to see Streams API support go along with this request, at least being anticipated if browser adoption is too low to justify support now (unless an existing streams API matches the planned one).

[whitlockjc]

The 2.0.0 API will have this. findRefs already has an options.filter that does type-based filtering or you can provide a function. Is this what you're thinking?

[whitlockjc]

Well, there is no streaming specifically.

[brettz9]

I was thinking more along the lines of:

var iterator = JsonRefs.findRefIterator(json);

function getNextRef () {
    var ref = iterator.next();
    if (ref && meetsCondition(ref)) {
        JsonRefs.resolveRefIn(ref, json).then(function () {
            getNextRef(); // Or we could conditionally call this upon a new event, say such
                          //   as when the user clicks to expand the rendered JSON in view
        });
    }
}

This avoids checking the entire JSON file instead of merely avoiding returning certain results.

[whitlockjc]

The new version of findRefs should allow this, which will also apply to resolveRefs. Here is an example:

JsonRefs.findRefs({...}, {
  filter: function (refDetails, path) {
    return meetsCondition(refDetails));
  }
});

That's not an exact example but the idea is there. For each valid JSON Reference, the filter function is called with the JSON Reference details/metadata and a path to where it was located. You'd still process the whole document but there is also an options.subDocPath to allow you to only search a portion of the document.

https://github.com/whitlockjc/json-refs/blob/master/docs/API.md#module_JsonRefs..RefDetailsFilter

[brettz9]

That's good to know but it won't quite meet my needs as I'd like to avoid traversing the whole document (unless and until needed). Are you open to a PR for this iterator idea whenever I may get around to it (i.e., and leave the issue open even if not making it into 2.0.0)?

[whitlockjc]

If I understand, you just need an "out" when parsing to stop processing. Other than that, the current 2.0.0 stuff does what you need. If not, can you tell me where it's lacking (specifically) so I can try to wrap my head around it? We could probably do it in the existing findRefs and upcoming resolveRefs.

[whitlockjc]

As for PRs, I'm always open to them.

[brettz9]

Yes, having an "out" but also having a "resume", so one can pick up the iteration again if and when desired (without the consumer needing to care or manage where we are exactly in the JSON object tree as your library would keep track of that for us). I think an iterator is the most common and convenient way for this kind of thing to be done. Does that make sense?

[whitlockjc]

It makes sense. I'll likely not put any time into this until 2.0.0 is at API parity with 1.x but PRs are fully appreciated. (I would suggest not doing this until resolveRefs is done as the API is in flux until that point.)

[brettz9]

Ok, very cool, thanks!

[brettz9]

And for my or others' reference, I see we might use the Iterator protocol and optionally with a polyfill if you wish to maintain compatibility with older browser.

[brettz9]

Btw, it seems that ES6 generators provide for a particularly elegant implementation of such iterators. Would you be amenable (if I find the energy) to my modifying the json-refs code (if not also the path-loader code), putting in a babel routine to convert ES6 back to ES5 until ES6 is more widely supported?

[whitlockjc]

If I had to guess, I'd think that moving to ES6 would likely happen after the initial 2.0.0 release. I'm not guaranteeing this but I think it could take some ramp up time on my part. I'm open to PRs of course, although please wait until resolveRefs is done. ;)

[brettz9]

Edited the OP to indicate a desire to see multiple strategies for the iterator if possible. Plan to get to your other items within the day.

[brettz9]

FWIW--and I haven't looked at your code carefully---the thought occurred to me that if possible, it may be helpful to use a generic JSON iterator (such as https://github.com/Floby/node-object-iterator/ ) to do the traversing, though an iterator which could jump ahead by type (e.g., object) and filter (e.g., only with $ref). I.e., though this is tangential to this request, I'm wondering whether you've contemplated making your iterator more generic to any kind of object traversal and including as a dependency (I recall you did have a dependency previously). Anyways, this is just a throw-away idea.