Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Propose better JSON-LD processing text #1302

Closed
wants to merge 30 commits into from

Conversation

OR13
Copy link
Contributor

@OR13 OR13 commented Oct 4, 2023

Copy link
Contributor

@decentralgabe decentralgabe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice and clear

index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
OR13 and others added 4 commits October 4, 2023 17:33
Co-authored-by: Brian Richter <[email protected]>
Co-authored-by: Brian Richter <[email protected]>
Co-authored-by: Brian Richter <[email protected]>
Co-authored-by: Brian Richter <[email protected]>
index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
common.js Outdated Show resolved Hide resolved
common.js Outdated Show resolved Hide resolved
Copy link
Contributor

@andresuribe87 andresuribe87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great for implementers

Copy link
Member

@msporny msporny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, happy with 80% of the text in this PR. The rest of it either needs to be made less vague, be corrected, or be removed. Concrete suggestions have been made where possible, questions raised for vague statements to get further clarity.

index.html Outdated
Comment on lines 4258 to 4259
in cases where the issuer has leveraged the <code>@context</code> to provide details regarding
their intended expression of the <a href="#verifiable-credential-graphs">Verifiable Credential Graphs</a>.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not clear what "intended expression of the Verifiable Credential Graphs" means, or how this is different from "Minimal JSON-LD Processing" (which is what we should rename the JSON Processing section to, IMHO).

Perhaps we should have these sections: "JSON-LD Processing", which contains two sub-sections: "Processing without a JSON-LD Library" and "Processing as RDF"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this suggestion, I will update to it

index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
index.html Outdated
Comment on lines 4346 to 4347
Implementers should be aware that object member, and array element order are not preserved by default in JSON-LD,
and as such, Issuers intending to communicate array order need to leverage the langauge features of JSON-LD.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Implementers should be aware that object member, and array element order are not preserved by default in JSON-LD,
and as such, Issuers intending to communicate array order need to leverage the langauge features of JSON-LD.
Implementers are advised that object member, and array element order are not preserved by default in JSON-LD,
and as such, Issuers intending to communicate array order need to leverage the langauge features of JSON-LD.

index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
index.html Show resolved Hide resolved
index.html Outdated
Comment on lines 4252 to 4253
Some scenarios where processing a <a>verifiable credential</a> or a <a>verifiable presentation</a> as RDF is
desirable include, but are not limited to:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Some scenarios where processing a <a>verifiable credential</a> or a <a>verifiable presentation</a> as RDF is
desirable include, but are not limited to:
Some scenarios where limiting processing to a limited set of expected
<a>verifiable credentials</a> or a <a>verifiable presentations</a> is
desirable include, but are not limited to:

Copy link
Contributor Author

@OR13 OR13 Oct 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd take "Some scenarios where limiting RDF processing"... but I won't take removal of RDF entirely.

index.html Outdated Show resolved Hide resolved
index.html Outdated
Comment on lines 4284 to 4286
Naive JSON processing is allowed as long as the document being consumed or
produced is a <a>conforming document</a>, however, this can lead to interoperability issues
and confusion about the specific information that has been secured between JSON-LD and RDF processors.
Copy link
Contributor

@dlongley dlongley Oct 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, "Naive JSON processing" is never allowed ... with any JSON. I guess I don't know what this means. When you are processing JSON ... or anything generally, you shouldn't be naive. There is also no special danger of confusion or deviation of interpretation from generalized processors if the rules are followed, so we should focus there.

An attempt at rewording:

Suggested change
Naive JSON processing is allowed as long as the document being consumed or
produced is a <a>conforming document</a>, however, this can lead to interoperability issues
and confusion about the specific information that has been secured between JSON-LD and RDF processors.
<a>Conforming documents</a> that are expressed using contexts and shapes that are
well-known to a consuming application can be consumed like any other JSON
document, that is, according to the rules of its associated specifications and interpreted
according to the <a>issuers</a> intent. <a>Issuers</a> might use additional
contexts in <code>@context</code> to provide details regarding the expression of
their <a>verifiable credential</a>, and consumers are advised to follow these rules
to ensure that the document matches their accepted contexts and document shapes:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think your suggestion makes it clearer. The issuer is required to use @context, since it's required in conforming documents.

Perhaps you mean additional context beyond the v2 core context? can you adjust your suggestion to take that into account?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've made an adjustment to say "Issues might use additional contexts... and consumers are advised...".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@OR13 please respond to this comment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to define "shapes that are well-known", assuming some version of this PR will land:

#1320

I can accept this suggestion as it exists.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@OR13,

I can accept this suggestion as it exists.

Yes, please, thanks!

index.html Outdated Show resolved Hide resolved
index.html Outdated
Comment on lines 4314 to 4315
These rules alone do not guarantee semantic interoperability between JSON and JSON-LD for
literal JSON keys mapped to URIs by the `@context` mechanism.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is true now with some clean up above and it's simpler:

Suggested change
These rules alone do not guarantee semantic interoperability between JSON and JSON-LD for
literal JSON keys mapped to URIs by the `@context` mechanism.
The rules above guarantee semantic interoperability between applications
that are hard-coded to specific contexts and applications that more
generally consume JSON-LD documents.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not true, but could be true modulo some defense language around contexts changing, and it would be nice to say it this way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dlongley @msporny I am not intending to accept this suggestion, in the interest of getting this PR to mergable state, I suggest we resolve any threads that we feel don't need further discussion on the PR, if you need issue markers to resolve a thread, please file issues, and I will address all issues filed in a subsequent PR.

index.html Show resolved Hide resolved
index.html Outdated
Comment on lines 4245 to 4249
While the media types describing conforming documents defined in this
specification always express JSON-LD, JSON-LD processing is not required to be
performed, since JSON-LD is JSON. Some scenarios where processing a
<a>verifiable credential</a> or a <a>verifiable presentation</a> as JSON is
specification always express JSON-LD, RDF processing is not required to be
performed, since JSON-LD is a concrete RDF syntax as described in [RDF11-CONCEPTS].
Hence, a JSON-LD document is both an RDF document and a JSON document and correspondingly represents an instance of an RDF data model.
See <a href="JSON-LD11##relationship-to-rdf">Relationship to RDF</a> for more details.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The notion of "RDF processing" is not defined, and it can mean many things. Better not go there. Besides, it took me some time to understand what (I think) you meant in this section... What about a rewrite of the whole paragraph along the lines of:

While the media types describing conforming documents defined in this specification all express JSON-LD, developers should be aware that JSON-LD is a serialization of the abstract RDF Model [[RDF11-CONCEPTS]]. Hence, a JSON-LD document is both a JSON document and the representation of an abstract RDF Dataset. See Relationship to RDF for more details.

Also, in view of what you write later, I have the feeling that you have in mind two type of "processing":

  • "RDF Processing" when you transform the JSON-LD into RDF using the JSON-LD rules
  • "JSON-LD Processing" when you operate on the JSON level only.

if that is correct, this may be the place to define these, because otherwise the text below is very unclear. That being said, I do not like the two terms; for me, "JSON-LD Processing" means to perform the processing steps defined in the JSON-LD 1.1 Processing Algorithms and API specification, and this seems to contradict this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ this is correct, my goal is to basically say this:

  1. You need to care about JSON-LD
  2. You can care about it and ignore the context, you will have a bad time.
  3. You can care about it and pay attention to the context, and W3C or any other 3rd party context host can cause you to have a bad time (by changing the context after you commit to it).
  4. In the case that all contexts are immutable and always available, you can still have equivocation, but at least its limited to differences between RDF and JSON, not RDF and JSON-LD (where there will be NO equivocation in this very special case).

Copy link
Contributor

@dlongley dlongley Oct 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can care about it and ignore the context, you will have a bad time.

No, this is not a thing. You never "ignore" @context. You can hard-code your application to only work with some specific contexts, that's fine. But you don't "ignore" @context; that is not at all the same thing. Perhaps this is part of a core misunderstanding here?

The comments about context immutability are a different kind of issue. We can have text here that says: "if you're going to hard-code to specific contexts, they must be immutable in order to guarantee the same interpretation as someone who doesn't hard-code". That is a simple fact and encouraging the use of immutable contexts for VCs is a good thing, IMO.

index.html Outdated Show resolved Hide resolved
index.html Outdated
Comment on lines 4262 to 4264
After performing JSON Schema validation, as described in Section
<a href="#data-schemas"></a>, in order to mitigate attacks
on the canonicalization process as described in <a href="https://w3c.github.io/rdf-canon/spec/#dataset-poisoning">Dataset Poisoning</a>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two comments on the same section

  • I am not sure why JSON schema is relevant in this respect. I guess the issue is that if the issuer decided to use the RDFC 1.0 algorithm for its security related processing, and that is completely orthogonal whether he/she also uses JSON schemas. The two are unrelated
  • RDFC 1.0 takes care of the dataset positing issue, so "mitigate may not be appropriate here. The bullet point may be:
Suggested change
After performing JSON Schema validation, as described in Section
<a href="#data-schemas"></a>, in order to mitigate attacks
on the canonicalization process as described in <a href="https://w3c.github.io/rdf-canon/spec/#dataset-poisoning">Dataset Poisoning</a>
In order to fine tune the <a href="https://www.w3.org/TR/rdf-canon/#canon-algo-algo">defense against the dataset poisoning issue</a> built-in the RDFC 1.0 algorithm. [[RDF-CANON]]

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw now @dlongley's comment that the whole bullet point should be removed in view of the recent changes in RDFC; I would be o.k. with that, too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think its better to give direct security advice here, conforming documents are JSON, schema validation can protect against poisoning... telling people to read the RDF spec without spelling this out is harmful to the security posture of our document.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dataset poisoning is an issue in general but, I have the impression, it is very rarely an issue with v. credentials or presentations, and the text we are writing here relates to those. (It has been my suspicion for a long time that a VC/VP dataset is, from an RDF point of view, "well-behaved" 90% of the time, which means that the "tricky" part of the RDFC 1.0 algorithm, that may run amok, is not applied in the first place. But I cannot prove that.) The VCDM spec does not even talk about defining a blank node explicitly, something that is usually done to create poisoned datasets.

(Let alone the fact that, per RDFC 1.0 spec, each implementation may have its own way of defending itself against poisoned datasets, whose parameters (that the user may fiddle with) are therefore also implementation dependent. The RDFC standard only stipulates that a defense must exist and have a reasonable default parameter setting; the test suite does include a few poisoned datasets that the implementation should pass.)

As a conclusion, I do not believe that going to the details of the RDFC 1.0 is really an issue for this text.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@OR13, please respond to this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decline to implement feedback, but perhaps a stronger rewarding regarding the security impact would be acceptable

index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
index.html Show resolved Hide resolved
index.html Outdated
</p>
</section>
<section>
<h2>Advanced RDF Processing</h2>
Copy link
Member

@iherman iherman Oct 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we find an alternative subtitle? There is nothing "Advanced" in what is in the section, at least not from an RDF point of view...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest we remove this subsection. It's not clear what "expressions of intention" would actually be. The closing paragraph in this section also states that...

Verifiers that do not understand or process JSON-LD, will not be aware of these differences, and confusion over the intention of the issuer and the holder could lead to unexpected processing behavior by verifiers.

The claims being made in the separate graphs are no more "separate" conceptually in JSON than in RDF and any "intention" toward their use would be on the Verifiers (or whomever) to "unite" as/when desired.

The other items about id/type and value ordering introduce more confusion than value in their current form. Folks familiar with processing RDF out of JSON-LD will know about aliasing and value ordering, and (as @iherman mentions) there's nothing "advanced" about it...just par for the course when using term aliasing and graph data.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intention was to clearly communicate that there is "JSON-LD processing" where you don't apply the context (This was previously called "JSON Processing"... and there is JSON-LD processing, where you do apply the context (this is currently titled "Advanced RDF Processing".).

I'm not attached to the names, I am attached to signaling that both processing rules are in the context of RDF.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not call the first "JSON-LD Processing", because it is very misleading. The term "JSON-LD Processing" is a term coined by the JSON-LD WG and has a well-defined meaning. It is actually closer to what you call "Advanced RDF Processing".

What we describe in the spec handling VC files in JSON-LD format but without a full JSON-LD Processing is, though fairly general, proprietary to the VC specification. Maybe using something like "Basic VC Processing" would do.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manu suggested calling in RDF processing, I think thats a fair compromise... its the "JSON-LD... but with transformations that care about effort put into the context" processing kind...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dlongley suggested "Static Processing" vs. "Dynamic Processing"... where "Static Processing" covers things like "Use a JSON Schema to check the well-formed-ness of the payload and then process it like you would a JSON payload... and where "Dynamic Processing" is using a JSON-LD processor to compact/expand or convert to NQuads.

Thoughts?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure about the "Dynamic" vs. "Static". I do not lie down on the road over this, but these two terms do not speak to me in this context.

index.html Outdated
Comment on lines 4338 to 4339
Implementers should be aware that <code>id</code> and <code>type</code> have been
aliased to <code>@id</code> and <code>@type</code>, and are marked <code>@protected</code>.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While this statement is correct, the usage of @protected is way more far-reaching than that: it is used to protect all terms define in this specification. That is what should be emphasized, not only these two imho.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, do you you have any concrete text, or maybe its better for us to just refer to https://www.w3.org/TR/json-ld11/#protected-term-definitions ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the JSON-LD spec alone is too terse for those who would really dive into this section. Something like

"Implementers should also be aware that all terms, defined in this specification, are marked as @protected [link to the JSON-LD spec], meaning that subsequent, application specific context files would not be able to change them."

I leave the details to native Anglo-Saxon speakers... :-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we addressed this, and we'll need to resolve things that are addressed so I can process the remaining blockers. can you confirm?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this addressed? The text has not changed...

To be more precise: in §6.1.1 of the spec, the role of @protected is indeed explained (in the last bullet), so is the fact that @id and @type are aliased. In fact, this paragraph does not add any information that would not be explicitly stated in §6.1.1. Maybe the best is to remove it altogether.

index.html Outdated Show resolved Hide resolved
index.html Outdated
<p>
Implementers should be aware that while naive JSON interpretation of claims might make it appear that claims are related,
the <code>https://www.w3.org/ns/credentials/v2</code> leverages <code>@container</code> and <code>@graph</code> to separate information graphs.
In particular, to separate the graphs related to <a>verifiable credentials</a>, <a>verifiable presentations</a>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK there is no property marked as "@container":"@graph" for VP-s. Only credentials and proofs are "separated" in their own graphs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What those mean is that a Verifiable Credential, resp. a proof, is put into a separate graph. You are right that this means, in practice, that the Verifiable Presentation that refers to the credential is therefore kept in its own (default) graph, but the sentence is nevertheless misleading.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its more misleading to ignore that RDF transformations make this distinction but JSON-LD (Without context processing) does not... how do you suggest we proceed?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove the reference to verifiable presentation. References to proofs and to verifiable credentials result in separate graphs, as you say in the text, and that is it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@OR13, thoughts?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is thread is actionable.

The context controls the shape of the RDF, not processing it (the latest version published by the authority, such as W3C) will lead to inconsistent processing.

I suggest we resolve this thread, or file seperate issues for follow up

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The context controls the shape of the RDF, not processing it (the latest version published by the authority, such as W3C) will lead to inconsistent processing.

I do not understand this statement.

Co-authored-by: Ivan Herman <[email protected]>
@OR13
Copy link
Contributor Author

OR13 commented Oct 9, 2023

In case you missed this, w3ctag/design-principles#239 (comment)

Very relevant to the guidance we are attempting on this PR, and on #1270

@OR13 OR13 mentioned this pull request Oct 9, 2023
@dlongley
Copy link
Contributor

dlongley commented Oct 9, 2023

In case you missed this, w3ctag/design-principles#239 (comment)

Very relevant to the guidance we are attempting on this PR, and on #1270

I don't think it's relevant, but I see how people could get confused that it is. Let's not add to that. Part of this confusion, I believe, comes from the fact that we keep trying to use "processing" terminology that is either under-defined or differs from how it is defined in the JSON-LD specification. This is one of Ivan's points above.

So, I think the most troublesome terminology is "JSON processing" and "JSON-LD processing", which everyone attributes their own (different) meaning to. We ought to avoid those terms. We are not presenting two different models for interpretation, just one. There can be different steps to get there based on localized optimizations, but whether you do steps X or Y, everyone must arrive at the same interpretation; that's what this section is about. This is a common concept for interfaces: the implementation details behind them should not matter.

In this section, we're telling people how to write software to get a consistent view of what pieces of the data they choose to consume. You can either consume JSON-LD by first transforming it to a context your application speaks (i.e., do "JSON-LD processing" as defined by the JSON-LD API spec by running compaction, etc.) or by limiting your inputs to data that your application natively accepts. Both of these must result in an interpretation that is according to "the context", whether that interpretation is hard-coded into your application like it is for every "application-specific vanilla JSON" consumer or if your application can work with multiple contexts and / or data shapes.

Your application either accepts only one specific expression that you know how to interpret properly and rejects everything else, or it accepts other expressions that it can transform, but once transformed, will result in the same interpretation as if it had started that way. The former is the only way software can be written when working with "application-specific vanilla JSON", because it does not have "transformation capabilities" (this is something that JSON-LD adds). But this is additive. You can still write applications that do not use these capabilities, you just can't pretend that you can interpret however you want. But that is never true with any format; you must always interpret according to the author's intent or else you will not interoperate. That the author can tag their data with machine-processable context helps widen where it can be used and accepted.

The fact that you can transform to another context doesn't mean you must nor does it mean that you must accept multiple expressions. In order to increase interoperability, the onus is on the producer to express the data in a way most will accept, which means in a way that simple applications that do not perform transforms will accept -- because those that do perform them can handle that expression too. The advantage for generalized processors is that they won't need to read your spec to transform to a context they accept -- because, with JSON-LD, your data carries that contextual information.

@iherman
Copy link
Member

iherman commented Oct 10, 2023

Adding to what @dlongley says:

In this section, we're telling people how to write software to get a consistent view of what pieces of the data they choose to consume. You can either consume JSON-LD by first transforming it to a context your application speaks (i.e., do "JSON-LD processing" as defined by the JSON-LD API spec by running compaction, etc.) or by limiting your inputs to data that your application natively accepts.

There is one aspect that should be emphasized: "limiting your inputs to data that your application natively accepts" means that what we are defining here is a processing strictly limited to Verifiable Credentials and Presentations. Ie, we have a general mechanism (JSON-LD) and we have a specialized version thereof for our purposes. That is why I wondered about the "Basic VC Processing" term for the latter. (I am not saying @dlongley did not put this in his comment, but it should be emphasized better nevertheless.)

Maybe an example from the past helps. In the Publication Manifest specification we defined manifest metadata in terms of JSON-LD (we did not even have a choice because we defined a profile and extension to schema.org). The dilemma about the general vs. specialized processing was the same. We were very meticulous, and we even defined the specialized processing steps of a manifest in great details (see §7.4 of that spec), and we also had two independent implementations for that algorithm to prove its correctness. I am not saying we have to go through the same level of details in the VCDM spec, but maybe it explains the general thought process.

@awoie
Copy link
Contributor

awoie commented Oct 10, 2023

IMO, nothing in the specification should suggest that JSON-only processors can be VCDM 2.0 conformant. Because this would mean all normative statements can be checked without RDF/JSON-LD processing. type having to be a URL is one example. To make this absolutely clear this should be explicitly put into the specification. Putting JSON-only processors into the mix is actually harmful.

@dlongley
Copy link
Contributor

@awoie,

IMO, nothing in the specification should suggest that JSON-only processors can be VCDM 2.0 conformant. Because this would mean all normative statements can be checked without RDF/JSON-LD processing.

I cannot understand the terms "JSON-only" or "RDF/JSON-LD processing" in the above. I do not think I have any robust shared understanding of what you mean by them. Instead of us trying to define these terms (which will inevitably differ in what seem to be important ways from how various people feel about them), we should just avoid them.

type having to be a URL is one example. To make this absolutely clear this should be explicitly put into the specification.

This speaks to my confusion. I'm guessing that some of the confusion comes from what happens in lexical space vs. value space and how we even talk about those things (or don't). JSON doesn't have a primitive for "URLs", it only has strings. So, from one perspective, there's no such thing as "JSON processing" of URLs ... or anything other than the core JSON primitive types. Is the suggestion that you can't put a URL in JSON? Clearly not. It's not the case that application developers have never created some JSON format that expresses a URL (as a JSON string).

From another perspective, "JSON processing" refers to reading a spec that says how to interpret everything you parse from a JSON string. "JSON processing", to me, refers to interpreting information expressed in JSON according to the author's very specific, idiosyncratic intent, as written in some spec they produced. Of course, this doesn't mean that authors can't reference, reuse, or rely upon other specs to describe their JSON format. For example, the JSON-LD spec. The number of specs you reference or build on top of certainly shouldn't change the definition of "JSON processing" or cause it to no longer apply.

The text this PR aims to improve should point out that relying on JSON-LD reduces idiosyncrasies (great value!) by requiring a tighter binding between the information model and JSON primitives (JSON objects really are objects in the data model and their JSON keys and values really are properties and values of those objects). It also enables expressing some remaining idiosyncrasies directly in the data via the @context property, to enable more generalized processing and tranformations to occur from one user's "context" to another's. It does all this, of course, by adding more spec text to read.

I agree that we shouldn't tell people NOT to read the specs VCDM builds on. But I also think we should cover common requirements directly in the VCDM spec, like around id and type. Sometimes we are more specific in the VCDM spec than the JSON-LD spec requires, further narrowing things for the purpose of increased interoperability with simpler applications.

But, in doing any of this, I feel we should avoid trying to define these "X processing" terms because I think that's likely to lead us further away from clarity than towards it. We should focus on specific items of concern, such as what you mentioned around type. VCs can use a type property and what its value can be is influenced by the @context property: valid values include simple strings that will be mapped to URLs via the @context or strings that express full URLs that would not be mapped (to the intended URL) by @context. This is the type of statement I'd expect to see in any "JSON author's spec" -- and if we're not saying that, we should say it more clearly. I don't think it's for this section, however, it should be said in the type section.

The goal of spelling that out in the type section should be to alleviate the need to read more referenced spec text if you want to only consume that property. And we should of course avoid creating any contradictions.

I'd rather us not further discuss specific properties here, though, to avoid derailing what I see as progress for this section, given all of the great suggestions above that we should get incorporated.

OR13 and others added 2 commits October 24, 2023 09:02
Co-authored-by: Manu Sporny <[email protected]>
Co-authored-by: Manu Sporny <[email protected]>
@OR13
Copy link
Contributor Author

OR13 commented Oct 24, 2023

@msporny thanks, that view is much easier, I think I covered everything. I did not merge all suggestions.

I suggest we resolve everything that we can resolve, so we can see what is still blocking.

index.html Outdated Show resolved Hide resolved
Copy link
Member

@msporny msporny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have resolved all conversations that I could. There are still a handful of statements that are vague or inaccurate. Removal of the changes, or the statements, would result in a +1 to merge the PR. I still believe this PR is moving in an acceptable direction.

index.html Outdated Show resolved Hide resolved
@iherman
Copy link
Member

iherman commented Oct 31, 2023

The issue was discussed in a meeting on 2023-10-31

  • no resolutions were taken
View the transcript

1.6. Propose better JSON-LD processing text (pr vc-data-model#1302)

See github pull request vc-data-model#1302.

Brent Zundel: I remember talking about this one recently and I thought we had a path forward.
… Have any steps been taken on that path?

Manu Sporny: Yes, steps are being taken towards the path -- Orie requested that those wanting changes ... resolve conversations that they feel are resolved. I certainly went through and did that. At this point Orie is not intending to accept a variety of suggestions that have been made.
… Speaking for myself, I'm not happy with the things that Orie is not wanting to change. And I say that -- kind of sad, because 85%-90% of the text that Orie is suggesting is good and should make it to the spec.
… This is one of those difficult ones.
… But if Orie is not going to pull back some of the changes, I don't agree to merging this.

Brent Zundel: Dave is pointing out in the chat that this is post-CR.

Ivan Herman: I put a number of comments into this one as well, I don't feel like I'm getting a reply from Orie. Or we are talking about different things, I certainly don't understand what he is saying. But I do agree it is post-CR.
… I am happy to take a deep breath and come back to it.

Brent Zundel: Ok, that's the course of action we're going to take and labeling it post-CR and hopefully in the intervening time we can come to an agreement on it.

@iherman
Copy link
Member

iherman commented Nov 1, 2023

The issue was discussed in a meeting on 2023-11-01

  • no resolutions were taken
View the transcript

2.5. JSON-only processors and VCDM 2.0 conformance (issue vc-data-model#1290)

See github issue vc-data-model#1290.

See github pull request vc-data-model#1302.

Brent Zundel: JSON only processor and VCDM 2.0 conformance. Raised by Oliver and not assigned to anyone. He brings up good points about how we define it in the spec. If no one gets assigned it can't move forward.

Dave Longley: "credential specific" is another to use if application-specific is too broad.

Manu Sporny: I'm wondering if we can get direction from the group. Its clear that people don't like the term "json processing". There have been two alternative terms proposed. We stop calling it JSON processing because certain workgroup members feel its misleading. Do we want to call it status versus dynamic or application specific versus generalized. Can we get feedback?

Dave Longley: or "context specific".

Ivan Herman: application specific is better than dynamic versus static. I think that application specific is really very broad. In our case what is relevant is that its credential specific. Its what we should do.

Brent Zundel: we do have a somewhat related PR open 1302 which is marked post-CR. Maybe it doesn't apply? Does it apply to this issue? How? As far as context specific, I agree that application specific doesn't tell me much. Credential specific might be better. But that begs the question that if there is credential specific processing, why haven't we defined it. I would expect that question to be asked.

Ivan Herman: +1 to be relevant to 1302.

Joe Andrieu: I think limited might be what we are talking about. We should do ranked poll and pick.

Manu Sporny: We could debate endlessly but a ranked choice poll would be a good option. I can put this out this week and run it for a week and we can review what comes back.

Dave Longley: some choices i heard: credential-specific, context-specific, application-specific, static vs. dynamic, limited, restricted.

Manu Sporny: Need to get all the options down. Please put your options into the minutes so it shows up in the poll.
… The other question was "is 1302 relevant". It is relevant but 1302 does other change beyond what we call these two things. By naming these two things the rest becomes easier to talk about.

Joe Andrieu: I think what we are trying to name is not application specific. Its the choice of the verifier what processing they want to do. I think we put the choice at the verifier. Its not a choice of the issuer, application or credential. Thats why they didn't resonate with me.

Dave Longley: Agree with Joe. We can fall into a lot of pitfalls if we specify the type of processing. Its really about the specific set of document that you accept. We don't want to confuse people into thinking that data would be understood in a different way. Its whether you accept a lot of things or just what you understand in your context.

Brent Zundel: We have another PR and poll going out. But no one is assigned to the issue.

Manu Sporny: I could pick it up once the poll is done and its clear what people want it to be changed to.

Ivan Herman: can we do it now.

Manu Sporny: Not sure over IRC. We have a ranked choice poll tool and I suggest we use it.

Brent Zundel: Look forward to seeing the poll. Moving on to the next issue.

index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
index.html Outdated Show resolved Hide resolved
@iherman
Copy link
Member

iherman commented Nov 14, 2023

The issue was discussed in a meeting on 2023-11-14

  • no resolutions were taken
View the transcript

1.3. Propose better JSON-LD processing text (pr vc-data-model#1302)

See github pull request vc-data-model#1302.

Brent Zundel: my understanding is that this is still not closer to consensus than it was. is this at all closer and worth talking about, or should we move on?
… moving on.

OR13 and others added 9 commits November 29, 2023 10:25
Co-authored-by: Manu Sporny <[email protected]>
Co-authored-by: Dave Longley <[email protected]>
Co-authored-by: Manu Sporny <[email protected]>
Co-authored-by: Manu Sporny <[email protected]>
Co-authored-by: Manu Sporny <[email protected]>
Co-authored-by: Manu Sporny <[email protected]>
Co-authored-by: Ted Thibodeau Jr <[email protected]>
Co-authored-by: Ted Thibodeau Jr <[email protected]>
Co-authored-by: Ted Thibodeau Jr <[email protected]>
@OR13
Copy link
Contributor Author

OR13 commented Nov 29, 2023

I accepts all suggestions, but I have lost the energy to continue to work on this, I suggest the PR be closed.

Hence, a JSON-LD document is both an RDF document and a JSON document and correspondingly represents an instance of an RDF data model.
</p>
<p>
RDF is a technology for modeling graphs of statements. Each statement is a single subject-property-value
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
RDF is a technology for modeling graphs of statements. Each statement is a single subject-property-value
RDF is a technology for modeling graphs of statements. Each statement is
a single <i>subject&#x2192;property&#x2192;value</i>
(also known as <i>entity&#x2192;attribute&#x2192;value</i>)

RDF is a technology for modeling graphs of statements. Each statement is a single subject-property-value
relationship, which is referred to as a <a>claim</a> in this specification. JSON-LD is a technology that
enables the expression of RDF using idiomatic JSON, enabling developers familiar with JSON to write
applications to consume RDF as JSON. In general, subjects are expressed as JSON objects with each property and value of the subject as a JSON key and value, respectively. A special affordance is made to express an identifier of a subject, if present, using the <code>@id</code> JSON key, which is aliased in this specification to <code>id</code>.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just line breaks

Suggested change
applications to consume RDF as JSON. In general, subjects are expressed as JSON objects with each property and value of the subject as a JSON key and value, respectively. A special affordance is made to express an identifier of a subject, if present, using the <code>@id</code> JSON key, which is aliased in this specification to <code>id</code>.
applications to consume RDF as JSON. In general, subjects are expressed as JSON objects
with each property and value of the subject as a JSON key and value, respectively. A
special affordance is made to express an identifier of a subject, if present, using
the <code>@id</code> JSON key, which is aliased in this specification to
<code>id</code>.

Comment on lines +4258 to +4266
As JSON can be used to express all different kinds of information, a consumer of a particular JSON
document can only properly interpret the authors intent if they possess information that contextualizes
and disambiguates it from other possible expressions. Information to assist with this interpretation can either
be wholly external to the JSON document or linked from within it. Compact JSON-LD documents include a <code>@context</code> property that internally expresses or links to contextual information and, per the JSON-LD specification, these documents follow object-oriented organizational principles to express <a>claims</a>. These features enable generalized processors to be written to convert JSON-LD documents from one context to another, but this is not needed when consumers receive JSON-LD documents that already use the context and shape that they expect. Authors of JSON-LD documents, such as <a>issuers</a> of <a>verifiable credentials</a>, are required to provide proper JSON-LD contexts and follow these rules in order to facilitate interoperability.
</p>
<p>
The text below helps consumers understand how to ensure a JSON-LD document is expressed in a context
and shape that their application already understands such that they do not need to transform it in
order to consume its contents. Notably, this does not mean that consumers do not need to understand any context at all, rather, consuming applications only need to understand a chosen set of contexts and document shapes to work with and not others. Issuers can publish contexts and information about their <a>verifiable credentials</a> to aid consumers who do not use generalized processors, just like would be done with any other JSON-formatted data.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just line breaks

Suggested change
As JSON can be used to express all different kinds of information, a consumer of a particular JSON
document can only properly interpret the authors intent if they possess information that contextualizes
and disambiguates it from other possible expressions. Information to assist with this interpretation can either
be wholly external to the JSON document or linked from within it. Compact JSON-LD documents include a <code>@context</code> property that internally expresses or links to contextual information and, per the JSON-LD specification, these documents follow object-oriented organizational principles to express <a>claims</a>. These features enable generalized processors to be written to convert JSON-LD documents from one context to another, but this is not needed when consumers receive JSON-LD documents that already use the context and shape that they expect. Authors of JSON-LD documents, such as <a>issuers</a> of <a>verifiable credentials</a>, are required to provide proper JSON-LD contexts and follow these rules in order to facilitate interoperability.
</p>
<p>
The text below helps consumers understand how to ensure a JSON-LD document is expressed in a context
and shape that their application already understands such that they do not need to transform it in
order to consume its contents. Notably, this does not mean that consumers do not need to understand any context at all, rather, consuming applications only need to understand a chosen set of contexts and document shapes to work with and not others. Issuers can publish contexts and information about their <a>verifiable credentials</a> to aid consumers who do not use generalized processors, just like would be done with any other JSON-formatted data.
As JSON can be used to express all different kinds of information, a consumer of
a particular JSON document can only properly interpret the authors intent if they
possess information that contextualizes and disambiguates it from other possible
expressions. Information to assist with this interpretation can either be wholly
external to the JSON document or linked from within it. Compact JSON-LD documents
include a <code>@context</code> property that internally expresses or links to
contextual information and, per the JSON-LD specification, these documents follow
object-oriented organizational principles to express <a>claims</a>. These features
enable generalized processors to be written to convert JSON-LD documents from one
context to another, but this is not needed when consumers receive JSON-LD documents
that already use the context and shape that they expect. Authors of JSON-LD
documents, such as <a>issuers</a> of <a>verifiable credentials</a>, are required
to provide proper JSON-LD contexts and follow these rules in order to facilitate
interoperability.
</p>
<p>
The text below helps consumers understand how to ensure a JSON-LD document is
expressed in a context and shape that their application already understands
such that they do not need to transform it in order to consume its contents.
Notably, this does not mean that consumers do not need to understand any
context at all, rather, consuming applications only need to understand a chosen
set of contexts and document shapes to work with and not others. Issuers can
publish contexts and information about their <a>verifiable credentials</a> to
aid consumers who do not use generalized processors, just like would be done
with any other JSON-formatted data.

<a href="#data-schemas"></a>.
</li>
<li>
When serializing or deserializing <a>verifiable credentials</a> or
<a>verifiable presentations</a> into systems that store or index their contents.
<a>verifiable presentations</a> into systems that store or index data in
Turtle [[?TURTLE]], N-Quads [[?N-QUADS]], or JSON-LD [[?JSON-LD11]].
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Turtle [[?TURTLE]], N-Quads [[?N-QUADS]], or JSON-LD [[?JSON-LD11]].
Turtle [[?TURTLE]], N-Quads [[?N-QUADS]], JSON-LD [[?JSON-LD11]], or other
RDF serializations.

That is, JSON processing is allowed as long as the document being consumed or
produced is a <a>conforming document</a>. If JSON processing is desired, an
implementer is advised to follow the following rule:
Instead of performing general JSON-LD processing, application-specific processing is allowed as long as the document being consumed or
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a line break

Suggested change
Instead of performing general JSON-LD processing, application-specific processing is allowed as long as the document being consumed or
Instead of performing general JSON-LD processing, application-specific
processing is allowed as long as the document being consumed or

Comment on lines +4334 to +4335
mechanism. With respect to JSON-LD-based processors, this is achieved in a
lightweight manner, without having to use RDF processing libraries.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
mechanism. With respect to JSON-LD-based processors, this is achieved in a
lightweight manner, without having to use RDF processing libraries.
mechanism. JSON-LD-based processors achieve this in a lightweight manner,
without having to use RDF processing libraries.

Comment on lines +4342 to +4343
not all Holders or Verifiers will understand expressions of intention that are only visible
after RDF processing has occured.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
not all Holders or Verifiers will understand expressions of intention that are only visible
after RDF processing has occured.
some <a>holders</a> and <a>verifiers</a> will not understand expressions of
intention that are only visible after RDF processing has occured.

Comment on lines +4346 to +4347
Some of the best features of JSON-LD are only possible by applying its language features to the
terms defined in the <code>@context</code>.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Some of the best features of JSON-LD are only possible by applying its language features to the
terms defined in the <code>@context</code>.
Some of the most powerful features of JSON-LD are only accessible by applying
its language features to the terms mapped in the <code>@context</code>.

<a href="https://www.w3.org/TR/json-ld11/#specifying-the-type">Specifying the Type</a>.
</p>
<p>
Implementers are advised that object member, and array element order are not preserved by default in JSON-LD,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Implementers are advised that object member, and array element order are not preserved by default in JSON-LD,
Implementers are advised that JSON-LD does not preserve the orders of
object members or array elements by default;

Comment on lines +4378 to +4380
be aware of these differences, and confusion between the intentions of the
<a>issuer</a> and the <a>holder</a> could lead to unexpected processing
behavior by <a>verifiers</a>.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
be aware of these differences, and confusion between the intentions of the
<a>issuer</a> and the <a>holder</a> could lead to unexpected processing
behavior by <a>verifiers</a>.
be aware of these differences, and confusion about the intentions of
<a>issuers</a> and/or <a>holders</a> could lead to unexpected processing
behavior by subsequent <a>holder</a> and/or <a>verifiers</a>, respectively.

@brentzundel
Copy link
Member

It is my understanding that this PR has been somewhat overtaken by events. Some of the changes sought here have already been included in other PRs, and there are a large number of conflicts that would need to be addressed in order to see what the remaining set of proposed changes looks like.

I am marking this PR pending close and will give it a week to see if anyone steps forward to try and clean it up in a way that is more likely to gain consensus.

@brentzundel brentzundel added the pending close Close if no objection within 7 days label Jan 17, 2024
@BigBlueHat
Copy link
Member

@brentzundel here is a PR that resolves the conflicts and brings in the text with @TallTed's suggestions:
#1413

@brentzundel
Copy link
Member

Closing in favor of #1413

@msporny msporny deleted the recommending-safe-json-ld-processing branch May 27, 2024 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pending close Close if no objection within 7 days post-CR
Projects
None yet
Development

Successfully merging this pull request may close these issues.