Host web5 vectors example

[phoebe-lew]

closes #62

[mistermoe]

@phoebe-lew mind if i add a did:jwk vector for an invalid did?

[andresuribe87]

It's unclear to me how errors is meant to work still. Specifically,

• when errors is true, what does the value of output do?
• what do errors translate to in each implementation?

My suggestion is to remove the errors field. Output can encode that an error is expected.

[mistermoe]

when errors is true, what does the value of output do?

you can omit output entirely if errors is true. that's why neither is a required field. if output is present, you can omit errors

what do errors translate to in each implementation?

nothing is prescribed here. errors: true effectively means the code in question should return or throw an error/exception.

general reason for keeping this here is to account for functions that don't return anything on success or throw exceptions

My suggestion is to remove the errors field. Output can encode that an error is expected.

can you provide a concrete example? we're looking to avoid magic strings that mean specific things as much as possible

[andresuribe87]

I was thinking of something along the lines of

{
  "output": {
    "type": "error",
    "message": "my expected error message"
  }
}

Regarding

we're looking to avoid magic strings that mean specific things as much as possible

Can you share the motivation? My first instinct is that "error" is already having a specific meaning. Is the line at the top level?

Having a Boolean field can be too vague to be useful in tests. Implementations may easily return a different type of error than what was intended, so I believe it's important to allow that flexibility so implementations can check error messaging.

If we decide that the current schema is the way to go, then I would suggest enforcing that only one of errors, output, can be populated, but not both. This can be done with a oneOf.

[mistermoe]

Can you share the motivation? My first instinct is that "error" is already having a specific meaning. Is the line at the top level?

was careful to say "avoid magic strings as much as possible" vs. at all because, like you said, the property names themselves are "magic". so more generally, just aiming to limit the use of them as much as possible regardless of level.

having a "type" provides quite a bit of flexibility, which, i'm not sure we necessarily need. could end up with an explosion of custom "types" when it doesn't seem needed (at least imo). doesn't feel like we need to go further than supporting:

• the function works and returns something that should match output
• the function throws an error (in whatever way the consuming language represents errors)
  • optionally, the error's output should match output

errors: true effectively means: expect an error in the consuming language.

• so in kotlin or js, whenever errors: true, an assertThrows would be added.
• in go i suppose you'd expect err to not be nil.
• in Rust, you'd want the match to occur on Result.Err instead of Result.Ok. it's intentionally vague so as not to be too prescriptive for any given implementation.
• I tried to provide a COBOL example but Chad made it apparent that COBOL tests are entirely separate programs that have to be written to run on mainframes.

If we decide that the current schema is the way to go, then I would suggest enforcing that only one of errors, output, can be populated, but not both. This can be done with a oneOf.

If we decide that the current schema is the way to go, then I would suggest enforcing that only one of errors, output, can be populated, but not both. This can be done with a oneOf.

i considered this as well initially, but decided not to make them mutually exclusive in order to provide the optionality of using output for error messages if desired

[andresuribe87]

This all makes sense to me. Thanks for elaborating.

I would only add that the way to use error and output should be documented in the description field. Specifically the stuff we've talked about in this thread: "when errors is set to true, the output field can be used for further validation. For example, when doing error message validation."

[mistermoe]

b14de73

[mistermoe]

🏅 excited about this