[test] Add jsontestsuite

[ellen364]

This is a hacky way of running nst/JSONTestSuite. It's a starting point for conversation, not something that's mergeable.

Run using python3 jsontestsuite/run_tests.py which produces a log.txt and 2x html files in results/

The original Python script assumes a particular directory structure and presence of certain files when producing results.

If one library is tested, the results only show failures and implementation specific tests. (The original Python script wants to compare results between libraries.) A simple way to get more output was to run another library (e.g. Bash) to compare and one that isn't installed (e.g. Awk, which the script assumes will be in a specific directory) and will always fail.

[andychu]

Thanks for figuring this out! That kind of thing is why I didn't get around to it myself :-/

I have to review some PRs for Melvin tonight

In the meantime, if you think the failure is a real bug (sounds like it), please file an issue on github, showing how Oils diverges from the JSON spec (and it would be interesting to see that Python/JavaScript conforms, etc.)

I linked this comment in the Zulip thread, and I'm more than happy to honor it - https://lobste.rs/s/1y9hjq/regular_json#c_8tvoie

[ellen364]

I've written it up as #2026.

[andychu]

This is excellent, thank you!

Actually I changed it to use the C++ version and it found an overflow bug too ... for big positive and negative integers

I am going to fix all these :)

[andychu]

Hm since this found some bugs, I wonder how test_transform works ...

there are fewer cases there

https://github.com/nst/JSONTestSuite/tree/master/test_transform

[andychu]

Hm somehow these other cases aren't automated in the original repo?

andy@hoover:~/git/oilshell/JSONTestSuite$ grep test_parsing run_tests.py 
TEST_CASES_DIR_PATH = os.path.join(BASE_DIR, "test_parsing")

andy@hoover:~/git/oilshell/JSONTestSuite$ grep transform run_tests.py