"Finishing" the grammar

[larsgw]

Some suggestions for the grammar. I applied the following changes and ran it on the examples.

• For decimal, something like below. The first rule all depends on what kind of decimals you want to support, but I think this should cover everything except 1. and .1.

  decimal := integer ("." [0-9]+)? exponent?
  exponent := ("e" | "E") integer
  integer := sign? [1-9] [0-9_]*
  sign := "+" | "-"
  

• Regular strings:

  string := '"' ("\\" ["\\] | [^"])* '"'
  

• I am a bit unsure how strict the node whitespace rules are but something like below. This has the "command" followed by any number of prop | values (separated by whitespace with one optional escaped newline), then optionally followed by whitespace with one optional escaped newline and a nested document. I also moved the trailing newline to document because otherwise all files would have to end with a newline (I guess that's preferred but maybe not mandatory?)

  node := identifier (node-space node-argument)* (node-space node-document)?
  node-argument := prop | value
  node-document := '{' linespace* document linespace* '}'
  node-space := ws* escline ws* | ws+
  

• For the comments maybe something like this (note: more stuff needed if */ should be able to be escaped):

  linespace := newline | ws | single_line_comment
  ws := ' ' | '\t' | multi_line_comment
  single_line_comment := '//' [^\n]+ newline
  multi_line_comment := '/*' ('*' [^\/] / [^*])* '*/'
  

• The character range of identifier is not complete when comparing to the example (missing the hat, single quote, pipe and angle brackets)
• Something like this for document? I don't like the double ? but I have not found anything else yet.

  document := linespace* (node (newline document)?)?
  

As for the raw string, I wanted to remove the *n to keep the grammar of the grammar simple but that's not very easy without backtracking. I thought something like this but that does not work.

raw-string := "r" raw-string-hash
raw-string-hash := "#" raw-string-hash "#" | raw-string-quotes
raw-string-quotes := '"' .* '"'

[larsgw]

Also I am not entirely sure how well the string approaches work with emojis etc., they seem to break off and I don't know if they can be put back together again after.

[larsgw]

kach/nearley#81 (comment) maybe has a solution.

[larsgw]

It does actually work in nearley though, even the r# strings (though not the emojis yet):

document -> linespace:* (node (newline document):?):?

node -> identifier (node_space node_argument):* (node_space node_document):?
node_argument -> prop | value
node_document -> "{" linespace:* document linespace:* "}"
node_space -> ws:* escline ws:* | ws:+

identifier -> [a-zA-Z] [a-zA-Z0-9~!@#$%\^&*\-_+.:'|<>?//]:* | string
prop -> identifier "=" value
value -> string | raw_string | number | boolean | "null"

string -> "\"" ("\\" [\"\\] | [^"]):* "\""

raw_string -> "r" raw_string_hash
raw_string_hash -> "#" raw_string_hash "#" | raw_string_quotes
raw_string_quotes -> "\"" .:* "\""

number -> decimal | hex | octal | binary

decimal -> integer ("." [0-9]:+):? exponent:?
exponent -> ("e" | "E") integer
integer -> sign:? [1-9] [0-9_]:*
sign -> "+" | "-"

hex -> "0x" [0-9a-fA-F] [0-9a-fA-F_]:*
octal -> "0o" [0-7] [0-7_]:*
binary -> "0b" ("0" | "1") ("0" | "1" | "_"):*

boolean -> "true" | "false"

escline -> "\\" newline
linespace -> newline | ws | single_line_comment
newline -> ("\r" "\n") | "\n"
ws -> " " | "\t" | multi_line_comment
single_line_comment -> "//" [^\n]:+ newline
multi_line_comment -> "/*" ("*" [^\/] | [^*]):* "*/"

[larsgw]

Nearley can also generate example files lol:

	""/**/\
/**D*/"B"=0o3623	/**/\
/**/null 	/*f*/ev=true	{//N*\*
}

//T
""
l		 \
""=r#"1"#\
	{ /**Pz*/	 //:@
}

Can there be nodes without arguments? It kind of makes sense but I did not see it in the examples.

[zkat]

yes, there can!

[zkat]

omg thanks so much for doing this! This is awesome. I'm amazed that my random grammar syntax "mostly" just worked with nearley!

Would you like to PR this grammar, so we can start iterating further on it?