Depending on pyyaml can make things slow.

[jonathan-s]

I ran some profiling on our code that runs pysigma. It turns out that a significant amount of time is spent in pyyaml. Would you be open to accept a PR using? https://pypi.org/project/ryaml/

I'm thinking that the best solution, if it turns out to be faster, would be to have pip install pysigma use pyyaml and pip install pysigma[rust] (or similar) to use ryaml.

[jonathan-s]

I have confirmed that the part parsing and loading yaml have completely disappeared, and takes a lot less time using ryaml.

[thomaspatzke]

Sounds great! It really takes a while to parse the whole SigmaHQ rule repository, so if there's the possibility to speed up, lets do it!

One thing that's customized with YAML parsing is the check for duplicate keys, an error that is sometimes made in writing Sigma rules. Is this something that can also be done with ryaml or is it possibly already there?

[jonathan-s]

@thomaspatzke, one thing that I bumped into, that you may have some insight into. There are several versions of the yaml spec. Version 1.1 uses on, yes and represents these as booleans. Version 1.2 of that yaml spec drops these and only uses true | True | TRUE | false | False | FALSE. The ryaml package only supports version 1.2 of the yaml spec.

Do you have any hunch to what extent this is a common practice to use the prior writing of booleans among people who write sigma rules?
 
Either way, that is one change that will be made if a user chooses to use the rust version.

[thomaspatzke]

I think we should stick on the 1.2 standard and only accept the boolean values supported there. I just see the version 2 specification already refers to YAML 1.2.2, therefore a rule that is compliant to the specification shouldn't contain version 1.1 YAML.