analysis

[KiaraGrouwstra]

analysis on synthesis:

• for each task fn I think the stats I may want include: total samples, number correct/wrong, ratio correct/wrong, then the number / ratio over all task fns.

see Jupyter PoC - reported usage issues with this here.

out of scope:

visualization:

• hyper-parameter vis:
  • seaborn grids
  • MS Power BI (needs Windows VM)
  • alternatives
  • sherpa (python)

analysis on generated datasets:

• further analysis -- I imagine a lot of factors may influence how much my synthesizer can learn from types, including perhaps number of types used, generics used, typeclasses / constraints used, arities of all these... if I can find the patterns here it will likely help find a dataset demonstrating added value of this additional info.
• per dataset, visualize distribution of sample i/o string length, so I can visually determine a sweet spot cut-off string length (esp. round numbers e.g. powers of 2) so as to statically determine a limited maximum string length (and corresponding dataset length) that will still allow most of the samples...

plot results for analysis

• hasktorch tensorboard issue
• regular plotting libraries in haskell seem inferior to seaborn so maybe just dump stats to csv then use that (old code: 1, 2, 3, 4)