Metric wishlist

[msto]

Hey @nh13 @tfenne ,

I'd like to consolidate a wishlist of features/improvements to the Metric class.

Their absence has blocked the use of Metric in client contexts (of note, #103 being blocked required me to re-implement much of the Metric functionality from scratch on a client project), and I'd greatly appreciate their inclusion in the fgpyo roadmap.

In rough order of perceived priority:

• Metric should not be dependent on pysam.
  • It would be helpful if Metric (and other non-pysam-dependent modules) could be spun out into a separate package, or if pysam could be made an optional dependency.
• Add MetricWriter to permit streamed writing of metrics #88
• Metric.write should support excluding attributes #90
• feat: Support reading Picard-style metrics files with comments above the header #103
• Metric.read should support PEP 604 syntax for Optional types #89
  • I think this would require dropping support for 3.8/3.9 and don't know if it's viable
• Metric.read and Metric.write should be context managers #70

I've written proof-of-concept solutions to these in https://github.com/msto/dataclass_io/, and am curious which might be considered for adoption by fgpyo. I would very much like to see Metric become the general solution for serializing/deserializing dataclasses in our client repos, but I've been hesitant to open additional PRs on fgpyo to address these issues due to lack of clarity around which of these features are compatible with the vision for Metric and fgpyo.

[tfenne]

@msto I have no objections to any of the issues linked; I might have input on implementation of those but I'm 100% on board with the functionality being added. With the possible exception of #89 if we really can't find a way to make it back-compat, or guarded by a check to see if the python version in use is high enough to support it.

Re pysam ... I am also on board with finding a way to use the rest of fgpyo without forcing the pysam dependency. I don't know enough about how optional dependencies work in python/pypi (and by extension conda). That said, I think I could easily 👍 a solution that looks something like:

• Move pysam to a dev dependency
• Clearly state in docs that fgpyo.sam requires a version of pysam >= x.y.z
• In the __init__.py for the sam module (or similar place) check to see if pysam is available and if it is not then raise an exception with a super-clear warning about what's going on

I think that would be preferable to trying to split fgpyo in two, but I'm open to being convinced of better solutions.

[msto]

Fantastic!

I'll open a number of PRs over the next few weeks with candidate implementations.

Re: pysam, I'm on board with your solution. The only modification I'd propose would be to make pysam its own extra/optional dependency, instead of adding it to the list of dev dependencies. (e.g. pip install fgpyo[pysam]) That way the user doesn't have to pull in all of the dev dependencies to access the sam functionality.

This is straightforward with pip/poetry - I can look into if/how conda supports it.

https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#optional-dependencies
https://python-poetry.org/docs/managing-dependencies/#optional-groups

[msto]

conda/conda#3299
conda/conda#7502

Looks like a conda solution may take some consideration

[tfenne]

I was thinking that rather than pull in the dev dependencies, users would just add pysam to their own dependencies and that way they also get to pick their preferred version.