Source-root confusion in presence of scalameta macro annotations

[ryan-williams]

Repro

git clone [email protected]:hammerlab/iterators.git
cd iterators
sbt clean coverage core/test core/coverageReport
…
[info] Waiting for measurement data to sync...
[info] Reading scoverage instrumentation [/Users/ryan/c/hl/iterator/core/target/scala-2.11/scoverage-data/scoverage.coverage.xml]
[info] Reading scoverage measurements...
[info] Generating scoverage reports...
[info] Written Cobertura report [/Users/ryan/c/hl/iterator/core/target/scala-2.11/coverage-report/cobertura.xml]
java.lang.RuntimeException: No source root found for '/Users/ryan/c/hl/iterator/<macro>' (source roots: '/Users/ryan/c/hl/iterator/core/src/main/scala/')
	at scoverage.report.BaseReportWriter.relativeSource(BaseReportWriter.scala:28)
	at scoverage.report.BaseReportWriter.relativeSource(BaseReportWriter.scala:16)
	at scoverage.report.CoberturaXmlWriter.klass(CoberturaXmlWriter.scala:42)
	at scoverage.report.CoberturaXmlWriter$$anonfun$pack$1.apply(CoberturaXmlWriter.scala:66)
	at scoverage.report.CoberturaXmlWriter$$anonfun$pack$1.apply(CoberturaXmlWriter.scala:66)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.immutable.List.foreach(List.scala:318)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.AbstractTraversable.map(Traversable.scala:105)
	at scoverage.report.CoberturaXmlWriter.pack(CoberturaXmlWriter.scala:66)
	at scoverage.report.CoberturaXmlWriter$$anonfun$xml$3.apply(CoberturaXmlWriter.scala:90)
	at scoverage.report.CoberturaXmlWriter$$anonfun$xml$3.apply(CoberturaXmlWriter.scala:90)
…

Full stack trace here

Discussion

This repo has two modules, core and macros. The latter defines a scalameta macro annotation that causes implicit conversions into a class to be synthesized, and the former uses that annotation to generate some code.

Some previous discussion at scoverage/sbt-coveralls#102

[gslowikowski]

First, I'm surprised nobody found this issue before. meta macros are available since the beginning of Scala 2.11 line.

I've tried to learn meta macros quickly. Found simple example to test Scoverage. @mappable annotation generates toMap method, but unfortunately removes all other methods. I'd like not to loose my code and see it covered by tests on Scoverage report.

@ryan-williams, first please check your project with my PR branch of Scoverage to verify that it works properly.
Additionally, could you provide simple test project? Ideally I would like to have unit test here and scripted test in sbt-scoverage. Maybe both of them can be created from one common code base.

Thank you in advance.

[ryan-williams]

Thanks again @gslowikowski, sorry for the delay.

I still see the error occurring with your PR branch. Here is a simple repro repo:

git clone [email protected]:ryan-williams/scala-bugs.git
cd scala-bugs
git checkout scov
sbt clean coverage test coverageReport
…
[info] All tests passed.
[success] Total time: 12 s, completed Nov 19, 2017 8:20:02 PM
[info] Waiting for measurement data to sync...
[info] Reading scoverage instrumentation [/Users/ryan/c/scalac-bug/target/scala-2.11/scoverage-data/scoverage.coverage.xml]
[info] Reading scoverage measurements...
[info] Generating scoverage reports...
java.lang.RuntimeException: No source root found for '/Users/ryan/c/scalac-bug/<macro>' (source roots: '/Users/ryan/c/scalac-bug/src/main/scala/')
	at scoverage.report.BaseReportWriter.relativeSource(BaseReportWriter.scala:28)
	at scoverage.report.BaseReportWriter.relativeSource(BaseReportWriter.scala:16)
	at scoverage.report.CoberturaXmlWriter.klass(CoberturaXmlWriter.scala:42)
	at scoverage.report.CoberturaXmlWriter$$anonfun$pack$1.apply(CoberturaXmlWriter.scala:66)
	at scoverage.report.CoberturaXmlWriter$$anonfun$pack$1.apply(CoberturaXmlWriter.scala:66)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.immutable.List.foreach(List.scala:318)

This uses my IteratorOps macro which generates implicit conversions from an Iterable, Iterator, or Array into an instance of an Ops-class which takes a single Iterator ctor argument.

I'm not sure why your fix doesn't work yet; am hoping to dig in as well but haven't had time yet. I was looking at the same part of Transformer.process as you modified though, so maybe there is hope for me to help 😃.

I'd like not to loose my code and see it covered by tests on Scoverage report.

Agreed that porting coverage/location information from code that gets moved/rearranged in a meta macro would be great; in particular in my case above it is important as all the interesting code I'd like coverage info about is in the body of a macro-tagged class, and gets wrapped in an extra trait layer but ends up looking similar (identical?) to how it was originally written.

Really appreciate your efforts here, hoping I can help more / more quickly if you continue to make progress!

[gslowikowski]

Thanks, I will check this tomorrow.

[gslowikowski]

You had to do something wrong. My fix works.

After publishing locally scalac-scoverage-plugin from my branch, add coverageScalacPluginVersion := "1.4.0-SNAPSHOT". You can use any of the latest sbt-scoverage plugin with this setting (I've tested with 1.5.1).
If you want to test with sbt-scoverage built from sources with changed default value of this setting, you have to change it in two places: build.sbt and ScoverageSbtPlugin.scala.

Anyway, I have zero instrumented statements in your project.

[ryan-williams]

ah, I had not changed the ScoverageSbtPlugin.scala value; thanks for pointing that out, and thanks a lot for the quick fix!

Unfortunately, libraries that I've been using such macros in are mostly comprised of code contained in the macro-annotated classes, so dropping those lines limits the usefulness of coverage metrics.

I'm guessing it's a significant project, but if you want to brainstorm how to measure macro-generated lines, I'm interested to try to help.

It seems to me coverage-instrumentation has to happen on post-macro-expansion code, otherwise positions/coverage seem ill-defined (for example, what location would be reported/highlighted for a quasiquote-generated line that is covered in one generated class and not another)?

I dug at it a bit tonight and the most interesting thing I came up with is this fork of scalameta/paradise that just hard-codes the original source path in place of the <macro> string; that was enough to get a report submitted to coveralls, but there are some conflicting / meaningless numbers floating around that I don't have a handle on.

Compilation and coverageReport output:

[info] [info] Instrumentation completed [7 statements]
…
[info] Statement coverage.: 42.86%
[info] Branch coverage....: 100.00%

which seems promising! And the "statement list" at target/scala-2.11/scoverage-report/index.html seems consistent and based on macro-generated code:

However, the "Code Grid" shows the pre-macro file, as does the Coveralls build.

Not sure when I'll have time to keep pluggin on this but if you have any thoughts I'd love to hear them!

[gslowikowski]

Ryan, sorry for not responding, but I have no time for scoverage recently.

What I can say now is:

• coverage instrumentation is performed after marco expansion
• what you see on statement list is based on real information scoverage plugin is getting from scala during compilation
• source file does not reflect the state of the code during compilation and I don't know, what we can do about it (even with the tricks like yours in paradise library)

Generally speaking generated code cannot be shown because it never had a source lines form.
We could try to save somewhere the result of toString called on generated statements, but it does not look good (I use it sometimes during debugging).

I will try to experiment more, when I will have more time.

[ryan-williams]

That sounds good, thanks @gslowikowski!

I'm relatively optimistic about some of these ideas; in my case the macro annotates a class that is the only top-level declaration in the file, so replacing the file contents wholesale with the post-macro-expansion code – and displaying/using that throughout coverage reporting – seems reasonable.

Additionally, the paradise hack I attempted at least convinced me that getting access to the post-macro code as a string isn't too hard. The fact that the "statement list" in the web UI seemed to have worked above is also encouraging.

I'm hoping I can find some time to push on it again as well, thanks!