Add feature to run testitems

[jwortmann]

This is intended to adopt the feature recently added in VSCode allowing to run individual @testitem blocks in Julia files from the editor UI.

Work in progress.

Todo:

• Prevent reset of all testitem results whenever the file is edited
• Add command with quick panel in command palette to show results of testitems and to run a particular testitem
• Allow to cancel a currently running testitem (maybe)

Preview:

testitem.webm

[ghyatzo]

Hello, I have been fiddling a bit with this branch. It's very nice, but I believe I have encountered a pretty serious issue (and it is why I believe in VS they use much more sophisticated parsing through TestItemRunner.jl

I have a dummy module with

module TestModule
using TestItems

function foo(x)
    return x*x
end

@testitem "Test for foo" begin
    x = foo("bar")

    @test x == "barbar"
    @test 1 + 1 == 2
end
end

(which is the example from TestItemRunner.jl package). But when I try to run the test It errors out with a ERROR: LoadError: UndefVarError: foo not defined

I believe that there is an issue of scope. More precisely, we do import the main package but the line
code = string('\n'^params.line, ' '^params.column, params.code) does not prepend unexported symbols with TestModule.

Changing x = foo("bar") to x = TestModule.foo("bar") does not incur in this issue and the test finishes correctly.

So either we put a disclaimer and tell users they have to explicitly call methods from the module, or we must find a way to "inherit" the scope of the calling @testitem block.

EDIT: I took a look at the TestitemRunner.jl internals more closely and I have to admit I really don't see the bit where the parent scope seems to be taken into account...

EDIT2: They dont, that example in the starting page is hella misleading... I didn't look close enough before lol

@testitem "compute_line_column" begin
    content = "abc\ndef\nghi"

    @test TestItemRunner.compute_line_column(content, 1) == (line=1, column=1)
    @test TestItemRunner.compute_line_column(content, 2) == (line=1, column=2)
    @test TestItemRunner.compute_line_column(content, 3) == (line=1, column=3)
    @test TestItemRunner.compute_line_column(content, 5) == (line=2, column=1)
    @test TestItemRunner.compute_line_column(content, 6) == (line=2, column=2)
    @test TestItemRunner.compute_line_column(content, 7) == (line=2, column=3)
    @test TestItemRunner.compute_line_column(content, 9) == (line=3, column=1)
    @test TestItemRunner.compute_line_column(content, 10) == (line=3, column=2)
    @test TestItemRunner.compute_line_column(content, 11) == (line=3, column=3)
end

(from within their own source code.

[ghyatzo]

Another small issue, inspired by julia-vscode/TestItemRunner.jl#28, is that the working directory of the code in the @testitem body is not the same directory as the file which contain the test item.

try

@testitem "Same Directory" begin
	@test pwd() == @__DIR__  

        # pwd() = os.path.join(cls.storage_path(), "LSP-julia", "testrunner")
        # @__DIR__ = /path/to/directory/that/contains/the/file/with/this/testitem/
end # error

@testitem "Force Same Directory" begin
	cd(@__DIR__)
	@test pwd() == @__DIR__
end # pass

[jwortmann]

Thanks!

The working directory should be fixed now.

But on Windows, the example @test pwd() == @__DIR__ still fails, because @__DIR__ seems to return a lowercase drive letter here. I'm not sure what exactly happens during the testitem evaluation, but I assume this is a result of a uri2filepath conversion function which I've copied from the VSCode extension, and unfortunately the LSP client in VSCode uses and expects lowercase drive letters from language servers. So I guess that this function is responsible that for the @__DIR__ being evaluated as a path with lowercase drive letter here.

It can be fixed by using

@testitem "Same Directory" begin
    @test pwd() == realpath(@__DIR__)
end

but I'm unsure whether this driveletter casing mismatch has any relevance in practice, so I'll leave it like that for now.

---

Regarding your first comment, perhaps you need to export foo in order to be able to use it directly without the module name prefix?

[ghyatzo]

Yeah exporting would also do the trick, but I also thought that it is not really realistic to expect that a package would export all the methods it is trying to test.

The example with soft-scope behaviour was in the main package README so i thought that was the case, but by looking at the code I could not really find what would enable that behaviour. The showcasing example from the video works like that, but probably he just exported the function he tests.
I was thinking of opening an issue and ask directly if that behaviour is some kind of long term goal not yet implemented or it was never intended and it is just me on a trip. I don't really know how it really behaves in VSCode though.

[jwortmann]

I've tried the example in VS Code, and I got the same error.
So I guess you are right and the example from the TestItemRunner.jl README is misleading, and one either needs to export all relevant methods or explicitly use the module name as a prefix.

[jwortmann]

I think this works quite well now. I will merge it and publish a new release soon. Then we can see if any missed bugs emerge.

[ghyatzo]

Did not want to open another issue. I thought that it would be useful to mention in the readme section on testitems the discussion about scope we talked about.

I can make a PR in the afternoon.