Example instances apply the hook too broadly

[sellout]

I opened see hspec/hspec#850, which captures this, but I removed hspec-golden from the example (thankfully preserving the bug).

Here is hopefully an equivalent version that shows hspec-golden’s issue:

.golden/should_fail/golden:

success content

golden-test.hs:

import Test.Hspec (around_, describe, hspec, it)
import Test.Hspec.Golden (defaultGolden)
import Test.Mockery.Directory (inTempDirectory)

main :: IO ()
main = hspec $ do
  describe "golden manual" $
    it "should fail" $
      inTempDirectory $ do
        -- do something in the temp directory
        defaultGolden "should_fail" "failure content"
  describe "golden around" $ around_ inTempDirectory $
    it "should fail" $ do
      -- do something in the temp directory
      defaultGolden "should_fail" "failure content"

I expect these two test cases to fail (as “failure content” does not match “success content”).

The first case fails as expected, but the second succeeds claiming something like “first run, creating golden file”. But the golden file is created in the temp directory, and the one that exists in the repo gets ignored.

@sol posted a solution to my non-golden example in hspec/hspec#850 (comment). The gist is that the Example instances do too much in the function passed to hook (action in the hspec-golden code). So it should look something like:

instance Eq str => Example (arg -> IO (Golden str)) where
  type Arg (arg -> IO (Golden str)) = arg
  evaluateExample golden _ action _ = do
    ref <- newIORef ""
    action $ writeIORef ref <=< golden
    r <- runGolden =<< readIORef ref
    pure $ fromGoldenResult r

(but I haven’t tested that)

[CristhianMotoche]

Hello @sellout Thanks for reporting this issue. I'll wrap my head around it during this week to understand the issue and the possible solution that you mentioned. Thanks!