Proposal: Structured Logging

[gasi]

First of all, thanks for this useful library!

We use monad-logger for logging in our system. We are now moving towards logging structured data using JSON, e.g.

Example

Before

$(logInfo) ("Identifying user token " <> showt jwt <> " resulted in " <> showt res)

{
  "time": "2018-09-24T17:43:31.761781Z",
  "type": "app",
  "level": "info",
  "message": "Identifying user token Token {getToken = \"eyJ0...\"} resulted in IRKnown \"cvid|9eb...\"",
  "location": {
    "file": "src/UserService/Token.hs:41:7",
    "module": "UserService.Token",
    "package": "user-service-1.0.0-7qau..."
  }
}

After

$(logInfo) "Identify user token" ["jwt" .= jwt, "result" .= res]

{
  "time": "2018-09-25T15:05:17.044166Z",
  "type": "app",
  "level": "info",
  "message": "Identify user token",
  "meta": {
    "jwt": {
      "getToken": "eyJ0..."
    },
    "result": {
      "contents": {
        "unCVID": "cvid|9eb..."
      },
      "tag": "IRKnown"
    }
  },
  "location": {
    "file": "src/UserService/Token.hs:40:7",
    "module": "UserService.Token",
    "package": "user-service-1.0.1-6Ss..."
  }
}

However, the current implementation of monad-logger doesn’t support this well as only a single ToLogStr message can be logged.

---

Proposal (draft)

Would you be open to extend the type-class to support structured logging, e.g.

-- Before
class Monad m => MonadLogger m where
    monadLoggerLog :: ToLogStr msg => Loc -> LogSource -> LogLevel -> msg -> m ()

-- After
class Monad m => MonadLogger m where
    monadLoggerLog :: ToLogStr msg => Loc -> LogSource -> LogLevel -> msg -> m ()
    monadLoggerLogJSON :: (ToLogStr msg, ToJSON meta) => Loc -> LogSource -> LogLevel -> msg -> meta -> m ()
    monadLoggerLogJSON loc src lvl msg meta = monadLoggerLog loc src lvl (toLogStr msg <> encode meta) {- untested -}

Prior Art

There are existing libraries such as monad-log that offer something similar but we have found them harder to integrate as they were missing some of the monad instances that monad-logger offers and were generally not as mature.

Workarounds

These are potential workarounds for the current setup:

• Give up on pluggable formatters and loggers. Do all the work in a custom logging function and simply send the output LogStr to monad-logger to reuse its monad instances.
• Fork monad-logger and extend it to support structured data.

Neither of these are very appealing.

Conclusion

As systems grow, logging becomes more critical. Aggregators such as Splunk, SumoLogic, etc. offer support for structured logs, e.g. JSON. monad-logger should support it out-of-the-box as well. The proposal above is one approach and feedback is welcome. If there is interest to add this to monad-logger, I’d be happy to contribute a PR. Thanks!

[snoyberg]

Some questions:

• This looks like a backwards change, ami right?
• Will this also include the addition of new logging functions to the library?
• Did you consider putting this method into a new subclass instead? What are some of the design tradeoffs going on here?

[gasi]

@snoyberg Thanks for the quick response!

Re: questions.

• Yes, the idea would be to make this backwards compatible.
• Yes, we could introduce new logging functions, e.g. adopt FPCO’s logDebugJ, logInfoJ, etc. functions that were exposed via monad-logger-json but with the ability to fully control the JSON output: https://github.com/fpco/monad-logger-json
• Re: subclass. Would that mean we’d have to write new monad transformer instances for new subclass? The idea here was to reuse as much of the existing code as possible.

Disclaimer: I’m an intermediate Haskeller and somewhat comfortable using monad transformers but don’t have experience writing them yet. I’d definitely appreciate some guidance. At the same time, our logging work has been lowered in priority so I’m not sure when I’ll be able to dedicate more time on this. There was also a recent announcement of co-log which looks like it has some interesting ideas we might explore: https://kowainik.github.io/posts/2018-09-25-co-log

The purpose of this issue was to assess the need / desire for structured logging on your end and bounce around ideas before starting to work on implementation 😄

[snoyberg]

Regardless of whether we do a subclass, I think you're going to need a new concrete monad transformer type to allow you the flexibility to provide your own special handling of JSON values.

[gasi]

@snoyberg Would you mind elaborating on that or give a small example?

[snoyberg]

To see it, try implementing the monadLoggerLogJSON method you proposed for the LoggingT datatype in monad-logger.