add Laplace approximation

[st--]

This should be sufficiently ready for a proper PR. It supports the AbstractGPs API, has an example notebook, and some docstrings. All of it I'm sure we can still improve on. :)

[trappmartin]

Would it be possible to add some inline documentation on the structs and functions?

Also, possibly a stupid question but does this do a Gauss-Newton approximation to the Hessian? I wasn't sure from the code what is going on.

[theogf]

Looks good! I have got to say I thought Laplace approx was easier to do 😆
More generally do you think it would make sense to rearrange the functions going from the most public ones to the most internal ones? Right now it's a bit tricky to navigate

[theogf]

You should avoid using @assert and do something like mean(fx) == zero(mean(fx)) || error("Non Zero-Mean prior not supported yet")

[willtebbutt]

(The reason being that @assert can be disabled sometimes -- the docstring for @assert is quite informative in this regard :) )

[st--]

yeaaah that's actually an argument for keeping it as an assert 😅

[willtebbutt]

Haha fair enough. I have no strong opinion either way!

[theogf]

Also, possibly a stupid question but does this do a Gauss-Newton approximation to the Hessian? I wasn't sure from the code what is going on.

From what I understood from the code, the hessian is just inv(K) + Diagonal(d^2 log p(y|f) / df^2) (with some minuses missing) so the second derivatives of the log likelihood are computed exactly with ForwardDiff

[willtebbutt]

Broadly LGTM. Have answered the remaining questions posed in the other PR as well.

[st--]

@theogf

More generally do you think it would make sense to rearrange the functions going from the most public ones to the most internal ones? Right now it's a bit tricky to navigate

I kinda did it the other way around - starting with the basic building blocks and have each function's dependencies above it.

Would you still want to keep separation between training & prediction?

[theogf]

@theogf

More generally do you think it would make sense to rearrange the functions going from the most public ones to the most internal ones? Right now it's a bit tricky to navigate

I kinda did it the other way around - starting with the basic building blocks and have each function's dependencies above it.

Would you still want to keep separation between training & prediction?

So what I had in mind was to have AbstractGPs.posterior(la::LaplaceApproximation, lfx::LatentFiniteGP, ys), approx_lml, build_laplace_objective and other high-end functions at the beginning and then have other internal functions in the order of their calls.

[st--]

@theogf: I reordered it; happy now?:)

[theogf]

I just have a couple of minor comments, but otherwise it looks great :D