Skip to content

Commit

Permalink
Merge pull request #252 from stan-dev/update-psis-paper-reference
Browse files Browse the repository at this point in the history
update psis paper reference
  • Loading branch information
jgabry committed Mar 2, 2024
2 parents 0d21e4d + 3dd00e6 commit 219be0d
Show file tree
Hide file tree
Showing 9 changed files with 16 additions and 14 deletions.
6 changes: 3 additions & 3 deletions R/diagnostics.R
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#' `threshold` value, or plot observation indexes vs. diagnostic estimates.
#' The **Details** section below provides a brief overview of the
#' diagnostics, but we recommend consulting Vehtari, Gelman, and Gabry (2017)
#' and Vehtari, Simpson, Gelman, Yao, and Gabry (2022) for full details.
#' and Vehtari, Simpson, Gelman, Yao, and Gabry (2024) for full details.
#'
#' @name pareto-k-diagnostic
#' @param x An object created by [loo()] or [psis()].
Expand All @@ -23,7 +23,7 @@
#' parameter \eqn{k} of the generalized Pareto distribution. The
#' diagnostic threshold for Pareto \eqn{k} depends on sample size
#' \eqn{S} (sample size dependent threshold was introduced by Vehtari
#' et al. (2022), and before that fixed thresholds of 0.5 and 0.7 were
#' et al. (2024), and before that fixed thresholds of 0.5 and 0.7 were
#' recommended). For simplicity, `loo` package uses the nominal sample
#' size \eqn{S} when computing the sample size specific
#' threshold. This provides an optimistic threshold if the effective
Expand Down Expand Up @@ -99,7 +99,7 @@
#' obtain the samples from the proposal distribution via MCMC the **loo**
#' package also computes estimates for the Monte Carlo error and the effective
#' sample size for importance sampling, which are more accurate for PSIS than
#' for IS and TIS (see Vehtari et al (2022) for details). However, the PSIS
#' for IS and TIS (see Vehtari et al (2024) for details). However, the PSIS
#' effective sample size estimate will be
#' **over-optimistic when the estimate of \eqn{k} is greater than**
#' \eqn{min(1-1/log10(S), 0.7)}, where \eqn{S} is the sample size.
Expand Down
2 changes: 1 addition & 1 deletion R/loo-glossary.R
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@
#'
#' The diagnostic threshold for Pareto \eqn{k} depends on sample size
#' \eqn{S} (sample size dependent threshold was introduced by Vehtari
#' et al., 2022, and before that fixed thresholds of 0.5 and 0.7 were
#' et al., 2024, and before that fixed thresholds of 0.5 and 0.7 were
#' recommended). For simplicity, `loo` package uses the nominal sample
#' size \eqn{S} when computing the sample size specific
#' threshold. This provides an optimistic threshold if the effective
Expand Down
4 changes: 2 additions & 2 deletions R/loo-package.R
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
#' *Stan Development Team*
#'
#' This package implements the methods described in Vehtari, Gelman, and
#' Gabry (2017), Vehtari, Simpson, Gelman, Yao, and Gabry (2022), and
#' Gabry (2017), Vehtari, Simpson, Gelman, Yao, and Gabry (2024), and
#' Yao et al. (2018). To get started see the **loo** package
#' [vignettes](https://mc-stan.org/loo/articles/index.html), the
#' [loo()] function for efficient approximate leave-one-out
Expand All @@ -33,7 +33,7 @@
#' fast and stable computations for approximate LOO-CV laid out in Vehtari,
#' Gelman, and Gabry (2017). From existing posterior simulation draws, we
#' compute LOO-CV using Pareto smoothed importance sampling (PSIS; Vehtari,
#' Simpson, Gelman, Yao, and Gabry, 2022), a new procedure for stabilizing
#' Simpson, Gelman, Yao, and Gabry, 2024), a new procedure for stabilizing
#' and diagnosing importance weights. As a byproduct of our calculations,
#' we also obtain approximate standard errors for estimated predictive
#' errors and for comparing of predictive errors between two models.
Expand Down
4 changes: 2 additions & 2 deletions R/loo.R
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
#' CV, efficient approximate leave-one-out (LOO) cross-validation for Bayesian
#' models using Pareto smoothed importance sampling ([PSIS][psis()]). This is
#' an implementation of the methods described in Vehtari, Gelman, and Gabry
#' (2017) and Vehtari, Simpson, Gelman, Yao, and Gabry (2022).
#' (2017) and Vehtari, Simpson, Gelman, Yao, and Gabry (2024).
#'
#' @export loo loo.array loo.matrix loo.function
#' @param x A log-likelihood array, matrix, or function. The **Methods (by class)**
Expand Down Expand Up @@ -500,7 +500,7 @@ mcse_elpd <- function(ll, lw, E_elpd, r_eff, n_samples = NULL) {
FUN.VALUE = numeric(1),
FUN = function(i) {
# Variance in linear scale
# Equation (6) in Vehtari et al. (2022)
# Equation (6) in Vehtari et al. (2024)
var_epd_i <- sum(w2[, i] * (lik[, i] - E_epd[i]) ^ 2) / r_eff[i]
# Compute variance in log scale by match the variance of a
# log-normal approximation
Expand Down
2 changes: 1 addition & 1 deletion R/loo_model_weights.R
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
#' Model averaging via stacking of predictive distributions, pseudo-BMA
#' weighting or pseudo-BMA+ weighting with the Bayesian bootstrap. See Yao et
#' al. (2018), Vehtari, Gelman, and Gabry (2017), and Vehtari, Simpson,
#' Gelman, Yao, and Gabry (2022) for background.
#' Gelman, Yao, and Gabry (2024) for background.
#'
#' @export
#' @param x A list of `"psis_loo"` objects (objects returned by [loo()]) or
Expand Down
2 changes: 1 addition & 1 deletion R/psis.R
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
#' Implementation of Pareto smoothed importance sampling (PSIS), a method for
#' stabilizing importance ratios. The version of PSIS implemented here
#' corresponds to the algorithm presented in Vehtari, Simpson, Gelman, Yao,
#' and Gabry (2022).
#' and Gabry (2024).
#' For PSIS diagnostics see the [pareto-k-diagnostic] page.
#'
#' @export
Expand Down
2 changes: 1 addition & 1 deletion R/waic.R
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
#' @seealso
#' * The __loo__ package [vignettes](https://mc-stan.org/loo/articles/) and
#' Vehtari, Gelman, and Gabry (2017) and Vehtari, Simpson, Gelman, Yao,
#' and Gabry (2022) for more details on why we prefer `loo()` to `waic()`.
#' and Gabry (2024) for more details on why we prefer `loo()` to `waic()`.
#' * [loo_compare()] for comparing models on approximate LOO-CV or WAIC.
#'
#' @references
Expand Down
5 changes: 3 additions & 2 deletions man-roxygen/loo-and-compare-references.R
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@
#' ([journal version](https://link.springer.com/article/10.1007/s11222-016-9696-4),
#' [preprint arXiv:1507.04544](https://arxiv.org/abs/1507.04544)).
#'
#' Vehtari, A., Simpson, D., Gelman, A., Yao, Y., and Gabry, J. (2022).
#' Pareto smoothed importance sampling.
#' Vehtari, A., Simpson, D., Gelman, A., Yao, Y., and Gabry, J. (2024).
#' Pareto smoothed importance sampling. *Journal of Machine Learning Research*,
#' accepted for publication.
#' [preprint arXiv:1507.02646](https://arxiv.org/abs/1507.02646)
#'
#' Sivula, T, Magnusson, M., Matamoros A. A., and Vehtari, A. (2022).
Expand Down
3 changes: 2 additions & 1 deletion man-roxygen/loo-and-psis-references.R
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
#' [preprint arXiv:1507.04544](https://arxiv.org/abs/1507.04544)).
#'
#' Vehtari, A., Simpson, D., Gelman, A., Yao, Y., and Gabry, J. (2022).
#' Pareto smoothed importance sampling.
#' Pareto smoothed importance sampling. *Journal of Machine Learning Research*,
#' accepted for publication.
#' [preprint arXiv:1507.02646](https://arxiv.org/abs/1507.02646)
#'

0 comments on commit 219be0d

Please sign in to comment.