Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

metric_set() has a less useful error message when no estimate is provided for classification metric sets #504

Open
mikemahoney218 opened this issue Apr 28, 2024 · 1 comment
Open

metric_set() has a less useful error message when no estimate is provided for classification metric sets #504

mikemahoney218 opened this issue Apr 28, 2024 · 1 comment
Labels
feature a feature request or enhancement

Comments

@mikemahoney218
Copy link
Member

The problem

For classification metrics, when truth or estimate is not a factor, metric_set() gives a much less informative error than the classification metric itself.

Reproducible example

yardstick::precision(Orange, age, Tree)
#> Error in `yardstick::precision()`:
#> ! `truth` should be a factor, not a a double vector.
#> Backtrace:
#>      ▆
#>   1. ├─yardstick::precision(Orange, age, Tree)
#>   2. └─yardstick:::precision.data.frame(Orange, age, Tree)
#>   3.   └─yardstick::class_metric_summarizer(...)
#>   4.     ├─rlang::inject(...)
#>   5.     ├─base::withCallingHandlers(...)
#>   6.     └─yardstick (local) fn(...)
#>   7.       └─yardstick::check_class_metric(...)
#>   8.         └─yardstick:::validate_factor_truth_factor_estimate(...)
#>   9.           └─cli::cli_abort(...)
#>  10.             └─rlang::abort(...)
yardstick::metric_set(yardstick::precision)(Orange, age, Tree)
#> Error in `metric_set()`:
#> ! Failed to compute `precision()`.
#> Caused by error:
#> ! Must select at least one item.
#> Backtrace:
#>      ▆
#>   1. ├─(yardstick::metric_set(yardstick::precision))(Orange, age, Tree)
#>   2. │ └─base::mapply(...)
#>   3. │   └─yardstick (local) `<fn>`(dots[[1L]][[1L]], dots[[2L]][[1L]])
#>   4. │     ├─base::tryCatch(...)
#>   5. │     │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
#>   6. │     │   └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
#>   7. │     │     └─base (local) doTryCatch(return(expr), name, parentenv, handler)
#>   8. │     └─rlang::eval_tidy(expr, data = data, env = env)
#>   9. ├─yardstick (local) `<clss_mtr>`(...)
#>  10. └─yardstick:::precision.data.frame(...)
#>  11.   └─yardstick::class_metric_summarizer(...)
#>  12.     └─yardstick:::yardstick_eval_select(...)
#>  13.       └─tidyselect::eval_select(...)
#>  14.         └─tidyselect:::eval_select_impl(...)
#>  15.           ├─tidyselect:::with_subscript_errors(...)
#>  16.           │ └─base::withCallingHandlers(...)
#>  17.           └─tidyselect:::vars_select_eval(...)
#>  18.             └─tidyselect:::check_empty(pos, allow_empty, call = error_call)
#>  19.               └─cli::cli_abort("Must select at least one item.", call = call)
#>  20.                 └─rlang::abort(...)

yardstick::precision(Orange, Tree, age)
#> Error in `yardstick::precision()`:
#> ! `estimate` should be a factor, not a a double vector.
#> Backtrace:
#>      ▆
#>   1. ├─yardstick::precision(Orange, Tree, age)
#>   2. └─yardstick:::precision.data.frame(Orange, Tree, age)
#>   3.   └─yardstick::class_metric_summarizer(...)
#>   4.     ├─rlang::inject(...)
#>   5.     ├─base::withCallingHandlers(...)
#>   6.     └─yardstick (local) fn(...)
#>   7.       └─yardstick::check_class_metric(...)
#>   8.         └─yardstick:::validate_factor_truth_factor_estimate(...)
#>   9.           └─cli::cli_abort(...)
#>  10.             └─rlang::abort(...)
yardstick::metric_set(yardstick::precision)(Orange, Tree, age)
#> Error in `metric_set()`:
#> ! Failed to compute `precision()`.
#> Caused by error:
#> ! Must select at least one item.
#> Backtrace:
#>      ▆
#>   1. ├─(yardstick::metric_set(yardstick::precision))(Orange, Tree, age)
#>   2. │ └─base::mapply(...)
#>   3. │   └─yardstick (local) `<fn>`(dots[[1L]][[1L]], dots[[2L]][[1L]])
#>   4. │     ├─base::tryCatch(...)
#>   5. │     │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
#>   6. │     │   └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
#>   7. │     │     └─base (local) doTryCatch(return(expr), name, parentenv, handler)
#>   8. │     └─rlang::eval_tidy(expr, data = data, env = env)
#>   9. ├─yardstick (local) `<clss_mtr>`(...)
#>  10. └─yardstick:::precision.data.frame(...)
#>  11.   └─yardstick::class_metric_summarizer(...)
#>  12.     └─yardstick:::yardstick_eval_select(...)
#>  13.       └─tidyselect::eval_select(...)
#>  14.         └─tidyselect:::eval_select_impl(...)
#>  15.           ├─tidyselect:::with_subscript_errors(...)
#>  16.           │ └─base::withCallingHandlers(...)
#>  17.           └─tidyselect:::vars_select_eval(...)
#>  18.             └─tidyselect:::check_empty(pos, allow_empty, call = error_call)
#>  19.               └─cli::cli_abort("Must select at least one item.", call = call)
#>  20.                 └─rlang::abort(...)

Created on 2024-04-28 with reprex v2.0.2

Session info 
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.0 (2023-04-21)
#>  os       macOS 14.4.1
#>  system   aarch64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       America/Chicago
#>  date     2024-04-28
#>  pandoc   3.1.12.3 @ /Applications/Positron.app/Contents/Resources/app/bin/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package      * version    date (UTC) lib source
#>  class          7.3-22     2023-05-03 [1] CRAN (R 4.3.0)
#>  cli            3.6.2.9000 2024-04-04 [1] Github (r-lib/cli@bcb5c78)
#>  codetools      0.2-19     2023-02-01 [1] CRAN (R 4.3.0)
#>  colorspace     2.1-0      2023-01-23 [1] CRAN (R 4.3.0)
#>  data.table     1.14.8     2023-02-17 [1] CRAN (R 4.3.0)
#>  dials          1.2.0      2023-04-03 [1] CRAN (R 4.3.0)
#>  DiceDesign     1.9        2021-02-13 [1] CRAN (R 4.3.0)
#>  digest         0.6.33     2023-07-07 [1] CRAN (R 4.3.0)
#>  dplyr          1.1.4      2023-11-17 [1] CRAN (R 4.3.1)
#>  evaluate       0.21       2023-05-05 [1] CRAN (R 4.3.0)
#>  fansi          1.0.6      2023-12-08 [1] CRAN (R 4.3.1)
#>  fastmap        1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
#>  foreach        1.5.2      2022-02-02 [1] CRAN (R 4.3.0)
#>  fs             1.6.3      2023-07-20 [1] CRAN (R 4.3.0)
#>  furrr          0.3.1      2022-08-15 [1] CRAN (R 4.3.0)
#>  future         1.33.0     2023-07-01 [1] CRAN (R 4.3.0)
#>  future.apply   1.11.0     2023-05-21 [1] CRAN (R 4.3.0)
#>  generics       0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
#>  ggplot2        3.4.2      2023-04-03 [1] CRAN (R 4.3.0)
#>  globals        0.16.2     2022-11-21 [1] CRAN (R 4.3.0)
#>  glue           1.7.0      2024-01-09 [1] CRAN (R 4.3.1)
#>  gower          1.0.1      2022-12-22 [1] CRAN (R 4.3.0)
#>  GPfit          1.0-8      2019-02-08 [1] CRAN (R 4.3.0)
#>  gtable         0.3.3      2023-03-21 [1] CRAN (R 4.3.0)
#>  hardhat        1.3.1      2024-02-02 [1] CRAN (R 4.3.1)
#>  htmltools      0.5.6      2023-08-10 [1] CRAN (R 4.3.0)
#>  ipred          0.9-14     2023-03-09 [1] CRAN (R 4.3.0)
#>  iterators      1.0.14     2022-02-05 [1] CRAN (R 4.3.0)
#>  knitr          1.43       2023-05-25 [1] CRAN (R 4.3.0)
#>  lattice        0.22-6     2024-03-20 [1] CRAN (R 4.3.1)
#>  lava           1.7.2.1    2023-02-27 [1] CRAN (R 4.3.0)
#>  lhs            1.1.6      2022-12-17 [1] CRAN (R 4.3.0)
#>  lifecycle      1.0.4      2023-11-07 [1] CRAN (R 4.3.1)
#>  listenv        0.9.0      2022-12-16 [1] CRAN (R 4.3.0)
#>  lubridate      1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
#>  magrittr       2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
#>  MASS           7.3-60.0.1 2024-01-13 [1] CRAN (R 4.3.1)
#>  Matrix         1.6-5      2024-01-11 [1] CRAN (R 4.3.1)
#>  munsell        0.5.0      2018-06-12 [1] CRAN (R 4.3.0)
#>  nnet           7.3-19     2023-05-03 [1] CRAN (R 4.3.0)
#>  parallelly     1.36.0     2023-05-26 [1] CRAN (R 4.3.0)
#>  parsnip        1.1.0      2023-04-12 [1] CRAN (R 4.3.0)
#>  pillar         1.9.0      2023-03-22 [1] CRAN (R 4.3.0)
#>  pkgconfig      2.0.3      2019-09-22 [1] CRAN (R 4.3.0)
#>  prodlim        2023.03.31 2023-04-02 [1] CRAN (R 4.3.0)
#>  purrr          1.0.2      2023-08-10 [1] CRAN (R 4.3.0)
#>  R.cache        0.16.0     2022-07-21 [1] CRAN (R 4.3.0)
#>  R.methodsS3    1.8.2      2022-06-13 [1] CRAN (R 4.3.0)
#>  R.oo           1.25.0     2022-06-12 [1] CRAN (R 4.3.0)
#>  R.utils        2.12.2     2022-11-11 [1] CRAN (R 4.3.0)
#>  R6             2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
#>  Rcpp           1.0.12     2024-01-09 [1] CRAN (R 4.3.1)
#>  recipes        1.0.6      2023-04-25 [1] CRAN (R 4.3.0)
#>  reprex         2.0.2      2022-08-17 [1] CRAN (R 4.3.0)
#>  rlang          1.1.3.9000 2024-04-04 [1] Github (r-lib/rlang@baedaca)
#>  rmarkdown      2.24       2023-08-14 [1] CRAN (R 4.3.0)
#>  rpart          4.1.19     2022-10-21 [1] CRAN (R 4.3.0)
#>  rsample        1.1.1      2022-12-07 [1] CRAN (R 4.3.0)
#>  scales         1.2.1      2022-08-20 [1] CRAN (R 4.3.0)
#>  sessioninfo    1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
#>  styler         1.10.1     2023-06-05 [1] CRAN (R 4.3.0)
#>  survival       3.5-5      2023-03-12 [1] CRAN (R 4.3.0)
#>  tibble         3.2.1      2023-03-20 [1] CRAN (R 4.3.0)
#>  tidyr          1.3.0      2023-01-24 [1] CRAN (R 4.3.0)
#>  tidyselect     1.2.1      2024-03-11 [1] CRAN (R 4.3.1)
#>  timechange     0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
#>  timeDate       4022.108   2023-01-07 [1] CRAN (R 4.3.0)
#>  tune           1.1.1      2023-04-11 [1] CRAN (R 4.3.0)
#>  utf8           1.2.4      2023-10-22 [1] CRAN (R 4.3.1)
#>  vctrs          0.6.5      2023-12-01 [1] CRAN (R 4.3.1)
#>  withr          3.0.0      2024-01-16 [1] CRAN (R 4.3.1)
#>  workflows      1.1.3      2023-02-22 [1] CRAN (R 4.3.0)
#>  xfun           0.40       2023-08-09 [1] CRAN (R 4.3.0)
#>  yaml           2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
#>  yardstick      1.3.1.9000 2024-04-28 [1] Github (tidymodels/yardstick@be744a3)
#> 
#>  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────
@mikemahoney218 mikemahoney218 mentioned this issue Apr 28, 2024
Closed
@mikemahoney218
Copy link
Member Author

Sorry, I'm realizing I misdiagnosed the issue here -- I think the real problem is that I didn't name the estimate argument (got too used to the signature for numeric metric sets!). I still the error message here could be more clear about what we need to do to select at least one item, though!

yardstick::metric_set(yardstick::precision)(Orange, age, estimate = Tree)
#> Error in `metric_set()`:
#> ! Failed to compute `precision()`.
#> Caused by error:
#> ! `truth` should be a factor, not a a double vector.
#> Backtrace:
#>      ▆
#>   1. ├─(yardstick::metric_set(yardstick::precision))(Orange, age, estimate = Tree)
#>   2. │ └─base::mapply(...)
#>   3. │   └─yardstick (local) `<fn>`(dots[[1L]][[1L]], dots[[2L]][[1L]])
#>   4. │     ├─base::tryCatch(...)
#>   5. │     │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
#>   6. │     │   └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
#>   7. │     │     └─base (local) doTryCatch(return(expr), name, parentenv, handler)
#>   8. │     └─rlang::eval_tidy(expr, data = data, env = env)
#>   9. ├─yardstick (local) `<clss_mtr>`(...)
#>  10. └─yardstick:::precision.data.frame(...)
#>  11.   └─yardstick::class_metric_summarizer(...)
#>  12.     ├─rlang::inject(...)
#>  13.     ├─base::withCallingHandlers(...)
#>  14.     └─yardstick (local) fn(...)
#>  15.       └─yardstick::check_class_metric(...)
#>  16.         └─yardstick:::validate_factor_truth_factor_estimate(...)
#>  17.           └─cli::cli_abort(...)
#>  18.             └─rlang::abort(...)

Created on 2024-04-28 with reprex v2.0.2

@mikemahoney218 mikemahoney218 changed the title metric_set() has a less useful error message than classification metric functions when truth or estimate is not a factor metric_set() has a less useful error message when no estimate is provided Apr 29, 2024
@mikemahoney218 mikemahoney218 changed the title metric_set() has a less useful error message when no estimate is provided metric_set() has a less useful error message when no estimate is provided for classification metric sets Apr 29, 2024
@EmilHvitfeldt EmilHvitfeldt added the feature a feature request or enhancement label Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature a feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@EmilHvitfeldt @mikemahoney218