90 average calibration functions in utils.jl #97

pasq-cat · 2024-06-22T09:29:08Z

added functions in utils.jl for computing the average empirical frequency and changed glm_predictive_distribution to return a normal distribution in case of regression. it should pass all tests

…operly due to the issue with mljflux.

…dict

…operly due to the issue with mljflux.

…dict

…m/JuliaTrustworthyAI/LaplaceRedux.jl into 90-average-calibration-in-utilsjl

src/baselaplace/predicting.jl

src/utils.jl

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions · 2024-06-22T09:42:42Z

src/utils.jl

+
+
+


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions · 2024-06-22T09:44:35Z

src/baselaplace/predicting.jl

+    normal_distr = [
+        Distributions.Normal(fμ[i], fstd[i]) for i in 1:size(fμ, 1)] 
+    return (normal_distr,fμ,fvar )


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

normal_distr = [

Distributions.Normal(fμ[i], fstd[i]) for i in 1:size(fμ, 1)]

return (normal_distr,fμ,fvar )

normal_distr = [Distributions.Normal(fμ[i], fstd[i]) for i in 1:size(fμ, 1)]

return (normal_distr, fμ, fvar)

github-actions · 2024-06-22T09:44:36Z

src/baselaplace/predicting.jl

@@ -75,11 +83,12 @@ predict(la, hcat(x...))
 function predict(
    la::AbstractLaplace, X::AbstractArray; link_approx=:probit, predict_proba::Bool=true
 )
-    fμ, fvar = glm_predictive_distribution(la, X)
+    normal_distr,fμ, fvar = glm_predictive_distribution(la, X)


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

normal_distr,fμ, fvar = glm_predictive_distribution(la, X)

normal_distr, fμ, fvar = glm_predictive_distribution(la, X)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions · 2024-06-22T09:45:50Z

src/baselaplace/predicting.jl

@@ -75,11 +82,12 @@ predict(la, hcat(x...))
 function predict(
    la::AbstractLaplace, X::AbstractArray; link_approx=:probit, predict_proba::Bool=true
 )
-    fμ, fvar = glm_predictive_distribution(la, X)
+    normal_distr,fμ, fvar = glm_predictive_distribution(la, X)


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

normal_distr,fμ, fvar = glm_predictive_distribution(la, X)

normal_distr, fμ, fvar = glm_predictive_distribution(la, X)

src/utils.jl

yeah basically juliaformatter being a pain for nothing Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

pat-alt

Really nice work here @Rockdeldiablo 👍🏽 just a few missing pieces:

check individual comments (minor things)
add a tutorial to the documentation
add unit tests (at least passing through functions and having simple tests for expected output type, then new issue for testing with synthetic data).

pat-alt · 2024-07-01T06:00:29Z

src/baselaplace/predicting.jl

I guess we could just keep this consistent and return everything in both cases?

Edit: my bad, let's indeed as discussed just add an option for classification to return distribution. By default, we should still return probabilities for now, but at least we give the option and add that to the docstring.

pat-alt · 2024-07-01T06:07:22Z

src/utils.jl

@@ -39,3 +40,128 @@ corresponding to the number of neurons on the last layer of the NN.
 function outdim(model::Chain)::Number
    return [size(p) for p in Flux.params(model)][end][1]
 end
+
+""" 


Nice detail here! Just a formatting thing: can you use maths notation that's recognized by Documenter.jl please? See here for an example and here for the docs.

i fixed the docstrings but for a new line ( without the empy line in the middle), i had to use the \ character.

i checked the results in the julia repl but i was not able to make the function appear in the documentation. i tried to go in docs/ and then type julia make.jl and i got "[ Info: SetupBuildDirectory: setting up build directory.
[ Info: Doctest: running doctests.
[ Info: ExpandTemplates: expanding markdown templates.
[ Info: CrossReferences: building cross-references.
[ Info: CheckDocument: running document checks.
[ Info: Populate: populating indices.
[ Info: RenderDocument: rendering document.
[ Info: HTMLWriter: rendering HTML pages.
┌ Warning: Unable to determine the repository root URL for the navbar link.
│ This can happen when a string is passed to the repo keyword of makedocs.
│
│ To remove this warning, either pass a Remotes.Remote object to repo to completely
│ specify the remote repository, or explicitly set the remote URL by setting repolink
│ via makedocs(format = HTML(repolink = "..."), ...).
└ @ Documenter.HTMLWriter C:\Users\Pasqu.julia\packages\Documenter\qoyeC\src\html\HTMLWriter.jl:732
[ Info: Automatic version="0.2.1" for inventory from ..\Project.toml
┌ Warning: Documenter could not auto-detect the building environment. Skipping deployment.
└ @ Documenter C:\Users\Pasqu.julia\packages\Documenter\qoyeC\src\deployconfig.jl:76 "
so i opened index.html with edge through vs code and my functions do not appear in the documentations when i search for them. do i have to modify some sort of flag?

pat-alt · 2024-07-01T06:10:53Z

src/utils.jl

+    return sharpness
+end
+
+""" 


Same as above comment on maths notation

pat-alt

Regarding your question on docs, just make sure that the function signature stated in the docstring matches the function

pat-alt · 2024-07-02T06:23:58Z

src/utils.jl

 """
 function sharpness_regression(sampled_distributions)
    sharpness = mean(var.(sampled_distributions))
    return sharpness
 end

-""" 
+@doc raw""" 
    empirical_frequency-classification(y_binary, sampled_distributions)


The dash should be an underscore

pat-alt · 2024-07-02T06:24:14Z

src/utils.jl

@@ -144,21 +148,22 @@ function empirical_frequency_binary_classification(
    return (num_p_per_interval, emp_avg, bin_centers)
 end

-""" 
-    sharpness-classification(y_binary,sampled_distributions)


Same as above, dash should be an underscore

…ests.jl to add the new file.

github-actions · 2024-07-02T15:36:25Z

src/utils.jl

+    - `bin_centers`: array with the centers of the bins. 
+
+"""
+function empirical_frequency_binary_classification(y_binary, sampled_distributions, n_bins::Int=20)


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

function empirical_frequency_binary_classification(y_binary, sampled_distributions, n_bins::Int=20)

function empirical_frequency_binary_classification(

y_binary, sampled_distributions, n_bins::Int=20

)

github-actions · 2024-07-02T15:36:25Z

test/calibration.jl

+            0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8;
+            0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8;

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2

github-actions · 2024-07-02T15:36:25Z

test/calibration.jl

+    mean_class_one, mean_class_zero = sharpness_classification(y_binary, sampled_distributions)
+    @test mean_class_one ≈ mean(sampled_distributions[1,[2,4,5,7]])
+    @test mean_class_zero ≈ mean(sampled_distributions[2,[1,3,6,8]])
+


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

mean_class_one, mean_class_zero = sharpness_classification(y_binary, sampled_distributions)

@test mean_class_one ≈ mean(sampled_distributions[1,[2,4,5,7]])

@test mean_class_zero ≈ mean(sampled_distributions[2,[1,3,6,8]])

mean_class_one, mean_class_zero = sharpness_classification(

y_binary, sampled_distributions

)

@test mean_class_one ≈ mean(sampled_distributions[1, [2, 4, 5, 7]])

@test mean_class_zero ≈ mean(sampled_distributions[2, [1, 3, 6, 8]])

github-actions · 2024-07-02T15:36:25Z

test/calibration.jl

+
+

[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2024-07-02T15:36:25Z

test/calibration.jl

+
+# Test for `sharpness_regression` function
+@testset "sharpness_regression tests" begin
+    sampled_distributions = [[0.1, 0.2, 0.3, 0.7, 0.6], [0.2, 0.3, 0.4, 0.3 , 0.5 ], [0.3, 0.4, 0.5, 0.9, 0.2]]


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

sampled_distributions = [[0.1, 0.2, 0.3, 0.7, 0.6], [0.2, 0.3, 0.4, 0.3 , 0.5 ], [0.3, 0.4, 0.5, 0.9, 0.2]]

sampled_distributions = [

[0.1, 0.2, 0.3, 0.7, 0.6], [0.2, 0.3, 0.4, 0.3, 0.5], [0.3, 0.4, 0.5, 0.9, 0.2]

]

github-actions · 2024-07-02T15:36:25Z

test/calibration.jl

+    sharpness = sharpness_regression(sampled_distributions)
+
+    @test sharpness ≈ mean_variance
+end


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

end

end

github-actions · 2024-07-02T16:10:44Z

test/calibration.jl

+
+@testset "sharpness_classification tests" begin
+
+


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2024-07-02T16:10:44Z

test/calibration.jl

+    # Test 1: Check that the function runs without errors and returns two scalars for a simple case
+    y_binary = [1, 0, 1, 0, 1]
+    sampled_distributions = [0.9 0.1 0.8 0.2 0.7; 0.1 0.9 0.2 0.8 0.3]  # Sampled probabilities
+    mean_class_one, mean_class_zero = sharpness_classification(y_binary, sampled_distributions)


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

mean_class_one, mean_class_zero = sharpness_classification(y_binary, sampled_distributions)

mean_class_one, mean_class_zero = sharpness_classification(

y_binary, sampled_distributions

)

github-actions · 2024-07-02T16:10:45Z

test/calibration.jl

+    @test typeof(mean_class_one) <: Real  # Check if mean_class_one is a scalar
+    @test typeof(mean_class_zero) <: Real  # Check if mean_class_zero is a scalar
+
+


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2024-07-02T16:10:45Z

test/calibration.jl

+            0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8;
+            0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8;

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2

github-actions · 2024-07-02T16:10:45Z

test/calibration.jl

+    mean_class_one, mean_class_zero = sharpness_classification(y_binary, sampled_distributions)
+    @test mean_class_one ≈ mean(sampled_distributions[1,[2,4,5,7]])
+    @test mean_class_zero ≈ mean(sampled_distributions[2,[1,3,6,8]])


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

mean_class_one, mean_class_zero = sharpness_classification(y_binary, sampled_distributions)

@test mean_class_one ≈ mean(sampled_distributions[1,[2,4,5,7]])

@test mean_class_zero ≈ mean(sampled_distributions[2,[1,3,6,8]])

mean_class_one, mean_class_zero = sharpness_classification(

y_binary, sampled_distributions

)

@test mean_class_one ≈ mean(sampled_distributions[1, [2, 4, 5, 7]])

@test mean_class_zero ≈ mean(sampled_distributions[2, [1, 3, 6, 8]])

github-actions · 2024-07-02T16:10:45Z

test/calibration.jl

+    # Test 4: Edge case with all zeros in y_binary
+    y_binary_all_zeros = [0, 0, 0]
+    sampled_distributions_all_zeros = [0.1 0.2 0.3; 0.9 0.8 0.7]
+    mean_class_one_all_zeros, mean_class_zero_all_zeros = sharpness_classification(y_binary_all_zeros, sampled_distributions_all_zeros)


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

mean_class_one_all_zeros, mean_class_zero_all_zeros = sharpness_classification(y_binary_all_zeros, sampled_distributions_all_zeros)

mean_class_one_all_zeros, mean_class_zero_all_zeros = sharpness_classification(

y_binary_all_zeros, sampled_distributions_all_zeros

)

github-actions · 2024-07-02T16:10:45Z

test/calibration.jl

+
+

[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2024-07-02T16:10:45Z

test/calibration.jl

+
+

[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2024-07-02T16:10:45Z

test/calibration.jl

+    @test typeof(sharpness) <: Real  # Check if the output is a scalar
+
+    # Test 2: Check the function with a known input
+    sampled_distributions = [[0.1, 0.2, 0.3, 0.7, 0.6], [0.2, 0.3, 0.4, 0.3 , 0.5 ], [0.3, 0.4, 0.5, 0.9, 0.2]]


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

sampled_distributions = [[0.1, 0.2, 0.3, 0.7, 0.6], [0.2, 0.3, 0.4, 0.3 , 0.5 ], [0.3, 0.4, 0.5, 0.9, 0.2]]

sampled_distributions = [

[0.1, 0.2, 0.3, 0.7, 0.6], [0.2, 0.3, 0.4, 0.3, 0.5], [0.3, 0.4, 0.5, 0.9, 0.2]

]

github-actions · 2024-07-02T16:10:45Z

test/calibration.jl

+
+
+end


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

end

end

pasq-cat · 2024-07-02T16:12:08Z

is it possible to stop this checks until i am ready? every commit github automatically runs checks .....

github-actions · 2024-07-02T22:16:21Z

test/calibration.jl

+
+@testset "sharpness_classification tests" begin
+
+


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2024-07-02T22:16:21Z

test/calibration.jl

+    # Test 1: Check that the function runs without errors and returns two scalars for a simple case
+    y_binary = [1, 0, 1, 0, 1]
+    sampled_distributions = [0.9 0.1 0.8 0.2 0.7; 0.1 0.9 0.2 0.8 0.3]  # Sampled probabilities
+    mean_class_one, mean_class_zero = sharpness_classification(y_binary, sampled_distributions)


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

mean_class_one, mean_class_zero = sharpness_classification(y_binary, sampled_distributions)

mean_class_one, mean_class_zero = sharpness_classification(

y_binary, sampled_distributions

)

github-actions · 2024-07-02T22:16:21Z

test/calibration.jl

+    @test typeof(mean_class_one) <: Real  # Check if mean_class_one is a scalar
+    @test typeof(mean_class_zero) <: Real  # Check if mean_class_zero is a scalar
+
+


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2024-07-02T22:16:21Z

test/calibration.jl

+            0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8;
+            0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8;

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2

github-actions · 2024-07-02T22:16:21Z

test/calibration.jl

+    mean_class_one, mean_class_zero = sharpness_classification(y_binary, sampled_distributions)
+    @test mean_class_one ≈ mean(sampled_distributions[1,[2,4,5,7]])
+    @test mean_class_zero ≈ mean(sampled_distributions[2,[1,3,6,8]])


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

mean_class_one, mean_class_zero = sharpness_classification(y_binary, sampled_distributions)

@test mean_class_one ≈ mean(sampled_distributions[1,[2,4,5,7]])

@test mean_class_zero ≈ mean(sampled_distributions[2,[1,3,6,8]])

mean_class_one, mean_class_zero = sharpness_classification(

y_binary, sampled_distributions

)

@test mean_class_one ≈ mean(sampled_distributions[1, [2, 4, 5, 7]])

@test mean_class_zero ≈ mean(sampled_distributions[2, [1, 3, 6, 8]])

github-actions · 2024-07-02T22:16:22Z

test/calibration.jl

+
+

[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2024-07-02T22:16:22Z

test/calibration.jl

+    y_binary = rand(0:1, 10)
+    sampled_distributions = rand(Normal(0.5, 0.1), 10, 6)
+    n_bins = 4
+    num_p_per_interval, emp_avg, bin_centers = empirical_frequency_binary_classification(y_binary, sampled_distributions,  n_bins=n_bins)


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

num_p_per_interval, emp_avg, bin_centers = empirical_frequency_binary_classification(y_binary, sampled_distributions, n_bins=n_bins)

num_p_per_interval, emp_avg, bin_centers = empirical_frequency_binary_classification(

y_binary, sampled_distributions; n_bins=n_bins

)

github-actions · 2024-07-02T22:16:22Z

test/calibration.jl

+
+

[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2024-07-02T22:16:22Z

test/calibration.jl

+    Y_cal =  [0, 1, 0, 1.2, 4]
+    sampled_distributions =  rand(Normal(0.5, 0.1), 5, 6)
+    @test_throws ArgumentError empirical_frequency_binary_classification(Y_cal, sampled_distributions, n_bins=10)
+


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

Y_cal = [0, 1, 0, 1.2, 4]

sampled_distributions = rand(Normal(0.5, 0.1), 5, 6)

@test_throws ArgumentError empirical_frequency_binary_classification(Y_cal, sampled_distributions, n_bins=10)

Y_cal = [0, 1, 0, 1.2, 4]

sampled_distributions = rand(Normal(0.5, 0.1), 5, 6)

@test_throws ArgumentError empirical_frequency_binary_classification(

Y_cal, sampled_distributions, n_bins=10

)

github-actions · 2024-07-02T22:16:22Z

test/calibration.jl

+    sampled_distributions =  rand(Normal(0.5, 0.1), 5, 6)
+    @test_throws ArgumentError empirical_frequency_binary_classification(Y_cal, sampled_distributions, n_bins=0)
+end


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

sampled_distributions = rand(Normal(0.5, 0.1), 5, 6)

@test_throws ArgumentError empirical_frequency_binary_classification(Y_cal, sampled_distributions, n_bins=0)

end

sampled_distributions = rand(Normal(0.5, 0.1), 5, 6)

@test_throws ArgumentError empirical_frequency_binary_classification(

Y_cal, sampled_distributions, n_bins=0

)

end

pasq-cat · 2024-07-03T01:16:05Z

src/baselaplace/predicting.jl

- `fμ::AbstractArray`: Mean of the predictive distribution if link function is set to `:plugin`, otherwise the probit approximation. The output shape is column-major as in Flux.
- `fvar::AbstractArray`: If regression, it also returns the variance of the predictive distribution. The output shape is column-major as in Flux.
+For classification tasks, LaplaceRedux provides different options:
+    -`normal_distr::Distributions.Normal`:the array of Normal distributions computed by glm_predictive_distribution  If the `link_approx` is set to :distribution


@pat-alt i am confused on how to proceed here. the issue is that the output of the chain may have already passed through a softmax layer, so the output should not be converted again. should we add a check for this or leave it to the educated reader lol?

Good point! I think we should probably add a check somewhere in the corestruct that contains the Flux chain. Let's open a separate issue for this.

either that or modify the predict function so that it checks that the la.model has a finaliser layer

pasq-cat · 2024-07-03T01:20:15Z

src/utils.jl

+@doc raw""" 
+    empirical_frequency_regression(Y_cal, sampled_distributions, n_bins=20)
+
+FOR REGRESSION MODELS.  \


@pat-alt maybe i can move this functions to a dedicated julia file ( calibration_functions.jl) , so that in the future i may add something else in a compartmentalized file

yes, good idea

github-actions · 2024-07-03T01:24:01Z

test/calibration.jl

+    sampled_distributions =  [rand(Distributions.Normal(1, 1.0),6) for _ in 1:5]
+    @test_throws ArgumentError empirical_frequency_regression(Y_cal, sampled_distributions, n_bins=0)
+


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

sampled_distributions = [rand(Distributions.Normal(1, 1.0),6) for _ in 1:5]

@test_throws ArgumentError empirical_frequency_regression(Y_cal, sampled_distributions, n_bins=0)

sampled_distributions = [rand(Distributions.Normal(1, 1.0), 6) for _ in 1:5]

@test_throws ArgumentError empirical_frequency_regression(

Y_cal, sampled_distributions, n_bins=0

)

github-actions · 2024-07-03T01:24:01Z

test/calibration.jl

+
+

[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2024-07-03T01:24:01Z

test/calibration.jl

+    y_binary = rand(0:1, 10)
+    sampled_distributions = rand(Normal(0.5, 0.1), 10, 6)
+    n_bins = 4
+    num_p_per_interval, emp_avg, bin_centers = empirical_frequency_binary_classification(y_binary, sampled_distributions,  n_bins=n_bins)


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

num_p_per_interval, emp_avg, bin_centers = empirical_frequency_binary_classification(y_binary, sampled_distributions, n_bins=n_bins)

num_p_per_interval, emp_avg, bin_centers = empirical_frequency_binary_classification(

y_binary, sampled_distributions; n_bins=n_bins

)

github-actions · 2024-07-03T01:24:01Z

test/calibration.jl

+
+

[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2024-07-03T01:24:01Z

test/calibration.jl

+    Y_cal =  [0, 1, 0, 1.2, 4]
+    sampled_distributions =  rand(Normal(0.5, 0.1), 5, 6)
+    @test_throws ArgumentError empirical_frequency_binary_classification(Y_cal, sampled_distributions, n_bins=10)
+


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

Y_cal = [0, 1, 0, 1.2, 4]

sampled_distributions = rand(Normal(0.5, 0.1), 5, 6)

@test_throws ArgumentError empirical_frequency_binary_classification(Y_cal, sampled_distributions, n_bins=10)

Y_cal = [0, 1, 0, 1.2, 4]

sampled_distributions = rand(Normal(0.5, 0.1), 5, 6)

@test_throws ArgumentError empirical_frequency_binary_classification(

Y_cal, sampled_distributions, n_bins=10

)

github-actions · 2024-07-03T01:24:01Z

test/calibration.jl

+    sampled_distributions =  rand(Normal(0.5, 0.1), 5, 6)
+    @test_throws ArgumentError empirical_frequency_binary_classification(Y_cal, sampled_distributions, n_bins=0)
+end


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

sampled_distributions = rand(Normal(0.5, 0.1), 5, 6)

@test_throws ArgumentError empirical_frequency_binary_classification(Y_cal, sampled_distributions, n_bins=0)

end

sampled_distributions = rand(Normal(0.5, 0.1), 5, 6)

@test_throws ArgumentError empirical_frequency_binary_classification(

Y_cal, sampled_distributions, n_bins=0

)

end

github-actions · 2024-07-03T04:52:04Z

test/calibration.jl

+    sampled_distributions = [rand(Distributions.Normal(1, 1.0),6) for _ in 1:5]
+    counts = empirical_frequency_regression(Y_cal, sampled_distributions, n_bins=n_bins)
+    @test typeof(counts) == Array{Float64, 1}  # Check if the output is an array of Float64
+    @test length(counts) == n_bins + 1 


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

sampled_distributions = [rand(Distributions.Normal(1, 1.0),6) for _ in 1:5]

counts = empirical_frequency_regression(Y_cal, sampled_distributions, n_bins=n_bins)

@test typeof(counts) == Array{Float64, 1} # Check if the output is an array of Float64

@test length(counts) == n_bins + 1

sampled_distributions = [rand(Distributions.Normal(1, 1.0), 6) for _ in 1:5]

counts = empirical_frequency_regression(Y_cal, sampled_distributions; n_bins=n_bins)

@test typeof(counts) == Array{Float64,1} # Check if the output is an array of Float64

@test length(counts) == n_bins + 1

github-actions · 2024-07-03T04:52:04Z

test/calibration.jl

+    sampled_distributions =  [rand(Distributions.Normal(1, 1.0),6) for _ in 1:5]
+    @test_throws ArgumentError empirical_frequency_regression(Y_cal, sampled_distributions, n_bins=0)
+


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

sampled_distributions = [rand(Distributions.Normal(1, 1.0),6) for _ in 1:5]

@test_throws ArgumentError empirical_frequency_regression(Y_cal, sampled_distributions, n_bins=0)

sampled_distributions = [rand(Distributions.Normal(1, 1.0), 6) for _ in 1:5]

@test_throws ArgumentError empirical_frequency_regression(

Y_cal, sampled_distributions, n_bins=0

)

github-actions · 2024-07-03T04:52:04Z

test/calibration.jl

+
+

[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2024-07-03T04:52:04Z

test/calibration.jl

+@testset "empirical_frequency_binary_classification tests" begin
+    # Test 1: Check that the function runs without errors and returns an array for a simple case
+    y_binary = rand(0:1, 10)
+    sampled_distributions = rand(2,10)


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

sampled_distributions = rand(2,10)

sampled_distributions = rand(2, 10)

github-actions · 2024-07-03T04:52:04Z

test/calibration.jl

+    y_binary = rand(0:1, 10)
+    sampled_distributions = rand(2,10)
+    n_bins = 4
+    num_p_per_interval, emp_avg, bin_centers = empirical_frequency_binary_classification(y_binary, sampled_distributions,  n_bins=n_bins)


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

num_p_per_interval, emp_avg, bin_centers = empirical_frequency_binary_classification(y_binary, sampled_distributions, n_bins=n_bins)

num_p_per_interval, emp_avg, bin_centers = empirical_frequency_binary_classification(

y_binary, sampled_distributions; n_bins=n_bins

)

github-actions · 2024-07-03T04:52:04Z

test/calibration.jl

+
+

[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2024-07-03T04:52:05Z

test/calibration.jl

+    Y_cal =  [0, 1, 0, 1.2, 4]
+    sampled_distributions =  rand(2,5)
+    @test_throws ArgumentError empirical_frequency_binary_classification(Y_cal, sampled_distributions, n_bins=10)
+


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

Y_cal = [0, 1, 0, 1.2, 4]

sampled_distributions = rand(2,5)

@test_throws ArgumentError empirical_frequency_binary_classification(Y_cal, sampled_distributions, n_bins=10)

Y_cal = [0, 1, 0, 1.2, 4]

sampled_distributions = rand(2, 5)

@test_throws ArgumentError empirical_frequency_binary_classification(

Y_cal, sampled_distributions, n_bins=10

)

github-actions · 2024-07-03T04:52:05Z

test/calibration.jl

+    sampled_distributions =  rand(2,5)
+    @test_throws ArgumentError empirical_frequency_binary_classification(Y_cal, sampled_distributions, n_bins=0)
+end


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

sampled_distributions = rand(2,5)

@test_throws ArgumentError empirical_frequency_binary_classification(Y_cal, sampled_distributions, n_bins=0)

end

sampled_distributions = rand(2, 5)

@test_throws ArgumentError empirical_frequency_binary_classification(

Y_cal, sampled_distributions, n_bins=0

)

end

pat-alt · 2024-07-03T07:14:07Z

is it possible to stop this checks until i am ready? every commit github automatically runs checks .....

You can remove this part here (I've already done this in #39). For now, please just run the formatter before you push, it takes 2s :)

pasq-cat · 2024-07-03T12:13:38Z

is it possible to stop this checks until i am ready? every commit github automatically runs checks .....

You can remove this part here (I've already done this in #39). For now, please just run the formatter before you push, it takes 2s :)

I know, I just forget to do it. I wish it was automatically applied before synchronization

…ct ( will address it in a different issue) fixed docstring for calibration.jl

pasq-cat added 19 commits June 9, 2024 05:49

function empirical_frequency

733312a

fixed the docstring.

25ea642

added sharpness and binary classification. i have yet to test them pr…

c290ed8

…operly due to the issue with mljflux.

added trapz to the list of dependencies.

4ff22f4

added Distributions to theproject

6a22210

working version

df3d60d

ops forgot to add sharpness for the classification case

09f25e8

working release.. changed changelog, glm_predictive_distribution, pre…

07b318f

…dict

function empirical_frequency

eafa7bd

fixed the docstring.

f66e08e

added sharpness and binary classification. i have yet to test them pr…

5355281

…operly due to the issue with mljflux.

added trapz to the list of dependencies.

2efaa99

added Distributions to theproject

26643ee

working version

b79ca39

ops forgot to add sharpness for the classification case

0d71736

working release.. changed changelog, glm_predictive_distribution, pre…

5f772cf

…dict

Merge branch '90-average-calibration-in-utilsjl' of https://github.co…

d146d1d

…m/JuliaTrustworthyAI/LaplaceRedux.jl into 90-average-calibration-in-utilsjl

changed docstrings in predicting.jl

7af9378

fixed glm_predictive_distribution

2c42236

pasq-cat linked an issue Jun 22, 2024 that may be closed by this pull request

Add a function to compute the average calibration in utils.jl #90

Closed

github-actions bot reviewed Jun 22, 2024

View reviewed changes

src/baselaplace/predicting.jl Outdated Show resolved Hide resolved

src/baselaplace/predicting.jl Outdated Show resolved Hide resolved

src/utils.jl Outdated Show resolved Hide resolved

src/utils.jl Outdated Show resolved Hide resolved

src/utils.jl Outdated Show resolved Hide resolved

pasq-cat and others added 2 commits June 22, 2024 11:41

Update src/utils.jl

9d67ddc

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update src/utils.jl

9f07583

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions bot reviewed Jun 22, 2024

View reviewed changes

src/utils.jl Outdated

Comment on lines 134 to 136

Copy link

Contributor

github-actions bot Jun 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

pasq-cat and others added 2 commits June 22, 2024 11:43

Update src/utils.jl

f81d226

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update src/baselaplace/predicting.jl

6cdc503

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions bot reviewed Jun 22, 2024

View reviewed changes

Update src/baselaplace/predicting.jl

89bb19b

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions bot reviewed Jun 22, 2024

View reviewed changes

JuliaTrustworthyAI deleted a comment from github-actions bot Jun 22, 2024

github-actions bot reviewed Jun 30, 2024

View reviewed changes

src/utils.jl Outdated Show resolved Hide resolved

pasq-cat and others added 2 commits July 1, 2024 00:45

fixed error in empirical_frequency_regression

3d2ebd6

Update src/utils.jl

4ab04f6

yeah basically juliaformatter being a pain for nothing Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

pat-alt requested changes Jul 1, 2024

View reviewed changes

pat-alt mentioned this pull request Jul 1, 2024

39 refactor mljflux interface #92

Merged

docstrings fixes and predict update

267b8f4

pat-alt requested changes Jul 2, 2024

View reviewed changes

pasq-cat added 2 commits July 2, 2024 14:37

fixed typos

d188daf

moved sharpness functions units tests in calibration.jl. changed runt…

270b70a

…ests.jl to add the new file.

github-actions bot reviewed Jul 2, 2024

View reviewed changes

more sharpness unit tests

3320063

github-actions bot reviewed Jul 2, 2024

View reviewed changes

fixes and more unit tests

3750dbe

github-actions bot reviewed Jul 2, 2024

View reviewed changes

pasq-cat commented Jul 3, 2024

View reviewed changes

JuliaTrustworthyAI deleted a comment from pat-alt Jul 3, 2024

pasq-cat commented Jul 3, 2024

View reviewed changes

small stuff

39d4bdc

github-actions bot reviewed Jul 3, 2024

View reviewed changes

fix. there is still an issue with the shape of the input to use.

56c3b66

github-actions bot reviewed Jul 3, 2024

View reviewed changes

JuliaTrustworthyAI deleted a comment from github-actions bot Jul 4, 2024

pasq-cat added 3 commits July 4, 2024 05:51

fixed logit.md ,moved functions to new file, removed changes to predi…

908c804

…ct ( will address it in a different issue) fixed docstring for calibration.jl

removed calibration_plots.md

f468803

test plot

459b2fe

pasq-cat closed this Jul 4, 2024

pasq-cat deleted the 90-average-calibration-in-utilsjl branch July 6, 2024 02:15

	normal_distr,fμ, fvar = glm_predictive_distribution(la, X)
	normal_distr, fμ, fvar = glm_predictive_distribution(la, X)

		0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8;
		0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2

		@test typeof(mean_class_one) <: Real # Check if mean_class_one is a scalar
		@test typeof(mean_class_zero) <: Real # Check if mean_class_zero is a scalar

		sampled_distributions = [rand(Distributions.Normal(1, 1.0),6) for _ in 1:5]
		@test_throws ArgumentError empirical_frequency_regression(Y_cal, sampled_distributions, n_bins=0)

	sampled_distributions = rand(2,10)
	sampled_distributions = rand(2, 10)

90 average calibration functions in utils.jl #97

90 average calibration functions in utils.jl #97

Conversation

pasq-cat commented Jun 22, 2024

github-actions bot Jun 22, 2024

Choose a reason for hiding this comment

github-actions bot Jun 22, 2024

Choose a reason for hiding this comment

github-actions bot Jun 22, 2024

Choose a reason for hiding this comment

github-actions bot Jun 22, 2024

Choose a reason for hiding this comment

pat-alt left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pat-alt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

pasq-cat commented Jul 2, 2024

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

github-actions bot Jul 2, 2024

Choose a reason for hiding this comment

pasq-cat Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pat-alt left a comment •

edited

Loading

pasq-cat Jul 3, 2024 •

edited

Loading