CREPO: An Open Repository to Benchmark Credal Network Algorithms

Setup

The latest version of the Python package for CREPO can be installed from PyPI as follows.

!pip install crepobenchmark

or from github source code:

!pip install git+https://github.com/IDSIA-papers/crepo.git@main#"subdirectory=src/python"

Import the package.

import crepobenchmark as crb

If you need to change the java binary used, the command

crb.info.java_bin = "/jdkpath/bin/java"

will make crepo use the java binary at /jdkpath/bin/ instead of the default one.

Benchmarking data

The following function will plot a table summarizing the features of the benchmarking dataset.

crb.describe()

	property	value	description
0	num_vert	[2, 4, 6]	Number of vertices in the credal sets. It is a...
1	max_degree	[6]	Maximum arc degree in the model.
2	max_indegree	[2, 4, 6]	Maximum arc indegree in the model.
3	max_values	[4]	Maximum cardinality plus 1
4	nodes	[4, 5, 6, 7, 8, 9, 10]	Number of nodes in the model.
5	method	[approxlp, cve, cve_ch10, cve_ch5]	Inference method.
6	kind	[sing, mult]	Topology of the DAG: singly or multy connected.
7	query_type	[marg, cond]	Marginal or conditional query.
8	Vmodels	378	Count of models with a vertex specification.
9	Hmodels	378	Count of models with a linear constraints spec...
10	rows	1920	Number of rows in the current benchmarking data.
11	columns	15	Number of rows in the current benchmarking data.

The data itself can be obtained as follows.

data = crb.get_benchmark_data()
data

	label	num_vert	max_degree	max_indegree	max_values	nodes	method	kind	query_type	filename	target	observed	barren	interval_result	time
0	hs_n4_mID2_mD6_mV4_nV2-1	2	6	2	4	4	approxlp	sing	marg	./networks/hmodel/hmodel-sing_n4_mID2_mD6_mV4_...	3	NaN	0	0.47661183887412617 0.6147556334817609 0.38524...	477.0
1	hs_n4_mID2_mD6_mV4_nV2-1	2	6	2	4	4	approxlp	sing	cond	./networks/hmodel/hmodel-sing_n4_mID2_mD6_mV4_...	0	3	0	NaN	NaN
2	hm_n4_mID2_mD6_mV4_nV2-1	2	6	2	4	4	approxlp	mult	marg	./networks/hmodel/hmodel-mult_n4_mID2_mD6_mV4_...	2	NaN	0	0.14280081709746253 0.524457784601085 0.100093...	1277.0
3	hm_n4_mID2_mD6_mV4_nV2-1	2	6	2	4	4	approxlp	mult	cond	./networks/hmodel/hmodel-mult_n4_mID2_mD6_mV4_...	0	2	0	NaN	NaN
4	hs_n5_mID2_mD6_mV4_nV2-1	2	6	2	4	5	approxlp	sing	marg	./networks/hmodel/hmodel-sing_n5_mID2_mD6_mV4_...	4	NaN	0	0.13952568805044846 0.40904652506095185 0.2338...	1037.0
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
1915	vm_n5_mID4_mD6_mV4_nV6-1	6	6	4	4	5	cve_ch10	mult	cond	./networks/vmodel/vmodel-mult_n5_mID4_mD6_mV4_...	0	4	0	NaN	NaN
1916	vs_n5_mID4_mD6_mV4_nV6-1	6	6	4	4	5	cve_ch5	sing	marg	./networks/vmodel/vmodel-sing_n5_mID4_mD6_mV4_...	4	NaN	0	0.02162501899836531 0.6233117954158697 0.04147...	140053.0
1917	vs_n5_mID4_mD6_mV4_nV6-1	6	6	4	4	5	cve_ch5	sing	cond	./networks/vmodel/vmodel-sing_n5_mID4_mD6_mV4_...	0	4	0	NaN	NaN
1918	vm_n5_mID4_mD6_mV4_nV6-1	6	6	4	4	5	cve_ch5	mult	marg	./networks/vmodel/vmodel-mult_n5_mID4_mD6_mV4_...	4	NaN	0	0.19665011233872 0.934857703437616 0.065142296...	139.0
1919	vm_n5_mID4_mD6_mV4_nV6-1	6	6	4	4	5	cve_ch5	mult	cond	./networks/vmodel/vmodel-mult_n5_mID4_mD6_mV4_...	0	4	0	NaN	NaN

1920 rows × 15 columns

Credal networks specificaitons

The specification of any of the models in UAI format can be obtained as follows.

modelname = "vs_n4_mID2_mD6_mV4_nV2-1"
print(crb.get_model(modelname))

V-CREDAL
4
4 4 2 2
4
1 0
2 0 1
2 1 2
2 2 3

8
0.218 0.48 0.254 0.048
0.296 0.603 0.086 0.015

8
0.241 0.617 0.119 0.023
0.004 0.989 0.003 0.004
8
0.197 0.711 0.048 0.044
0.039 0.948 0.006 0.007
8
0.919 0.004 0.009 0.068
0.002 0.017 0.001 0.98
8
0.049 0.108 0.13 0.713
0.095 0.428 0.427 0.05

4
0.423 0.577
0.752 0.248
4
0.616 0.384
0.667 0.333
4
0.875 0.125
0.087 0.913
4
0.007 0.993
0.517 0.483

4
0.674 0.326
0.677 0.323
4
0.361 0.639
0.464 0.536

This specification can also be saved into a file:

crb.save_model(modelname, "model.uai")
!ls | grep model

model.uai

Running inference with CREMA

Finally, an inference task can be done by invoking Crema:

crb.run_crema("model.uai", target=0, method="cve")

-t --method=cve -r 1 -w 0 -x 0 -y 1 --timeout=600 model.uai





{'time': 55.0,
 'interval_result': [0.0026153357047309983,
  0.7449456975772765,
  0.060534723389944506,
  0.982702139276319,
  0.0008971229475704658,
  0.9140697811019305,
  0.002723271185938287,
  0.18491484184914841],
 'err_msg': '',
 'arg_str': '-t;--method=cve;-r;1;-w;0;-x;0;-y;1;--timeout=600;model.uai'}