Hierarchical: restructuring and add relative to InnerCalculatorCollec…

…tor (#1245)

* Hier. par. plotting for fides

* hierarchical 'hess' bug

* Fix storing of inner parameters in result

Inner parameters were not being stored when the result was stored. This was due to a problem with putting a dictionary into an HDF5 format. I've transformed it into 2 lists now: a INNER_PARAMETERS_VALUES and INNER_PARAMETER_NAMES list.

* adjusted documentation of hierarichal opt.

* Fix some type issues

* Small fix

* Save only values, not names

* Extend to all optimizers

* Remove redundant

* Small update

* Update parameters.py

* Small typo

* Fix some bugs, use lists for saving instead

* Fix tests

* Update hierarchical parameter plot test

* Largescale renaming and restructuring

- Restructuring of the hierarchical scaling+offset method. The base classes used by all non-quantitative and semi-quantitative data types are put in the `base_...` files. All classes related to the scaling+offset method (for relative data) is moved to the `relative` folder, analogous to other data types.
- Renaming: the naming scheme doesn't follow the method, but the data type now. It's more direct and removes the ambiguity of `OptimalScaling` for ordinal data and `scaling_offset` for relative data.
- Optimal scaling -> ordinal
- Spline approximation -> semiquantitative
- Scaling + offset -> relative

* Update base_problem.py

* Fix Docstrings

* Daniel&Dilan review changes

* Small change

* More Dilan Review changes

* Fix testing

* Fix base test

* Rename notebooks (again)

* Move inner pars to decorator

* Collect decorators for minimize

* Maren review changes

* Rename & fix notebooks

* Include notebook changes

* FIx tests

* Add output

* Include relative into Collector

Included the relative calculator into the Inner calculator collector. Now relative data can be used together with any other non-quantitative data type. A lot of TODOs to still clean up. But it works nicely on all examples, except one which I think is a bad one (Boehm_mixed_test with nonlinearities: 1 spline 1 relative 1 known: very bad convergence... fits look okayish...)

* Small documentation update

* Documentation update

* Update the notebooks

* Update spline_approximation.py

* Change data_types from list to set

* Use add_sim_grad_to_opt_grad

* Rename to semiquantitative in example

* mode RES with only RELATIVE data

* Add petab data type validation

* DO TODOs

* Small TODO

* FIx api.rst module name changes

* Update CODEOWNERS module name changes

* Fix underline too short & api.rst

* Fix plotting routines

* Inline literal error

* Testing error

* Testing error 2

* Test literal error

* Fix missing indentation

* Fix inline

* Daniel, Fabian & Stephan review changes

* Notebook changes

* Dilan & Daniel review changes

* Change title of relative_data.ipynb

* Remove observableParameter from solvers

* Notebook changes from #1246

* Fabian review change

---------

Co-authored-by: Lea@Mac <[email protected]>

.github/CODEOWNERS

@@ -3,25 +3,26 @@
 # default owners
 * @ICB-DCM/pypesto-maintainers
 
-/doc/example/censored.ipynb @Doresic
+/doc/example/censored_data.ipynb @Doresic
 /doc/example/hdf5_storage.ipynb @PaulJonasJost
-/doc/example/hierarchical.ipynb @dilpath @dweindl
 /doc/example/julia.ipynb @PaulJonasJost
 /doc/example/model_selection.ipynb @dilpath
-/doc/example/nonlinear_monotone.ipynb @Doresic
-/doc/example/ordinal.ipynb @Doresic
+/doc/example/ordinal_data.ipynb @Doresic
 /doc/example/petab_import.ipynb @dweindl @FFroehlich
+/doc/example/relative_data.ipynb @dilpath @dweindl
 /doc/example/sampler_study.ipynb @dilpath
 /doc/example/sampling_diagnostics.ipynb @dilpath
+/doc/example/semiquantitative_data.ipynb @Doresic
 /doc/example/store.ipynb @PaulJonasJost
 /doc/example/synthetic_data.ipynb @dilpath
 /docker/ @dweindl
 /pypesto/engine/ @PaulJonasJost
 /pypesto/engine/mpi_pool.py @PaulJonasJost
 /pypesto/ensemble/ @dilpath @PaulJonasJost
 /pypesto/hierarchical/ @dweindl @Doresic
-/pypesto/hierarchical/optimal_scaling_approach/ @Doresic
-/pypesto/hierarchical/spline_approximation/ @Doresic
+/pypesto/hierarchical/ordinal/ @Doresic
+/pypesto/hierarchical/relative/ @dweindl @Doresic
+/pypesto/hierarchical/semiquantitative/ @Doresic
 /pypesto/history/ @PaulJonasJost
 /pypesto/objective/ @PaulJonasJost
 /pypesto/objective/amici/ @dweindl @FFroehlich

doc/api.rst

@@ -7,8 +7,9 @@ API reference
    pypesto.engine
    pypesto.ensemble
    pypesto.hierarchical
-   pypesto.hierarchical.optimal_scaling
-   pypesto.hierarchical.spline_approximation
+   pypesto.hierarchical.ordinal
+   pypesto.hierarchical.relative
+   pypesto.hierarchical.semiquantitative
    pypesto.history
    pypesto.logging
    pypesto.objective

doc/example.rst

@@ -51,10 +51,10 @@ Algorithms and features
    example/store.ipynb
    example/model_selection.ipynb
    example/julia.ipynb
-   example/hierarchical.ipynb
-   example/ordinal.ipynb
-   example/censored.ipynb
-   example/nonlinear_monotone.ipynb
+   example/relative_data.ipynb
+   example/ordinal_data.ipynb
+   example/censored_data.ipynb
+   example/semiquantitative_data.ipynb
 
 Application examples
 --------------------

doc/example/censored.ipynb → doc/example/censored_data.ipynb

@@ -115,10 +115,15 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "For censored measurements, the `measurement` column will be ignored. For the `Ybar` observable we didn't specify a measurement type, so those will be used as quantitative.\n",
-    "\n",
+    "For censored measurements, the `measurement` column will be ignored. For the `Ybar` observable we didn't specify a measurement type, so those will be used as quantitative."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
     "#### Note on inclusion of additional data types:\n",
-    "It is possible to include observables with different types of data to the same `petab_problem`. Refer to the notebooks on using [nonlinear-monotone data](nonlinear_monotone.ipynb) and [ordinal data](ordinal.ipynb) for details on integration of other data types. Additionally, as shown in this example, if the `measurementType` column is left empty for all measurements of an observable, the observable will be treated as quantitative."
+    "It is possible to include observables with different types of data to the same `petab_problem`. Refer to the notebooks on using [semiquantitative data](semiquantitative_data.ipynb), [relative data](relative_data.ipynb) and [ordinal data](ordinal_data.ipynb) for details on integration of other data types. If the `measurementType` column is left empty for all measurements of an observable, the observable will be treated as quantitative."
    ]
   },
   {
@@ -136,9 +141,9 @@
    "source": [
     "Now when we construct the `objective`, it will construct all objects of the optimal scaling inner optimization:\n",
     "\n",
-    "- `OptimalScalingInnerSolver`\n",
-    "- `OptimalScalingAmiciCalculator`\n",
-    "- `OptimalScalingProblem`\n",
+    "- `OrdinalInnerSolver`\n",
+    "- `OrdinalCalculator`\n",
+    "- `OrdinalProblem`\n",
     "\n",
     "As there are no censored data specific inner options, we will pass none to the constructor."
    ]
@@ -149,15 +154,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "objective = importer.create_objective()"
+    "model = importer.create_model(verbose=False)\n",
+    "objective = importer.create_objective(model=model)"
    ]
   },
   {
    "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now let's construct the pyPESTO problem and optimizer. We're going to use a gradint-based optimizer for a faster optimization, but gradient-free optimizers can be used in the same way:"
+    "Now let's construct the pyPESTO problem and optimizer. We're going to use a gradient-based optimizer for a faster optimization, but gradient-free optimizers can be used in the same way:"
    ]
   },
   {
@@ -168,7 +174,7 @@
    "source": [
     "problem = importer.create_problem(objective)\n",
     "\n",
-    "engine = pypesto.engine.SingleCoreEngine()\n",
+    "engine = pypesto.engine.MultiProcessEngine(n_procs=3)\n",
     "\n",
     "optimizer = optimize.ScipyOptimizer(\n",
     "    method=\"L-BFGS-B\",\n",

doc/example/example_semiquantitative/example_semiquantitative.yaml

@@ -0,0 +1,11 @@
+format_version: 1
+parameter_file: parameters_example_semiquantitative.tsv
+problems:
+- condition_files:
+  - experimentalCondition_example_semiquantitative.tsv
+  measurement_files:
+  - measurementData_example_semiquantitative.tsv
+  observable_files:
+  - observables_example_semiquantitative.tsv
+  sbml_files:
+  - model_example_semiquantitative.xml

doc/example/example_semiquantitative/example_semiquantitative_linear.yaml

@@ -0,0 +1,11 @@
+format_version: 1
+parameter_file: parameters_example_semiquantitative.tsv
+problems:
+- condition_files:
+  - experimentalCondition_example_semiquantitative.tsv
+  measurement_files:
+  - measurementData_example_semiquantitative_linear.tsv
+  observable_files:
+  - observables_example_semiquantitative.tsv
+  sbml_files:
+  - model_example_semiquantitative.xml

doc/example/example_semiquantitative/measurementData_example_semiquantitative.tsv

@@ -0,0 +1,19 @@
+observableId	preequilibrationConditionId	simulationConditionId	measurement	time	observableParameters	noiseParameters	observableTransformation	noiseDistribution	measurementType
+Activity		Inhibitor_0	7.68240348241022	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_3	7.87610717856578	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_10	8.31458694362777	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_25	9.13091469917684	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_35	8.07849405536742	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_50	5.45211611461129	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_75	2.69874576106127	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_100	1.673154390125	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_300	0.392886259997627	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_0	0	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_3	0.744411111757651	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_10	2.31052444293466	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_25	4.23805611211676	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_35	4.6428594963941	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_50	4.83287946040586	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_75	4.89868421035517	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_100	4.91379066100035	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_300	4.92900825318423	5		1	lin	normal	semiquantitative

doc/example/example_semiquantitative/measurementData_example_semiquantitative_linear.tsv

@@ -0,0 +1,19 @@
+observableId	preequilibrationConditionId	simulationConditionId	measurement	time	observableParameters	noiseParameters	observableTransformation	noiseDistribution	measurementType
+Activity		Inhibitor_0	15.24029495	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_3	15.97660789	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_10	17.89265379	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_25	23.18714697	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_35	16.81210375	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_50	9.17312936	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_75	4.15092812	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_100	2.53355252	5		1	lin	normal	semiquantitative
+Activity		Inhibitor_300	0.5896329	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_0	0	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_3	0.05999885	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_10	0.19999376	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_25	0.49904277	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_35	0.65916874	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_50	0.81495435	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_75	0.91638297	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_100	0.94898071	5		1	lin	normal	semiquantitative
+Ybar		Inhibitor_300	0.98813045	5		1	lin	normal	semiquantitative