Proposed new feature to improve console output

[Cole-Monnahan-NOAA]

The current ADMB console output is quite verbose and often difficult to interpret. I think in most cases this output should be condensed down into something that is more useful. In some cases the user may want to suppress standard output altogether as well. I argue that improved console output is important because it helps users catch issues and understand optimization of their model much better.

To this end I propose a new feature of ADMB accessed by the command line argument"-output" which allows the user to control standard console output and warnings. I modified the code on this fork ( edit: on this PR) to create a beta version of this feature. This issue is to alert users/developers of this work and to request feedback and ideas.

The user invokes the output with -output N where

• N=0 suppresses all standard output
• N=1 is a new compact output, I propose this as the new default
• N=2 is the current default
• N=3 could be added later to give more information for debugging purposes

For instance the packaged catage model now looks like this:

Starting optimization of 'catage' in phase 1 of 3 at Wed Nov 24 13:25:01 2021
phase= 1 | nvar=  4 | iter=  0 | nll=9.493e+03 | mgc=+5.937e+03
phase= 1 | nvar=  4 | iter= 10 | nll=5.146e+02 | mgc=+3.764e-03
phase= 2 | nvar= 28 | iter=  0 | nll=5.146e+02 | mgc=-5.187e+02
phase= 2 | nvar= 28 | iter= 10 | nll=4.618e+02 | mgc=-4.958e+02
phase= 2 | nvar= 28 | iter= 20 | nll=4.595e+02 | mgc=-6.737e+02
phase= 2 | nvar= 28 | iter= 30 | nll=3.801e+02 | mgc=+7.843e+02
phase= 2 | nvar= 28 | iter= 40 | nll=3.547e+02 | mgc=+1.815e+01
phase= 2 | nvar= 28 | iter= 50 | nll=3.547e+02 | mgc=-1.256e+00
phase= 2 | nvar= 28 | iter= 60 | nll=3.547e+02 | mgc=+4.896e-04
phase= 3 | nvar= 40 | iter=  0 | nll=3.547e+02 | mgc=+2.418e+02
phase= 3 | nvar= 40 | iter= 10 | nll=3.358e+02 | mgc=-3.926e+02
phase= 3 | nvar= 40 | iter= 20 | nll=3.345e+02 | mgc=-3.548e+02
phase= 3 | nvar= 40 | iter= 30 | nll=3.339e+02 | mgc=-3.454e+02
phase= 3 | nvar= 40 | iter= 40 | nll=3.325e+02 | mgc=-2.508e+02
phase= 3 | nvar= 40 | iter= 50 | nll=3.307e+02 | mgc=-2.755e+01
phase= 3 | nvar= 40 | iter= 60 | nll=3.304e+02 | mgc=-1.101e+00
Optimization complete with final statistics:
phase= 3 | nvar= 40 | iter= 68 | nll=3.304e+02 | mgc=+8.741e-05

Calculating Hessian: 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%... done!
Applying delta method to dependent variables... done!
Inverting Hessian: 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%... done!
Starting standard error calculations... done!

Finished running model 'catage' after 0.203 seconds.

The optimization reports a condense output, printing every iprint iteration (the same as current, defaulting to 10). Gradients and initial, intermediate, final statistics are also suppressed. If iprint=0 then only the final statistics are printed during optimization, like this:

Starting optimization of 'catage' in phase 1 of 3 at Wed Nov 24 13:42:09 2021
Optimization complete with final statistics:
phase= 3 | nvar= 40 | iter= 68 | nll=3.304e+02 | mgc=+8.741e-05

After optimization it reports a live progress bar on key steps in uncertainty calculations. This has the advantage that when an error is thrown it's clear in which step it occurs. In this model there are no errors. I also modified how warnings and errors are reported. As an example if I add an unused parameter (no link to NLL) then it adds this message:

Inverting Hessian: 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
 Warning: Parameter 40 appears to have identically 0 derivative.. check model
... done!
Starting standard error calculations... done!

Which alerts the user to check the .std file and indeed the SE is meaningless:

40 dummypar2 0.0000e+00 1.0000e+10

If I add two dummy parameters that are completely confounded then this gets printed:

 Warning: Parameter 1 appears to have identically 0 derivative.. check model
 Warning: Parameter 40 appears to have identically 0 derivative.. check model
 Error: Estimated variance of parameter 1 is -4.71308e+42, failed to write admodel.cov.
        Do not trust model output files. Fix model structure and reoptimize

This covers models that are well behaved, but presumably there are many other ways that they can fail that could use improved error messages/console feedback. Another advantage is the experimental "hess_step" feature is now much easier to interpret:

Number of hess_steps not specified, using default of 1
Experimental feature to take up to 1 Newton step(s) using the inverse Hessian
Initial max gradient=5.3001e-05 and min gradient= 0
Step 1: Updated max gradient=2.19925e-11 and min gradient= 0

Starting optimization of 'catage' in phase 3 of 3 at Wed Nov 24 14:09:46 2021

Calculating Hessian: 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%... done!
Applying delta method to dependent variables... done!
Inverting Hessian: 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%... done!
Starting standard error calculations... done!

The 1 Hessian step(s) reduced maxgrad from 5.3001e-05 to 2.19925e-11.
All output files should be updated, but confirm as this is experimental still.
The fact this was successful gives strong evidence of convergence to a mode
with quadratic log-likelihood surface.

Finished running model 'catage' after 0.109 seconds.

You can see that my strategy to update files after the successful hess_step is to run the optimizer with maxfn=0 and in the final phase... so that shows up.

So I think I have a pretty good beta version of this. There are some aesthetic tweaks to make and other general tweaks to improve output. But for now I would like to request feedback and suggestions for further improvements.

Edit: This also applies to RE models (logistic example shown below). I disabled the inner optimization output which may not be a good idea in most cases. Maybe I should put it back in just for the final statistics?

Starting optimization of model 'logistic' in phase 1 of 2 total at Mon Nov 29 08:32:39 2021
phase=1 | nvar=5 | iter=0 | nll=3.073e+02 | mgc=-8.022e+02
phase=1 | nvar=5 | iter=10 | nll=1.822e+02 | mgc=1.579e-04
phase=2 | nvar=6 | iter=0 | nll=1.723e+02 | mgc=2.397e+02
phase=2 | nvar=6 | iter=10 | nll=1.033e+02 | mgc=-4.162e-01
Optimization complete with final statistics:
phase=2 | nvar=6 | iter=15 | nll=1.033e+02 | mgc=6.478e-05 | exit code=1
30 6

Calculating Hessian: 0%... done!
Applying delta method to dependent variables... done!
Inverting Hessian: 0%... done!
Finished running model 'logistic' after 2.24 seconds.

Edit 2: I forgot to mention that if an sdreport variable is constant (no derivative) it prints warnings during that step, e.g.,

Starting standard error calculations...
 Warning: Non-positive variance of sdreport variables: 423, 424, 425, 442, 452
 done!

Some general questions:

• Do you have general suggestions to further improve?
• Do you agree w/ using this as the new default?
• Would you be willing to test the fork on your own models before I do a PR?

Thanks,
Cole

[iantaylor-NOAA]

Nice work @Cole-Monnahan-NOAA

1. I don't have any suggestions for further improvements at this time.
2. I like the proposed new default.
3. I'm happy to help test.

[IJStewart]

As another long-time ADMB user, I think these improvements to the default outputs would be very helpful.
Ian

[k-doering-NOAA]

Agree with Ian and Ian, thanks for working on this, Cole!

[Cole-Monnahan-NOAA]

I made another round of updates, mostly aesthetic. I swapped out mgc for mag, added timings to each step, cleaned up output, and fixed a few more warnings. I think we should also set the default iprint to 20 or even higher when using the new option.

Please see pull request #220 to review code changes.

Here's what the glmmadmb ADMB-RE exmaple model looks like now (iprint 30):

Starting RE optimization of 'glmmadmb' in phase 1 of 4 at Thu Dec 02 16:13:40 2021
phase= 1 | nvar=  4 | iter=  0 | nll=5.787e+02 | mag=4.238e+00
phase= 3 | nvar=  6 | iter=  0 | nll=5.576e+02 | mag=2.482e+00
phase= 3 | nvar=  6 | iter= 30 | nll=3.460e+02 | mag=3.029e+00
Hessian type 4
phase= 4 | nvar= 11 | iter=  0 | nll=2.245e+02 | mag=4.616e-01
phase= 4 | nvar= 11 | iter= 30 | nll=2.135e+02 | mag=6.719e-02
phase= 4 | nvar= 11 | iter= 60 | nll=2.027e+02 | mag=4.069e-02
phase= 4 | nvar= 11 | iter= 90 | nll=1.990e+02 | mag=1.407e-02
phase= 4 | nvar= 11 | iter=120 | nll=1.987e+02 | mag=6.544e-03
Optimization successful after 2.9 minutes with final statistics:
 nll=198.671159 | mag=6.07401e-06

Calculating Hessian: 0%, 20%, 40%, 60%, 80%, 100% done! (35.8 s)
Differentiating 3 derived quantities: 0, 1, 2, 3 done! (0.1 s)
Inverting Hessian: 0%, 20%, 40%, 60%, 80%, 100% done! (0 s)
Starting standard error calculations...  done! (0.3 s)

Finished running model 'glmmadmb' after 3.54 mins.

I also ran a stock synthesis model (iprint 100). Any model with its own output will necessarily collide with the output, but I still think it's an improvement. Note how the warnings about maxfn being exceeded are much clearer now. It's stalling out and not making progress in some of the phases. But appears to finish in the final phase. I tried -maxfn 150 and it failed to optimize correctly. This suggests to me there are some paramterization issues interacting with phasing. I hope this cleaner output helps identify issues like this.

Starting optimization of 'ss' in phase 1 of 5 at Thu Dec 02 16:02:53 2021
0 0 -log(L): 0  between 
 MG setup OK 

[truncated]

 OK with obj_func 8311.74
phase= 1 | nvar= 48 | iter=  0 | nll=8.312e+03 | mag=9.234e+03
1 101 -log(L): 3183.16  between 
phase= 2 | nvar= 58 | iter=  0 | nll=3.183e+03 | mag=1.183e+04
Exiting without success due to excessive function evaluations (maxfn=100) | mag=99.625
2 203 -log(L): 2118.97  between 
phase= 3 | nvar= 82 | iter=  0 | nll=2.119e+03 | mag=9.324e+01
3 285 -log(L): 2090.16  between 
phase= 4 | nvar= 85 | iter=  0 | nll=2.090e+03 | mag=8.462e+01
Exiting without success due to excessive function evaluations (maxfn=100) | mag=1.56763
4 387 -log(L): 2069.53  between 
phase= 5 | nvar=224 | iter=  0 | nll=2.070e+03 | mag=1.100e+03
phase= 5 | nvar=224 | iter=100 | nll=1.870e+03 | mag=1.673e+02
phase= 5 | nvar=224 | iter=200 | nll=1.868e+03 | mag=1.325e+00
Optimization successful after 59.1 seconds with final statistics:
 nll=1867.836289 | mag=1.84876e-05
 writing big output now 
Wrote bigoutput and bodywt for last_phase in REPORT_SECTION and before hessian, no benchmark or forecast 

Calculating Hessian: 0%, 20%, 40%, 60%, 80%, 100% done! (1.2 mins)
Differentiating 8 derived quantities: 0do benchmark and forecast if requested in sdphase
, 1, 2, 3, 4, 5, 6, 7, 8 done! (51.4 s)
Inverting Hessian: 0%, 20%, 40%, 60%, 80%, 100% done! (1 s)
Starting standard error calculations...  done! (7.6 s)

In final section 
Finish time: Thu Dec 02 16:06:06 2021
Elapsed time: 0 hours, 3 minutes, 13 seconds.
 Iterations: 1591 -log(L): 1867.84
Final gradient: 1.84876e-05

 finished COVAR.SSO
 [Truncated]
Global_MSY:  finished 

!!  Run has completed  !!            See warning.sso for N warnings: 3

Finished running model 'ss' after 3.35 mins.

[JaneSullivan-NOAA]

Thanks for your work on this Cole! I think it's great and a huge improvement to the existing console output. There were just a couple questions/thoughts that came up for me while reviewing:

1. Could you document here why you switched from mgc to mag and what the 'a' stands for?
2. Would it be too much of a lift to add a check for parameters hitting bounds?

[Cole-Monnahan-NOAA]

Hi Jane, thanks for the feedback.

1. mag= maximum absolute gradient=abs(mgc). Just ignores the sign which cleans things up a bit on the console. Very minor.
2. Other users had this same idea previously and I opened an issue about it: Report parameters estimated at or near bounds #221 . Not many ideas over there for now.

[Cole-Monnahan-NOAA]

Hi all, so we are close to getting this working. @wStockhausen and I made some new improvements to this feature.

1. It works for all RE and autodiff models
2. Instead of printing the parameter number, it prints the parameter name. This makes it much easier to interpret the output.
3. A new feature to check for parameters near bounds is implemented. It runs right after optimization, before the Hessian calculations. This output goes to the console but is also appended to the log file.
4. Improved hess_step functionality and console output.

See model output examples below.

We have tested this thoroughly locally and @johnoel will test it further and do code review. This in combination with a major ADMB release led us to decide setting this as the new default to be the best option. So if you have any objections to that please let us know ASAP. It is possible that the new features will error out for some existing models, and in that case the user can just run with -output 2 to revert to the old behavior.

Here's a model without bound issues:

Starting optimization of 'catage' in phase 1 of 3 at Wed Feb 02 16:04:35 2022
phase= 1 | nvar=  2 | iter=  0 | nll=9.49e+03 | mag=5.94e+03 | par=log_q
phase= 2 | nvar= 26 | iter=  0 | nll=5.15e+02 | mag=5.19e+02 | par=log_sel_coff[1]
phase= 2 | nvar= 26 | iter= 50 | nll=3.55e+02 | mag=1.26e+00 | par=log_relpop[6]
phase= 3 | nvar= 38 | iter=  0 | nll=3.55e+02 | mag=2.42e+02 | par=effort_devs[1]
phase= 3 | nvar= 38 | iter= 50 | nll=3.31e+02 | mag=2.76e+01 | par=log_sel_coff[4]
Optimization completed after 0 seconds with final statistics:
 nll=330.403817 | mag=8.74065e-05 | par=log_relpop[2]

Checking for estimated parameters on bounds... done!
Calculating Hessian: 0%, 20%, 40%, 60%, 80%, 100% done! (0 s)
Differentiating 4 derived quantities: 0, 1, 2, 3, 4 done! (0 s)
Inverting Hessian: 0%, 20%, 40%, 60%, 80%, 100% done! (0 s)
Starting standard error calculations...  done! (0 s)

Finished running model 'catage' after 0.1 s.

and here is one with bound issues:

Starting optimization of 'bound_checks' in phase 1 of 1 at Wed Feb 02 16:04:14 2022
phase= 1 | nvar= 16 | iter=  0 | nll=2.04e+02 | mag=7.85e+01 | par=parvec[1]
phase= 1 | nvar= 16 | iter= 10 | nll=1.26e+01 | mag=1.08e+00 | par=parvec[7]
phase= 1 | nvar= 16 | iter= 20 | nll=1.23e+01 | mag=1.32e-02 | par=parvec[1]
phase= 1 | nvar= 16 | iter= 30 | nll=1.23e+01 | mag=4.83e-04 | par=parvec[5]
phase= 1 | nvar= 16 | iter= 40 | nll=1.23e+01 | mag=2.30e-03 | par=parvec[7]
Optimization completed after 0 seconds with final statistics:
 nll=12.322236 | mag=6.05871e-05 | par=parvec[7]

Checking for estimated parameters on bounds...
   par1 is near lower bound: 0 < 0.000100554 < 1
   par2 is near lower bound: 0 < 2.00202e-06 < 1
   par3 is near upper bound: 0 < 0.999899 < 1
   par4 is near upper bound: 0 < 0.999998 < 1
   parvec[1] is near lower bound: 0 < 2.00854e-08 < 1
   parvec[2] is near lower bound: 0 < 9.63058e-05 < 1
   parvec[3] is near lower bound: 0 < 9.76947e-05 < 1
   parvec[4] is near lower bound: 0 < 0.000113818 < 1
   parvec[7] is near upper bound: 0 < 1 < 1
   parvec[8] is near upper bound: 0 < 0.999898 < 1
   parvec[9] is near upper bound: 0 < 0.999902 < 1
   parvec[10] is near upper bound: 0 < 0.999886 < 1
 done!
Calculating Hessian: 0%, 20%, 40%, 60%, 80%, 100% done! (0 s)
Inverting Hessian: 0%, 20%, 40%, 60%, 80%, 100% done! (0 s)
Starting standard error calculations...  done! (0 s)

[iantaylor-NOAA]

I support the choice to implement this by default in the next release. I think it will be more widely used and appreciated that way, and if there are indeed model configurations where it doesn't work, they will be found faster this way.
Thank you @Cole-Monnahan-NOAA, @johnoel, and @wStockhausen.

[JaneSullivan-NOAA]

I echo Ian's gratitude to @Cole-Monnahan-NOAA, @johnoel, and @wStockhausen and support making this feature default in the next release.
A couple thoughts:
The output would benefit from being just a little more explicit about the results. For example (see *** for proposed revisions):

Starting optimization of 'catage' in phase 1 of 3 at Wed Feb 02 16:04:35 2022
phase= 1 | nvar=  2 | iter=  0 | nll=9.49e+03 | mag=5.94e+03 | par=log_q
phase= 2 | nvar= 26 | iter=  0 | nll=5.15e+02 | mag=5.19e+02 | par=log_sel_coff[1]
phase= 2 | nvar= 26 | iter= 50 | nll=3.55e+02 | mag=1.26e+00 | par=log_relpop[6]
phase= 3 | nvar= 38 | iter=  0 | nll=3.55e+02 | mag=2.42e+02 | par=effort_devs[1]
phase= 3 | nvar= 38 | iter= 50 | nll=3.31e+02 | mag=2.76e+01 | par=log_sel_coff[4]
Optimization completed after 0 seconds with final statistics:
 nll=330.403817 | mag=8.74065e-05 | par=log_relpop[2]

Checking for estimated parameters on bounds... done! ****No parameters estimates near bounds.****
Calculating Hessian: 0%, 20%, 40%, 60%, 80%, 100% done! (0 s)
Differentiating 4 derived quantities: 0, 1, 2, 3, 4 done! (0 s)
Inverting Hessian: 0%, 20%, 40%, 60%, 80%, 100% done! (0 s)
Starting standard error calculations...  done! (0 s)

Finished running model 'catage' after 0.1 s.

And the example with bounding issues...

Starting optimization of 'bound_checks' in phase 1 of 1 at Wed Feb 02 16:04:14 2022
phase= 1 | nvar= 16 | iter=  0 | nll=2.04e+02 | mag=7.85e+01 | par=parvec[1]
phase= 1 | nvar= 16 | iter= 10 | nll=1.26e+01 | mag=1.08e+00 | par=parvec[7]
phase= 1 | nvar= 16 | iter= 20 | nll=1.23e+01 | mag=1.32e-02 | par=parvec[1]
phase= 1 | nvar= 16 | iter= 30 | nll=1.23e+01 | mag=4.83e-04 | par=parvec[5]
phase= 1 | nvar= 16 | iter= 40 | nll=1.23e+01 | mag=2.30e-03 | par=parvec[7]
Optimization completed after 0 seconds with final statistics:
 nll=12.322236 | mag=6.05871e-05 | par=parvec[7]

Checking for estimated parameters on bounds...***One or more parameter(s) estimated on or near bounds***
   par1 is ***on or*** near lower bound: 0 < 0.000100554 < 1
   par2 is ***on or***near lower bound: 0 < 2.00202e-06 < 1
   par3 is ***on or*** near upper bound: 0 < 0.999899 < 1
   par4 is near upper bound: 0 < 0.999998 < 1
   parvec[1] is near lower bound: 0 < 2.00854e-08 < 1
   parvec[2] is near lower bound: 0 < 9.63058e-05 < 1
   parvec[3] is near lower bound: 0 < 9.76947e-05 < 1
   parvec[4] is near lower bound: 0 < 0.000113818 < 1
   parvec[7] is near upper bound: 0 < 1 < 1
   parvec[8] is near upper bound: 0 < 0.999898 < 1
   parvec[9] is near upper bound: 0 < 0.999902 < 1
   parvec[10] is near upper bound: 0 < 0.999886 < 1
 done!
Calculating Hessian: 0%, 20%, 40%, 60%, 80%, 100% done! (0 s)
Inverting Hessian: 0%, 20%, 40%, 60%, 80%, 100% done! (0 s)
Starting standard error calculations...  done! (0 s)

Are you all planning to add documentation on this feature in the user manual? If so I would be happy to review. I think a novice user would benefit from having the following questions answered: (1) What does it mean if a parameter is hitting a bound? (e.g. is the parameter not estimable? is the model not actually converged? does it render other parameter estimates invalid?), and then of course (2) What do I do next? (e.g. fix parameters? transformation? re-parameterize?)

Oh, and sorry to bring it up again but mag should defined in the documentation or in the output itself. Its otherwise not googleable unless someone makes it onto this thread.

Thanks again for your work.

[Cole-Monnahan-NOAA]

@JaneSullivan-NOAA Good ideas. Done!

Yeah we can update the documentation. Those great and "big" questions that depend a lot on the context and I'm not sure I can answer them, or if they should be in the ADMB dox. Open to thoughts on that.

Checking for estimated parameters on bounds... none found!

Checking for estimated parameters on bounds...
  Warning: the following parameters had issues:
   par1 is within 0.01% of lower bound: 0 < 0.000100554 < 1
   par2 is within 0.001% of lower bound: 0 < 2.00202e-06 < 1
  [truncated]
   parvec[10] is within 0.01% of upper bound: 0 < 0.999886 < 1
 done!

[JaneSullivan-NOAA]

@Cole-Monnahan-NOAA the new output looks fantastic. I especially like the idea of presenting it as a warning msg.

Great point about the importance of context. I think a simple, adequately caveated statement would go a long way towards enabling a new user to troubleshoot on their own or ask more informed questions.

For example:
"This diagnostic is presented as a warning message in ADMB because the underlying cause is context-dependent and could have varying levels of severity with respect to statistical inference. When a parameter hits a bound, it could be an indication that the true estimate lies outside the bounds, that the model has strong parameter correlation or is mis-specified, or that there is insufficient data to estimate the parameter in question. Potential ways to alleviate this issue include fixing the parameter, applying a transformation that constrains the parameter within the desired bounds (e.g. log or logit), putting a penalty or prior on the parameter if appropriate, increasing the limits of the parameter bounds, or reparameterizing the model."

In terms of whether or not it should be in the ADMB documentation... From my perspective ADMB is a statistical software, which makes its documentation is a great place for information on model diagnostics. This is something that Stan does exceptionally well, albeit with a much larger team of developers.

[johnoel]

Changes for new output display (Issue #221) was merged (Pull #230) into the dev branch. Please test and provide feedback.

[arni-magnusson]

Great stuff, Cole!
In pull request #248 I propose tiny improvements.

[kellijohnson-NOAA]

@johnoel thank you for merging these changes in and everyone, especially @Cole-Monnahan-NOAA, for all of the development. I am curious on the status of documenting the output to the console. After reading the thread here it was clear but when I went to the manual on the ADMB website, the last update to the pdf was from 2021 and it is for version 12.3. Should I be looking somewhere else for the newest documentation?

[Cole-Monnahan-NOAA]

I have not done any documentation and would welcome some help there.

I guess I never answered @JaneSullivan-NOAA specific questions above. (1) If a parameter is on a bound it violates the MLE statistical assumptions used for asymptotic uncertainty calculations (estimate is in the interior of the parameter space) so the resulting uncertainty estimates for at least that parameter are unreliable. You see this in practice as nonsensical SEs that are negligible or gigantic (although this could be alleviated with this proposed feature). I would assume any parameter correlated with that one will also be unreliable. Yes it is inestimable. Is it converged? I suppose it depends on your definition. It might have a small gradient. I would hesitate to call it converged though. (2) What to do? Well it's already transformed so that's not an option. You could (a) widen the bounds if that makes sense; (b) fix the parameter at the bound or a more reasonable value; (c) add prior/penalty if appropriate.

[JaneSullivan-NOAA]

Thanks @Cole-Monnahan-NOAA and @kellijohnson-NOAA. I looked at the docs/manuals structure and am not sure I have the expertise to update the manual efficiently. Sorry... but I'd be happy to review that section of the documentation when ready! :)

[Cole-Monnahan-NOAA]

Just to be clear, there is no new functionality in this new version. We just made the optimizer behavior more transparent and clear about issues. So in some sense there should be nothing new to add to the manual, although I admit that's not a very fair argument. @johnoel any thoughts on how to proceed?

[johnoel]

Continued in #286.