Documentation

[willtebbutt]

The purpose of this issue is to plan out the first iteration of documentation in Tapir, and to figure out what order to do things in.

Structure plan:

Developer Docs

This is probably the most important documentation to do well, because it will make it possible to on-board other people. My basic plan is to start simple, and build up.

All of these documentations will be built around diagrams -- these will probably be hand-drawn to minimise the amount of time I have to spend messing around with annoying printing.

Explain the rrule!! abstraction #175 dealt with this

This page will explain in detail what rrule!!s and things output by build_rrule must do, with a couple of worked examples. A non-exhaustive list of points to discuss includes:

1. why data with addresses and data without addresses require different treatment
2. what can go wrong if we don't require that each memory address has a unique tangent memory address (i.e. all of the stuff around aliasing) -- provide a simple example
3. a simple mathematical model for the computation that rrules do

Differentiation a Function Comprising a Single Block

With the rrule abstraction introduced, we can discuss

1. how in principle rrules can be chained together to differentiate a composition of functions for which we have rules, and
2. how this is done in practice.

At the end of this section the reader should have a solid understanding of what's going on with AD in Tapir.jl. In particular, we'll be able to discuss the transformation which happen to each primal line with needing to think about control flow, and will be able to discuss the basic mechanisms by which we share information between the forwards and reverse passes, how we use + generate OpaqueClosures, etc.

Handling Dynamic Control Flow and CFG Transformations

With the above done, we can turn out attention to handling dynamic control flow. The basic idea is to convince the reader that each possible trajectory through the CFG corresponds to a different linearisation of the control

At this point, we'll need to explain how control flow is handled in SSA (phi nodes, goto / goto-if-not nodes etc), and then jointly explain how

1. each basic block gets transformed,
2. the overall CFG gets transformed.

These two steps are necessarily somewhat coupled.

BBCode Explainer

TODO: plan this

Tutorials

TODO: plan these

Interface Documentation

TODO: plan these

Misc:

1. Expand on argument about uniquely typed tangents giving conditional type stability
2. Worked hand-written example with control flow

[yebai]

The following paragraph is too brief for users to define new tangent types independently. Let's provide a complete example illustrating all (or more) necessary functions.

The point here is that you can manually resolve the circular dependency using a data structure which mimics the primal type. You will, however, need to implement similar methods for zero_tangent, randn_tangent, etc, and presumably need to implement additional getfield and setfield rules which are specific to this type.

https://compintell.github.io/Tapir.jl/stable/known_limitations/#Tangent-Types

[willtebbutt]

@yebai and I discussed today that a side-by-side comparison of Tapir.rrule!! with ChainRules.rrule should be included somewhere in the docs.