NN.Optimisers.Instances

Definitions

GD : Neg pType => {auto mon : ComMonoid pType} -> FromDouble pType => {default 0.001 _ : pType} -> Optimiser (Const pType) ()

  Gradient descent optimiser. Has trivial state
  @lr is the learning rate

Totality: total
Visibility: public export

GA : Neg pType => {auto mon : ComMonoid pType} -> FromDouble pType => {default 0.001 _ : pType} -> Optimiser (Const pType) ()

  Gradient ascent optimiser. Has trivial state
  @lr is the learning rate

Totality: total
Visibility: public export

momentumUpdate : Neg pType => pType -> pType -> pType -> pType -> (pType, pType)

Totality: total
Visibility: public export

lookAhead : Num pType => pType -> pType -> pType -> pType

Totality: total
Visibility: public export

GDMomentum : Neg pType => {auto mon : ComMonoid pType} -> FromDouble pType => {default False _ : Bool} -> {default 0.001 _ : pType} -> {default 0.9 _ : pType} -> Optimiser (Const pType) pType

  Gradient Descent with momentum, optionally with Nesterov acceleration

Totality: total
Visibility: public export

adamUpdate : Neg pType => Fractional pType => Sqrt pType => FromDouble pType => Materialise pType => Double -> Double -> pType -> pType -> pType -> pType -> Double -> Double -> pType -> (pType, (pType, (pType, (Double, Double))))

  Adam step. The moments are parameter-shaped; the bias-correction powers
  beta^t are the same scalar at every coordinate, so they are `Double`

Totality: total
Visibility: public export

Adam : Neg pType => {auto mon : ComMonoid pType} -> FromDouble pType => Materialise pType => Fractional pType => Sqrt pType => {default 0.001 _ : pType} -> {default 0.9 _ : Double} -> {default 0.999 _ : Double} -> {default 1e-8 _ : pType} -> Optimiser (Const pType) (pType, (pType, (Double, Double)))

  Adam optimiser (Kingma & Ba, 2014)
  State: the two moments and the two scalar bias-correction powers
  @lr is the learning rate
  @beta1 is the exponential decay rate for the first moment estimate
  @beta2 is the exponential decay rate for the second moment estimate
  @epsilon is a small constant for numerical stability

Totality: total
Visibility: public export