NN.Architectures.Transformer.Attention

Definitions

crossAttention : Num a => {auto acif : NewAxisConsistent inputStructure [features]} -> {auto accf : NewAxisConsistent crossStructure [features]} -> {auto acci : NewAxisConsistent crossStructure [inputStructure]} -> TensorMonoid (inputStructure .cont) => TensorMonoid (features .cont) => AllAlgebra [inputStructure, features] a => {default id _ : (Tensor [crossStructure, inputStructure] a -> Tensor [crossStructure, inputStructure] a)} -> (Tensor [inputStructure] a -> Tensor [inputStructure] a) -> Tensor [inputStructure, features] a -> Tensor [inputStructure, features] a -> Tensor [crossStructure, features] a -> Tensor [crossStructure, features] a

  Generalised form of attention

Visibility: public export

selfAttention : Num a => {auto {conArg:17753} : NewAxisConsistent inputStructure [features]} -> TensorMonoid (inputStructure .cont) => TensorMonoid (features .cont) => AllAlgebra [inputStructure, features] a => {default id _ : (Tensor [inputStructure, inputStructure] a -> Tensor [inputStructure, inputStructure] a)} -> (Tensor [inputStructure] a -> Tensor [inputStructure] a) -> Tensor [inputStructure, features] a -> Tensor [inputStructure, features] a -> Tensor [inputStructure, features] a -> Tensor [inputStructure, features] a

  Self-attention is cross-attention where inputStructure = crossStructure

Visibility: public export

record SelfAttentionParams : Axis -> Type -> Type

  Data structure for holding parameters of self-attention

Totality: total
Visibility: public export
Constructor: 

MkSAParams : Tensor [features, features] a -> Tensor [features, features] a -> Tensor [features, features] a -> SelfAttentionParams features a

Projections:

.keyMatParam : SelfAttentionParams features a -> Tensor [features, features] a

.queryMatParam : SelfAttentionParams features a -> Tensor [features, features] a

.valueMatParam : SelfAttentionParams features a -> Tensor [features, features] a

.queryMatParam : SelfAttentionParams features a -> Tensor [features, features] a

Visibility: public export

queryMatParam : SelfAttentionParams features a -> Tensor [features, features] a

Visibility: public export

.valueMatParam : SelfAttentionParams features a -> Tensor [features, features] a

Visibility: public export

valueMatParam : SelfAttentionParams features a -> Tensor [features, features] a

Visibility: public export

.keyMatParam : SelfAttentionParams features a -> Tensor [features, features] a

Visibility: public export

keyMatParam : SelfAttentionParams features a -> Tensor [features, features] a

Visibility: public export

SAImpl : Num a => {auto ac : NewAxisConsistent inputStructure [features]} -> TensorMonoid (inputStructure .cont) => TensorMonoid (features .cont) => AllAlgebra [inputStructure, features] a => {default id _ : (Tensor [inputStructure, inputStructure] a -> Tensor [inputStructure, inputStructure] a)} -> (Tensor [inputStructure] a -> Tensor [inputStructure] a) -> DPair (Tensor [inputStructure, features] a) (const (SelfAttentionParams features a)) -> Tensor [inputStructure, features] a

  Forward pass of self-attention, from input

Visibility: public export

SelfAttention : Num a => {auto {conArg:18689} : NewAxisConsistent inputStructure [features]} -> TensorMonoid (inputStructure .cont) => TensorMonoid (features .cont) => AllAlgebra [inputStructure, features] a => {default id _ : (Tensor [inputStructure, inputStructure] a -> Tensor [inputStructure, inputStructure] a)} -> (Tensor [inputStructure] a -> Tensor [inputStructure] a) -> Tensor [inputStructure, features] a -\-> Tensor [inputStructure, features] a

  Self-attention as a parametric map

Visibility: public export

causalMask : Num a => Exp a => InterfaceOnPositions (c .cont) MOrd => TensorMonoid (c .cont) => Tensor [c, c] a -> Tensor [c, c] a

Visibility: public export