Encoder-decoder models

less than 1 minute read


Encoder decoder is a deep neural network architecture that consists of 2 components:

  • Encoder: input -> encoder memory (real-valued vector),
  • Decoder: encoder memory -> output. Input is variable length, encoder memory is fixed length.

Attachments/Pasted image 20220622113403.png 9.6. Encoder-Decoder Architecture — Dive into Deep Learning 0.17.5 documentation

The encoder and decoder portions can be swapped out. Sometimes the encoder is initialized with pretrained weights, for example in CodeBERT.

They are often used for Seq2seq tasks such as Neural Machine Translation (NMT).