NOTES of NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE by Dzmitry Bahdanau et al. (2016)

Traditional

An encoder neural network reads and encodes a source sentence
into a fixed-length vector.

A decoder then outputs a translation from the encoded vector.

The whole encoder–decoder system, which consists of the encoder and the decoder for a language pair,
is jointly trained to maximize the probability of a correct translation given a source sentence.

Issue: a neural network needs to be able to compress all the necessary information of a source sentence into a fixed-length vector, difficult to cope with long sentences (especially longer than those in the training corpus).

Our work

Align and translate jointly.

Each time the proposed model generates a word in a translation, it (soft-)searches for a set of positions in a source sentence where the most relevant information is concentrated.

The model predicts a target word based on the context vectors associated with these source positions and all the previous generated target words.

It does not attempt to encode a whole input sentence into a single fixed-length vector. Instead, it encodes the input sentence into a equence of vectors and chooses a subset of these vectors adaptively while decoding the translation. Allow a model cope better with long sentences.

Problem depiction of translation

Translation is equivalent to finding a target sentence y$y$ that maximizes
the conditional probability of y$y$ given a source sentence x$x$, i.e., argmaxyp(y|x)$argma{x}_{y}p(y|x)$.

In neural machine translation, we fit a parameterized model to maximize the conditional probability of sentence pairs using a parallel training corpus.

background-basic work

In the Encoder–Decoder framework, an encoder reads the input sentence, a sequence of vectors x=(x1;...;xTx)$x=(x_1;...;x_{T_x})$, into a vector c$c$. The most common approach is to use an RNN such that

(1)
and

where is a hidden state at time t$t$, and c$c$ is a vector generated from the sequence of the hidden states. f$f$ and q$q$ are some nonlinear functions. (Sutskever et al.) used an LSTM as f$f$ and q({h1;...;hTx})=hT$q(\{h_1;...;h_{T_x}\})=h_T$ , for instance.

The decoder is often trained to predict the next word yt′$y_{t^{\prime }}$ given the context vector c$c$ and all the previously predicted words {y1;...;yt′−1}$\{y_1;...;y_{t^{\prime }-1}\}$. In other words, the decoder defines a probability over the translation y by decomposing the joint probability into the ordered conditionals:

(2)

wherey=(y1,...,yTy)$y=(y_1,...,y_{T_y})$. With an RNN, each conditional probability is modeled as

(3)
here g$g$ is a nonlinear, potentially multi-layered, function that outputs the probability of yt$y_t$, and st$s_t$ is the hidden state of the RNN.

Learing to align and translate

decoder

define each conditional probability
in Eq. (2) as:

(4)

where si$s_i$ is an RNN hidden state for time i$i$, computed by

here the probability is conditioned on a distinct context vector ci$c_i$ for each target word yi$y_i$.

The context vector ci$ci$ depends on a sequence of annotations (h1;...;hTx)$(h_1;...;h_{T_x})$ to which an encoder maps the input sentence. Each annotation hi$h_i$ contains information about the whole input sequence with a strong focus on the parts surrounding the i-th word of the input sequence.

(5)

and αij=exp(eij)∑Txk=1exp(eik)${\alpha }_{ij}=\frac{exp(e_{ij})}{\sum _{k=1}^{T_x}exp(e_{ik})}$

where eij=a(si−1,hj)$e_{ij}=a(si-1,hj)$

is an alignment model which scores how well the inputs around position j$j$ and the output at position i$i$ match. The score is based on the RNN hidden state si−1$s_{i-1}$ (just before emitting yi$y_i$, Eq. (4)) and the
j-th annotation hj$h_j$ of the input sentence.

encoder

we would like the annotation of each word to summarize not only the preceding words, but also the following words. Hence, we propose to use a bidirectional RNN.

A BiRNN consists of forward and backward RNN’s. The forward RNN f→$\overrightarrow{f}$ reads the input sequence as it is ordered (from x1$x_1$ to xTx$x_{T_x}$ ) and alculates a sequence of forward hidden states (h1→;...;hTx−→)$(\overrightarrow{h_1};...;\overrightarrow{h_{T_x}})$.

The backward RNN f←$\overleftarrow{f}$ reads the sequence in the reverse order (from xTx$x_{T_x}$ to x1$x_1$), resulting in a sequence of backward hidden states (h1←;...;hTx←−)$(\overleftarrow{h_1};...;\overleftarrow{h_{T_x}})$.

In this way, the annotation hj$h_j$ contains the summaries of both the preceding words and the following words.