CS224n：深度学习的自然语言处理（2017年冬季）1080p https://www.bilibili.com/video/av28030942/

How to represent meaning of words

meaning=denotation:
signifier (symbol) $\Leftrightarrow$⇔ signified (idea or thing)

Usable meaning in computer:
WordNet (synonym sets & hypernyms)

Shortcomings:

1. missing nuance
2. missing words’ new meaning
3. subjective
4. human labour requirement
5. hard to computer accurate word similarity

discrete symbols of representing words:
one-hot vectors
Vector dimension = number of words in vocabulary

shortcomings:
All vectors are orthogonal, no natural notion of similarity

Solution:
learn to encode similarity in the vectors themselves

Representing words by their context:
Since the meaning of word can be represented by those words around, the nearby context can be used to build up a representation of the centre word.

Method of building word vectors: (word embedding/representation)
Build a dense vector for each word, which is similar to vectors of words that appear in similar contexts. (words appear in similar contexts share similar vectors)

Word2vec

Overview:
steps:

1. a large corpus of text
2. represent every word in a fixed vocabulary with a vector
3. Go through each position t (with a centre word c and context/outside words o) in the text
4. Use word vectors’ similarity of c and o to calculate the probability $p(o|c)$p(o∣c) (probability of o given c)
5. adjusting word vectors to maximize the probability

example:

Here, $w_t$wt​ = “into”, is the previously mentioned centre word c.
The center word keeps changing as the position changes (centre word “into” --> “banking” --> “crises”…)

Objective function:
The likelihood of context words given centre word:
$L(\theta )=\prod_{t=1}^{T}\prod_{-m\leq j\leq m,j\neq 0}P(w_{t+j}|w_t;\theta)$L(θ)=t=1∏T​−m≤j≤m,j̸​=0∏​P(wt+j​∣wt​;θ)
Here, $\theta$θ represents variables to be optimised, $w_t$wt​ is the centre word, $w_{t+j}$wt+j​ is the context word, m stands for the size of window.
The objective function $J(\theta)$J(θ) is average negative log likelihood:

Tips: minimising the objective function $\Leftrightarrow $ Maximizing predictive accuracy

How to calculate $P(w_{t+j}|w_t;\theta)$P(wt+j​∣wt​;θ):

1. Use two vectors for each word w:
   $v_w$vw​ when w is centre word
   $u_w$uw​ when w is context word
2. For centre word c and context/outside word o:
   

example:

Here “into” is center word, thus $v_into$vi​nto is used, other words are context, thus their $u$u vectors are used, window size is 2.

The probability of o given c here is an example of softmax function.
(max: amplifies probability of largest value; soft: still assigns probability to small values)

Parameters optimization: (gradients are used)

The dimension of $\theta$θ: $\mathbb{R}^{2dV}$R2dV
Here, d is the dimension of vectors, V is the number of words, each word has 2 vectors (as center and as context)

Derivations of gradient

Why two vectors?
Easier optimization. Average both at the end.

Two models:

1. Skip-grams (SG): Predict context/outside words (position independent) given center word (used in the above example)
2. Continuous Bag of Words (CBOW): Predict center word from (bag of) context words

Additional efficiency in training:
Negative sampling (the above focus on softmax)

Gradient Descent:

This part will not be specifically covered here, just pay attention to the learning rate $alpha$alpha and use stochastic gradient descent (SGD) for saving time.