Dynamic topic model

Topics

Similarly to LDA and pLSA, in a dynamic topic model, each document is viewed as a mixture of unobserved topics. Furthermore, each topic defines a multinomial distribution over a set of terms. Thus, for each word of each document, a topic is drawn from the mixture and a term is subsequently drawn from the multinomial distribution corresponding to that topic.

The topics, however, evolve over time. For instance, the two most likely terms of a topic at time t could be "network" and "Zipf" (in descending order) while the most likely ones at time t+1 could be "Zipf" and "percolation" (in descending order).

Model

Define

α t as the per-document topic distribution at time t. β t , k as the word distribution of topic k at time t. η t , d as the topic distribution for document d in time t, z t , d , n as the topic for the nth word in document d in time t, and w t , d , n as the specific word.

In this model, the multinomial distributions α t + 1 and β t + 1 , k are generated from α t and β t , k , respectively. Even though multinomial distributions are usually written in terms of the mean parameters, representing them in terms of the natural parameters is better in the context of dynamic topic models.

The former representation has some disadvantages due to the fact that the parameters are constrained to be non-negative and sum to one. When defining the evolution of these distributions, one would need to assure that such constraints were satisfied. Since both distributions are in the exponential family, one solution to this problem is to represent them in terms of the natural parameters, that can assume any real value and can be individually changed.

Using the natural parameterization, the dynamics of the topic model are given by

β t , k | β t − 1 , k ∼ N ( β t − 1 , k , σ 2 I )

and

α t | α t − 1 ∼ N ( α t − 1 , δ 2 I ) .

The generative process at time slice 't' is therefore:

1. Draw topics β t , k | β t − 1 , k ∼ N ( β t − 1 , k , σ 2 I ) ∀ k
2. Draw mixture model α t | α t − 1 ∼ N ( α t − 1 , δ 2 I )
3. For each document:
   1. Draw η t , d ∼ N ( α t , a 2 I )
   2. For each word:
      1. Draw topic Z t , d , n ∼ Mult ( π ( η t , d ) )
      2. Draw word W t , d , n ∼ Mult ( π ( β t , Z t , d , n ) )

where π ( x ) is a mapping from the natural parameterization x to the mean parameterization, namely

π ( x i ) = exp ⁡ ( x i ) ∑ i exp ⁡ ( x i ) .

Inference

In the dynamic topic model, only W t , d , n is observable. Learning the other parameters constitutes an inference problem. Blei and Lafferty argue that applying Gibbs sampling to do inference in this model is more difficult than in static models, due to the nonconjugacy of the Gaussian and multinomial distributions. They propose the use of variational methods, in particular, the Variational Kalman Filtering and the Variational Wavelet Regression.

Application

In the original paper, a dynamic topic model is applied to the corpus of Science articles published between 1881 and 1999 aiming to show that this method can be used to analyze the trends of word usage inside topics. The authors also show that the model trained with past documents is able to fit documents of an incoming year better than LDA.

A continuous dynamic topic model was developed by Wang et al. and applied to predict the timestamp of documents.