2 Graphical Models in Action

• From course Probabilistic Models and Inference Algorithms for Machine Learning, Prof. Dahua Lin
• All contents here are from the course and self understandings.

基本步骤

1. 理解你要求解的某个问题
   
   • what kind of entities/factors are involved?
   • How do they interact with each other?
   • Any constraints to take into account?
2. 建立模型
   
   • Introduce variables
   • Specify relations among them: make assumptions and modeling choices
   • Formalize the graphical model
3. Derive the inference & estimation algorithms

Gaussian Mixture Model (GMM)

1. Motivation & Assumptions

• Observation: clusters （如图中所示）
• Assumptions:
  
  • latent components
  • independent generation of components （一个component指一个如图所示的cluster）
  • independent generation of points in each component.

2. Formulate Gaussian Mixture Model

• Variables:
  
  • sample points: xi
  • component indicators: zi
• Generative procedure: for each i
  
  • choose a component: zi∈π
  • generate a point: xi∈N(μzi,Σzi) (每个sample point 服从正态分布，mean和方差为μzi 和 Σzi)
• Model parameters:
  
  • component parameters: {(μk,Σk)}k=1:K
  • choice prior: π=(π2,⋯,πK)
• Joint distribution:
  
  p(X,Z|Θ)=∏i=1NpC(zi|π)pN(xi|μzi,Σzi)
• 该模型的图形表达如下：
  
  
  • 但是以上的模型不能泛化：当有多个groups 的数据时候，每一个 group Gm 都有一个 prior: πm，这样每一个group一个先验，对于新来的数据，并没有一个generalized的 πnew. 因此需要一个 Group-wise 的GMM

3. Extend Gaussian Mixture Model

• 一般来说，要泛华某个先验（prior），我们可以采用一些常见的分布，如下，这里我们采用Dirichlet
  
• Group-wise GMM: Generalizable to New Groups
  
  • Introduce a Dirichlet Prior over πm to allow the generation of new groups.
  • Formulation:
  • • For each group Gm:πm∼Dir(α).
  • • Generate the i-th point in Gm:
      zi∼πm
      xi∼N(μk,Σk)
  • • 注意 πm 现在是个 隐变量 (latent variable)
      
• Temporal Structures:
  
  • 在实际世界中，时域上的变化是很常见的，那对于模型来说，也需要对应的dynamics作用到 不同的变量上
  • Three ways to incorporate dynamics：
    Dynamics on xi
    Dynamics on zi
    Dynamics on π