KL-Divergenz

Einfach erklärt

KL-Divergenz misst, wie unterschiedlich zwei Wahrscheinlichkeitsverteilungen sind.

Die Formel:

DKL(P||Q) = Σ P(x) × log(P(x) / Q(x))

P = "wahre" Verteilung
Q = approximierte Verteilung

Beispiel:

P = [0.5, 0.5]  (faire Münze)
Q = [0.9, 0.1]  (unfaire Münze)

DKL(P||Q) = 0.5×log(0.5/0.9) + 0.5×log(0.5/0.1)
          = 0.5×(-0.59) + 0.5×(1.61)
          = 0.51 bits

Technischer Deep Dive

KL-Divergenz berechnen

import numpy as np

def kl_divergence(p, q):
    # Vermeidet log(0)
    p = np.clip(p, 1e-10, 1)
    q = np.clip(q, 1e-10, 1)
    return np.sum(p * np.log(p / q))

p = np.array([0.5, 0.5])
q = np.array([0.9, 0.1])
print(f"KL(P||Q) = {kl_divergence(p, q):.3f}")  # 0.510
print(f"KL(Q||P) = {kl_divergence(q, p):.3f}")  # 0.368 (nicht gleich!)

VAE Loss mit KL

def vae_loss(x, x_recon, mu, logvar):
    # Reconstruction Loss
    recon_loss = F.binary_cross_entropy(x_recon, x, reduction='sum')
    
    # KL Divergence: N(mu, sigma) vs N(0, 1)
    kl_loss = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    
    return recon_loss + kl_loss