> For the complete documentation index, see [llms.txt](https://windmising.gitbook.io/mathematics-basic-for-ml/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://windmising.gitbook.io/mathematics-basic-for-ml/src/information/divergence.md).

# Divergence

## f-divergence

$$
D\_f(P||Q) = \int\_x q(x)f\left(\frac{p(x)}{q(x)}\right)dx
$$

且f(x) is convex && f(1) = 0

$D\_f(P||Q)$用于评估分布P和分布Q的差异。\
f(1) = 0 ==> 当p(x)=q(x)时，Df = 0\
f is convex ==> Df >= f(1) = 0

令$f(x) = x\log x$，则Df是KL divergence

$$
D\_f = \int\_x p(x)\log\left(\frac{p(x)}{q(x)}\right)
$$

令$f(x) = -\log x$，则Df是reverse KL divergence

$$
D\_f = \int\_x q(x)\log\left(\frac{q(x)}{p(x)}\right)
$$

令$f(x) = (x-1)^2$，则Df是Chi Square Divergence

$$
D\_f = \int\_x\frac{(p(x)-q(x))^2}{q(x)}dx
$$

## KL散度 KL divergence

有两个独立的分布P(x)和Q(x)，KL散度用于衡量这两个分布的差异。

$$
D\_{KL}(P||Q) = E\_{X\sim P}\left\[\log \frac{P(x)}{Q(x)}\right] = E\_{X\sim P}\[\log P(x) - \log Q(x)]
$$

### 交叉熵 cross-entropy

$$
H(P, Q) = -E\_{X\sim P} \log Q(x)
$$

最小化交叉熵等价于最小化KL散度。

## JS散度 Jensen-Shannon

<http://blog.sina.com.cn/s/blog\\_18bdda0da0102xzpw.html>