Bayesian Decision Methods

Type
Generative Classification Model

Also see, 4. Bayesian Decision Theory

Given observation x, the decision is based on posterior probability:

• Decide $y_1$, if $P(y_1|x)>P(y_2|x)$
• Decide $y_2$, if $P(y_2|x)>P(y_1|x)$

Note that, $P(y_1|x)=\frac{P(x|y_1).P(y_1)}{P(x)}$ and $P(y_2|x)=\frac{P(x|y_2).P(y_2)}{P(x)}$ so the probability $P(x)$ does not matter in our decision (same for both).

Probability of error:

$$\begin{array}{r}P(error|x)=\{\begin{array}{ll}P(y_1|x),& \text{if decide }{y}_2\text{ but }{y}_1\text{ is true}\\ P(y_2|x),& \text{if decide }{y}_1\text{ but }{y}_2\text{ is true}\end{array}\end{array}$$

Goal is to minimize error (based on single instance):

$P(error|x)=min[P(y_1|x),P(y_2|x)]$

Minimizing average error:

$P(error)={\int }_{-\infty }^{\infty }P(error,x)dx={\int }_{-\infty }^{\infty }P(error|x)p(x)dx$

Generalizing for more classes:

• Feature vector $x=(x1,x2,...,x_d)$ ∈ $R^d$: allow use of more than one feature
• $y1,y2,...,y_c$: finite set of c states of nature, i.e., categories (can be more than two)
• ${\alpha }_1,{\alpha }_2,...,{\alpha }_a$: a finite set of possible actions
• $\lambda ({\alpha }_i|y_i)$: loss function, describes the loss incurred for taking action ${\alpha }_i$ when state of nature is $y_i$
• $P(y_i)$: prior probability that state of nature is $y_i$
• $p(x|yi)$: state conditional probability for $x$

The expected loss, or conditional risk, of taking action ${\alpha }_i$ is:

$R({\alpha }_i|x)={\sum }_{j=1}^c\lambda ({\alpha }_i|y_j)P(y_j|x)$

Choose $\alpha (x)$ that minimizes overall risk:

$R=\int R(\alpha (x)|x)p(x)dx$

$R({\alpha }_i|x)={\sum }_{j=1}^c\lambda ({\alpha }_i|y_j)P(y_j|x)$

${\alpha }^{\ast }=argmi{n}_{{\alpha }_i}R({\alpha }_i|x)$