Continuous-time Markov chain

In probability theory, a continuous-time Markov chain is a stochastic process { X(t) : t ≥ 0 } that enjoys the Markov property and takes values in a set called the state space, each member of which is a state of the process. The Markov property states that at any times s > t > 0, the conditional probability distribution of the process at time s given the whole history of the process up to and including time t, depends only on the state of the process at time t. In effect, the state of the process at time s is conditionally independent of the history of the process before time t, given the state of the process at time t.

Intuitively, one imagines that during an infinitely small amount dt of time, the probability of transition from state i to state j is an infinitely small number p_ij dt. The probability that there has not been a transition after time dt is therefore 1 − p_ij dt. After some finite time t, the probability that there has been no transition is therefore

$(1-p_{ij}\,dt)^{t/dt}=e^{-p_{ij}t}.$

Therefore, the probability distribution of the waiting time until the first transition is a memoryless exponential distribution. Given that a process that started in state i has experienced a transition by time dt (an event with an infinitely small probability), the conditional probability that that transition is into state j is

${p_{ij}\,dt \over \sum _{k}p_{ik}\,dt}=p_{ij}$

the probability that more than one transition has occurred before time dt is neglected because dt is infinitesimal. Let P be the matrix whose entries are p_ij. The the probability that the process is in state j at time t, given that it started in state i, is the i,j entry of the matrix

$e^{Pt}=\sum _{n=0}^{\infty }{P^{n}t^{n} \over n!}$

(see matrix exponential). This is a semigroup of matrices whose infinitesimal generator is P.