The spinodal instability, i.e. the self-amplified deviation from the equilibrium state during a first order phase transition, of the baryonic rich quark matter is studied by both using the linear response theory and solving the Boltzmann equations with the test particle method. The former approach includes the quantum effect but only works near equilibrium, while the second one is semi-classical but capable of describing a highly non-equilibrated system. In the first approach, we obtain both the spinodal boundaries of the unstable modes of different wavelengths and the growth rates of them at a certain temperature and baryon density in the early stage of phase separation.  In the second approach, we study the spinodal instability of the baryonic rich quark matter in both a closed box and an expanding fireball by investigating the time evolution of the quantities such as the scaled density moments, the event distribution of the particle numbers in a sub-volume, the event distribution of the anisotropic flows, and the dilepton yield.