Topological insulators (TIs) are novel quantum materials with topologically nontrivial band structure induced by strong spin-orbit coupling. Breaking the time reversal symmetry (TRS) in TIs has been predicted to create a variety of exotic topological magnetoelectric effects such as image magnetic monopole and quantized anomalous Hall effect. In this talk we report transport studies of magnetically doped TI ultrathin films grown by molecular beam epitaxy (MBE), aiming to reveal the unique properties of the topological surface states and realize the quantum anomalous Hall effect.

In the Bi2Se3 films doped with Cr, we found a systematic crossover from weak antilocalization to weak localization induced by magnetic doping. We show that the evolution of the localization behavior indicates the transformation of the system from a topologically nontrivial TI to a topologically trivial dilute magnetic semiconductor. In a new ternary TI system (Bi1-xSbx)2Te3 with depleted bulk carriers, Cr dopants induce a long-range ferromagnetic ordering. More interestingly, the ferromagnetism exists both in the presence of hole- and electron-type Dirac fermions with widely varied carrier concentrations. The carrier-independent ferromagnetism in TIs is consistent with the Van Vleck mechanism mediated by the band electrons. This picture is further supported by recent observations of a topology-driven magnetic quantum phase transition, in which ferromagnetic ordering is strongly favored by the nontrivial bulk band topology. More recently, we have experimentally observed the quantum anomalous Hall effect, i.e., the quantum Hall effect without a magnetic field, in magnetic TIs.