Abstract
Using spin- and angle-resolved spectroscopy and relativistic many-body calculations, we investigate the evolution of the electronic structure of (Bi_1-xIn_x)₂Se₃ bulk single crystals around the critical point of the trivial to topological insulator quantum-phase transition. By increasing x, we observe how a surface gap opens at the Dirac point of the initially gapless topological surface state of Bi₂Se₃, leading to the existence of massive fermions. The surface gap monotonically increases for a wide range of x values across the topological and trivial sides of the quantum-phase transition. By means of photon-energy dependent measurements, we demonstrate that the gapped surface state survives the inversion of the bulk bands which occurs at a critical point near x=0.055. The surface state exhibits a non-zero in-plane spin polarization which decays exponentially with increasing x, and that persists on both the topological and trivial insulator phases. Its out-of-plane spin polarization remains zero demonstrating the absence of a hedgehog spin texture expected from broken time-reversal symmetry. Our calculations reveal qualitative agreement with the experimental results all across the quantum-phase transition upon the systematic variation of the spin-orbit coupling strength. A non-time reversal symmetry breaking mechanism of bulk-mediated scattering processes that increase with decreasing spin-orbit coupling strength is proposed as explanation.