Abstract
We study the ground state and the thermal phase diagram of a two-species Bose-Hubbard model, with U(1)× Z₂ symmetry, describing atoms and molecules on a 2D optical lattice interacting via a Feshbach resonance. Using quantum Monte Carlo simulations and mean field theory, we show that the conversion between the two-species, coherently coupling the atomic and molecular states, has a crucial impact on the Mott-Superfluid transition and stabilizes an insulating phase with a gap controlled by the conversion term -- \textit{the Feshbach insulator} -- instead of a standard Mott insulating phase. Depending on the detuning between atoms and molecules, this model exhibits three phases: the Feshbach insulator, a molecular condensate coexisting with non condensed atoms and a mixed atomic-molecular condensate. Employing a finite-size scaling method, we observe 3D XY (3D Ising) transition when U(1) ( Z₂) is broken whereas the transition is first-order when both U(1) and Z₂ symmetries are spontaneously broken. The finite temperature phase diagram is also discussed. The thermal disappearance of the molecular superfluid leads to a Berezinskii-Kosterlitz-Thouless transition with unusual universal jump in the superfluid density. The loss of the quasi-long-range coherence of the mixed atomic and molecular superfluid is more subtle since only atoms exhibit conventional Berezinskii-Kosterlitz-Thouless criticality. We also observe a classical first-order transition between the mixed superfluid and the normal Bose liquid at low temperature.