We propose a scheme to realize universal quantum gates between nitrogen-vacancy (NV) centers in an optically trapped nanodiamond, through uniform magnetic field induced coupling between the NV centers and the torsional mode of the levitated nanodiamond. The gates are tolerant to the thermal noise of the torsional mode. By combining the scheme with dynamical decoupling technology, it is found that the high fidelity quantum gates are possible for the present experimental conditions. The proposed scheme is useful for NV-center-based quantum network and distributed quantum computation