Abstract
Quantum coherence constitutes a fundamental physical mechanism essential to the study of quantum algorithms. We study the coherence and decoherence in generalized Shor's algorithm where the register A is initialized in arbitrary pure state, or the combined register AB is initialized in any pseudo-pure state, which encompasses the standard Shor's algorithm as a special case. We derive both the lower and upper bounds on the performance of the generalized Shor's algorithm, and establish the relation between the probability of calculating r when the register AB is initialized in any pseudo-pure state and the one when the register A initialized in arbitrary pure state. Moreover, we study the coherence and decoherence in noisy Shor's algorithm and give the lower bound of the probability that we can calculate r.