We demonstrate that the output statistics of a quantum annealing protocol run on D-Wave 2000Q can be explained by static disorder garnishing an otherwise ideal device hardware. A Boltzmann-like distribution over distinct output states emerges with increasing problem size, and significantly reduces the chances for a correct identification of the sought-after optimal solutions.