Consider the case of moving only one particle at a time which originally had the following time scaling for one transition:

$$O_R(N)+O_\mathrm{inverse}(N^2)$$

For the factorized determinants one of the two determinants is obviously unaffected by the change so that it cancels from the ratio $R$.