My first thought is to not time stretch your drum loop. If you tempo match and/or stretch your drum break/loop to your working tempo then it won't have that choppy feel.
The only other obvious thing I can think of is, a long collected/curated library of drum samples. For example, a kick pulled...