It depends on when the lack of moisture is prevalent during their growing cycle. It's my understanding that dry winters have an adverse effect on top antler growth. In other words, their 4th/5th/6th points lack any length/mass. On the other hand, if it's dry during spring/early summer, their fronts (brow tines & 3rds) are weak.
A dry winter/spring/summer scenario can negatively impact antler growth quite heavily.
This can easily result in a 20" to 40" deficit compared to antler growth in wetter years.