For peaches the biggest effect on size has been the really quick warm-up, as well as the very dry conditions. As you recall, after peaches set, it got very hot very quick. This reduced the cell division of the peachlets resulting in a drastic reduction in cell count of the fruit.
All the peach growers in the KC area experienced this. One grower who grows 20 acres of peaches wrote Bill Shane about what we are all seeing. Growers are seeing very few 2" fruit (we’ve probably averaged 2-1/4" on these first varieties because we sell it softer where it has more time to swell). These are on irrigated and unirrigated orchards alike.
Bill Shane wrote back and confirmed it was probably the hot weather before pit hardening that did it. He sent a research link about it (see below). I’m certain drought is also contributing to the small size. I’ve seen dry years where fruit was significantly smaller. Everything is working against size this year.