Fruit trees have limited capacity to produce fruit, which is proportional to their root volume among other things. So given a fixed capacity a young tree can produce a lot of small fruits vs. a smaller number of healthy size fruits. Also worth considering is the capability of younger branches to hold the fruit without excessive bending (or breaking). My father taught me that on peach/nectarine - any branch with diameter less than a pencil is going to have a difficult time -- except for those locations immediately at its base next on a robust branch. Also, on 1/2" to 1" branches there can be too much fruit -- try to space them about 4 fingers apart.
Finally, here is a general rule of thumb about "allowing" fruit on young trees: each 1/2 pound of fruit you let develop could have been 3 feet of branch growth.