One of the first things you need to do is to figure out if you have curculio or oriental fruit moth or both. They require very different treatments. Find a worm and put it on your hand, if it crawls in a definite direction its OFM, if it just flails about its a PC (no legs).
If you have PC they can be controlled without poisons, I find Surround (edible clay particles that irritates them) works very well. But it does require 3-4 sprays in the spring. At this point the PC is done damaging for the year so its more planning for next year if thats the problem you have.
The OFM is more challenging organically, you need to spray spinosad and/or Bt when they are exposed (around egg laying time) and it requires spraying throughout the season. Surround also helps with them. If you didn’t find it yet I made a guide for my spray schedule which gives some hints on how to use Surround.
The major problem with growing peaches and plums organically is the rots. You need to spray a whole lot of sulphur to try to control them and will still lose a good amount. It depends on how humid it is where you are though. I now use synthetic disease control sprays to deal with the rots, but I was all organic for ten years or so and found it could work with effort.