I know exactly what you speak of. We had so many yellow jackets attack late peaches last year it was almost a cloud at times. I started using thick leather gloves to pick peaches.
This year I’ve maintained sprays longer for SWD, which has also slowed the wasps.
You don’t have to spray ripe fruit. They seem to practice avoidance spraying non ripe fruit. That said, I don’t know how well it will work once they’ve moved in.
There are some insecticides with short PHIs. For peaches Bathroid is a week and I believe malathion is 3 days.