Where are you located? That will tell how much pest and disease pressure you have.
I do spray some chemicals so I do not plants flowering plants under or close to my fruit trees for the reason you mention.
If you want, you can do the low impact approach Scott Smith uses.
Low-Impact Spray Schedule (2019 Edition).
Don’t be discouraged by it. Familiarize yourself with Surround and how it work. You can learn how bagging works.
Another thought. Do not plant spring flowers that bloom the same time your fruit trees bloom. Plant summer flowers or thing like hostas instead. By the time your flowers bloom, may be you have bagged all your fruit and there is no need to spray.
.