It is not so much that you did anything wrong as that you hoped too much.
The efficacy of Neem oil and other organics depends on where the orchard is. Depends on the dirt also. Some places just have too many uninvited guests for Neem oil to work as well has you would like. Good example would be most of Virginia (especially the Shenandoah Valley) and northern NC. PC, RAA, SJS, JB, CM, and OFM, may be a bit discouraged but they will not be controlled so you can crop your trees. For that, you have to resort to things like Imidan, Assail, Altacor, Centaur, Delegate, Dipel, Verdepryn, and for those of you with licenses, Lorsban (Pilot 4E), (1x year), Lamba CY, Abemectin (AgriMek), Lannate, etc. The thing to do is get the Spray Bulletin from the local Land Grant Extension University, like Cornell, Penn State, Michigan State, University of Minnesota, Rutgers, VPI, etc. There is just so much going on - Fungus control is also important (Captan, Manzate Pro Stick, Ziram, Rally - all inexpensive - Syllit, Merivon, Pristine more expensive but they work) (mixing some together really works well - see spray Bulletin). Moreover, when the birds poke an apple the other bugs jump into their holes - that could mean nets. Also. If you do not spray calcium, you will get results that look like bugs (bitter pit) - same with Boron (corky fruit almost like bitter pit (spray at pink) if needed by a soil test (the Land Grant colleges also do this - usually for free - contact your county extension agent). Finally, Fireblight is a real terror to non-resistant rootstock and scions and it takes Copper (with dormant oil spray early) and Harbour (streptomyicin) (plus strict hygiene and prayer) to fight that. Finally, keep in mind that some cultivars, e.g., Honeycrisp, are better off without Phosphorus or much Potassium and it is better to avoid nitrogen (except as a carrier for Calcium or Boron) until after harvest. Bottom line: Integrated Pest Management. I practice hygiene (pruning, cleaning around the trees and use organics (like Neem oil and Regalia) when I can, but Delegate is much cheaper than its organic cousin Spinosad, which costs upward of $400/gallon.
I am find of saying the hardest work know to man is dairy farming (our county is #1 in Va for that) but that is closely followed by being an Orchardist.