My guess would be that when they are young, there is almost no self-shading. Then, as they get bigger, parts of the tree shade other parts of the tree at various points in the day and then none of it gets enough sun to set fruit.
That is the problem with living in a populated area. I have the same issue- you can pay to cut down a neighbors trees (preferably with permission), but they will rarely agree to let you take down their house 
I’m starting to get more than the few initial fruit ripening now. Here was this morning’s haul. Until today, I was getting ~6 per day for the last 3-4 days.
It is supposed to be pretty cloudy and rainy for the next few days (8 of the next 9) and I’m interested in how the trees will handle it. I’m hoping that there is already plenty of sugar (24 vs 28 wouldn’t be all that noticeable…) and this will just add juice and crispness (and maybe a bit of cracking).