They purportedly do best if planted in an area where the soil is heavily amended with sand or loam. They don't like super heavy soils. @clarkinks-- This might be your problem? And a problem I will need to work on...
I know a guy, Jim Davis, who has the world's biggest pawpaw farm, named "Deep Run" -- located north of Westminster, Maryland, near the Pennsylvania border. His property is on a bluff. After begging him on the phone, he let me and my family visit on two separate occasions.
At the bottom of his hilly property, he has two beach plum trees planted near the edge of a forest, in loamy soil. I have seen them twice-- in successive visits to the property over two early Septembers (to sample the pawpaws!).
The beach plum trees are about 10 feet tall - each, the picture of health - and both summers were LOADED with fruits. One has red plums. The other has blue plums. Mr. Davis said they are improved cultivars, but could not remember their names or sources of origin. He says he planted them many years ago. He does not spray them. He shows them nothing but neglect, and most years they reward his abuse by putting out an outrageous fruit load. He says he can't eat them all-- too many. Can you imagine?