Thanks @Drew51!
I want to keep it like a short (ish) fig bush and have branches/etc. I think you had mentioned doing something like that on another post I saw. Not exactly sure how big, but definitely not like a massive tree.
Also found this thread where you and other experienced growers have given advice and it seemed like the consensus was to prune it back a bit. I asked the OP on that thread how their plant is doing a few years out 
Thanks for the notes on the stone fruit.
I already did the “cut old wood by 1/2 and then cut new growth by 1/2” technique I heard about and read about. Unfortunately due to some brown rot of stone fruit, most of my new growth on both these peaches was affected or rotted out as it was budding. I managed to get the brown rot under control for now and we will see how they grow this year. Last year we also had rain and overcast skies for 4-5 months almost continuously and I think that contributed significantly to the issues. I think I’ll let the fruit set and then see later in the summer if I can do some structural pruning.
@Olpea, @scottfsmith, and @fruitnut - wondering if you had any thoughts on the peaches and stone fruit as well. (Or of course, on the fig).