It depends on the deer I guess, the tree I planted at my parents house got sampled for a while. But that was a doe who raised a fawn in their back lot so they tolerated them when they’d come in the yard and they eat everything now. At the farm fig trees are about the only thing deer won’t eat.
The bucks will go after them during the rut, they should regrow the next year so it is not necessarily a big problem for small trees. But sometimes they will work them over so hard they loosen up in the soil or split a bad crotch, and they like to hit the same ones every year.