Before considering how to also handle the last case Rob mentioned (if you choose to do so), I think it may be helpful to simplify the code a bit. Consider:
Should setChildOnly() and setParentOnly() be public at all? It seems to me that the public API should only be methods which will leave the two objects in a consistent state, ensuring bidirectional links. What reason does anyone outside this class have for calling setChildOnly() or setParentOnly()? For that matter, I'm not sure if anyone
inside this class really needs both of these methods. That can be decided later.
Do we need complex synchronization code inside both setChild() and setParent()? Or can we just call one method from the other?
In fact I'm not sure both these methods need to be part of the public API, as it's easy for someone to mistakenly think they need to call both of them - they don't. And anyone who wants to call a.setParent(b) can easily call b.addChild(a) instead, and be done.
Now, "complex stuff" could be exactly what Rob wrote for addChild() - it works well for everything except the case he noted, two nodes (accidentally?) made children of each other. If you want to also handle that case, I think it's probably easiest to avoid using synchronization entirely, and instead use either (a) methods from java.util.locks.Lock, or (b) optimistic locking techniques using AtomicInteger or AtomicLong. I recommend
Java Concurrency in Practice if you want to learn about these techniques in detail.