The normal
Java strategy is this: Try not to keep references to objects which you don't need. You seem to be trying to follow that strategy, but in a way which looks strange to me.
Let me put it this way: you have a B object which contains references to a set of A objects. But you say at some point you don't need those A objects any more, so you're going to change your B object so it doesn't refer to them any more and so they can be garbage collected. That's all very well if the design of B says that it just needs the A objects at the beginning and doesn't need them later. But if that isn't part of the design, then your drive to save memory by dereferencing the A objects might be damaging the B objects.
And if your B objects are actually designed so that they hang on to A references after the point where they need them, then there must be a point in the processing where the design says they are not needed any more, and something could kick in which dereferences them. But I can't see that being the case for all 150 classes in your application.