There are a couple ways to look at this.
One is to say the Swing client has the complete MVC inside it, and remote communications are something the model does when it has to. That makes remote communications unrelated to the client MVC. Oh, I almost passed up a chance to say "orthogonal".
The server can have another complete MVC model inside as well, if you like.
It's more complex to imagine remote communications between MVC components. You could try to just ignore the remote aspect and have the components talk to each other as usual. For server to talk to client asynchronously (say as a result of a server event, not a client request) it would have to have a socket or some connection to push data. Unfortunately, ignoring the remote aspect leads us to design overly busy network communications, so we probably don't want to go there.
I'd love to make up a third answer that perfectly balances all your concerns, but I have to go watch a movie with the wife. And I don't know one.
Did those thoughts mesh with what you're trying to do?