The default for anonymous users is not to display any "registered" user links (private messages, my profile, and the like). This is generally done in the templates using the "logged" context variable. E.g., <#if logged>... registered stuff...</#if>
The logged variable is set via the SessionFacade class's makeLogged and makeUnlogged. These can called based on a wide variety of conditions. But many of the "is this an anonymous user" decisions come via the anonymous.userId config parameter.
The anonymous.userId parameter is the DB Id number that matches your anonymous user's ID. Normally this is created as the first user and is 1. However, if due to your SSO setup you are using a different id for the anonymous user, you will need to set this to match your situation.
One thing to beware of is that I believe there are still a few places in the template files that use a hard coded "user.id == 1" condition. If you find one of these, put a Jira bug entry on it so it will get fixed.
If this is not the behavior you want, you should now know enough to modify the templates to make it do what you want.