The ultimate line of defense is the user login itself. The Tomcat manager and admin webapps both use the
J2EE container-managed security system, which unlike the tissue-paper security that's the overwhelming norm for do-it-yourself login systems, was designed by security specialists and has never, to my knowledge, been defeated in Tomcat or any other webapp server. As long as best practices for formulation of login IDs and passwords is adhered to, it's as close to impervious as any software system can be.
If you want to limit access to the manager apps themselves to specific source IP ranges (your intranet, for example), you can add a Valve to their Context definitions or to the Host definition (although if you do it there, ALL webapps for that host will be restricted.
There are other safeguards you can take. Commonly outside access to Tomcat webapps is mediated by some sort of reverse proxy such as Apache HTTP, Nginx, IIS, or the like. It allows having mixed-language webapps on a common server and keeps Tomcat from having to run as a privileged user in order to listen on ports 80 and 443. In that case you can simply define your proxy to not route any external requests addressed to the Manager and Admin webapps and Tomcat will never have to deal with them.
Of, of course, you can simply delete these webapps from your Tomcat server altogether. They're just webapps, and although they do provide a convenient web-based management console, Tomcat can run just fine without them.