I usually see the opposite of this question asked:
http://localhost and
http://127.0.0.1 work but
http://hostname and
http://99.99.99.99 don't.
The key is in how the app server opens the port. If it opens port using IP address 0.0.0.0, then the port is bound to all IP addresses known to the system. Thus the port is bound to 127.0.0.1 and all IP addresses assigned to all of the NICs. On the other hand, if the app server opens the port using a specific IP address, such as 99.99.99.99, then you can access the port only via that IP address - in other words, 127.0.0.1 cannot be used.
You can use netstat (or tcpview on Windows) to see how the app server opened the port.
However, I do not understand why remote systems can access
http://10.3.20.456 but when on that system (the one with that IP address) that URL is not valid. But then again there could be something screwy in the network. I would first try a ping and then a tracert to try to solve the problem.