A certificate is just part of the story. A certificate contains 3 basic pieces of information -
1) the public key of the (public key,private key) pair of the certificate owner (the private key is kept very very private and not released to anyone - ever),
2) information about the owner of the key pair (name, address etc),
and 3) a signature by someone to say that the owner is who he says he is.
When a server uses a Verisign signed certificate then Verisign say that they have checked upon the owner of the certificate and he/she is who they say they are. This does not mean they are trustworthy, just that they are who they say they are. If Al Capone were alive today then Verisign could issue a certificate to Al once they checked he was the real Al Capone but would you trust him? The trust part is your responsibility and not Verisign's.
During the SSl handshake the server sends it's certificate and information signed using the private key associated with the certificate. The certificate signature is checked against the Verisign certificate stored in the SSL truststore. If this proves to be valid then the signature of the signed information is checked using the certificate and if this is valid then you have proved that the site owner is who he says he is. The truststore contains ALL the certificates of certificate authorities such as Verisign that are trusted.
Now, by 'custom certificate' you seem to mean a self signed certificate which means that the owner of the certificate is certifying that he is who he says he is. I could easily create a self signed certificate supposedly owned by Microsoft but since there is no certificate authority (CA) such as Verisign certifying that I am Microsoft then if I present the certificate via a server then there is no certificate in the truststore to verify the signature so it will be reject during the SSL handshake.
The whole story is a little more complicated than that since the certificate may be a chain of certificates i.e. Verisign vouches for A who vouches for B ... who vouches for the certificate owned by the server. During the handshake the whole chain is checked with the ultimate check being against a certificate in the truststore. Of course it is possible to put the server's self-signed certificate in the client's truststore but this is not realistic in real world production environments as there is no proof of ownership. OK for
testing though.
It is not very usual but one can also have client side authentication whereby the client has a certified certificate and presents that as part of the SSL handshake. The server will then reject the client if the client's certificate chain does not have one of the certificates in it's truststore at the end of the chain.
How you get Verisign to issue a certificate is another story.
1
