No doubt the first step would be to study the existing allocator and collectors, and come to understand, at a minimum, how they interface to the rest of the JVM. There is no published interface for this. Sun's JVM is certainly not designed for other folks to write plug-in garbage collectors, so you're not going to find much -- if any -- documentation on doing so.
Of course, you'll need to be a pretty darn good C/C++ programmer; you can't write a garbage collector in Java!
What gave you this idea? Do you have a new algorithm you want to try out?
I just don't know where to start.. some issues like "where do I create a C++ file to start programming", "what libs can I use", etc..
OK, then I'd suggest one more step before my "first step" above: compile and deploy the source you've downloaded. TO do that, of course, you'll have to set up a build environment that works. Then you'd be able to add more source files in the same places as the existing ones, and (prudently) use the libraries the other code uses. So still, what I'm saying is, start working with the existing implementation, and learn about it. That's really the only way to get into this.
Having seen and played with the C++ source code... I have to say that it is not for the faint of heart. It's really complex -- and it will be unlikely that you will be able to get much help (if any), at the ranch, to help you through it.