In short, i am not able to send emails to a particular group through java mail.
A "group" in Notes (if I remember right) is not an address as such, rather a type of document, yes? So a group does not represent a recognizable address as far as SMTP is concerned - there is some conversion going on on the server (as Rovas Kram points out).
You can't use a CVS list of addresses, because Java Mail's InternetAddress class will parse the CSV to make sure it is RFC822 complient. But look at the Transport class's send methods - you'll notice that two take an array of InternetAddresss. Presuming you know the addresses in the Lotus Group, you can send to all of them this way.
I have a
vague memory that there is an IBM supplied transport implementation for Lotus which ships with Domino. You might like to search IBM's site for such a thing - if there is it might let you use groups directly.