Do we have to create Topic/Queue explicitly in application server console ? wondering if it is specified in ejb.xml or something like that, should it be enough ?
The ejb.xml (or annotation) provides a reference/pointer to the queue. You still need to create an actual queue. This could be done through the admin console of your application server or via the command line.