Your browser will try to treat the output like an XLS file and use Excel to render it.
If Excel can't understand what you're sending, it will not render it correctly and/or give errors.
It's up to you to format the output correctly. CSV format works for simple data, SLK for more complex stuff.
Or you may want to look at packages like POI to generate binary Excel data and try to send that.
IMO you'd better not use a
JSP to generate Excel data, a
servlet (possibly using an XSL transformation or Jython script) is far better suited.