Problem with encoding and polish characters

[Karol Matysiak]

Hello.

So:
I'm using windows 7 (sadly),
I've got MySQL server with database encoded to latin2
I've got glassfish3 server with hibernate (3.2.5 ? or something like that, I can't check it exactly right now)
I'm using JSP/Servlets to create webapp

When I select data in my mysql console, everything works great, I have a polish characters, but when I want to retrive data from the db and put it into the JSP page, data displays without polish characters. I have set the encoding to latin2everywhere I could (jsp pages, hibernate.cfg - also in the hibernate.connerction.url property, filter), but without any success.

I had a problem with displaying the polish characters in the page title, but when I have added a filter (found on the net) everything was fine, but the filter changed encoding to UTF-8. So I have also changed it to latin2 encoding.

When I set utf-8 in my mysql database I don't get polish characters when I'm selecting data, so I can't use utf-8 ;/

Do I need to change encoding somewhere else? In glassfish server or jdbc driver?

[jatan bhavsar]

Hi Karol,

Right direction is to work with the filter and UTf-8.
Try to work this way and let me know what type of error are you getting i might be able to help you.
Also have you set the UTf-8 encoding type for the glassfish server like <parameter-encoding default-charset="UTF-8"/>.

It is require in tomcat to change the default encoding to UTF-8 with encodeuri when you are working for i18n so check the same for the glassfish .

Regards
jatan

[Vishal Shaw]

Karol Matysiak wrote: When I set utf-8 in my mysql database I don't get polish characters when I'm selecting data, so I can't use utf-8

This is wrong, as UTF-8 is the most accepted encoding, until unless you want to use UTF-16 :)

As for the character encoding filter, is it the first filter (in the filter chain) in your xml? Otherwise it will not work as intended. Also did you tried setting the connection encoding to the required encoding, like

<property name="hibernate.connection.useUnicode">true</property> 
<property name="hibernate.connection.characterEncoding">UTF-8</property> 

I did it for UTF-8, you can use your encoding , if you desire.
Also is your page encoding using the required one, or is it using the default one?

Try these and let us know.

[Karol Matysiak]

jatan bhavsar wrote:Hi Karol,

Try to work this way and let me know what type of error are you getting i might be able to help you.
Also have you set the UTf-8 encoding type for the glassfish server like <parameter-encoding default-charset="UTF-8"/>.

I don't get error. Data is displayed, but I've got something like Imię: Andrzej Nazwisko: Krokosiäski tel.: [phone], instead of Imię: Andrzej Nazwisko: Krokosiński tel.: [phone] . Word "imię" got polish character, but that word is not from the database.
I just notice, that I didn't changed the filter. After change (it's now encoding to latin2, same as mysql server) the data which was in the database is displaying as earlier - Krokosiäski, but when I will add somethign through the form on index page, everything is fine (it displays Krokosiński ).

This is wrong, as UTF-8 is the most accepted encoding, until unless you want to use UTF-16 :)

I know, but I can't set up my db server to utf-8 :/ I'm trying this since few days.

As for the character encoding filter, is it the first filter (in the filter chain) in your xml? Otherwise it will not work as intended. Also did you tried setting the connection encoding to the required encoding, like

<property name="hibernate.connection.useUnicode">true</property> 
<property name="hibernate.connection.characterEncoding">UTF-8</property> 

I did it for UTF-8, you can use your encoding , if you desire.
Also is your page encoding using the required one, or is it using the default one?

Try these and let us know.

I have that in my hibernate.cfg.xml - but it didn't solved my problem. Of course in each page I have set up encoding to latin2.
And about my filters, I have only that one filter, so it must be first and the last filter in the chain :)
Here is my hibernate.cfg.xml

<?xml version="1.0" encoding="latin2"?>
<!DOCTYPE hibernate-configuration PUBLIC "-//Hibernate/Hibernate Configuration DTD 3.0//EN" "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
  <session-factory>
    <property name="hibernate.dialect">org.hibernate.dialect.MySQLDialect</property>
    <property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
    <property name="hibernate.connection.url">jdbc:mysql://localhost:3306/WebAppDB?autoReconnect=true&useUnicode=true&characterEncoding=latin2</property>
    <property name="hibernate.connection.username">root</property>
    <property name="show_sql">true</property>
    <property name="hibernate.hbm2ddl.auto">update</property>  
    <property name="hibernate.connection.useUnicode">true</property>
    <property name="hibernate.connection.characterEncoding">latin2</property>
    <mapping resource="webHiber/Czlowiek.hbm.xml"/>
  </session-factory>
</hibernate-configuration>

hibernate reveng.xml

<?xml version="1.0" encoding="latin2"?>
<!DOCTYPE hibernate-reverse-engineering PUBLIC "-//Hibernate/Hibernate Reverse Engineering DTD 3.0//EN" "http://hibernate.sourceforge.net/hibernate-reverse-engineering-3.0.dtd">
<hibernate-reverse-engineering>
  <schema-selection match-catalog="WebAppDB"/>
  <table-filter match-name="czlowiek"/>
</hibernate-reverse-engineering>

JSP with the form

<%@page contentType="text/html"  pageEncoding="latin2"%>
<!DOCTYPE html>
<html>
    <jsp:include page="template/Header.jsp">
        <jsp:param name="pageName" value="Strona Główna" />
    </jsp:include>
    <body>

        <h1>Hello World!</h1>
        <form action="HibernateTest" method="POST">
           Podaj swoje imię: <input type="text" name="imie" /><br />
           Podaj swoje nazwisko: <input type="text" name="nazwisko"/>
           Podaj numer telefonu: <input type="text" name="nrTel" />
           <input type="submit" value="wyślij" />
        </form>
    </body>
</html>

JSP with the data from db

<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %>
<%@page contentType="text/html" pageEncoding="latin2"%>
<!DOCTYPE html>
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=latin2">
        <title>JSP Page</title>
    </head>
    <body>
        <h1>Lista ludzi z bazy danych</h1>
    <c:forEach var="czlowiek" items="${ludzie}">  
            Imię: ${czlowiek.imie} Nazwisko: ${czlowiek.nazwisko} tel.: ${czlowiek.nrTel} 
      
        </c:forEach>  
   
    </body>
</html>

servlet

package webHiber;

import org.hibernate.Session;  
import org.hibernate.SessionFactory;  
import org.hibernate.Transaction;  
import org.hibernate.cfg.Configuration;  
import java.io.IOException;  
import java.io.PrintWriter;  
import java.util.List;
import org.hibernate.Query;
import org.hibernate.cfg.AnnotationConfiguration;
  
public class HibernateTest extends javax.servlet.http.HttpServlet {  
    protected void doPost(javax.servlet.http.HttpServletRequest request, javax.servlet.http.HttpServletResponse response) throws javax.servlet.ServletException, IOException {  
  doGet(request,response);
    }  
  
    protected void doGet(javax.servlet.http.HttpServletRequest request, javax.servlet.http.HttpServletResponse response) throws javax.servlet.ServletException, IOException {  
        response.setContentType("text/html");  
        PrintWriter out = response.getWriter();  
        String imie = request.getParameter("imie");  
        String nazwisko = request.getParameter("nazwisko"); 
        String nrTel = request.getParameter("nrTel");
        Session session = null;  
        SessionFactory sessionFactory = new Configuration().configure().buildSessionFactory();  
        session = sessionFactory.openSession();  
        Transaction transaction = session.beginTransaction();  
          
        Czlowiek czlowiek = new Czlowiek();  
        czlowiek.setImie(imie);  
        czlowiek.setNazwisko(nazwisko);  
        czlowiek.setNrTel(nrTel);
        session.save(czlowiek);  
        Query queryResult = session.createQuery("from Czlowiek");  
        List<Czlowiek> humanList;  
        humanList = queryResult.list(); 
        
        transaction.commit();
        session.flush();
        System.out.println("Database contents delivered..."); 
        session.close();   
        
       request.setAttribute("ludzie", humanList);
            request.getRequestDispatcher("/listaLudzi.jsp")
                .forward(request, response);
          
        
         
    }  
}  

I don't post the java mapping file because it is a bunch of setters and getters.>

[Vishal Shaw]

Karol Matysiak wrote: I know, but I can't set up my db server to utf-8

Did you mean, you are trying but can't set it, or you have been forbidden to set it?

Karol Matysiak wrote: After change (it's now encoding to latin2, same as mysql server) the data which was in the database is displaying as earlier - Krokosiäski, but when I will add somethign through the form on index page, everything is fine (it displays Krokosiński )

I think your filter is encoding only request parameters not the response object. So, you can try setting it on response objects too, alternatively try adding

 
response.setCharacterEncoding( "*******" );
    

in your HibernateTest.java

[Karol Matysiak]

Vishal Shaw wrote:
Did you mean, you are trying but can't set it, or you have been forbidden to set it?

Yes, I tried, it's not any buisness project so I can set up what only I want, but I have set up everything to utf-8 in my database and polish characters wasn't displayed well.

Vishal Shaw wrote:

I think your filter is encoding only request parameters not the response object. So, you can try setting it on response objects too, alternatively try adding

 
response.setCharacterEncoding( "*******" );
    

in your HibernateTest.java

I will try to add this when I will be at home.

[Karol Matysiak]

Nope, it still doesn't work. Only records which I add from the mysql server console display without polish characters.

[Bear Bibeault]

"rok"?

[Paul Clapham]

Bear Bibeault wrote:"rok"?

The Polish word for "year". (Hope that helps!)

[Karol Matysiak]

My mistake, it should be "work". Now it's ok;p

[Paul Clapham]

Karol Matysiak wrote:I'm using windows 7 (sadly)

So am I.

I've got MySQL server with database encoded to latin2

My instance of MySQL is configured to use UTF-8. This handles letters specific to Polish (and Slovak for that matter) correctly.

When I set utf-8 in my mysql database I don't get polish characters when I'm selecting data, so I can't use utf-8 ;/

You can't just change the configuration and expect it to work correctly. All that does is to assert that data which was already encoded in latin2 is now encoded in UTF-8.
Which of course it isn't. You have to export all the data from the database to a text file, then change the configuration, then import all the data back from that text file.

[Karol Matysiak]

Paul Clapham wrote:
You can't just change the configuration and expect it to work correctly. All that does is to assert that data which was already encoded in latin2 is now encoded in UTF-8.
Which of course it isn't. You have to export all the data from the database to a text file, then change the configuration, then import all the data back from that text file.

Of course your're right, but I have even created new database to check it, without any success. In best configuration I had something like that on my server:
query:

 SHOW VARIABLES LIKE "coll%";

result :

+----------------------+-----------------+
| Variable_name | Value |
+----------------------+-----------------+
| collation_connection | utf8_general_ci |
| collation_database | utf8_polish_ci |
| collation_server | utf8_polish_ci |
+----------------------+-----------------+

Also it was set everywhere utf8_general_ci, and still no polish characters
And result for query

SHOW VARIABLES LIKE "char%";

Result was:

+--------------------------+----------------------------------------------------
-----+
| Variable_name | Value
|
+--------------------------+----------------------------------------------------
-----+
| character_set_client | utf8
|
| character_set_connection | utf8
|
| character_set_database | utf8
|
| character_set_filesystem | BINARY
|
| character_set_results | utf8
|
| character_set_server | utf8
|
| character_set_system | utf8
|
| character_sets_dir | C:\Program Files\MySQL\MySQL Server 5.5\share\chars
ets\ |
+--------------------------+----------------------------------------------------

And I have also created db in InnoDB engine, with setchar=utf8 attributes, I created even tables with encoding set to utf8 - and still it didn't worked. I was searching solution for few days, and the only thing i found was for PHP developing to send mysql query "SET NAMES 'utf-8' ". Here's my.ini file:

[WinMySQLAdmin]
Server="C:/Program Files/MySQL/MySQL Server 5.5/bin/mysqld.exe"

[client]
default-character-set=utf8

[mysql]
default-character-set=utf8

[mysqladmin]
default-character-set=utf8

[mysqlcheck]
default-character-set=utf8
[mysqldump]
default-character-set=utf8

[mysqlimport]
default-character-set=utf8

[mysqlshow]
default-character-set=utf8
[mysqld]
character-set-server=utf8
collation-server=utf8_polish_ci
init_connect='SET collation_connection = utf8_polish_ci'
init_connect = 'SET NAMES utf8'

[Paul Clapham]

I think you're going to have to tell us what "didn't work" means.

If you're using the Windows console to test with, that quite likely isn't going to show you non-ASCII characters correctly unless you have your Windows code page set correctly. But instead of messing about with that, you could use a small Swing application to check your database configuration. That's the context I was describing, which works perfectly well with a MySQL database configured to use UTF-8.

In any case you shouldn't use a web application to test that, because now you're testing the web app and the database connection at the same time, so you don't know where to look when you don't get the correct result.

[Karol Matysiak]

Yes, I'm using windows console.

Ok but if I set encoding to latin2 - in windows console I have all characters which I want, when I will change it to utf-8, those characters are missing.

Anyway, like you said, I wrote simple app that is connecting to the same server as my JSP, and still the polish characters are not displayed :|

[Karol Matysiak]

Sorry for double post, but I have to update few things. So:

I have switched to linux, now I have mysql server 5.1, jboss 7.1.1, eclipse ide.

MySQL encoding is now set to utf-8 and I have polish characters in linux command line, but of course i don't have those characters on my web page, and not only in data retrived from database, but also on template text in jsp, also in title of jsp. Everything is the same but it's of course doesn't work...

Any ideas what can be wrong now?