Can AJAX be used to dynamically update any scoped variable storing a list of objects

[Rama Krishna]

I have never used AJAX before. I really require to display huge data in an html table (Struts - display tag API) where the data is stored in a session scoped variable (rather any variable) containing a list of objects obtained as a request-response. However, currently our slow database queries and the time taken for the data to be displayed to the user makes the whole thing appear very very slow.

We currently only show a couple of thousand records each time rather than showing all the records at once and fetch the next ones on newer requests. Can AJAX be used to display a couple of 1000 records initially and then dynamically append/update data by calling methods that fetch and update the existing table data with new data until the whole data is obtained??

Can AJAX allow us to fetch any kind of data and format to possibly update a variable/parameter such as in our case containing the display table data without requiring to refresh the page?

Something like

<script type="text/javascript">
 
var xmlHttp=null;
 
function showHint(str)
{
if (str.length==0)
  { 
  document.getElementById("txtHint").innerHTML="";
  return;
  }
try
  {// Firefox, Opera 8.0+, Safari, IE7
  xmlHttp=new XMLHttpRequest();
  }
catch(e)
  {// Old IE
  try
    {
    xmlHttp=new ActiveXObject("Microsoft.XMLHTTP");
    }
  catch(e)
    {
    alert ("Your browser does not support XMLHTTP!");
    return;  
    }
  }
var url="getNext1000Records.do?" + fetchNext1000Records;
xmlHttp.open("GET",url,false);
xmlHttp.send(someFlagField);
 
// IS IT POSSIBLE TO DO USE A JSP SCRIPTLET TO UPDATE THE PARAMETER CONTAINING THE TABLE DATA WITHOUT REFRESHING THIS TABLE
 
document.forms[0].VariableStoringTableData = VariableStoringTableData +  xmlHttp.responseText;
// 
 
}
</script>

Any suggestions or comments to this AJAX newbie are really appreciated!!

-Thanks
Rama

[ May 19, 2008: Message edited by: Rama Krishna ]
[ May 19, 2008: Message edited by: Rama Krishna ]

[Rama Krishna]

Could somebody move this to HTML and JavaScript as well.

I have accidentally created a post here in the wrong location because of the previous post which is also related to AJAX.

[Bryce Martin]

In a quick answer, yes. It can be done. One word of caution. When displaying 1000's of records in an HTML table the render time will be terrible. Also, the way to accomplish what you want to do involves rewriting parts of the DOM with javascript. This is perfectly ok to do, but be forewarned that when doing this in DOMs with 1000's of elements you will see performance issues. I currently use JQuery and its jqGrid plugin to handle displaying table data on a webpage with large result sets. This allows for paging to specific pages as well.

[Rama Krishna]

Impressive!

With my knowledge of java script programming, I felt surprised by how much and how well you can develop things using AJAX/JavaScript. Unfortunately we have already developed most of our data tables using (Struts/Hibernate) Display Tag library. It's a pretty clean solution with options to export, sort, group and what not!

Can jqGrid work for a struts/jsp html parameter (list of objects) and display particular fields provided the field names? Finally any idea if we can sort, group, subtotal, filter and export the table data? If so, I can spend a night or two and convince my Hibernate/Struts based team to switch to using this!

-Cheers
Rama

PS: I am seeing a work around to see if I can use another JSP rather hidden and whose javascript fetches more records at regular intervals and appends to the base table <td> records without refreshing the page at all. Hopefully this should work!

[Bryce Martin]

I had toss around the idea of doing something like you are trying with your alternative jsp work around. Load the initial page and then make ajax calls at a certain interval to load more table rows. This is possible.

If you can feed the Hibernate table data as xml or json then jqgrid can digest it. You define the table column header names and then define the fields by giving them names, lengths, alignment properties, and others. Then you just name the xml tag or json key the name of the corresponding column. Its quite slick and has added a lot of new features recently. I'm not sure about the ability to total rows and the likes. I'm sure you could do it and just summarize it outside of the table itself....

Have a look here...

http://trirand.com/jqgrid/jqgrid.html#

I've done some pretty complex stuff with this. I've done ajax requests after loading the table and changed cell backgrounds and all kinds of stuff. It has so much flexibility.

Like I said, if you can feed the Hibernate/Struts table row data in an XML or JSON(my personal favorite) feed then it can work for you.

I hope this helps
Bryce

[Rama Krishna]

Successful. Although tedious, I was able to somehow have the hidden iframe ping the server continuously and get every couple of thousand records (request/response type) and append this data to the parent table tbody innerHTMl, until I get all the records. The only problem here is the page heaviness (say more than 50,000 records) as you said so. We need to display the complete data for a couple of important reasons (comment if these are wrong requirements or if there are work arounds); be able to use our own javascript based filtering on complete data and use display tag API based grouping, sorting, more importantly use available display tag API to export COMPLETE data into various cvs, xls, rtf and pdf formats with custom header and footers! Our display tag API, the current display table allows grouping, sub-totaling, pagination (not dynamic), except that it does not allow searching the whole content just like AJAX in jqGrid allows you to do! We really liked the cool feature of filtering, selecting data from the whole data (just like the Advanced/Search Big Sets option in jqGrid Demo) rather than the displayed data! I do not see why I may not be able to do the samething with iframe as well. Other pros in jqGrid include the cool UI with VB or excel based static header and scrollable bars, something I was not able to achieve for display table.

Anyways, I think converting a hibernate object to an XML document should not be hard at all right? I have not done it quite a while ago while learning XML parsers and marshalling. And why doesn't AJAX or rather jqGrid support java based list or array data? One thing I also note is that, equivalent XML data is even heavier and takes more time to transport from the server to the client than the object, isn't it?

There is also Hibernate support for XML documents.

http://www.hibernate.org/hib_docs/reference/en/html/xml.html

Ill have to quickly learn AJAX by this weekend and test one of the pages to see if everything goes well and hopefully start using jqGrid along with JQuery. By the way, JQuery as I see has a totally different programming syntax and is a lot different from Java Script.

I do not have much experience in suggesting a major change. Do you think the transition to AJAX based jqGrid and using JQuery (learning involved as well as JQuery syntax seems to be a little different from javascript) be more useful. I am more interested in its user friendliness, free (is it open source and can it be used in our company application which will be sold?), free example css style sheets, jsp/servlet based examples rather than php based, user groups for free support?

-Cheers
Rama

[ May 21, 2008: Message edited by: Rama Krishna ]
[ May 21, 2008: Message edited by: Rama Krishna ]

[Bear Bibeault]

JQuery as I see has a totally different programming syntax and is a lot different from Java Script.

No, jQuery is JavaScript. What makes you think it's not JavaScript?

If the fact that a $ is used as a function name confuses you, just use the jQuery alias instead.

jQuery is just a JavaScript library like any other. It will help you write concise and powerful JavaScript statements, and enable you to do a lot more functionality with a lot less code.
[ May 21, 2008: Message edited by: Bear Bibeault ]

[Rama Krishna]

Wow, a response faster than I could finish my Heineken 25oz beer!

I may be am wrong and so I take back that statement. I had a quick look at JQuery and felt it has its own syntax and language.

I am eager to get answers for my following previous post!

Originally posted by Rama Krishna:
Successful. Although tedious, I was able to somehow have the hidden iframe ping the server continuously and get every couple of thousand records (request/response type) and append this data to the parent table tbody innerHTMl, until I get all the records. The only problem here is the page heaviness (say more than 50,000 records) as you said so. We need to display the complete data for a couple of important reasons (comment if these are wrong requirements or if there are work arounds); be able to use our own javascript based filtering on complete data and use display tag API based grouping, sorting, more importantly use available display tag API to export COMPLETE data into various cvs, xls, rtf and pdf formats with custom header and footers! Our display tag API, the current display table allows grouping, sub-totaling, pagination (not dynamic), except that it does not allow searching the whole content just like AJAX in jqGrid allows you to do! We really liked the cool feature of filtering, selecting data from the whole data (just like the Advanced/Search Big Sets option in jqGrid Demo) rather than the displayed data! I do not see why I may not be able to do the samething with iframe as well. Other pros in jqGrid include the cool UI with VB or excel based static header and scrollable bars, something I was not able to achieve for display table.

Anyways, I think converting a hibernate object to an XML document should not be hard at all right? I have not done it quite a while ago while learning XML parsers and marshalling. And why doesn't AJAX or rather jqGrid support java based list or array data? One thing I also note is that, equivalent XML data is even heavier and takes more time to transport from the server to the client than the object, isn't it?

There is also Hibernate support for XML documents.

http://www.hibernate.org/hib_docs/reference/en/html/xml.html

Ill have to quickly learn AJAX by this weekend and test one of the pages to see if everything goes well and hopefully start using jqGrid along with JQuery. By the way, JQuery as I see has a totally different programming syntax and is a lot different from Java Script.

I do not have much experience in suggesting a major change. Do you think the transition to AJAX based jqGrid and using JQuery (learning involved as well as JQuery syntax seems to be a little different from javascript) be more useful. I am more interested in its user friendliness, free (is it open source and can it be used in our company application which will be sold?), free example css style sheets, jsp/servlet based examples rather than php based, user groups for free support?

-Cheers
Rama

[ May 21, 2008: Message edited by: Rama Krishna ]

[ May 21, 2008: Message edited by: Rama Krishna ]

[Bear Bibeault]

Originally posted by Rama Krishna:
I may be am wrong and so I take back that statement. I had a quick look at JQuery and felt it has its own syntax and language.

Nope, just JavaScript. Again, the use of the $ (which Prototype also uses) might be a bit disconcerting at first. The selector syntax (which is key to using jQuery effectively) is based on CSS, so that's not new syntax either.

If you're not used to using JavaScript as a functional language (and many people are not), some of the more advanced statements and constructs can seem odd. But functional JavaScript is a skill that all page authors should know whether they are using jQuery or not.

[Bryce Martin]

I totally agree with Bear on this. I came into this whole page authoring bit just about 8 or 9 months ago. Learning javascript and the jquery library have been invaluable. A couple of answers to your questions. JQuery is completely open source. Use it, abuse it, build your own plug ins for it. JQgrid is just another library that supports JQuery's style of javascript coding (namely its selector functionality). The reason why jqGrid does not handle java objects is because its not java. Its javascript. Javascript cannot work directly with java objects. They need to be JSONized and pushed back to the browser for digestion. jqGrid works so smoothly because it does the heavy lifting on the server side and just sits int he browser to digest and display the data you send back to it. For example, if you want to change the sort order from ascending to descending jqGrid just sends a request to the server telling it to give it data in the new order. You have to have the code on the server side to do the sort (usually just changing a keyword in sql from (asc to desc) and giving back the requested number of rows.

You need to remember that the browser is for DISPLAY. All the heavy work needs to be done on the server. Converting of data to various formats such as cvs, xls, etc... need to be done on the server. The browser is just used to show the user results.

I hope this helped to clear some things up for you.

Bryce

[Gregg Bolinger]

If getting values from Java to jqGrid seems difficult you might want to take a look at this. Should ease the pain a bit.

[Justin Fox]

all you have to do is build the html on the server side, and echo or response.write or whatever to the page so the xmlHttpRequest can just set the .innerHTML of the div to the response.

Justin Fox

[Rama Krishna]

Appreciate all of your suggestions!

We are trying to convince ourselves to continue to use the display tag library with all of its advantages I mentioned in my earlier post. I also found that I can convert a java object very easily to JSON, not sure if I can convert my huge (as much as 100,000 row and 20 column) list of objects to JSON very quickly as well.

Gregg Bolinger's idea
http://www.greggbolinger.com/blog/2008/04/14/1208228820000.html
seems to be of help here

Although, I was able to use iframe to pump in the huge data into the parent table data without the page refresh, the page being heavier was very much visible so we decided that we will run a query for a couple of 1000 records and populate the remaining data to create the next 1000 record buckets on the server side only, rather than waiting for the user to click the next 1000 records and then running the query. Display tag table has problems with creating a static header and scrollable bars (normal div creates cross browser incompatibility in both IE and FF) which mimics an excel table.

Keeping this aside, I am now at a completely new problem:

As for the sorting I am referring to a SQL order by kind of sorting. Most of the data that we display in our display tables is dynamic (selection from a bunch of pre-defined fields) along with a couple of calculated fields. I have no clue how we can group, perform sorting of the type of SQL order by query and filter the final data which is a result of performing some calculations on the database resultset and includes calculated fields??

Our display tag does a decent job in grouping and subtotaling but somehow it fails in creating the same pdf out of the displayed data. I wonder if I can do the order, sort and filter in the displayed html data and also be able to create a pdf out of this?

Bryce here is the problem:

You said: You have to have the code on the server side to do the sort (usually just changing a keyword in sql from (asc to desc) and giving back the requested number of rows.

Almost all of our table data takes time to populate due to database query and processing the records for display. So I was looking for alternative solutions as discussed above in our particular case!

[ May 22, 2008: Message edited by: Rama Krishna ]

[ May 22, 2008: Message edited by: Rama Krishna ]

[ May 22, 2008: Message edited by: Rama Krishna ]

[ May 22, 2008: Message edited by: Rama Krishna ]
[ May 22, 2008: Message edited by: Rama Krishna ]

[Rama Krishna]

Justin Fox,

I am not sure how I may be able to use your idea! The display tag takes a session scoped variable, containing a list of objects and converts into its own HTML table, custom style, formatting, buttons, links... i.e., we do not have the html table to be created on the server side and so not an easy task to be able to pump any kind of table data into the innerHTML of the div.

Wonder what your quote
"You down with OOP? Yeah you know me!"
means

-Cheers
Rama

Originally posted by Justin Fox:
all you have to do is build the html on the server side, and echo or response.write or whatever to the page so the xmlHttpRequest can just set the .innerHTML of the div to the response.

Justin Fox

[Bryce Martin]

Hey Rama,

SQL has the ability to do calculations and add columns to the resultset as it processes. Take a look at SQL column functions. Things like Count() and Sum() and Avg(). These might help you to summarize your data. If you know how you want to group your data you can use GROUP BY as well. There are tons of things that can be done with SQL that will get you extremely close to what you want if not all the way there.

Thanks
Bryce