• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

JSON Restrictions?

 
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Is there a limit to the size of data to be returned as a JSON response by an HTTP server;

Suppose that returns the records of a DB, which are thousands...
 
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Assuming the server can write the JSON as a HTTPResponse output stream one item at a time - rather than building the whole thing in a buffer and then sending it, any limit will be on the client side, not the server side.

If you are thinking about a browser client that a human being has to interact with, returning more than a few screens worth of data is a very bad design.

Bill


 
John Astralidis
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

William Brogden wrote:Assuming the server can write the JSON as a HTTPResponse output stream one item at a time - rather than building the whole thing in a buffer and then sending it, any limit will be on the client side, not the server side.

If you are thinking about a browser client that a human being has to interact with, returning more than a few screens worth of data is a very bad design.

Bill




The client will be a java-swing desktop application that presents in a JTable all the DB records. So, what I'm thinking is the client having a method to request the data (eg getInventoryList()) and the server replying with a json stream containing all that data from DB.

If this is bad design what should be the correct ones?

Any advice would be highly appreciated.
 
Java Cowboy
Posts: 16084
88
Android Scala IntelliJ IDE Spring Java
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
It's not useful for a user of any program to be presented with a screen containing a list with many thousands of items. People are not going to scroll through a list containing thousands of items and find what they need.

Design your program in such a way that you don't need to display thousands of records in your GUI at once. For example, by implementing paging - display at most a few tens of records at once, with buttons to get the previous or next page of results. Provide search fields so that the user can find the relevant data quickly, instead of having to look through thousands of records.

Doing queries that return huge amounts of data will also make your application slow, because it takes time to transmit a lot of data, and it takes a lot of memory to keep all the data in memory.
 
William Brogden
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

If this is bad design what should be the correct ones?



What Jesper said!

If this was my problem I would be talking to potential users, storyboarding various ways they might interact with the database. Nothing worse than working for days and having a user say "thats not what I wanted at all."

I would say: What kind of output do you want? What sort of selection capability? Do you need statistical summaries? Graphical summaries?

If they really are convinced that they want the whole DB on the desktop application maybe you should build them a local copy - maybe an in-memory DB?

Can the server return any other form besides JSON?

Bill
 
John Astralidis
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

William Brogden wrote:...
Can the server return any other form besides JSON?
Bill



This is all about a client application that I have already implemented but it's current architecture was not intended to run on multiple devices/platforms. So, I'm just an enthusiast junior developer who wants to practice and expand his knowledge.

In this project, I'm thinking to make use of RESTful web services and JSON format data transmission between server and clients.

This is my project
Web service for data synchronization?
Updating tables by simultaneous users
 
Sheriff
Posts: 67746
173
Mac Mac OS X IntelliJ IDE jQuery TypeScript Java iOS
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I don't see how the choice of JSON is any way relevant to the issue; and is, in fact, the interchange format currently in vogue.

The key to large datasets is paging and filtering. As said by many before me: blatting out a ton of data to the user is useless.
 
John Astralidis
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Bear Bibeault wrote:I don't see how the choice of JSON is any way relevant to the issue; and is, in fact, the interchange format currently in vogue.

The key to large datasets is paging and filtering. As said by many before me: blatting out a ton of data to the user is useless.



How would you suggest to implement the data exchange between server and clients, if JSON is not an option?
 
Bear Bibeault
Sheriff
Posts: 67746
173
Mac Mac OS X IntelliJ IDE jQuery TypeScript Java iOS
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I wouldn't suggest not using JSON. Prior to JSON, XML was the flavor of the day. Proprietary formats have their place, but not recommended without really good reasons.
 
John Astralidis
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Bear Bibeault wrote:I wouldn't suggest not using JSON. Prior to JSON, XML was the flavor of the day. Proprietary formats have their place, but not recommended without really good reasons.



To sum up, I should use pagination/filtering and JSON/XML/etc (I find JSON much simpler though) for data exchange...
 
Bear Bibeault
Sheriff
Posts: 67746
173
Mac Mac OS X IntelliJ IDE jQuery TypeScript Java iOS
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I would not recommend XML unless you have a really good reason.

But yes, allow the user to filter the data to what they want to see; then page the results so that they aren't inundated with a deluge of data.
 
John Astralidis
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Bear Bibeault wrote:I would not recommend XML unless you have a really good reason.

But yes, allow the user to filter the data to what they want to see; then page the results so that they aren't inundated with a deluge of data.



What my customer needs is a list/table (or whatelse we -the developers- call it) that has all the products that are stored in the warehouse with specific details (type, price, domain, etc). Doesn't care if it's to scroll or to press next/prev to cycle through all the products.
 
William Brogden
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Why not spit out a CSV file suitable for importing to a spreadsheet program? That would be very compact in transmission and let the spreadsheet do the heavy lifting.

XML would only be indicated if we were dealing with a hierarchical database.

Bill
 
John Astralidis
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

William Brogden wrote:Why not spit out a CSV file suitable for importing to a spreadsheet program? That would be very compact in transmission and let the spreadsheet do the heavy lifting.

XML would only be indicated if we were dealing with a hierarchical database.

Bill



This is not the case, but a custom application (will run on multiple devices PC, android, tamplet, etc) with access from many users simultaneously on the DB (read, write, update, etc). In scope to add some other features in the future.

Thank you very much for your support so far!

You gave me great ideas guys! Thank you so much!

Now I'm thinking of an interface just like web-mails (gmail, yahoo, etc) with pagination, like you proposed. But a new question came in my mind...

In pagination a GUI (say gmail...) says 1-50 of 400.

So, in order to know the amount of data records (eg 400) I have to get the last product_id and implement an algorithm as to select the required subset of data each time (1-50, 51-100,...,351-400), right?
 
Bear Bibeault
Sheriff
Posts: 67746
173
Mac Mac OS X IntelliJ IDE jQuery TypeScript Java iOS
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Most databases, as well as ORMs, will let you specify an offset and limit, which can be use to return just the results for the current "page". Figuring out what these value should be is a simple matter of math.
 
John Astralidis
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Bear Bibeault wrote:Most databases, as well as ORMs, will let you specify an offset and limit, which can be use to return just the results for the current "page". Figuring out what these value should be is a simple matter of math.



I tried to google it, but I didn't find anything relevant. Perhaps I use wrong keywords. Could you please give me some examples?

 
Bear Bibeault
Sheriff
Posts: 67746
173
Mac Mac OS X IntelliJ IDE jQuery TypeScript Java iOS
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

John Astralidis wrote:

Bear Bibeault wrote:Most databases, as well as ORMs, will let you specify an offset and limit, which can be use to return just the results for the current "page". Figuring out what these value should be is a simple matter of math.



I tried to google it, but I didn't find anything relevant. Perhaps I use wrong keywords. Could you please give me some examples?


What database or ORM are you using? The syntax is DB-specific.
 
John Astralidis
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Bear Bibeault wrote:

John Astralidis wrote:

Bear Bibeault wrote:Most databases, as well as ORMs, will let you specify an offset and limit, which can be use to return just the results for the current "page". Figuring out what these value should be is a simple matter of math.



I tried to google it, but I didn't find anything relevant. Perhaps I use wrong keywords. Could you please give me some examples?


What database or ORM are you using? The syntax is DB-specific.



I use MySQL with innoDB engine.
 
Ranch Hand
Posts: 63
Spring Java Google App Engine
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
JSON is not restricted, but returning large sets of data is really a bad design. get a set of records first display at client, and if more records are required make a call to server again to fetch the records.
 
reply
    Bookmark Topic Watch Topic
  • New Topic