JSON Restrictions?

[John Astralidis]

Is there a limit to the size of data to be returned as a JSON response by an HTTP server;

Suppose that returns the records of a DB, which are thousands...

[William Brogden]

Assuming the server can write the JSON as a HTTPResponse output stream one item at a time - rather than building the whole thing in a buffer and then sending it, any limit will be on the client side, not the server side.

If you are thinking about a browser client that a human being has to interact with, returning more than a few screens worth of data is a very bad design.

Bill

[John Astralidis]

William Brogden wrote:Assuming the server can write the JSON as a HTTPResponse output stream one item at a time - rather than building the whole thing in a buffer and then sending it, any limit will be on the client side, not the server side.

If you are thinking about a browser client that a human being has to interact with, returning more than a few screens worth of data is a very bad design.

Bill

The client will be a java-swing desktop application that presents in a JTable all the DB records. So, what I'm thinking is the client having a method to request the data (eg getInventoryList()) and the server replying with a json stream containing all that data from DB.

If this is bad design what should be the correct ones?

Any advice would be highly appreciated.

[Jesper de Jong]

It's not useful for a user of any program to be presented with a screen containing a list with many thousands of items. People are not going to scroll through a list containing thousands of items and find what they need.

Design your program in such a way that you don't need to display thousands of records in your GUI at once. For example, by implementing paging - display at most a few tens of records at once, with buttons to get the previous or next page of results. Provide search fields so that the user can find the relevant data quickly, instead of having to look through thousands of records.

Doing queries that return huge amounts of data will also make your application slow, because it takes time to transmit a lot of data, and it takes a lot of memory to keep all the data in memory.

[William Brogden]

If this is bad design what should be the correct ones?

What Jesper said!

If this was my problem I would be talking to potential users, storyboarding various ways they might interact with the database. Nothing worse than working for days and having a user say "thats not what I wanted at all."

I would say: What kind of output do you want? What sort of selection capability? Do you need statistical summaries? Graphical summaries?

If they really are convinced that they want the whole DB on the desktop application maybe you should build them a local copy - maybe an in-memory DB?

Can the server return any other form besides JSON?

Bill

[John Astralidis]

William Brogden wrote:...
Can the server return any other form besides JSON?
Bill

This is all about a client application that I have already implemented but it's current architecture was not intended to run on multiple devices/platforms. So, I'm just an enthusiast junior developer who wants to practice and expand his knowledge.

In this project, I'm thinking to make use of RESTful web services and JSON format data transmission between server and clients.

This is my project
Web service for data synchronization?
Updating tables by simultaneous users

[Bear Bibeault]

I don't see how the choice of JSON is any way relevant to the issue; and is, in fact, the interchange format currently in vogue.

The key to large datasets is paging and filtering. As said by many before me: blatting out a ton of data to the user is useless.

[John Astralidis]

Bear Bibeault wrote:I don't see how the choice of JSON is any way relevant to the issue; and is, in fact, the interchange format currently in vogue.

The key to large datasets is paging and filtering. As said by many before me: blatting out a ton of data to the user is useless.

How would you suggest to implement the data exchange between server and clients, if JSON is not an option?

[Bear Bibeault]

I wouldn't suggest not using JSON. Prior to JSON, XML was the flavor of the day. Proprietary formats have their place, but not recommended without really good reasons.

[John Astralidis]

Bear Bibeault wrote:I wouldn't suggest not using JSON. Prior to JSON, XML was the flavor of the day. Proprietary formats have their place, but not recommended without really good reasons.

To sum up, I should use pagination/filtering and JSON/XML/etc (I find JSON much simpler though) for data exchange...

[Bear Bibeault]

I would not recommend XML unless you have a really good reason.

But yes, allow the user to filter the data to what they want to see; then page the results so that they aren't inundated with a deluge of data.

[John Astralidis]

Bear Bibeault wrote:I would not recommend XML unless you have a really good reason.

But yes, allow the user to filter the data to what they want to see; then page the results so that they aren't inundated with a deluge of data.

What my customer needs is a list/table (or whatelse we -the developers- call it) that has all the products that are stored in the warehouse with specific details (type, price, domain, etc). Doesn't care if it's to scroll or to press next/prev to cycle through all the products.

[William Brogden]

Why not spit out a CSV file suitable for importing to a spreadsheet program? That would be very compact in transmission and let the spreadsheet do the heavy lifting.

XML would only be indicated if we were dealing with a hierarchical database.

Bill

[John Astralidis]

William Brogden wrote:Why not spit out a CSV file suitable for importing to a spreadsheet program? That would be very compact in transmission and let the spreadsheet do the heavy lifting.

XML would only be indicated if we were dealing with a hierarchical database.

Bill

This is not the case, but a custom application (will run on multiple devices PC, android, tamplet, etc) with access from many users simultaneously on the DB (read, write, update, etc). In scope to add some other features in the future.

Thank you very much for your support so far!

You gave me great ideas guys! Thank you so much!

Now I'm thinking of an interface just like web-mails (gmail, yahoo, etc) with pagination, like you proposed. But a new question came in my mind...

In pagination a GUI (say gmail...) says 1-50 of 400.

So, in order to know the amount of data records (eg 400) I have to get the last product_id and implement an algorithm as to select the required subset of data each time (1-50, 51-100,...,351-400), right?

[Bear Bibeault]

Most databases, as well as ORMs, will let you specify an offset and limit, which can be use to return just the results for the current "page". Figuring out what these value should be is a simple matter of math.

[John Astralidis]

Bear Bibeault wrote:Most databases, as well as ORMs, will let you specify an offset and limit, which can be use to return just the results for the current "page". Figuring out what these value should be is a simple matter of math.

I tried to google it, but I didn't find anything relevant. Perhaps I use wrong keywords. Could you please give me some examples?

[Bear Bibeault]

John Astralidis wrote:

Bear Bibeault wrote:Most databases, as well as ORMs, will let you specify an offset and limit, which can be use to return just the results for the current "page". Figuring out what these value should be is a simple matter of math.

I tried to google it, but I didn't find anything relevant. Perhaps I use wrong keywords. Could you please give me some examples?

What database or ORM are you using? The syntax is DB-specific.

[John Astralidis]

Bear Bibeault wrote:

John Astralidis wrote:

Bear Bibeault wrote:Most databases, as well as ORMs, will let you specify an offset and limit, which can be use to return just the results for the current "page". Figuring out what these value should be is a simple matter of math.

I tried to google it, but I didn't find anything relevant. Perhaps I use wrong keywords. Could you please give me some examples?

What database or ORM are you using? The syntax is DB-specific.

I use MySQL with innoDB engine.

[Radhakrishna Sharma Gorenta]

JSON is not restricted, but returning large sets of data is really a bad design. get a set of records first display at client, and if more records are required make a call to server again to fetch the records.