• Post Reply Bookmark Topic Watch Topic
  • New Topic

Working with really long strings?  RSS feed

 
Mike London
Ranch Hand
Posts: 1505
11
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello,

I have a third-party application program which is generating a long list of words that it will send to a web service to sort. The web service code is done (simple), but the problem is that the string is too long for java (15,000 words) and maybe too long for the POST length, too.

So, trying to think about how to solve that problem AND still have performance be acceptable, I'm thinking if possibly instead doing a JDBC connection to that third-party application would be better. Do a SQL against the third-party app's database, do the sort, and then do an INSERT via JDBC with the sorted list.

Would appreciate other ideas how to deal with super long strings and passing them to Java web services (or even being able to create them at all).

Thanks,

- mike
 
Junilu Lacar
Sheriff
Posts: 11494
180
Android Debian Eclipse IDE IntelliJ IDE Java Linux Mac Spring Ubuntu
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Those are all the kind of things that keep a security-minded person up at night. Using a POST request to handle large amounts of data opens you up to DoS (denial of service) attacks. Opening up a JDBC connection to a third-party application is a horrific idea. Thinking of making that connection as something you'd do in the UI layer is even more horrific (you didn't say anything specific about this but I imagine the thought wasn't far from the back of your mind).

When it comes to web services, the first option people consider is usually SOAP. Other options you might want to investigate: WebSockets, Message Queues, and the old tried and true FTP or SFTP. See http://stackoverflow.com/questions/405800/large-amount-of-data-what-is-the-best-way-to-send-them for more discussion.


 
Ron McLeod
Bartender
Posts: 1603
232
Android Angular Framework Eclipse IDE Java Linux MySQL Database Redhat TypeScript
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Mike London wrote:.. the problem is that the string is too long for java (15,000 words) and maybe too long for the POST length, too.

I think that as long as you have enough heap space, a String can have a length of Integer.MAX_VALUE, so I don't think there should be a problem with 15,000 words.

And for POST, I have some web service methods which move 20MB+ of XML data, so your POST payload shouldn't be a problem.  There may however depending on the particular client and/or server implementation, be a limit to the number of bytes which you can include in the query string - often capped at 2048 bytes, but you shouldn't be putting the payload in the query string anyway - it belongs in the request body.
 
Mike London
Ranch Hand
Posts: 1505
11
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ron McLeod wrote:
Mike London wrote:.. the problem is that the string is too long for java (15,000 words) and maybe too long for the POST length, too.

I think that as long as you have enough heap space, a String can have a length of Integer.MAX_VALUE, so I don't think there should be a problem with 15,000 words.

And for POST, I have some web service methods which move 20MB+ of XML data, so your POST payload shouldn't be a problem.  There may however depending on the particular client and/or server implementation, be a limit to the number of bytes which you can include in the query string - often capped at 2048 bytes, but you shouldn't be putting the payload in the query string anyway - it belongs in the request body.


Thanks for the replies....I'm specifically talking about this error: "Error37, 22) java: constant string too long" in just a test java program. Though it could be a heap space issue. Not sure.

This is a compile time error.

I tried to set the VM options in Intellij to -Xmx 8096m with no effect.

If there is some way I can send you my test data file that would be great. JavaRanch won't accept any version of it (size, file type, etc.)

Here's the (testing) code and data.

 

===========

Thanks,

- mike
 
Paul Clapham
Sheriff
Posts: 22835
43
Eclipse IDE Firefox Browser MySQL Database
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes, Java string literals can't exceed 65535 characters in length. So if you have a constant string that long, maybe you could read it from a file or maybe you could break it into parts like this:


 
Mike London
Ranch Hand
Posts: 1505
11
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Paul Clapham wrote:Yes, Java string literals can't exceed 65535 characters in length. So if you have a constant string that long, maybe you could read it from a file or maybe you could break it into parts like this:




I see, so it's really not the String object per se, but rather each string constant. I get it.

I was actually just trying to do a SQL from a third party application and send that data to a web service and have it be fast. Breaking up the strings may slow things down a bit, but I don't really see another way around this.

Thanks for your note! :)

- mike
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!