Nancy Joe

Greenhorn
+ Follow
since Feb 09, 2019
Cows and Likes
Cows
Total received
0
In last 30 days
0
Total given
0
Likes
Total received
0
Received in last 30 days
0
Total given
0
Given in last 30 days
0
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by Nancy Joe

Thank you. I am exploring the streaming option with the JsonToken.
2 days ago
Hi  Tim,

Yeah, thats what our workflow looks like in short. Interesting that you worked on some of the pentaho source code. Basically in our case, pentaho makes one rest call to the service to pull tons of data (in GBs). The inbuilt rest client step in pentaho did not work for us since we got the gzipped json response back from the service and pentaho did not know how to decompress it. So we created a java library(which calls the rest service and decompresses the gzip to json string) which we use in the user defined java step and pass it back to a json input step in pentaho where it decodes the fields. But the user defined java step is not able to handle so much data in memory. We are using the com.fasterxml.jackson.core.JsonParser, but that also stores everything in memory when reading the json tree from the mapper.  Is there any other option than calling the rest service in batches?

Thanks.
3 days ago
Thanks for all the responses. Basically, we are using an ETL tool to load the data into the DB. But the intermediate step is the rest client which uses a java library where I am having the problem. Once I send the json string to pentaho, it does the rest.
5 days ago
Hi Stephan,

Thanks for your response. It is huge, can run in MB and GB. After reading this data, it needs to be loaded into a DB, so it needs to be in the memory.
5 days ago
Hi all,

When trying to read a large gzip response from a BufferedReader , I get this exception on this line: while ((line = bf.readLine()) != null) {

I tried increasing the heap size xmx , it still doesn't work. Any thoughts on how to fix this?

Thanks.

This is the exception:

java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
at java.lang.StringBuilder.append(StringBuilder.java:190)
at java.io.BufferedReader.readLine(BufferedReader.java:370)
at java.io.BufferedReader.readLine(BufferedReader.java:389)


This is the code:

6 days ago
Thanks, appreciate all your responses.  I dont see the  application/x-gzip in the MediaType class. How do we include that? The sample response from postman is:

�

the above pattern goes on forever which is ofcourse gibberish.


Like you all suggested, I modified some of the code, here is the new snippet I have:
In the headers,

     


I am also sending query parameters along with the params variable as seen above.

When I call :

ResponseEntity loginResponse = restTemplate.exchange(url, HttpMethod.POST, entity, Object.class);


It fails with the below exception:

org.springframework.web.client.RestClientException: Error while extracting response for type [class java.lang.Object] and content type [application/json;charset=utf-8]; nested exception is org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in string value; nested exception is com.fasterxml.jackson.core.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in string value
at [Source: (PushbackInputStream); line: 1, column: 127]



Tried with bytes as well as below:
ResponseEntity<byte[]> responseEntity = restTemplate.exchange(url, HttpMethod.POST, entity, byte[].class);


With bytes, I get this exception:

org.springframework.web.client.RestClientException: Error while extracting response for type [class [B] and content type [application/json;charset=utf-8]; nested exception is org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Cannot deserialize instance of `[B` out of START_OBJECT token; nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `[B` out of START_OBJECT token
at [Source: (PushbackInputStream); line: 1, column: 1]



I am not sure at this point, if it is a problem with the service provider or with the request I am sending. The service provider tells me that other clients are using it, maybe not the same API but a similar one.

Thanks so much for the help!
3 weeks ago
Hi Rob,

Thanks for the response. There is no content-encoding. But there are these:

Content-Disposition: attachment; filename=response.json.gz
Content-Type: application/json;charset=utf-8
Transfer-Encoding: chunked


I am pretty sure the response is in gzip format since few other clients are already leveraging it. I am using the rest template constructed using ClientHttpRequestFactory. The response body that I get is not a readable format. Hence I use the following code to decompress it.
This is a snippet of the code I use:

byte[] inputBytes = responseEntity.getBody().getBytes("UTF-8");
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(inputBytes);
GZIPInputStream gis = new GZIPInputStream(byteArrayInputStream );

At this point, I get exception. java.util.zip.ZipException: Not in GZIP format

I have also tried to to decode using Base64 as:
inputBytes = Base64.decodeBase64(inputBytes );

That also did not work. Please help.

Thanks.

3 weeks ago
Hi,

I am trying to read a .json.gz response through a rest call. When decompressing into GZIP InputStream, I get an error "Not in GZIP format". I tried to check for the GZIP Magic number as well as per this reference:
https://www.programcreek.com/java-api-examples/?class=java.util.zip.GZIPInputStream&method=GZIP_MAGIC

Nothing seems to work. Any ideas how to get around it?

Thanks.
3 weeks ago
Hi all,

I have a scenario where a cluster in weblogic has 4 nodes and all nodes are up. But looks like 3 nodes are not accepting requests for some reason. There is also a apache front end to take care of load balancing.
Any ideas why that might be?

Thanks!
2 years ago
We have a huge json payload. When deserializing it with 2.8.11, it works just fine. When using 2.9.8, one of the embedded objects is null. What is the best way to debug the deserializer? I dont see any exceptions in the log. If I go through the debugger, it takes forever and I never reach the end. Please help. Thanks.
2 years ago
I have the mail.jar and activation.jar in the lib folder of the WEB-INF directory of my application. I have the bean Id configured as:


The emails are still not getting sent. What else do I need to do? Thanks.
2 years ago
Thanks for all the responses. Dave, this was legacy code. We did not have any app context then. We had to do all the wiring ourselves. Now we are porting it over to spring context.
I removed the singleton instance and just having it as a normal bean. It is working now. It is fetching the resource bean now just fine. I am guessing the singleton instance is created when the classes load, the creation of the beans happens later when the bean factory is instantiated. Hence, the singleton instance did not have the bean reference in it.
2 years ago
Thanks for the quick response. Sorry my bad. Let me post that code right here. I am accessing the bean from a  singleton instance like the below. Now that you say, is it because I am using a singleton instance that it is not able to find the bean?? Thanks!



2 years ago
Thanks for the response. Here is the code:



I have masked some of the package names and field names. Basically, that is the structure that I have.
2 years ago