Atul Mishra

Hi all,

We have to create a new Oracle table using subset of data from 3 existing Oracle tables.
The data is huge, around 90M.

We have tried procedures which use Oracle JOINs across the three tables. The performace is not as quick.
Could anyone please suggest any tools/ faster mechanisms through which we can achive this ?

Thanks

Thank you Paul.

Thanks Paul.
Sorry one more question.

If I use the method you specified below, it will give me all NodeLists with this namespace. right ?

The problem is, "Node City" can belong to other Elements as well.

So I have to extract the Node which is specific to an Element.

Here is again XML smippet


- <Location> - <city:city xmlns="http://www.w3.org/TR/REC-html40" xmlns:City="http://www.w3.org/TR/REC-html40"> <state>IL</state> <postalCode>60014</postalCode> </City:City> </Location> - <MainLocation> - <city:city xmlns="http://www.w3.org/TR/REC-html40" xmlns:City="http://www.w3.org/TR/REC-html40"> <state>IL</state> <postalCode>60014</postalCode> </City:City> <cityId>M36b</cityId> </City:City> </MainLocation>

So in this case I need to get the node under the element "MainLocation".

Thanks much

So in both cases you have a City element in the "http://www.w3.org/TR/REC-html40"; namespace. There wasn't any change. Therefore, since your code isn't treating the two elements the same way, your code doesn't handle namespaces properly.

Thanks Paul.

This is what the code does today.
Element city= (Element) XMLHelper.findNodeByName(rootElement, "City");


XMLHelper class contains this for the method findNodeByName

public static final org.w3c.dom.Node findNodeByName( org.w3c.dom.Node parent, String name) { if (parent == null) return parent; String parentName = parent.getNodeName(); if (parentName.equalsIgnoreCase(name)) return parent; org.w3c.dom.NodeList nodeList = parent.getChildNodes(); int length = nodeList.getLength(); for (int l = 0; l < length; l++) { org.w3c.dom.Node node = nodeList.item(l); String nodeName = node.getNodeName(); //String test = node.getNamespaceURI(); // ignore comments if (nodeName.startsWith("#")) continue; if (nodeName.equalsIgnoreCase(name)) return node; org.w3c.dom.Node foundNode = findNodeByName(node, name); if (foundNode != null) return foundNode; } return null; }


Please let me know what modifications to be made for this to handle namespaces.

And for my understanding,
Since the namespace URI is the same in both cases, the only difference you see is in the introuction of a prefix. right ?
If the code handled namespaces, introduction of prefixes is something which code would have handled right ?

Thanks

Hi all,

I am working with an application which is existing for 6 years now.
Here is an issue I am facing with one of our clients who send us an XML and we parse it on our end.

In the XML there was a Node which looked like this:

<City xmlns="http://www.w3.org/TR/REC-html40">

The existing code uses Xerces parser and uses findNodeByName and gives node name as "City" to parse this.

Now recently the clients starting sending us XMLs where the above tag looks like

<City:City xmlns="http://www.w3.org/TR/REC-html40" xmlns:City="http://www.w3.org/TR/REC-html40"> // So they changed the namespace.

Unfortunately we as the backend application didnt know about this namespace change.

My question is, Should we have "assumed" that this namespace change can occur ?

And while parsing should we have accomodated namespaces ?

Please help
Thanks
Atul

Oh sorry..
Could anyone please suggest any ways to analyze this in AIX environment ?

Thanks

Tried using .phd file, didnt work
Somehow the .hprof file is not getting created

Here is my java home settings and classpath entries

-Dcom.ibm.oti.vm.bootstrap.library.path=/usr/java5_64/jre/bin
-Dsun.boot.library.path=/usr/java5_64/jre/bin
-Djava.library.path=/usr/java5_64/jre/bin:/usr/java5_64/jre/bin:/usr/java5_64/jre/bin/classic:/usr/java5_64/jre/bin:/appl/local/nsp/lib/oracle:/usr/java5_64/jre/bin/j9vm:/appl/local/nsp/lib/oracle:/usr/lib
-Djava.home=/usr/java5_64/jre

I did the above and am getting three files only

heapdump.20091105.003746.1380464.phd
Snap0001.20091105.003746.1380464.trc
javacore.20091105.003746.1380464.txt

the other file didnt produce.

Please suggest as of what am I missing

Can i use the .phd file against Visual VM ?

Thanks

Peter,

I should just get VisualVM from Jdk 1.6 ?
Trying to download 1.6 and install it on my machine.
Then I will change the jvm option in the AIX machine and get the file.

Will let you know what I find out.

I did it, and added this to the shell script which calls the program
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=file:///oom.hprof

but I dont see a file. Am i missing any ?


Thanks
Atul

Hi all,

First-thanks for the replies:

1. satya - sorry I used "quotes".. will use code tags going ahead.
What I am thinking is since I am storing all objects in the collection, Collections[Lists] eat up memory.

Hence I am trying to see whether I can get chunks of resultset and processs the chunks rather than loading all 20K [or even 50K] to collection at once..

Do you agree ?

Thanks Peter and Kri Shan
Peter-we dont have 1.6 yet, still in 1.5
and also i dont think the company will allow modifying prod file to do it.
I will give a try

Thanks

Could any one please provide inputs ?

Thanks

Paul,

Could you please give me some more inputs ?
I tried using sublist method. looks like that is eating so much memory..

Again I want to commit at every 1000 transactions

Thanks

Hi all,

I am dealing with an out of memory exception while calling an Oracle stored procedure using Callable statement.

I have called 20000 objects with this procedure, procedure doesnt complain. Then when the objects are 24000 I got the error.

So what I did is, I created sublists with 1000 objects each and then for those 1000 objects, I call the stored procedure.

Still I deal with the same problem..

Any ideas will be appreciated.

Right now when I generate sublists, I am creating connection objects and then closing them also for each 1000 sets.

so its like one connection object will work for 1000 objects.
Callable statement is also registered once for every 1000 objects

csValues = conn.prepareCall("{ ? = call getvalues_proc(?,?,?,?,?,?,?) }");
csValues .registerOutParameter(1, oracle.jdbc.OracleTypes.CURSOR);

then before i stop working on these 1000 objects, I dispose the connection object and the csValues callablestatement also.

But still it fails saying Out of memory exception.

This is what it says exactly

Exception in thread "main" java.lang.OutOfMemoryError
at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java:879)
at oracle.jdbc.driver.T4CResultSetAccessor.getCursor(T4CResultSetAccessor.java:315)
at oracle.jdbc.driver.OracleCallableStatement.getCursor(OracleCallableStatement.java:1659)

Please provide any thoughts

Now here is how I generate sublists.. I am thinking that I keep on creating list objects.. so even though Its working on 1000 objects at a time, the previously creating objects are still existing in memory.

Am I correct ?

this is a sample method I have which I use to create sublist

public void createBatch(List <TestObj> objs ){ int size = objs.size(); int i = 0; if(size > 1000){ while(i < size){ if((size-i) < 1000) { List<TestObj> batchObjs = objs .subList(i, size); processTestObj(batchObjs,conn); }else { List<TestObj> batchObjs = objs .subList(i, i+1000); // my doubt is if the total number of items in list is huge, the number of lists getting created are becoming more. log.info("generating sublist" + i); processTestObj(batchObjs,conn); } i = i+1000; } }else{ processTestObj(batchObjs,conn); } }

Thanks
Atul


[NK: Added code tags]

Hi all,

This is an exception handling question which I have.
lets say that I have a List in which I have all 50 states in US.
My main program has a process each list method and this calls other classes and processes each communities belonging to different states.

Lets say that while processing one state, a null pointer exception came or a SQL exception came. Rather than stopping and exiting the whole program, is there anyway to say, Ok. this state had an exception but go ahead and process the next one..

Thanks
Atul

I have a question while using bulk inserts/ bulk update

How do I do batch commits ?

Like if i have 10000 records to be inserted, how will I say to commit every 1000 transactions.

Thanks
Atul