sanchit dua

The following curl is working in Windows Environment

curl --user abc@xyz.com:testing --form file=@Testing.csv <URL>/data

But when I'm trying to introduce the commons HTTPClient 4.x for this request, I'm facing the error:

HTTP/1.1 500 Server Error

From the code snippet I'm passing the file and its value inside the BasicNameValuePair. So, whenever I'm passing nothing like

new BasicNameValuePair("file", "")

the response is coming

200 Ok

and obviously the content over the server gets replaced with Empty data.

I got one success in passing the content in Text (the CSV received content) by using StringEntity like

httpPost.setEntity(new StringEntity("a, b, c, d", "UTF-8"));

, but its again giving me 2 thoughts:

1. If I try to use the Text - I mostly face the Java Heap Space issue.
2. I'm not able to pass the FileName from inside StringEntity - like

new StringEntity("file=@Testing.csv")

Please let me know what I'm doing wrong.

This is the following code I'm using:

CredentialsProvider credsProvider = new BasicCredentialsProvider(); credsProvider.setCredentials( new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT), new UsernamePasswordCredentials("abc@xyz.com", "test")); CloseableHttpClient httpclient = HttpClients.custom() .setDefaultCredentialsProvider(credsProvider).build(); HttpPost httpPost = new HttpPost("<URL>/data"); List <NameValuePair> nvps = new ArrayList <NameValuePair>(); nvps.add(new BasicNameValuePair("file", "@Testing.csv")); httpPost.setEntity(new UrlEncodedFormEntity(nvps)); CloseableHttpResponse response2 = httpclient.execute(httpPost); try { System.out.println(response2.getStatusLine()); HttpEntity entity2 = response2.getEntity(); // do something useful with the response body // and ensure it is fully consumed EntityUtils.consume(entity2); } finally { response2.close(); }

Generics introduce following advantages over non-generic classes-
1. Type Safety
2. Explicitly type conversions are not required.

We get the same output then why we need Generics when we can get same output by ordinary method?

The following example demonstrates the use of Generics.

Program A.java package generics; class A<T> { T x; public A(T a) { x=a; } public void display() { System.out.println("Value of x is : " + x); } }



package generics; public class NonGenericTest{ public static void main(String... args){ B strObj = new B("It is a non generic class"); B intObj = new B(5); System.out.println("Value of String instance..."); strObj.display(); System.out.println("Value of Integer instance..."); intObj.display(); intObj = strObj; // Doesn't make sense, must not allowed (shall compile) } }

Now, we are doing some changes in the last program to make GenericTest.java.
package generics; public class GenericTest{ public static void main(String... args){ A<String> strObj = new A<String>("It is a generic class"); A<Integer> intObj = new A<Integer> (5); strObj.display(); System.out.println("Value of Integer instance..."); intObj.display(); intObj = strObj; // Make sense, must allowed (shall not compile) }

// Compile time Error:
Type mismatch: cannot convert from A<String> to A<Integer>


Type safety is the feature of detecting incompatible assignments at compilation time. In case of Generics each reference variable is given a type at the time of declaration hence, incompatible assignments are detected whereas incase of non-generic classes incompatible assignments are not detected usually when references of parent type are used to hold child objects.

Please let me know if it answers your question.

Regards,

Sanchit

Hi Anayonkar,

Anayonkar Shivalkar wrote:

1) ColWithData has two fields - col and data. The col field will contain a field name of TableClass field and data will contain the actual data. Now, how do you manage to link these two things? I mean, if your code has 50 TableClass object, how would you map which ColWithData object contains data for which TableClass object? Maybe you are maintaining the list of ColWithData inside TableClass.

Yeah the list of ColWithData is being maintained such that it will contain the 2 elements (the variable name and the value).

2) Why to do duplication of ALL data? I mean, if TableClass is having 10 fields, then you'll have 11th field - which is a list of ColWithData, and it will again contain 10 member, right? Basically, you are duplicating all the data just for the sake of hashCode and equals method.

The duplication is required to do the Mathematical operations (mainly calculating summary) over some of the fields like hours or amount if come. I have to separate the unique one and then perform the related summation if their related List of records contains the duplicates otherwise not.

3) What if value of a field in TableClass changes? Are you also going to do similar modification in cwdList? This way, you'll degrade your code performance, plus the code would be difficult to maintain (and note that by duplicating the data, you've already increased the memory footprint of your code).

If we're concerned about this application then during its execution the fields won't change while it may change for another executing instance. There are some predefined operations we need to perform out of them, which will be provided by the user as an input.

4) What if a field in TableClass cannot be expressed in String format? e.g. if TableClass contains a DB connection, what would be data field in corresponding ColWithData object?

The fields will contain the String values or null but not any other objects like connection of a database.

Frankly, I would follow the simple approach:
1) Whenever a field is added/removed, make corresponding changes in hashCode/equals method (just like your original code where you were considering 4 fields).

2) Or, obtain types and data of all fields via reflection - this way you'll never have to change your code even if you add/remove the field of TableClass.

What if I would like to follow the 2nd approach suggested?
Won't it make the code more complex like handling all those variables and injecting their values in equals and hashCode. Like if we make use of javassist library.

Reviewing my approach:
Yeah it seems very complex approach if we need to deal with the varying fields also it will take too much heap space if the data is going to be in thousands or even lacs.

I would really like to know if I should proceed for #2 approach from your side.

I think I have found the solution and its working for me. Please let me know if there could be the simple or any other way out finding the solution for this problem.

The code snippet is:

package mypack; import java.util.*; public class TableClass { public List<ColWithData> cwdList; public List<ColWithData> getCwdList() { return cwdList; } public void setCwdList(List<ColWithData> cwdList) { this.cwdList = cwdList; } @Override public boolean equals(Object obj) { TableClass tClass = (TableClass) obj; boolean returnVal = true; for(ColWithData cwd: this.getCwdList()){ for(ColWithData innerCwd: tClass.getCwdList()){ if(cwd.getCol().equalsIgnoreCase(innerCwd.getCol())){ if(!cwd.getData().equalsIgnoreCase(innerCwd.getData())) returnVal = false; } } } return returnVal; } @Override public int hashCode() { int hCode = 0; for(ColWithData cwd: this.getCwdList()){ hCode = hCode+cwd.getData().hashCode(); } return hCode; } }

And finally made a map as said:

Map<TableClass, List<TableClass>> map = new LinkedHashMap<TableClass, List<TableClass>>();
displaying the things as desired.

To compare the different objects of the same class with their contents like jobTitleId, classificationId, deptId & classificationId was to be done and do some manipulations later using Set and Map. I was able to do that by simply overriding the equals and hashCode methods of Object class and was able to fetch the information (like in the following Map).

Map<LocationData, List<LocationData>>

The following is the class I used (its been shown to you so that it can be referred for my problem statement):

LocationData class package com.astreait.bulkloader; public class LocationData { String locId, deptId, jobTitleId, classificationId; @Override public boolean equals(Object obj) { LocationData ld = (LocationData)obj; return this.deptId.equals(ld.deptId) && this.jobTitleId.equals(ld.jobTitleId) && this.classificationId.equals(ld.classificationId) && this.locId.equals(ld.locId); } @Override public int hashCode() { return deptId.hashCode() + jobTitleId.hashCode() + classificationId.hashCode() +locId.hashCode(); } }

Problem:

I'm already known to which all fields of this object I need to make the comparison.
i.e I'm bound to use the variables named classificationId, deptId, jobTitleId & locId etc.

Need:
I need to customize this logic such that the fields Names (classificationId, deptId, jobTitleId & locId etc) can be pulled dynamically along with their values. So, as far as my understanding I made use of 2 classes (TableClass and ColWithData) such that the List of ColWithData is there in TableClass object.

I'm thinking what if I override the same two methods `equals() & hashCode();`
such that the same can be achieved.


TableClass class #1 class TableClass{ List<ColWithData> cwdList; @Override public boolean equals(Object obj) { boolean returnVal = false; // I need to have the logic to be defined such that // all of the dynamic fields can be compared return returnVal; } @Override public int hashCode() { int returnVal = 0; // I need to have the logic to be defined such that // all of the dynamic fields can be found for their individual hashCodes return returnVal; } } ColWithData class #2 class ColWithData{ String col; // here the jobTitleId, classificationId, deptId, locId or any other more fields info can come. String data; // The corresponding data or value for each jobTitleId, classificationId, deptId, locId or any other more fields. }

Please let me know if I'm proceeding in the right direction or I should make some any other approach. If it is ok to use the current approach then what should be performed in the equals and hashCode methods?

Finally I need to make the map as: (Its not the concern how I will make, but can be considered as my desired result from this logic)

Map<TableClass, List<TableClass>> finalMap;

The main purpose of using Microsoft Access Databases is:
1. Another .Net application installed on several systems are already making use of MS Access database and we have to refer to those databases every time a change is reflected from them from the Java Application.
2. We can't do the Database migration here since we can have the data integrity issues like several relationships of tables and all.

I would also like to have some another from the solution I proposed earlier. As I have realized it would not provide the cost benefits by developing the given scenario.
For this I think I need to perform the following:
1. I have to first install some dedicated Application/Web Server over there on the Remote Server.
2. Then, based on the online accessibility (getting the remote accessibility) of this server I will perform the callouts from outside.

Its gonna cost me the overhead of maintaining the Web Server as well as so many web services callouts (Not much thinking but as I was asking about writing of web service, it is already consuming 3 Soap based web services).

So, not going into much details can you please let me know what should be more possibilities of dealing with this situation considering the Microsoft Access Databases solely?

What if I'm able to call another java application residing on the Server housing the Microsoft Access Databases? Can you please let me know if it can be the solution such that I will have the division of my application into two:
1. The Online java Application performing 2 things: Accepting the requests coming in and waiting for the response to come from another application running on the Server housing the Microsoft Access Databases.
2. The Java application installed on the Server housing the Microsoft Access Databases only responsible for connecting and returning the DTO (Data Transfer Objects) -or some kind of objects in Collections.

So, I would expect a serverside program to handle the communication with the DB and the client application would communicate with the server app.

Please let me know if I should proceed in this manner.

Do we have some free or open source Type-4 Driver for this case? As it seems to be not fitting for my pocket.

Currently:
Currently I have a Java application which performs all I wanted and the limitation
in it is "It only resides on the Local machine for which it needs to work". I have the Access Databases located on a Remote Server and I install the Java app on the remote server to connect the databases using a connectionString "jdbc:odbc:DRIVER={Microsoft Access Driver (*.mdb)}; DBQ=D:\\DATA\\DPPI\\DATA\\DPPI\\DPPIPR01.mdb";
It connects to Access Databases and do all the bussiness logics with them.

Enhancements (to be done):
1. Making the application online - not a big issue.
2. Connecting the databases remotely - its the only concern.
So, if I make the Application Online the Access Databases need to be connecting through network (somewhat like ip or some web address).

Problem:
But after going through this http://stackoverflow.com/questions/11388277/connect-to-access-database-remotely It seems my requirement can not be completed.
Is there any way if I'm able to upload the database to the online web hosting server and access the database from there only? If yes, then how should I do the connectivity since Access Database connects by giving the fully qualified name containing drive letter and all.
But how should I proceed further so that I may be able to connect the mdb Databases remotely?

ADO Solution:
But after going through the link: http://webcheatsheet.com/ASP/access_connection_strings.php, it seems we can perform remote Access Database connectivity.


The solution for the ADO is something like:

connectionString="Provider=MS Remote; Remote Server=http://your_remote_server_ip;" &_ "Remote Provider=Microsoft.Jet.OLEDB.4.0; Data Source=c:\your_database_name.mdb"


I'm a tad confused to how Access Database we can connect.

EDIT
There is one more question in my head.
I'm making use of Type-1 Driver. Should the Type-4 Driver support the remote connectivity?

If you just want to wait for a set of threads to finish, you can use for example a CountDownLatch in the main thread, initialized to the amount of threads. The main thread then waits for each thread to finish and notify the latch.

Repeat for the next sets.

Which ultimately solved my problem.

I have the following scenario which needs to be changed as soon as possible:
1. A group of sequential threads executes (contains 3 threads)
2. A group of another sequential threads executes (contains 2 threads)
3. A group of parallel threads executes (contains 9 threads such that producer-consumer approach of threads is applied correctly)

Why I created this scenario?
To execute a particular threads first (point-1: containing 3 threads), after their processing is done i required to execute another 2 threads (point-2: containing 2 threads). If and only if the 1 and 2 points are covered I required to process point-3.

Initially, I required to execute all the threads to process parallell operations, but I left that idea as the parallel threads would execute simultaneusly.

For the above situation everything is working smooth until some problem occured.

Why I need to change the above scenario?
Sometimes, I get an Exception which can be handled well in case of parallel threads. While the same Exception if I get in the Sequential threads that becomes unable to handle. Since, all other processing of sequential threads comes under waiting condition until the first completes.


So, I need to take the benefit of parallel threads but I chosen this way to be easy which has become a tough to handle situation for the application.
So, I need to create such a scenario in which I should execute different parallel threads for a fixed order. E.g:
Order-1st = process the first 3 parallel threads
Order-2nd = process the next 2 parallel threads
Order-3rd = process the next 9 parallel threads

Also, the above case is somewhat restricted, though I need a generic solution for this. Like if in future I want to add 2 more threads from Order-1 or, if I even remove some threads for any order the sequence of the groups should be executing as desired.
Is there any way in java through which we can make a group of parallel threads such that they can be executed for whatever order we may specify?


Edit
I have the following code snippet if you can help me modifying:

// Level-1 Thread[] threads = new Thread[list1.size()]; int j=0; for(list1.size()){ Object delegator1 = new Object("Level-1"); Thread th = new Thread(delegator1); threads[j]=th; th.start(); j++; } for(Thread thread: threads){ thread.join(); }
// Level-2 threads = new Thread[list2.size()]; j=0; for(list2.size()){ Object delegator2 = new Object("Level-2"); Thread th = new Thread(delegator2); threads[j]=th; th.start(); j++; } for(Thread thread: threads){ thread.join(); }
// Level-3 & 4 --> Applied Producer-Consumer using wait-notify working fine. for(list3.size()){ Object delegator3 = new Object("Level-3 & 4") Thread th = new Thread(delegator3); th.start(); }
I want these different levels to execute parallelly but if first level finishes off the second level should thereafter executes.

Thanks for all helping me out thinking out-of-box. The problem has been resolved by removing line-breaks for each cell.
Cheers

But the parsing logic for other files having newlines (\n) in data is working fine. Before writing data to csv file, I've assured there are no " (quotes) present. There is something malicous going on in Writing of file, Reading of file or Creating Temporary file. Regarding Manual Search of "qoute:
From last searching backwards for few records manually, I 've not found missing quotes.
Regarding Programmatical Search of "qoute:
For searching "qoute within quote, I'm not able to find them normally, for this I need to manually insert \ before "qoute. e.g "abc"",,"def" will not be searched instead by doing "abc\"",,"def".

I tried deleting some csv records from bottom, up-till a point came when I delete that record, a batch gets created with "numberRecordsProcessed=0" and it waits saying "waiting results-1" and then nothing happens. The record also doesn't look malicious.. Its something like "3745","TEST TEST1
12345 TEST TEST
TEST, TEST 43215","DPPI-3745"
And if I don't delete this record the error message still comes stating "Failed to parse CSV. Found unescaped quote. A value with quote should be within a quote"

Thanks for your reply Ulf Dittmer.
Yes, exactly it is data problem. I'm not able to discover what is causing behind this. Can you please let me know how can I perform its search from a CSV file of 28k records.