• Post Reply Bookmark Topic Watch Topic
  • New Topic

Filter and deleting data in table  RSS feed

 
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello guys!
I am really having a headache with this problem.
I am new in handling data in java. Here is my problem.
I have an ArrayList<ArrayList<String>> where my data is stored. It simulates a table with specific number of columns containing the information, for example, product, quality, packaging and the column with quantity in units.
I need to make a filter on the data of the array, for example I need to filter just product 1 and 2, whit first and second quality, and with packaging type 1. All the data not fixing that conditions, must be discarded. My really issue is that, i did that method of filtering and deleting by "for" cycles walking line by line, evaluating each position. Then, since the array can have so much lines, the proccedure becomes extremely slow(20 seconds for example filtering just one array with about 15 thousand lines). I really need something very very much faster because I have to execute the process so many times in my program and the computational efficiency is one of my objectives.
Do you know about something that can help me?
 
Ranch Hand
Posts: 614
9
BSD Java
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
There are few ways to do it.
1. Use comparator
For that it is better to use List<POJO> instead of ArrayList<ArrayList<String>>
2. Fetch only the required records from database.

You can choose what suits your application best.
 
Bartender
Posts: 10575
66
Eclipse IDE Hibernate Ubuntu
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Daniel Espinoza wrote:I have an ArrayList<ArrayList<String>> where my data is stored. It simulates a table with specific number of columns containing the information, for example, product, quality, packaging and the column with quantity in units.

Well that sounds to me like your first problem right there. An ArrayList<String> is NOT a good substitute for a proper Java class that includes the information you list above as member fields (called Stock perhaps?). Then, as Tapas said, you can store the information in an ArrayList<Stock>.

For more info, have a look at the StringsAreBad page.

However, it's possible that you've been given this structure for a class exercise, and if so you should use it (although IMO, it's bad form). Tapas also touched on another point, and that is that even if your list is an ArrayList, you should always refer to it as a List.
The only place your program needs to know what type of List it is is when it's created, viz:

List<List<String>> data = new ArrayList<ArrayList<String>>();

Daniel Espinoza wrote:My really issue is that, i did that method of filtering and deleting by "for" cycles walking line by line, evaluating each position...

Why do you need to do that? Even if you do have to use a List<List<...>>, if your data is a fixed column positions, then you can simply go directly to the "column", eg:
although, as Tapas also mentioned, you could also do this with a Comparator.

However, my first suggestion would be to find out if you have to use that horrible List<List<...>> structure, because it's really not needed.

HIH

Winston
 
Daniel Espinoza
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Winston Gutkowski wrote:. . .

Thanks a lot you guys Tapas and Winston.
I understand your suggestions. But, here is another point. What I am doing is to develop a framework to optimize by using linear programming solved through metaheuristics. Then, in the example, the ArrayList<ArrayList<String>> is representing a variable of the model. So that, if I want to represent that variable as a List<POJO>, then the POJO object cannot be static, I mean, the lenght of the variables can vary, and one variable can have just two columns and other one can have 3 or more.
How can I represent that POJO object in a dynamic way?
For example if the variable is "stock", then the variable has the attributes: product, variety, quality, time period and quantity(stock) ; but if the variable is "production", then it has the attributes: production plant,product, production line, variety, quality, time period and quanitity(production).
Then, each of that two variables can have thousands of lines, each of them with a specific unrepeatable combination of the attributes of the variable.
Naturally, I will use that variables to make operations over them. One operation is to filter and delete lines that not fit a predefined condition.
Can you please guide me in what's the best way to code this?
 
Marshal
Posts: 56600
172
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Welcome to the Ranch

There is no need to quote the whole of a preceding post, so I removed the quote.

Please explain more about what you are trying to do. At the moment it looks like something where you are asking, “What is the best way to code this?” and I think there is no best way. That sounds like something which is inconsistent with object‑oriented programming and which may be exceedingly difficult and error‑prone to achieve.
 
Campbell Ritchie
Marshal
Posts: 56600
172
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You can easily filter objects from a List using a Stream. The List can put all its elements into a Stream and the Stream can use its filter() method to create another Stream containing those elements which pass the filter. But it would be very difficult to implement if you are simply passing many Strings. As you have been told, create a Stock object.
And how are you going to compare Stock with Production? They talk about comparing apples and oranges, but here you are more comparing apples with trees. That looks infeasible to me at this stage.
 
Daniel Espinoza
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Sheriff
No no, I will not compare stock with production. I just can actually make operations(Multiply, divide, sum, subtraction) just over variables of the same lenght(production + production for example). I mentioned stock and production just to clarify that I will generate different variables with different lenght. So that, I will need a dynamic object called "variable" for example that can serve to generate the stock and production objects. I can't do it separately(two different objects, stock and production) because, it is supposed that this program operates generally. So the user must be able to have "variables" and each of them can have different lenght. Again, the operations will take place just over variables of the same lenght. My questiong would be, how can I design that "variable" object, taking into account that I will need to make operations with variables of the same lenght(multiply...) and operations of filtering and deleting over one same variable?
 
Campbell Ritchie
Marshal
Posts: 56600
172
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I presume you are getting result sets from the database? If you have a result set, how do you tell from it whether you have a stock result or a production result? I presume that all the columns for production are in the same order as the columns for other production (and similarly for stock). If you can write that down, then you have a means for getting stock objects and production objects, and putting them into the right List/Stream/Whatever.

But why are you not getting the database to do such hard work? Add a WHERE myTable.type = 'PRODUCTION' to your SQL query and let the database worry about it. Remember database programs can do that sort of query on 10⁹‑record databases in a second or two.
 
Saloon Keeper
Posts: 3330
46
Eclipse IDE Firefox Browser Java MySQL Database VI Editor Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sounds something like a spreadsheet. You have a table with rows and columns. I'm assuming that for any given table that the row lengths are the same. You could have your List<List<String>> as you said or you could create a Row (or "Record") class that has a List<String> and then your table is a List<Row>. I found using a Row class preferable. You could have a SubTable class that has a reference to the table, and then beginningRowNumber, beginningColumnNumber, width, and height and perform operations on two SubTables.

I had to do something similar years ago and it was messy. I tried very hard to keep the application that made use of my tables from having to deal with the internal structure (encapsulation is your friend).
 
Sheriff
Posts: 11494
180
Android Debian Eclipse IDE IntelliJ IDE Java Linux Mac Spring Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You're trying to generalize either too much or too early or both. I prefer to start with specific implementations and get at least three similar instances so that I can see a pattern start to emerge. This works for me in the small as well as in the large. That is, the general approach is the same whether I'm looking for a few duplicate lines of code to eliminate or a repetitive series of calls that can be parameterized and modularized better. If I were doing this, I'd start with a List<SomePojo> as suggested, then do another List<AnotherPojo>. If I'm not sure I'll need to do the same thing again, I'd just leave it that and live with that little ugliness. However, if a need for a third instance of List<YetAnotherPojo> arises, then I would start looking at the implementations and see where the similarities lie and try to modularize and parameterize.

Just how many different variations of this "pattern" do you think you'll have? If you're going to have dozens of different types of data that you want tabulate and filter, then it might be worth the effort to try to generalize, otherwise, maybe it's not worth the effort. If you do decide to generalize, I suggest you do so incrementally so that you don't get overwhelmed with the complexity of trying to do everything at one go.
 
Junilu Lacar
Sheriff
Posts: 11494
180
Android Debian Eclipse IDE IntelliJ IDE Java Linux Mac Spring Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Daniel Espinoza wrote:I really need something very very much faster because I have to execute the process so many times in my program and the computational efficiency is one of my objectives.
Do you know about something that can help me?

Is this for a school assignment or for work? Are you allowed to use third party libraries? Because if you are, then hand-crafting something generic is not a good investment in time and effort, IMO. Something like MongoDb would be a better and more robust alternative to a home-spun framework and you'll probably spend about as much time trying to learn it as you would trying to come up with your own thing and get it working reasonably well.
 
Daniel Espinoza
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Ok guys, It seems that I have to give you a better contextualization. There are a lot of privative solvers in market to solve linear programming models. It exists also a lot of open solvers that are usually considerably less efficient than privatives. The issue is that, once the size of the variables of the model becomes very big(what is known as NP-Hard in optimization), the model gets unsolvable through conventional solvers. That's when we look for alternative methodologies like Metaheuristics(genetic algorithms, swarm optimization, etc..). The problem with metaheuristics, is they have not been applied to face NP-Hard problems(there is a lot of methods that researchers use to handle this kind of problems, like solve it partially, etc, etc..). So what I am trying to do, is to focus my effors in making a pretty simple and clear modelling framework that can be easily adapted to a metaheuristic(I already have a very good framework of metaheuristics in java, but no one has applied it to solve a very large model). We did an initial experiment achieving very good results with a middle scale problem(working with a very bad designed proccedure to make the variable representation Object[][]). The only thing I need to make this works fine, is to achieve fast and efficient operation methods among variables(multiply, divide, sum, subraction, basically), because the hard work of searching for a solutions is a task of the metaheuristic and it can do it very well. So, regarding some questions in the forum, I don't read data from a data base each time I use a variable. I read parameters from the data base like "product price" for example just one time at the begining of the proccedure, then I have to build a variable called production, the values of the variable production(the rows in that variable) must be supplied by the metaheuristic in the solution seeking process(that's other task appart of this forum already solved). Then, If I want to calculate the earns for example(objective function), I have to multiply the production*product price.
So my problem in this moment is, what is the best way to build that variables in order to make the operations easy and fast to handle?I have tried with Object[][], String[][], ArrayList<ArrayList<>>, I have read about hashtables, hashmaps, JTables, etc.. but since I am not experienced developer, I dont know what is the best option to achieve computational efficiency. If you guys recomend me to use List<POJO> sintaxis, I will use it, but I need to construct that pojo object dinamically (able to construct variables with different attributes or lengh), because trough the same procedure I have to build different variables(stock, production, supply, costs, earns, etc...). How can I do that?
The data what I will upload to the variable is not coming from a data base, but it is not a problem because I already have the proccedure to obtain it. This is part of my doctoral thesis but, if it works, we could apply it to a real case, then it is not the best option for us to use privative methods or "third party libraries".

Thanks a lot in advance for your help guys.
 
Junilu Lacar
Sheriff
Posts: 11494
180
Android Debian Eclipse IDE IntelliJ IDE Java Linux Mac Spring Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Off the top of my head, it seems like it would be best to look at using some of the new features in Java 8 such as method references, streams, and lambda expressions. Because of the level of abstraction you'll need to generalize this, it could get a bit complicated but the new Java features can help you manage the complexity of the code. I don't know how dynamic you can get it though. Seems to me like you could have a pretty decently generalized framework and pass in method references to make retrieving values more flexible and use lambda expressions for flexibility in the actual computations involved. You'd still have to write case-specific code for each different type of data/calculation and compile it though. With proper modularization, however, you can make each case something that can be plugged-and-played into your framework. This still goes back to my earlier suggestion to incrementally work in generalizations after seeing a pattern emerge from specific instances though.
 
Winston Gutkowski
Bartender
Posts: 10575
66
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Daniel Espinoza wrote:If you guys recomend me to use List<POJO> sintaxis, I will use it, but I need to construct that pojo object dinamically...

And there might be your problem. Java is NOT a dynamic language, and was never intended to be. It is statically-typed, so however you do it (and there are patterns), creating a dynamic type on the fly is likely to be a kludge.

The question therefore remains: Do you need to create a genuinely dynamic type, or merely one of several predefined choices? If the answer is (a), then I'd say that Java is probably not the best choice of language; if it's (b) then the chances are you'll be fine, but it may involve some duplication or boilerplate code.

There are also "middleware" products like Hibernate, that help to translate database tables and/or views to POJOs, but I get the impression that your requirement is a bit more like data mining - ie, to extract specific items from a database by column name and "marshal" them into an ordered collection of "rows", each of which constitutes an object of some dynamic type. And that is likely to be hard - unless, as I say, these types can be predefined.

About the only other advice I can give you is to avoid Reflection. It's OK in small doses, and might seem like a perfect solution to your problem, but it's arcane, error-prone, and SLOW; and an application founded on Reflection is probably best written in another language.

Otherwise, listen to Junilu: He speaketh not with forked tongue. :-)

Winston
 
Don't get me started about those stupid light bulbs.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!