• Post Reply Bookmark Topic Watch Topic
  • New Topic

How to improve performance?  RSS feed

 
Tushar Goel
Ranch Hand
Posts: 934
4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I got my first project in which i need to receive some data from multiple client on network regularly(say every second) and then process that data and stored it to the database and then GUI needs to be created which display data accordingly.

While thinking for the design i thought blocking queue to implement the same. Like producer receive data from the port into a string builder and put that into the LinkedBlockingQueue. Consumer will take this data , process some work on it and store it into the database.
I am using sybase ase 15 as a database.

My question is as i am receiving data very frequently which require insertion , updation and search almost every second into the database. What is the best way to improve the performance.

say i have 2 types of data is coming one having keyword 'Alarm' and other having 'ceased' so i need to use something like this

common fields in both is name and number.

if (ceased)
search into the database for particular event for which this ceased receive using name and numbe rand update the another field into it
Also make separate entry for it
else
make a entry

Also as i am using 2-3 thread for consumers and thread = Number of client for the producer. So i thought to use synchronization at the consumer side which writes data into the database such that only 1 thread can write it. Any idea to improve further.

 
Jesper de Jong
Java Cowboy
Sheriff
Posts: 16059
88
Android IntelliJ IDE Java Scala Spring
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
First of all, do you actually have a problem where you have a working piece of software that is running too slow, or are you only anticipating about possible future performance problems?

Most of the time it is very hard to predict performance problems, so it makes not much sense to try to optimize things before you know that there actually is a real performance problem.

Updating and searching in a database "almost every second" should not be a problem; a normal relational database system can easily handle hundreds or thousands of transactions per second (ofcourse it depends on a lot of factors, including what hardware the database is running on, what other software is running on the same machine, etc.).

How you design your database, what indexes and constraints you set on it etc. can also have a big impact on performance, but it's hard to say in general what is and what is not a good idea.

Assuming you don't choose a strategy or algorithm which has obvious performance issues, the best way to investigate performance issues is to let the system run, and use tools such as a profiler to measure how long each part of the program takes to run, and then focus on improving the parts that run the slowest.
 
Tushar Goel
Ranch Hand
Posts: 934
4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
or are you only anticipating about possible future performance problems?


Thanks Jasper, i am only anticipating it.

My database is only having 2 tables having 13 and 12 fields only. I do not have very good idea about database so i am also getting same high level aspect like indexes, cursor etc..

Assuming you don't choose a strategy or algorithm which has obvious performance issues


can you put some light on this? I have not undertaken any data structure classes.
 
Winston Gutkowski
Bartender
Posts: 10575
66
Eclipse IDE Hibernate Ubuntu
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tushar Goel wrote:Thanks Jasper, i am only anticipating it.

Then the best advice I can give is: DON'T.

Write your app and get it working first. Then worry about optimisation if (and only if) you find that you need to.

can you put some light on this? I have not undertaken any data structure classes.

Well, one thing might be to show your db design to someone who knows about these things. I was a DBA for more than 10 years, so if you want to post it here, I'll be happy to comment.

However, one thing I will say is that you appear to be worrying too much about how you're going to this - LinkedBlockingQueue's, Threads, and the like - when what you should be concentrating on is what needs to be done. Get that down - preferably on paper - because all those other implementation decisions are usually interchangeable.

And when you've got the "what", concentrate on writing clear, simple, unoptimized code to action it. That in itself is likely to yield a solution that is fast enough; and even if it doesn't, it will almost certainly make any optimization you need to add much simpler.

You might want to have a look at the WhatNotHow (←click) page for more details.

HIH

Winston
 
Tushar Goel
Ranch Hand
Posts: 934
4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Thanks Winston.. Actually , i implemented the design on paper only. I spent almost 3-4 days on it and then asked question what i thought while working on paper (At my present level i tried my best to implement the same on paper and thought for it).

I will share db design once it is ready. Please give me 2-3 days.
 
Tushar Goel
Ranch Hand
Posts: 934
4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am facing another problem while implementation..

Now i start the server it works fine but when i want to stop it does not stop it as thread was waiting at accept and so if i shutdown the executor service it doesnot come out of the loop. I tried to think like close the server socket which throws socket exception then give shutdown to the executor. This is working but any thing better than this.. sample code is:



stop method as below
 
Tushar Goel
Ranch Hand
Posts: 934
4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

still no able to figure out what other ways other than closing socket i can come out the loop. As cursor is on accept call so if i shutdown the executor it still not checked the flag and keep it there...
 
Tushar Goel
Ranch Hand
Posts: 934
4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Winston,

I have identified the fields which needs to be present in the database. There are 2 tables FMA_alarm_list and FMA_alarm_log. I am attaching screenshot of list table . Log almost have same column and data as list having except data present in this table permanently(as per retention period) but in the list, data will be removed once it get acknowledged by the user on the GUI.

Could you please tell me how to decide the size of the database and also about indexing to improve performance of the database.

I am using sybase ASE 15.
Selection_204.png
[Thumbnail for Selection_204.png]
Selection_205.png
[Thumbnail for Selection_205.png]
 
Narayana Padmanabhuni
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tushar Goel wrote:
or are you only anticipating about possible future performance problems?


Thanks Jasper, i am only anticipating it.

My database is only having 2 tables having 13 and 12 fields only. I do not have very good idea about database so i am also getting same high level aspect like indexes, cursor etc..

Assuming you don't choose a strategy or algorithm which has obvious performance issues


can you put some light on this? I have not undertaken any data structure classes.


ADF
 
Tushar Goel
Ranch Hand
Posts: 934
4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
what is ADF?? i Google this word, do you mean 'Oracle Application development framework' ? i do not have any idea.
 
Winston Gutkowski
Bartender
Posts: 10575
66
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tushar Goel wrote:I have identified the fields which needs to be present in the database. There are 2 tables FMA_alarm_list and FMA_alarm_log. I am attaching screenshot of list table . Log almost have same column and data as list having except data present in this table permanently(as per retention period) but in the list, data will be removed once it get acknowledged by the user on the GUI.

That seems redundant to me. Why not just have a single table with an "acknowledged" field (or date, as you already appear to)? I could see a possible reason for two tables if the "permanent" one is going to get enormous, and you need to retrieve "unacknowledged" entries often; but I suspect that may be something for another day, and you could easily just add an index to make that particular query faster.

However, I reckon that you will need more than one table though...just not those two.

There are a few fields - specifically: Alarm_ID, Record_Type, and Severity - that look like foreign keys to me. For example, rather than validating severity as a piece of text, why not just have a Severity table? Eg:
# - Description
1 - NSA (?)
2 - Minor
3 - Major
4 - Critical
and add a relationship between the two. That way, you'll never be able to add an Alarm without a severity, and the "correctness" check will be done by the database. Same thing with 'Record Type'.

I think I'd need to know a bit more about why you think you need two tables and what this 'Alarm_ID' is all about before I could comment further. In fact, a bit more background about what an 'Alarm' is would also be useful.

HIH

Winston
 
Tushar Goel
Ranch Hand
Posts: 934
4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Winston..

Actually, alarm is a string which i am receiving from the several client. something like this. I underlined some parts which i have to pull out from it..
________________________________________________
*** ALARM ceased 1621 A3/APT "CUTBSC1//G10B//"U 140415 2350
RADIO TRANSMISSION IP GATEWAY FAULT

IPDEV GW
RTIPGPH-13 GW1

END
______________________________________

In this Alarm number = 1621
recordtype = "ceased" or else "alarm"
severity : A3 = Minor (A1- Critical , A2 - Major , A3 Minor , A4 NSA)
Event time: 140415:2350
node: CUTBSC1


Alarm id is automatic increment assigned to all those entries in which ceased keyword is not there and the for those alarms in which ceased keyword is there we need to search in database(there is one to one mapping with alarms and ceased alarms. First alarm comes and if issue is fixed then its ceased comes otherwise it still active). Alarms and ceased alarms having same alarm number , node name , severity. So as per these 3 entries we need to search alarm id in the database and then add same id to the ceased alarm entries. This will help to link alarm and its ceased.

I thought of 2 tables because of following reasons:

1) list table have all the alarms which are not acknowledged and these many alarms will be visible to the user GUI which i need to create as well. By doing this i can have limited number of alarms to be displayed on the GUI. Count can go upto several thousands as well. once user acknowledged it will be removed from this table but will be present in the log table.

2) log table i used because once alarm is acknowledged then user is not care about this. May be we can use to make reports etc later on. So if i user single table then it will take unnecessary user space and everything will be displayed. Second as size grows i can zip/backup data or delete as well as these are those on which some work already done.

3) There are chances that problem take more time to resolve so ceased type alarm will not come even for few days. so if i use single table and if i have to remove/zip alarms for older than n days than this alarm will also be archieved/removed which i dont want. by using separate table i am having a flexibility to remove/archieve without impacting those alarms whose ceased are not come and also not acknowledged.

I thought above points. I may be wrong.

Regarding severity and record type i am getting your point. I should have separate table for them to control things.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!