• Post Reply Bookmark Topic Watch Topic
  • New Topic

How to find a troubleshooter consultant ASAP  RSS feed

 
Sheriff
Posts: 6037
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
My current project ran into a stumbling block. We're having deadlock problems on our application server (Weblogic 7.0 using Oracle 9i). The EJB/DB person on the team is rapidly running out of ideas as to how to fix it. We need to find someone to come in and clear up the problem. It's just this one problem, and my guess is it'll take a few days, most of it spent going over the code.
I'm thinking I can call Oracle and/or BEA (I'm not even sure what departments). I know Sun runs a "911 call" group--although they used to charge $350/hr. As this is an academic project, that might be a bit pricey.
Anyone know of any other people/places to call?
Anyone have experience hiring for something like this?
--Mark
 
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
you should first give some info about your project, such as requirement, framework...
 
Ranch Hand
Posts: 925
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
why not post the stacktrace to the appropriate forum? thats what everyone else does!
have you enough rollback segments? (1st guess)
 
Ranch Hand
Posts: 532
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I used to work for BEA. If the problem is at the app server level, call your rep and get someone from their professional services out there. If you've got some money to burn, call Oracle too and get one of their people out. Oracle has some great tools that their support folk can use to diagnose a DB. I know we're talking $$$ but sometimes it's the only way to fix the problem.
 
Mark Herschberg
Sheriff
Posts: 6037
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Simon Lee:
why not post the stacktrace to the appropriate forum? thats what everyone else does!
have you enough rollback segments? (1st guess)

I've done that often enough with other problems, but I think the scope of this problems prevents it from being practical over the internet. We're running into deadlock issues. This means different parts of our code is grabbing locks. The stack trace will only show you the code that didn't get the lock; it won't show the 5 other classes that also content for it. now we could show you those 5 other classes, too, but if those were the only ones, then we'd be able to figure it ourselves. I think the problem is that we're missing someone who is grabbing the lock (some class or method or something).
But for those who are interested, my colleague did post a summary of the problem here.
--Mark
 
Jim Baiter
Ranch Hand
Posts: 532
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yeah judging from his post you should probably only use the $$$ consultant as a last resort. Try going to BEA NEWSGROUPS I copied the post there. Some of those folks manning that group are really sharp.
[ April 04, 2003: Message edited by: Jim Baiter ]
 
Ranch Hand
Posts: 42
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
if you can reproduce the scenario, you can try to use some profiling tool, such as OptimizeIt Thread Debugger, to get a real-time view of threads
 
Ranch Hand
Posts: 346
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Mark Herschberg:

I've done that often enough with other problems, but I think the scope of this problems prevents it from being practical over the internet. We're running into deadlock issues. This means different parts of our code is grabbing locks. The stack trace will only show you the code that didn't get the lock; it won't show the 5 other classes that also content for it. now we could show you those 5 other classes, too, but if those were the only ones, then we'd be able to figure it ourselves. I think the problem is that we're missing someone who is grabbing the lock (some class or method or something).
But for those who are interested, my colleague did post a summary of the problem here.
--Mark


Mark, although service and support is pricely I think once one fo the slaes engineers finds out you are academic that they might lend a hand at no rate charged..as they generally want their systems view in a good light..
I am not saying that its their fault I am just saying if you play on their motivations a little bit you might get the answer at the academic price
 
Ranch Hand
Posts: 2712
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Fred Grott:
although service and support is pricely I think once one of the sales engineers finds out you are academic that they might lend a hand at no rate charged

Not likely.
 
Jim Baiter
Ranch Hand
Posts: 532
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yeah, only presales would.
 
Greenhorn
Posts: 27
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The next time the application deadlocks, here are some things to try to determine if the deadlock is a result of a synchronizing lock in the java code or a result of a row lock in the database.
Generate a thread dump for all threads in the jvm.
For unix machines: kill -QUIT process_id
For windows machines: CTRL+BREAK in the console window.
Look for threads in the "wait" state, and check out the stacktrace for those threads.
To identify which threads are in the wait state, read here:
http://developer.java.sun.com/developer/technicalArticles/Programming/Stacktrace/
If there are multiple threads in the wait state waiting for the same object/lock, that could point you to the code that is deadlocking
or which finder methods are being called.
View the sql statements currently being run against oracle. I don't know the exact sql, but the v$sqltext and v$session tables will be of use. Do a search in deja.com for some usable sql. Depending on what sql statements you see and
how many occurances of the same sql statement you see, that will give you some hints into which tables/rows might be causing the deadlock.
http://groups.google.com/groups?selm=59ccou%24gqc%40atglab10.atglab.bls.com&output=gplain
select a.sid, a.username, b.sql_text from v$session a,v$sqltext b
where a.sql_address = b.address and a.sql_hash_value=b.hash_value
and a.username is not null and a.status = 'ACTIVE' and a.user# != uid
order by a.sid, b.piece;
Hope some of this helps you identify the deadlocking problem.
Eddy
 
SJ Adnams
Ranch Hand
Posts: 925
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
well IMO, TRANSACTION_READ_COMMITTED_FOR_UPDATE means your ejbs are doing 'SELECT FOR UPDATE' queries. which I may or may not be correct for your application.
you also say that the table is locked? do you mean there are row level locks or table level?
And why all the seperate queries? try bundling them up into one update statement rather than multiple queries - that way you dont need to do select for update.
 
Mark Herschberg
Sheriff
Posts: 6037
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Over the weekend my colleague worked to minimize the amount of DB access. He seems to have mostly solved the problem and applying the solution to the final areas of the system should work, so I think we might be ok.
Thanks for all the advice.
(For continued technical discussion--I'm still learning about transactions isolation levels with EJBs--see the technical thread here.
--Mark
 
Saloon Keeper
Posts: 18800
74
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Well, by cracky, once upon a time back before every job required 15 years Java experience, 12 of Oracle 9i, 17 of MQSI (rant, rant, rant for another 21 unlikely job skills) I used to get paid for resolving problems like that.
You'd be amazed at what you can learn just by wrapping the chokepoints with simple tracing statements.
Of course, the Heisenberg principle can sometimes get in the way. And/or you may be working within an environment where I/O can't be done. For cases like that you can often get what you need by stuffing info in a buffer for output by a trace-output thread. Just make sure that the interlocks between the trace loggers and the printing thread don't add yet another set of deadlocks!
Deadlocks are as old as computers, or darn near so. The only sure way of avoiding them is to design your resource guards in such a way that no two resources are interdependent. There's a number of tricks that can be used to that end.
[ April 08, 2003: Message edited by: Tim Holloway ]
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!