Issue with WeblogicServer

[Piyush Kumar Sharma]

Hi All

I am getting below mantioned error while stoping the webserver

Failed to connect to t3://MYSERVER:8098: Destination unreachable; nested exception is:
java.net.SocketException: Broken pipe; No available router to destination

Please help me out.....

[Amit Kumar.]

We get this when there are any Connection Reset Errors. Check your connection pool and data source.

[Amit Ghorpade]

"Amit Kumar Hcl " please check your private messages for an important administrative matter. You can check them by clicking the My Private Messages link above.

[Marcos Maia]

Piyush Kumar Sharma wrote:Hi All

I am getting below mantioned error while stoping the webserver

Failed to connect to t3://MYSERVER:8098: Destination unreachable; nested exception is:
java.net.SocketException: Broken pipe; No available router to destination

Please help me out.....

Usually that's exactly what means it can't resolve MYSERVER:8098 as a weblogic server instance(i.e- there's no such server or is not reacheble) Please post complete stack trace if you need further help on this.

[]s

[Maurizio Villa]

Hi All,
please, can you support me on the same issue?
I have a problem with a customer of mine.
We're facing BROKEN PIPE issue, on Weblogic ALSB 2.5.
This is teh error: java.net.SocketException: Broken pipe

This is the stack:

<Mar 17, 2009 9:48:03 AM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in HttpInboundMessageContext.close:
java.net.SocketException: Write failed: Broken pipe
<Mar 17, 2009 9:48:41 AM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in
HttpInboundMessageContext.close: java.net.SocketException: Write failed: Broken
pipe
<Mar 17, 2009 9:48:42 AM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in HttpInboundMessageContext.close: java.net.SocketException: Write
failed: Broken pipe


.....

Following other details of the error:

<Mar 23, 2009 12:30:59 PM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in
HttpTransportServlet.service: java.net.SocketException: Write failed: Broken
pipe
java.net.SocketException: Write failed: Broken pipe
at jrockit.net.SocketNativeIO.writeBytesPinned(I[BII)V(Native Method)
at jrockit.net.SocketNativeIO.socketWrite(Ljava.io.FileDescriptor;[BII)V(Unknown
Source)
at java.net.SocketOutputStream.socketWrite0(Ljava.io.FileDescriptor;[BII)V(SocketOu
tputStream.java:???)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
Truncated. see log file for complete stacktrace
>
<Mar 23, 2009 12:31:00 PM CET> <Error> <HTTP> <BEA-101017>
<[weblogic.servlet.internal.WebAppServletContext@12affa8 - appName: 'XBus
Kernel', name: '/Portal/PX_CommonUtilityService', context-path:
'/Portal/PX_CommonUtilityService'] Root cause of
ServletException.
java.net.SocketException: Write failed: Broken pipe
at jrockit.net.SocketNativeIO.writeBytesPinned(I[BII)V(Native Method)
at
jrockit.net.SocketNativeIO.socketWrite(Ljava.io.FileDescriptor;[BII)V(Unknown
Source)
at java.net.SocketOutputStream.socketWrite0(Ljava.io.FileDescriptor;[BII)V(SocketOu
tputStream.java:???)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
Truncated. see log file for complete stacktrace


The issue is not dependent to the service in use.
The problem has a great priority due to the fact is in production env. and is causing loss of service.

Thanks in advance to everyone.
Maury

[Marcos Maia]

<Mar 17, 2009 9:48:41 AM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in
HttpInboundMessageContext.close: java.net.SocketException: Write failed: Broken
pipe

Hi,

My first impression on this error is problably:

1) The client access the application
2) Some resource(DB, JCA connection, etc) is slow or some synchronized block in application is overloading the server and it takes too long to respond.
3) The client closes browser before receive the response and maybe try to start all over...
4) When the server tries to get back to client it's gone for that http cycle and that causes the Broken pipe... HttpInboundMessageContext.close....


My suggestion is for you to take some thread dumps when you realize the server is becaming overloaded and than you can find the problem(external resource or application dead lock).

[]s

P.S - To take a thread dump in Unix environments find the jvm process for a instance than use "kill -3 PID"




>

[Maurizio Villa]

Thanks Marcos,
please look this attachments, we're encountering queuing in thread pool unjustified.
have a look to this graph.

any hints?
regards.

[Marcos Maia]

That's it, you need few thread dumps from server in the moment the Queue and Hogging threads are happening.
In an interval from 13:27 to 13:34 minutes(or in a moment like this) you take some thread dumps, my sugestion is 2 or 3 per minute and than with this thread dump in hands you'll be able to find out exactly what's causing the queue and Hogging threads in your app.
Also problably you have a high number configured to consider threads as stuck(problably the default 600s - 10min) so you don't have stuck threads..

Take the thread dumps and you'll find out what's causing this problem(external resources or sync block code).
Are you using Jrockit JVM?

[]s

[Maurizio Villa]

Hi Marcos,
yes, JVM is JRockit -> Java Version: 1.5.0_06
The problem to get a full thread dump is to be so fast to intervene exactly when isssue is happening.

We also suspect some limit on DB, perhaps waits or something else.
I dont' know if it is a reasonable feeling.

Thanks for any kind of hints for addressing this issue.
Regards.
Maury

[Marcos Maia]

Well,

You don't need to do gess it, you can configure a notification to take diagnostic images(TDumps included), send snmp traps to notify someone who could run a script to take dumps, or automate TDumps in some flag situation, it's all about configuring or writting some code. you may also use WLST to take these dumps in some specific situation.

Take a look at the following link to read about notifications, I also have some WLST scripts who take dumps in some situation, if you wanna go this way let me know and I'll post some examples.

http://edocs.bea.com/wls/docs103/wldf_configuring/config_notifications.html

This is latest release but It's been around since 9.2 as far as I can remember.

[]s

[Maurizio Villa]

Hi M.,
thanks for suggestion, i really appreciate if you may share some of your WLST scripts, or send them to me via emai at
this address: [e-mail] so that i could perform some tests.
thanks in advance, i'll let you know.
regards.
Maurizio

[Marcos Maia]

Hi,

follows a sample script you can use as starting point, explanations follows. Unfotunatelly my primary language is Portuguese and you'll find comments on this language on this script.

    
from java.util 	import Properties
from java.io 	import FileInputStream
from java.io 	import File
from java.util 	import Enumeration

from string import split
#============CARREGA DADOS ARQUIVO PROPRIEDADES====================================
#arquivo de propriedades a ser carregado
localizacaoArquivoPropriedadesConfiguracao = 'servidores.properties'

myProps = Properties()

#carrega arquivo de propriedades
myProps.load(FileInputStream(File(localizacaoArquivoPropriedadesConfiguracao)))

#========SEPARA ELEMENTOS SERVIDOR DA LISTA DE PROPRIEDADES CARREGADA==============
#array para armazenar as chaves dos servidores definidos no arquivo de propriedades.
servidores = []

propertyNames = myProps.propertyNames()

while propertyNames.hasMoreElements():
    #carrega os nomes de todas as chaves carregadas do arquivo de propriedades
    chave = propertyNames.nextElement()
    #recupera tipo da chave para verificar se é um servidor se for adiciona no array de servidores.
    elemento = split(chave, '_')
    #checa se elemento carregado do arquivo de properties é um servidor e adiciona na lista de servidores
    if elemento[0] == 'SERVIDOR':
        servidores.append(chave)
        
print 'A seguinte lista de servidores será monitorada....'
for a in servidores:
    print 'Nome do servidor ==>> ' + myProps.get(a)
#====FIM SEPARA ELEMENTOS SERVIDOR DA LISTA DE PROPRIEDADES CARREGADA==============
#==========================CRIACAO DE INSTANCIAS===================================
print 'Iniciando monitoracao dos servidores........'


portaHttps = myProps.get('PORTA_HTTPS_SERVIDORES')
usuario = myProps.get('USUARIO_ADMIN')
senha = myProps.get('SENHA_ADMIN')
parametroTDump = myProps.get('PARAMETRO_THREAD_DUMP')

def monitoraServidores() :
    for servidor in servidores:
        nomeServidor = myProps.get(servidor)
        ipServidor = myProps.get('IP_'+servidor)
        print('\n\n\n===============================INICIANDO MONITORAMENTO SERVIDOR ==>> ' + nomeServidor + '==============================')
        print('=============================================================================================================================')
        try:
            connect(usuario, senha, 't3s://' + ipServidor + ':443')
            serverRuntime()
            print('Checando a memoria...')
            cd('serverRuntime:/JVMRuntime/' + nomeServidor)
            jvmDescription = cmo.getJVMDescription()
            memAlocada = cmo.getHeapSizeCurrent() / 1048576
            porcentagemMemDisponivel = cmo.getHeapFreePercent()
            memDisponivel = cmo.getHeapFreeCurrent()/1048576
            print ('=======INFORMACOES DE JVM==========')
            print('Descricao da  JVM :' + jvmDescription)
            print('Memoria alocada em MB: ' + str(memAlocada))
            print('Percentual de memoria disponivel: ' + str(porcentagemMemDisponivel))
            print('Total de memoria disponivel em MB: ' + str(memDisponivel))
            print('\n')
            
            cd('serverRuntime:/')
            quantidadeSocketsAbertos = cmo.getOpenSocketsCurrentCount();
            print('Quantidade de sockets abertos: ' + str(quantidadeSocketsAbertos));
        
            cd('serverRuntime:/ThreadPoolRuntime/ThreadPoolRuntime/')
            quantidadeRequisicoesAtendidas = cmo.getCompletedRequestCount()
            statusServidor = cmo.getHealthState()
            hoggingThreads = cmo.getHoggingThreadCount()
            totalThreads = cmo.getExecuteThreadTotalCount()
            threadsLivres = cmo.getExecuteThreadIdleCount()
            requisicoesPendentes = cmo.getPendingUserRequestCount()
            tamanhoFila = cmo.getQueueLength()
            requisicoesPorSegundo = cmo.getThroughput()
            if(hoggingThreads > int(parametroTDump)) :
                threadDump(writeToFile='true', fileName='ThreadDump_' + nomeServidor + '.txt')
            print('========INFORMACOES DE THREADS======')
            print('Status do servidor: ' + str(statusServidor))
            print('Total de requisicoes de usuarios atendidas: ' + str(quantidadeRequisicoesAtendidas))
            print('Total de threads no servidor: ' + str(totalThreads))
            print('Total de threads livres: ' + str(threadsLivres))
            print('Total de threads com delay na resposta: ' + str(hoggingThreads))
            print('Total de requisicoes de usuarios pendentes: ' + str(requisicoesPendentes))
            print('Tamanho da fila aguardando por threads: ' + str(tamanhoFila))
            print('Media de requisicoes por segundo(Throughput): ' +str(requisicoesPorSegundo))
            disconnect()
            print('\n\n\n===============================TERMINANDO MONITORAMENTO SERVIDOR ==>> ' + nomeServidor + '==============================')
            print('==============================================================================================================================')
            print('==============================================================================================================================')
        except:
            print('Erro ao tirar estatísticas do servidor + ' + nomeServidor + ' ignorando este servidor, indo para o próximo!!!!');

monitoraServidores()
exit()

First part of this script is used to load a properties file(follows) where you configure servers you wanto to monitor and load wladmin url, pwd, etc...

The function is the one who does the job.. the fragment:

    
 if(hoggingThreads > int(parametroTDump)) :
                threadDump(writeToFile='true', fileName='ThreadDump_' + nomeServidor + '.txt')

Test how many hogging threads the server has and take a thread dump write to the file with pattern ThreadDump_SERVERNAME if the conditions is met.....

In your case I would modify this script to have a fuction wich would take for example one tdump each 20sec for a few cycles in case the hogging threads are > than certain number..

I would also modify it to keep conected to a desired server and each 1 min do a check on it..

This would do the job.

Follows the sample proerties:

    
#este arquivo de propriedades é utilizado para configurar os 
#servidores em um ambiente. Todas as chaves devem ter o formato
#do nome utilizando sempre _ (underscore) pois é o que define a 
#busca dos nomes dos elementos e separa durante a configuarção 
#no script criaInstancias.py
USUARIO_ADMIN=user
SENHA_ADMIN=pwd

PORTA_HTTPS_SERVIDORES=443

#parametro que configura com que quantidade de threads com delay o thread dump deve ser tirado para o servidor.
PARAMETRO_THREAD_DUMP=10

#Dados dos servidores a serem monitorados pelo script
SERVIDOR_1=server1
IP_SERVIDOR_1=xxxxxx

SERVIDOR_2=server2
IP_SERVIDOR_2=xxxxxxx

SERVIDOR_3=server3
IP_SERVIDOR_3=xxxxxx

SERVIDOR_4=server4
IP_SERVIDOR_4=xxxxxx

.... you may add as many servers as you want here...

The final consideration is the weakness of this approach which is related to servers being too overloaded in such case maybe you won't be able to take thread dumps or connect to the server(doesn't looks like your case and is a rare situation anyway).

[]s

[Maurizio Villa]

Thanks a lot.

I'll try them asap.

thanks for the moment.

As per your update thereupon the issue of broken pipe.
We saw even if we have stand by thread, when bea consumes all idle threads it starts to enque requests without using alla vailable stand by threads.
have you a concrete idea for this behaviour and the difference between idle and stand by status?
thanks.
regards

[Marcos Maia]

That's really strange to me! Can't remember from top of my mind what's the difference but as far as I remember Stand By threads are suppose to get picked to serve client requests.. really confused by this information.. Never seen this behavior.. I'll take a look and let you know if I find out something about this. It would be nice If you share your progress on that as well..

regards,
Marcos Maia.

Maurizio Villa wrote:Thanks a lot.

I'll try them asap.

thanks for the moment.

As per your update thereupon the issue of broken pipe.
We saw even if we have stand by thread, when bea consumes all idle threads it starts to enque requests without using alla vailable stand by threads.
have you a concrete idea for this behaviour and the difference between idle and stand by status?
thanks.
regards

[Pavan BhavaniShekhar]

Dear friends

To giving more flexibility to understand the ThreadPool Monitoring script using WLST when we use default Self-tune Work Manager.
I had renovated the script with English strings, Comments

Check it out!!

http://wlstbyexamples.blogspot.com/2009/06/self-tuned-thread-pool-count.html
suggest your needs and comments on the script.