• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Paul Clapham
  • Jeanne Boyarsky
  • Junilu Lacar
  • Henry Wong
Sheriffs:
  • Ron McLeod
  • Devaka Cooray
  • Tim Cooke
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Frits Walraven
  • Tim Holloway
  • Carey Brown
Bartenders:
  • Piet Souris
  • salvin francis
  • fred rosenberger

Issue with ActiveMQ and springboot app

 
Ranch Hand
Posts: 198
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have a spring boot app which is deployed on tomcat on RHEL server which is trying to connect to ActiveMQ and it keeps on throwing following error in tomcat logs. Things were working fine till yesterday and today morning.

My application.properties of springboot app has following configuration:

 


   

 
Sheriff
Posts: 3122
459
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Jack Tauson wrote:


Do you have ActiveMQ running on the local host?

You can see if there is anything listen on port 61616 with the command line: netstat -natp | grep 61616
 
Jack Tauson
Ranch Hand
Posts: 198
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Ron McLeod wrote:

Jack Tauson wrote:


Do you have ActiveMQ running on the local host?

You can see if there is anything listen on port 61616 with the command line: netstat -natp | grep 61616



It was running before but not right now for some reason after I stopped it. How do I start it again since it's not starting? Here are few thing that I did:

[activemq@myservername servers]$ cd apache-activemq-5.15.11/

[activemq@myservername apache-activemq-5.15.11]$ bin/activemq start
INFO: Loading '/web/servers/apache-activemq-5.15.11//bin/env'
INFO: Using java '/bin/java'
INFO: Starting - inspect logfiles specified in logging.properties and log4j.properties to get details
INFO: pidfile created : '/web/servers/apache-activemq-5.15.11//data/activemq.pid' (pid '6996')
[activemq@myservername apache-activemq-5.15.11]$ bin/activemq start
INFO: Loading '/web/servers/apache-activemq-5.15.11//bin/env'
INFO: Using java '/bin/java'
INFO: Process with pid '6996' is already running

[activemq@myservername apache-activemq-5.15.11]$ bin/activemq start
INFO: Loading '/web/servers/apache-activemq-5.15.11//bin/env'
INFO: Using java '/bin/java'
INFO: Process with pid '6996' is already running

[activemq@myservername apache-activemq-5.15.11]$ bin/activemq stop
INFO: Loading '/web/servers/apache-activemq-5.15.11//bin/env'
INFO: Using java '/bin/java'
INFO: Waiting at least 30 seconds for regular process termination of pid '6996' :
Java Runtime: Oracle Corporation 1.8.0_242 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-1.el7.x86_64/jre
 Heap sizes: current=62976k  free=61991k  max=932352k
   JVM args: -Xms64M -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/web/servers/apache-activemq-5.15.11//conf/login.config -Dactivemq.classpath=/web/servers/apache-activemq-5.15.11//conf:/web/servers/apache-activemq-5.15.11//../lib/: -Dactivemq.home=/web/servers/apache-activemq-5.15.11/ -Dactivemq.base=/web/servers/apache-activemq-5.15.11/ -Dactivemq.conf=/web/servers/apache-activemq-5.15.11//conf -Dactivemq.data=/web/servers/apache-activemq-5.15.11//data
Extensions classpath:
 [/web/servers/apache-activemq-5.15.11/lib,/web/servers/apache-activemq-5.15.11/lib/camel,/web/servers/apache-activemq-5.15.11/lib/optional,/web/servers/apache-activemq-5.15.11/lib/web,/web/servers/apache-activemq-5.15.11/lib/extra]
ACTIVEMQ_HOME: /web/servers/apache-activemq-5.15.11
ACTIVEMQ_BASE: /web/servers/apache-activemq-5.15.11
ACTIVEMQ_CONF: /web/servers/apache-activemq-5.15.11/conf
ACTIVEMQ_DATA: /web/servers/apache-activemq-5.15.11/data
Connecting to pid: 6996
INFO: failed to resolve jmxUrl for pid:6996, using default JMX url
Connecting to JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
.INFO: Broker not available at: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
..............................
INFO: Regular shutdown not successful,  sending SIGKILL to process
INFO: sending SIGKILL to pid '6996'

[activemq@myservername apache-activemq-5.15.11]$



So whenever I check whether its running or not using command  "bin/activemq start" I keep on getting "INFO: Process with pid '6996' is already running" as shown above. However, I am no longer seeing the admin console and also the command you mentioned "netstat -natp | grep 61616" is not returning any results. What might be the issue?



On my other server where it runs properly, it does show the following after running "netstat -natp | grep 61616"


 
Ron McLeod
Sheriff
Posts: 3122
459
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
This is what I see ...

It was started when you tried the first time:

Then, while it was already running, you tried and start it again, and you are told that in-fact, it is already running:

Same thing again:

 
Jack Tauson
Ranch Hand
Posts: 198
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Ron McLeod wrote:This is what I see ...

It was started when you tried the first time:

Then, while it was already running, you tried and start it again, and you are told that in-fact, it is already running:

Same thing again:



Right. In spite of it telling me that it's already running, I am wondering why I'm not able to see it listening on port 61616?

The following command doesn't shows anything :



Whereas, on the other RHEL server, where it is running, I can see the following:


 
Ron McLeod
Sheriff
Posts: 3122
459
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Jack Tauson wrote:Right. In spite of it telling me that it's already running, I am wondering why I'm not able to see it listening on port 61616?

The following command doesn't shows anything :
Whereas, on the other RHEL server, where it is running, I can see the following:


Well .. my first guess would be that it is not configured to use port 61616.  Check your configuration to verify (I think it is an attribute of the transportConnector element in the activemq.xml file).
 
Jack Tauson
Ranch Hand
Posts: 198
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Ron McLeod wrote:

Jack Tauson wrote:Right. In spite of it telling me that it's already running, I am wondering why I'm not able to see it listening on port 61616?

The following command doesn't shows anything :
Whereas, on the other RHEL server, where it is running, I can see the following:


Well .. my first guess would be that it is not configured to use port 61616.  Check your configuration to verify (I think it is an attribute of the transportConnector element in the activemq.xml file).



I compared activemq.xml of both the server and they both are identical.

I noticed the following things though on both the server:

At myservername:





Whereas, in other server, it connects fine as shown below:



Could this be an issue?
 
Ron McLeod
Sheriff
Posts: 3122
459
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Jack Tauson wrote:


Do you have a web server (Tomcat?) running on the local host and listening on port 8161?  What does netstat show?
 
Ron McLeod
Sheriff
Posts: 3122
459
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Ron McLeod wrote:Well .. my first guess would be that it is not configured to use port 61616.  Check your configuration to verify (I think it is an attribute of the transportConnector element in the activemq.xml file).


Jack Tauson wrote:I compared activemq.xml of both the server and they both are identical.


Maybe it's picking-up its configuration from a different configuration file?

Can you enable detailed logging on ActiveMQ and see what configuration it being used when the broker starts?
 
Jack Tauson
Ranch Hand
Posts: 198
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Ron McLeod wrote:

Jack Tauson wrote:


Do you have a web server (Tomcat?) running on the local host and listening on port 8161?  What does netstat show?



Yes, I do have a tomcat running and this is what it shows with the server:



and



Is tomcat supposed to listnen on port 8161 & 61616. I have removed my apps from tomcat since it was messing up with the starting of other apps with the following error and was not letting other apps to start.




On the server where everything works fine, it shows like this:



and




 
Jack Tauson
Ranch Hand
Posts: 198
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Ron McLeod wrote:

Ron McLeod wrote:Well .. my first guess would be that it is not configured to use port 61616.  Check your configuration to verify (I think it is an attribute of the transportConnector element in the activemq.xml file).


Jack Tauson wrote:I compared activemq.xml of both the server and they both are identical.


Maybe it's picking-up its configuration from a different configuration file?

Can you enable detailed logging on ActiveMQ and see what configuration it being used when the broker starts?



How to enable detail logging?

I haveactivemq.log inside  /web/servers/apache-activemq-5.15.11/data. Is this enough?
 
Ron McLeod
Sheriff
Posts: 3122
459
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Jack Tauson wrote:How to enable detail logging?

I don't know - what logging framwork are you using?

Jack Tauson wrote:I haveactivemq.log inside  /web/servers/apache-activemq-5.15.11/data. Is this enough?

I don't know that either - have you looked at the log to see if there are any clues about what might be going on?
 
Ron McLeod
Sheriff
Posts: 3122
459
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Jack Tauson wrote:Is tomcat supposed to listnen on port 8161 & 61616


Tomcat and ActiveMQ listen on the ports as configured.  By default Tomcat will listen for HTTP traffic on port 8080; ActiveMQ's default port is 61616.
 
Jack Tauson
Ranch Hand
Posts: 198
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Ron McLeod wrote:

Jack Tauson wrote:How to enable detail logging?

I don't know - what logging framwork are you using?

Jack Tauson wrote:I haveactivemq.log inside  /web/servers/apache-activemq-5.15.11/data. Is this enough?

I don't know that either - have you looked at the log to see if there are any clues about what might be going on?



I enabled it like this "log4j.logger.org.apache.activemq=DEBUG" (https://activemq.apache.org/how-do-i-enable-debug-logging) but not sure if it made any difference.

When I went inside activemq.log, this is what I see for today. I stopped and started the server and did nothing else:

 
Jack Tauson
Ranch Hand
Posts: 198
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Ron McLeod wrote:

Jack Tauson wrote:Is tomcat supposed to listnen on port 8161 & 61616


Tomcat and ActiveMQ listen on the ports as configured.  By default Tomcat will listen for HTTP traffic on port 8080; ActiveMQ's default port is 61616.



Yeah, and as soon as I start activemq, I would imagine that port 61616 should show up with the command netstat -a | grep 61616
 
Marshal
Posts: 25594
69
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Jack Tauson wrote:Yeah, and as soon as I start activemq, I would imagine that port 61616 should show up with the command netstat -a | grep 61616



And if it doesn't, there are two possibilities for that. One is that it's configured to start on some other port, and the other reason is that it failed to start for some reason other than something is already using port 61616. So you need to consider those, for a start.
 
Ron McLeod
Sheriff
Posts: 3122
459
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Jack Tauson wrote:... When I went inside activemq.log, this is what I see for today. I stopped and started the server and did nothing else


This message seems to explain the problem:
ActiveMQ Master/Slave behavior
 
Jack Tauson
Ranch Hand
Posts: 198
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Paul Clapham wrote:

Jack Tauson wrote:Yeah, and as soon as I start activemq, I would imagine that port 61616 should show up with the command netstat -a | grep 61616



And if it doesn't, there are two possibilities for that. One is that it's configured to start on some other port, and the other reason is that it failed to start for some reason other than something is already using port 61616. So you need to consider those, for a start.



Instead of figuring this out, I went ahead and asked my server team to install it fresh from the tar file. I'm now seeing the following :



However, I'm still not sure why it's picking up "tcp6" instead of "tcp" above.

And on my other server, I see two more items listed as shown below (two ESTABLISHED ones), which are not showing above for some reason:


 
Ron McLeod
Sheriff
Posts: 3122
459
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Jack Tauson wrote:Instead of figuring this out, I went ahead and asked my server team to install it fresh from the tar file.


Personally, I would have kept digging so it is happened again I would know what to do.  In your case, it could have just been a stale lock file(?).


Jack Tauson wrote:
However, I'm still not sure why it's picking up "tcp6" instead of "tcp" above.


That server appears to be configured for IPv6 (as well as IPv4).
 
Jack Tauson
Ranch Hand
Posts: 198
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Personally, I would have kept digging so it is happened again I would know what to do.  In your case, it could have just been a stale lock file(?).


I agree. Not sure about the state lock file. I never did anything related to that so not sure why it was showing like that in previous installation.

I checked your post - https://coderanch.com/t/731494/frameworks/ActiveMQ-springboot-app#3403572

Someone mentioned in that post that it's normal. But again, I didn't do anything related to kahab stuff. I don't know what that even means.



That server appears to be configured for IPv6 (as well as IPv4).


Right. I hope it always shows tcp6 protocol while checking netstat ( I mean after several shutdown and restarts ) because for applications deployed on that server, I would have to configure my application.properties file  with "tcp6" instead of "tcp".
 
Ron McLeod
Sheriff
Posts: 3122
459
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Jack Tauson wrote:

Ron McLeod wrote:That server appears to be configured for IPv6 (as well as IPv4).


Right. I hope it always shows tcp6 protocol while checking netstat ( I mean after several shutdown and restarts ) because for applications deployed on that server, I would have to configure my application.properties file  with "tcp6" instead of "tcp".


I would assume that it is listening for both IPv4 and IPv6 connections, so no need to change.

You can verify by either running your application as-is, or use something like curl to verify it is possible to make a connection with IPv4.

For example (the data transfer information is unimportant):
 
Jack Tauson
Ranch Hand
Posts: 198
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

I would assume that it is listening for both IPv4 and IPv6 connections, so no need to change.



But if my application tries to connect over "tcp" and on the server, it is listening over "tcp6", wouldn't it fail?


You can verify by either running your application as-is, or use something like curl to verify it is possible to make a connection with IPv4.

For example (the data transfer information is unimportant):



Sorry I didn't understand this. Does the above information shows that the server is configured for IPV4 and 6?
 
Ron McLeod
Sheriff
Posts: 3122
459
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Jack Tauson wrote:Sorry I didn't understand this. Does the above information shows that the server is configured for IPV4 and 6?


In the first example, a connection to the specified port is able to be successfully established using IPv4.  This is what you want to verify

In the second example, a connection to the specified port fails with: "couldn't connect to host" because there is nothing listening on the port.  I included this so you could see what a failed connection attempt with curl looks like.

Even though curl isn't really intended to be used in this way, I was suggesting to use it to determine if you can make an IPv4 connection because it is available on almost all distributions and I have no idea what other tools might be available to you.  If it is more confusing than helpful, then just run your application configured for IPv4 and verify that it can connect to the broker.
 
Jack Tauson
Ranch Hand
Posts: 198
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Ron McLeod wrote:

Jack Tauson wrote:Sorry I didn't understand this. Does the above information shows that the server is configured for IPV4 and 6?


In the first example, a connection to the specified port is able to be successfully established using IPv4.  This is what you want to verify

In the second example, a connection to the specified port fails with: "couldn't connect to host" because there is nothing listening on the port.  I included this so you could see what a failed connection attempt with curl looks like.

Even though curl isn't really intended to be used in this way, I was suggesting to use it to determine if you can make an IPv4 connection because it is available on almost all distributions and I have no idea what other tools might be available to you.  If it is more confusing than helpful, then just run your application configured for IPv4 and verify that it can connect to the broker.



Thanks. Could you also answer this question that I asked above :

But if my application tries to connect over "tcp" and on the server, it is listening over "tcp6", wouldn't it fail?


Thanks very much !
 
Jack Tauson
Ranch Hand
Posts: 198
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Ron McLeod wrote:

Jack Tauson wrote:Sorry I didn't understand this. Does the above information shows that the server is configured for IPV4 and 6?


In the first example, a connection to the specified port is able to be successfully established using IPv4.  This is what you want to verify

In the second example, a connection to the specified port fails with: "couldn't connect to host" because there is nothing listening on the port.  I included this so you could see what a failed connection attempt with curl looks like.

Even though curl isn't really intended to be used in this way, I was suggesting to use it to determine if you can make an IPv4 connection because it is available on almost all distributions and I have no idea what other tools might be available to you.  If it is more confusing than helpful, then just run your application configured for IPv4 and verify that it can connect to the broker.



I saw the following:

[activemq@myservername1 apache-activemq-5.15.11]$ curl -4 localhost:61616 > /dev/null
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
 0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (52) Empty reply from server



Does this mean that even though netstat is showing "tcp6" I should not change it to "tcp6" in my application.properties and keep it as "tcp" only?
 
Ron McLeod
Sheriff
Posts: 3122
459
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Jack Tauson wrote:Thanks. Could you also answer this question that I asked above :
But if my application tries to connect over "tcp" and on the server, it is listening over "tcp6", wouldn't it fail?

Thanks very much !



I don't think I can answer that question any better than I already have.

Ron McLeod wrote:I would assume that it is listening for both IPv4 and IPv6 connections, so no need to change.

You can verify by either running your application as-is, or use something like curl to verify it is possible to make a connection with IPv4.


Since you have access to the platform - give it a try and see what happens.
 
Jack Tauson
Ranch Hand
Posts: 198
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator


I don't think I can answer that question any better than I already have.



Ok, and if I want to check connection establishment over IPv6, then I would just use number "6" instead of "4" in the curl request you used for testing IPv4?

 
Ron McLeod
Sheriff
Posts: 3122
459
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Jack Tauson wrote:Ok, and if I want to check connection establishment over IPv6, then I would just use number "6" instead of "4" in the curl request you used for testing IPv4?


Yes.
 
It was the best of times. It was the worst of times. It was a tiny ad.
Devious Experiments for a Truly Passive Greenhouse!
https://www.kickstarter.com/projects/paulwheaton/greenhouse-1
    Bookmark Topic Watch Topic
  • New Topic