This week's book giveaway is in the Reactive Progamming forum.
We're giving away four copies of Reactive Streams in Java: Concurrency with RxJava, Reactor, and Akka Streams and have Adam Davis on-line!
See this thread for details.
Win a copy of Reactive Streams in Java: Concurrency with RxJava, Reactor, and Akka Streams this week in the Reactive Progamming forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Liutauras Vilda
  • Junilu Lacar
  • Jeanne Boyarsky
  • Bear Bibeault
Sheriffs:
  • Knute Snortum
  • Tim Cooke
  • Devaka Cooray
Saloon Keepers:
  • Ron McLeod
  • Stephan van Hulst
  • Tim Moores
  • Tim Holloway
  • Carey Brown
Bartenders:
  • Piet Souris
  • Frits Walraven
  • Ganesh Patekar

Apache Seems to Have Gone Deaf

 
Ranch Hand
Posts: 200
Eclipse IDE Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
So our test server was running Apache with Shibboleth and Tomcat. (For running Liferay). I had Web SSO set up to log me into the test Liferay and all was peachy...

Until the server crashed and got restarted.

I don't know what caused the crash but now Apache seems to be having issues. I've shut down Tomcat and Shibboleth to test just the Apache by itself and nothing. Unable to connect. Tried getting a response from Apache using the command line:

telnet localhost 80

and connected, so I sent the command

HEAD / HTML/1.0

and got this back:

HTTP/1.1 500 Internal Server Error
Date: Tue, 01 Nov 2011 18:11:49 GMT
Expires: 01-Jan-1997 12:00:00 GMT
Cache-Control: private,no-store,no-cache
Content-Length: 960
Connection: close
Content-Type: text/html; charset=UTF-8

Now, I know the machine has the appropriate ports open because I was able to access Liferay through Tomcat when Tomcat was running... So the machine is live, it's listening on port 80, but it seems Apache has decided to sit out. I've restarted httpd with no effect.

Looking at the error logs, I'm not finding any errors of any kind. Am I missing something stupidly obvious?

I've made NO configuration changes since the crash, and I've checked all my config files anyway just to be sure they weren't corrupted.
 
Bartender
Posts: 1210
25
Android Python PHP C++ Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Should it not be
HEAD / HTTP/1.0
?
 
Guy deLyonesse
Ranch Hand
Posts: 200
Eclipse IDE Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Karthik Shiraly wrote:Should it not be
HEAD / HTTP/1.0
?



Yes, thanks. That was a typo on my part here. On the server I did enter "HTTP."
 
Karthik Shiraly
Bartender
Posts: 1210
25
Android Python PHP C++ Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If httpd logs are not giving any clue, increase the LogLevel to debug (in httpd.conf) and see if there's anything unusual.
What OS is this? Assuming OS is *nix, check if syslogs hold any clues at the time of crash and later.

I don't think httpd is hanging - it's just not giving any HTTP response.
What file should be served at that path "/"?
If it's a PHP file, I've seen this happen if there's some syntax error or runtime error in PHP files. That error is captured in the PHP log, but if PHP logging is turned off, then it's difficult to find out. If using PHP, check that php.conf has log_errors set to on and error_log pointing to a writeable path.

Other things to check:
- file permissions - the apache process should be able to read the file and its parent directories
- not enough disk space for logs, caches, etc
- if using mod_jk, check the log file specified by JkLogFile in mod_jk configuration
 
Guy deLyonesse
Ranch Hand
Posts: 200
Eclipse IDE Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hey Karthik, thanks for the reply.

Karthik Shiraly wrote:If httpd logs are not giving any clue, increase the LogLevel to debug (in httpd.conf) and see if there's anything unusual.
What OS is this? Assuming OS is *nix, check if syslogs hold any clues at the time of crash and later.



This is CentOS. Looking at the syslog in this case didn't help because it looks like there are no log files at all that remain from before the crash. I don't know why this is. I'll be meeting with the sysadmin later. Maybe he knows.

Karthik Shiraly wrote:
I don't think httpd is hanging - it's just not giving any HTTP response.
What file should be served at that path "/"?
If it's a PHP file, I've seen this happen if there's some syntax error or runtime error in PHP files. That error is captured in the PHP log, but if PHP logging is turned off, then it's difficult to find out. If using PHP, check that php.conf has log_errors set to on and error_log pointing to a writeable path.



There's a generic index.html file there. (In the DocumentRoot directory) for testing purposes. Prior to the crash, I used mod_ajp to connect to a Tomcat servlet container.

Karthik Shiraly wrote:
Other things to check:
- file permissions - the apache process should be able to read the file and its parent directories
- not enough disk space for logs, caches, etc
- if using mod_jk, check the log file specified by JkLogFile in mod_jk configuration



File permissions and the apache user are unchanged from before, and the disk has plenty of space. We're not using mod_jk.

At this point, all I really want to do is get the basic index.html page to display in the browser to use as a starting point and I can't even get that.

Interestingly, after a server reboot I ran the HEAD / HTTP/1.0 command again and got this:


HTTP/1.1 200 OK
Date: Wed, 02 Nov 2011 15:54:19 GMT
Server: Apache/2.2.3 (CentOS)
Last-Modified: Thu, 11 Aug 2011 18:55:45 GMT
ETag: "24f2c4-6f-4aa3f58200e40"
Accept-Ranges: bytes
Content-Length: 111
Connection: close
Content-Type: text/html; charset=UTF-8


But still no response when using a web browser.
 
Karthik Shiraly
Bartender
Posts: 1210
25
Android Python PHP C++ Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
That's very odd.
What HTTP code does the browser receive, if any?
I notice the telnet was done from localhost. Is the browser on another machine, and if so, is httpd listening on that LAN IP address?


 
Guy deLyonesse
Ranch Hand
Posts: 200
Eclipse IDE Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The browser receives no code at all.

I also tested a GET request over telnet to make sure it was finding the correct DocumentRoot folder ans I got the HTML output to the terminal, so it's definitely running and finding the file.

The browser is indeed on another machine, and httpd is configured to Allow from all on the DocumentRoot folder.
 
Karthik Shiraly
Bartender
Posts: 1210
25
Android Python PHP C++ Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
So, browsing from the client machine gets a "could not connect" kind of browser error screen, but telnet from the same client machine gets a response?! It gets odder .
Is it a browser proxy issue or something like that, independent of the actual apache crash problem? Does clearing the browser cache help? Sorry, at this point, I've run out of ideas.
 
Guy deLyonesse
Ranch Hand
Posts: 200
Eclipse IDE Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Well no, when I use telnet it's when I'm connected to the server through ssh. That's why I can use localhost.

I don't think Apache is crashing... I think it's just not receiving the http request. No configuration has changed from before so I think it may be a corrupted file somewhere...
 
Karthik Shiraly
Bartender
Posts: 1210
25
Android Python PHP C++ Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ah ok, so port 80 is not reachable from remote machines. I'd look at "netstat -anp" output to ensure it's listening on that LAN IP too or on all IPs (0.0.0.0:80).
If that's also ok, then check if some firewall configuration has changed (/sbin/iptables).
If it was a corrupt file, then I think it would have manifested in the telnet output too, but that's not happening.
So perhaps it's an inadvertent configuration change / update done by sysadmins.
 
Guy deLyonesse
Ranch Hand
Posts: 200
Eclipse IDE Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If any of those weren't open, wouldn't it also prevent a request from reaching Tomcat? Or am I misunderstanding something and Tomcat circumvents this?
 
Karthik Shiraly
Bartender
Posts: 1210
25
Android Python PHP C++ Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes you're right - if apache is fronting tomcat on port 80 and there's a firewall problem, the liferay path should not have worked either. Guess that's not the problem then - I missed your statement about having tried tomcat after the crash.
 
Guy deLyonesse
Ranch Hand
Posts: 200
Eclipse IDE Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
SUCCESS!

It looks like what changed was the IP tables. The sysadmin had suggested I look at it before but at the time I had another idea in mind of what I expected to find, but on your advice I looked again and this time I saw it...

Port 80 and 443 are being redirected to 8080 and 8443, respectively... and Apache was only listening on 80 and 443. Tomcat responded because it IS listening on 8080 (and then doing a redirect... The way I had my configuration with Shibboleth, Tomcat would only be connected to through 8009 per the ajp config.)

This is not how it was prior to the crash, as far as I know.

To test this, I set Apache to listen on 8080 and 8443 and sure enough, it worked! So I re-enabled Shibboleth and that worked too. So now I need to get the iptables set back to the way they were so I can put my configuration back the way it was and all will be well.

Thanks very much for your help!
 
Karthik Shiraly
Bartender
Posts: 1210
25
Android Python PHP C++ Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You're welcome! These environmental issues can be major time wasters - good that this one got solved relatively quickly.
 
A berm makes a great wind break. And Iwe all like to break wind once in a while. Like this tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!