4

In a CentOS 7 server with apache httpd 2.4 set up as an ssl reverse proxy for tomcat, I am getting an error indicating that httpd is not able to connect with tomcat. How can I resolve this error so that httpd serves up the content generated by tomcat?

The ssl_error_log says: 

[Tue Dec 16 20:20:15.007630 2014] [proxy:error] [pid 12784] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed
[Tue Dec 16 20:20:15.007727 2014] [proxy:error] [pid 12784] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 60s
[Tue Dec 16 20:20:15.007752 2014] [proxy_http:error] [pid 12784] [client client.ip.address:48100] AH01114: HTTP: failed to make connection to backend: localhost

I have read the catalina.out file, which clearly indicates that tomcat successfully launched the root app after I ran startup.sh. I also read this other posting, which put the blame on SELinux. But I made sure that /etc/sysconfig/selinux has SELINUX=disabled, so it seems my problem is caused by something else.

The reverse proxy was working perfectly earlier today. The only changes made on the server today were to upload new versions of the root war into tomcat a couple of times, and to stop then restart tomcat and httpd each time a new root war was uploaded. This might have involved changing the shutdown port in server.xml from -1 to a valid port number to allow tomcat to shutdown.

You can view the complete ssl.conf by clicking on this link. Please note that the VirtualHost is completely defined in ssl.conf and NOT in httpd.conf. You can also read the complete server.xml by clicking on this link.

EDIT:

Following IanMcGowan's advice gave the following results: 

[[email protected] ~]# telnet localhost 8080
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
[[email protected] ~]# GET / HTTP/1.0
<HTML>
<HEAD>
<TITLE>Directory /</TITLE>
<BASE HREF="file:/">
</HEAD>
<BODY>
<H1>Directory listing of /</H1>
<UL>
<LI><A HREF="./">./</A>
<LI><A HREF="../">../</A>
<LI><A HREF=".autorelabel">.autorelabel</A>
<LI><A HREF=".readahead">.readahead</A>
<LI><A HREF="bin/">bin/</A>
<LI><A HREF="boot/">boot/</A>
<LI><A HREF="db/">db/</A>
<LI><A HREF="dev/">dev/</A>
<LI><A HREF="etc/">etc/</A>
<LI><A HREF="home/">home/</A>
<LI><A HREF="lib/">lib/</A>
<LI><A HREF="lib64/">lib64/</A>
<LI><A HREF="media/">media/</A>
<LI><A HREF="mnt/">mnt/</A>
<LI><A HREF="opt/">opt/</A>
<LI><A HREF="proc/">proc/</A>
<LI><A HREF="root/">root/</A>
<LI><A HREF="run/">run/</A>
<LI><A HREF="sbin/">sbin/</A>
<LI><A HREF="srv/">srv/</A>
<LI><A HREF="sys/">sys/</A>
<LI><A HREF="tmp/">tmp/</A>
<LI><A HREF="usr/">usr/</A>
<LI><A HREF="var/">var/</A>
<LI><A HREF="www/">www/</A>
</UL>
</BODY>
</HTML>

I also opened up 8080 to test if tomcat is running as follows:

When I typed in http : / / mydomain.com and http : / / my.ip.address , nothing was served to the browser. When I type https : / / mydomain.com , I get the same error stating the service is unavailable.

The last two lines of catalina.out have not changed since I restarted the server yesterday, and are as follows: 

16-Dec-2014 20:19:35.967 INFO [localhost-startStop-1] org.apache.catalina.start$
16-Dec-2014 20:19:35.970 INFO [main] org.apache.catalina.startup.Catalina.start$ Server startup in 46065 ms

EDIT #2:

To test if tomcat is running, I did the following: 

[[email protected]]# ps aux | grep tomcat
root      6858  0.7 18.5 3826248 1095780 ?     Sl   Dec12  51:03 java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/tomcat/endorsed -classpath /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
root     12524  0.2 20.1 3891788 1187888 ?     Sl   Dec15   6:20 java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/tomcat/endorsed -classpath /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
root     16404  0.2 15.0 3630784 887836 ?      Sl   Dec16   2:36 java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/tomcat/endorsed -classpath /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
root     17987  0.0  0.0 112640   964 pts/0    R+   14:20   0:00 grep --color=auto tomcat
Jeff Schaller
  • 66,199
  • 35
  • 114
  • 250
CodeMed
  • 5,079
  • 45
  • 100
  • 147
  • Are you sure tomcat is running? A common problem at startup is that something is already listening on a port it needs (e.g. your 8005 shutdown port). Can you post the complete catalina.out? You do not have to restart tomcat when making changes to the Apache proxy. – Ian McGowan Dec 17 '14 at 04:11
  • 1
    "telnet localhost 8080" from a shell on the CentOS server to make sure that tomcat is listening. If telnet gives an error, you are not connected, if nothing appears tomcat is up. You can type "GET / HTTP/1.0[enter][enter]" to request the default page. – Ian McGowan Dec 17 '14 at 04:17
  • @IanMcGowan: +1, but you shouldn't get "nothing" from `telnet` if Tomcat is up and running on port 8080. You'll get 3 or so lines, something like `Trying localhost...` then `Connected to localhost.` then `Escape character is ^]`. Also, `telnet` isn't installed by default on CentOS 7, so you may need to say `yum install telnet` first. – Warren Young Dec 17 '14 at 06:18
  • @WarrenYoung - you're right! To my shame I was thinking of the windows telnet command. On *nix I tend to use netcat nowadays... – Ian McGowan Dec 17 '14 at 07:15
  • @CodeMed: Why are you bringing a second computer into the discussion at all? The mantra of troubleshooting is to keep it as simple as possible, and test only one thing at a time. So, do *everything* here on a single machine. I don't care if you do it all on your development box or all on the production box. `telnet localhost 8080` only works *within* a single machine. That's what `localhost` means. – Warren Young Dec 17 '14 at 08:22
  • Why are you using generic proxying when there is a [specific solution](http://tomcat.apache.org/connectors-doc/generic_howto/quick.html) designed to link Tomcat and Apache? – Warren Young Dec 17 '14 at 16:30
  • @IanMcGowan I posted the results of your `telnet` commands in an edit to my OP above. I hesitate to post `catalina.out` because it is a massive file that really only shows that `tomcat` has loaded the app on the server, with no requests afterwards. I will try to open `8080` to the world and access it by http to confirm `tomcat` is working. – CodeMed Dec 17 '14 at 18:56
  • @IanMcGowan I added the results of opening `port 8080`, along with the last 2 lines of of `catalina.out.` – CodeMed Dec 17 '14 at 19:10
  • @WarrenYoung please see the edit to my OP above. – CodeMed Dec 17 '14 at 19:11
  • Tomcat is not running, or is not listening on 127.0.0.1. Is there someone local who can help you with this? Stackoverflow is a Q/A site, not a troubleshooting service. Is this the best approach to getting your site running? – Ian McGowan Dec 17 '14 at 19:12
  • @IanMcGowan The server is on a hosting company's network. I only call them with the types of questions that help me trouble shoot myself. I got the server in order to learn it by trial and error. I have had success with many similar questions on SO over the years. – CodeMed Dec 17 '14 at 19:15
  • Can you post the output of: `cat /etc/sysconfig/network`? – eyoung100 Dec 17 '14 at 19:19
  • @eyoung100 `cat /etc/sysconfig/network` produced only one single line of output as follows: `# Created by anaconda` – CodeMed Dec 17 '14 at 19:21
  • That's on your Devbox right? – eyoung100 Dec 17 '14 at 19:24
  • @eyoung100 No. That (and everything else in this post) is on the remote web server. I did not intend any confusion. Everything works on my local machine. – CodeMed Dec 17 '14 at 19:26
  • Can you remove Tomcat and Catalina, and your reverse proxy, so we have a clean slate? – eyoung100 Dec 17 '14 at 19:32
  • @eyoung100 If I remove tomcat and catalina, I am just going to have to rebuild the reverse proxy and ssl, which I need up and running tomorrow. – CodeMed Dec 17 '14 at 19:34
  • Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/19561/discussion-between-eyoung100-and-codemed). – eyoung100 Dec 17 '14 at 19:34
  • @eyoung100 this site still will not let me log into chat, but we could talk on the SO chat room again if you are willing. here is the link: http://chat.stackoverflow.com/rooms/67058/private – CodeMed Dec 17 '14 at 19:37
  • It's easiest to 1. Get all the services you need running first then add the reverse proxy, then add the SSL, or 2. Setup the reverse proxy, add all your services, and hen add SSL. – eyoung100 Dec 17 '14 at 19:38
  • @eyoung100 My host did a clean install of CentOS. I installed httpd and tomcat. The war is served by tomcat through 8080, but I am having trouble getting httpd to connect with tomcat. Are you willing to look at this again? I logged into this chat room: http://chat.stackoverflow.com/rooms/67058/private – CodeMed Dec 18 '14 at 21:27
  • @eyoung100 Thank you for all the help you have given me. I appreciate it. – CodeMed Dec 19 '14 at 17:24
  • @eyoung100 If you are still willing to help me with this, I am logged on to the server as root, and I am in the chat room reviewing what you typed before: http://chat.stackoverflow.com/rooms/67058/private – CodeMed Dec 22 '14 at 21:26
  • @eyoung100 I did the digitalocean tutorial, and now it is giving me a different error about not being able to reliably resolve the servername. I tried a few things that have not resolved the problem. Not sure if I should rip everything out and start over, or if there is an easier way to resolve this new error. I have the chat room open while I am researching options. – CodeMed Dec 23 '14 at 19:41
  • @eyoung100 I wrote up the results of your changes to `/etc/hosts` as an edit to another question. If you would like to read it, this link is here: http://unix.stackexchange.com/questions/175558/apache-will-not-restart-after-adding-link-to-virtual-host-conf-file-why – CodeMed Dec 24 '14 at 01:42

1 Answers1

9

To fix this, first test by setting the boolean dynamically (not permanent yet):

/usr/sbin/setsebool httpd_can_network_connect 1

If that works, you can set it so that the default policy is changed and this setting will persist across reboots:

/usr/sbin/setsebool -P httpd_can_network_connect 1

Credit: https://web.archive.org/web/20190313023936/http://sysadminsjourney.com/content/2010/02/01/apache-modproxy-error-13permission-denied-error-rhel/

Thomas Ruiz
  • 103
  • 4
Ramon R.
  • 91
  • 1
  • 2