Error 503 Service Unavailable on Varnish, but varnishlog shows no error

Question

I have a server running Varnish and nginx. The frontend website reports a 503 error:

Error 503 Service Unavailable

Service Unavailable

Guru Meditation:
XID: 317911182

Varnish cache server

This is the content of /etc/varnish/foo.vcl:

backend default {
  .host = "127.0.0.1";
  .port = "8080";
  .connect_timeout = 1s;
  .first_byte_timeout = 90s;
  .between_bytes_timeout = 90s;
}

varnishlog apparently shows no issue:

0 CLI          - Rd ping
0 CLI          - Wr 200 19 PONG 1486463718 1.0
0 CLI          - Rd ping
0 CLI          - Wr 200 19 PONG 1486463721 1.0
0 CLI          - Rd ping
0 CLI          - Wr 200 19 PONG 1486463724 1.0
0 CLI          - Rd ping
0 CLI          - Wr 200 19 PONG 1486463727 1.0

However, there is no service running on port 8080 on localhost:

server# netstat -anp | grep 8080
server# telnet localhost 8080
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

and in fact the nginx log reports an error while trying to connect to it:

2017/02/07 11:51:11 [error] 2008#0: *188 connect() failed (111: Connection refused) while connecting to upstream, client: 10.2.3.4, server: _, request: "GET /mydir/ HTTP/1.1", upstream: "http://127.0.0.1:8080/mydir/", host: "myhost.example.com"

What could I check to further troubleshoot the issue?

Answer 1

The following lines show normal operation - Varnish is checking if it's caches are ok. This is normal and not related to connectivity to backend.

     0 CLI            - Rd ping
     0 CLI            - Wr 200 19 PONG 1486473078 1.0

If the backend has configured health check, you would be seeing something similar to this:

varnishlog output:

     0 CLI            - Rd ping
     0 CLI            - Wr 200 19 PONG 1486473075 1.0
     0 Backend_health - boot.default Still sick ------- 1 3 3 0.000000 0.000000
     0 Backend_health - boot.default Still sick ------- 0 3 3 0.000000 0.000000

or

 32771 Timestamp      b Start: 1486473576.634500 0.000000 0.000000
 32771 BereqMethod    b GET
 32771 BereqURL       b /
 32771 BereqProtocol  b HTTP/1.1
 32771 BereqHeader    b Host: 127.0.0.1:8080
 32771 BereqHeader    b User-Agent: curl/7.51.0
 32771 BereqHeader    b Accept: */*
 32771 BereqHeader    b X-Forwarded-For: 127.0.0.1
 32771 BereqHeader    b Accept-Encoding: gzip
 32771 BereqHeader    b X-Varnish: 32771
 32771 VCL_call       b BACKEND_FETCH
 32771 VCL_return     b fetch
 32771 FetchError     b no backend connection

default.vlc:

backend default {
    .host = "127.0.0.1";
    .port = "8081";
    .probe = {
        .url = "/status";
        .timeout = 60 ms;
        .interval = 10s;
        .window = 3;
        .threshold = 3;
    }
}