0

I have an nginx container that runs as part of a service. This is the function code used to start all the containers including nginx:

    if [[ ! -s ${SCRIPT_DIR}/.container-info-patches.txt ]]; then
      patches_echo "Patches must be set up before running. Please run 'patches setup' first."
      exit 1
    fi

    all_containers_running=false

    while true; do
      all_containers_running=true

      for container in "${containers[@]}"; do
        patches_echo "Checking container status: $container"
        status=$(podman container inspect -f '{{.State.Status}}' "$container" 2>/dev/null)

        if [[ "$status" != "running" ]]; then
          all_containers_running=false

          patches_echo "Starting container: $container"
          podman start "$container" >/dev/null 2>&1 || patches_echo "Container not found: $container" --error

          if [[ $container == "patches-nginx" ]]; then
            check_nginx_status
          elif [[ $container == "patches-psql" ]]; then
            wait_for_postgresql
          fi
        fi
      done

      if [[ $all_containers_running == "true" ]]; then
        break
      fi

      if [[ $CONTINUOUS != "TRUE" ]]; then
        break
      fi

      sleep 1
    done

If I run that manually, it runs just fine and all the containers start. I have a user systemd file that I have set up with linger to run on startup. It starts all the containers EXCEPT nginx without issue. When I check the logs with podman logs patches-nginx after a reboot I see:

}[grant@patches ~]$podman logs patches-nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: IPv6 listen already enabled
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up

        10.89.0.5 - - [02/Jun/2023:18:48:43 +0000]
        "GET / HTTP/1.1" 400 255
        "-" "curl/7.76.1"
        Certificate: "-"
        Client Key: "-"

which tells me other than the fact that it received a single request that issued a 400 - it's all happy and there should be no reason for it to exit. Even if it did exit, I have specifically engineered the aforementioned start function, to continuously retry to start nginx for 60 seconds. What is absolutely bizarre is that the code exits and ignores the code that is meant to keep trying to start it well beneath that 60 second window.

Again, if I run exactly the same code that the service file should be running manually, everything works exactly as I expect. That is to say, if I login and run patches.sh start --continuous everything comes up immediately including nginx which is ostensibly all that service file should be doing.

Something about the service file is mucking up the works and I can't figure out what it is because everything except nginx works exactly as expected.

The service file is straightforward:

[Unit]
Description=Patches Service
Wants=network.target
After=network.target
Requires=user@${USER}.service

[Service]
Type=oneshot
TimeoutStartSec=10min
ExecStart=/bin/bash ${SCRIPT_DIR}/patches.sh start --continuous

[Install]
WantedBy=default.target
Grant Curell
  • 507
  • 4
  • 17

1 Answers1

0

This is part of the answer - something is sending SIGTERM to nginx. I checked the dockerfile and didn't see anything that stood out as a healthcheck or something that might be killing it.

I figured it out thanks to this helpful answer. I was seeing the same behavior on the httpd container and on a hunch swapped both it and nginx to --restart-always and now they both work.

I'll leave this open for someone else to perhaps come along and figure out what's causing the SIGTERM.

Grant Curell
  • 507
  • 4
  • 17