Always Free: NO NETWORK after Upgrade from Ubuntu 20.04 LTS ARM to Ubuntu 22.04 LTS ARM

Question

Scenario is the following:

I upgraded from Ubuntu 20.04 LTS to Ubuntu 22.04 LTS. However, after upgrading I lost total connection to the host and I'm only able to access it through Cloud Shell Console.

Observed behavior:

• SSH is no accesible through Public IP. Only Cloud Shell through Serial Console. Tested if key exchange RSA was the issue since now its recommended ECDSA, but this doesn't seem to be the issue.

• Unable to reach any source while doing sudo apt update

• nslookup google.es won't give any output (unreachable)

• ip addr IP Interfaces are up:

  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host 
         valid_lft forever preferred_lft forever
  2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
      link/ether 02:00:17:00:95:86 brd ff:ff:ff:ff:ff:ff
      inet 10.0.0.62/24 metric 100 brd 10.0.0.255 scope global dynamic ens3
         valid_lft 62767sec preferred_lft 62767sec
      inet6 fe80::17ff:fe00:9586/64 scope link 
         valid_lft forever preferred_lft forever
  

• ip route output:

  default via 10.0.0.1 dev ens3 proto dhcp src 10.0.0.62 metric 100 
  10.0.0.0/24 dev ens3 proto kernel scope link src 10.0.0.62 metric 100 
  10.0.0.1 dev ens3 proto dhcp scope link src 10.0.0.62 metric 100 
  169.254.169.254 via 10.0.0.1 dev ens3 proto dhcp src 10.0.0.62 metric 100
  

• iptables -L output:

  Chain INPUT (policy DROP)
  target     prot opt source               destination         
  
  Chain FORWARD (policy DROP)
  target     prot opt source               destination         
  
  Chain OUTPUT (policy ACCEPT)
  target     prot opt source               destination
  

• The output for iptables-save -c out is as follows:

  :INPUT DROP [71948:4580785] 
  :FORWARD DROP [0:0]  
  :OUTPUT ACCEPT[59782:3909997] COMMIT
  # Completed on Tue Aug 16 08:10:43 2022
  

• Netplan has only 1 configuration filed called 50-cloud-init.yaml which contains the following:

    ethernets:
        ens3:
            dhcp4: true
            match:
                macaddress: 02:00:17:00:95:86
            set-name: ens3
    version: 2
  

• /etc/resolv.conf output:

  nameserver 127.0.0.53 
  options edns0 trust-ad
  search vcn09040100.oraclevcn.com
  

• Im not sure if this an Oracle Cloud configuration issue, although I havent touched anything and it was working before and hosting a website. VCN vcn-20210904-0043 is assigned with a subnet of 10.0.0.0/24 and the following Ingress Security List

  

I'm running out of ideas and any help is GREATLY appreciated. I have other instances running on the same tenant sharing same VCN and have no problems whatsoever....so I'm inclined to think this is a OS issue.

Answer 1

You have iptables rules that prevent network operations. Despite having no apparent rule, there's still one rule applied: the default DROP policy for filter/INPUT. This means that any traffic received by the system is dropped before an application has a chance to use it. DNS resolution fails. TCP connections (obviously directly to an IP address since DNS failed) stay in SYN-SENT state. etc.

I don't know what rules are still staying around in the system before they were flushed, because nothing about this is written in the question. Removing existing rules to a ruleset is a dangerous operation with iptables: it doesn't revert the default policy to ACCEPT. So to manually and blindly remove all filter rules, this should be done:

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -F
iptables -X

Note that there are other possible tables in use (nat, mangle, raw) but OP's iptables-save -c command doesn't show they are in use.

The first command above will revert the filter/INPUT policy to ACCEPT in case it was DROP and only after it removes rules and (now empty) user chains.

Note: nftables doesn't suffer from this flaw, and nft flush ruleset would probably do all of above for all tables in one single command, assuming iptables (as iptables-nft) is now using the nftables API.

---

What still has to be done: find what part of the configuration in the system sets firewall rules and fix it.

This could be a firewalling package, such as firewalld or ufw, or manual rules. You'll have to investigate what correction should be done there.

For direct manual rules in configuration files, this command might help find such candidate files:

grep -r 'INPUT DROP' /etc

Meanwhile the system has no firewall rule, which isn't a good idea. But it's still protected by the firewall provided by the cloud environment.