0

I have a question regarding something what i consider unusual behavior related to the logging file /var/log/wtmp

There is a server which had Ubuntu 14.04.5 LTS minimal server installed and running for a while. At some point in time, it was decided to reinstall the base OS and change that to CentOS Linux release 7.8.2003 minimal server install. The installation was done and verified. All works well. However i noticed something really strange i.e. when issuing the command

last

I see successful authentications to the box before the time point of re-installation of the Server with CentOS! How is this possible at all. The IP from where the authentication occurred is valid and also expected!, however it is really interesting to me how this information got added to the /var/log/wtmp after the installation of the CentOS.

Because of this i started looking into the logs in detail i.e. using utmpdump

utmpdump < /var/log/wtmp

and there among some valid logs i see thing like :

[7] [123933] [ts/6] [user1   ] [pts/6       ] [                    ] [192.168.74.2   ] [Wed Oct 07 00:34:20 2020    ]
[8] [123933] [    ] [        ] [pts/6       ] [                    ] [0.0.0.0        ] [Wed Oct 07 02:45:41 2020    ]
[8] [120363] [    ] [        ] [pts/5       ] [                    ] [0.0.0.0        ] [Wed Oct 07 03:30:31 2020    ]

at first I noticed that high level process 123933 and 120363 and started looking immediately in /proc/ and "ps auxwf".
Those 2 last entries were not shown by last command output. Also I was not able to find any processes pointing to deleted files in proc!

I supposed that the second and third line means just the timeout ssh session, since they share the same process id, i.e. 123933 with the first line, however not sure about that. According to utmp man pages

type 8 means "DEAD_PROCESS"

The server has only one service available i.e. ssh on a non standard port. Authentication is allowed only with certificates, as well as fail2ban in configured, basically the standard ssh hardening guide is being followed there. i.e. things like AllowUsers, ClientAliveInterval, Protocol 2, No root user login, no passwords authentication etc. running 3.10.0-1127.el7.x86_64 kernel. SeLinux was enabled all of the time.

I do not see any listening raw sockets, there i.e. ss -w -a is empty , and the firewall has only that one open tcp port for ssh and nothing else. No funny dot "..." directories. The rest of the logs seems intact.

So far i can not explain one thing i.e. the wtmp log file. Is there some constellation where the installation of Centos could have read the wtmp logs from the previous server which could have been stored on some Non-volatile memory and imported them into the new installation, since i do see in the current CentOS wtmp log valid authentications that practically have occurred before the CentOS installation i.e. while the server had Ubuntu as an OS not CentOS !

In what why this information could have been carried over to CentOS, during the installation?

The other reason what i can think about is that the machine was compromised and someone copied/edited the wtmp log file, in this case the server was compromised.

Is there the possibility that wtmp is stored also on some other place except the current hard drive, i.e. some non volatile memory or anything of that kind, which is some default CentOS constellation. How those valid logins on the server that are seen in the wtmp with the time stamp preceding the installation date, could be explained?

Tito
  • 101
  • 1
  • There are far too many exclamation marks in this question, and no explanation of how `/var/log` residing on "the hard drive" is somehow "volatile". – JdeBP Oct 08 '20 at 02:40
  • @JdeBP, my question here is could the OS somehow store the /var/log/wtmp on a NVRAM per default. I think no but i can not exaplin valid entires in the wtmp that happened before the installation of the system. – Tito Oct 08 '20 at 06:31
  • I think it's far more likely that the system clock was not set correctly when the first logins after the CentOS installation happened, and resulted in incorrect login times to be logged. Or was the installation done by preserving the existing partitions and their formatting? That might have caused an old wtmp file to not get overwritten... but could mean that your system might now have other Ubuntu left-overs mixed in with the CentOS system files, taking up space and potentially causing confusion. – telcoM Oct 08 '20 at 06:49
  • @JdeBP basically i am searching for an explanation other then network breach. – Tito Oct 08 '20 at 06:50
  • @telcoM thank you for your reply. I am thinking about that too. The installation was done by inserting completely new drives into the system and formatting them together into hardware raid level 1. On top of the raid the CentOS was installed. Everything was formatted! i Did that myself. I did not configured the NTP straight away but on the next day. Basically i see a logins that we matched with those source machines. I mean that those logging indeed occurred, because we see that in TACACS as well. but how in hell did those logins survived re-installation. – Tito Oct 08 '20 at 07:08

0 Answers0