I have searched everywhere and what people say is that you have to give the nodelay argument to the pam_unix.so in /etc/pam.d/system-auth to remove delay. I tried doing that and it didn't work. I even added nodelay to every pam_unix.so in the /etc/pam.d/ folder and it didn't work either. When that didn't work I tried adding this line: auth optional pam_faildelay.so delay=0 in /etc/pam.d/system-auth and it didn't work either. On the other hand, if I set the delay to more than 3 seconds it adds the delay but below 3 seconds it does not work. I am really confused and would appreciate if someone could answer me.
I also tried changing the FAIIL_DELAY to 0 in my /etc/login.defs and that also didn't work. Lastly, I looked for the auth required pam_unix.so try_first_pass nullok line in my /etc/pam.d/system-auth file but couldn't even find that line. The closest line that resembled that was this line: auth [success=2 default=ignore] pam_unix.so try_first_pass nullok. I then added the nodelay argument to that but it didn't work too.