You can turn off access time recording for some file systems (look for noatime in /etc/fstab). To test this, you could try to touch -a file; ls -l --time=atime file, then repeat touch -a file; ls -l --time=atime file to see if it changes.
The ls behavior is explained by man ls: ls -l --time=atime shows access time in the date/time column. To sort by this key you need to use --sort=time as well.
If you by "the time is not changed" mean that it shows the same result, that is either because atime is disabled for the file system (see /etc/fstab) or because the access time is really the same as the modification time (quite common).
Note that listing files does not change their access time - The files are not accessed by ls, only the inode cache.