If you REALLY do not want to resort to logrotate, use find not to parse the output of ls, while printing out the last found file's modification time in seconds since 1970.01.01.
$ find ./* -maxdepth 0 -name "binlog*" -printf "%T@ %f\0" \
| sort -z -k 1 -r \
| cut -z -d " " -f2- \
| tail -z -n +4 \
| xargs -0 -I {} mv -f "{}" /path/to/directory/
The above assumes GNU coreutils, which accept -z (sort, cut and tail) or -0 (xargs) to use NUL-delimited lines, meaning the code works correctly even with newlines in the filenames.
If you can be sure there are no newlines in the file names, it turns out that you don't need to terminate each found file with a null (\0) because you are sorting line-wise on the first field (files' last modification times) which never contains problematic characters. You can simplify the above to :
$ find ./* -maxdepth 0 -name "binlog*" -printf "%T@ %f\n" \
| sort -k 1 -r \
| cut -d " " -f2- \
| tail -n +4 \
| xargs -I {} mv -f "{}" /path/to/directory/