I ran grub-mkconfig -o /boot/grub/grub.cfg using Ansible, it created the same .new extention.
But when I used grub-mkconfig > /boot/grub/grub.cfg, it ran successfully and created grub.cfg.
-o might not be the same as >.
Looking at the source code after @oldfred 's comment, found out that the .new extension is created when there is an error in either /etc/default/grub or /etc/grub.d/* files. See here:
if test "x${grub_cfg}" != "x" ; then
if ! ${grub_script_check} ${grub_cfg}.new; then
# TRANSLATORS: %s is replaced by filename
gettext_printf "Syntax errors are detected in generated GRUB config file.
Ensure that there are no errors in /etc/default/grub
and /etc/grub.d/* files or please file a bug report with
%s file attached." "${grub_cfg}.new" >&2
echo >&2
exit 1
else
# none of the children aborted with error, install the new grub.cfg
mv -f ${grub_cfg}.new ${grub_cfg}
fi
fi