I don't consider my answer here to be an in depth response from a security point of view, but it might help a bit.
Do you trust your "small group of people" and do they trust each other with their accounts? If no, then you still need munge.
Are you preventing other systems from reaching the "SlurmdPort" and "SlurmctldPort" on your slurm server? If no, then you still need munge.
Possibly the most basic reason that you still need munge is: Are you running a version of slurm that has any other option besides "munge" for "AuthType"? The latest version of slurm (22.05) does not even list any alternative but munge, as seen in the latest man page of slurm.conf.
I know this next statement is not what you asked, but I'll write it here in case it helps: munge is very light weight to run and quite easy to configure. For a slurm installation, it's likely that configuring munge will take up very little of your time compared to the rest of the slurm config.