Yes it is safe, but before you just throw it all away check what is in there. The files under .cache are used to speed up your programs e.g. by not causing downloads when not necessary. Some content there might immediately have to be downloaded or recreated again.
du -sm ~/.cache/*
will give you a list and you can selectively remove any of those subdirectories based on your usage of the programs, disc space used etc.
You could logout and back in as Lekensteyn indicated, but at most the programs owning the subdirectories have to be restarted if running.