I cannot comment so posting as an answer. (On a second thought, it is too much for just a comment.)
Looks like I have the same problem. I don't have a solution, but I think I found what causes the problem and have a (kind of) workaround.
I am using Linux Mint 17.3 with Cinnamon. When I try to switch to custom resolution using xrandr, screen blinks and I see something like this in Xorg log (it is /var/log/Xorg.0.log for me):
[ 13767.590] (II) intel(0): switch to mode [email protected] on VGA1 using pipe 1, position (0, 0), rotation normal, reflection none
[ 13767.696] (II) intel(0): switch to mode [email protected] on VGA1 using pipe 1, position (0, 0), rotation normal, reflection none
So it actually changes mode (or at least tries to), but reverts it back immediately.
When I do all the same in an empty Xorg server (sudo Xorg :1 & xterm -display :1), everything works as expected. This means that the problem is not caused by Xorg or xrandr.
Most likely that is Cinnamon (its Gnome heritage, to be precise) trying to automatically configure monitors based on ~/.config/monitors.xml which is modified by "Display settings". I don't know how to prevent this interference, apart from not using Gnome.
Update: It so happened that I've been passing close to this question again, so I decided to re-check it. Two things:
Most likely xrandr plugin of gnome-settings-daemon (cinnamon-settings-daemon in my case) is responsible, so disabling it should help. (I have a toggle for it at Settings>Startup applications right now, so I didn't look further into it. If the toggle is not present there should be other ways.)
I cannot reproduce that problem at all at the moment, xrandr can now change modes just fine. That's Cinnamon version 3.6.7 (cinnamon-settings-daemon 3.6.2), not sure what the situation is with Gnome itself. My system setup is exactly the same as it was. (I even got the fried LCD replaced under warranty.)