It is possible to only launch the X server with one application, you can do it using the startx script from the tty
$ startx "/usr/bin/application --key value"
More info on startx / xinit on the archwiki. (The archwiki is an invaluable ressource even if you are not using archlinux)
It should also be possible to configure your login screen (probably lightdm) to do so, if you want a graphical login
But honestly, if you want to avoid a heavy desktop manager, you can also use a small window manager (such as openbox), which won't be as heavy, but will still allow you to, for example, resize your window, or launch several applications