Assuming you are running on a modern Linux or similar system, the encryption part should be fairly simple.
(Edit: Reading your question again, I see you started by asking about authentication and then asked the specific question about RSA encryption which I've answered here. You probably want to set things up as I've described here, then go and investigate pam_pkcs11 which is designed to do this kind of thing.)
Setup
Make sure your Yubikey has enabled 'PIV' mode, and you've installed the
OpenSC PKCS#11 module which can talk to it. Also install the OpenSSL
PKCS#11 'engine'.
Use the yubico-piv-tool or any other method you like to install an RSA
key into the slot of your choice, let's say the 'Card
Authentication slot 9e for example. You can either import an existing
key, or create one that's never existed outside the Yubikey.
Signing
Now it's simple. Any software that can use an RSA key from a file
ought to accept a standard PKCS#11 URI identifying the key in the
Yubikey instead.
The URI for the key in the Card Authentication slot will be something
like 'pkcs11:manufacturer=piv_II;id=%04'.
So you can do something like
echo "test payload" | openssl dgst -sha256 -sign 'pkcs11:manufacturer=piv_II;id=%04' -hex
... except OpenSSL doesn't quite get this right yet, so you have to add
-engine pkcs11 -keyform engine
to the above command line because all crypto software authors hate
their users and it doesn't bother to infer those obvious things from
the fact that you gave it a PKCS#11 URI. This is also slightly out of
date because newer OpenSSL uses "providers" instead of engines, and I
think it still doesn't just get things right for itself so you may need
a slightly different pointless arcane incantation to make it do so.
Decent software won't need those extra hints, and will just take the
URI in place of a filename and do the right thing. File bugs if not.
Example
$ yubico-piv-tool -s9e -ARSA2048 -agenerate | tee pubkey.pem
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAr9LrzjNbRABqhDQrGi3l
VcQhmUu0lls8k4XnO8c/U0oS6IvH4H7HuqXFfwThYofxIgA2eIXuRXf+V/CSWtXN
40Bb10QKcTXVATm05+KFNWg1GCVg2yrvsUOQSd6MOxAa5goUUi2xOjeLFZRvXuvt
YmLytwY77YqE0WOHYfYuk9kolueZHhq4BSOVRmQpZxKd6/MkWlT46SPc3Bwbyx41
t2U42vlnHYuma3NF6qI+a+LaMyvkFVBkRM6A1WB2u5jjl/ZQmyYsuqg2e8xu7P8m
5/GUH9HjD074+ea1NEdMncPhKjO+pL24BSebtIPUmzEJIh6kCaweJYiMKYuLx15H
HQIDAQAB
-----END PUBLIC KEY-----
Successfully generated a new private key.
$ echo Test | openssl pkeyutl -encrypt -pubin -inkey pubkey.pem > Test.enc
$ openssl pkeyutl -decrypt -in Test.enc -engine pkcs11 -keyform engine -inkey 'pkcs11:manufacturer=piv_II;id=%04'
Engine "pkcs11" set.
Test
You can also do signatures...
$ echo Test | openssl dgst -sha256 -sign 'pkcs11:manufacturer=piv_II;id=%04' -engine pkcs11 -keyform engine > signature.bin
$ echo Test | openssl dgst -sha256 -verify pubkey.pem -signature signature.bin
Verified OK
TL;DR
The different slots in the Yubikey PIV have different policies, especially around whether the PIN is required. Read the documentation I linked. See https://www.infradead.org/openconnect/pkcs11.html for more hints on finding the URI for a given object in a token.