2

I am trying to build small CEPH cluster on single box - testing in the lab, before switching bigger probject to it. Run into memory problems and can not find way to control it.

We have old Core2Duo CPU, 4gb ram. Creating 3 OSDs with 3 local 8tb disks.

ceph-deploy will make it and everything runs nice, expcet that for each of 3 OSDs 1 tmpfs partition is being created, which is 2gb and after copying ~50gb of data to CephFS Bluestore - box starts agressively using RAM and ends up with using all the swap. I failed to find correct setting to control, how much RAM OSD process is allowed to use.

It is ok to use more IO, instead of RAM. Looking for help (if that is possible). :) I am building everything on Centos 7.

Anton Aleksandrov
  • 23
  • 1
  • 4

2 Answers2

3

The Ceph 13.2.2 release notes says the following...

The bluestore_cache_* options are no longer needed. They are replaced by osd_memory_target, defaulting to 4GB. BlueStore will expand and contract its cache to attempt to stay within this limit. Users upgrading should note this is a higher default than the previous bluestore_cache_size of 1GB, so OSDs using BlueStore will use more memory by default. For more details, see the BlueStore docs.

This caught me by surprise. My osds were going absolutely wild with resident memory usage. The kernel was oom-killing osd processes.

Changing over to the new key and bouncing the osd processes has given me stable performance.

Mazzystr
  • 46
  • 2
  • Do you think that increasing this value could boost performance on a system with plentiful memory? – Tmanok Sep 26 '21 at 15:29
  • Ceph is very memory dependent. It will eat up whatever memory you thrown at it. Memory is tight for me. There's nothing I can do about is with shallow pockets. So I set the following kernel parameters as well to ensure a stable system * vm.min_free_kbytes = 524288 * vm.dirty_background_ratio = 50 * vm.dirty_ratio = 50 * vm.swappiness = 50 * vm.vfs_cache_pressure = 50 – Mazzystr Sep 29 '21 at 15:30
  • If you find you're swapping a lot, drop the swappiness and increase the cache pressure way way up. Otherwise looks good. I'll test giving say 6GB per OSD on my own systems at a later date and report back. – Tmanok Sep 29 '21 at 15:40
1

This is a common issue, the ceph mailing list is a very helpful archive for these questions, see this. You can reduce the bluestore_cache_size values, the defaults are 3GB for a SSD and 1 GB for a HDD OSD:

# If bluestore_cache_size is zero, bluestore_cache_size_hdd or bluestore_cache_size_ssd will be used instead.

host1:~ # ceph daemon osd.3 config show | grep bluestore_cache_size
    "bluestore_cache_size": "0",
    "bluestore_cache_size_hdd": "1073741824",
    "bluestore_cache_size_ssd": "3221225472",

But according to our monitoring in a production cluster the residual memory is about 3 GB for a HDD-OSD, so you'll have to tweak those values to your needs. The config reference is here.

I would recommend to start with one OSD first then watch the performance of your node; then add another OSD if the memory (or other measurements) is ok. If you hit the limits with one or two OSDs already you'll have to adjust the configs according to your needs. The values can be changed online running:

host1:~ # ceph daemon osd.<ID> config set bluestore_cache_size[_hdd|_ssd] <VALUE>

Permanent changes of configs have to be stored in /etc/ceph/ceph.conf

eblock
  • 956
  • 4
  • 7