13

In Linux, I want to migrate the memory pages of a KVM virtual machine from one NUMA node to another NUMA node on runtime. But I cannot find any interfaces to do that in the KVM hypervisor or using the libvirt API. Then I tried to use the numa_migrate_pages function in -lnuma , and migrate the memory pages of the VM process. But I found that the numa_migrate_pages function can only migrate some pages, it cannot migrate all the pages. For example, the text below shows the memory pages distribution of this VM process:

Node0:  0             pages
Node1:  1538          pages
Node2:  270641        pages
Node3:  552           pages

And I want to migrate all the pages in Node2 to Node0. But after using numa_migrate_pages function, only some pages are migrated, as text below shows:

Node0:  7952          pages
Node1:  1538          pages
Node2:  262113        pages
Node3:  552           pages

Then I open the /proc/[pid of VM process]/numa_maps file, and find that most of the pages left on Node2 are anonymous and dirty pages:

7f572c000000 default anon=262143 dirty=262143 N2=262113 ...

So why can't all the pages in Node2 be migrated to Node0? What's the problem here?

Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
Jenson
  • 161
  • 5
  • 3
    don't post images when pasting the text will do. images can't be searched for or pasted into answers, and can't even be read on monitors significantly different to yours (e.g. unreadably tiny fonts on larger, higher-resolution monitors) or by sight-impaired users using screen-reader software like Jaws. – cas Jun 14 '16 at 09:21
  • I presume that you are running the numa migrate pages as root? – Lmwangi Jul 25 '16 at 18:30
  • @Lmwangi yes, I am running as root. – Jenson Jul 27 '16 at 10:29
  • Are there enough free pages on Node0 to complete the transaction? – Aaron Walerstein Mar 23 '17 at 07:32

1 Answers1

3

You want the migratepages binary in the numactl package.

Usage & Example

sudo migratepages $VM_PID $SRC_NODE $DEST_NODE
sudo migratepages 12345 2 0

Limitations

VM hardware

Pages may be locked to a node, eg. if they related to hardware pass-through and they represent hardware located on a specific node.

Free Memory & Page size

You obviously need enough free memory on the destination node, but it also needs to not too fragmented to move large pages. If one of the pages is a large order contiguous allocation, and the destination node free memory has no free regions large enough, then moving the large page might fail (depending on compaction being triggered & succeeding).

robbat2
  • 3,599
  • 20
  • 32