Volume Management: How to move space from one partition to another?

Question

I am setting up a redhat ec2 instance and by default the software I am using (called qradar) created the following volumes on the two 500g ebs storage devices attached to the instance:

$ lvs
  LV        VG        Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  storetmp  rootrhel  -wi-ao----   20.00g                                                    
  varlog    rootrhel  -wi-ao----  <20.00g                                                    
  store     storerhel -wi-ao---- <348.80g                                                    
  transient storerhel -wi-ao----  <87.20g 

$ df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/xvda2                       500G  1.4G  499G   1% /
devtmpfs                          16G     0   16G   0% /dev
tmpfs                             16G     0   16G   0% /dev/shm
tmpfs                             16G   17M   16G   1% /run
tmpfs                             16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/storerhel-store      349G   33M  349G   1% /store
/dev/mapper/storerhel-transient   88G   33M   88G   1% /transient
/dev/mapper/rootrhel-storetmp     20G   33M   20G   1% /storetmp
/dev/mapper/rootrhel-varlog       20G   35M   20G   1% /var/log
tmpfs                            3.2G     0  3.2G   0% /run/user/1000

I need my storetmp to be 100g. How can I move 80g of storage from store to storetmp?

It also seems that I may need to shift some space from xvdb3 to xvdb2:

# lsblk
NAME                    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
xvda                    202:0    0   500G  0 disk 
├─xvda1                 202:1    0     1M  0 part 
└─xvda2                 202:2    0   500G  0 part /
xvdb                    202:16   0   500G  0 disk 
├─xvdb1                 202:17   0    24G  0 part [SWAP]
├─xvdb2                 202:18   0    40G  0 part 
│ ├─rootrhel-varlog     253:2    0    20G  0 lvm  /var/log
│ └─rootrhel-storetmp   253:3    0    20G  0 lvm  /storetmp
└─xvdb3                 202:19   0   436G  0 part 
  ├─storerhel-store     253:0    0 348.8G  0 lvm  /store
  └─storerhel-transient 253:1    0  87.2G  0 lvm  /transient

Note that the directories are currently being used by the software running on the box and are not empty, so deleting them is out of the question, I need this to be done on-the-fly:

$ ls -l /dev/mapper/storerhel-transient
lrwxrwxrwx 1 root root 7 Aug 10 16:00 /dev/mapper/storerhel-transient -> ../dm-3
$ ls -l /dev/mapper/rootrhel-varlog 
lrwxrwxrwx 1 root root 7 Aug 10 16:00 /dev/mapper/rootrhel-varlog -> ../dm-0
$ ls -l /dev/mapper/storerhel-store 
lrwxrwxrwx 1 root root 7 Aug 17 04:10 /dev/mapper/storerhel-store -> ../dm-2

Answer 1

An extra 80 GB in EC2 EBS costs something under $12 per month. On-line manipulations are likely to take more than one hour of your work, and a risk of downtime if something goes wrong - how much is that worth for you?

Pay for some extra capacity, add it to your instance as a third disk xvdc, initialize it as a LVM PV (you don't even have to put a partition table on it: just pvcreate /dev/xvdc will be sufficient). Then add the new PV to your rootrhel VG (vgextend rootrhel /dev/xvdc) and now you can extend your /storetmp with the added capacity.

lvextend -L +80G /dev/mapper/rootrhel-storetmp
xfs_growfs /storetmp  #or the appropriate tool for your filesystem type 

With your immediate problem solved, you can now schedule for some downtime at suitable time.

If you are using XFS filesystem (as RHEL/CentOS 7 does by default), then during the next scheduled downtime, you'll create tarballs of the current contents of /store and /transient, unmount and remove the entire storerhel VG, add its PV xvdb3 to the rootrhel VG and then recreate the LVs for /store and /transient filesystems using more realistic estimates for their capacity needs, and restore the contents of the tarballs. End of downtime.

Now your rootrhel VG has three PVs: xvdb2, xvdb3 and xvdc, and plenty of space for your needs.

If you want to stop paying for xvdc, you can use pvmove /dev/xvdc to automatically migrate the data within the VG off the xvdc and onto unallocated space within xvdb2 and/or xvdb3. You can do this on-line; just don't do it at the time of your peak I/O workload to avoid taking a performance hit. Then vgreduce rootrhel /dev/xvdc, echo 1 > /sys/block/xvdc/device/delete to tell the kernel that the xvdc device is going away, and then tell Amazon that you don't need your xvdc disk any more.

I have nearly 20 years of experience working with LVM disk storage (first with HP-UX LVM, and later with Linux LVM once it matured enough to be usable in enterprise environment). These are the rules of thumb I've come to use with LVM:

• You should never create two VGs when one is enough.

In particular, having two VGs on a single disk is most likely a mistake that will cause you headache. Reallocating disk capacity within a VG is as flexible as your filesystem type allows; moving capacity between VGs in chunks smaller than one already-existing PV is usually not worth the hassle.

• If there is uncertainty in your disk space requirements (and there always is), keep your LVs on the small side and some unallocated space in reserve.

As long as your VG has unallocated capacity available, you can extend LVs and filesystems in them on-line as needed with one or two quick commands. It's a one-banana job for a trained monkey junior sysadmin.

If there is no unallocated capacity in the VG, get a new disk, initialize it as a new PV, add it to the VG that needs capacity, and then go on with the extension as usual. Shrinking filesystems is more error-prone, may require downtime or may even be impossible without backing up & recreating the filesystem in smaller size, depending on filesystem type. So you'll want to avoid situations that require on-line shrinking of filesystems as much as possible.

• Micro-management of disk space can be risky, and is a lot of work. Work is expensive.

Okay. Technically you could create a 80 GB file on /store, losetup it into a loop device, then make that into a PV you could add into your rootrhel VG... but doing that would result in a system that would most likely drop into a single user recovery mode at boot unless you set up a customized start-up script for these filesystems and VGs and got it right at the first time.

Get it wrong, and the next time your system is rebooted for any reason you'll have to take some unplanned downtime for troubleshooting and fixing, or more realistically recreating the filesystems from scratch and restoring the contents from backups because it's simpler than trying to troubleshoot this jury-rigged mess.

Or if you are using ext4 filesystem that can be on-line reduced, you could shrink the /store filesystem, shrink the LV, use pvmove --alloc anywhere to consolidate the free space to the tail end of the xvdb3 PV, shrink the PV, shrink the partition, run partprobe to make the changes effective without a reboot, then create a new partition xvdb4, initialize it as a new PV and add it to rootrhel VG...

BUT if you make one mistake in this sequence so that your filesystem/PV extends beyond its LV/partition container, and your filesystem gets switched into read-only mode with an error flag that can be only reset by running a filesystem check, resulting in mandatory unplanned downtime.