63

On a Linux VM I would like to TEST the NAGIOS monitoring more deeply than just switching off the VM or disconnecting the virtual NIC; I would like to test or "enforce a disk space alarm" through occupying several % of free space for a short period of time.

I know that I could just use a 

dd if=/dev/zero of=/tmp/hd-fillup.zeros bs=1G count=50

or something like that... but this takes time and loads the system and requires again time when removing the test files with rm.

Is there a quick (almost instant) way to fill up a partition that does not load down the system and takes a lot of time ? im thinking about something that allocates space, but does not "fill" it.

Rui F Ribeiro
  • 55,929
  • 26
  • 146
  • 227
Axel Werner
  • 978
  • 1
  • 9
  • 19

4 Answers4

88

The fastest way to create a file in a Linux system is using fallocate:

fallocate -l 50G file

From man:

fallocate is used to manipulate the allocated disk space for a file, either to deallocate or preallocate it.
For filesystems which support the fallocate system call, preallocation is done quickly by allocating blocks and marking them as uninitialized, requiring no IO to the data blocks. This is much faster than creating a file by filling it with zeros.
Supported for XFS (since Linux 2.6.38), ext4 (since Linux 3.0), Btrfs (since Linux 3.7) and tmpfs (since Linux 3.5).

Rui F Ribeiro
  • 55,929
  • 26
  • 146
  • 227
  • 1
    Why are you running it through `sudo`? – gerrit Jun 29 '16 at 23:31
  • 1
    @gerrit Added that point to the answer. – Rui F Ribeiro Jun 30 '16 at 06:53
  • 4
    *"`fallocate` needs root privileges"* Not on my system (Linux Mint 17.3, downstream of Ubuntu, thus Debian). (ext4 file system) – T.J. Crowder Jun 30 '16 at 07:48
  • 1
    +1 although the OP explicitly mentioned that his filesystem is ext3. – syneticon-dj Jun 30 '16 at 07:59
  • @syneticon-dj I noticed that too, however it is trivial to upgrade an ext3 system without much complications and without needing to boot from another medium to ext4. Txs for the mod up. – Rui F Ribeiro Jun 30 '16 at 08:40
  • Strictly speaking, `truncate` is the fastest way. On my system at least, it is significantly faster than `fallocate`. On the other hand, `truncate`'s file doesn't affect `du` or `df` output while `fallocate` does. Presumably because ir isn't sparse. – terdon Jun 30 '16 at 14:12
  • `fallocate` does not need sudo (maybe you did your tests on a root-owned directory?) and at this level, it is not more "direct manipulation" than normal file editing or truncate. The Re: paragraph is based on false assumptions, and contrasts with the philosophy of this command/syscall. – ignis Jun 30 '16 at 15:12
  • That paragraph is actually incorrect (except for "the result [with this example command] is not a sparse file"). fallocate creates a file and it's a normal file. "A normal user cannot really manipulate the file system in that way." This is untrue (and nonsense, since it's not a security issue and I can't see another reason why anyone would think that). I appreciate trying to provide a factual answer with a clear example, but I don't like that this answer expresses a poster's original invention "as if" it were the rationale of the input/output syscall and commands design. – ignis Jun 30 '16 at 15:16
  • I stand corrected about not needing root privileges. Just had the time to test it now. Thanks. – Rui F Ribeiro Jun 30 '16 at 15:46
  • @RuiFRibeiro thanks for the tips! but since we use SLES 11 SP4 here (and we are not allowed to change that yet) it seems it wont support ext4 partitions for read/write. - but anyway... maybe for another case, would you please link or show me how you would usually convert/upgrade ext3 to ext4 , keeping all ACLs and meta data ? – Axel Werner Jul 01 '16 at 11:11
  • Sure thing. A bit Debian specific, but it applies. https://debian-administration.org/article/643/Migrating_a_live_system_from_ext3_to_ext4_filesystem ; We used to be a SuSE shop before I migrated almost everything to Debian...coincidentally I am using a OpenSuSE 11.4 for testing fallocate and it seems happy with ext4. – Rui F Ribeiro Jul 01 '16 at 11:23
  • (or as it is a test system, instead of migrating, add a new ext4 filesystem and point Nagios to it) – Rui F Ribeiro Jul 01 '16 at 11:34
  • 1
    @RuiFRibeiro, thanks! for sles11sp4 ive been able to create a file, format it with ext4, but where unable to mount it in RW mode. later i found a kernel message in /var/log/messages that said, ext4 is supported only as read-only. :/ – Axel Werner Jul 01 '16 at 11:40
  • 1
    `fallocate` doesn't exist on Mac OS nor does Homebrew have it, but `mkfile 50G file` does the same thing, as described here https://stackoverflow.com/questions/11497567/fallocate-command-equivalent-in-os-x – Noah Sussman Feb 19 '20 at 18:17
  • 1
    is it safe to say that fallocate doesn't wear flash storage (via program/erase cycles) making it preferable to dd for that reason as well? – Michael Feb 09 '22 at 04:09
  • @Michael Indeed, very good selling point form a modern perspective – Rui F Ribeiro Feb 09 '22 at 08:59
14

Other alternatives include:

  1. to change the alarm thresholds to something near or below the current usage, or
  2. to create a very small test partition with limited inodes, size, or other attributes.

Being able to test things such as running into the root reserved percentage, if any, may also be handy.

cat
  • 3,428
  • 4
  • 22
  • 50
thrig
  • 34,333
  • 3
  • 63
  • 84
  • root reserved percentage *normally* is 10% unless you tweak it - it ends up a too big waste of system in big partitions/modern disks. When defining alarms, you´d better already take it in account. – Rui F Ribeiro Jun 29 '16 at 16:59
  • +1 for the first thing. a hundred times true. why the hell should I actually create something on a machine disk? what if something (like coredump, batch job generating big temporary files, ...) happens at the time of my testing and whole disk gets accidentally eaten up? – Fiisch Jun 29 '16 at 18:30
  • 2
    @Fisch - Why? To make sure your alerting threshold is correct and that you're not doing something like accidentally setting the inode free percentage instead of the disk space free percentage (which I've seen done before). If something fails because you filled up a disk to the alerting threshold, then your alerting threshold is too low - the whole point of alerting is that it's supposed to alert you *before* things start to break. – Johnny Jun 29 '16 at 20:06
  • Cat, good point. But no solution for me. I dont have controll over the VM configuration (cant alter partitions or virtual disks), nor have controll over the NAGIOS Server. – Axel Werner Jun 30 '16 at 06:37
  • 2
    @AxelWerner Can you loopback-mount a file as "fake" partitiion? That still would allow you to test without seriously affecting anything. Format it with one of the supported filesystems and and you can play around with fallocate too. – Tonny Jun 30 '16 at 14:09
  • @RuiFRibeiro `man 8 mke2fs` says for `-m` that the default is 5% on both Debian Wheezy (July 2012) and Jessie (August 2014). Where are you getting the figure 10% from? I can't imagine that the default percentage has been *increased* in recent years. – user Jun 30 '16 at 15:28
  • @Tonny, thats a good idea. should be possible, asuming the NAGIOS guys do dynamically test for mounted partitions or use "df". will test this soon. thanks – Axel Werner Jun 30 '16 at 15:55
  • The Suggestion of @Tonny worked a treat! The load to the system is minimal and result is sufficient. This is what i did: `dd if=/dev/zero of=nagios-hd-space-test.ext3 bs=1M count=100 ; mkfs.ext3 nagios-hd-space-test.ext3 ; mkdir –p /mnt/test ; mount nagios-hd-space-test.ext3 /mnt/test/ ; dd if=/dev/zero of=/mnt/test/hd-filler-file.dd bs=1M count=82 ; sleep 400 ; umount /mnt/test ; rm nagios-hd-space-test.ext3` – Axel Werner Jul 01 '16 at 11:05
  • @MichaelKjörling 10% is (very) old school, pre-Linux, thanks for correcting me. – Rui F Ribeiro Jul 01 '16 at 11:27
  • @AxelWerner Nice ! – Tonny Jul 01 '16 at 14:53
12

  1. fallocate -l 50G big_file

  2. truncate -s 50G big_file

  3. dd of=bigfile bs=1 seek=50G count=0

As those three ways can all fill up a partition quickly.

If you like use dd, usually you can try it with seek. Just set seek=file_size_what_you_need and set count=0. That will tell the system there is a file, and its size is what you set, but the system will not create it actually. And used this way, you can create a file which is bigger than the partition size.

Example, on an ext4 partition with less than 3G available. Use dd to create a 5T file which exists as metadata -- requiring virtually no block space.

df -h . ; dd of=biggerfile bs=1 seek=5000G count=0 ; ls -log biggerfile ; df -h .

Output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda9        42G   37G  2.8G  94% /home
0+0 records in
0+0 records out
0 bytes copied, 4.9296e-05 s, 0.0 kB/s
-rw-rw-r-- 1 5368709120000 Jun 29 13:13 biggerfile
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda9        42G   37G  2.8G  94% /home
agc
  • 7,045
  • 3
  • 23
  • 53
Se ven
  • 323
  • 1
  • 10
  • 1
    can you add some more information to your answer? – cat Jun 29 '16 at 15:18
  • I just add more thinking in a finished question for people which search the same question other way. ignore it what if you are not. – Se ven Jun 29 '16 at 15:50
  • This `count=0` method is quite interesting, I've added an example. – agc Jun 29 '16 at 16:47
  • 8
    Note that the `dd` example above may well allocate a sparse file. In that case the file size is 50G, it's actually only using a block (or not even) and so the disk is not getting full. YMMV. – MAP Jun 29 '16 at 19:49
  • `dd: failed to truncate to 53687091200000 bytes in output file ‘biggerfile’: File too large` – gerrit Jun 29 '16 at 23:33
  • 2
    I tested your suggestion on my ext3 filesystem. it did not work as expected. truncate and dd did create a file with a large file size, but "df -h" did not recognize it. still shows the same free hd space. – Axel Werner Jun 30 '16 at 06:53
  • Quite interesting knowing dd can create a sparse file. – Rui F Ribeiro Jun 30 '16 at 06:56
5

You could also take advantage of the stress-ng tool that is supported on a wide number of Linux-based systems:

stress-ng --fallocate 4 --fallocate-bytes 70% --timeout 1m --metrics --verify --times
theosophe74
  • 51
  • 1
  • 2
  • Exactly what I was looking for to test autoscaling functionality. Being able to allocate disk-space dependent on the total available size was a must-have feature. – Martin Bories Nov 27 '21 at 11:39