3

Let's say I have two bash files. The first one is called diskFile.bash:

for i in {1..10000}
do
    file=$(mktemp)
    cat > $file <<- "SIGN"
        "Hello World"
SIGN
done
echo "finished"

And the second one is called ramFile.bash:

for i in {1..10000}
do
    file=$(mktemp -p /dev/shm/)
    cat > $file <<- "SIGN"
        "Hello World"
SIGN
done
echo "finished"

On Linux, files created inside /dev/shm are files inside the RAM memory of the system. But if I try executing the first file with time bash diskFile.bash, I get:

finished
15.56user 5.74system 0:20.25elapsed 105%CPU (0avgtext+0avgdata 6324maxresident)k

And with time bash ramFile.bash:

finished
15.20user 5.37system 0:19.45elapsed 105%CPU (0avgtext+0avgdata 6380maxresident)

The difference of time doesn't look like a significant one considering /dev/shm should be using RAM memory. Why creating temporary files inside /dev/shm is not faster than creating temporary files inside /tmp? Is there any way of speeding up the process of creating files using bash?

raylight
  • 361
  • 1
  • 5
  • 13
  • 2
    Are you sure `/tmp` is on disk? (It’s commonly in memory now.) – Stephen Kitt Mar 22 '22 at 15:31
  • 1
    @StephenKitt In my case, I'm sure it's not mounted on RAM since I see that space on my hard disk is used when I put huge files inside it. I've made a different test here creating a tmp folder inside `$HOME` and I've executed the code with `file=$(mktemp -p /home/myuser/tmp)` (to be sure that this folder is in disk)... I still got `15.58user 6.41system 0:20.91elapsed 105%CPU (0avgtext+0avgdata 6392maxresident)k` as result... – raylight Mar 22 '22 at 15:39
  • I would guess that temp file are created **first** in RAM, then synchronized to disk at a latter time, outside the scope of yout `time` command. – Archemar Mar 22 '22 at 15:57
  • 2
    That was going to be my next question — does it make any difference if you `sync`? The `/tmp` files are probably only in the page cache (in memory) for the duration of your test. – Stephen Kitt Mar 22 '22 at 16:00
  • (1) You say “… I see that space on my hard disk is used when I put huge files inside it.”  (I presume that “it” refers to `/tmp`.)  How are you measuring / observing “space on [your] hard disk”? (2) This is not really a question *about* bash, and I doubt that it will be answered with bash knowledge. – G-Man Says 'Reinstate Monica' Mar 22 '22 at 16:17
  • @G-ManSays'ReinstateMonica' It's a question about optimizing the way of creating files in the shell. I'd understand if that was just the time it takes to accomplish this task, but I think it's weird that it takes the same amount of time for creating files in `/dev/shm` and `/any-other-folder`. About the way I'm checking space on disk, I'm using `df -h` and I see the size of my OS partition increasing... – raylight Mar 22 '22 at 16:37
  • @StephenKitt Sorry, I didn't get the question "does it make any difference if you `sync`?". How would using `sync` solve this issue? – raylight Mar 22 '22 at 16:39
  • It wouldn’t solve it, it would show the real difference between creating files in memory and creating files on disk (by ensuring that you’re actually hitting the disk). – Stephen Kitt Mar 22 '22 at 16:40
  • (1)  Yes, it’s somewhat peculiar that it takes the same amount of time to create files in ``/dev/shm`` and other directories, but …  Can you program in C, C++, Java, Python, or any other programming language?  If so, what happens if you translate your test code into one of those other languages?  If you get results for C (or one of the others) that are radically unlike the results for bash, then you will have a ***really** interesting* question.  Until then, your question is about the file system and the operating system, and (probably) not about bash.  … (Cont’d) – G-Man Says 'Reinstate Monica' Mar 22 '22 at 17:02
  • (Cont’d) …  (2) If your ``/tmp`` directory is really a directory on your root partition, and the free space on `/` goes down when you create files in `/tmp`, you should probably say so.  (3) I accept the terms as pretty much interchangeable, but some people here get annoyed when you refer to directories (in Unix / Linux) as “folders”. – G-Man Says 'Reinstate Monica' Mar 22 '22 at 17:02
  • Unfortunately i dont have an answer- but i have wondered this EXACT same question. I can also CONFIRM your findings in my real world scripts (some that create 500+ small temporary files). I have compared results both writing to a directory (non-SSD even) VS to a memory /dev/shm file (same mktmp command), and i can NEVER find a performance improvement. I cant say 100%, but via watching output of : iostat -x -m 1 100 , (it does *SEEM/Look like* /dev/shm is NOT writting to the disks as opposed to redirection to a file which DOES seem to write to disk ) -- would LOVE an answer! tks – James Gaul Sep 21 '22 at 18:40

0 Answers0