8

I am looking for a stackable file system (Like UnionFS, aufs, and OverlayFS) that works similar to bcache or lvmcache but on file level - not on block level:

  • Reads will be done from top-layer. If the file is not there, read the file from bottom-layer and store it in top-layer.
  • Writes will be done to top-layer. After write is done, it will also be written to bottom-layer (like write-back/write-behind caching)
  • When the top-layer reaches a certain size, files will be erased from the top-layer (like an LRU cache). Files will only be removed if they have been written to bottom-layer.

For my use bottom-layer will be 10-1000 times bigger than top-layer - in other words: bottom-layer will not fit in top-layer (again similar to bcache).

Does such stackable file system exist? Can it be done with aufs or similar?

Background

I love file systems, and use all sorts of Fuse-based file systems. But file systems like sshfs can be rather slow, and it would be really handy to have a way of speeding that up if you know that the bottom-layer will only be changed through this system.

Using an sshfs mounted dir as the bottom-layer is just one example, so I am not looking for a specialized approach, that deals with sshfs. I am looking for a general approach where the top-layer and the bottom-layer are just treated as dirs.

Ole Tange
  • 33,591
  • 31
  • 102
  • 198

1 Answers1

-1

What you describe can be handled by the use of standard LVM cache volumes. Typically the top layer is handled by a faster device (eg. SSD). LVM cache volumes are highly configurable, for example you can specify writethrough caching (writes to the lower layer happen at the same time = no data loss) or writecache (writes to the lower layer are delayed = better performance).

lvmcache(7) describes how to convert a standard logical volume (LV) into a volume which makes use of a fast device as a caching top-layer.

John Hawthorne
  • 708
  • 6
  • 7
  • It's not stackable though, is it...? – roaima Mar 04 '20 at 10:14
  • It's stackable in the sense that reads and writes go through the top layer of cache. Admittedly it's not a file-system as the caching is done on the block level but it allows a file-system to make use of caching. – John Hawthorne Mar 04 '20 at 10:49
  • 1
    I still don't think I can stack it on top of, say, [S3QL](http://www.rath.org/s3ql-docs/about.html) or even CIFS, though. – roaima Mar 04 '20 at 10:53
  • `lvmcache` and `bcache` are pretty much the same thing and not what I am looking for. I have elaborated the question why a block based approach will not solve my issue. – Ole Tange Mar 04 '20 at 14:06