1

Hi my use case is quite specific.

I have 20 Windows 7 machines constantly creating files in my storage; around 98% of these files are 2.1 MB. On average we create 24 million files every 3 days, and this number may increase in the near future as we may need to add new clients to our system.

I do not modify files (just create, read, copy and delete). I have seen Reiser4, which looks promising, but I also would like to have the capability to replicate the files across multiple storage nodes across the network, so I can have a fault tolerance system in place.

Any suggestion?

Faheem Mitha
  • 34,649
  • 32
  • 119
  • 183
masber
  • 129
  • 1
  • 1
  • 7
  • 1
    So the small files are the remaining 2%. Why would you want Reiser4 for that percentage of your files? Isn't the performance on the remaining 98% more important? – Anthon Mar 29 '16 at 04:33
  • 3
    I'm voting to close this question as off-topic because it is about Windows. – dr_ Mar 29 '16 at 07:24
  • @dr01 this question is not about Windows, the server we are running is Linux CentOS 6.5 hence the filesystem is going to be installed on linux, that is why I put this question here – masber Mar 29 '16 at 11:21
  • @Anthon 2.1MB files are small and they don't perform same as 2GB files. I can prove this because we are using Panasas (PanFS) as file system and we get bandwidth of 400 MBs on big files (2Gigs) and 10 MBs on 2.1MB files. – masber Mar 29 '16 at 11:23
  • @dr01 The reason I mentioned that our clients are Windows 7 is to give you more details so you can think that I am using CIFS as protocol to share files from the clients. So this is still a Linux question as my problem is the solution I need to install on my CentOS server. – masber Mar 29 '16 at 11:28
  • @ManuelSopenaBallesteros I thought you were looking into tail packing for small files ( < 1Kb ) for other files Reiser4 doesn't do anything special IIRC. – Anthon Mar 29 '16 at 15:54

1 Answers1

2

Check out GlusterFS. I had similar needs a few months ago and tested it on VM and physical nodes. Replication is quick and setup isn't that difficult.

One caveat. If you do go with this, you need a dedicated machine to act as the file server. That machine will then handle replication between the slave nodes. I tried editing a file on one of the end nodes - i.e. to see if multiple nodes could act as the file server - to see if replication would sync the changes. It didn't; but when I edited the file via the master/controller the changes were synced.

user208145
  • 2,437
  • 2
  • 20
  • 21