1

We write software that runs in third party devices. On one of the devices we support, the manufacture tells us not to write to the flash drive, or we risk using up the limited write operations it supports. Unfortunately, one of the requirements of our application is to persist some data across boots and we have no other alternative.

I don't know exactly what the drive inside the device is, nor how it is configured, so one question is how can I go about finding this information? Some information I have managed to find: bash-3.2$ df | grep mtd /dev/mtdblock5 65536 7824 57712 12% /apps

bash-3.2$ dmesg | grep -i mtd Kernel command line: root=/dev/mtdblock4 rootfstype=jffs2 rw ip=none console= mem=128M init=/sbin/init mtdparts=mtd:512k(bootloader),512k(env),2M(kernel_a),2M(kernel_b),59M(filesystem),64M(user) loglevel=3 panic=5 reboot=h 6 cmdlinepart partitions found on MTD device <NULL> Creating 6 MTD partitions on "<NULL>":

I've had a look in proc and sysfs and didn't find anything useful. The device environment doesn't have any useful tools installed such as hdparam, lshw, etc that I can find.

Another question is whether there are any heuristics software could use to detect whether the 'write limit' is approaching?

Finally, is there any best practices that could be observed while writing to the disk to limit the negative effects? For example, are small bursts of writing better than sustained write operations? Is it data throughput that's the problem or is it a file-system thing? If I open a file without closing it and continue to stream data there, is it better than if I open, write and close for each new piece of data?

Many thanks for any help you can provide, Dan.

Dan
  • 113
  • 2

1 Answers1

0

If I open a file without closing it and continue to stream data there, is it better than if I open, write and close for each new piece of data?

No. Closing or not closing a file where output is buffered makes a difference as to whether/when the data is visible to be read from the file, but this is distinct from whether/when it is physically written to disk.

In other words, when you flush a filehandle (e.g. by closing it), a separate process reading from the same file will now be able to read the data you flushed to the file, but this does not necessarily mean that file has literally been written out by the kernel. If it's in use, it is possibly cached, and it may only be that cache which is effected.

System disk caches are flushed (-> written out to a device) when sync is called on an entire filesystem. AFAIK there is no way to do this for a single file.

Another question is whether there are any heuristics software could use to detect whether the 'write limit' is approaching?

I very much doubt it, especially since you do not know much about the device. Numbers like that will be approximate and conservative, which is why I image devices are generally not built to fail at a pre-defined point: They fail when they fail, and since they could fail at any point, you might as well do what you can to check for and protect against loss because of that, period, rather than assuming everything is okay until ~N operations.

Run fsck whenever feasible (before mounting the filesystems). If this is a long-running device, determine a way to umount and fsck at intervals when the system is idle-ish.

goldilocks
  • 86,451
  • 30
  • 200
  • 258
  • Unfortunately we don't have the kind of access to the device that would allow unmounting and checking with fsck. Thanks a lot, very informative. – Dan May 01 '14 at 08:04
  • One more question, would compressing data before writing it help? I.e. less data written would mean less writes per sector (or whatever mtd devices have) over time? – Dan May 01 '14 at 08:16
  • If you can afford the processor cycles and the compression ratio is high enough I guess so. MTD's often (I think: usually now) use [wear leveling](http://en.wikipedia.org/wiki/Wear_levelling) so you needn't bother thinking about "writes per sector", etc -- that may not reflect anything about the physics of the device as the same sector may not be used repeatedly even if it appears to be. But a lower total volume of data is a total lower volume, if the significant unit of a "write cycle" is blocks or sectors then fewer in terms of total volume is better. – goldilocks May 01 '14 at 11:29
  • Thanks again. I've accepted your answer, but one more question occurs to me. I've read that typically one can expect 10k write cycles from this device, do you know if that is that per sector or for the whole drive? – Dan May 02 '14 at 07:59
  • It will be in relation to the total volume -- read that article on wear leveling and maybe have a [look here](http://unix.stackexchange.com/questions/96774/stress-testing-sd-cards-using-linux), point being you do not need to think about each sector or any other unit that is supposed to represent a particular physical location on the device *because there are no such units*. While the device uses sectors, etc. these are not exposed literally. When you write, e.g., to the first sector over and over again, that's just the first sector in the representation and not literally the first physical.. – goldilocks May 02 '14 at 10:44
  • ..meaning it won't be the same physical sector each time. The device itself juggles this to spread the wear out. So if you have 2GB total and write to (ostensibly) the *exact same* 4kB sector over and over, you can't do it just 10k times, you can do it 2000000000 / 4096 * 10000 ~= 9.7 billion times. I.e. if you have a decent volume and are only writing modest amounts, you should not experience any problem on most of the units; the fact that it is the same couple of files over and over doesn't make a difference. – goldilocks May 02 '14 at 10:50