29

Following an unclean shutdown on an SD card based device, I took the SD card out to fsck the root filesystem. This led to variations on the following:

e2fsck 1.43.1 (08-Jun-2016)
/dev/sdc2: recovering journal
Superblock needs_recovery flag is clear, but journal has data.
Run journal anyway<y>? no
Clear journal<y>? no
e2fsck: unable to set superblock flags on /dev/sdc2

Here I've answered "no" both times but there is no sequence of yes/no that does not immediately lead to the same outcome.

The filesystem can be mounted and on casual inspection appears okay; it also works fine in the device, and that's the root filesystem (actually it turned out to be not quite fine, see comments; tldr some irretrievably corrupted directories).

I dd'd the partition (8 GB) to a file, and tried fsck on that. Interestingly:

e2fsck 1.43.1 (08-Jun-2016)
plush.rootfs: recovering journal
Clearing orphaned inode 18290 (uid=0, gid=0, mode=0100644, size=34096)
Clearing orphaned inode 18270 (uid=0, gid=0, mode=0100644, size=38916)
Clearing orphaned inode 18250 (uid=0, gid=0, mode=0100644, size=1128076)
Clearing orphaned inode 11411 (uid=0, gid=0, mode=0100644, size=293108)
Setting free inodes count to 406127 (was 408580)
Setting free blocks count to 1305622 (was 1347486)
plush.rootfs: clean, 60209/466336 files, 604906/1910528 blocks (check after next mount)

A subsequent fsck passed clean, the image can be mounted, and fsck -f after that passes as well.

But the filesystem on the card from which the raw block copy image was created still has the same problem -- except that the systemd-fsck which takes place during boot logs the filesystem as "clean". Subsequently though, a proper shutdown, taking the card out, and trying fsck again from another box presents the same error.

Whenever the original is mounted on another machine, syslog notes:

kernel: EXT4-fs (sdc2): 4 orphan inodes deleted
kernel: EXT4-fs (sdc2): recovery complete

Since I have it all backed up, I'm open to trying anything here. I could simply forget about this and reburn the partition from the apparently fixed image, but that does not seem like a very satisfactory solution, since it means assuming fsck cryptically failed at solving a minor looking problem.

I suspect this is going to turn into a "request for official documentation" question regarding stuff like needs recovery_flag (or just plain "What does this mean?" question), so any suggestions along those lines are appreciated.

goldilocks
  • 86,451
  • 30
  • 200
  • 258
  • Anything in the kernel logs about device errors? This wouldn't be the first time an SD card suddenly became read-only. – Mark Plotnick Dec 03 '16 at 23:22
  • @MarkPlotnick No, and it is writable. The last thing in the log from before the problem was systemd restarting (the device is headless and became unresponsive following a long `apt upgrade`). After that it logs a normal boot -- and the systemd-fsck says "clean" (I'll edit that in), but trying fsck outside of that context still fails. – goldilocks Dec 03 '16 at 23:36
  • Your fsck on the copy cleared 4 inodes, but fixed the free inode count by reducing it by 2453 inodes! That's enormous. Check that the device is getting enough power. – meuh Dec 04 '16 at 11:08
  • @meuh I notice whenever it's mounted on the big box syslog refers to those 4 inodes (edited in above). Some stuff on the fs did turn out to be messed up (updated kernel modules! \O/) so I've burned a new card and will hang on to the old one in case I have a chance to dig into this more. It isn't exactly new -- an unbranded bargain bin class 10 card, in (light duty) use 24/7 for possibly a few years so...I don't think there's any way to verify an SD card is definitively defunct, but I guess it could be that. The power should be okay but might be iffy under certain conditions. – goldilocks Dec 04 '16 at 16:48
  • 2
    Doesn't it really suck when the very tool that's supposed to fix your problem, doesn't work because of the nature of the problem? Conclusion: The tool is bad and should be fixed. – Marc.2377 Jul 07 '17 at 02:42
  • Comment posted as (now probably deleted) answer from @TomásMayr : *"Just came across this answer while trying to fix my Samsung Evo 16gb microSD, so to complement Malvineous, it might have something to do with this specific model."* – peterh Oct 01 '17 at 00:58

5 Answers5

28

I just ran into this same problem. After debugging the issue with the e2fsck maintainer, we realised that the SD card was broken. It was accepting writes without error, but it wasn't actually writing the data to the card. The SD card was effectively read only.

It seems the card had gone into some sort of failsafe mode, where the data could still be read, but nothing written.

The e2fsck message unable to set superblock flags means it tried to write to the superblock to mark the journal as processed, which happened without error, but when it went to read the superblock back again it still indicated that the journal needed to be replayed. In other words, the changes written to the superblock were not saved on the storage medium.

The card I am using that has this problem is a Samsung Evo 16GB microSD, which I mention just in case it's a common problem with these cards.

I was able to test this by using dd to write 4096 bytes from /dev/zero onto the card at block 0, then I read back from the card and instead of getting all zeroes as I should, I still got the original unchanged ext4 superblock.

I'm now in the process of moving the data onto a new card and then seeing if I can get a replacement from Samsung, who appear to offer a 10 year warranty on SD cards.

UPDATE: Samsung replaced the 16GB card with a 32GB one in the same Evo series, so guess I can't complain too much!

Malvineous
  • 6,524
  • 5
  • 52
  • 76
  • *"where the data could still be read, but nothing written"* -> The fs was writable. – goldilocks Oct 01 '17 at 13:05
  • 1
    @goldilocks: Sounds like your fs superblock may not have been writable though. Also, my fs appeared writable thanks to caching, it was only after unmounting and remounting I noticed any changes were lost. – Malvineous Oct 02 '17 at 01:27
  • It was not an illusion due to caching. – goldilocks Oct 02 '17 at 12:57
  • 1
    amazing: I have the same issue, with the same card ! looks like it is "read only" from now on, that is weird for sure – MoonCactus Jun 20 '20 at 19:37
  • @MoonCactus: Interesting! I've since had it happen to another two cards of the same model, so looks like these particular ones have longevity issues. – Malvineous Jun 22 '20 at 11:55
  • 1
    @Malvineous thanks, this is good to know! I bought from Samsung specifically for the hope I was reducing such a risk, and it ends up the other way. It gets statistically significant enough for me to try something else... but what? May be it was only a batch of these 16GB cards... There might be no extrapolation to be made from one product to another even in the same brand! – MoonCactus Jun 23 '20 at 13:24
16

I know this is an old thread, but, I figured I'd offer some insight.

This seems to be the way sd cards die a natural death. The number of read/write cycles sd cards can endure is substantially lower than most other mediums considered 'read/write'. When that has been exhausted, the card will go into read only mode, but won't inform you of such. Lots of things will think they're writing to the card thanks to OS caching, etc., but nothing ever sticks.

A great way to kill an sd card is to mount it as a swap partition or something that's very read/write intensive. You'd be surprised how quickly you can kill a card that way. I've found that running knoppix off of an sd card or usb thumb drive will only last a month or two, depending on the quality of the card and the intensity of knoppix usage. (I've since switched to running knoppix off of a usb SSD drive which has lasted a couple of years now).

Joe
  • 161
  • 1
  • 2
  • 2
    If you can, please provide a source for it. It would be great. – Lyokolux Jun 22 '21 at 12:46
  • It's interesting because after the first Samsung SD card did this for me, I made sure to mount the next one read-only, so that the OS would never write anything to it. Even with no writes beyond the initial set up, the cards still went read-only after a similar amount of time. I'm not really sure what would explain that behaviour, but it certainly wasn't exhausting the number of write cycles. On the plus side, the Raspberry Pi each card was in would still boot and run just fine, because even though the card was read-only, it was configured to be mounted read-only too, so it wasn't a problem. – Malvineous Mar 30 '23 at 07:02
1

Had the same problem, unable to mount, unable to read superblock, unable to set superblock flags, unable to use superblock backup.

The problem I had was write lock switch on the card adapter ;)

sebo
  • 31
  • 2
1

For me, the problem disappeared after using the card with another microSD-to-USB adapter. Then it worked fine with the first adapter as well. I figure in my case it was a bad contact issue.

sizif
  • 121
  • 1
  • 4
-1

Twice now I've had similar problems for a microSD and it turned out to be the SD adapter, not the microSD. Changing to a different SD adapter, even when the first one was the same brand as the microSD, fixed the problem of being unable to write superblock flags, and having I/O errors.

brianc
  • 1