On 17 Apr 2015, at 13:20, Mark Rogers <mark@more-solutions.co.uk> wrote:

There is a big discrepancy between what "du /" and "df" can see: 800MB vs 2.6GB used* - there must be a way to "find" this?

There’s always the blunt hammer approach:- find / -xdev -type f -size +100M

On 17 April 2015 at 14:47, Chris Walker <alug_cdw@the-walker-household.co.uk> wrote:

In the root of the card might be a folder called $Recycle.Bin which, if it's there will contain the files.

Not there :-( And in any case the more I think about this, if it were there then it would have shown in 'sudo du -h /' which it doesn't. Which is why William's suggestion to search for large files using "find" doesn't help either (I did try it, no files found). The file isn't "there" to find, however the space is still allocated.

If there's nothing important on the card, can you not format it?

It's the root filesystem of the device I'm SSH'ing into, so yes, it is important... Where does "du" get it's information from? Are there any clues in the following? The "1 large file" sounds helpful but I can''t find any reference to what that means in practice. Note that fsck thinks I have 24% blocks used, which is about right, so why does "df" think I've used 74% of the disk? Aside: I tried creating a large file to fill the disk (cp /dev/zero ~/tmpfile) and it exhausted available disk space at about 900M, agreeing with df. $ sudo /sbin/fsck.ext3 -fnv /dev/sda2 e2fsck 1.42.5 (29-Jul-2012) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information 27366 inodes used (13.92%, out of 196608) 21 non-contiguous files (0.1%) 22 non-contiguous directories (0.1%) # of inodes with ind/dind/tind blocks: 1268/18/0 189248 blocks used (24.06%, out of 786432) 0 bad blocks 1 large file 22051 regular files 2444 directories 11 character device files 276 block device files 2 fifos 411 links 2572 symbolic links (2471 fast symbolic links) 1 socket $ df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 3756092 2638136 927156 74% / /dev/root 3756092 2638136 927156 74% / devtmpfs 256800 0 256800 0% /dev tmpfs 51376 212 51164 1% /run tmpfs 5120 0 5120 0% /run/lock tmpfs 102740 0 102740 0% /run/shm Mark -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER

On 17 April 2015 at 15:45, Brett Parker <iDunno@sommitrealweird.co.uk> wrote:

OK - So, the general reason for df and du to disagree is if there was a huge file, and the file was deleted *but* something still had a handle open on it, the space will not have been released...

It's consistent over reboots and appears to have started after a ~2GB file was deleted via Ext2Fsd from Windows. It's as if the space is still allocated but no longer in any directory listing, similar to what would happen if an open file was deleted but apparently different.

You probably want to try something akin to:

sudo lsof | grep DEL

Nothing other than a few entries relating to /dev/zero. -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER

On 17/04/15 18:01, Mark Rogers wrote:

OK, so it looks like I've been missing a fairly major point: my boot drive (which is also my SD card) is sdb not sda.

fsck.ext3 -vnf /dev/sdb2 shows me loads of errors.

What can I do to force this partition to be checked (unattended) on reboot? "touch /forcefsck" hasn't done it (although /forcefsck has been removed after reboot).

Unattended is important, I won't have any access to it until it reboots and I can SSH into it.

Just a thought - did you create forcefsk in the the correct place? Try creating one in each "root" directory of each SD card, then rebooting. Sucking eggs time: If you for example had the root partition \ on sda1, but then had the \home directory provided by partition sdb1. To fsck both sd cards, you'd create forcefsck using touch \forcefsck, and touch \home\forcefsc. (At least I think that's what you'd do!) HTH Steve

On 20/04/15 09:42, Mark Rogers wrote:

On 17 April 2015 at 20:05, <steve-ALUG@hst.me.uk> wrote:

...

Just a thought - did you create forcefsk in the the correct place? Try creating one in each "root" directory of each SD card, then rebooting. There is only one SD card, although the device has on-board flash which isn't being used (and which is /dev/sda, which confused me previously).

I suspect it's irrelevant, but if you have multiple partitions on the SD card, create a forcefsck in each of the partitions and try again, but I suspect that's futile. I'll comment below.

It isn't mounted, although may well be where grub is installed (how do I check?)

er.... Let Me Google That For You.... http://lmgtfy.com/?q=where+is+grub+installed

"mount" says that / is /dev/root, and /dev/root is a symlink to /dev/sdb2

Something isn't quite right here; looking at the output of dumpe2fs the disk is well past the point at which a scheduled fsck should have run but it isn't happening, any ideas? (Mount count = 62, vs maximum mount count 23; last check Sept 2011, check interval 6 months, next check Feb 2012...)

(Also of note: the "Last mounted on" doesn't make any sense, as clearly it was last mounted as / on the device, but what it's showing is from mounting in a card reader on a colleagues' PC months ago.)

$ sudo /sbin/dumpe2fs /dev/sdb2 dumpe2fs 1.42.5 (29-Jul-2012) Filesystem volume name: dp-rootfs Last mounted on: /media/glenn/dp-rootfs Filesystem UUID: e671a042-62bc-4888-b8f6-7f68dbe6b941 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file Filesystem flags: unsigned_directory_hash Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 238560 Block count: 954009 Reserved block count: 47700 Free blocks: 277956 Free inodes: 212179 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 232 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 7952 Inode blocks per group: 497 Filesystem created: Thu Sep 1 18:00:30 2011 Last mount time: Fri Apr 17 16:48:48 2015 Last write time: Fri Apr 17 16:48:48 2015 Mount count: 62 Maximum mount count: 23 Last checked: Thu Sep 1 18:00:30 2011 Check interval: 15552000 (6 months) Next check after: Tue Feb 28 18:00:30 2012 Lifetime writes: 1736 kB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 First orphan inode: 135223 Default directory hash: half_md4 Directory Hash Seed: 947f9517-4f4a-4c2d-b87a-7315cc8869eb Journal backup: inode blocks Journal features: journal_incompat_revoke Journal size: 64M Journal length: 16384 Journal sequence: 0x00253b6b Journal start: 8660

Mark

I suspect it's borked, big-time rub-a-dub-style. I don't think you're going to fix this with a power-on fsck. I think you're going to have to attach it to another computer and run fsck on it whilst the drive isn't mounted on the file system. I suspect that copying off the files, then formatting it and starting again is going to be the quickest way of proceeding. Alternatively, if you need to keep the machine running mostly, could you duplicate the sd card with DD or similar, then swap to using the duplicated sd card, then send the original faulty card to you for you to check at your leisure? Good luck, Steve