I have an ext3 filesystem on a 4GB SD card in a remote unit.
It had a large log file causing the disk to be full. At that point I was unable to SSH into it so I had someone on site put the SD card into a Windows laptop and used Ext2FS to delete the log file (and to fix the network issues that were preventing my logging in).
I can now SSH in and that log file is gone but the disk space isn't back, leaving me running with only a few MB available.
What do I do?
I am restricted to what I can do via SSH logged into the unit (ie with the filesystem mounted and in use), or via TeamViewer to a Windows 8 laptop with an SD card reader in it.
On Fri, 17 Apr 2015 12:09:23 +0100 Mark Rogers mark@more-solutions.co.uk wrote:
I have an ext3 filesystem on a 4GB SD card in a remote unit.
It had a large log file causing the disk to be full. At that point I was unable to SSH into it so I had someone on site put the SD card into a Windows laptop and used Ext2FS to delete the log file (and to fix the network issues that were preventing my logging in).
I can now SSH in and that log file is gone but the disk space isn't back, leaving me running with only a few MB available.
What do I do?
Has Windows put the file into the Recycle bin, thus still taking up disc space?
I am restricted to what I can do via SSH logged into the unit (ie with the filesystem mounted and in use), or via TeamViewer to a Windows 8 laptop with an SD card reader in it.
Does Windows show the disc space as low on the card?
On 17 April 2015 at 12:50, Chris Walker alug_cdw@the-walker-household.co.uk wrote:
Has Windows put the file into the Recycle bin, thus still taking up disc space?
Any idea where it would show in the Linux filesystem if this is the case? At the moment the guy on site isn't available to move the card over to the laptop.
There is a big discrepancy between what "du /" and "df" can see: 800MB vs 2.6GB used* - there must be a way to "find" this?
* I found a separate 1GB directory of unused files which I have deleted, from Linux, since the initial post. Hence the figures above not matching the "few MB" remaining from 4GB, in case anyone was wondering.
On 17 Apr 2015, at 13:20, Mark Rogers mark@more-solutions.co.uk wrote:
There is a big discrepancy between what "du /" and "df" can see: 800MB vs 2.6GB used* - there must be a way to "find" this?
There’s always the blunt hammer approach:- find / -xdev -type f -size +100M
On Fri, 17 Apr 2015 13:20:58 +0100 Mark Rogers mark@more-solutions.co.uk wrote:
On 17 April 2015 at 12:50, Chris Walker alug_cdw@the-walker-household.co.uk wrote:
Has Windows put the file into the Recycle bin, thus still taking up disc space?
Any idea where it would show in the Linux filesystem if this is the case? At the moment the guy on site isn't available to move the card over to the laptop.
In the root of the card might be a folder called $Recycle.Bin which, if it's there will contain the files.
There is a big discrepancy between what "du /" and "df" can see: 800MB vs 2.6GB used* - there must be a way to "find" this?
If there's nothing important on the card, can you not format it?
On 17 April 2015 at 14:47, Chris Walker alug_cdw@the-walker-household.co.uk wrote:
In the root of the card might be a folder called $Recycle.Bin which, if it's there will contain the files.
Not there :-(
And in any case the more I think about this, if it were there then it would have shown in 'sudo du -h /' which it doesn't. Which is why William's suggestion to search for large files using "find" doesn't help either (I did try it, no files found). The file isn't "there" to find, however the space is still allocated.
If there's nothing important on the card, can you not format it?
It's the root filesystem of the device I'm SSH'ing into, so yes, it is important...
Where does "du" get it's information from? Are there any clues in the following? The "1 large file" sounds helpful but I can''t find any reference to what that means in practice. Note that fsck thinks I have 24% blocks used, which is about right, so why does "df" think I've used 74% of the disk?
Aside: I tried creating a large file to fill the disk (cp /dev/zero ~/tmpfile) and it exhausted available disk space at about 900M, agreeing with df.
$ sudo /sbin/fsck.ext3 -fnv /dev/sda2 e2fsck 1.42.5 (29-Jul-2012) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information
27366 inodes used (13.92%, out of 196608) 21 non-contiguous files (0.1%) 22 non-contiguous directories (0.1%) # of inodes with ind/dind/tind blocks: 1268/18/0 189248 blocks used (24.06%, out of 786432) 0 bad blocks 1 large file
22051 regular files 2444 directories 11 character device files 276 block device files 2 fifos 411 links 2572 symbolic links (2471 fast symbolic links) 1 socket $ df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 3756092 2638136 927156 74% / /dev/root 3756092 2638136 927156 74% / devtmpfs 256800 0 256800 0% /dev tmpfs 51376 212 51164 1% /run tmpfs 5120 0 5120 0% /run/lock tmpfs 102740 0 102740 0% /run/shm
Mark
On 17 Apr 15:01, Mark Rogers wrote:
On 17 April 2015 at 14:47, Chris Walker alug_cdw@the-walker-household.co.uk wrote:
In the root of the card might be a folder called $Recycle.Bin which, if it's there will contain the files.
Not there :-(
OK - So, the general reason for df and du to disagree is if there was a huge file, and the file was deleted *but* something still had a handle open on it, the space will not have been released...
You probably want to try something akin to:
sudo lsof | grep DEL
Cheers,
On 17 April 2015 at 15:45, Brett Parker iDunno@sommitrealweird.co.uk wrote:
OK - So, the general reason for df and du to disagree is if there was a huge file, and the file was deleted *but* something still had a handle open on it, the space will not have been released...
It's consistent over reboots and appears to have started after a ~2GB file was deleted via Ext2Fsd from Windows.
It's as if the space is still allocated but no longer in any directory listing, similar to what would happen if an open file was deleted but apparently different.
You probably want to try something akin to:
sudo lsof | grep DEL
Nothing other than a few entries relating to /dev/zero.
OK, so it looks like I've been missing a fairly major point: my boot drive (which is also my SD card) is sdb not sda.
fsck.ext3 -vnf /dev/sdb2 shows me loads of errors.
What can I do to force this partition to be checked (unattended) on reboot? "touch /forcefsck" hasn't done it (although /forcefsck has been removed after reboot).
Unattended is important, I won't have any access to it until it reboots and I can SSH into it.
On 17/04/15 18:01, Mark Rogers wrote:
OK, so it looks like I've been missing a fairly major point: my boot drive (which is also my SD card) is sdb not sda.
fsck.ext3 -vnf /dev/sdb2 shows me loads of errors.
What can I do to force this partition to be checked (unattended) on reboot? "touch /forcefsck" hasn't done it (although /forcefsck has been removed after reboot).
Unattended is important, I won't have any access to it until it reboots and I can SSH into it.
Just a thought - did you create forcefsk in the the correct place? Try creating one in each "root" directory of each SD card, then rebooting.
Sucking eggs time: If you for example had the root partition \ on sda1, but then had the \home directory provided by partition sdb1. To fsck both sd cards, you'd create forcefsck using touch \forcefsck, and touch \home\forcefsc.
(At least I think that's what you'd do!)
HTH Steve
On 17 April 2015 at 20:05, steve-ALUG@hst.me.uk wrote:
Just a thought - did you create forcefsk in the the correct place? Try creating one in each "root" directory of each SD card, then rebooting.
There is only one SD card, although the device has on-board flash which isn't being used (and which is /dev/sda, which confused me previously). It isn't mounted, although may well be where grub is installed (how do I check?)
"mount" says that / is /dev/root, and /dev/root is a symlink to /dev/sdb2
Something isn't quite right here; looking at the output of dumpe2fs the disk is well past the point at which a scheduled fsck should have run but it isn't happening, any ideas? (Mount count = 62, vs maximum mount count 23; last check Sept 2011, check interval 6 months, next check Feb 2012...)
(Also of note: the "Last mounted on" doesn't make any sense, as clearly it was last mounted as / on the device, but what it's showing is from mounting in a card reader on a colleagues' PC months ago.)
$ sudo /sbin/dumpe2fs /dev/sdb2 dumpe2fs 1.42.5 (29-Jul-2012) Filesystem volume name: dp-rootfs Last mounted on: /media/glenn/dp-rootfs Filesystem UUID: e671a042-62bc-4888-b8f6-7f68dbe6b941 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file Filesystem flags: unsigned_directory_hash Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 238560 Block count: 954009 Reserved block count: 47700 Free blocks: 277956 Free inodes: 212179 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 232 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 7952 Inode blocks per group: 497 Filesystem created: Thu Sep 1 18:00:30 2011 Last mount time: Fri Apr 17 16:48:48 2015 Last write time: Fri Apr 17 16:48:48 2015 Mount count: 62 Maximum mount count: 23 Last checked: Thu Sep 1 18:00:30 2011 Check interval: 15552000 (6 months) Next check after: Tue Feb 28 18:00:30 2012 Lifetime writes: 1736 kB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 First orphan inode: 135223 Default directory hash: half_md4 Directory Hash Seed: 947f9517-4f4a-4c2d-b87a-7315cc8869eb Journal backup: inode blocks Journal features: journal_incompat_revoke Journal size: 64M Journal length: 16384 Journal sequence: 0x00253b6b Journal start: 8660
Mark
On 20/04/15 09:42, Mark Rogers wrote:
On 17 April 2015 at 20:05, steve-ALUG@hst.me.uk wrote:
Just a thought - did you create forcefsk in the the correct place? Try creating one in each "root" directory of each SD card, then rebooting.
There is only one SD card, although the device has on-board flash which isn't being used (and which is /dev/sda, which confused me previously).
I suspect it's irrelevant, but if you have multiple partitions on the SD card, create a forcefsck in each of the partitions and try again, but I suspect that's futile. I'll comment below.
It isn't mounted, although may well be where grub is installed (how do I check?)
er.... Let Me Google That For You.... http://lmgtfy.com/?q=where+is+grub+installed
"mount" says that / is /dev/root, and /dev/root is a symlink to /dev/sdb2
Something isn't quite right here; looking at the output of dumpe2fs the disk is well past the point at which a scheduled fsck should have run but it isn't happening, any ideas? (Mount count = 62, vs maximum mount count 23; last check Sept 2011, check interval 6 months, next check Feb 2012...)
(Also of note: the "Last mounted on" doesn't make any sense, as clearly it was last mounted as / on the device, but what it's showing is from mounting in a card reader on a colleagues' PC months ago.)
$ sudo /sbin/dumpe2fs /dev/sdb2 dumpe2fs 1.42.5 (29-Jul-2012) Filesystem volume name: dp-rootfs Last mounted on: /media/glenn/dp-rootfs Filesystem UUID: e671a042-62bc-4888-b8f6-7f68dbe6b941 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file Filesystem flags: unsigned_directory_hash Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 238560 Block count: 954009 Reserved block count: 47700 Free blocks: 277956 Free inodes: 212179 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 232 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 7952 Inode blocks per group: 497 Filesystem created: Thu Sep 1 18:00:30 2011 Last mount time: Fri Apr 17 16:48:48 2015 Last write time: Fri Apr 17 16:48:48 2015 Mount count: 62 Maximum mount count: 23 Last checked: Thu Sep 1 18:00:30 2011 Check interval: 15552000 (6 months) Next check after: Tue Feb 28 18:00:30 2012 Lifetime writes: 1736 kB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 First orphan inode: 135223 Default directory hash: half_md4 Directory Hash Seed: 947f9517-4f4a-4c2d-b87a-7315cc8869eb Journal backup: inode blocks Journal features: journal_incompat_revoke Journal size: 64M Journal length: 16384 Journal sequence: 0x00253b6b Journal start: 8660
Mark
I suspect it's borked, big-time rub-a-dub-style. I don't think you're going to fix this with a power-on fsck. I think you're going to have to attach it to another computer and run fsck on it whilst the drive isn't mounted on the file system. I suspect that copying off the files, then formatting it and starting again is going to be the quickest way of proceeding. Alternatively, if you need to keep the machine running mostly, could you duplicate the sd card with DD or similar, then swap to using the duplicated sd card, then send the original faulty card to you for you to check at your leisure?
Good luck, Steve
On 20 April 2015 at 21:07, steve-ALUG@hst.me.uk wrote:
I suspect it's borked, big-time rub-a-dub-style. I don't think you're going to fix this with a power-on fsck.
I think you might be right, or at least while it's working the risk of forcing a power-on fsck might outweigh the benefits.
The disk (SD card) was created from a disk image of a working system and looking at the output from dumpe2fs the details predate the creation of this card. It''s almost like some of the settings there have become read-only (is that even possible?).
-
Brett Parker -
Chris Walker -
Mark Rogers -
steve-ALUG＠hst.me.uk -
William Hill