On 20/12/12 15:27, Mark Rogers wrote:
It appears that the uptime was >20 days which means I didn't reboot it when I thought I had. I just rebooted it and the array is back.
Hmm, it seems to have happened again, only this time it's still recent enough to be in the logs. Coincidentally (or otherwise) the uptime on the box is now 19 days, so similar to last time.
As things stand: I have two RAID5 arrays, one (md0) comprising four local disks and is fine, the other (md1) comprising four disks (partitions sd[fghi]1) in a USB3 caddy, which has vanished. At 13:29 yesterday I received five emails from mdadm telling me, respectively, FailSpare event on sdi1, Fail events on sdf1, sdg1 and sdh1, and finally Fail event on the array md1.
I wasn't in the office when this happened but having come in this morning the external drive is up and running and according to fdisk all four partitions are currently present, but md1 is still inaccessible. Output from mdstat/fdisk/syslog included below.
My interpretation is that (a) "something" happened to cause all the drives to become inaccessible at once, (b) the "something" resolved itself (within a very short period of time - 2s in this case), (c) the array didn't come back up (not that I'd necessarily expect it to, but I might like to find a way to achieve this in future).
Two questions: (1) How can I bring the array back up without restarting the server, and (b) any clues as to what the "something" was and how to avoid it?
Hopefully relevant info follows (with no further comment from me)...
Mark
$ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md1 : active raid5 super 1.2 level 5, 512k chunk, algorithm 2 [4/0] [____]
md0 : active raid5 sdc1[2] sdd1[3] sda1[0] sdb1[1] 5860535808 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
unused devices: <none>
$ sudo fdisk -l /dev/sd[fghi] [...] Device Boot Start End Blocks Id System /dev/sdf1 2048 3907029167 1953513560 fd Linux RAID autodetect [...] /dev/sdg1 2048 3907029167 1953513560 fd Linux RAID autodetect [...] /dev/sdh1 2048 3907029167 1953513560 fd Linux RAID autodetect [...] /dev/sdi1 2048 3907029167 1953513560 fd Linux RAID autodetect
$ cat /var/log/syslog.1 [...] Jan 8 13:29:43 FileServer kernel: [1635170.968066] usb 6-2: USB disconnect, device number 2 Jan 8 13:29:43 FileServer kernel: [1635170.968072] sd 6:0:0:2: Device offlined - not ready after error recovery Jan 8 13:29:43 FileServer kernel: [1635170.968092] sd 6:0:0:2: [sdh] Unhandled error code Jan 8 13:29:43 FileServer kernel: [1635170.968097] sd 6:0:0:2: [sdh] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK Jan 8 13:29:43 FileServer kernel: [1635170.968104] sd 6:0:0:2: [sdh] CDB: Read(10): 28 40 de c4 0b 28 00 00 f0 00 Jan 8 13:29:43 FileServer kernel: [1635170.968123] end_request: I/O error, dev sdh, sector 3737389864 Jan 8 13:29:43 FileServer kernel: [1635170.968568] sd 6:0:0:2: rejecting I/O to offline device Jan 8 13:29:43 FileServer kernel: [1635170.968917] sd 6:0:0:2: rejecting I/O to offline device Jan 8 13:29:43 FileServer kernel: [1635170.969272] sd 6:0:0:2: rejecting I/O to offline device Jan 8 13:29:43 FileServer kernel: [1635170.969681] sd 6:0:0:2: rejecting I/O to offline device Jan 8 13:29:43 FileServer kernel: [1635170.969973] sd 6:0:0:2: rejecting I/O to offline device Jan 8 13:29:43 FileServer kernel: [1635170.970263] sd 6:0:0:2: rejecting I/O to offline device Jan 8 13:29:43 FileServer kernel: [1635170.970583] sd 6:0:0:0: [sdf] Unhandled error code Jan 8 13:29:43 FileServer kernel: [1635170.970588] sd 6:0:0:0: [sdf] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jan 8 13:29:43 FileServer kernel: [1635170.970594] sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 de c4 0b a8 00 00 f0 00 Jan 8 13:29:43 FileServer kernel: [1635170.970612] end_request: I/O error, dev sdf, sector 3737389992 Jan 8 13:29:43 FileServer kernel: [1635170.971069] sd 6:0:0:1: [sdg] Unhandled error code Jan 8 13:29:43 FileServer kernel: [1635170.971074] sd 6:0:0:1: [sdg] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jan 8 13:29:43 FileServer kernel: [1635170.971081] sd 6:0:0:1: [sdg] CDB: Read(10): 28 20 de c4 09 b8 00 00 f0 00 Jan 8 13:29:43 FileServer kernel: [1635170.971100] end_request: I/O error, dev sdg, sector 3737389496 Jan 8 13:29:43 FileServer kernel: [1635170.971832] sd 6:0:0:3: [sdi] Unhandled error code Jan 8 13:29:43 FileServer kernel: [1635170.971837] sd 6:0:0:3: [sdi] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jan 8 13:29:43 FileServer kernel: [1635170.971844] sd 6:0:0:3: [sdi] CDB: Read(10): 28 60 de c4 09 58 00 00 f0 00 Jan 8 13:29:43 FileServer kernel: [1635170.971865] end_request: I/O error, dev sdi, sector 3737389400 Jan 8 13:29:43 FileServer kernel: [1635170.972329] sd 6:0:0:3: [sdi] Unhandled error code Jan 8 13:29:43 FileServer kernel: [1635170.972334] sd 6:0:0:3: [sdi] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jan 8 13:29:43 FileServer kernel: [1635170.972341] sd 6:0:0:3: [sdi] CDB: Read(10): 28 60 de c4 0a 48 00 00 f0 00 Jan 8 13:29:43 FileServer kernel: [1635170.972360] end_request: I/O error, dev sdi, sector 3737389640 Jan 8 13:29:43 FileServer kernel: [1635170.972854] sd 6:0:0:3: [sdi] Unhandled error code Jan 8 13:29:43 FileServer kernel: [1635170.972858] sd 6:0:0:3: [sdi] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jan 8 13:29:43 FileServer kernel: [1635170.972864] sd 6:0:0:3: [sdi] CDB: Read(10): 28 60 de c4 0b 38 00 00 f0 00 Jan 8 13:29:43 FileServer kernel: [1635170.972881] end_request: I/O error, dev sdi, sector 3737389880 Jan 8 13:29:43 FileServer kernel: [1635170.973282] sd 6:0:0:3: [sdi] Unhandled error code Jan 8 13:29:43 FileServer kernel: [1635170.973286] sd 6:0:0:3: [sdi] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jan 8 13:29:43 FileServer kernel: [1635170.973292] sd 6:0:0:3: [sdi] CDB: Write(10): 2a 60 de c4 09 58 00 00 60 00 Jan 8 13:29:43 FileServer kernel: [1635170.973309] end_request: I/O error, dev sdi, sector 3737389400 Jan 8 13:29:43 FileServer kernel: [1635170.973617] md/raid:md1: Disk failure on sdi1, disabling device. Jan 8 13:29:43 FileServer kernel: [1635170.973620] md/raid:md1: Operation continuing on 3 devices. Jan 8 13:29:43 FileServer kernel: [1635170.973626] sd 6:0:0:3: [sdi] Unhandled error code Jan 8 13:29:43 FileServer kernel: [1635170.973635] sd 6:0:0:3: [sdi] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jan 8 13:29:43 FileServer kernel: [1635170.973641] sd 6:0:0:3: [sdi] CDB: Read(10): 28 60 de c4 0c 28 00 00 f0 00 Jan 8 13:29:43 FileServer kernel: [1635170.973679] end_request: I/O error, dev sdi, sector 3737390120 Jan 8 13:29:43 FileServer kernel: [1635170.973707] md/raid:md1: read error not correctable (sector 3737388072 on sdi1). Jan 8 13:29:43 FileServer kernel: [1635170.973734] md/raid:md1: read error not correctable (sector 3737388080 on sdi1). Jan 8 13:29:43 FileServer kernel: [1635170.973743] md/raid:md1: read error not correctable (sector 3737388088 on sdi1). Jan 8 13:29:43 FileServer kernel: [1635170.973759] md/raid:md1: read error not correctable (sector 3737388096 on sdi1). Jan 8 13:29:43 FileServer kernel: [1635170.973785] md/raid:md1: read error not correctable (sector 3737388104 on sdi1). Jan 8 13:29:43 FileServer kernel: [1635170.973799] md/raid:md1: read error not correctable (sector 3737388112 on sdi1). Jan 8 13:29:43 FileServer kernel: [1635170.973818] md/raid:md1: read error not correctable (sector 3737388120 on sdi1). Jan 8 13:29:43 FileServer kernel: [1635170.973850] md/raid:md1: read error not correctable (sector 3737388128 on sdi1). Jan 8 13:29:43 FileServer kernel: [1635170.973868] md/raid:md1: read error not correctable (sector 3737388136 on sdi1). Jan 8 13:29:43 FileServer kernel: [1635170.973880] md/raid:md1: read error not correctable (sector 3737388144 on sdi1). Jan 8 13:29:43 FileServer kernel: [1635170.974029] sd 6:0:0:3: [sdi] Unhandled error code Jan 8 13:29:43 FileServer kernel: [1635170.974034] sd 6:0:0:3: [sdi] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jan 8 13:29:43 FileServer kernel: [1635170.974038] sd 6:0:0:3: [sdi] CDB: Read(10): 28 60 de c4 0d 18 00 00 f0 00 Jan 8 13:29:43 FileServer kernel: [1635170.974075] end_request: I/O error, dev sdi, sector 3737390360 Jan 8 13:29:43 FileServer kernel: [1635170.974277] sd 6:0:0:3: [sdi] Unhandled error code Jan 8 13:29:43 FileServer kernel: [1635170.974293] sd 6:0:0:3: [sdi] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jan 8 13:29:43 FileServer kernel: [1635170.974308] sd 6:0:0:3: [sdi] CDB: Read(10): 28 60 de c4 0e 08 00 00 f0 00 Jan 8 13:29:43 FileServer kernel: [1635170.974343] end_request: I/O error, dev sdi, sector 3737390600 Jan 8 13:29:43 FileServer kernel: [1635170.974546] sd 6:0:0:3: [sdi] Unhandled error code Jan 8 13:29:43 FileServer kernel: [1635170.974564] sd 6:0:0:3: [sdi] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jan 8 13:29:43 FileServer kernel: [1635170.974579] sd 6:0:0:3: [sdi] CDB: Read(10): 28 60 de c4 0e f8 00 00 f0 00 Jan 8 13:29:43 FileServer kernel: [1635170.974630] end_request: I/O error, dev sdi, sector 3737390840 Jan 8 13:29:43 FileServer kernel: [1635170.974817] sd 6:0:0:3: [sdi] Unhandled error code Jan 8 13:29:43 FileServer kernel: [1635170.974839] sd 6:0:0:3: [sdi] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jan 8 13:29:43 FileServer kernel: [1635170.974876] sd 6:0:0:3: [sdi] CDB: Read(10): 28 60 de c4 0f e8 00 00 f0 00 Jan 8 13:29:43 FileServer kernel: [1635170.974907] end_request: I/O error, dev sdi, sector 3737391080 Jan 8 13:29:43 FileServer kernel: [1635170.975091] sd 6:0:0:3: [sdi] Unhandled error code Jan 8 13:29:43 FileServer kernel: [1635170.975108] sd 6:0:0:3: [sdi] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jan 8 13:29:43 FileServer kernel: [1635170.975127] sd 6:0:0:3: [sdi] CDB: Read(10): 28 60 de c4 10 d8 00 00 80 00 Jan 8 13:29:43 FileServer kernel: [1635170.975159] end_request: I/O error, dev sdi, sector 3737391320 Jan 8 13:29:43 FileServer kernel: [1635170.976385] sd 6:0:0:2: rejecting I/O to offline device Jan 8 13:29:43 FileServer kernel: [1635170.976659] md/raid:md1: Disk failure on sdh1, disabling device. Jan 8 13:29:43 FileServer kernel: [1635170.976662] md/raid:md1: Operation continuing on 2 devices. Jan 8 13:29:43 FileServer kernel: [1635170.977277] md/raid:md1: Disk failure on sdg1, disabling device. Jan 8 13:29:43 FileServer kernel: [1635170.977280] md/raid:md1: Operation continuing on 1 devices. Jan 8 13:29:43 FileServer kernel: [1635170.978051] md/raid:md1: Disk failure on sdf1, disabling device. Jan 8 13:29:43 FileServer kernel: [1635170.978053] md/raid:md1: Operation continuing on 0 devices. Jan 8 13:29:43 FileServer kernel: [1635170.978809] md: md1: data-check done. Jan 8 13:29:43 FileServer kernel: [1635170.979427] RAID conf printout: Jan 8 13:29:43 FileServer kernel: [1635170.979433] --- level:5 rd:4 wd:0 Jan 8 13:29:43 FileServer kernel: [1635170.979437] disk 0, o:0, dev:sdf1 Jan 8 13:29:43 FileServer kernel: [1635170.979441] disk 1, o:0, dev:sdg1 Jan 8 13:29:43 FileServer kernel: [1635170.979445] disk 2, o:0, dev:sdh1 Jan 8 13:29:43 FileServer kernel: [1635170.979449] disk 3, o:0, dev:sdi1 Jan 8 13:29:43 FileServer kernel: [1635170.992052] RAID conf printout: Jan 8 13:29:43 FileServer kernel: [1635170.992057] --- level:5 rd:4 wd:0 Jan 8 13:29:43 FileServer kernel: [1635170.992062] disk 0, o:0, dev:sdf1 Jan 8 13:29:43 FileServer kernel: [1635170.992065] disk 1, o:0, dev:sdg1 Jan 8 13:29:43 FileServer kernel: [1635170.992068] disk 3, o:0, dev:sdi1 Jan 8 13:29:43 FileServer kernel: [1635170.992076] RAID conf printout: Jan 8 13:29:43 FileServer kernel: [1635170.992078] --- level:5 rd:4 wd:0 Jan 8 13:29:43 FileServer kernel: [1635170.992081] disk 0, o:0, dev:sdf1 Jan 8 13:29:43 FileServer kernel: [1635170.992083] disk 1, o:0, dev:sdg1 Jan 8 13:29:43 FileServer kernel: [1635170.992086] disk 3, o:0, dev:sdi1 Jan 8 13:29:43 FileServer kernel: [1635171.012042] RAID conf printout: Jan 8 13:29:43 FileServer kernel: [1635171.012051] --- level:5 rd:4 wd:0 Jan 8 13:29:43 FileServer kernel: [1635171.012057] disk 1, o:0, dev:sdg1 Jan 8 13:29:43 FileServer kernel: [1635171.012061] disk 3, o:0, dev:sdi1 Jan 8 13:29:43 FileServer kernel: [1635171.012069] RAID conf printout: Jan 8 13:29:43 FileServer kernel: [1635171.012072] --- level:5 rd:4 wd:0 Jan 8 13:29:43 FileServer kernel: [1635171.012074] disk 1, o:0, dev:sdg1 Jan 8 13:29:43 FileServer kernel: [1635171.012077] disk 3, o:0, dev:sdi1 Jan 8 13:29:43 FileServer kernel: [1635171.032038] RAID conf printout: Jan 8 13:29:43 FileServer kernel: [1635171.032043] --- level:5 rd:4 wd:0 Jan 8 13:29:43 FileServer kernel: [1635171.032047] disk 1, o:0, dev:sdg1 Jan 8 13:29:43 FileServer kernel: [1635171.032054] RAID conf printout: Jan 8 13:29:43 FileServer kernel: [1635171.032057] --- level:5 rd:4 wd:0 Jan 8 13:29:43 FileServer kernel: [1635171.032060] disk 1, o:0, dev:sdg1 Jan 8 13:29:43 FileServer kernel: [1635171.060015] RAID conf printout: Jan 8 13:29:43 FileServer kernel: [1635171.060020] --- level:5 rd:4 wd:0 Jan 8 13:29:43 FileServer mdadm[1354]: Fail event detected on md device /dev/md1, component device /dev/sdg1 Jan 8 13:29:43 FileServer mdadm[1354]: Fail event detected on md device /dev/md1, component device /dev/sdh1 Jan 8 13:29:43 FileServer kernel: [1635171.373837] md: unbind<sdf1> Jan 8 13:29:43 FileServer kernel: [1635171.373942] md: export_rdev(sdf1) Jan 8 13:29:43 FileServer mdadm[1354]: Fail event detected on md device /dev/md1 Jan 8 13:29:43 FileServer kernel: [1635171.530156] md: unbind<sdg1> Jan 8 13:29:43 FileServer kernel: [1635171.540089] md: export_rdev(sdg1) Jan 8 13:29:43 FileServer kernel: [1635171.549711] md: unbind<sdh1> Jan 8 13:29:43 FileServer kernel: [1635171.564043] md: export_rdev(sdh1) Jan 8 13:29:43 FileServer kernel: [1635171.567696] md: unbind<sdi1> Jan 8 13:29:43 FileServer kernel: [1635171.584038] md: export_rdev(sdi1) Jan 8 13:29:43 FileServer mdadm[1354]: FailSpare event detected on md device /dev/md1, component device /dev/sdi1 Jan 8 13:29:43 FileServer mdadm[1354]: RebuildFinished event detected on md device /dev/md1 Jan 8 13:29:43 FileServer mdadm[1354]: Fail event detected on md device /dev/md1, component device /dev/sdf1 Jan 8 13:29:43 FileServer mdadm[1354]: SpareActive event detected on md device /dev/md1, component device /dev/sdg1 Jan 8 13:29:43 FileServer mdadm[1354]: SpareActive event detected on md device /dev/md1, component device /dev/sdh1
Jan 8 13:29:45 FileServer kernel: [1635173.360025] usb 6-2: new high-speed USB device number 3 using xhci_hcd Jan 8 13:29:45 FileServer kernel: [1635173.381147] usb 6-2: ep 0x81 - rounding interval to 32768 microframes, ep desc says 0 microframes Jan 8 13:29:45 FileServer kernel: [1635173.381158] usb 6-2: ep 0x2 - rounding interval to 32768 microframes, ep desc says 0 microframes Jan 8 13:29:45 FileServer kernel: [1635173.384334] scsi7 : usb-storage 6-2:1.0 Jan 8 13:29:46 FileServer kernel: [1635174.385997] scsi 7:0:0:0: Direct-Access WDC WD20 EARX-00PASB0 PQ: 0 ANSI: 2 CCS Jan 8 13:29:46 FileServer kernel: [1635174.386284] scsi 7:0:0:1: Direct-Access WDC WD20 EARX-00PASB0 PQ: 0 ANSI: 2 CCS Jan 8 13:29:46 FileServer kernel: [1635174.386571] scsi 7:0:0:2: Direct-Access WDC WD20 EARX-00PASB0 PQ: 0 ANSI: 2 CCS Jan 8 13:29:46 FileServer kernel: [1635174.386843] scsi 7:0:0:3: Direct-Access WDC WD20 EARX-00PASB0 PQ: 0 ANSI: 2 CCS Jan 8 13:29:46 FileServer kernel: [1635174.387945] sd 7:0:0:0: Attached scsi generic sg6 type 0 Jan 8 13:29:46 FileServer kernel: [1635174.390973] sd 7:0:0:1: Attached scsi generic sg7 type 0 Jan 8 13:29:46 FileServer kernel: [1635174.391731] sd 7:0:0:0: [sdf] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB) Jan 8 13:29:46 FileServer kernel: [1635174.392563] sd 7:0:0:0: [sdf] Write Protect is off Jan 8 13:29:46 FileServer kernel: [1635174.392571] sd 7:0:0:0: [sdf] Mode Sense: 28 00 00 00 Jan 8 13:29:46 FileServer kernel: [1635174.393512] sd 7:0:0:2: Attached scsi generic sg8 type 0 Jan 8 13:29:46 FileServer kernel: [1635174.394209] sd 7:0:0:0: [sdf] No Caching mode page present Jan 8 13:29:46 FileServer kernel: [1635174.394530] sd 7:0:0:0: [sdf] Assuming drive cache: write through Jan 8 13:29:46 FileServer kernel: [1635174.395241] sd 7:0:0:3: Attached scsi generic sg9 type 0 Jan 8 13:29:46 FileServer kernel: [1635174.397299] sd 7:0:0:0: [sdf] No Caching mode page present Jan 8 13:29:46 FileServer kernel: [1635174.397566] sd 7:0:0:0: [sdf] Assuming drive cache: write through Jan 8 13:29:46 FileServer kernel: [1635174.410408] sdf: sdf1 Jan 8 13:29:46 FileServer kernel: [1635174.410931] sd 7:0:0:1: [sdg] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB) Jan 8 13:29:46 FileServer kernel: [1635174.411088] sd 7:0:0:2: [sdh] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB) Jan 8 13:29:46 FileServer kernel: [1635174.411276] sd 7:0:0:3: [sdi] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB) Jan 8 13:29:46 FileServer kernel: [1635174.412160] sd 7:0:0:1: [sdg] Write Protect is off Jan 8 13:29:46 FileServer kernel: [1635174.412169] sd 7:0:0:1: [sdg] Mode Sense: 28 00 00 00 Jan 8 13:29:46 FileServer kernel: [1635174.413011] sd 7:0:0:2: [sdh] Write Protect is off Jan 8 13:29:46 FileServer kernel: [1635174.413019] sd 7:0:0:2: [sdh] Mode Sense: 28 00 00 00 Jan 8 13:29:46 FileServer kernel: [1635174.413879] sd 7:0:0:1: [sdg] No Caching mode page present Jan 8 13:29:46 FileServer kernel: [1635174.414192] sd 7:0:0:1: [sdg] Assuming drive cache: write through Jan 8 13:29:46 FileServer kernel: [1635174.414784] sd 7:0:0:3: [sdi] Write Protect is off Jan 8 13:29:46 FileServer kernel: [1635174.414792] sd 7:0:0:3: [sdi] Mode Sense: 28 00 00 00 Jan 8 13:29:46 FileServer kernel: [1635174.415662] sd 7:0:0:2: [sdh] No Caching mode page present Jan 8 13:29:46 FileServer kernel: [1635174.416074] sd 7:0:0:2: [sdh] Assuming drive cache: write through Jan 8 13:29:46 FileServer kernel: [1635174.417177] sd 7:0:0:3: [sdi] No Caching mode page present Jan 8 13:29:46 FileServer kernel: [1635174.417528] sd 7:0:0:3: [sdi] Assuming drive cache: write through Jan 8 13:29:46 FileServer kernel: [1635174.420515] sd 7:0:0:0: [sdf] No Caching mode page present Jan 8 13:29:46 FileServer kernel: [1635174.420837] sd 7:0:0:0: [sdf] Assuming drive cache: write through Jan 8 13:29:46 FileServer kernel: [1635174.421144] sd 7:0:0:0: [sdf] Attached SCSI disk Jan 8 13:29:46 FileServer kernel: [1635174.421416] sd 7:0:0:1: [sdg] No Caching mode page present Jan 8 13:29:46 FileServer kernel: [1635174.421791] sd 7:0:0:1: [sdg] Assuming drive cache: write through Jan 8 13:29:46 FileServer kernel: [1635174.437545] sdg: sdg1 Jan 8 13:29:46 FileServer kernel: [1635174.439003] sd 7:0:0:3: [sdi] No Caching mode page present Jan 8 13:29:46 FileServer kernel: [1635174.439361] sd 7:0:0:3: [sdi] Assuming drive cache: write through Jan 8 13:29:46 FileServer kernel: [1635174.440001] sd 7:0:0:2: [sdh] No Caching mode page present Jan 8 13:29:46 FileServer kernel: [1635174.448689] sd 7:0:0:2: [sdh] Assuming drive cache: write through Jan 8 13:29:46 FileServer kernel: [1635174.467770] sdi: sdi1 Jan 8 13:29:46 FileServer kernel: [1635174.493225] sdh: sdh1 Jan 8 13:29:46 FileServer kernel: [1635174.495137] sd 7:0:0:1: [sdg] No Caching mode page present Jan 8 13:29:46 FileServer kernel: [1635174.504732] sd 7:0:0:1: [sdg] Assuming drive cache: write through Jan 8 13:29:46 FileServer kernel: [1635174.513962] sd 7:0:0:1: [sdg] Attached SCSI disk Jan 8 13:29:46 FileServer kernel: [1635174.523285] sd 7:0:0:3: [sdi] No Caching mode page present Jan 8 13:29:46 FileServer kernel: [1635174.531683] sd 7:0:0:3: [sdi] Assuming drive cache: write through Jan 8 13:29:46 FileServer kernel: [1635174.541065] sd 7:0:0:3: [sdi] Attached SCSI disk Jan 8 13:29:46 FileServer kernel: [1635174.548886] sd 7:0:0:2: [sdh] No Caching mode page present Jan 8 13:29:46 FileServer kernel: [1635174.558578] sd 7:0:0:2: [sdh] Assuming drive cache: write through Jan 8 13:29:46 FileServer kernel: [1635174.567333] sd 7:0:0:2: [sdh] Attached SCSI disk Jan 8 14:00:06 FileServer kernel: [1636994.816122] Buffer I/O error on device md1, logical block 751866032 Jan 8 14:00:06 FileServer kernel: [1636994.824972] lost page write due to I/O error on md1 Jan 8 14:00:06 FileServer kernel: [1636994.824996] JBD2: Detected IO errors while flushing file data on md1-8 Jan 8 14:00:06 FileServer kernel: [1636994.825053] Aborting journal on device md1-8. Jan 8 14:00:06 FileServer kernel: [1636994.834255] Buffer I/O error on device md1, logical block 732463104 Jan 8 14:00:06 FileServer kernel: [1636994.845362] lost page write due to I/O error on md1 Jan 8 14:00:06 FileServer kernel: [1636994.845422] JBD2: I/O error detected when updating journal superblock for md1-8. Jan 8 14:00:35 FileServer kernel: [1637023.164090] Buffer I/O error on device md1, logical block 0 Jan 8 14:00:35 FileServer kernel: [1637023.173657] lost page write due to I/O error on md1 Jan 8 14:00:35 FileServer kernel: [1637023.173680] EXT4-fs error (device md1): ext4_journal_start_sb:327: Detected aborted journal Jan 8 14:00:35 FileServer kernel: [1637023.173692] EXT4-fs (md1): Remounting filesystem read-only Jan 8 14:00:35 FileServer kernel: [1637023.173697] EXT4-fs (md1): previous I/O error to superblock detected Jan 8 14:00:35 FileServer kernel: [1637023.174810] Buffer I/O error on device md1, logical block 0 Jan 8 14:00:35 FileServer kernel: [1637023.174833] lost page write due to I/O error on md1 Jan 8 14:00:35 FileServer kernel: [1637023.174916] EXT4-fs (md1): ext4_da_writepages: jbd2_start: 1024 pages, ino 94180140; err -30 Jan 8 20:00:01 FileServer kernel: [1658589.772603] EXT4-fs (md1): previous I/O error to superblock detected Jan 8 20:00:01 FileServer kernel: [1658589.792110] Buffer I/O error on device md1, logical block 0 Jan 8 20:00:01 FileServer kernel: [1658589.803060] lost page write due to I/O error on md1 Jan 8 20:00:01 FileServer kernel: [1658589.803085] EXT4-fs error (device md1): ext4_find_entry:935: inode #94179991: comm BackupPC_dump: reading directory lblock 0 Jan 8 20:00:01 FileServer kernel: [1658589.951838] EXT4-fs (md1): previous I/O error to superblock detected Jan 8 20:00:01 FileServer kernel: [1658589.964244] Buffer I/O error on device md1, logical block 0 Jan 8 20:00:01 FileServer kernel: [1658589.976811] lost page write due to I/O error on md1 Jan 8 20:00:02 FileServer kernel: [1658589.976830] EXT4-fs error (device md1): ext4_find_entry:935: inode #94179991: comm BackupPC_dump: reading directory lblock 0 Jan 8 20:00:02 FileServer kernel: [1658590.037737] EXT4-fs (md1): previous I/O error to superblock detected Jan 8 20:00:02 FileServer kernel: [1658590.050969] Buffer I/O error on device md1, logical block 0 Jan 8 20:00:02 FileServer kernel: [1658590.064255] lost page write due to I/O error on md1 Jan 8 20:00:02 FileServer kernel: [1658590.064278] EXT4-fs error (device md1): ext4_find_entry:935: inode #94179991: comm BackupPC_dump: reading directory lblock 0 Jan 8 21:00:01 FileServer kernel: [1662189.887900] EXT4-fs (md1): previous I/O error to superblock detected Jan 8 21:00:01 FileServer kernel: [1662189.902258] Buffer I/O error on device md1, logical block 0 Jan 8 21:00:01 FileServer kernel: [1662189.916412] lost page write due to I/O error on md1 Jan 8 21:00:01 FileServer kernel: [1662189.916458] EXT4-fs error (device md1): ext4_find_entry:935: inode #94179991: comm BackupPC_dump: reading directory lblock 0 Jan 8 21:00:02 FileServer kernel: [1662190.000254] EXT4-fs (md1): previous I/O error to superblock detected Jan 8 21:00:02 FileServer kernel: [1662190.015366] Buffer I/O error on device md1, logical block 0 Jan 8 21:00:02 FileServer kernel: [1662190.030395] lost page write due to I/O error on md1 Jan 8 21:00:02 FileServer kernel: [1662190.030413] EXT4-fs error (device md1): ext4_find_entry:935: inode #94179991: comm BackupPC_dump: reading directory lblock 0 Jan 8 21:00:02 FileServer kernel: [1662190.073627] EXT4-fs (md1): previous I/O error to superblock detected Jan 8 21:00:02 FileServer kernel: [1662190.089554] Buffer I/O error on device md1, logical block 0 Jan 8 21:00:02 FileServer kernel: [1662190.105590] lost page write due to I/O error on md1 [...]