Wisdom (or otherwise) of using software RAID5 across 4 USB drives
I have a USB caddy that takes 4 drives, on USB3. I have four 2TB SATA disks in it and have just set up a RAID5 array across the four disks, formatted to ext4. cat /proc/mdstat currently reports: md1 : active raid5 sdi1[4] sdh1[2] sdg1[1] sdf1[0] 5860145664 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_] [>....................] recovery = 1.2% (24142492/1953381888) finish=1134.2min speed=28348K/sec That's about 20hrs to complete the build, running at max speed (/proc/sys/dev/raid/speed_limit_max is 500000 so I'm not hitting the upper limit there). To be honest, rebuilding an array comprising 4x2TB disks over USB in 20hrs seems pretty good to me, so I'm not overly concerned. But should I be? Is running this configuration likely to cause me grief in future? The purpose of the array is to store backups, so read/write speed is relatively unimportant. At this point the array is empty so wiping and starting again isn't a big deal if someone has better ideas. (As an aside, does the above look right for USB3 speeds? It's my first experience of USB3 and I'm using a cheap PCI-E USB3 card so I have no idea how well it's performing.) Mark -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER
I think if you really really don't care about speed then it'll be ok. 20 hours sounds about right to me considering the USB interface is CPU bound and everything is having to go through the USB Mass Storage subsystem. On a direct interface like SATA/SAS I have seen arrays of this size rebuild in a lot less time but that is to be expected. The only other thing you might see is performance problems across the system when working this array hard, really depends on the spec of the machine it is attached to but it's going to get fairly tied up with stuff sitting in IO wait and depending on what processes that happens to it can block other things. You might be able to fix a bit of that by making sure write caching is enabled on the array.
On 30/11/12 08:21, Wayne Stallwood wrote:
I think if you really really don't care about speed then it'll be ok.
Well the backups will be going to the USB drives regardless, so it's going to be relatively slow whatever I do. How much slower is putting RAID5 on it going to make it?
20 hours sounds about right to me considering the USB interface is CPU bound and everything is having to go through the USB Mass Storage subsystem. On a direct interface like SATA/SAS I have seen arrays of this size rebuild in a lot less time but that is to be expected.
It's a pretty old 3GHz Pentium D. It's already a file server (software RAID5 across 4 SATA internal disks), and the only other thing it's going to be doing is the backups. It's old and slow in modern CPU terms but I'm assuming its still way more powerful that whatever CPU I'd find in a dedicated RAID NAS box?
The only other thing you might see is performance problems across the system when working this array hard, really depends on the spec of the machine it is attached to but it's going to get fairly tied up with stuff sitting in IO wait and depending on what processes that happens to it can block other things. You might be able to fix a bit of that by making sure write caching is enabled on the array.
Assuming backups run at night, there won't be anything else going on when this happens. Backups will be mostly from local data (the RAID5 array on internal disks) to the USB array, but will also include some data pulled across the LAN from other machines, where the LAN will probably be a bigger bottleneck than the USB interface. With caching: I would have assumed that the worst case in the event of power loss would be loss of the backup that was still cached, but a quick Google suggests that the whole array could be corrupted? That would kinda defeat the point of using it for backups, so I'd appreciate advice on that. RAID10 would be substantially faster but obviously I'd lose a third of my array capacity. I think I'm going to have to just see how bad it is and make a judgement after that. Mark -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER
On 30/11/12 09:52, Mark Rogers wrote: []
It's a pretty old 3GHz Pentium D. It's already a file server (software RAID5 across 4 SATA internal disks), and the only other thing it's going to be doing is the backups.
[] With caching: I would have assumed that the worst case in the event of power loss would be loss of the backup that was still cached, but a quick Google suggests that the whole array could be corrupted? That would kinda defeat the point of using it for backups, so I'd appreciate advice on that.
I can't comment on the backup corruption, but if it's a file server, and a backup server, then I suggest an uninterpretable power supply (UPS). I'm sure you'll be able to pick one up on ebay without breaking the bank. Change the batteries at least every 2 years, yearly if you can afford it. APC is a good make, and with that, you can install APCUPSD (APC UPS Daemon) which will safely shutdown the PC a configurable time after a power failure. It should be able to supply power to your external raid too if required. HTH Steve
On 30/11/12 10:13, steve-ALUG@hst.me.uk wrote:
I can't comment on the backup corruption, but if it's a file server, and a backup server, then I suggest an uninterpretable power supply (UPS). I'm sure you'll be able to pick one up on ebay without breaking the bank. Change the batteries at least every 2 years, yearly if you can afford it. APC is a good make, and with that, you can install APCUPSD (APC UPS Daemon) which will safely shutdown the PC a configurable time after a power failure. It should be able to supply power to your external raid too if required.
We do already have a UPS although I'm not sure it's suitably sized - that's one of the things on my list. I'm pretty sure it's not currently being monitored (historic reasons - the kit that used to monitor it has now gone) which makes it only slightly better than no UPS at all, and I am sure it needs new batteries. Up until now it wasn't powering anything that mattered so that wasn't a big deal. Assuming I can get to a point where the UPS will keep the server running for a few minutes, and notify it to shut down, then is write caching on the external RAID array "safe"? Mark Rogers -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER
On 30/11/12 10:28, Mark Rogers wrote:
We do already have a UPS although I'm not sure it's suitably sized - that's one of the things on my list. I'm pretty sure it's not currently being monitored (historic reasons - the kit that used to monitor it has now gone) which makes it only slightly better than no UPS at all, and I am sure it needs new batteries. Up until now it wasn't powering anything that mattered so that wasn't a big deal.
Assuming I can get to a point where the UPS will keep the server running for a few minutes, and notify it to shut down, then is write caching on the external RAID array "safe"?
If the UPS is monitored, detects a power failure, and initiates a shutdown, I would expect the O/S would request the backup program to terminate, and then kill it if it didn't respond quickly enough. The backup program will stop writing because it's been shut down, and I'm sure the o/s will flush its own write cache to disk before terminating. I'm reasonably sure that any hardware disk cache will get written to disk before the power is removed so I think you'll be OK. A monitored UPS at least gives you the chance to attempt to shutdown gracefully, which an un-monitored one doesn't. Steve Steve
On 30/11/12 11:07, steve-ALUG@hst.me.uk wrote:
A monitored UPS at least gives you the chance to attempt to shutdown gracefully, which an un-monitored one doesn't.
I've just pulled the UPS out for a bit of TLC. It's an APC Smart-UPS SC1500 so not a bad bit of kit; I've just ordered replacement batteries on the basis that the old ones are, well, old. [*] The UPS has a serial connector for monitoring, so handy that my old server also has serial. So hopefully I can get all that set up this afternoon without too much grief. Thanks for all the advice! [*] Off-topic: Anyone know what sort of solar panel would do a good job of charging one of the 7.2Ah 12v batteries I'm taking out of the UPS? I have some cheap 12v LED Christmas lights that came with solar panels, which I haven't used yet but I figure won't keep much charge (running 100 LEDs off a single 800mAh 1.2v NiMH battery doesn't fill me with confidence). So I'd like to get a half decent solar panel and charge a bigger 12v battery instead, and I now have 4 spare 12v batteries.... -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER
On 30/11/12 11:44, Mark Rogers wrote:
On 30/11/12 11:07, steve-ALUG@hst.me.uk wrote:
A monitored UPS at least gives you the chance to attempt to shutdown gracefully, which an un-monitored one doesn't.
I've just pulled the UPS out for a bit of TLC. It's an APC Smart-UPS SC1500 so not a bad bit of kit; I've just ordered replacement batteries on the basis that the old ones are, well, old. [*]
The UPS has a serial connector for monitoring, so handy that my old server also has serial. So hopefully I can get all that set up this afternoon without too much grief.
I suggest you install apcupsd Comprehensive website here <http://www.apcupsd.com/manual/manual.html#supported-upses-and-cables>http://www.apcupsd.com/manual/manual.html#frequently-asked-questions But see Q 1 & 4 re cables. APC uses customised serial cables not standard ones. You either have to buy one from APC (they do come with the UPS), or build one - there's a link. I built one - I got a cheap 9 pin serial cable from Maplin, with a non-sealed connector, then un-soldered and re-soldered a couple of wires. My apcupsd config was simple. Edit the config file to say it's using a serial port, and the serial port name. UPSCABLE smart DEVICE /dev/ttyS0 (for serial port 1 etc) (NB, I'm now using it via my custom cable, that's plugged into a Serial to USB converter! and set that up as DEVICE /dev/ttyUSB0 or 1...) Set Timeout to 0. With a SmartUPS, apcupsd will then monitor the battery level, and initiate a shutdown when it falls below a certain level. If the power comes back on before then, then there's no shutdown. I've also set my server's bios so that when power appears at the power socket it powers on. I also use gapcmon to monitor the UPS and tell me of any power events. apcupsd can also drive a web-based power monitoring page if you prefer. HTH Steve
On 30/11/12 12:48, steve-ALUG@hst.me.uk wrote:
I suggest you install apcupsd
Done and now working, thanks for your pointers. Looks like I have about an hour of battery runtime supplying this server and the USB caddy, using the old batteries. That's a bit more than I was expecting! I do have another box I'd like to put through it but even then 30mins backup is plenty of time to ride out a short power cut and still shut down before the power fails.
I also use gapcmon to monitor the UPS and tell me of any power events. apcupsd can also drive a web-based power monitoring page if you prefer.
I can't find gacpmon - gacpmon.sourceforge.net no longer resolves. Is it still maintained? For now I'm happy with "apcaccess status" to get my UPS status, but running a GUI on my desktop wouldn't be a terrible idea. Mark -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER
On 30/11/12 15:28, Mark Rogers wrote:
On 30/11/12 12:48, steve-ALUG@hst.me.uk wrote:
I suggest you install apcupsd
Done and now working, thanks for your pointers. Looks like I have about an hour of battery runtime supplying this server and the USB caddy, using the old batteries. That's a bit more than I was expecting! I do have another box I'd like to put through it but even then 30mins backup is plenty of time to ride out a short power cut and still shut down before the power fails.
I also use gapcmon to monitor the UPS and tell me of any power events. apcupsd can also drive a web-based power monitoring page if you prefer.
I can't find gacpmon - gacpmon.sourceforge.net no longer resolves. Is it still maintained?
For now I'm happy with "apcaccess status" to get my UPS status, but running a GUI on my desktop wouldn't be a terrible idea.
Mark
http://gapcmon.sourceforge.net works for me, as does http://sourceforge.net/projects/gapcmon/ Last new version was in 2008 though. It's a gnome tray app though - dunno if it'll work in Ubuntu Unity or KDE. Suggest that one day, when you're feeling brave, switch the power off at the mains and see if the machine shuts down gracefully as expected. Also check it comes back on when the power is re-connected. If you have a webserver running somewhere on your network, you can monitor the UPS via that (even if the ups is on a different machine to the webserver) Having a quick look, there seems to be a new app called NUT Monitor, which may work better for you. Suggest you just open a package manager and type in UPS and see what comes up! :-) If you're running two machines off one machine, then I suggest that you install apcupsd on both and configure as listed below, so that both shutdown gracefully http://www.apcupsd.com/manual/manual.html#nis-server-client-configuration-us... Checking the documentation, it suggest batteries should last 2 years - I guess I've been buying cheap ones that don't last so long! Cheers Steve
On 30/11/12 09:52, Mark Rogers wrote:
It's a pretty old 3GHz Pentium D. It's already a file server (software RAID5 across 4 SATA internal disks), and the only other thing it's going to be doing is the backups.
It's old and slow in modern CPU terms but I'm assuming its still way more powerful that whatever CPU I'd find in a dedicated RAID NAS box?
That is true...however a dedicated Raid NAS box wouldn't have the overhead of the drives being on a CPU bound overhead....md on its own isn't so much of a drag on resources...md talking to 4 USB drives may be. :)
participants (3)
-
Mark Rogers -
steve-ALUG＠hst.me.uk -
Wayne Stallwood