I need to set up an FTP server to host fairly large text files (.CSV).
At the FTP client level these need to be uncompressed but I'd like to reduce server disk requirements by using compression. I guess I could either do this at a file system (FUSE) level or transparently within an FTP server, assuming there is an FTP server that can do this.
Anyone got any experience with this or suggestions?
Hi Mark,
I'd do it at the file system level. I would ZFS for compression and deduplication.
Cheers,
Ewan
On 13 May 2014 11:05, Mark Rogers mark@quarella.co.uk wrote:
I need to set up an FTP server to host fairly large text files (.CSV).
At the FTP client level these need to be uncompressed but I'd like to reduce server disk requirements by using compression. I guess I could either do this at a file system (FUSE) level or transparently within an FTP server, assuming there is an FTP server that can do this.
Anyone got any experience with this or suggestions?
--
Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG
main@lists.alug.org.uk http://www.alug.org.uk/ http://lists.alug.org.uk/mailman/listinfo/main Unsubscribe? See message headers or the web site above!
On 14 May 2014 15:20, Ewan Slater ewan.slater@googlemail.com wrote:
I'd do it at the file system level. I would ZFS for compression and deduplication.
The final solution is going to end up on a hosted virtual server somewhere; I've not found any yet that offer ZFS, which is why I was looking at something like FUSE to run something on top of the default filesystem (generally ext4).
I have had a quick play with avfs on my Ubuntu desktop and that looks useful - effectively allowing me to just gzip (or bzip2) the files and have them appear uncompressed to the FTP server, although it takes a bit of messing around (the transparently uncompressed version of ~/foo.gz becomes ~/.avfs/home/mark/foo.gz#, but I can create a symlink in the FTP root to make it appear there as "foo").
fusecompress isn't in the repos so I need to work a bit harder on that one.
Mark
But if that's the case you're going to need to tweak the VM anyway, so why not use ZFS?
I've certainly used ZFS on AWS VMs.
Sent from my iSheep
On 14 May 2014, at 17:04, Mark Rogers mark@quarella.co.uk wrote:
On 14 May 2014 15:20, Ewan Slater ewan.slater@googlemail.com wrote:
I'd do it at the file system level. I would ZFS for compression and deduplication.
The final solution is going to end up on a hosted virtual server somewhere; I've not found any yet that offer ZFS, which is why I was looking at something like FUSE to run something on top of the default filesystem (generally ext4).
I have had a quick play with avfs on my Ubuntu desktop and that looks useful - effectively allowing me to just gzip (or bzip2) the files and have them appear uncompressed to the FTP server, although it takes a bit of messing around (the transparently uncompressed version of ~/foo.gz becomes ~/.avfs/home/mark/foo.gz#, but I can create a symlink in the FTP root to make it appear there as "foo").
fusecompress isn't in the repos so I need to work a bit harder on that one.
Mark
Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG
main@lists.alug.org.uk http://www.alug.org.uk/ http://lists.alug.org.uk/mailman/listinfo/main Unsubscribe? See message headers or the web site above!
On 14/05/14 17:21, Ewan Slater wrote:
But if that's the case you're going to need to tweak the VM anyway, so why not use ZFS?
I've certainly used ZFS on AWS VMs.
Is Fusecompress any use?
https://code.google.com/p/fusecompress/wiki/Usage
Steve
On 14 May 2014 22:44, steve-ALUG@hst.me.uk wrote:
Is Fusecompress any use?
It's looking the most likely, I just haven't tried it yet (it's not in the repos so I'm reluctant to just try it on my desktop, which means setting up a VM to play with, and I haven't had chance yet).
If anyone has any experience of fusecompresss I'd love to hear about it!
On 14 May 2014 17:21, Ewan Slater ewan.slater@googlemail.com wrote:
But if that's the case you're going to need to tweak the VM anyway, so why not use ZFS?
I've certainly used ZFS on AWS VMs.
The sort of VM I'm talking about is in a different league I'm afraid: the sort of league where £5/mo gets you 40GB storage and everything is provisioned using one of their standard images, all of which are are ext4.
Who is the hosting partner out of morbid interest?
On 15 May 2014 11:34, Mark Rogers mark@quarella.co.uk wrote:
On 14 May 2014 17:21, Ewan Slater ewan.slater@googlemail.com wrote:
But if that's the case you're going to need to tweak the VM anyway, so why not use ZFS?
I've certainly used ZFS on AWS VMs.
The sort of VM I'm talking about is in a different league I'm afraid: the sort of league where £5/mo gets you 40GB storage and everything is provisioned using one of their standard images, all of which are are ext4.
-- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG
main@lists.alug.org.uk http://www.alug.org.uk/ http://lists.alug.org.uk/mailman/listinfo/main Unsubscribe? See message headers or the web site above!
On 15 May 2014 11:58, Ewan Slater ewan.slater@googlemail.com wrote:
Who is the hosting partner out of morbid interest?
We usually use Memset (mostly through habit!).
The application here is pretty trivial though; we just need to store a few years' worth of data for a few devices that generate it in CSV format, and need a limited amount of scripting capability on the server (primarily for mirroring to clients servers) which means that a simple cloud storage package isn't sufficient.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Dropbox or a NoSQL database in the cloud?
On 15 May 2014 13:52:17 BST, Mark Rogers mark@quarella.co.uk wrote:
On 15 May 2014 11:58, Ewan Slater ewan.slater@googlemail.com wrote:
Who is the hosting partner out of morbid interest?
We usually use Memset (mostly through habit!).
The application here is pretty trivial though; we just need to store a few years' worth of data for a few devices that generate it in CSV format, and need a limited amount of scripting capability on the server (primarily for mirroring to clients servers) which means that a simple cloud storage package isn't sufficient.
-- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG
main@lists.alug.org.uk http://www.alug.org.uk/ http://lists.alug.org.uk/mailman/listinfo/main Unsubscribe? See message headers or the web site above!
- -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On 15 May 2014 13:59, Ewan Slater ewan.slater@googlemail.com wrote:
Dropbox or a NoSQL database in the cloud?
We currently use some pretty cheap cloud storage which is fine, but we now have a requirement to run some scripts on the data so we need to move away from pure storage.
In truth though, no matter where the data is stored I have an instinctive dislike for storing large text files in an uncompressed format. Maybe this goes back to using Stacker under DOS all those many years ago, but the fact is that CPUs are faster than disks and generally it's quicker to retrieve a compressed file and uncompress it than it is to retrieve an uncompressed file, so as long as it can be done transparently that would be my preference. One of the few features I miss from Windows is the NTFS compression option, which I regularly use in areas where it makes sense (eg log directories). In contrast Linux makes it incredibly easy to work with compressed files (eg .gz/.bz2), but not so much when the compression needs to be "invisible".
Mark
-
Ewan Slater -
Mark Rogers -
steve-ALUG＠hst.me.uk