On 25 January 2018 at 17:23, Martijn Koster mak-alug@greenhills.co.uk wrote:
Once you have your ordered stream, with awk you could do e.g:
awk -F '-' '{f = "split"$1".log";print >> f}' log.txt
which creates split20180114.log, split20180115.log and so forth.
Great thanks, I'll give that a go. Can awk either write directly to gzipped files or else can the above be modified to pipe through gzip? It's not just the relative efficiency of doing it one step but also the volume of disk space I'm going to chew up otherwise. I know that PHP can write .gz directly but that feels like a horrible tool for the job in general.
As for best/efficient, there may be ways that are faster, but I'd not optimise prematurely; if the above simple way gets the job done and works fast enough.
Absolutely agree about not prematurely or over-optimising.