Re: Comparing large folders of mostly identical file contents with different filenames (Mark Rogers)
A query, not a suggestion, because you are all far, far better informed than me about this stuff! I was curious and looked up what tools are available for this, and came on:
rdfind Fdupes dupGuru
which all seem to find dupes by content. I am sure Mark is well aware of these and more. So why can't one of them be used to find the dupes? Also, is any of them significantly better than the others?
Different but related issue, the app I have used to sync folders in the past is Unison, which I found very simple and effective. Also Grsync. Do people use these? Or is something else preferable?
Peter
On Sun, 22 May 2022 12:00:03 +0100 main-request@lists.alug.org.uk wrote:
Send main mailing list submissions to main@lists.alug.org.uk
To subscribe or unsubscribe via email, send a message with subject or body 'help' to main-request@lists.alug.org.uk
You can reach the person managing the list at main-owner@lists.alug.org.uk
When replying, please edit your Subject line so it is more specific than "Re: Contents of main digest..."
(First time reply using FairEmail on my phone, which I think solves the plain text issue but I can't see how to do inline quotes properly)
The issue with standard dedup tools is that I have a lot of files, some of them quite large, and a dedup tool won't take into account all the features of the file structure that save our from having to, in effect, check every file against every other.
Also, with what I know now, they'd have failed horribly at detecting files which are identical except for whitespace differences
My script evolved to take the first 20k of message data after the header, remove whitespace, hash the first 10k of that, and compare those hashes, but only against files in the same associated sub directory. This was quite a quick way to find the things I need to find (where a message had failed to transfer from one server to another) but highly unlikely to be much use in a general case.
Incidentally, Google have now said that legacy Workspace accounts which are used for private (non commercial) purposes only can be retained, so the exercise has proved to be unnecessary after all the effort, but such is life!
23 May 2022 08:47:11 Peter peter.northerly@gmail.com:
A query, not a suggestion, because you are all far, far better informed than me about this stuff! I was curious and looked up what tools are available for this, and came on:
rdfind Fdupes dupGuru
which all seem to find dupes by content. I am sure Mark is well aware of these and more. So why can't one of them be used to find the dupes? Also, is any of them significantly better than the others?
Different but related issue, the app I have used to sync folders in the past is Unison, which I found very simple and effective. Also Grsync. Do people use these? Or is something else preferable?
Peter
On Sun, 22 May 2022 12:00:03 +0100 main-request@lists.alug.org.uk wrote:
Send main mailing list submissions to main@lists.alug.org.uk
To subscribe or unsubscribe via email, send a message with subject or body 'help' to main-request@lists.alug.org.uk
You can reach the person managing the list at main-owner@lists.alug.org.uk
When replying, please edit your Subject line so it is more specific than "Re: Contents of main digest..."
To unsubscribe send an email to main-leave@lists.alug.org.uk http://www.alug.org.uk/ Unsubscribe? See message headers or the web site above!
PS: Unison, now that's a blast from the past! Used to be a heavy user but probably haven't touched it in 10 years or so.
What are you trying to sync? I use version control (generally SVN) for some things, SyncThing for others, rsync for others still - they're all great for different use cases.
Might have to dig out unison for some nostalgia though!
I was looking at the post about comparing large amount of emails. My first thought was, why bother? Why not use rm -r * ;)
If you haven't used/reached to those files for a long time, isn't it to part with them?
As to the Fairmail on android; I have to say it is an amazing app. Simple yet having plenty of options, protecting you from spying images, tracking links etc. One issue is that it does not work with google mail... Which is a bit of a shame, but for those without gmail I really recommend.
Cheerio, Bart
pon., 23 maj 2022 o 21:10 Mark Rogers mark@more-solutions.co.uk napisał(a):
PS: Unison, now that's a blast from the past! Used to be a heavy user but probably haven't touched it in 10 years or so.
What are you trying to sync? I use version control (generally SVN) for some things, SyncThing for others, rsync for others still - they're all great for different use cases.
Might have to dig out unison for some nostalgia though! _______________________________________________ To unsubscribe send an email to main-leave@lists.alug.org.uk http://www.alug.org.uk/ Unsubscribe? See message headers or the web site above!
On Mon, 23 May 2022 21:59:56 +0100 B D dzidek23@gmail.com allegedly wrote:
As to the Fairmail on android; I have to say it is an amazing app. Simple yet having plenty of options, protecting you from spying images, tracking links etc. One issue is that it does not work with google mail... Which is a bit of a shame, but for those without gmail I really recommend.
K-9 can do all of that. I deliberately do not allow images in (any of) my email clients (claws on the desktop). HTML email is an abomination and not to be tolerated by anyone who supports proper standards (Harrumph!)
I'm no fan of google (I use lineageos on my phone and get my apps from f-droid) but I understand that they have kicked Fairemail off the play store amidst accusations that it is "spyware".
Mick
--------------------------------------------------------------------- Mick Morgan gpg fingerprint: FC23 3338 F664 5E66 876B 72C0 0A1F E60B 5BAD D312 https://baldric.net/about-trivia ---------------------------------------------------------------------
On Tue, 24 May 2022 at 13:37, mick mbm@rlogin.net wrote:
K-9 can do all of that.
What it doesn't do is play well with XOAUTH (ie Gmail accounts). There are workarounds but not good ones.
I deliberately do not allow images in (any of) my email clients (claws on the desktop). HTML email is an abomination and not to be tolerated by anyone who supports proper standards
As much as I agree with the sentiment I gave that up as a battle at least a decade ago. I still reply inline and maintain a short 2-line .sig with proper separate r(that only formats correctly in plain text but doesn't break horribly otherwise), but that's about all now.
I understand that they have kicked Fairemail off the play store amidst accusations that it is "spyware".
Indeed. The developer is struggling to get an answer from Google as to what the trigger is but not getting an answer. His best guess is that it relates to him sending email addresses to a server without adequate user policy to allow it. As in: if your email account uses an email address as its login, then Fairemail has to send that address to the IMAP server... He keeps making tweaks, which work, until they don't.
He was hoping to talk to a real person at Google today via a backchannel, I don't know if that happened,
The source is of-course all there on Github for anyone who wants to audit it!
On Tue, 24 May 2022 17:52:52 +0100 Mark Rogers mark@more-solutions.co.uk allegedly wrote:
On Tue, 24 May 2022 at 13:37, mick mbm@rlogin.net wrote:
K-9 can do all of that.
What it doesn't do is play well with XOAUTH (ie Gmail accounts). There are workarounds but not good ones.
google are attempting to force everyone to do things /their/ way (in the name of "security", but which has naff all to do with the user's security and everything to do with google's security).
But the k-9 devs seem to have caved in to the behemoth and will add xoauth2 in 6.200
https://forum.k9mail.app/t/the-plan-for-k-9-mail-6-200/4590
Why the f*** anyone uses gmail is beyond me.
Mick
--------------------------------------------------------------------- Mick Morgan gpg fingerprint: FC23 3338 F664 5E66 876B 72C0 0A1F E60B 5BAD D312 https://baldric.net/about-trivia ---------------------------------------------------------------------
On this one point, because without a gmail address using an Android phone is impossible. (I do know of a non Google app store but it's not as extensive.)
Bev
https://forum.k9mail.app/t/the-plan-for-k-9-mail-6-200/4590
Why the f*** anyone uses gmail is beyond me.
Mick
On 24 May 2022 20:05:24 BST, Bev Nicolson lumos@gmx.co.uk wrote:
On this one point, because without a gmail address using an Android phone is impossible. (I do know of a non Google app store but it's not as extensive.)
Bev
https://forum.k9mail.app/t/the-plan-for-k-9-mail-6-200/4590
Why the f*** anyone uses gmail is beyond me.
Mick
To unsubscribe send an email to main-leave@lists.alug.org.uk http://www.alug.org.uk/ Unsubscribe? See message headers or the web site above!
No it is not. This Motorola phone runs lineageos. No google apps at all. 🙂
On Tue, May 24, 2022 at 08:05:24PM +0100, Bev Nicolson wrote:
https://forum.k9mail.app/t/the-plan-for-k-9-mail-6-200/4590
Why the f*** anyone uses gmail is beyond me.
On this one point, because without a gmail address using an Android phone is impossible. (I do know of a non Google app store but it's not as extensive.)
Not true. You can tie a non-Google email address to a Google account without any GMail involvement. I do not have a GMail account, but my phone is setup for the Play store.
J.
On 25/05/2022 15:38, Jonathan McDowell wrote:
On Tue, May 24, 2022 at 08:05:24PM +0100, Bev Nicolson wrote:
https://forum.k9mail.app/t/the-plan-for-k-9-mail-6-200/4590
Why the f*** anyone uses gmail is beyond me.
On this one point, because without a gmail address using an Android phone is impossible. (I do know of a non Google app store but it's not as extensive.)
Not true. You can tie a non-Google email address to a Google account without any GMail involvement. I do not have a GMail account, but my phone is setup for the Play store.
J.
Well I'm glad others have managed it (3 so far.) I did have a look but couldn't see any instructions how one did this. (This is not a plea for those instructions, by the way. Thanks.)
Bev
On Tue, 24 May 2022 at 19:29, mick mbm@rlogin.net wrote:
But the k-9 devs seem to have caved in to the behemoth and will add xoauth2 in 6.200
That's good to hear.
Why the f*** anyone uses gmail is beyond me.
Well for me: back when I started using it seriously (15+y ago) it was by far the best at filtering spam, as indeed logically it ought to be since it has a far larger pool to train its algorithms on. Combined with the fact I could run an entire domain for work on it up to 50 users without paying a penny it was hard to resist. (Prior to that I had run my own servers, using dspam for spam management which was a great bit of software but it never took off the way Spamassassin did and it stopped being maintained.)
Frankly I didn't want the hassle of supporting something for work, for myself, or for anyone else. If Gmail goes down, everyone just accepts that Gmail is down and doesn't blame me!
On Mon, 23 May 2022 at 22:00, B D dzidek23@gmail.com wrote:
I was looking at the post about comparing large amount of emails. My first thought was, why bother? Why not use rm -r * ;)
If you haven't used/reached to those files for a long time, isn't it to part with them?
Actually I do refer to them quite frequently. Not all of them obviously, and a clear-out is on the cards. However the immediate issue was the need to migrate to a new account (which has now ceased to be an issue due to Google policy changes).
As to the Fairmail on android; I have to say it is an amazing app. Simple yet having plenty of options, protecting you from spying images, tracking links etc. One issue is that it does not work with google mail... Which is a bit of a shame, but for those without gmail I really recommend.
It works fine on Google for me (far better than K-9).
Sadly the developer has given up fighting with Google to get his apps onto Google's Play Store (in the past few days) and as things stand development has ceased. It's currently too early to see where the final resting place will be.
Mark
That was very instructive, particularly the part about not catching differences in white space.
I was trying to sync two folders with music files. I had been very careless about just using one as the master. The other was on a removable drive. So I had the situation where there were some files on one and not on the other in both directions, along with a lot of duplicates. Also the structure was different. For instance, Couperin's pieces might be in a file called French Baroque on one, but in a file at the top level of the other called Couperin, as I had reorganized the files to make the structure more intuitive. And then there would be some Couperin pieces that were only in one of the folders, in both directions. It was a complete mess.
I spent forever trying to figure out how to do this and finally resorted to Unison, which seemed to work OK, though I did end up with a lot of empty directories on the new master and had to purge them by hand.
The thing I learned, which I really knew already, was to be rigorous. Have a master and a backup, and only make changes to the master. Obvious enough! It crept up on me, because at first there were very few files. But as I converted more and more CDs to flac, to use on a music player, and used the backup independently, it got out of hand.
Peter
On Mon, 23 May 2022 21:10:21 +0100 (GMT+01:00) Mark Rogers mark@more-solutions.co.uk wrote:
PS: Unison, now that's a blast from the past! Used to be a heavy user but probably haven't touched it in 10 years or so.
What are you trying to sync? I use version control (generally SVN) for some things, SyncThing for others, rsync for others still - they're all great for different use cases.
Might have to dig out unison for some nostalgia though!
On 24/05/2022 08:21, Peter wrote:
I was trying to sync two folders with music files. I had been very careless about just using one as the master. The other was on a removable drive. So I had the situation where there were some files on one and not on the other in both directions, along with a lot of duplicates. Also the structure was different. For instance, Couperin's pieces might be in a file called French Baroque on one, but in a file at the top level of the other called Couperin, as I had reorganized the files to make the structure more intuitive. And then there would be some Couperin pieces that were only in one of the folders, in both directions. It was a complete mess.
I spent forever trying to figure out how to do this and finally resorted to Unison, which seemed to work OK, though I did end up with a lot of empty directories on the new master and had to purge them by hand.
The thing I learned, which I really knew already, was to be rigorous. Have a master and a backup, and only make changes to the master. Obvious enough! It crept up on me, because at first there were very few files. But as I converted more and more CDs to flac, to use on a music player, and used the backup independently, it got out of hand.
It's probably moot now. Have you come across MusicBrains Picard?
It looks up your music and adjusts file names, folder names and MP3/Ogg tags, but only if you want to. I use it to standardise my music file names.
You could have used this to standardise names then de-duplicate the files.
Hope that helps
Steve
-
B D -
Bev Nicolson -
Jonathan McDowell -
Mark Rogers -
Mick -
mick
-
Peter -
steve-ALUG＠hst.me.uk