I'm shortly aiming to merge a load of mail messages, they stored in various different mbox hierarchies. It should be pretty easy to write a little script to simply merge all the files in the same place in the hierachy together but then I'll need to remove duplicate messages as I know quite large chunks will be the same messages which have been stored in different places.
Are there any tools out there which will take a mbox file and remove duplicate messages? Removal on the basis of Message-Id: would be fine.