1

I have a large image gallery, and a few years later I realized that I get more and more duplicates. :) Basically, my gallery consists of 7-Zip archives containing the image sets. I saw there are some commercial image duplicate scanners for Windows, but, unfortunately, they deal with RAR and ZIP formats only.

So I would like to find a tool that can scan the regular directories and such archives like they were a regular directories. (Perhaps, there could be some kind of a virtual file system mount if possible, that could make something like a file system mirror where all archives are expanded to regular files.) The target system is Windows, though Cygwin-related ideas are also really welcome.

Can anyone help? Your help would be very appreciated.

Lyubomyr Shaydariv
  • 885

1 Answers1

2

I don't know any tool which does that but if you're interested to try for yourself a way to achieve that would be to write a script wrapping the 7z l -slt archive.7z which for each file would append to a file:

  • the name of the file
  • the size of the file
  • its CRC

From here you just have to find duplicates (defined as 'Equal CRC and size' to limit collisions) and delete them using 7z d archive.7z filename.

edit:

You may want to look at this project, even though it doesn't support writing yet it could evolve:
fuse-7z

Many other solutions exist for other archive formats, here is a list: http://sourceforge.net/apps/mediawiki/fuse/index.php?title=ArchiveFileSystems

Shadok
  • 3,970
  • Thank you for the reply. Actually, that is not guaranteed that an image name reflects its content or even marks the original timestamp. In that case I could write a simple batch file to get all files content entries and process the output in some custom analyzer written e.g. in Java or so (not familiar with Perl, etc). The reason is exactly to find the image duplicates with some similarity threshold... That would be great... – Lyubomyr Shaydariv Feb 14 '12 at 22:21
  • The reason I kept the name of the file is because you need it to delete the file from the archive, not for discriminating between images :) You sure could pipe each file to a tool which would compare the pixels in the image but then isn't it simpler to uncompress everything on another machine and recompress them after having deleted the duplicates ? You wouldn't be limited at all in the tools you can use. – Shadok Feb 15 '12 at 09:47
  • Yes, I had a similar thought in my mind: simply decompress and then recompress not duplicates. But this requires really a lot of free HDD space and really much time at least for compression (yeah, 7-Zip over JPEG compression may look weird, but 7-Zip really gave a lot of free space). So that's why I'm interested in in-archives scanners: either direct scanner, or via some virtual file system. :) – Lyubomyr Shaydariv Feb 15 '12 at 12:53
  • For curiosity's sake, given the images of your gallery are in 7z format, how do you read them ? (presumably to serve them over http or whatever) – Shadok Feb 15 '12 at 13:55
  • As I just saw you were running windows, I'm wondering if you already tried using NTFS compression on your files ? You will surely take a big hit in size-wise but the transparency of it may be worth the try. Also running a NAS box with a file system supporting compression and deduplication could be interesting if you can invest a little. – Shadok Feb 15 '12 at 14:08
  • Ah, I meant the local personal storage only. No, I didn't use NTFS compression, but I guess it really has the sense. Actually saying, I never considered that attribute before. Also thanks for fuse-7z - that is also one of the things I was looking for. You gave a lot of ideas, thanks. :) – Lyubomyr Shaydariv Feb 15 '12 at 16:22
  • You're welcome ! Have a look at this post, it has great details on DIY NAS: http://superuser.com/questions/124978/how-do-i-build-a-diy-nas – Shadok Feb 15 '12 at 16:51