Page 1 of 2 12 LastLast
Results 1 to 15 of 18
  1. #1
    Bronze Lounger
    Join Date
    Jun 2001
    Location
    New York, New York, Lebanon
    Posts
    1,449
    Thanks
    1
    Thanked 1 Time in 1 Post

    Bit-wise File Compareing Utility

    <img src=/S/hello.gif border=0 alt=hello width=25 height=29> Loungers

    Can someone point me to a utility, hopefully freeware, that would do a Bit-for Bit comparison and tells what files are the same?

    I have about 60,000 JEPG images that I need to make sure that we don't have any duplicates, and also some MS-Word, MS-Excel, and MS-PowerPoint stuff that I don't mind sifting through and checking for uniquness.

    TIA

    Wassim
    <img src=/S/compute.gif border=0 alt=compute width=40 height=20> in the <img src=/S/bagged.gif border=0 alt=bagged width=22 height=22>

  2. #2
    Plutonium Lounger
    Join Date
    Oct 2001
    Location
    Lexington, Kentucky, USA
    Posts
    12,107
    Thanks
    0
    Thanked 1 Time in 1 Post

    Re: Bit-wise File Compareing Utility

    Take a look at this one. I haven't used it, but it's <img src=/S/free.gif border=0 alt=free width=30 height=15>

    (I use something bought a few years ago, but rather expensive nowadays, so I won't bore you with it)

    Added: The reason I like and support MidnightBlue Software is that I HAVE used their SuperJPG and SST for a few years.

  3. #3
    Plutonium Lounger
    Join Date
    Mar 2002
    Posts
    84,353
    Thanks
    0
    Thanked 29 Times in 29 Posts

    Re: Bit-wise File Compareing Utility

    Warning: if you open an Office document and save it under another name without changing anything in the contents, the copy and the original will NOT be bitwise identical.

  4. #4
    Super Moderator jscher2000's Avatar
    Join Date
    Feb 2001
    Location
    Silicon Valley, USA
    Posts
    23,112
    Thanks
    5
    Thanked 93 Times in 89 Posts

    Re: Bit-wise File Compareing Utility

    DOS used to have something built in. Perhaps it still exists in newer versions of Windows? Or do you need a programmable COM object?

  5. #5
    Uranium Lounger
    Join Date
    Dec 2000
    Location
    Salt Lake City, Utah, USA
    Posts
    9,508
    Thanks
    0
    Thanked 6 Times in 6 Posts

    Re: Bit-wise File Compareing Utility

    Still does, 'comp.exe', but it only takes two file arguments at a time, and so won't be too useful for Wassim's 60,000 files unless he writes a killer batch loop for it. <img src=/S/grin.gif border=0 alt=grin width=15 height=15>
    -John ... I float in liquid gardens
    UTC -7ąDS

  6. #6
    Uranium Lounger
    Join Date
    Dec 2000
    Location
    Salt Lake City, Utah, USA
    Posts
    9,508
    Thanks
    0
    Thanked 6 Times in 6 Posts

    Re: Bit-wise File Compareing Utility

    I still have PCMag's Dupeless but I haven't run it on 2000 or XP; it will definitely find bitwise dupes.
    -John ... I float in liquid gardens
    UTC -7ąDS

  7. #7
    Platinum Lounger
    Join Date
    Jan 2001
    Location
    Quedgeley, Gloucester, England
    Posts
    5,333
    Thanks
    0
    Thanked 1 Time in 1 Post

    Re: Bit-wise File Compareing Utility

    Well, John, funny you should mention a BATch solution, which in many circumstances is infinitely preferable to a GUI one!

    But in any event a little preliminary systems analysis is required...

    Obviously the way to reduce the number of tests to rather less than 60000! (factorial) is to realise that only those files of identical size need be compared. Then COMP could be used, in the usual way. Say there are four files of a particular size, A, B, C and D. You'd have to compare A with B, and with C and with D; then B with C and with D; then C with D. Six tests here. Someone mathematical can tell me the general formula for the number of tests needed to compare n files!
    <font face="Script MT Bold"><font color=blue><big><big>John</big></big></font color=blue></font face=script>

    Ita, esto, quidcumque...

  8. #8
    Plutonium Lounger
    Join Date
    Nov 2001
    Posts
    10,550
    Thanks
    0
    Thanked 7 Times in 7 Posts

    Re: Bit-wise File Compareing Utility

    > Someone mathematical can tell me the general formula for the number of tests needed to compare n files!

    Somone not very mathematical can tell you too! Although it would be easier if I had a Sigma key on this keyboard.
    To compare n files you need (n-1) + (n-2) + ... + 1 comparisons. This us usually written as Sigma(n-1), and is equal to n * (n-1) / 2

    So to compare 9 files you would need 9 * 8 / 2 = 36 comparisons.

    StuartR

  9. #9
    Uranium Lounger
    Join Date
    Dec 2000
    Location
    Salt Lake City, Utah, USA
    Posts
    9,508
    Thanks
    0
    Thanked 6 Times in 6 Posts

    Re: Bit-wise File Compareing Utility

    Which command line function are you going to use to compare file sizes? Or are you just going to sort by size, and if so then how do you feed the batch file files of same size?
    -John ... I float in liquid gardens
    UTC -7ąDS

  10. #10
    Platinum Lounger
    Join Date
    Jan 2001
    Location
    Quedgeley, Gloucester, England
    Posts
    5,333
    Thanks
    0
    Thanked 1 Time in 1 Post

    Re: Bit-wise File Compareing Utility

    > Which command line function are you going to use to compare file sizes?

    From my previous post: "Then COMP could be used, in the usual way"

    > Or are you just going to sort by size, and if so then how do you feed the batch file files of same size?

    Almost everything in BATch land works off the FOR /F command. You surely don't want me to write the actual BAtch file??!
    I wasn't sure that the OP was actually serious!
    <font face="Script MT Bold"><font color=blue><big><big>John</big></big></font color=blue></font face=script>

    Ita, esto, quidcumque...

  11. #11
    Uranium Lounger
    Join Date
    Dec 2000
    Location
    Salt Lake City, Utah, USA
    Posts
    9,508
    Thanks
    0
    Thanked 6 Times in 6 Posts

    Re: Bit-wise File Compareing Utility

    Confound my inability to articulate! In the context of my earlier question, and knowing how good you are with batch files (having seen some of your work, and deciding that flattery might help <img src=/S/grin.gif border=0 alt=grin width=15 height=15>) my question rather is, within the BATch file, how will you feed the 60k files to COMP? I know how to write a simple batch FOR IN DO loop, but I can't imagine how to process 60,000 files, in pairs, with it.

    No, I wasn't really expecting you to write the BATch file, just to elucidate how it could be done. Which I suppose means writing it to demonstrate. <img src=/S/evilgrin.gif border=0 alt=evilgrin width=15 height=15>
    -John ... I float in liquid gardens
    UTC -7ąDS

  12. #12
    Platinum Lounger
    Join Date
    Nov 2001
    Location
    Melbourne, Victoria, Australia
    Posts
    5,016
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Bit-wise File Compareing Utility

    Wassim,

    This utility is decidedly not for bitwise comparison, but rather is aimed at weeding out images that "look" the same:

    "Dup Detector finds duplicate and near duplicate images by opening and reading image pixel data. Large collections and large image file sizes can slow Dup Detector but it works well running in the background. Large image collections often are filled with duplicate images under different names. Use Dup Detector to clean them.
    DupDetector reads 9 image file types from a folder or its subfolders or from an image list."

    It might be a useful <img src=/S/free.gif border=0 alt=free width=30 height=15> adjunct to what's already been recommended.

    Alan

  13. #13
    3 Star Lounger
    Join Date
    Feb 2001
    Posts
    369
    Thanks
    2
    Thanked 1 Time in 1 Post

    Re: Bit-wise File Compareing Utility

    <P ID="edit" class=small>(Edited by DaveA on 31-Jan-04 10:47. added URL code)</P>http://www.foldermatch.com

    It ain't free, but it will find your dupes, and lets you choose the criteria for calling something a duplicate. Their system for deleting dupes is klunky with a kapital k, but it does the trick nicely if you don't expect to find too many duplicates.

    Earlier versions of foldermatch didnt' have this function, but the current one does.

  14. #14
    5 Star Lounger
    Join Date
    Mar 2002
    Location
    Buenos Aires, Argentina
    Posts
    877
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Bit-wise File Compareing Utility

    Hey Stuart,

    <hr>To compare n files you need (n-1) + (n-2) + ... + 1 comparisons. This us usually written as Sigma(n-1), and is equal to n * (n-1) / 2<hr>
    I had never read of the Sigma(n-1) notation! I don't know exactly how to name it in English but it has to do with combinatorics. The number is nC2 (sub-n, C, sub-2) which indicates the number of possible combinations of n numbers taking two at a time. It can be written n! /[2! . (n-2)!], which ends up being what you said: n . (n-1) / 2.
    <img src=/w3timages/blue3line.gif width=33% height=2>
    <img src=/S/flags/Argentina.gif border=0 alt=Argentina width=30 height=18> <big><font color=4682b4><font face="Comic Sans MS">Diegol</font face=comic></font color=4682b4> </big>

  15. #15
    Platinum Lounger
    Join Date
    Jan 2001
    Location
    Quedgeley, Gloucester, England
    Posts
    5,333
    Thanks
    0
    Thanked 1 Time in 1 Post

    Re: Bit-wise File Compareing Utility

    Diegol

    Don't you have a course at American universities called "Math 101", which would deal with Permutations and Combinations? (all those sigmas and dabbas)

    (It's the sort of maths that would now be done, in simple form, in British schools at about age 14...) <img src=/S/clever.gif border=0 alt=clever width=15 height=15> <img src=/S/evilgrin.gif border=0 alt=evilgrin width=15 height=15>
    <font face="Script MT Bold"><font color=blue><big><big>John</big></big></font color=blue></font face=script>

    Ita, esto, quidcumque...

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •