Results 1 to 13 of 13
  1. #1
    New Lounger
    Join Date
    Dec 2009
    Location
    Hamilton Ontario Canada
    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Need a File Management tool

    Hi Folks,

    I know the following request is a little old school but please humour me; there is method in my madness. For archival purposes, I am looking to organize and then store a bunch of image files on DVDs. At the moment the files are in a directory structured by the date on which the image was created, i.e. the top level directory is called 2015, then there are 30+ directories called 2015-01-24, 2015-01-25, etc. within each directory are pairs of files that need to stay together - for example files named image0001.nef and image0001.xmp need to be kept in the same directory when written to DVD. There can be as few as 2 files in each subdirectory and as many as 1600. The total number of files is in the 30,000 range.

    I would like to copy the image files from their current directories into a set of sequentially number directories that will then be burned to a disk. The destination directories, which must be close to but less than 4.5gb in size, will be called RAW_001, RAW_002... with just the paired image files with their original file names preserved in each new RAW_### directory - no date structure. Finally, the files that get burned to the DVD must be readable without any de-compression or back-up software. Also, DVDs are pretty cheap so I don't really need to squeeze every last byte of storage out of each disk - it is far more important that I be able to keep the paired files together.

    I have looked at a number of tools including WinRAR, KC's Ignition, Comodo Back-up, Acronis True Image, and others but I haven't found anything that will to the job. Any suggestions??

  2. #2
    Silver Lounger wavy's Avatar
    Join Date
    Dec 2009
    Location
    ny
    Posts
    2,378
    Thanks
    235
    Thanked 147 Times in 136 Posts
    Quote Originally Posted by DWTNA View Post
    Hi Folks,

    I know the following request is a little old school but please humour me; there is method in my madness. For archival purposes, I am looking to organize and then store a bunch of image files on DVDs. At the moment the files are in a directory structured by the date on which the image was created, i.e. the top level directory is called 2015, then there are 30+ directories called 2015-01-24, 2015-01-25, etc. within each directory are pairs of files that need to stay together - for example files named image0001.nef and image0001.xmp need to be kept in the same directory when written to DVD. There can be as few as 2 files in each subdirectory and as many as 1600. The total number of files is in the 30,000 range.
    ?
    As in HDD images? How big are those files??
    David

    Just because you don't know where you are going doesn't mean any road will get you there.

  3. #3
    Super Moderator jwitalka's Avatar
    Join Date
    Dec 2009
    Location
    Minnesota
    Posts
    6,796
    Thanks
    117
    Thanked 799 Times in 720 Posts
    Quote Originally Posted by wavy View Post
    As in HDD images? How big are those files??
    I would guess they are photo images.

    Jerry

  4. #4
    Silver Lounger wavy's Avatar
    Join Date
    Dec 2009
    Location
    ny
    Posts
    2,378
    Thanks
    235
    Thanked 147 Times in 136 Posts
    Perhaps, I am not familiar w/ the extension mentioned. One seems to be for metadata. I am wondering about the need to rename directories with sequential numbers.

    David

    Just because you don't know where you are going doesn't mean any road will get you there.

  5. #5
    jwoods
    Guest
    I would put them on an external hard drive (1 TB +).

    Then, you won't have to worry about exceeding the size limit of the DVD and having the pairs separated.

    You might consider a program like Beyond Compare...

    http://scootersoftware.com/

    With that, you could also use your original folder names on the external drive, which is helpful if you ever need to restore something down the road and forget what RAW_001, etc. matches up to.
    Last edited by jwoods; 2015-11-10 at 21:14.

  6. #6
    New Lounger
    Join Date
    Dec 2009
    Location
    Hamilton Ontario Canada
    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by wavy View Post
    As in HDD images? How big are those files??
    No, not HDD images, just raw files from my camera. Average size is about 26mb but there are some that I have run through photoshop that can up to 750+mb.

  7. #7
    New Lounger
    Join Date
    Dec 2009
    Location
    Hamilton Ontario Canada
    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by wavy View Post
    Perhaps, I am not familiar w/ the extension mentioned. One seems to be for metadata. I am wondering about the need to rename directories with sequential numbers.

    The sequentially number directories are needed as part of a cataloguing system. I am making a redundant backup system based on a model outlined in a book called "The Dam Book - Digital Asset Management for Photographers, 2nd ed." The DVDs/BLURAYs are for secondary offsite backup and directory/file structure on the DVDs has to match the structure that is used in the primary backup.
    Last edited by DWTNA; 2015-11-11 at 00:57.

  8. #8
    New Lounger
    Join Date
    Dec 2009
    Location
    Hamilton Ontario Canada
    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by jwoods View Post
    I would put them on an external hard drive (1 TB +).

    Then, you won't have to worry about exceeding the size limit of the DVD and having the pairs separated.

    You might consider a program like Beyond Compare...

    http://scootersoftware.com/

    With that, you could also use your original folder names on the external drive, which is helpful if you ever need to restore something down the road and forget what RAW_001, etc. matches up to.
    I intend to use external HDs as the primary onsite back-up media, the DVDs (actually will probably use BLURAYs) are secondary backups. The cataloging software keeps track of which directory a particular file is in so I don't have to worry about it - provided I always have a good back-up strategy for my catalogue database

    Thanks for the tip about Beyond Compare - I will have a look at it to see if it will do the job.

  9. #9
    WS Lounge VIP
    Join Date
    Dec 2009
    Location
    Earth
    Posts
    8,191
    Thanks
    48
    Thanked 986 Times in 916 Posts
    I expect good quality DVDs will last 10 years but you need to check them regularly and have 2 copies in case of a failure. A hard disk should last at least 5 years, especially if it's turned off and stored, but again, it needs to be checked regularly. Of the two, a hard disk is much easier to create / manage / store / check, so that is what I use.

    It's also worth mentioning file verification and recovery as Windows file systems are poor in this respect. NTFS does not support file recovery through checksums / redundant data, although hard disks do this internally. Writing everything to a ZIP file can give you recovery of corrupted sections of the file, but it's a lot of work and hard to check later. A better method may be to calculate the checksum of the entire disk and test this regularly, although recovery is not guaranteed. ZFS would be a solution, but it's not available in Windows. Ultimately, a commercial backup program with its own recoverable file format provides a good storage solution and regular checks of those files is your housekeeping task.

    Some further reading: http://www.pcmech.com/article/how-lo...up-media-last/
    Big Disks: http://blog.fosketts.net/2014/12/19/...rity-checking/
    File Verification (don't trust the file system to do this for you): http://www.techsupportalert.com/Bril...cation-Utility
    Microsoft PowerShell Checksum Utility: https://gallery.technet.microsoft.co...cksum-e57dcd67

    cheers, Paul
    Last edited by Paul T; 2015-11-11 at 03:05.

  10. #10
    New Lounger
    Join Date
    Dec 2009
    Location
    Hamilton Ontario Canada
    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Paul T View Post
    I expect good quality DVDs...
    cheers, Paul
    Thanks for your input. I was aware of the issues with OM vs HDD and I haven't decided yet but I leaning towards DVD-RAM because of its build-in file recovery system and the archival shelf-life. If the prices of SSD come down a bit more then it might make a good secondary. My preference would be tape but I can't justify the cost.

    I am using a file verification system that is very similar to PsFCIV but in addition to a key-hash checksum database it also checks the file structure of the image files against their published specifications. Nevertheless, I like the idea of ExactFile creating an executable that can be recorded on each media. If stored as the autoexec file, a disk could check itself simply by inserting it into the drive.

    Anyway, what I really need is a means to organize the files and directory structures on both the primary and secondary backups is identical but scalable, future-proof, cost effective and not too time consuming to use. It is beginning to look like I might have to write something myself. It may be time to brush up on my REXX.

    Take Care and thanks again,
    Doug

  11. #11
    Super Moderator RetiredGeek's Avatar
    Join Date
    Mar 2004
    Location
    Manning, South Carolina
    Posts
    9,436
    Thanks
    372
    Thanked 1,457 Times in 1,326 Posts
    Quote Originally Posted by DWTNA View Post
    T It is beginning to look like I might have to write something myself. It may be time to brush up on my REXX.

    Take Care and thanks again,
    Doug
    Doug,

    Ah REXX I remember it fondly. I used to use it on IBM mainframes and just loved it. I used it back in the 80's as the controlling language for a major system that tied together Database, Email, Messaging, and a few things I can't mention.

    I've tried a few PC implementations but never found one I really felt was properly integrated the way it was on VM/CMS. If you decide to DIY you may want to look into PowerShell as I know you can do it with PS and it should be around for quite a while since it is MS's new command language and is fully OO.

    As Bob Hope used to say: "Thanks for the memories!"

    HTH
    May the Forces of good computing be with you!

    RG

    PowerShell & VBA Rule!

    My Systems: Desktop Specs
    Laptop Specs

  12. #12
    New Lounger Zahid iqbal's Avatar
    Join Date
    Oct 2015
    Posts
    21
    Thanks
    0
    Thanked 1 Time in 1 Post
    You can use Total Commander, Google Drive, DropBox, Evernote Business.

  13. #13
    5 Star Lounger
    Join Date
    Dec 2009
    Location
    Delaware, US
    Posts
    1,171
    Thanks
    19
    Thanked 99 Times in 88 Posts
    There are a number of file managers you can get, I use Total Commander. But you have more of an organizational issue than anything. And with 30,000 files, it's more than a weekend project.

    I'm a database guy so my solution would be to read the directories and generate CMD files to create directories and move files. But that's a strictly custom solution as would be the notion of writing some kind of program.

    I have no doubt that there are programs that will do some kind of organization, but they would be business level and likely quite expensive. But if the book doesn't mention anything, it might not exist.
    Graham Smith
    DataSmith, Delaware
    "For every expert there is an equal and opposite expert.", Arthur C. Clarke (1917 - 2008)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •