Results 1 to 13 of 13
  1. #1
    2 Star Lounger
    Join Date
    Dec 2009
    Location
    near Boulder, Colorado, USA
    Posts
    112
    Thanks
    22
    Thanked 4 Times in 4 Posts

    Slow COPY on large files

    When I copy very large files (65GB Macrium backup file) from one USB HD to another USB HD, it takes 2 days or more (and the speed is 250 kB/sec or less), and often will not complete even in that time, giving a "failure to read file" error and stopping. If I copy a 5 GB file in the same way between the same two HDs, it goes fine (speed 11 MB/sec) and completes in 2 hours or so.

    Is this normal? Is there anything I can do to speed up the large file (aside from splitting it into pieces)?

  2. #2
    Star Lounger
    Join Date
    Apr 2011
    Posts
    99
    Thanks
    5
    Thanked 18 Times in 17 Posts
    Fat32 format has file size limits. Could that be the problem?

  3. The Following User Says Thank You to riffraff For This Useful Post:

    cosmlou (2014-10-09)

  4. #3
    Silver Lounger
    Join Date
    Mar 2014
    Location
    Forever West
    Posts
    2,072
    Thanks
    0
    Thanked 259 Times in 248 Posts
    Right, 4GB is the maximum single file size for FAT32-formatted HDDs, actually it's 1 Byte less than 4GB. FAT32 has other limitations:
    http://www.techrepublic.com/blog/win...nd-32gb-limit/

    I've noticed there is a performance hit when copying from one USB drive to another USB drive that does not exist
    when copying from an internal drive to a USB drive. Basically put, copying from one external drive to another requires first copying [temporarily] to the internal drive before sending the data to the other external drive. It's a lot like a print job going to the print spooler [buffer] before being sent to the printer, pretty much in the background/invisibly.

  5. The Following User Says Thank You to Berton For This Useful Post:

    cosmlou (2014-10-09)

  6. #4
    Bronze Lounger DrWho's Avatar
    Join Date
    Dec 2009
    Location
    Central Florida
    Posts
    1,501
    Thanks
    30
    Thanked 205 Times in 163 Posts
    Just for general information.

    As files and programs get bigger and bigger, the old USB 2.0 just isn't fast enough.
    So I've sworn off buying any more USB 2.0 drives of any kind.

    For the past year, I will buy only USB 3.0 devices, regardless of how cheap the USB 2.0 devices get.
    My motherboard does not have USB 3.0 ports on it, so.....
    I bought a USB 3.0 adapter card to plug into my motherboard, with four USB 3.0 ports on it.
    Now I can do quick data backups to both a 32 GB flash drive and a 1TB external USB 3.0 hard drive.

    And yes, if you're saving files over 4GB in size, your drives need to be formatted NTFS. (not FAT-32)
    (most flash drives, for instance, come pre-formatted as FAT-32 and will need to be RE-formatted to NTFS)

    Good Luck!

    The Doctor
    Experience is truly the best teacher.

    Backup! Backup! Backup! GHOST Rocks!

  7. The Following User Says Thank You to DrWho For This Useful Post:

    cosmlou (2014-10-27)

  8. #5
    WS Lounge VIP Coochin's Avatar
    Join Date
    Jun 2014
    Location
    Queensland, Australia
    Posts
    2,150
    Thanks
    31
    Thanked 305 Times in 265 Posts
    Quote Originally Posted by DrWho View Post
    Just for general information...
    Don't intend to "knock" what you have stated, but would like to point something out.

    Around the late 1990s (as I recall) the CEO of Intel Corp. stated that "computing power doubles every eighteen months".

    That assessment held fairly true until about 2005 or so, but since then the "doubling" seems to have accelerated.

    About 2005 we would have thought a 120GB or 160GB HDD to be huge, but now (2014) we would probably not bother with such a "small" HDD.

    My first PC (about 1995) had a 40MB HDD with Win3.1 and Win95 partitions, and I was wondering what to do with all that unused space.

    Many of us are now using 2, 3, or even 4TB HDDs quite happily, and I have seen quite a lot of discussion lately about the next level of HDD size, i.e.: "Exabyte" (1,000 Terabytes). Yikes!

    Re: USB2 vs USB3. Who knows? Is USB4 in the pipeline yet? (Haven't really looked yet).
    Computer Consultant/Technician since 1998 (first PC was Atari 1040STE in 1988).
    Most common computing error is EBKAC: Error Between Keyboard And Chairback
    AMD FX8120 (8-core @ 3.1GHz) CPU, Gigabyte GA-990FXA-D3 motherboard, 8GB (2x4GB) DDR3 1866MHz RAM, ATI-AMD Radeon HD6770 PCI-E VGA, 480GB Kingston SSD, 2TB Seagate SATA3.0 HDD, ASUS DVD/RW.

  9. The Following User Says Thank You to Coochin For This Useful Post:

    cosmlou (2014-10-27)

  10. #6
    2 Star Lounger
    Join Date
    Mar 2010
    Location
    Tampa, FL, USA
    Posts
    114
    Thanks
    11
    Thanked 10 Times in 9 Posts
    Quote Originally Posted by cosmlou View Post
    When I copy very large files (65GB Macrium backup file) from one USB HD to another USB HD, it takes 2 days or more (and the speed is 250 kB/sec or less), and often will not complete even in that time, giving a "failure to read file" error and stopping. If I copy a 5 GB file in the same way between the same two HDs, it goes fine (speed 11 MB/sec) and completes in 2 hours or so.

    Is this normal? Is there anything I can do to speed up the large file (aside from splitting it into pieces)?
    What process are you using to perform the copy? Drag and drop, ctrl-c/ctrl-v, Xcopy, other? If using native Windows (DnD or Ctrl-C/V), I'd suggest trying Teracopy or another "turbo-speed" copy program such as the programs reviewed in Gizmo's article on File Copiers. The ability to recover from errors or pause and resume can be invaluable for large files or large numbers of files

    HTH
    PJ in FL

  11. The Following 2 Users Say Thank You to pjustice57 For This Useful Post:

    cosmlou (2014-10-27),RolandJS (2014-11-05)

  12. #7
    Super Moderator
    Join Date
    Aug 2012
    Location
    Durham UK
    Posts
    6,606
    Thanks
    147
    Thanked 869 Times in 831 Posts
    You can tweak the performance of the USB HDs by right clicking on them in Device Manager/Disk drives - select Properties and the options are under the Policies tab.

  13. The Following User Says Thank You to Sudo15 For This Useful Post:

    cosmlou (2014-10-27)

  14. #8
    2 Star Lounger
    Join Date
    Dec 2009
    Location
    near Boulder, Colorado, USA
    Posts
    112
    Thanks
    22
    Thanked 4 Times in 4 Posts
    Thanks to all. Further experimentation/exploration shows all drives have the Policies set to high performance. (They are all NTFS.)

    When I copy a 6GB file from the 1TB USB drive to the internal HD of the computer, it still slows down to 1MB/sec or less after the first 15 seconds or so, and the projected time to complete the transfer is about 1 day. I have not yet tried to move such a file from the internal HD to the other USB drive.

    The usual way I manually copy a file is to right-click it in one directory, select Copy, and then right-click in the target directory and select Paste (or Move). When I write a batch file, I usually use Robocopy and use the Max size and MaxAge switches.
    I just tried Teracopy, but it too chokes at around 150MB copied (out of 6GB), and appears to stop working without any indication of what the problem is.

    When this problem first started, I reformated the 1TB USB drive (it took 4 or 5 days, as I recall, because QuickFormat would not run). So it is a fairly clean drive. I am beginning to wonder if it is just defective. Running dskchk on it shows no problems, tho.

  15. #9
    WS Lounge VIP Coochin's Avatar
    Join Date
    Jun 2014
    Location
    Queensland, Australia
    Posts
    2,150
    Thanks
    31
    Thanked 305 Times in 265 Posts
    Quote Originally Posted by cosmlou View Post
    ...When this problem first started, I reformated the 1TB USB drive (it took 4 or 5 days, as I recall, because QuickFormat would not run). So it is a fairly clean drive. I am beginning to wonder if it is just defective...
    A "quick" format on a 1TB HDD (even via USB) should take less than 2min.

    A "full" format on a 1TB HDD (even via USB) should not take more than about 2 hours.

    Seems there is something very wrong with your 1TB USB HDD.
    Computer Consultant/Technician since 1998 (first PC was Atari 1040STE in 1988).
    Most common computing error is EBKAC: Error Between Keyboard And Chairback
    AMD FX8120 (8-core @ 3.1GHz) CPU, Gigabyte GA-990FXA-D3 motherboard, 8GB (2x4GB) DDR3 1866MHz RAM, ATI-AMD Radeon HD6770 PCI-E VGA, 480GB Kingston SSD, 2TB Seagate SATA3.0 HDD, ASUS DVD/RW.

  16. The Following User Says Thank You to Coochin For This Useful Post:

    cosmlou (2014-11-15)

  17. #10
    Silver Lounger RolandJS's Avatar
    Join Date
    Dec 2009
    Location
    Austin metro area TX USA
    Posts
    1,722
    Thanks
    95
    Thanked 126 Times in 123 Posts
    Coochin, not long ago, I full formatted both of my usb external HDs, I think the Great Wall of China took less time to build. Anyway, I did that to ensure that any bad sector would be locked out. My backups and restores are very important.
    "Take care of thy backups and thy restores shall take care of thee." Ben Franklin revisited.
    http://collegecafe.fr.yuku.com/forum...-Technologies/

  18. The Following User Says Thank You to RolandJS For This Useful Post:

    cosmlou (2014-11-15)

  19. #11
    2 Star Lounger
    Join Date
    Dec 2009
    Location
    near Boulder, Colorado, USA
    Posts
    112
    Thanks
    22
    Thanked 4 Times in 4 Posts
    /Coochin/ L: is my 1-TB USB HD.
    I ran chkdsk L: /f /v and it showed no problems in any area. Is there a better test of disk health that I should run?

  20. #12
    Silver Lounger RolandJS's Avatar
    Join Date
    Dec 2009
    Location
    Austin metro area TX USA
    Posts
    1,722
    Thanks
    95
    Thanked 126 Times in 123 Posts
    Copying large files or large number of files, I also use Teracopy. While it does have a speed-performance hit under certain conditions, it appears to be problem-free as far as read/write errors go. I have mine set to allow my choice of Teracopy or Explore -- because they both have feature/benefit differences.
    "Take care of thy backups and thy restores shall take care of thee." Ben Franklin revisited.
    http://collegecafe.fr.yuku.com/forum...-Technologies/

  21. #13
    WS Lounge VIP Coochin's Avatar
    Join Date
    Jun 2014
    Location
    Queensland, Australia
    Posts
    2,150
    Thanks
    31
    Thanked 305 Times in 265 Posts
    Quote Originally Posted by cosmlou View Post
    /Coochin/ L: is my 1-TB USB HD.
    I ran chkdsk L: /f /v and it showed no problems in any area. Is there a better test of disk health that I should run?
    Look up the drive's model number on the manufacturer's website; there should be diagnostic software available, and there might be a firmware upgrade available. I have upgraded the firmware on several 1TB and 2TB Seagate drives which Windows was complaining about "controller errors".
    Computer Consultant/Technician since 1998 (first PC was Atari 1040STE in 1988).
    Most common computing error is EBKAC: Error Between Keyboard And Chairback
    AMD FX8120 (8-core @ 3.1GHz) CPU, Gigabyte GA-990FXA-D3 motherboard, 8GB (2x4GB) DDR3 1866MHz RAM, ATI-AMD Radeon HD6770 PCI-E VGA, 480GB Kingston SSD, 2TB Seagate SATA3.0 HDD, ASUS DVD/RW.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •