Results 1 to 2 of 2
  1. #1
    Star Lounger
    Join Date
    Jan 2001
    Location
    Charlottetown, Prince Edward Island, Canada
    Posts
    76
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Syncsort on a sun platform

    i am trying to use syncsort to perform a sort and sum of a variable length file that contains anywhere from 300-400 million rows of data. the issue is the amount of disk spaces required to do this is maxing out resources. It is running on a sun system. The file itself is actually fed by 24 independent files that are merged into one before a final step.

    are there any chances of efficiencies here or maybe a better way of using syncsort prior to the last step or during the last step?

    thanks
    mike

    ps... Eileen thought this might be a better forum to place this question.

  2. #2
    Star Lounger
    Join Date
    Jan 2001
    Posts
    68
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Syncsort on a sun platform

    If you only need the sum, you don't need to sort the file at all. That might be a SyncSort option. And you could sum each of the original files and then sum the sums -- manually if no other way.

    If you really need to a sorted copy of the combined files, try sorting each source file first. Then merge the sorted source files. The individual sorts should require less resources (at least CPU and disk space) because the individual files are smaller and the sort work areas can be reused. The merge requires far less resources than a sort because no actual sorting is done (so very little if any disk work space is required). For little files, the extra steps aren't worth it. For big files, it can be faster to sort in steps (this is the way it was done back when mainframes had the computing power of today's PDA).

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •