Results 1 to 6 of 6
  1. #1
    2 Star Lounger
    Join Date
    Mar 2002
    Location
    Germany, Germany
    Posts
    169
    Thanks
    0
    Thanked 0 Times in 0 Posts

    How to deal with huge data volume? (XP/SP3)

    Hello everybody!

    I have done some calculations and the results are collected in an CSV-file. This file contains about 6.800.000 lines, each is unique, and every line has about 25 values. Now I have to find 130 lines within this file with the result being dependent on some filter criteria. The lines have to be reorganized and the extract has to be displayed as a 3D surface chart. Now I wonder what would be the fastest way to access and filter the data. The CSV-file is about 1.5 GB. Would it be best to convert it in some kind of database, or should it be kept as a CSV file? For filtering, is it a good idea to work with a Pivot table on external data? One request on a possible solution is, that it has to be very efficient. The idea is to have a form with some sliders on it which determine the filter criteria. I hope for a solution that enables me to get the result immediately after moving the slider - but maybe this is just a dream ;-)
    I have never had to deal with such a data volume and have no idea which way to go. So if anybody has some hints, they are very much appreciated.

  2. #2
    Plutonium Lounger
    Join Date
    Mar 2002
    Posts
    84,353
    Thanks
    0
    Thanked 29 Times in 29 Posts

    Re: How to deal with huge data volume? (XP/SP3)

    I'd import the data into an Access database, or even better, a SQL Server database. You can then index the imported table on the relevant fields, so that sorting and filtering on those fields will be much faster.

  3. #3
    2 Star Lounger
    Join Date
    Mar 2002
    Location
    Germany, Germany
    Posts
    169
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: How to deal with huge data volume? (XP/SP3)

    Hello Hans,

    thank you for your fast reply. I have feared somebody would answer this. For the SQL solution: would it mean to run an SQL server on the target computer? I have to deliver the data and the related excel sheet to a customer. I know he has Excel, and I know that he will not be allowed to install additional software on his computer. So could an SQL database still be a solution? Would it work with an Access database ( I do not know whether Access is available at our customer)?
    Thanks in advance.

  4. #4
    Uranium Lounger
    Join Date
    Jan 2001
    Location
    South Carolina, USA
    Posts
    7,295
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: How to deal with huge data volume? (XP/SP3)

    An SQL database would be better because it would be faster and more efficient. If the customer does not have and will not get SQL, then Access should work but would be much slower. Once the data is in the data base, the relevent 130 lines of data could be filtered out and exported to Excel for analysis if desired, or depending on the filter process might be extracted into Excel using MS Query.
    Legare Coleman

  5. #5
    2 Star Lounger
    Join Date
    Mar 2002
    Location
    Germany, Germany
    Posts
    169
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: How to deal with huge data volume? (XP/SP3)

    Thanks a lot for your suggestions. I will try with an SQL database and see how I can deal with it.

  6. #6
    Plutonium Lounger
    Join Date
    Mar 2002
    Posts
    84,353
    Thanks
    0
    Thanked 29 Times in 29 Posts

    Re: How to deal with huge data volume? (XP/SP3)

    If the client is not allowed to install software, SQL Server is out of the question. You could store the data in an Access database and create a pivot table in Excel based on the Access table, or use MS Query to import the rows you need. The user doesn't need to have Access installed for this. But, as Legare remarked, don't expect stellar performance.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •