Results 1 to 5 of 5
  1. #1
    Platinum Lounger
    Join Date
    Nov 2001
    Location
    Melbourne, Victoria, Australia
    Posts
    5,016
    Thanks
    0
    Thanked 0 Times in 0 Posts

    VBS to download web pages (Win98)

    I wasn't really sure of where to post this query, but thought it might lend itself to scripting. I'm wanting to cycle through a series of URLs and download/save all of the web pages. These links are cgi links, with pages generated at the server. I know that there are about 70 pages in the "series", with their URLs varying only by one number. For example:
    http://www.somesite.com/cgi-bin/showlist.p...s&Showpage=<font color=red>*</font color=red>&Listings=30

    The <font color=red>*</font color=red> would take on any value from 0 to (about) 70, for the pages of interest.

    I know that I can type this to a browser (say, with *=34) and get the correct page. But efforts to cycle through the series with a download manager won't work, for whatever reason. I'd like a script to do the looping and downloading. I'd only need to save the HTML on the page (graphics and anything else not needed).

    Any help with this would be appreciated.

    Alan

  2. #2
    Super Moderator jscher2000's Avatar
    Join Date
    Feb 2001
    Location
    Silicon Valley, USA
    Posts
    23,112
    Thanks
    5
    Thanked 93 Times in 89 Posts

    Re: VBS to download web pages (Win98)

    I hope you don't plan to do this to the Lounge! Our server is working hard enough as it is. <img src=/S/grin.gif border=0 alt=grin width=15 height=15>

    The ADODB Stream object so much in the news of late as a conduit for spyware could be your answer. I've never used it, but clearly it can do the job. Or should I say, could do the job. You probably would have to reverse the kill bit in the registry in order to script it... probably a good idea to reset it after you're done.

    If the server denies connections that do not supply appropriate referrer or useragent strings, or saved cookies, then the Stream object probably won't help you. It is a bit of work to script Internet Explorer itself, but that is another option. (I've only done that from Word VBA, not from VBS.)

  3. #3
    Star Lounger
    Join Date
    Mar 2001
    Location
    Atlanta, Georgia, USA
    Posts
    64
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: VBS to download web pages (Win98)

    Actually, I've done this, but I can't find the details at the moment. If you do an internet search, using 'readyState' and 'WSH' as keywords, you should find some good examples of reading a web page through scripting.

  4. #4
    Platinum Lounger
    Join Date
    Nov 2001
    Location
    Melbourne, Victoria, Australia
    Posts
    5,016
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: VBS to download web pages (Win98)

    <hr>I hope you don't plan to do this to the Lounge! <hr>It hadn't crossed my mind, but it's an interesting idea, now that you mention it. <img src=/S/evilgrin.gif border=0 alt=evilgrin width=15 height=15>

    I will have a look at the ADODB Stream object. I always forget about the VBA possibility <img src=/S/stupidme.gif border=0 alt=stupidme width=30 height=30>. I guess it's one of those old habits - if it's too trivial for a program, then go for a script.

    I actually managed to coerce a download manager to recurse a list of the repetitious URLs, which I built using Excel. It wouldn't entertain a file on HD, so I had to upload it as a HTML webpage, and use the URL of that for the base of the download "tree". This particular software deals with "complex" URLs (form submissions, cgi etc.) by setting up a "special" temporary proxy address in the browser, then somehow sucking out the appropriate URLs for downloading. Don't ask me how.

    Thanks

    Alan

  5. #5
    Platinum Lounger
    Join Date
    Nov 2001
    Location
    Melbourne, Victoria, Australia
    Posts
    5,016
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: VBS to download web pages (Win98)

    This is a very useful tip Briana <img src=/S/cheers.gif border=0 alt=cheers width=30 height=16>. This page is heading towards exactly what I want - not just the pages, but processed information from them. I can spot the possibility for a recursive loop just glancing at this example. Now that I've managed to get the files though <post#=406081>post 406081</post#> I might try to adapt this for post-processing.

    Thank you Briana, for a great lead <img src=/S/thankyou.gif border=0 alt=thankyou width=40 height=15>.

    Alan

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •