Page 1 of 2 12 LastLast
Results 1 to 15 of 16
  1. #1
    New Lounger
    Join Date
    Sep 2011
    Posts
    22
    Thanks
    5
    Thanked 0 Times in 0 Posts

    https vs http results in different file download

    Can someone explain what may be going on here...

    I was trying to update a Broadcom Bluetooth driver (BTW_12.0.1.940_win8_10_x64.zip) for a Win10 install on an older laptop. Initially (Aug 8), I used Https to connect to Broadcom's site but Firefox complained about an untrusted intermediate certificate; so did Chrome, but let me connect. I downloaded the driver file over https on Chrome. I also downloaded the file on a different PC using http with Firefox and noticed a different file size and sha256sum.

    No matter what machine/browser/OS I use, whether I use a proxy or not or if I use a different ISP, the file I receive over https is always 77 bytes larger than the one over http. Each version has a consistent sha256sum but different from each other.

    I had VirusTotal check both versions of the file and they came back clean. SSLLabs indicates an incomplete server certificate chain for Broadcom, but yesterday FF didn't complain as much and let me connect without asking to make an exception.

    What would explain the files having a different sha256sum and size just because it was downloaded over this https environment? What security/vulnerability is here?

  2. #2
    WS Lounge VIP mrjimphelps's Avatar
    Join Date
    Dec 2009
    Location
    USA
    Posts
    3,401
    Thanks
    447
    Thanked 404 Times in 376 Posts
    Anytime you use https, it will look for a valid certificate. If it doesn't find one, it will give you an error message. Getting the error message does not necessarily indicate a security violation; rather, it indicates that the certificate has some error (e.g. perhaps it is expired). If the certificate has some error, then you can't trust the security of the link, just like you can't trust it when using http.

    As to your question about the different sizes of downloads, I don't have an answer for that one.

  3. #3
    Super Moderator Rick Corbett's Avatar
    Join Date
    Dec 2009
    Location
    South Glos., UK
    Posts
    2,143
    Thanks
    101
    Thanked 579 Times in 464 Posts
    Quote Originally Posted by WS Subber
    What would explain the files having a different sha256sum and size just because it was downloaded over this https environment?
    The downloaded files have different SHA checksums and file sizes because their attached 'zone information' metadata, stored in each file as an alternate data stream (with a stream name of :Zone.Identifier:$DATA), is different.

    It's this 'zone information' that causes Windows to display a warning about files that originated 'elsewhere', e.g. downloaded from the internet or transferred from another device.

    Hope this helps...

    EDIT... more info:

    You may be able to see this for yourself using Nir Sofer's AlternateStreamView. For more info, including how to control this behaviour, have a look at Disable downloaded files from being blocked in Windows 10.
    Last edited by Rick Corbett; 2016-09-13 at 09:11. Reason: Added more info

  4. The Following 3 Users Say Thank You to Rick Corbett For This Useful Post:

    Lugh (2016-09-15),mrjimphelps (2016-09-13),WS Subber (2016-09-13)

  5. #4
    New Lounger
    Join Date
    Sep 2011
    Posts
    22
    Thanks
    5
    Thanked 0 Times in 0 Posts
    Thanks Jim, Rick.

    The idea of 'zone information' metadata is a new one for me so I'll have to check up on it. I understand http is not secure and trustworthy. Neither is an https connection with an incomplete certificate chain.

    So the fact of getting different files over https vs http is not in and of itself so unusual?

  6. #5
    Super Moderator Rick Corbett's Avatar
    Join Date
    Dec 2009
    Location
    South Glos., UK
    Posts
    2,143
    Thanks
    101
    Thanked 579 Times in 464 Posts
    Quote Originally Posted by WS Subber
    So the fact of getting different files over https vs http is not in and of itself so unusual?
    It's the same file, only the metadata is different. Here's a different analogy: You go to the store and buy a dozen eggs. The checkout operator gives you them in a bag. You ask for them to be double-bagged. The differing metadata is the difference between one bag or two... but the eggs remain the same.

    For more info about ADS see Windows Alternate Data Streams.

    For info about removing ADS using PowerShell, see Alternate Data Streams in NTFS.

    Hope this helps...
    Last edited by Rick Corbett; 2016-09-13 at 09:54. Reason: Added more info

  7. The Following User Says Thank You to Rick Corbett For This Useful Post:

    WS Subber (2016-09-13)

  8. #6
    New Lounger
    Join Date
    Sep 2011
    Posts
    22
    Thanks
    5
    Thanked 0 Times in 0 Posts
    Rick, those were some very good resources on ADS.

    I have had a chance now to review them and have checked out my two files (versions). Both SysInternals' Streams64 and Get-Item from W10 PowerShell indicate there are no ADS related to my files (just the :$DATA). I even unzipped each of them and ran Streams64 with -s but it found no ADS. I am at a loss to know why the same file downloaded from the same site is different based on whether downloaded via https or http.

  9. #7
    Super Moderator Rick Corbett's Avatar
    Join Date
    Dec 2009
    Location
    South Glos., UK
    Posts
    2,143
    Thanks
    101
    Thanked 579 Times in 464 Posts
    Strange. How can Windows know that a file is not from Zone 0 (i.e. local computer), and has come from 'elsewhere' if it doesn't have an ADS containing at least one stream showing the zone identifier e.g. Zone 3 (internet)? The only reason I can think of is if the filesystem is not NTFS.

    Have you considered using something like Beyond Compare, DiffVue or UltraCompare and carrying out a Hex comparison.
    Last edited by Rick Corbett; 2016-09-14 at 04:39. Reason: Added more info

  10. #8
    Silver Lounger RolandJS's Avatar
    Join Date
    Dec 2009
    Location
    Austin metro area TX USA
    Posts
    1,730
    Thanks
    95
    Thanked 128 Times in 125 Posts
    Quote Originally Posted by WS Subber View Post
    ...I am at a loss to know why the same file downloaded from the same site is different based on whether downloaded via https or http.
    A previous poster gave this nugget:
    "...It's the same file, only the metadata is different. Here's a different analogy: You go to the store and buy a dozen eggs. The checkout operator gives you them in a bag. You ask for them to be double-bagged. The differing metadata is the difference between one bag or two... but the eggs remain the same." -- a snippit from a Rick Corbett post
    Apparently, the same web site, download site, either single-bags or double-bags the file.
    "Take care of thy backups and thy restores shall take care of thee." Ben Franklin revisited.
    http://collegecafe.fr.yuku.com/forum...-Technologies/

  11. #9
    Lounger
    Join Date
    May 2014
    Posts
    45
    Thanks
    0
    Thanked 2 Times in 2 Posts
    There is no requirement for HTTP and HTTPS to point at the same file, even though it's a reasonable expectation.

    Could be a caching issue; try downloading again later.

    --Scott.

  12. #10
    5 Star Lounger Lugh's Avatar
    Join Date
    Jun 2010
    Location
    Indy
    Posts
    620
    Thanks
    166
    Thanked 77 Times in 68 Posts
    Quote Originally Posted by Scott McNay View Post
    Could be a caching issue
    That was my thought also. My novice 'understanding' is that one of the main differences between HTTP and HTTPS is that the latter doesn't allow caching. So you might be getting an older version of the file from HTTP.
    Lugh.
    ~
    Windows 10 Pro x64 1607; Office 2016 (365 Home) x32; Win Defender, MBAM Pro

    ASRock H97 Anniversary; Xeon E3-1231V3 (like i7)
    Gigabyte GeForce GTX 970; 12GB Crucial DDR3 1600
    Logitech MX Master mouse; Roccat Isku kb

  13. #11
    WS Lounge VIP Browni's Avatar
    Join Date
    Dec 2009
    Location
    Rochdale, UK
    Posts
    1,651
    Thanks
    38
    Thanked 161 Times in 139 Posts
    A hex comparison of the files would be interesting.

  14. #12
    New Lounger
    Join Date
    Sep 2011
    Posts
    22
    Thanks
    5
    Thanked 0 Times in 0 Posts
    I am conscious of trying to stay on track here, so bear with me...

    Quote Originally Posted by Rick Corbett View Post
    Strange. How can Windows know that a file is not from Zone 0 (i.e. local computer), and has come from 'elsewhere' if it doesn't have an ADS containing at least one stream showing the zone identifier e.g. Zone 3 (internet)? The only reason I can think of is if the filesystem is not NTFS.
    You're correct. I have different machines available to me depending on where I am during the day. When I first tried to test the ADS issue, I had a Win7 PC and the -stream option was bugging out. I guess it doesn't work with Win7. So I used a Win10 install within a VM. The use of the VM apparently had stripped out the zone identifier streams (all ADS I guess). When doing the ADS checks on the Win10 laptop itself, those zone identifier streams were present. They were the same for all the files and no other ADS streams were there.

    Quote Originally Posted by Rick Corbett View Post
    Have you considered using something like Beyond Compare, DiffVue or UltraCompare and carrying out a Hex comparison.
    I'll check those out to try and find what is different between the two versions of the zip files.

    Quote Originally Posted by Scott McNay View Post
    There is no requirement for HTTP and HTTPS to point at the same file, even though it's a reasonable expectation.

    Could be a caching issue; try downloading again later.
    It was my assumption that they would point to the same file, just without/with encrypted communications. Re: caching, the differences have been consistent for at least a week.

    The original Firefox certificate issue and sslLabs comment of an incomplete certificate chain raised my suspicions, but the fact that SHA checksums and file sizes were different (ADS now ruled out as a possible cause) stoked the fire. It is hard to know what to have trust in.
    Last edited by WS Subber; 2016-09-15 at 11:54. Reason: error correction

  15. #13
    New Lounger
    Join Date
    Sep 2011
    Posts
    22
    Thanks
    5
    Thanked 0 Times in 0 Posts
    After doing the Hex comparison (I'd never done that before), the differences are accounted for by 77 instances of various pointers to the broadcom website. The https version included the extra "s" character 77 times. See attached screen grab.
    BC_hex_pic4.png

  16. #14
    WS Lounge VIP mrjimphelps's Avatar
    Join Date
    Dec 2009
    Location
    USA
    Posts
    3,401
    Thanks
    447
    Thanked 404 Times in 376 Posts
    Very interesting comparison. You'll note that the hexadecimal numbers in the left column of each window show the total number of bytes up to that point. Each time the "s" is missing on the left side, the total number of bytes shown in the next row is one less with the left window vs the right window.

    The hexadecimal digits are 0 1 2 3 4 5 6 7 8 9 A B C D E F. That's why, when you have eight bytes, you count from 28 to 30, whereas when you have seven bytes, you count from 28 to 2F -- ...25, 26, 27, 28, 29, 2A, 2B, 2C, 2D, 2E, 2F, 30, 31, 32...

    By comparison, the decimal digits are 0 1 2 3 4 5 6 7 8 9. Counting in decimal would be: ...25, 26, 27, 28, 29, 30, 31, 32...

    Decimal is base 10, whereas hexadecimal is base 16.

    It's very handy the way the Hex Compare program keeps everything aligned between the two windows so that you can easily see what is missing on the left.
    Last edited by mrjimphelps; 2016-09-16 at 14:50.

  17. #15
    Super Moderator Rick Corbett's Avatar
    Join Date
    Dec 2009
    Location
    South Glos., UK
    Posts
    2,143
    Thanks
    101
    Thanked 579 Times in 464 Posts
    Quote Originally Posted by WS Subber
    After doing the Hex comparison (I'd never done that before), the differences are accounted for by 77 instances of various pointers to the broadcom website. The https version included the extra "s" character 77 times.
    The file you are querying with BeyondCompare is the 'Broadcom WIDCOMM Bluetooth Driver 12.0.1.940' so it's no surprise that there are references to the Broadcom website within the HEX comparison of the driver ZIP file.

    As the only apparent differences between the two downloads are the inclusion of 's' (as in HTTPS) and VirusTotal comes back clean on both files I don't think you have anything to worry about.

    Hope this helps...

Page 1 of 2 12 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •