XP SP3 triggers false positives in security apps

Scott dunn By Scott Dunn

Installing Windows XP Service Pack 3 can cause your anti-malware programs to report the presence of Trojans and keyloggers that aren’t there.

The false positives have blocked important system files in some cases, and in others they have misled users into reinstalling XP.

SP3 causes some malware scanners to cry “wolf”

Comments on a PC Tools forum confirm customer reports that the company’s Spyware Doctor program generates a false positive on systems with Windows XP SP3.

Get our unique weekly Newsletter with tips and techniques, how to's and critical updates on Windows 10, Windows 8, Windows 7, Firefox, Internet Explorer, Google, etc. Join our 460,000 subscribers!

Enter your email above to receive messages about offerings by Penton, its brands, affiliates and/or third-party partners, consistent with Penton's Privacy Policy.
The Windows 7, Vol 3 (Excerpt)

Subscribe and get our monthly bonuses - free!

The Windows 7 Guide, Volume 3: Advanced maintenance and troubleshooting provides advanced tools for keeping Microsoft's premier operating system up and running smoothly. Get this excerpt and other 4 bonuses if you subscribe FREE now!

Similarly, at least one site claims that Symantec’s Norton Internet Security software identifies a common system file as a keylogger.

ReviewSaurus reports that XP SP3 causes Norton Internet Security to identify ctfmon.exe as a keylogger (a kind of malware that records your keystrokes to capture passwords and other important data).

In reality, the ctfmon.exe file in your WindowsSystem32 folder is a Microsoft system file that enables alternative input methods such as speech, tablet, or on-screen keyboard.

A spokesperson for Symantec was not immediately available for comment.

In the case of Spyware Doctor, the popular antispyware tool from PC Tools detects Trojan-Spy.Pophot.WX in RunDLL32.exe even if the system is uninfected. RunDLL32.exe is a system file that Windows uses to run code in dynamic link library (DLL) files.

The scan may also implicate other related system files, according to a report on the blog A Healthy Fear of Botulism.

By default, Spyware Doctor prevents any files it identifies as infected from running. If an important system file such as RunDLL32.exe is flagged incorrectly, the result can be disastrous for your PC. For example, users may be blocked from opening Windows Control Panel or using System Restore, among other operations.

One user who contacted us noted that blocking RunDLL32.exe created “an endless loop of scanning to remove the file, rebooting, finding the file again.”

“I’ve lost more than two days trying to fix something that was never broken,” he adds. “As far as mistakes go, this is pretty major.”

Other Spyware Doctor customers just gave up: “I had the same problem today,” reported Dave (screen name doz3r). “I got tired of fighting with it and just reinstalled the OS.”

For its part, PC Tools claims that a patch is in the works. “We are implementing a fix immediately,” wrote Super Moderator Anthony Chen on the PC Tools forum.

As of Wednesday evening, PC Tools has yet to make a fix available through the company’s Smart Update feature.

Until there’s a fix, there’s a workaround

In the case of the Norton Internet Security, ReviewSaurus advises users to ignore the false warning about ctfmon.exe.

Until a fix is available from PC Tools, Chen advises customers to add RunDLL32.exe to the global action list manually. The workaround consists of the following steps:

Step 1. In the Spyware Doctor window, click the Settings button on the left.

Step 2. Click Global Action List to the right of that.

Step 3. At the bottom of the window, click Add.

Step 4. In the New Rule dialog box, choose “File on disk” from the “Select data type” drop-down list.

Step 5. To the right of the text box below, click the … button to browse for a file. Locate and select RunDLL32.exe in the WindowsSystem32 folder.

Step 6. Make sure “Always allow” is selected in the drop-down list at the bottom and click the Add button.

Other XP SP3 compatibility problems may yet loom

This is not the first problem created by Microsoft’s latest (and last) service pack for Windows XP. Earlier this month, some HP PCs with an AMD processor experienced endless reboots after SP3 was installed.

These and other issues are documented by Windows Secrets columnist Susan Bradley’s Patch Watch column in the paid section of this week’s newsletter, as well as in her May 15 column. Bradley also provides advice on preparing for SP3 in the paid section of the May 1 issue.

If you are concerned about the effect the collection of patches that comprise XP SP3 will have on your PCs, wait a while before downloading and installing the service pack.

Check the support sites of the vendors of your most important products for news of compatibility issues with SP3. As the problems experienced by users of these anti-malware programs show, a collection of patches as large as SP3 may require some patches of its own.

Readers receive a gift certificate for a book, CD, or DVD of their choice for sending tips we print. Send us your tips via the Windows Secrets contact page.

Scott Dunn is associate editor of the Windows Secrets Newsletter. He has been a contributing editor of PC World since 1992 and currently writes for the Here’s How section of that magazine.
= Paid content

All Windows Secrets articles posted on 2008-05-22: