May 2002

Special Report

Combating Nonviral Malware

Trojans, sniffers and spyware, oh my!

BY Jay Heiser

Antivirus software is a commodity. Almost every company runs AV scanners on client machines, and most have AV at the gateway, too. The assumption is that AV protects us from malicious code, and, by and large, it does. As long as the scanning engine recognizes the viral code, it will stop it before it has a chance to infect your systems.

There's a huge category of unwanted code, however, against which AV software is powerless: nonreproducing malware. Electronic burglar tools like password crackers, traffic sniffers, keystroke loggers, data scroungers and remote access Trojans (RATs) are being used by attackers both inside and outside the organization to capture passwords, spy on network traffic, record private communications, and stealthily receive and transmit unauthorized commands to and from remote hosts. End users and even IT staff download the latest P2P or remote access program--not necessarily a hostile act, but one that nevertheless opens up unauthorized holes in the corporate firewall.

Overburdened IT security departments may consider these threats insignificant compared to other priorities. But the problem of nonviral malware is growing, and defending against it is a nontrivial task.

Nature of the Beast

The formal definition of malware is "malicious software." The only safe assumption is to treat all unwanted code as "malicious." While viruses and worms are the most visible forms of malware, "unwanted code" describes a broad range of software that potentially violates an organization's security policy.

Spyware. Surprisingly, undesirable code often arrives with commercial software distributions. The term spyware refers to software that makes unauthorized use of a system's Internet connection to communicate with its developer. Unless you've installed a host-based firewall that reports network activity, you may not be aware that many common personal productivity programs, such as Microsoft Money, communicate with the vendor at regular intervals. While most spyware is innocuous--particularly when it's part of a commercial software package--some freeware programs contain spyware that scans systems for proprietary data and communicates with unauthorized remote hosts.

RATs. As the name implies, remote access Trojans are programs that allow unauthorized network access once installed on a victim computer. Programs such as NetBus, Back Orifice and SubSeven often come disguised as some other program, such as an e-mail executable or Web site download, but they may also be manually installed on a PC by someone with physical or network access.

Zombies. Not all malware is directed inward--many Internet sites have been attacked by malware that was resident on other hosts. Denial-of-service programs called zombies can be simultaneously installed on multiple hosts through automated attack scripts that exploit vulnerabilities in network services. These hosts are effectively "secondary" victims that are marshaled together to attack one or more targets (the "primary" victims).

Crackers and sniffers. The degree to which utilities such as password crackers, network sniffers and vulnerability scanners constitute "malware" is a matter of perspective. Many IT security departments employ these tools to identify holes and weaknesses in their networks and systems. However, such tools are also used illicitly by corporate insiders to spy on their colleagues or gain access to unauthorized resources. Unlike RATs, these tools aren't "network aware," meaning they don't run autonomously and transmit their findings back to a remote server.

Keystroke loggers also walk the fine line between security administration and security breach. As with scanners and sniffers, the question comes down to intent. Some companies use commercial loggers, such as PC Activity Monitor from Raytown Corp. (http://www.keyloggers.com/) to monitor employees suspected of resource misuse. While most employees might feel this invades their privacy, such monitoring can be useful for capturing corporate spies. Law enforcement agencies may also use keystroke loggers to gather evidence on suspected criminals, as in the recent FBI investigation of Nicodemo S. Scarfo. On the other hand, staffers may download freeware keystroke loggers from hacker warez sites to turn the tables and illicitly spy on their boss or coworkers.

P2P applications. An ever-increasing number of Web-based peer-to-peer applications (such as Napster, AIM or Groove) and remote access tools (such as Gotomypc) tunnel through corporate firewalls via HTTP and other open ports, effectively allowing employees to create their own ad hoc VPNs. Gotomypc is especially risky because it initiates the connection from an internal PC out to the Gotomypc host. The subscriber can then access the office PC remotely by connecting to Gotomypc, which arranges a link with the incoming office PC connection.

Logic bombs. One form of malware that's often overlooked is the logic bomb, a program whose only purpose is to destroy data and applications. Typically planted by disgruntled employees, logic bombs and time bombs can cause massive damage to networks and systems. In 1996, a network administrator named Timothy Lloyd set off a software bomb at his former employer, Omega Engineering, causing an estimated $10 million in damages. Lloyd was recently sentenced to 41 months in prison for the crime.

Malware Rx

The reason commercial AV products don't scan for nonreproducing malware is simple, says Steve Trilling, senior director of research at Symantec 's Security Response Center (http://www.symantec.com/). Customers aren't demanding it.
"Every customer must legitimately agree that the utilities identified by AV software are ones that they want to be warned about," Trilling says. The loose definition of what is considered "hostile" in this environment precludes AV vendors from flagging malware in every situation. While most AV products scan for well-known Trojans such as NetBus and SubSeven, Trilling says most customers don't want their AV vendor to make "arbitrary" decisions on whether most nonviral code should be considered hostile.

Specific code that has already been discovered in the wild and judged as hostile can be recognized by distinctive internal patterns of bits. By and large, this is how AV software works, using what is commonly referred to as signature scanning. Given the fact that AV vendors don't include signatures for most nonviral code--and that marketing, technical, and philosophical issues discourage it--security managers will have to adopt other detection and mitigation techniques.

Specialty signature scanners. A handful of security vendors offer software with signature databases specifically aimed at Trojans, sniffers, keyboard loggers and other nonreproducing code. PestPatrol (http://www.pestpatrol.com/ ) offers a software product, also called PestPatrol, that comes with a database of more than 32,000 signatures, updated biweekly. Like AV software, PestPatrol can perform on-demand and on-access scanning, enabling users to ignore, quarantine or delete detected "pests." Unlike AV scanners, PestPatrol removes unwanted programs even if they are already running, according to CTO David Stang.

PestPatrol is available in both consumer and corporate editions. The consumer version (designed to run directly on host PCs and laptops) is available as a stand-alone product. The corporate edition, released in March, is a server-based tool with host agents. The product allows users to schedule scans, store logs centrally and configure e-mail alerts for designated admins. Encouraging its use in parallel with traditional AV products to stop all forms of undesirable code at the perimeter, the corporate edition integrates with Check Point Software Technologies' SVC (http://www.checkpoint.com/), and it can easily be configured to work with Clearswift 's Mailsweeper http://www.clearswift.com/).

Heuristics. Some nonviral malware can be detected using heuristics, a psychological term that means "rule of thumb." In the context of IT security, heuristics refers to the ability to infer that a particular binary might be hostile based on typical sequences of operation within the object code.

AV software can sometimes detect previously unknown hostile code by using heuristic techniques to identify reproductive capabilities. Likewise, Trojan-terminating utilities like PestPatrol and Raytown's Anti-keylogger (http://www.anti-keylogger.com/) can smell out the distinctive characteristics of keyboard loggers based on heuristics. On the downside, the Raytown product is limited only to keystroke attacks and won't protect against the myriad other forms of hostile code.

Several solutions from Finjan Software (http://www.finjan.com/ ) also use heuristic techniques to recognize previously unknown or disguised Trojans. Finjan's SurfinGate and SurfinShield products, in addition to controlling hostile Java and ActiveX, have behavior-detection capabilities that run executables in a sandbox, evaluating them for hostile intent. The "sandbox" approach is important because attackers are increasingly using a variety of "packing" tools that restructure binary object files without affecting their ability to run. A program that has been manipulated this way has a different signature, and won't be detected by a signature-based detector unless its definition file has been updated to include that specific binary packed in the same way.

SurfinGate and SurfinShield are available in a bundle with McAfee Security's VirusScan (http://www.mcafeeb2b.com/), thereby offering "complete protection from both known and unknown attacks," according to Finjan. However, the scanner only examines software being downloaded or received from the Internet, and won't catch Trojans that are manually placed on a system by insiders or uploaded by external attackers.

Behavior blockers are a class of controls that prevent hostile operations from executing on a host, such as a command that attempts to write to the boot sector. A number of vendors offer "trusted OS" and "intrusion prevention" tools that harden host systems, controlling a program's ability to access file and network resources (see "Different Approaches, Same Goal" ). Any system request not explicitly permitted by design or policy is default denied.

Personal firewalls, such as BlackICE PC Protection from Internet Security Systems (http://www.iss.net/), Norton Personal Firewall from Symantec, and Zone Lab's ZoneAlarm (http://www.zonelabs.com/), which is now available in a bundle with PestPatrol, can block most unwanted spyware, Trojans and P2P apps by denying inbound or outbound network connections. Because they run directly on a PC, personal firewalls have an advantage over network firewalls in that they control which specific executables can initiate or receive network connections. This is a very precise form of behavior blocking that can prevent spyware from contacting its spymaster and remote-control servers from responding to remote connections. However, while personal firewalls are common for home users directly connected to the Internet, they are not normally used inside the corporate network. Also, at least one Trojan has demonstrated an ability to circumvent ZoneAlarm by invoking Internet Explorer and using it as a sort of covert channel.

RAT traps. If your primary concern is remote access Trojans, two utilities might be worth investigating: LockDown Corp.'s LockDown Millennium Pro (http://www.lockdowncorp.com/) and Diamond Computer Systems' TDS-3 (http://www.diamondcs.com.au/). LockDown takes a multilayered approach of signature scanning and monitoring for programs attempting to access the Internet. Diamond, an Australian company, also offers a generic detection capability in its TDS-3 product that uses heuristics to evaluate executables for "Trojaneous qualities." It will identify known Trojans by name, even if they have been packed or otherwise manipulated. The product also checks for changes to system startup files, the areas most likely to contain evidence of the insertion of unauthorized code.

If you're mainly worried about remote-access attacks, these solutions provide a comprehensive approach in a single package. However, they don't protect against nonnetworked malware like password crackers or sniffers.

Integrity checkers. If you can't prevent hostile code from being installed, or executed, your last-ditch defense is to examine the system for changes. Tripwire (http://www.tripwire.com/) is one of the few malware controls available for both Windows and Unix. Integrity testers create a baseline record of the files on a system, against which later scans will be compared to determine changes. Even one-off attacks, such as time bombs, will be recognized. ISS's BlackICE has also recently added system baselining.

Mixed Results

The plethora of point solutions in the malware-defense space demonstrates that there's no magic wand for protecting against all forms of unwanted code. Because a defense-in-depth approach is the only way to control malware, expect to see continued
announcements of multiproduct bundling, market consolidation and acquisitions, and new types of controls being added to existing products.

The good news is that basic system hygiene is always helpful. A healthy system is naturally more resistant to all forms of attack. Extra tight configuration management will prevent or discourage users from installing code. Perhaps most importantly--especially when protecting workstations--make full use of your "biological" countermeasures. Teach your users well, so they avoid risky activities and know to call the help desk when unusual events happen.

---

JAY HEISER, CISSP (jheiser@infosecuritymag.com), works for a large European bank in London. His most recent book is Computer Forensics: Incident Response Essentials (Addison-Wesley, 2001).

May 2002 Table of Contents