Achei essa fonte bem esclarecedora sobre essa questão...
Rootkits attack techniques have matured over the past few years, posing a realistic threat to commodity operating systems. Comprehensive detection of such advanced rootkits is still an open research problem. The new attack techniques used by rootkits have in turn triggered the development of novel techniques to detect their presence. The evolution of rootkits and techniques to detect them continues to be an arms race between attackers and defenders. Rootkits have evolved from manipulating user space binaries and shared libraries to modifying control and non-control data in the kernel. The latest rootkits install themselves below the operating system.
Early rootkits operate by modifying system binaries and shared libraries replacing them with trojaned versions. The goal of these trojaned binaries is to hide malicious objects or grant privileged access to malicious processes. For example, a trojaned ps binary will not list the malicious processes running on the system. A trojaned login process can give root privileges to a malicious user. To detect trojaned system binaries and shared libraries, tools such as Tripwire and AIDE were developed. These tools generate checksums of authentic binaries when run on a clean system and store them in a database. A user can examine the system at later points in time, using these tools, and compare the checksums of system binaries with those previously stored in the database. A mismatch in checksum indicates the presence of the trojaned binary. Other detection tools used an anti-virus like approach, where the presence of a rootkit is detected using a database of known signatures, such as a specific sequence of bytes in memory, or by the presence of certain files on disk. This approach does not protect the system against newer unknown rootkits. Rootkits could thwart such detectors by using polymorphic and metamorphic techniques for code obfuscation, traditionally used by viruses to escape detection from anti-virus programs.
Fonte: Advanced Operating Systems and Kernel Applications: Techniques and Technologies