Detection Mechanisms

Written by: Iron Hulk Published: May 25, 2025 Reading time: Iron Hulk
← Back to Blogs

بسم الله الرحمن الرحيم


Introduction

AV and EDR Concept

When you install an antivirus or an endpoint detection and response (EDR) on your machine, you probably don’t give much thought to what’s happening behind the scenes. Most people see a green checkmark, maybe a few scan notifications, and assume their system is safe. But underneath that simple interface lies a complex engine built to detect and stop some of the most advanced threats on the planet.

Antivirus (AV) software was originally designed to identify and remove known malicious programs by scanning files for specific signatures, unique patterns, of bytes that reveal the presence of malware. Over time, AV has evolved to include heuristic analysis and limited behavioral detection, but its focus remains largely on preventing infections through signature matching and lightweight file analysis.

Endpoint Detection and Response (EDR) picks up where traditional antivirus leaves off. Instead of just scanning files, EDR continuously monitors system behavior, tracks processes, collects telemetry data, and allows security teams to detect, investigate, and respond to threats in real time. EDRs are built for visibility and response, not just prevention.

In this post, we’re going to pull back the curtain on both AV and EDR. We’ll walk through the major categories of detection: starting with static analysis, where the software inspects files on disk for known signatures; moving into heuristic detection, which looks for suspicious traits and behaviors inside code; then into behavioral monitoring, where actions are tracked in real time as the program runs; followed by anomaly-based detection, which flags deviations from normal system or user behavior; and finally sandboxing, where suspicious files are safely detonated in isolated environments to observe their true intent. Each layer plays a critical role in how modern security tools detect and respond to threats.


1. Static/Signature-Based Detection

Static or Signature-based detection are the oldest and most straightforward technique, the AV maintains a massive database of “fingerprints” (byte sequences or hashes) that correspond to known malware. When you download a file or an incoming email attachment, the AV scans its contents and compares them against the signature database. A match means “known bad,” and the file is quarantined or deleted. This approach is very fast and has a low false-positive rate for malware that matches exactly, but it falls short when malware authors change even a few bytes or pack their payloads dynamically.

🔍 Deep Dive into the Mechanics:

Imagine you have a phonebook of criminals, each entry lists a name, a photo, and some identifying characteristics. Whenever someone new shows up, you flip through this phonebook to see if the face or name matches. Signature-based AV does something similar, except instead of mugshots, it looks for byte-patterns or cryptographic hashes (Ex: MD5, SHA-256) that uniquely identify a malicious file.

  • A specific sequence of bytes at offset 0x100 in the file, common to a particular ransomware family.
  • The hash of a library or executable that is known to contain a trojan.
  • A set of instructions or strings (such as “Decrypting payload” or HttpPost /steal.php) that consistently appear in malware samples.

When the AV vendor discovers a new piece of malware, lets say a banking trojan, they analyze it, extract several reliable byte-patterns, and push those patterns out in the next signature update. Your AV, upon receiving the update, knows to look for those patterns on disk or in memory. If it finds them, it can immediately flag the file as malicious.

⚖️ Strengths and Limitations:

  • Speed and simplicity: Signature checks are usually just a matter of comparing short patterns, which modern AV engines do with highly optimized data structures (like Bloom filters or trie-based indices).
  • Low false positives: If a byte-pattern matches exactly, it’s almost certainly the same malware family.
  • Poor against minor modifications: Malware authors can recompile code, insert junk instructions, or pack/encrypt their payload so that the byte-pattern changes just enough to evade detection.
  • Reactive by nature: Signature databases are always cat and mouse game. Only after a threat is discovered, analyzed, and signatures extracted can the AV detect it. This leaves a window of “zero-day vulnerability” before the update arrives.

2. Heuristic-Based Detection

To catch threats that don’t yet have a signature, AV vendors introduced heuristics, rules that look for suspicious characteristics inside a file, rather than an exact byte match. These heuristics often involve scanning for “red flags” like encoded or obfuscated code, calls to low-level system functions that typically appear in malware (Ex: VirtualAlloc, CreateRemoteThread), or even suspicious combinations of API calls. Rather than saying “this file is exactly the same as TrojanXYZ,” a heuristic-based engine might say “this file behaves a lot like ransomware because it encrypts files and then writes a ransom note.”

🧩 Deep Dive into the Mechanics:

Think of a heuristic engine like a detective who knows that safe-crackers often carry lock-picking tools, wear gloves, and loiter around banks late at night. Even if the detective has never seen this particular individual before, seeing those telltale clues can raise suspicion.

In AV terms, heuristics might include:

  • Unpacking routines: Many malware samples pack their real code inside multiple layers of encryption or compression. The heuristic engine might look for unusual patterns of “self-modifying” code, for instance, a loop that decrypts a buffer and then jumps into it.
  • Suspicious API sequences: If a file calls CreateProcess and then immediately injects code into another process (via WriteProcessMemory and CreateRemoteThread), that’s a strong heuristic indicator of process injection, a common malware technique.
  • Embedded shellcode: A heuristic might scan the file for a sequence of bytes that starts with typical shellcode instructions (like a PUSH/POP combination or a short JMP to relative address), even if it’s encrypted.
  • Abnormal PE characteristics: Executables usually follow a standard Portable Executable (PE) format, correct section sizes, valid import tables, and so on. If an AV sees an EXE with an oversized .text section full of zeroes, or one with an invalid header, it may flag it as malicious.
  • Heuristic scoring: Often, each suspicious attribute adds points to a cumulative “malware score.” If the sum of scores crosses a threshold, the engine declares the file malicious or “potentially malicious” and quarantines it for further analysis.
  • Entropy checks: Some heuristic engines measure the entropy of file sections. Extremely high entropy may signal encrypted or packed content, often used to hide malicious payloads.

⚖️ Strengths and Limitations:

  • Spots new versions of malware: By focusing on behavior indicators and code structures, heuristics can detect malware families that have changed just enough to evade signatures.
  • Higher false-positive risk: Because heuristics are based on general behaviors (Ex: “calls VirtualAlloc then writes to memory”), benign software that packs itself (like some legitimate installers) or debuggers that perform in-memory patching might trip the heuristics.
  • Tunable thresholds: AV vendors calibrate heuristic thresholds carefully, if they’re too strict, many harmless programs get flagged; if too lenient, malware slips through.
  • Requires expert tuning: The folks designing these rules need deep knowledge of Windows internals, common malware patterns, and the behaviors of legitimate software, so they don’t accidentally trash useful applications.

3. Behavior-Based and Behavioral Monitoring

While heuristic detection often examines a file before it runs—looking for suspicious patterns or traits behavior-based detection waits until the program actually executes, observing what it does in real time. The AV/EDR hooks into various operating system functions such as kernel callbacks, user-mode API interceptions, or event tracing to observe actions like file creation, registry modifications, process injection, or network connections. If a process starts encrypting every “.docx” file it finds (classic ransomware behavior) or injects code into system processes, the EDR flags and quarantines it.

🧠 Deep Dive into the Mechanics:

Picture a security guard watching CCTV footage in real-time. They don’t care if the intruder is wearing a disguise (packed or obfuscated code), because the guard can see that the person is breaking into the server room (Ex: encrypting user data).

Behavioral detection mechanisms include:

  • OS-Level Hooks:
    1. Kernel-mode callbacks: On Windows, functions such as PsSetCreateProcessNotifyRoutine, PsSetCreateThreadNotifyRoutine, and PsSetLoadImageNotifyRoutine let a security driver receive events when a process starts, a thread spawns, or an image loads. An EDR running in kernel mode can kill or quarantine the offender immediately.
    2. User-mode API hooking: The agent injects a small DLL into every process and patches the prologue of APIs like OpenProcess, VirtualAlloc, WriteProcessMemory, or uses IAT rewriting. The hook logs or blocks the call if parameters look malicious.
  • Real-Time File and Registry Monitoring:
    1. File system filter drivers: An EDR can install a kernel-mode filter that sees every call to CreateFile, WriteFile, or DeleteFile. If a process starts modifying thousands of Office documents within 30 seconds, the system recognizes a sudden mass-encryption pattern and can halt the process.
    2. Registry callbacks: Using kernel functions like CmRegisterCallbackEx an EDR can monitor registry operations. If a process writes to HKLM\Software\Microsoft\Windows\CurrentVersion\Run, attempting to persist on reboot, it might be trying to create a malicious autorun entry. The EDR can stop or flag that behavior.

Beyond local monitoring, advanced EDRs correlate telemetry across multiple hosts—detecting patterns like simultaneous PowerShell execution across several endpoints, which could indicate lateral movement or coordinated attacks.

⚖️ Strengths and Limitations:

  • Accuracy: Watching real behavior reduces false positives, if a user is legitimately editing a document, that’s far different from a hidden service encrypting dozens of files.
  • Late-stage detection: Because the AV/EDR only flags things when code actually runs, there is still a small window where the malware can begin its damage (Ex: delete backups, exfiltrate data) before being stopped.
  • Performance overhead: Kernel callbacks, file filters, and API hooks add latency to every system call. Good EDR vendors optimize heavily, but if you install two memory scanners or multiple hooking products, you can see noticeable slowdowns.
  • Complexity of evasion: Advanced attackers sometimes use “living off the land” techniques, utilizing legitimate system utilities like PowerShell or WMI to do malicious work, thereby blending into normal behavior. Detecting these threats requires more sophisticated logic such as “Anomaly-Based Detection” and “Machine Learning”.
  • Delayed/staged execution evasion: Some malware executes in multiple stages with long delays or waits for specific conditions, often slipping past real-time behavior monitoring unless extended observation is used.

4. Anomaly-Based Detection (Statistical and Behavioral Analytics)

Rather than relying on pre-defined rules, anomaly-based detection profiles “normal” behavior for endpoints and then looks for deviations. For example, if a user’s workstation usually uses 128 MB of network bandwidth per hour but suddenly starts transmitting 2 GB while contacting an uncommon high-numbered port, the EDR raises a red flag. Likewise, if a process that normally reads its data from “C:\Program Files\MyApp” suddenly tries to write into “C:\Windows\System32,” that’s anomalous. Anomaly-based detection uses statistical baselines, machine learning models, and clustering algorithms to spot outliers.

🧠 Deep Dive into the Mechanics:

Think about your home security system, it knows when you’re usually at home, when you generally unplug appliances, or when your smart lock logs typical unlock times. If something happens outside that profile (let’s say a door unlocks at 3 a.m. when you’re away) it triggers an alert.

Anomaly Detection Examples:

  • User & Entity Behavior Analytics (UEBA):
    1. Baseline creation: The EDR agent spends hours learning how a particular user’s machine behaves: what applications are typically launched, how much CPU is used during normal business hours, which servers the machine contacts.
    2. Statistical models: Using regression or clustering, the system learns that 90% of the time, Chrome.exe makes HTTP connections to “cdn.jsdelivr.net” and “fonts.googleapis.com.” Suddenly, if Chrome.exe tries SSL connections to an unknown IP in Russia, that deviation flags an alert.
    3. Adaptive learning: When a user legitimately starts using a new cloud-based app, the EDR model gradually incorporates that behavior into the baseline, avoiding repetitive false positives.
  • Process & Thread Behavior:
    1. Unexpected child processes: A common technique for some malware is to spawn “cmd.exe /c regsvr32 /s malicious.dll ” If Word.exe (winword.exe) spawns cmd.exe, that’s unusual behavior. Word normally doesn’t launch a shell. The anomaly engine sees that “winword.exe → cmd.exe” spawning chain as a red flag.
    2. Code injection patterns: If a non-privileged process calls WriteProcessMemory from Notepad.exe into explorer.exe, that’s statistically rare. A model trained on millions of normal interactions quickly flags it.
    3. DLL load anomalies: If a system process like svchost.exe loads a suspicious DLL from “C:\Users\Public\somefolder\evil.dll” instead of “C:\Windows\System32,” the anomaly engine sees it as a major deviation.
    4. Process hollowing anomalies: If a trusted process (Ex: svchost.exe) is started suspended, then hollowed and injected with malicious code, the behavioral baseline may catch the divergence from normal startup behavior."

⚖️ Strengths and Limitations:

  • Adaptive and future-proof: Instead of requiring constant rule updates, anomaly engines learn the unique “fingerprint” of a network or device. New malware that behaves differently from any known benign process can get flagged immediately.
  • Complex to tune: Building a reliable baseline across a diverse environment is challenging. Some departments legitimately run memory-intensive applications or have high network usage; others don’t. A one-size-fits-all anomaly threshold results in noise.
  • Explainability challenges: Security teams need to know why a particular process was flagged. Some machine learning models are “black boxes,” making it hard to justify why exactly a certain behavior is suspicious. Newer research in explainable AI (XAI) is helping, but it’s still an area in flux.

5. Sandboxing and Dynamic Analysis

Sandboxing means running a suspicious file inside a controlled, isolated environment, often a virtual machine or a micro-Virtual Machine, where its behavior is observed over a short period (usually a few seconds to minutes). Modern AV/EDR vendors often send suspicious executables to cloud-based sandboxes, where Linux or Windows VMs execute the file and log everything: files created, registry keys touched, network connections attempted, whether it dropped a second-stage payload, and so on. If the sample launches a known banking trojan or ransomware, the sandbox engine records it and sends back a verdict. The endpoint agent then quarantines or blocks similar samples on other machines.

🧬 Deep Dive into the Mechanics:

Imagine dropping an unknown chemical into a lab’s test tube to see if it bubbles, emits heat, or changes color.

Sandbox Techniques:

  • Lightweight VMs (Hypervisor-based):
    1. Full system emulation: The sandbox spins up a Windows 10 VM with snapshotting enabled. It clones the base snapshot, runs the executable, and watches kernel events. Everything from file system changes to DirectX calls can be logged. After the analysis period, the VM reverts to a clean snapshot.
    2. Bare-metal or hardware-assisted introspection: Some sandboxes run on specialized hardware or use virtualization extensions (Intel VT-x or AMD-V) to inspect the guest’s kernel structures without installing agents inside the VM. By watching the guest’s page tables or hooking into VMware’s hypervisor APIs, the sandbox sees exactly which code pages execute, which regions end up being de-obfuscated, and what network traffic flows out.
  • Behavior Logging & Scoring:
    1. File-system exploration: Does the malware create a ransom note in every user’s Documents folder? Does it drop a malicious DLL into C:\Windows\System32?
    2. Network communication: Does it try to connect to a known command-and-control domain? Does it launch an HTTPS session on port 443 but craft its own TLS handshake (indicating a custom C2 protocol)?
    3. Registry tampering: Does it add keys under HKCU\Software\Microsoft\Windows\CurrentVersion\RunOnce or modify AutoPlay handlers to persist on USB insertion?
    4. Process spawning: Does it spawn Manage-bde.exe (BitLocker management utility) repeatedly to disable encryption? Does it run “vssadmin delete shadows” to erase Volume Shadow Copies?

Once the sandbox finishes its “run,” it aggregates these findings into a verdict. If the sample is confirmed malicious, the AV vendor automatically extracts new signatures, YARA rules, or behavioral indicators and pushes them out to endpoints worldwide.

⚖️ Strengths and Limitations:

  • Catch strongly obfuscated threats: Many packers and crypters can evade simple signature or heuristic checks. In a sandbox, the packer unpacks itself and reveals the malicious code.
  • Scalability challenges: Running thousands of VMs or CPU emulations demands massive infrastructure. Cloud-based sandboxes try to parallelize widely, but a very complex malware might delay its malicious logic for hours or check for virtualization artifacts (Ex: unusual device names).
  • Evasion by anti-sandbox techniques: Malware may detect that it’s inside a VM (by checking for VMware tools processes, looking at CPU timing discrepancies, or trying to communicate with a nonexistent router). If detected, it might abort execution or behave benignly to slip through the analysis phase.

5. Summary: Layered Defense and the Role of Each Detection Method

Modern AV and EDR solutions don’t rely on just one detection method, they layer them together to form a defense-in-depth approach. Each layer catches what the others might miss. Signature-based detection is fast and reliable for known threats. Heuristics add flexibility by identifying code that “looks” suspicious. Behavioral monitoring tracks what programs do in real time, while anomaly detection provides adaptive insight based on past activity. Sandboxing isolates execution for safe analysis, revealing behaviors that can’t be seen through static inspection. No single method is perfect, but together they create a much more resilient security posture.

Signature-Based

Strength: Fast, low false positives

Weakness:
- Misses unknown or obfuscated malware
- Ineffective against new or obfuscated malware

Use: Detecting known malware variants

Heuristic-Based

Strength: Catches slightly altered threats

Weakness:
- False positives if too aggressive
- Can cause false positives if overly broad

Use: Detecting polymorphic or packed malware

Behavioral Monitoring

Strength: Real-time execution insights

Weakness:
- May detect too late
- Resource intensive

Use: Stopping ransomware or process injection

Anomaly Detection

Strength: Adaptive to new threats

Weakness:
- Hard to explain results
- Training time needed

Use: Detecting insider threats, rare behaviors

Sandboxing

Strength: Observes real behavior in safe space

Weakness:
- Resource-heavy
- Can be evaded by sandbox-aware malware

Use: Validating suspicious files or scripts