Auditd is a user-space component of Linux auditing subsystem that collects and logs messages sent by the kernel. This tool can be used to collect critical security information so you can ensure you know what is happening on your systems. Auditd does not provide any form of additional security, however it does provide insights and audit trails regarding what is done on a server. Not only is this information useful in troubleshooting, it also allows certification of many compliance guides.
When auditd is running, logs are typically sent to /var/log/audit/audit.log. By default, the only messages logged will be related to authentication, authorization, selinux, and sudo. However, administrators can configure additional rules to audit any form of file operations, system calls, commands ran by users, and security events.
While auditd is the service that makes all the magic happen, auditctl is the command administrators can use to manage audit rules.
# service auditd status
auditd (pid 400918) is running...
# auditctl --help
usage: auditctl [options]
-a <l,a> Append rule to end of <l>ist with <a>ction
-A <l,a> Add rule at beginning of <l>ist with <a>ction
-b <backlog> Set max number of outstanding audit buffers
allowed Default=64
-c Continue through errors in rules
-C f=f Compare collected fields if available:
Field name, operator(=,!=), field name
-d <l,a> Delete rule from <l>ist with <a>ction
l=task,exit,user,exclude
a=never,always
-D Delete all rules and watches
-e [0..2] Set enabled flag
-f [0..2] Set failure flag
0=silent 1=printk 2=panic
-F f=v Build rule: field name, operator(=,!=,<,>,<=,
>=,&,&=) value
-h Help
-i Ignore errors when reading rules from file
-k <key> Set filter key on audit rule
-l List rules
-m text Send a user-space message
-p [r|w|x|a] Set permissions filter on watch
r=read, w=write, x=execute, a=attribute
-q <mount,subtree> make subtree part of mount point's dir watches
-r <rate> Set limit in messages/sec (0=none)
-R <file> read rules from file
-s Report status
-S syscall Build rule: syscall name or number
-t Trim directory watches
-v Version
-w <path> Insert watch at <path>
-W <path> Remove watch at <path>
Auditctl works in a similar fashion to iptables rules, where new rules are added to the bottom of the chain, unless otherwise specified in the command. So you will need to ensure that previous rules do not take precedence over new rules. For example if you have a rule to monitor all modifications to /etc, a new rule to monitor writes /etc/sysconfig will need to be placed above the /etc rule.
Persistent audit rules are stored in /etc/audit/rules.d/audit.rules. Just like iptables, when you add an audit rule via auditctl, it is not persistent. It must be written to configure to survive a reboot. Adding an audit rule to audit.rules is the same syntax as running the auditctl command without auditctl iteself. For example, if your rule is:
auditctl -w /etc/passwd -p wa -k user-edit
Making it persistent is as easy as adding the following to the bottom of /etc/audit/rules.d/audit.rules
-w /etc/passwd -p wa -k user-edit
So lets take a closer look at the previous example. We are using the -w flag to add the most common audit rule, a watch. Watch rules can be set on file and directories and trigger an audit log when a certain action is performed, such as a read, write, attribute change, or execution. We can add a watch rule to any file or directory.
The -p flag is specifying which action, or filter, we want to log. When we run “auditctl –help” we can see the options: r=read, w=write, x=execute, a=attribute. So in our example, we want auditd to log whenever an action writes or an attribute changes on /etc/passed.
When a number of audit rules are created, you will find that the audit log can become pretty bloated with useful information. So we want to ensure that we can make heads or tails of that information, which we ensure with the -k flag. -k adds a ‘filter key’ to the log. This filter key is whatever we want it to be so we can later search for events based on this key. We set this example rule to ‘user-edit’ to we can later search the audit logs for all ‘user-edit’ events.
With this audit rule in place, we can see our message log when a new user is created.
# adduser testuser
# tail -f /var/log/audit/audit.log | grep "user-edit"
type=SYSCALL msg=audit(1502003073.746:4875440): arch=c000003e syscall=2 success=yes exit=5 a0=7f97cfb44d00 a1=20902 a2=0 a3=0 items=1 ppid=657991 pid=659388 auid=502 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=1602311 comm="adduser" exe="/usr/sbin/useradd" key="user-edit"
type=CONFIG_CHANGE msg=audit(1502003073.859:4875443): auid=502 ses=1602311 op="updated_rules" path="/etc/passwd" key="user-edit" list=4 res=1
type=SYSCALL msg=audit(1502003073.859:4875444): arch=c000003e syscall=82 success=yes exit=0 a0=7ffe7ef1ebd0 a1=7f97cfb44d00 a2=7ffe7ef1eaa0 a3=0 items=5 ppid=657991 pid=659388 auid=502 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=1602311 comm="adduser" exe="/usr/sbin/useradd" key="user-edit"
Now, the information logged is a bit intense, but we can use ausearch make sense of it. ausearch is a command that allows us to query audit daemon logs and filter our query to certain events. There are many available filters to search by; toss –help to ausearch to see a full list
# ausearch --help
usage: ausearch [options]
-a,--event <Audit event id> search based on audit event id
--arch <CPU> search based on the CPU architecture
-c,--comm <Comm name> search based on command line name
--checkpoint <checkpoint file> search from last complete event
--debug Write malformed events that are skipped to stderr
-e,--exit <Exit code or errno> search based on syscall exit code
-f,--file <File name> search based on file name
-ga,--gid-all <all Group id> search based on All group ids
-ge,--gid-effective <effective Group id> search based on Effective
group id
-gi,--gid <Group Id> search based on group id
-h,--help help
-hn,--host <Host Name> search based on remote host name
-i,--interpret Interpret results to be human readable
-if,--input <Input File name> use this file instead of current logs
--input-logs Use the logs even if stdin is a pipe
--just-one Emit just one event
-k,--key <key string> search based on key field
-l, --line-buffered Flush output on every line
-m,--message <Message type> search based on message type
-n,--node <Node name> search based on machine's name
-o,--object <SE Linux Object context> search based on context of object
-p,--pid <Process id> search based on process id
-pp,--ppid <Parent Process id> search based on parent process id
-r,--raw output is completely unformatted
-sc,--syscall <SysCall name> search based on syscall name or number
-se,--context <SE Linux context> search based on either subject or
object
--session <login session id> search based on login session id
-su,--subject <SE Linux context> search based on context of the Subject
-sv,--success <Success Value> search based on syscall or event
success value
-te,--end [end date] [end time] ending date & time for search
-ts,--start [start date] [start time] starting data & time for search
-tm,--terminal <TerMinal> search based on terminal
-ua,--uid-all <all User id> search based on All user id's
-ue,--uid-effective <effective User id> search based on Effective
user id
-ui,--uid <User Id> search based on user id
-ul,--loginuid <login id> search based on the User's Login id
-uu,--uuid <guest UUID> search for events related to the virtual
machine with the given UUID.
-v,--version version
-vm,--vm-name <guest name> search for events related to the virtual
machine with the name.
-w,--word string matches are whole word
-x,--executable <executable name> search based on executable name
That is quite the list, so lets just look at some important filters you might use.
-a,--event <Audit event id> search based on audit event id
-i,--interpret Interpret results to be human readable
-k,--key <key string> search based on key field
-r,--raw output is completely unformatted
-f,--file <File name> search based on file name
-ts,--start [start date] [start time] starting data & time for search
Since we set the key to “user-edit”, we can search for just that one key. I also like human readable results, so lets add the -i
# ausearch -i -k user-edit
----
type=PATH msg=audit(08/06/2017 02:16:40.724:4875833) : item=0 name=/etc/passwd inode=1178864 dev=08:02 mode=file,644 ouid=root ogid=root rdev=00:00 nametype=NORMAL
type=CWD msg=audit(08/06/2017 02:16:40.724:4875833) : cwd=/root
type=SYSCALL msg=audit(08/06/2017 02:16:40.724:4875833) : arch=x86_64 syscall=open success=yes exit=5 a0=0x7fb02eaf8d00 a1=O_RDWR|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW a2=0x0 a3=0x0 items=1 ppid=657991 pid=670512 auid=malasuk uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts1 ses=1602311 comm=adduser exe=/usr/sbin/useradd key=user-edit
----
type=CONFIG_CHANGE msg=audit(08/06/2017 02:16:40.888:4875836) : auid=malasuk ses=1602311 op="updated_rules" path=/etc/passwd key=user-edit list=exit res=yes
----
type=PATH msg=audit(08/06/2017 02:16:40.888:4875837) : item=4 name=/etc/passwd inode=1178862 dev=08:02 mode=file,644 ouid=root ogid=root rdev=00:00 nametype=CREATE
type=PATH msg=audit(08/06/2017 02:16:40.888:4875837) : item=3 name=/etc/passwd inode=1178864 dev=08:02 mode=file,644 ouid=root ogid=root rdev=00:00 nametype=DELETE
type=PATH msg=audit(08/06/2017 02:16:40.888:4875837) : item=2 name=/etc/passwd+ inode=1178862 dev=08:02 mode=file,644 ouid=root ogid=root rdev=00:00 nametype=DELETE
type=PATH msg=audit(08/06/2017 02:16:40.888:4875837) : item=1 name=/etc/ inode=1177345 dev=08:02 mode=dir,755 ouid=root ogid=root rdev=00:00 nametype=PARENT
type=PATH msg=audit(08/06/2017 02:16:40.888:4875837) : item=0 name=/etc/ inode=1177345 dev=08:02 mode=dir,755 ouid=root ogid=root rdev=00:00 nametype=PARENT
type=CWD msg=audit(08/06/2017 02:16:40.888:4875837) : cwd=/root
type=SYSCALL msg=audit(08/06/2017 02:16:40.888:4875837) : arch=x86_64 syscall=rename success=yes exit=0 a0=0x7ffe19102210 a1=0x7fb02eaf8d00 a2=0x7ffe191020e0 a3=0x0 items=5 ppid=657991 pid=670512 auid=malasuk uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts1 ses=1602311 comm=adduser exe=/usr/sbin/useradd key=user-edit
Now we have a tremendous about of information on what happened. We can see that the epoch timestamp is now readable, and we know that the malasuk user, escalated as root, ran /usr/sbin/useradd and added a new user. The logs are quite verbose as they reflect the system calls that the Linux kernel is making to modify /etc/passwd.
Great, so with that one audit rule, we can determine who, when, and how someone added a user. What else can we do?
Maybe we want to watch everything in the /etc/httpd directory so when any user reads, writes, or executes anything in the Apache configuration, we get a log. We can add a new audit rule to do just that
# auditctl -w /etc/httpd/ -p rwa -k httpd-access
As a logical system administrator, I don’t trust any of my user’s and I want to log all their commands in the event they decide to delete their bash history.
auditctl -w /bin -p x -k command-ran
Then to make these audit rules are persistent across reboots, ensure they are added to the audit.rules configuration file.
# Log user edits
-w /etc/passwd -p wa -k user-edit
# log Apache configuration access
-w /etc/httpd/ -p rwa -k httpd-access
# log commands ran
-w /bin -p x -k command-ran
In conclusion, auditing is not just about maintaining compliance and paper trails, it can prove invaluable in troubleshooting and root cause analysis. Having proper audit rules in place could save you quite some time investigating the next compromised server where the user thought they covered their trail.