What's New in ModSecurity
Real-Time Audit Log Aggregation
In the new version, ModSecurity introduces an entirely new audit log format, one that is suitable for real-time log aggregation. There are two cases when you will need real-time aggregation:
- You run more than one web server and want to deploy ModSecurity at each one.
- You have only one web server, but you want to do something meaningful (or simply more useful) with the information from the audit logs (for example, store them into a database for a future reference).
The old audit log format is a good choice if you need to occasionally look at the logs, but it suffers from two problems. First, it uses one file to store multiple entries. To make the one-file approach work properly, the web server needs to use a server-wide lock to serialize access to the file. The second problem is that the file format was making the log difficult to parse. The new log format (also called the concurrent audit log format) uses one file per audit log entry. It also uses a modular format that is much easier to parse and can be extended over time. Here is a configuration example:
SecAuditEngine RelevantOnly
SecAuditLogType Concurrent
SecAuditLog /var/www/audit_log/index
SecAuditLogStorageDir /var/www/audit_log/
The index file will contain one line per audit log entry:
192.168.2.101 192.168.2.11 - - [15/Jul/2005:11:56:52 +0100]
"POST /form.php HTTP/1.1" 403 3 "http://192.168.2.101:8080/form.php"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.8) Gecko/20050511
Firefox/1.0.4" G3yTd38AAAEAAAM7BLwAAAAA "-"
/20050715/20050715-1156/20050715-115652-G3yTd38AAAEAAAM7BLwAAAAA 0 1031
md5:dc910f6d647d47b32ae6e47326f0ca42
You will recognize most of the index entries. Each line begins with a combined log format, but it then adds the following fields:
- The
UNIQUE_ID (generated by mod_unique_id)./li>
- A session ID (reserved for future expansion)./li>
- The name of the file containing the audit log entry, relative to the root of the repository./li>
- The offset of the file where the audit log entry begins (to allow the repacking of multiple audit log entries into a single file)./li>
- The total length of the audit log entry./li>
- The MD5 hash of the complete audit log entry./li>
Each audit log entry will look like this:
--67458b6b-A--
[15/Jul/2005:11:56:52 +0100] G3yTd38AAAEAAAM7BLwAAAAA
192.168.2.11 4236 192.168.2.101 8080
--67458b6b-B--
POST /form.php HTTP/1.1
Host: 192.168.2.101:8080
User-Agent: Mozilla/5.0
Accept: */*
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://192.168.2.101:8080/form.php
Content-Type: application/x-www-form-urlencoded
Content-Length: 5
--67458b6b-C--
f=111
--67458b6b-E--
403 (Response body)
--67458b6b-F--
HTTP/1.1 403 Forbidden
Last-Modified: Fri, 08 Jul 2005 14:25:30 GMT
ETag: "decb4-3-34b96a80"
Accept-Ranges: bytes
Content-Length: 19
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html
--67458b6b-H--
Message: Pattern match "111" at POST_PAYLOAD
[id "1"] [rev "2"] [msg "3"] [severity "4"]
Apache-Handler: application/x-httpd-php
Stopwatch: 1126536042708000 11024 (7276* 7375 9842)
--67458b6b-Z--
You can see how each audit log entry consists of multiple parts. Parts A (header), H (meta-data), and Z (trailer boundary) are mandatory. You can enable and disable everything else at will. If you look carefully, you will see that it is now even possible to log the complete response body (part E), but this feature only works with Apache 2.x.
The key to real-time audit log processing is the index file. If, instead of a path to the file on disk, you supply a piped logging script that accepts index entries, the script will be able to handle the audit log entries as they are written to disk. We are currently busy creating a central logging console that automates the process of audit log collection. The console consists of agents that collect audit log entries from ModSecurity sensors, central database that keep the entries at one place, and a web-based GUI used to access and inspect the data.
While most users will want to use the audit log engine to store only the relevant requests (SecAuditEngine RelevantOnly), in 1.9 there are improvements that allow you to do far more than that. The new actions auditlog and noauditlog allow you to log or not log a request to the audit log explicitly. This works well with a rule:
SecFilterSelective SOME_VARIABLE CONDITION auditlog
It is now also possible to log something to the audit log simply based on the response code. The argument of the SecAuditLogRelevantStatus directive is a regular expression to apply to the response status code. If it matches, the request will be logged to the audit log. For example:
# Log internal server errors to the audit log
SecAuditLogRelevantStatus ^5
Prev [1] [2] [3] [4] Next
