Why Did Our Web Application Crash? Leveraging WAF Logging Data

Submitted by Ryan Barnett 1/15/2009

More Than A Blocking Device
Unfortunately, most people have pigeon-holed WAFs as only "Attack Blocking Devices" and that is but one use-case option. This Blog post will highlight another interesting use-case/benefit of a web application firewall - capturing full HTTP transactional logging when web applications generate error conditions. Due to the fact that a WAF is constantly monitoring live communications between clients and web applications, it is able to identify if or when a web application generates errors. This may of course be caused by an actual attack in the reqest payload, however I can speak from personal experience in working with both Breach customers and the ModSecurity users in stating that many times applications bomb out due to either a back-end communication problem or from abnormal, non-malicious client input.

Why Are These Errors Occuring?
A typical scenario is that a normal client is interacting with web application and they receive a 500 level responst status code from the web application and they have no idea why. If this were to happen to a large number of clients and/or if these were reported to the site owners, the next step would be for administrators to try and trouble-shoot and figure out what happened to cause the failure. The sad reality is that web application exception handling and logging mechanisms are sorely lacking. How is an administrator supposed to conduct analysis to identify what happened when all they have are terse or cryptic error message such as the following Apache error_log entry -

[Sat Jan 11 08:52:42 2009] [notice] child pid 24906 exit signal Segmentation fault (11)

Review of the access_log data shows the following -

202.99.11.91 - [11/Jan/2009:08:52:42 +0200] "POST /service/soap/GetItemRequest HTTP/1.1" 500 21364 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.13) Gecko/20080325 Fedora/2.0.0.13-1.fc8 Firefox/2.0.0.13"

This request seems to match up with the time stamp of the error_log entry, however unfortunately, the request uses the POST request method which means that there was data passed in the request body that is not logged by the Common Log Format (CLF). Good luck figuring out what caused the problem with this data...

Detailed server-side error and debug logging mechanisms are commonly disabled due to performance/disk space reasons, in favor of allowing the web application to include detailed stack dump data within the error pages presented to clients. This is not an ideal situation as non-malicious clients probably won't know what to do with these technical details, while malicious clients may leverage the data to fine tune an attack.

Capturing Critical Transaction Data
The audit logging capabilties of web application firewalls allow them to address the two deficiencies outlined above: 1) Capturing the full request data, and 2) Capturing the full application response. By inspecting the full transactional data, it can be determined if the inbound request was malicious or not and any detailed application error message can be reviewed.

Shortening Trouble-shooting Time
In my previous life, I was the head of operational security for a government client's web applications. The approach that I found successful when introducing a web application firewall to operational personnel was to demonstrate to them how the auditing capabilities could help them to expedite their trouble-shooting steps. Believe me, when your job requirements are tied to system uptime, this can be a life saver.

Helping QA/Security Testing
If the cause of the application error was traced to some abnormality in the client's request, then it is a good idea to utilize that same request data as a part of future QA unit tests to ensure that the updated software correctly handles it.

The web application firewall audit logs that capture "real" attacks lauched at your site are also very useful for security testing future software. If you have ever been responsible for QA/Security testing of web applications, then you are familiar with the with the challenges of developing a comprehensive set of attack or abuse test cases. These audit logs can help your organization create a repository of attack data.

Data Sharing Is The Key
They key take away from this post is that web application firewalls gather a wealth of information that could be utilized by other business units. Therefore, it is critical that processes are setup to share this information.