Identifying Denial of Service Conditions Through Performance Monitoring

Submitted by Ryan Barnett 10/06/2009

Here is an interesting web application threat modeling exercise for you - how do you plan to identify and mitigate web application level denial of service conditions on your web sites?

This is one of those pesky security questions that, on the surface, seems pretty straight forward and then when you start peeling back the layers of complexity and interactions it becomes much more challenging. Here are some items to keep in mind.

Network DoS vs. Web App DoS

Whereas network level DoS attacks aim to flood your pipe with lower-level OSI traffic (SYN packets, etc...), web application layer DoS attacks can often be achieved with much less traffic. Just take a look at Rsnake's Slowloris app if you want to see a perfect example of the fragility of web server availability. The point here is that the amount of traffic which can often cause an HTTP DoS condition is often much less than what a network level device would identify as anomalous and therefore would not report on it as they would with traditional network level botnet DDoS attacks.

Rate Limiting

A common identification/mitigation implementation is to attempt Rate Limiting. This is essentially done by setting request threshold limits over a predefined period of time and monitoring request traffic for violations. While this is certainly useful for identify aggressive automated attacks, it has its own limitations.

• What resources to protect?
  

While protecting a web application login page is straight forward, many web site owners have not properly identified which resources are both critical and susceptible to DoS conditions. There are many web apps that are extremely resource intensive and take a long time to complete - for example any reporting interface that needs to query a back-end database to generate large reports. These apps are perfect targets for a DoS attack as the overall number of requests needed to consume open http sockets and RAM is much lower than a request for a static resource.

• What threshold to set?
  

Rate limiting is not a "one-size fits all" approach. It is highly dependent upon the resource itself. The threshold you would set against a login page to identify a brute force attack is much different then what you might set in order to identify a data scraping or DoS attack. The challenge for the defender is knowing ahead of time what to set. This is not easy as most users are missing a significant piece of the puzzle - correlating web application performance statistics. You may set an inbound rate limiting threshold for a resource that is either much too high and the application could fail due to the load (false negative), or you might set it much too low and start firing off alerts when in fact the application is able to handle the load quite fine (false positive).
Web Application Performance Monitoring

The best method for identifying fragile web resources and potential DoS thresholds is to actually monitor and track web application transaction processing times. Breach Security today announced that WebDefend 4.0 has a new Performance monitoring capability that aims to fill this important need.


With performance monitoring, the WAF user can track the average processing time including the combined average request time, web server processing time and response time. The following definitions apply in this pane:

• Request time is measured from the first packet to the last packet of the request.

• Web server processing time is measured from the last packet of the request to the first packet of the response.

• Response time is measured from the first packet to the last packet of the response.

With this information, it is easy to quickly identify the top URLs with high response latency and to pinpoint whether this is an application processing or networking issue. This data is a much truer picture of DoS conditions vs. rate limiting thresholds. The main advantages that this data brings to DoS threat modeling are identification of fragile resources that would be susceptible to attacks and to identify the an estimated threshold setting.

WASC Honeypots - Apache Tomcat Admin Interface Probes

Submitted by Ryan Barnett 10/05/2009

We have seen some probes similar to the following in our WASC Distributed Open Proxy Honeypots -

GET /manager/html HTTP/1.1
Referer: http://obscured:8080/manager/html
User-Agent: Mozilla/4.0 (compatible; MSIE

5.01; Windows NT 5.0; MyIE 3.01)
Host: obscured:8080
Connection: Close
Cache-Control: no-cache
Authorization: Basic YWRtaW46YWRtaW4=

This appears to be a probe attempt to access the Apache Tomcat Admin interface. This is due to the combination of URI "/manager/html" and port 8080. It looks as though the client is submitting authentication data in the Authorization header. If you decode the base64 data, it shows the credentials as "admin:admin" which is the default username/password combination when Tomcat is installed.

WASC Honeypot participant Erwin Geirnaert has seen similar activity and provides more data here. The attackers are conducting brute force scans trying different passwords for the "manager" account -

manager:Test
manager:adminservermanager:sqlserver
manager:2009
manager:159753
manager:1234qwerasdfzxcv
manager:1234qwerasdf
manager:1234qwer
manager:123qwe
manager:123qweasd

What do the attackers want to do once they gain access to the Tomcat server? Install backdoor/command WAR files so that they can execute code. Time to double check your default account passwords and implement those ACLs to only allow authorized clients to your Management interfaces...

Distributed Brute Force Attacks Against Yahoo

Submitted by Ryan Barnett 09/14/2009

As part of the WASC Distributed Open Proxy Honeypot Project (DOPHP), we have been able to track some pretty extensive distributed brute force attacks against Yahoo end-user email accounts. Valid email accounts and/or obtaining valid account credentials are a huge commodity for SPAMMERS. Identifying valid accounts is important as it allows them to only send SPAM messages to real accounts and they can also be able to sell lists of valid accounts to other SPAMMERS. Taking this a step further, if the SPAMMERS are able to enumerate valid credentials for an account (username and password) they can then hijack the account and use it for SPAMMING.

Normal Web Login

This methodology is not new and Yahoo is obviously aware of these attacks aim at their Yahoo mail web login interface page. This login page looks like this -


When a client clicks submit, the request looks similar to the following -

POST /config/login? HTTP/1.1
Host: login.yahoo.com
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: https://login.yahoo.com/
Cookie: B=ffetg09557ar5&b=3&s=od; cna=zwISA2sCdzgBAS+RbUtyRRes; Y=%2e
Content-Type: application/x-www-form-urlencoded
Content-Length: 296

.tries=1&.src=&.md5=&.hash=&.js=&.last=&promo=&.intl=us&.bypass=&.partner=&.u=007ofj55asupi&.v=0&
.challenge=hKhk9.OX5y0EOqJ3c4yxAH_rSrx5&.yplus=&.emailCode=&pkg=&stepid=&.ev=&hasMsgr=0&.chkP=Y&.done=http%3A%2F%2Fmy.yahoo.com&.pd=_ver%3D0%26c%3D%26ivt%3D%26sg%3D&login=foo&passwd=bar&.save=Sign+In

Notice the in the post payload that the application is tracking how many "tries" have been attempted. This is useful for throttling automated attacks and once a client goes over a limit, Yahoo then presents the user with an added CAPTCHA challenge -


Also notice that the login page is presenting the end user with a generic error message indicating that the credentials were not correct but it does not inform the user whether it was the login or password that was wrong. All of this type of anti-automation defense is good. The problem is - is Yahoo applying this type of defense consistently throughout their entire infrastructure? Are there any ways for the SPAMMERS to find a backdoor? Unfortunately, yes.

Web Services App

The WASC DOPHP has identified a large scale distributed brute force attack against what seems to be a web services authentication systems aimed at ISP or partner web applications. The authentication application is named "/config/isp_verify_user". Google links for the "isp_verify_user" app are here. One thing you will notice in looking at these results is that there is an incredibly large number of Yahoo authentication subdomains that are hosting this application and are able to authenticate clients. If you click on one of the links, you will see that the response data returned in the browser is terse. It is simply 1 line of data such as this -

ERROR:210:Required fields missing (expected l,p)

The format of this data is obviously not intended for end users, but it more tailored for parsing by web service applications. It very well could be that many front-end web applications are validating the credentials submitted by clients to these isp_verify_user app. This particular error message is returned when a client does not submit the l (login) and p (password) parameters.

If a client sends a request for a login/username that does not exist, the app will return a message of -

ERROR:102:Invalid Login

Remember the generic error message presented on the normal login web page? Not here - it is easy for a SPAMMER to automate sending requests and cycling through various login names to identify if/when they hit on a valid Yahoo account name. When this happens, the application gives a different Invalid Password error message -

ERROR:101:Invalid Password

Note that this application does not implement any of the same CAPTCHA mechanisms that the standard login page does. This means that the attackers have an unimpeded avenue of testing login credentials. If the client sends the correct credentials, they will receive a message similar to the following (where username is the data submitted in the "l" parameter) -

OK:0:username

With this information, the SPAMMERS can then log into the enumerated email account and abuse it as they wish.

Scanning Methodology

Here is small snippet of some of the transactions that were captured -

Get http://l33.login.scd.yahoo.com/config/isp_verify_user?l=kneeling@ort.rogers.com&p=qwerty HTTP/1.0
Get http://l06.member.kr3.yahoo.com/config/isp_verify_user?l=kneading@ort.rogers.com&p=000000 HTTP/1.0
Get http://69.147.112.199/config/isp_verify_user?l=kitbags@ort.rogers.com&p=333333 HTTP/1.0
Get http://217.12.8.235/config/isp_verify_user?l=kirk@ort.rogers.com&p=yankees HTTP/1.0
GET http://69.147.112.217/config/isp_verify_user?l=__miracle&p=weezer HTTP/1.0
GET http://69.147.112.202/config/isp_verify_user?l=123#@!.._69_&p=weezer HTTP/1.0
GET http://68.142.241.129/config/isp_verify_user?l=__lance_&p=weezer HTTP/1.0
GET http://202.86.7.115/config/isp_verify_user?l=__kitty__69__&p=weezer HTTP/1.0

The attackers used a three dimensional scanning methodology as described below -

1. Distributing the scanning traffic through multiple open proxy systems. This changes the source IP address as seen by the target web application so basic tracking/throttling is more challenging.
2. Distributing the traffic across different Yahoo subdomains. The advantage to this is that even if some form of failed authentication tracking is taking place, it is more difficult to synchronize this data across all systems.
3. Diagonal scanning - submitting different username/password combinations on each request. This is instead of vertical scanning which is choosing 1 username and cycling through passwords or horizontal scanning which is choosing 1 common password and cycling through userenames.
   

*Diagonal, vertical, horizontal and three dimensional brute force scanning terminology is taken from a forthcoming book by Robert "Rsnake" Hanson.

Defensive Takeaways

1. Implement proper ACLs against all web services apps. In this case, the isp_verify_user app was clearly not intended for direct client usage however there are no ACLs that prevent an end user from accessing them.
2. Need to identify any rogue web application authentication interfaces. This is a big problem for organizations that are either newly deploying distributed web services apps or those who have newly acquired a business partner.
3. Every web application must have some form of anti-automation capability in order to identify when clients are sending these requests.

Identifying Anomalous Behavior

Submitted by Ryan Barnett 09/10/2009

A quick test for you - can you tell what is abnormal about this web application request transaction that was captured by the WASC Distributed Open Proxy Honeypots Project?

GET http://www.example.com/ HTTP/1.1
User-Agent: User-Agent:Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10
Accept-Language: zh-cn
Accept: */*
Host: www.example.com
Cookie: dedifa=3984320578.43783.3716814272.929907673, BIGipCookie=0000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000, ASPSESSIONIDCSCBAQDQ=KMBPKNLCKCHMOMJAOPDPEPDF, pmaCookieVer=4, phpMyAdmin=98kdlkphdefb4lr6g5q9pke4if6gh0hg, pma_fontsize=82%25, session-id=00710064d8f2a4412ad4aeff56e96a2d, 802db0210e6b5f898c3d7fb3f82e11c0=-, _WealthCity_session=BAh7BzoPc2Vzc2lvbl9pZCIlN2NiMjM4MDM1Njk5ZDRlZTllMTY
4ZmZjYjE1NTVmNDU6EF9jc3JmX3Rva2VuIjE3YjVld0xiRkFvRy9zcnRJc1p1cDhsRldaZ
01TRTVqQ1l3RlhHUlNUNndVPQ%3D%3D--72c082556f241f5e62a26209b7c23cc42dbf
ae29, SQMSESSID=8dddae5eis8o9l2g6aul2o3ip4, JSESSIONID=678dcb81bdc1ce2e82346199c86d, SERVERID=A, CMSSESSID3aab33f1=96d98c3e54be906ecdf12195ada689a6, Compaq-HMMD=3BE1E1BD3B3B4AFED8970001A6AACE4862D267BC50C270927260D36E, _sm_au_d=1, SOrder=DatePr%2DDOWN%2D0%2D0%2D0%2D0, SRecInPage=30, ASPSESSIONIDCCRBACRC=DFEPKLMCAJDOEIPMMHNKMMCA, PHPSESSID=cf753ceefc14a51281818d11471552d4, _bz_session=BAh7BiIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsY
XNo%0ASGFzaHsABjoKQHVzZWR7AA%3D%3D--1eb7a63eabc98dec0e0f418633d652fb97f5a8db, _session_id=8e87b0524f883f7c820ec6a136f7438b, SATSATSQID=goZXTx8HbQ5eBfjGevkYJ5-Lv-M8ChUHe-NfvvDycOHkc8CTM2SrJ4F_Y_IPU6Sc, ARPT=ZXJIWKS10.32.254.104CKMWW, SESSdab19e82e1f9a681ee73346d3e7a575e=fbc279a6c6c2e66cac0a6aba173bb261, vb_session=77e75d1912c7b6d796dae865fb95149a, BAIDUID=2A880F37E13E5EB37286E3EFF5BF43AA:FG=1
Proxy-Connection: Keep-Alive

Two anomalous items of interest in this request -

1) Bogus User-Agent payload
Specifically the string "User-Agent" actually appears at the beginning of the header payload. This looks like a botched script that tried to spoof the User-Agent data.

Defensive recommendation - look for this string in the User-Agent field and tag the request as an automated client that is spoofing request header data.

2) Number of diverse SessionIDs
The number of SessionID related cookies in this request is certianly larger than normal. Also note that there are SessionIDs for different web application technologies -

ASPSESSIONID - for ASP web apps
JSESSIONID - for Java web apps
PHPSESSID - for PHP web apps

What are the odds that this website is running all three of those web technologies? Pretty slim...

My take is that the scripted client is just populating bogus SessionID data for a bunch of different apps with the hopes that this would pass basic filters that force a SessionID name to exist but don't have knowledge of valid/active values. The most likely candidate is a SPAM bot that is looking to post data to blogs, forums, etc...

Defensive recommendations -

A. Count the number of SessionIDs/Cookies submitted. If it is too large, then alert as appropriate.

B. Look for SessionIDs/Cookie names that do not match your web application technology.

There are numerous other methods to identify anomalous web application activity. Security applications that are able to automatically generate web application learning and profiling (such as web application firewalls and web fraud systems) and correlate data from application users are able to identify deviations from the norm. These are complex systems that have advanced logic components to identify anomalous traffic such as that which is presented here.

WASC Distributed Open Proxy Honeypot Update - XSS in User-Agent Field

Submitted by Ryan Barnett 8/18/2009

In case you missed it, the WASC Distributed Open Proxy Honeypot Project launched Phase III at the end of July. We have a few sensors online and as we start gathering data, we are starting our analysis. Our goal is to be able to release "events of interest" to the community to try and raise awareness of web-based attacks.

As part of my day job working with web application firewalls, I often get asked about why certain signatures should be applied in certain locations. Why not just apply signatures to parameter payloads? This would certainly cut down on potential false positives and also increase performance. While it is true that the most likely attack vector locations are parameter payloads, these are not the only ones. Where else should you look for attacks?

Well, in looking at the honeypot logs today, I noticed an interesting XSS attack vector - injecting the XSS code into the request User-Agent string. Here is an example of the captured data -

GET http://www.example.com/image-2707303-10559226 HTTP/1.1

Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */* Referer: http://www.example.co.uk/ Accept-Language: en-us User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; <script>window.open('http://www.medchecker.com/side.htm','_search')&lt;/script>) Host: www.example.com Connection: Keep-Alive

Notice the window.open javascript code in the UA payload? The intent here seems to be to target any web-based log analysis tool. So, now that you know that the User-Agent data is a possible attack vector, the question is are you applying proper input validation/signature checking there? Are you logging this data to know if clients are attempting this attack?

WASC WHID 2009 Bi-Annual Report - Social Media Sites Top Most Attacked Vertical Market

Submitted by Ryan Barnett 8/17/2009

Do you remember that line from the movie Field of Dreams: "If you build it, they will come"? Well, according to the data captured from the Web Application Security Consortium (WASC) Web Hacking Incidents Database (WHID) project, online criminals are re-enforcing that movie quote. The fact is that profit driven criminals have learned that they can utilize social networking types of web sites (such as Twitter, Facebook and MySpace) as a means to target the huge number of end users.

Breach Security Labs, a WHID contributor, has just released a whitepaper report that analyzes the WHID events from the first half of 2009. In the report, it was found that Social Networking sites (such as Twitter) that utilize Web 2.0 types of dynamic, user-content driven data, are the #1 targeted vertical market. The reason for this is really two-fold:

1) Criminals are now directly targeting the web application end-users. The bad guys are using flaws within web applications to attempt to send malicious code to end users. Popular websites that have large user bases are now ripe targets for criminals. These are target rich environments.

2) Social networking sites are so popular partly because they allow their users to customize and update their accounts with user-driven content, widgets and add-ons. These features make the sites dynamic and fun for the end users, however they also unfortunately also significantly increase the cross-site scripting (XSS) and cross-site request forgery (CSRF) attack surfaces.

The combination of these two points resulted in a number of different social media WHID 2009 Entries:

WHID 2009-2: Twitter accounts of the famous hacked
WHID 2009-4: Twitter Personal Info CSRF
WHID 2009-11: Lil Kim Facebook Hacked
WHID 2009-15: Kayne West has been Hacked
WHID 2009-23: Miley Cyrus Twitter Account Hit By Sex-Obsessed Hacker
WHID 2009-31: Double Clickjacking worm on Twitter
WHID 2009-32: 750 Twitter Accounts Hacked
WHID 2009-37: Twitter XSS/CSRF worm series

These examples clearly show that social networking sites that utilize Web 2.0 technology are the #1 attacked vertical market in WHID. This is important as social networking were grouped in the Other category in 2008. I would suspect the trend of targeting large pools of end users to continue in the future as the bad guys work on methods of automating and scaling their attacks.

Lessons Learned From Casino Surveillance

Submitted by Ryan Barnett 7/28/2009

Perhaps it is the fact that I am here in Las Vegas for the Blackhat conference, but I can't help but think about how much web application security monitoring could take a few pointers from Casino surveillance. Here are my thoughts -

Paradigm Shift
Traditional network security and brick-and-mortar banks have a similar philosophy with regards to security - keep the bad buys out! For banks, the goal is to keep the money in the vaults and for network security the aim is to keep outsiders from accessing internal systems and ports.

Web application security monitoring, on the other hand, is very similar to Casino surveillance in that the goal is not to keep the bad guys out since - you have to let the people play! The very nature of both a Casino and a web application is to allow people access to the resources.

So, the bottom line is not so much who you are but what you are doing.

Combination of recording and live analysis
All Casino cameras record all data and this information is stored for later use such as settling game disputes. If there are any problems, they can review the tapes to identify what happened. In addition to the recorded data, all Casinos have staff who are constantly monitoring and moving cameras to zero in on suspicious activity. In web application security monitoring, this is similar to having alerting systems based on rules/signatures and then supplementing that with full audit logging.

Surveillance is not a luxury
Implementation of proper surveillance inside a Casino is not a luxury but is actually mandated by law (gaming commission). Boy, it sure would be nice to have a mandate outlining exactly what *must* be logged/monitored with regards to web application transactions...

Proper Coverage
Casino surveillance cameras must be able to observe all aspects of the games including the equipment, staff and players. This includes the table layouts, the rack, chips and even view player's faces. In typical web application security logging, only a small subset of data is actually reviewed and/or logged. Most times, request and response bodies are excluded which leaves a gaping blind spot.

Anyone can be a cheat
It would be fool hearty to only focus on stereotypes when attempting to identify cheats. Cheats come in all shapes, sizes and ages. Once again, it is not who you are but what you are doing. Similarly, in webappsec, while there is some useful IP reputation data that can be used, you must actually review what the web transaction is actually doing in order to be able to identify possible malicious behavior.

Just Doesn't Look Right (JDLR)
Following proper procedures in Casinos is absolutely critical for identifying scams and cheating behavior. When staff or players deviate from these procedures, then something just doesn't look right (jdlr) and the surveillance staff can then call up increased camera coverage to focus in on the suspects. This is somewhat similar to scenarios where web application firewalls have automated learning/profiling and create positive security rules for the expected web application behavior. If a client deviates from this profile, then anomaly events can be generated. It is possible to then increase the audit logging and "tag" these clients actions for recording their traffic.

Two Types of Crimes
Casinos typically have two types of crimes, crimes against the casino and crimes against the patrons. Crimes against the casino might be where scam artists work in teams to distract staff and pass cards between themselves or possible using tools/electronics against the computerized slot machines. In web application security, these would be similar to SQL Injection types of attacks where the attacker is aiming to attack the application itself.

Casino crimes against the patrons are scenarios where cheaters try and snatch other players chips, etc... In webappsec, this would be similar to XSS/CSRF types of attacks that aim to exploit the end user through their browser.

Any other similarities you can think of?