HTTP error handling in w3af

Exceptions

• Sub-classes of urllib2.URLError are raised by w3af when one HTTP request fails

• HTTPRequestException is raised by w3af when one HTTP request fails

• ScanMustStopException is raised by the extended_urllib.py when multiple HTTP requests fail in a row, potentially indicating that the remote server is unreachable. There are two important sub-classes of ScanMustStopException:

  • ScanMustStopByKnownReasonExc: To be used when we know the base exception which generated many HTTP request fails
  • ScanMustStopByUnknownReasonExc: To be used when the base error is unknown

• ScanMustStopByUserRequest is a subclass of ScanMustStopException which is raised when we want to stop the scan. This exception is raised by the extended_urllib.py only when the user clicks "stop" in the UI.

Code sections where exceptions are raised

• ScanMustStopByKnownReasonExc , ScanMustStopByUnknownReasonExc, HTTPRequestException and ScanMustStopByUserRequest are usually raised by the extended_urllib.py but might be raised in other places.

• Sub-classes of urllib2.URLError are usually raised by the keep alive handler.

Code sections where exceptions are handled

• One "hidden" section where exceptions are handled is in plugins.py, where UrlOpenerProxy will catch any HTTPRequestException exceptions and ignore them. This is useful to avoid having that try/except code in all the plugins.

• The last part of the code where exceptions before they reach the custom ExceptionHandler is w3afCore

Retry

All urllib2 handlers (which are used by the extended_urllib.py module) raise exceptions and might have errors, but the only place where we retry to send an HTTP request is in ExtendedUrllib._retry.

Avoid retries in any other code section, since that might lead to "multiple retries":

• Your wants to send an HTTP request using ExtendedUrllib.send
• For some reason that request fails
• ExtendedUrllib._retry is called three times to retry sending the request
• The request still fails and a HTTPRequestException is raised
• Your code catches HTTPRequestException and re-sends the request. It does this in a loop, three times.
• The result is that the request was sent (or at least w3af tried to) 9 times.

Overriding the default exception handler

If all fails, w3af overrides the default python exception handler with ExceptionHandler. Ideally we should never get here, but it has proven to be a really important source of bug reports.

Related with HTTP requests, we'll reach the custom ExceptionHandler when ScanMustStopByUnknownReasonExc is raised by extended_urllib.py