-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Memory usage in w3af
w3af has a long history of high memory usage, and just lately (April 2015) I've been focusing on fixing these issues. Here are some tasks I've been working on:
- https://github.com/andresriancho/w3af/issues/9873
- https://github.com/andresriancho/w3af/issues/9626
- https://github.com/andresriancho/w3af/issues/7503
While working on these issues I understood and improved implementations of caches and base classes used by w3af. This document explains the main things I've found.
HTTPResponse instances use a lot memory because of the unicode body and the dom. In most cases the dom (lxml parsed body) will use ~3Mb and the body will use ~500k. Having a list of these objects in memory will kill performance.
These objects are kept in memory when referenced by BaseParser, and parsers are kept in memory by the ParserCache.
BaseParser sub-classes are using the most memory. These are referenced by ParserCache. I always need to make sure that the parsers get properly un-referenced when they go out of the LRU.
Is a memory pig. We're using it for the XML-based parsers, and it sucks.
These don't use a lot of memory but we might have a few thousand in memory during the audit (fuzzing, mutants, etc.) phase.