Metasploit: msfproxy, crawler and wmap

This is my first version of a ruby simple crawler. The main idea of this tool is to use it while interacting with the wmap database and populate the request table. Because this is my first version the database interaction is not as clean as I would like but I still believe it can be used. Now, I know for a fact that ET (creator of wmap) is already working on a crawler of its own with a lot of modularity and cleaner options. However since I already had this code with me I thought that it was a good idea to post it. Who knows maybe it will help someone :)

Things that you have to know before you start using this crawler is that it uses hpricot as its primary library. If hpricot is not installed it will use a function that I developed. Hpricot can be installed on ruby by typing:

sudo gem install hpricot

The first thing that you have to do is download the code and save it into the mf3/tools/ .

crawler.rb

Now, there are two options to run the crawler and then being able to run wmap modules based on your discovery:

OPTION 1. Two consoles (you can any crawler with this option)

1. Open two separate consoles and go in both of them to the mf3/tools/ folder.

2. On the first console open a metasploit msfconsole and follow the steps bellow:

| | _) |
__ `__ \ _ \ __| _` | __| __ \ | _ \ | __|
| | | __/ | ( |\__ \ | | | ( | | |
_| _| _|\___|\__|\__,_|____/ .__/ _|\___/ _|\__|
_|


=[ metasploit v3.3.4-dev [core:3.3 api:1.0]
+ -- --=[ 320 exploits - 99 auxiliary
+ -- --=[ 217 payloads - 20 encoders - 6 nops
=[ svn r8140 updated 8 days ago (2010.01.16)

msf > cd tools
msf > db_driver sqlite3
[*] Using database driver sqlite3
msf > db_create wm_req.db
[*] Creating a new database instance...
[*] Successfully connected to the database
[*] File: wm_req.db
msf > load db_wmap
[*] =[ WMAP v0.6 - et [ ] metasploit.com
[*] Successfully loaded plugin: db_wmap
msf > db_connect wm_req.db
[*] Successfully connected to the database
[*] File: wm_req.db

msf > ruby msfproxy.rb -w wm_req.db

3. On the second console run the following command:

ruby crawler.rb -h

Usage: crawler.rb

OPTIONS:

-a Print website structure
-c Create database to interact with wmap
-h Display this help information
-i Listening proxy IP address
-j Number of iterations. If not selected the site will be crawled until all links are visited (This can take a while)
-l Log visited websites in a text file /msf3/tools/crawler.log
-m Add discovered links to wmap database as targets. Options: -c or -w required
-n Print new possible targets during crawling
-p Listening proxy port
-v Verbose
-w Store requests to database .

ruby crawler.rb -i 127.0.0.1 -p 8080 -v -j 1 -t http://www.target.com/

4. After the crawler ends return to the metasploit console, stop the msfproxy and type the following

msf > wmap_targets -r
msf > wmap_run -e

OPTION 2. One console

____________
<>
------------
\ ,__,
\ (oo)____
(__) )\
||--|| *


=[ metasploit v3.3.2-release [core:3.3 api:1.0]
+ -- --=[ 463 exploits - 218 auxiliary
+ -- --=[ 192 payloads - 22 encoders - 8 nops
=[ svn r7808 updated 40 days ago (2009.12.10)

Warning: This copy of the Metasploit Framework was last updated 40 days ago.
We recommend that you update the framework at least every other day.
For information on updating your copy of Metasploit, please see:
http://dev.metasploit.com/redmine/projects/framework/wiki/Updating

msf > cd tools
msf > db_driver sqlite3
[*] Using database driver sqlite3
msf > db_create wm_req.db
[*] Creating a new database instance...
[*] Successfully connected to the database
[*] File: wm_req.db
msf > load db_wmap
[*] =[ WMAP v0.6 - et [ ] metasploit.com
[*] Successfully loaded plugin: db_wmap
msf > db_connect wm_req.db
[*] Successfully connected to the database
[*] File: wm_req.db
msf > ruby crawler.rb -v -j 1 -w wm_req.db -t http://www.target.com/
.
.
.
msf > wmap_targets -r
msf > wmap_run -e

Hope this helps someone.

Thanks

c4an.