Gobuster for directory, DNS and virtual hosts bruteforcing

Tue 17 Mar 2020

/Hacking/Cracking

In this article we are going to explore a "busting" tool called Gobuster. It is a really active project with many followers, which means that we get to see improvements and fixes for bugs often and ofcourse new features are being added as time passes by. The project is available on Github.

We are going to see some advantages Gobuster has over other similar tools like dirbuster and we are also going to explore its features and see examples of it. Finally a cheat-sheet is included in the end to have the most used cases in one place.

Features

What is so particularly interesting in this tool?

There are three main things that put Gobuster first in our list of busting tools.

Availability in the command line.
Able to brute force folders and multiple extensions at once.
Speed

Gobuster is written in Go and therefore good with concurrency which leads to better speeds while bruteforcing. Further, the fact that it does not require you to specify explicitly to brute force folders or files is a major advantage as it simplifies our tasks as penetration testers.

It has three main modes it can be used with:

dir - the classic directory brute-forcing mode
dns - DNS subdomain brute-forcing mode
vhost - virtual host brute-forcing mode (not the same as DNS!)

Running the help gives us the following.

er@erev0s:~$ gobuster help
Usage:
  gobuster [command]

Available Commands:
  dir         Uses directory/file brutceforcing mode
  dns         Uses DNS subdomain bruteforcing mode
  help        Help about any command
  vhost       Uses VHOST bruteforcing mode

Flags:
  -h, --help              help for gobuster
  -z, --noprogress        Don't display progress
  -o, --output string     Output file to write results to (defaults to stdout)
  -q, --quiet             Don't print the banner and other noise
  -t, --threads int       Number of concurrent threads (default 10)
  -v, --verbose           Verbose output (errors)
  -w, --wordlist string   Path to the wordlist

Use "gobuster [command] --help" for more information about a command.

We start by commenting on the top three most useful global flags. These are the flags that are usually included in every execution of the tool

The -q flag, which allows us to focus only on the output that matters.
The -t flag, which we use to define how many concurrent threads we would like to use. The number you want to use is basically up to two parameters, the bandwidth of your internet connection and the the resources of the server you are bruteforcing. Usually 50-100 is a good number to start with.
The -w flag, where we state which wordlist we want to use.

Lets explore a bit in more detail each mode now.

DIR mode

er@erev0s:~$ gobuster dir --help
Uses directory/file brutceforcing mode

Usage:
  gobuster dir [flags]

Flags:
  -f, --addslash                      Apped / to each request
  -c, --cookies string                Cookies to use for the requests
  -e, --expanded                      Expanded mode, print full URLs
  -x, --extensions string             File extension(s) to search for
  -r, --followredirect                Follow redirects
  -H, --headers stringArray           Specify HTTP headers, -H 'Header1: val1' -H 'Header2: val2'
  -h, --help                          help for dir
  -l, --includelength                 Include the length of the body in the output
  -k, --insecuressl                   Skip SSL certificate verification
  -n, --nostatus                      Don't print status codes
  -P, --password string               Password for Basic Auth
  -p, --proxy string                  Proxy to use for requests [http(s)://host:port]
  -s, --statuscodes string            Positive status codes (will be overwritten with statuscodesblacklist if set) (default "200,204,301,302,307,401,403")
  -b, --statuscodesblacklist string   Negative status codes (will override statuscodes if set)
      --timeout duration              HTTP Timeout (default 10s)
  -u, --url string                    The target URL
  -a, --useragent string              Set the User-Agent string (default "gobuster/3.0.1")
  -U, --username string               Username for Basic Auth
      --wildcard                      Force continued operation when wildcard found

Now for the dir specific flags, the help is pretty self-explainatory but it is best to spend some time on commenting some flags that might come up more often in our tests.

In many cases especially in CTF like events the SSL certifications are self signed and therefore not verified. Gobuster using the flag -k allows us to skip SSL verification and continue our pentesting unbothered. Another thing that comes in pretty handy is that we are able to define which status codes are valid for our test and which are not. This is done through the -s and -b flags where the -s flag works like a whitelist filter and the -b as a blacklist filter.

A question that comes up often is, how to specify Gobuster which files to search for? This is done using the -x flag where we can specify the file extensions we are looking for. For example if I am looking for images i could use something similar to -x jpg,png,gif.

Following there is a basic example running gobuster in dir mode.

er@erev0s:~$ gobuster dir -u https://erev0s.com -w awesome_wordlist.txt
===============================================================
Gobuster v3.0.1
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@_FireFart_)
===============================================================
[+] Url:            https://erev0s.com
[+] Threads:        10
[+] Wordlist:       awesome_wordlist.txt
[+] Status codes:   200,204,301,302,307,401,403
[+] User Agent:     gobuster/3.0.1
[+] Timeout:        10s
===============================================================
2020/02/01 18:46:25 Starting gobuster
===============================================================
/blog (Status: 301)
===============================================================
2020/02/01 18:46:36 Finished
===============================================================

Here we simply run gobuster against erev0s.com using a very small wordlist. We can see some details for the attack and below we can see which paths were found along with the status. It only found one path blog, which is actually the only correct path that was included in the wordlist.

DNS mode

In the DNS mode we are looking to find subdomains of a specific domain. This is very important in penetration testing as it might reveal areas not as well protected as others.

Lets start here also by showing the help.

er@erev0s:~$ gobuster help dns
Uses DNS subdomain bruteforcing mode

Usage:
  gobuster dns [flags]

Flags:
  -d, --domain string      The target domain
  -h, --help               help for dns
  -r, --resolver string    Use custom DNS server (format server.com or server.com:port)
  -c, --showcname          Show CNAME records (cannot be used with '-i' option)
  -i, --showips            Show IP addresses
      --timeout duration   DNS resolver timeout (default 1s)
      --wildcard           Force continued operation when wildcard found

The first thing we notice in the help is the -d flag which is used to specify the domain name we want. As you can figure out from the name of the mode, gobuster actually tries to DNS resolve the subdomains it tries so it can verify if they exist or not. As there are cases in pentesting where a specific DNS server is required to be used, Gobuster gives us the possibility to do so using the -r flag.

We are not going to spend time analyzing all flags as it is pretty straightforward to understand them. We are going directly to see an example of running Gobuster in dns mode.

er@erev0s:~$ gobuster dns -d erev0s.com -w awesome_wordlist.txt -i
===============================================================
Gobuster v3.0.1
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@_FireFart_)
===============================================================
[+] Domain:     erev0s.com
[+] Threads:    10
[+] Show IPs:   true
[+] Timeout:    1s
[+] Wordlist:   awesome_wordlist.txt
===============================================================
2020/02/01 23:21:12 Starting gobuster
===============================================================
Found: dok.erev0s.com [185.165.40.5]
===============================================================
2020/02/01 23:21:12 Finished
===============================================================

As it happened with the dir mode here as well we have a similar structure of the results. On top some details related to our dns busting, and since we have active the -i flag, besides the subdomain that was found, we also are given the IP it holds.

VHOST mode

This mode should not be mistaken to be the same as the DNS mode. In the DNS mode the tool attempts to DNS resolve the subdomains and based on that we are given the result. In vhosts mode the tool is checking if the subdomain exists by visiting the formed url and verifying the IP address. The following shows the output from the help command

er@erev0s:~$ gobuster vhost --help
Uses VHOST bruteforcing mode

Usage:
  gobuster vhost [flags]

Flags:
  -c, --cookies string        Cookies to use for the requests
  -r, --followredirect        Follow redirects
  -H, --headers stringArray   Specify HTTP headers, -H 'Header1: val1' -H 'Header2: val2'
  -h, --help                  help for vhost
  -k, --insecuressl           Skip SSL certificate verification
  -P, --password string       Password for Basic Auth
  -p, --proxy string          Proxy to use for requests [http(s)://host:port]
      --timeout duration      HTTP Timeout (default 10s)
  -u, --url string            The target URL
  -a, --useragent string      Set the User-Agent string (default "gobuster/3.0.1")
  -U, --username string       Username for Basic Auth

You can see that the flags are a subset from the flags in DIR mode. Once again it is straight forward what its flag is useful for so we are not spending time on that.

Lets go directly and see an example running the vhost mode.

er@erev0s:~$ gobuster vhost -u erev0s.com -w awesome_wordlist.txt
===============================================================
Gobuster v3.0.1
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@_FireFart_)
===============================================================
[+] Url:          http://erev0s.com
[+] Threads:      10
[+] Wordlist:     awesome_wordlist.txt
[+] User Agent:   gobuster/3.0.1
[+] Verbose:      true
[+] Timeout:      10s
===============================================================
2020/02/02 21:58:08 Starting gobuster
===============================================================
Found: www.erev0s.com (Status: 301) [Size: 0]
===============================================================
2020/02/02 21:58:09 Finished
===============================================================

In the example we are trying to find subdomains of erev0s.com using the awesome_wordlist.txt, which is a custom wordlist we made with only five lines. We can see that it found the www.erev0s.com, as it indeed exists and it redirects to non-www version of the domain.

A website behind cloudflare can mess up a scan like this!

For example when erev0s.com is behind Cloudflare protection then the following is being reported.

er@erev0s:~$ gobuster vhost -u erev0s.com -w awesome_wordlist.txt -v
===============================================================
Gobuster v3.0.1
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@_FireFart_)
===============================================================
[+] Url:          http://erev0s.com
[+] Threads:      10
[+] Wordlist:     awesome_wordlist.txt
[+] User Agent:   gobuster/3.0.1
[+] Verbose:      true
[+] Timeout:      10s
===============================================================
2020/02/02 21:57:42 Starting gobuster
===============================================================
Found: sgdfsd.erev0s.com (Status: 530) [Size: 3671]
Found: agsfdg.erev0s.com (Status: 530) [Size: 3671]
Found: hi.erev0s.com (Status: 530) [Size: 3659]
Found: fgsdfg.erev0s.com (Status: 530) [Size: 3671]
Found: www.erev0s.com (Status: 301) [Size: 338]
===============================================================
2020/02/02 21:57:42 Finished
===============================================================

As you can see Gobuster reports as found all the entries we have in the dummy wordlist we created. This is due to the fact that Cloudflare uses a different http code to respond to such a request that it does not exist. As you can see the rest of the subdomains except www.erev0s.com report that the status it was returned is Status: 530. As you can see here, this error is the origin DNS error from Cloudflare and since Gobuster does not recognize it, it considers it as found.

It would be helpful for vhost mode to have similar flags to -s and -b like in dir mode where we could whitelist/blacklist any status code we might want. In the mean time I have createdan issue in the github of gobuster and lets wait and see how the developers will react to it!

Conclusion

In this article we saw how Gobuster works and some basic examples of it. It is a pretty neat tool and very fast and it is considered a tool that every pentester will use eventually.

Following there is a cheat sheet with basic/common use cases.

We assume that you already have your awesome list to start your brute forcing, if that is not the case you can find some nice wordlists here.

Description	Command
web interesting files	gobuster dir -u mytarget.com -w path/to/my/awesome/wordlist.txt -e -t 100 -x php,txt,html,htm
find images	gobuster dir -u mytarget.com -w path/to/my/awesome/wordlist.txt -e -t 100 -x jpg,jpeg,png,gif,ico
Skip SSL verification	gobuster dir -u mytarget.com -w path/to/my/awesome/wordlist.txt -k
Bypass Basic Auth	gobuster dir -u mytarget.com -w path/to/my/awesome/wordlist.txt -U BasicAuthUser -P BasicAuthPass
Custom DNS server	gobuster dns -d mytarget.com -w path/to/my/awesome/wordlist.txt -r 10.10.10.10 -i -v