Wed 20 Apr 2022
In this article we are going to see a list of several popular automated security scanning tools and how well they perform against VAmPI, a vulnerable REST API. We discussed VAmPI and saw an overview of the vulnerabilities it contains in a previous article here.
Here is an overview:
When scanning a target of an unknown security status it is impossible to know if the tool used, performed well or not. So the purpose of this experiment is to establish the success rates of the known security scanning tools against a target that is intentionally vulnerable.
Before proceeding into evaluating the tools, we need to set our expectations. This means we need to define what is expected to be found when we do not configure anything for the tool and what should be found once configured. We should note that each tool has it's own configurations and not all configurations are equal in the details required or the time needed.
Expectations
There are 13 URLs included in the OpenAPI specification provided with VAmPI as also shown in the table included on the github page. Some are accessible without authentication while other require a token to authenticate. It is expected therefore, that without any configuration the pages that require an authentication would return an error (401). For this reason, by default only a selected set of vulnerabilities would be possible to be discovered by the automated scanners. These vulnerabilities are:
- Excessive Data Exposure
- SQLi Injection
- Mass Assignment
- User and Password Enumeration
- Lack of Resources & Rate Limiting
We can go one step further and assign a difficulty based on the complexity of each attack. This helps in predicting which vulnerability is most likely to be found, with "1" meaning that it is high likely to be found and "3" not likely.
| Vulnerability | Complexity(1-3) |
|---|---|
| Excessive Data Exposure | 1 |
| SQLi Injection | 2 |
| Mass Assignment | 3 |
| User and Password Enumeration | 2 |
| Lack of Resources & Rate Limiting | 1 |
The excessive data exposure along with the lack or resources and rate limiting are graded with "1" as the complexity involved in identifying these issues is low. The user and password enumeration requires to identify the login page and test both cases to figure out that a different error response is being returned in each case. The SQLi although it is in the simplest form it can be it requires from the tool to be able to handle such an attack. Finally the mass assignment is considered the most complex as one additional value should be added to the existing post body.
Once the user is authenticated it should be possible for an automated scanner to discover the rest of the vulnerabilities as well. The following table shows these vulnerabilities along with the complexity score assigned:
| Vulnerability | Complexity(1-3) |
|---|---|
| Unauthorized Password Change | 3 |
| Broken Object Level Authorization | 3 |
| RegexDOS (Denial of Service) | 2 |
The unauthorized password change and the broken object level authorization require from the tools to analyze the existing token and compare it to what the user should be able to access or not. For this reason they are both classified with complexity "3". The denial of service through Regex should be easy to get discovered as soon as access to the endpoint is available but due to it crashes the application it becomes hard to establish a result.
Using the concepts we defined above we are going to start the tests. Each security scanner has its own configurations on how to pass parameters, authenticate etc, so in each case we will explain if and what adjustment we do. The token's time to live for VAmPI is set to several minutes so it will not interrupt our flow.
ZAP (OWASP)
ZAP is being used from the docker image available. The command we used to run it with the default configuration is shown below along with the results against VAmPI:
➜ default docker run --add-host=host.docker.internal:host-gateway -v VAmPI/:/zap/wrk/:rw -t owasp/zap2docker-weekly zap-api-scan.py -t openapi_specs/openapi3.yml -f openapi -r default.html
2022-04-19 20:46:46,401 Could not find custom hooks file at /home/zap/.zap_hooks.py
2022-04-19 20:46:54,871 Number of Imported URLs: 14
Total of 105 URLs
PASS: Directory Browsing [0]
PASS: Vulnerable JS Library [10003]
[...]
PASS: WSDL File Detection [90030]
PASS: Loosely Scoped Cookie [90033]
PASS: Cloud Metadata Potentially Exposed [90034]
WARN-NEW: X-Content-Type-Options Header Missing [10021] x 7
http://host.docker.internal:5000/createdb (200 OK)
http://host.docker.internal:5000/ (200 OK)
http://host.docker.internal:5000/users/v1 (200 OK)
http://host.docker.internal:5000/users/v1/_debug (200 OK)
http://host.docker.internal:5000/users/v1/register (200 OK)
WARN-NEW: Server Leaks Version Information via "Server" HTTP Response Header Field [10036] x 9
http://host.docker.internal:5000/createdb (200 OK)
http://host.docker.internal:5000/ (200 OK)
http://host.docker.internal:5000/users/v1 (200 OK)
http://host.docker.internal:5000/users/v1/_debug (200 OK)
http://host.docker.internal:5000/users/v1/register (200 OK)
FAIL-NEW: 0 FAIL-INPROG: 0 WARN-NEW: 2 WARN-INPROG: 0 INFO: 0 IGNORE: 0 PASS: 112It imported 14 urls and overall it did 1224 requests to get the final results. Inspecting the requests shows 484 cases of 401 responses. Having that in mind, we expect that this is the reason why the results of the scan were indicating that nothing is vulnerable with it, given that 112 checks passed and only two warns were returned.
As a next step we will provide a configuration file to ZAP which will be used in order to authenticate to it and to pass some parameters used in the requests. There is handy way to pass all the information we need using a properties file. The following properties file was used:
replacer.full_list(0).description=auth1
replacer.full_list(0).enabled=true
replacer.full_list(0).matchtype=REQ_HEADER
replacer.full_list(0).matchstr=Authorization
replacer.full_list(0).regex=false
replacer.full_list(0).replacement=Bearer ey[...]
formhandler.fields.field(0).fieldId=username
formhandler.fields.field(0).value=name1
formhandler.fields.field(0).enabled=true
formhandler.fields.field(0).fieldId=password
formhandler.fields.field(0).value=pass1
formhandler.fields.field(0).enabled=true
formhandler.fields.field(0).fieldId=email
formhandler.fields.field(0)[email protected]
formhandler.fields.field(0).enabled=true
formhandler.fields.field(0).fieldId=book_title
formhandler.fields.field(0).value=bookTitle1
formhandler.fields.field(0).enabled=trueThe results from 1129 requests made are shown in the following snippet:
➜ with_parameters docker run --add-host=host.docker.internal:host-gateway -v VAmPI/:/zap/wrk/:rw -t owasp/zap2docker-weekly zap-api-scan.py -t openapi_specs/openapi3.yml -f openapi -z "-configfile /zap/wrk/options.prop" -r with-parameters.html
2022-04-19 20:57:25,345 Could not find custom hooks file at /home/zap/.zap_hooks.py
2022-04-19 20:57:33,686 Number of Imported URLs: 14
Total of 105 URLs
PASS: Directory Browsing [0]
PASS: Vulnerable JS Library [10003]
[...]
PASS: WSDL File Detection [90030]
PASS: Loosely Scoped Cookie [90033]
PASS: Cloud Metadata Potentially Exposed [90034]
WARN-NEW: X-Content-Type-Options Header Missing [10021] x 9
http://host.docker.internal:5000/createdb (200 OK)
http://host.docker.internal:5000/ (200 OK)
http://host.docker.internal:5000/users/v1 (200 OK)
http://host.docker.internal:5000/users/v1/_debug (200 OK)
http://host.docker.internal:5000/users/v1/register (200 OK)
WARN-NEW: Server Leaks Version Information via "Server" HTTP Response Header Field [10036] x 9
http://host.docker.internal:5000/createdb (200 OK)
http://host.docker.internal:5000/ (200 OK)
http://host.docker.internal:5000/users/v1 (200 OK)
http://host.docker.internal:5000/users/v1/_debug (200 OK)
http://host.docker.internal:5000/users/v1/register (200 OK)
FAIL-NEW: 0 FAIL-INPROG: 0 WARN-NEW: 2 WARN-INPROG: 0 INFO: 0 IGNORE: 0 PASS: 112We see that we have the exact same results as to when we passed no further configuration to the tool. It should be noted that the Denial of Service vulnerability was triggered by ZAP, but due to it resulted in crashing the application ZAP returned a timeout exception.
In an attempt to better understand the results we create the following table which includes the HTTP codes returned in both cases.
| HTTP Status Codes | No Configuration | With Configuration |
|---|---|---|
| 200 | 622 | 681 |
| 400 | 0 | 330 |
| 401 | 484 | 1 |
| 404 | 118 | 118 |
| 500 | 0 | 0 |
We can see that the successful requests increased slightly with the additional configuration. The number of 401 requests was reduced to just one, which shows that authentication helped here. Still the results although slightly better than the unauthenticated state, they do not include any of the findings we were expecting. The case of the SQL Injection is unexpected, not in the sense that it was not found but despite the fact that it was mentioned that ZAP checked for sql injection, a single ' would suffice to create a response of HTTP 500 which did not happen at all as can be seen by the table provided above.
Overall out of the eight findings ZAP was unable to discover any of them.
Burp
It should be noted that in order to use Burp for active scanning a professional version is required, as in the community edition this feature is not available!
The postman collection available from VAmPI was used to pass the endpoints into Burp. We set it up in a way so Burp was a proxy for every request that was made through Postman. Running the collection automatically passes all the endpoints into the "HTTP History" tab of Burp. We should note that since the collection runs the "Login" endpoint right in the beginning the endpoints will already be authenticated in that case meaning that also the parameters passed in the requests will be valid so we will not have a two step process in this case like before with ZAP.
The following figure shows the list of the endpoints in "HTTP History"
In order to scan the entire list at once we need to visit the "Target" tab, select the host, in my case localhost:5000, right click and select "Actively Scan this host". This produces the following results:
Overall Burp made 6502 requests and found 3 high findings and some informational ones. The 3 high findings are all the same finding, the SQL injection which Burp identified correctly but due to the matching in the path name with other endpoints it classified them as different injections.
Also one of the informational findings is about "Email addresses disclosed" which is a descent attempt to inform us about sensitive data discovered although it failed to mark the plaintext passwords next to that emails.
Burp managed to find the SQL Injection and also to partially inform us about about the excessive data exposure. Overall Burp found 1.5 out of the 8 vulnerabilities.
Astra
Astra is another open sourced tool available on Github, that specializes in automated security testing for Rest APIs. This makes it a suitable candidate for our tests. Unfortunately Astra has stopped being maintained for several years already and has many bugs and problems as it seems. Some time was invested in setting it up so it can be tested and evaluate the results. The CLI version failed to properly proceed with the available postman collection and led to exceptions being thrown on the back-end as can be seen below:
In order to make some use out of it, it was decided to use the GUI version. The problem with the GUI is that it does not support the import of a collection for automated scanning. It requires the user to provide the details of a URL to be scanned along with other parameters. The SQLi available in VAmPI is feasible to be found without any authentication and without any extra parameters as it is a GET request. For this reason it seemed a good idea to scan just this URL and see if the tool would be capable of identifying that. The following image shows the configuration in the GUI:
The results did not take a long time to show up:
As it can be seen, it did not manage to discover the SQLi but reported some informational findings about missing headers. Due to this fail it was deemed better to not spend any more time scanning individually the rest of the URLs as our chances of success are slim.
Conclusion & Other Tools
Overall we saw how automated scanners with default to little configuration are not able to produce impressive results. The low quality of the results is rather unexpected given how trivial some of the vulnerabilities are, on the other hand though it points out the significance of the tester who needs to filter and evaluate the results rather trusting them blindly. Adjustments to the configuration or plugins/extensions that each tool may have, might be able to yield better results than the ones seen above but that is left to the reader if willing.
Please feel free to suggest other tools capable of security scanning APIs and given enough time they can be tested and added to this list.
Disclaimer: I am not affiliated with any of the automated security scanning tools mentioned and the purpose of this article is experimental.