Scan evaluation criteria

AST's scan evaluation criteria help you set up rules based on which scans pass or fail. These evaluation criteria help you integrate AST scans with your CI/CD pipeline. Configuring the correct evaluation criteria helps you set up your CI/CD pipeline so that if vulnerabilities are found in a scan, the pipeline may stop proceeding. Scan evaluation criteria are available when you trigger the scans using CLI.

Evaluation criteria are nothing but a set of rules. The evaluation criteria may have one or more than one rule. Each rule in the evaluation criteria is evaluated independently, and then the result of each rule is clubbed together to arrive at the final result. The result of each rule is either true or false. The result of each rule could either be ANDed or ORed. If R1, R2, R3, and R4 are the rules, then they can either be:

R1 & R2 & R3 & R4

or 

R1 | R2 | R3 | R4

You can configure the evaluation criteria using the Traceable platform. For more information, see the below section.

Configure using UI

To configure the same evaluation criteria using the Traceable Platform, navigate to Testing → Settings → Evaluation Criteria. Traceable provides the following three default criteria, which cannot be edited.

• FailOnAny—This defines the criteria for the pipeline to stop if any vulnerability is found in your CI/CD pipeline.

• FailOnCritical — This defines the criteria such that if any Critical vulnerability is found in the CI/CD pipeline, then the pipeline will stop.

• FailOnHighAndAbove — This defines the criteria that if any High and above (Critical) vulnerability is found in the CI/CD pipeline, then the pipeline will stop.

In addition to the default criteria, you can define your own criteria by choosing a combination of the following. You can add one or more criteria. By default, criteria are ORed. However, you can choose the criteria to be evaluated based on AND condition.

• API Endpoints — All or new

• Vulnerability — Any or new

• Severity — Low, medium, high, or critical

• Operators — Greater than, greater than or equal, less than, less than or equals

• Threshold — The number of vulnerabilities

For example, in the above screenshot, a custom evaluation criteria is defined where if 3 or more vulnerabilities of medium severity are found on any new API Endpoint, then the CI/CD pipeline would stop.

Understand the scan evaluation result

Running a scan from CLI gives you a host of information, including the result of scan evaluation criteria. Following is a sample output of a scan result:

Name        : cliscan3
ID          : c876ad68-fd97-4d80-a479-cb3f6523bd5a
Created at  : 2023-06-20 08:47:09
State       : Completed
Scan URL    : https://app.traceable.ai/api-testing/scan/c876ad68-fd97-4d80-a479-cb3f6523bd5a?time=1d

Summary of Responses Per API Name
API Name                        | Requests Count | Errors/Timeouts | 200 | 302 | 400 | 403
------------------------------- | -------------- | --------------- | --- | --- | --- | ---
GET /cart                       | 20             | 0               | 10  | 0   | 0   | 10
POST /config/getVersion         | 58             | 0               | 0   | 0   | 0   | 58
GET /product/{product-id}       | 201            | 56              | 24  | 0   | 1   | 120
GET /order/{order-id}           | 151            | 0               | 10  | 0   | 1   | 140
GET /user/testURI               | 2              | 0               | 0   | 0   | 1   | 1
GET /                           | 17             | 0               | 8   | 0   | 0   | 9
POST /                          | 70             | 0               | 0   | 14  | 0   | 56
POST /cart                      | 117            | 0               | 0   | 0   | 2   | 115
POST /config/updateStatus       | 58             | 0               | 0   | 0   | 0   | 58
POST /setCurrency               | 59             | 0               | 0   | 58  | 0   | 1
GET /pastorders                 | 231            | 0               | 0   | 0   | 1   | 230

Summary of Responses Per Plugin
Plugin                           | Requests Count | Errors/Timeouts | 200 | 302 | 400 | 403
-------------------------------- | -------------- | --------------- | --- | --- | --- | ---
referrer_policy_misconfiguration | 31             | 0               | 27  | 0   | 1   | 3
sqli_blind                       | 550            | 88              | 0   | 44  | 0   | 418
os_command_injection             | 192            | 0               | 0   | 0   | 0   | 192
bola                             | 31             | 2               | 0   | 0   | 0   | 29
sweet32                          | 3              | 0               | 2   | 1   | 0   | 0
java_log4shell                   | 16             | 0               | 0   | 0   | 6   | 10
weak_ciphers                     | 3              | 0               | 2   | 1   | 0   | 0
bfla                             | 66             | 6               | 0   | 0   | 0   | 60
nosqli_blind                     | 400            | 64              | 0   | 32  | 0   | 304
ssrf_blind                       | 3              | 0               | 0   | 2   | 0   | 1
self_signed_certificate          | 3              | 0               | 2   | 1   | 0   | 0
lucky13                          | 3              | 0               | 2   | 1   | 0   | 0
revoked_certificate              | 3              | 0               | 2   | 1   | 0   | 0
certificate_name_mismatch        | 3              | 0               | 2   | 1   | 0   | 0
integer_overflow_error           | 6              | 0               | 0   | 0   | 0   | 6
drown                            | 3              | 0               | 2   | 1   | 0   | 0
broken_certificate_chain         | 3              | 0               | 2   | 1   | 0   | 0
unauthenticated_access           | 23             | 0               | 0   | 1   | 0   | 22
xss_reflected                    | 208            | 40              | 0   | 16  | 0   | 152
expired_certificate              | 3              | 0               | 2   | 1   | 0   | 0
beast                            | 3              | 0               | 2   | 1   | 0   | 0
sqli_error_based                 | 260            | 50              | 0   | 20  | 0   | 190
buffer_overflow                  | 12             | 0               | 0   | 0   | 0   | 12
crime                            | 3              | 0               | 2   | 1   | 0   | 0
hsts_header_misconfiguration     | 31             | 0               | 27  | 0   | 1   | 3
logjam                           | 3              | 0               | 2   | 1   | 0   | 0
poodle                           | 3              | 0               | 2   | 1   | 0   | 0
tls_not_implemented              | 3              | 0               | 2   | 1   | 0   | 0

==== VULNERABILITIES ====
Plugin Category             | Plugin Subcategory                                                                   | Vulnerabilities Found | Executed/Generated Tests | Severity
--------------------------- | ------------------------------------------------------------------------------------ | --------------------- | ------------------------ | --------
Access Control              | Remote File Inclusion                                                                | 0                     | 0/674                    | -
Access Control              | Rate Limiting                                                                        | 0                     | 0/0                      | -
Access Control              | Local File Inclusion                                                                 | 0                     | 0/840                    | -
Authentication              | Weak Password                                                                        | 0                     | 0/0                      | -
Authentication              | Unauthenticated Access                                                               | 1                     | 23/23                    | Critical
Authorization               | Broken Function Level Authorization                                                  | 0                     | 66/66                    | -
Authorization               | Broken Object Level Authorization                                                    | 0                     | 31/31                    | -
Business Logic              | Mass Assignment                                                                      | 0                     | 0/84                     | -
Business Logic              | Parameter Tampering                                                                  | 0                     | 0/3                      | -
Cross Site Scripting        | Reflected Cross Site Scripting                                                       | 0                     | 208/208                  | -
Cross Site Scripting        | Stored Cross Site Scripting                                                          | 0                     | 0/0                      | -
Data Exposure               | Excessive Data Exposure                                                              | 0                     | 0/17                     | -
Improper Asset Management   | Default Landing Page                                                                 | 0                     | 0/33                     | -
Improper Asset Management   | Multiple Versions of API                                                             | 0                     | 0/0                      | -
Insecure Design             | XSLT Injection                                                                       | 0                     | 0/0                      | -
Insecure Design             | HTTPS Content Available via HTTP                                                     | 0                     | 0/0                      | -
Insecure Design             | HTTP Redirect                                                                        | 0                     | 0/2                      | -
Insecure Design             | Anti-CSRF Tokens Check                                                               | 0                     | 0/0                      | -
Insecure Design             | Cloud Metadata Potentially Exposed                                                   | 0                     | 0/49                     | -
Insecure Design             | GET for POST                                                                         | 0                     | 0/12                     | -
Json Web Token              | JWT Token Expiry                                                                     | 0                     | 0/0                      | -
Json Web Token              | JWT Missing Audience Claim                                                           | 0                     | 0/0                      | -

==============================
    Scan Evaluation Result
==============================
             FAIL
------------------------------
-------- Failed Rules --------
------------------------------
Asset Type | Asset Selection | Vulnerability Selection | Severity | Operator     | Threshold | Actual Count
---------- | --------------- | ----------------------- | -------- | -------------| --------- | ------------
ENDPOINT   | All             | Any                     | CRITICAL | GREATER_THAN | 0         | 1

Scan URL: https://app.traceable.ai/api-testing/scan/c876ad68-fd97-4d80-a479-cb3f6523bd5a?time=1d

The output shows the Scan Evaluation Result at the end of the scan result. As shown above, the scan evaluation result displays the conditions of the scan evaluation criteria along with the result. In the above case, the scan evaluation result has failed because the threshold was set to 0 with the GREATER_THAN operator. This means that the scan evaluation would fail even if one critical vulnerability is found.