Updates (January 2025 to March 2025)
Updated the Configure using UI section to add information about the Vulnerability Age parameter. Also, added examples for criteria containing one and more than one rule.
AST's scan evaluation criteria help you set up rules based on which scans pass or fail. These evaluation criteria help you integrate AST scans with your CI/CD pipeline. Configuring the correct evaluation criteria helps you set up your CI/CD pipeline so that if vulnerabilities are found in a scan, the pipeline may stop proceeding. Scan evaluation criteria are available when you trigger the scans using CLI.
Evaluation criteria are nothing but a set of rules. The evaluation criteria may have one or more than one rule. Each rule in the evaluation criteria is evaluated independently, and then the result of each rule is clubbed together to arrive at the final result. The result of each rule is either true or false. The result of each rule could either be ANDed or ORed. If R1, R2, R3, and R4 are the rules, then they can either be:
R1 & R2 & R3 & R4or
R1 | R2 | R3 | R4You can configure the evaluation criteria using the Traceable platform. For more information, see the section below.
Configure using UI
To configure the same evaluation criteria using the Traceable Platform, navigate to Testing → Settings → Evaluation Criteria. Traceable provides the following three default criteria, which cannot be edited.
Default Criteria | Description |
|---|---|
FailOnAny | This defines the criteria for the pipeline to stop if any vulnerability is found in your CI/CD pipeline. |
FailOnCritical | This defines the criteria such that if any Critical vulnerability is found in the CI/CD pipeline, then the pipeline will stop. |
FailOnHighAndAbove | This defines the criteria that if any High and above (Critical) vulnerability is found in the CI/CD pipeline, then the pipeline will stop. |
In addition to the default criteria, you can define your own criteria by choosing a combination of the following attributes. You can add one or more criteria according to your requirements. A criteria rule consists of a combination of two scopes:
Scopes | Description |
|---|---|
Assets | The API Endpoints or Services where the criteria should apply. |
Vulnerability | The security gaps corresponding to the above assets where the criteria should apply. |
By default, criteria are OR’ed. However, you can choose the criteria to be evaluated based on AND condition.
The following table describes the attributes in a criteria along with their values:
Scan Evaluation Criteria
Attribute | Description | Values |
|---|---|---|
API Endpoints (default) / Service | The assets within which the criteria should apply. |
|
Vulnerability | The vulnerabilities corresponding to the above endpoints or services. |
|
Severity | The severity of the vulnerability where Traceable should evaluate the criteria. |
|
Operators | The operator to compare the vulnerabilities of the selected severity and the threshold. |
|
Threshold | The threshold value corresponding to the vulnerabilities of the selected severity. | Any value |
Vulnerability Age | The time for which the vulnerabilities should be open. | Any value between 1-60 |
Example
One Rule
.png)
Evaluation Criteria Containing One Rule
In the above screenshot, All endpoints are selected as part of the criteria. Then, Traceable looks for Any vulnerability having Critical severity in these endpoints. If the number of critical vulnerabilities in the endpoints is Greater Than 1, and the vulnerabilities have been open for more than 15 days, the rule fails the scan, else it passes.
Multiple Rules
Evaluation Criteria Containing Multiple Rules
The above screenshot contains multiple rules.
Rule 1 — New endpoints are selected as part of the criteria. Then, Traceable looks for Any vulnerability having High severity in these endpoints. Traceable also checks if the number of vulnerabilities in the endpoints is Less Than or Equals to 5, and if the vulnerabilities have been open for more than 30 days.
Rule 2 — All services are selected as part of the criteria. Then, Traceable looks for New vulnerabilities having Low severity in these endpoints. Traceable also checks if the number of vulnerabilities in the endpoints is Less Than 1.
The results of these two rules are OR’ed to process the evaluation criteria result, which means if either of the above rules evaluates to True, the evaluation criteria fails the scan.
Understand the scan evaluation result
Running a scan from CLI gives you a host of information, including the result of scan evaluation criteria. Following is a sample output of a scan result:
Name : cliscan3
ID : c876ad68-fd97-4d80-a479-cb3f6523bd5a
Created at : 2023-06-20 08:47:09
State : Completed
Scan URL : https://app.traceable.ai/api-testing/scan/c876ad68-fd97-4d80-a479-cb3f6523bd5a?time=1d
Summary of Responses Per API Name
API Name | Requests Count | Errors/Timeouts | 200 | 302 | 400 | 403
------------------------------- | -------------- | --------------- | --- | --- | --- | ---
GET /cart | 20 | 0 | 10 | 0 | 0 | 10
POST /config/getVersion | 58 | 0 | 0 | 0 | 0 | 58
GET /product/{product-id} | 201 | 56 | 24 | 0 | 1 | 120
GET /order/{order-id} | 151 | 0 | 10 | 0 | 1 | 140
GET /user/testURI | 2 | 0 | 0 | 0 | 1 | 1
GET / | 17 | 0 | 8 | 0 | 0 | 9
POST / | 70 | 0 | 0 | 14 | 0 | 56
POST /cart | 117 | 0 | 0 | 0 | 2 | 115
POST /config/updateStatus | 58 | 0 | 0 | 0 | 0 | 58
POST /setCurrency | 59 | 0 | 0 | 58 | 0 | 1
GET /pastorders | 231 | 0 | 0 | 0 | 1 | 230
Summary of Responses Per Plugin
Plugin | Requests Count | Errors/Timeouts | 200 | 302 | 400 | 403
-------------------------------- | -------------- | --------------- | --- | --- | --- | ---
referrer_policy_misconfiguration | 31 | 0 | 27 | 0 | 1 | 3
sqli_blind | 550 | 88 | 0 | 44 | 0 | 418
os_command_injection | 192 | 0 | 0 | 0 | 0 | 192
bola | 31 | 2 | 0 | 0 | 0 | 29
sweet32 | 3 | 0 | 2 | 1 | 0 | 0
java_log4shell | 16 | 0 | 0 | 0 | 6 | 10
weak_ciphers | 3 | 0 | 2 | 1 | 0 | 0
bfla | 66 | 6 | 0 | 0 | 0 | 60
nosqli_blind | 400 | 64 | 0 | 32 | 0 | 304
ssrf_blind | 3 | 0 | 0 | 2 | 0 | 1
self_signed_certificate | 3 | 0 | 2 | 1 | 0 | 0
lucky13 | 3 | 0 | 2 | 1 | 0 | 0
revoked_certificate | 3 | 0 | 2 | 1 | 0 | 0
certificate_name_mismatch | 3 | 0 | 2 | 1 | 0 | 0
integer_overflow_error | 6 | 0 | 0 | 0 | 0 | 6
drown | 3 | 0 | 2 | 1 | 0 | 0
broken_certificate_chain | 3 | 0 | 2 | 1 | 0 | 0
unauthenticated_access | 23 | 0 | 0 | 1 | 0 | 22
xss_reflected | 208 | 40 | 0 | 16 | 0 | 152
expired_certificate | 3 | 0 | 2 | 1 | 0 | 0
beast | 3 | 0 | 2 | 1 | 0 | 0
sqli_error_based | 260 | 50 | 0 | 20 | 0 | 190
buffer_overflow | 12 | 0 | 0 | 0 | 0 | 12
crime | 3 | 0 | 2 | 1 | 0 | 0
hsts_header_misconfiguration | 31 | 0 | 27 | 0 | 1 | 3
logjam | 3 | 0 | 2 | 1 | 0 | 0
poodle | 3 | 0 | 2 | 1 | 0 | 0
tls_not_implemented | 3 | 0 | 2 | 1 | 0 | 0
==== VULNERABILITIES ====
Plugin Category | Plugin Subcategory | Vulnerabilities Found | Executed/Generated Tests | Severity
--------------------------- | ------------------------------------------------------------------------------------ | --------------------- | ------------------------ | --------
Access Control | Remote File Inclusion | 0 | 0/674 | -
Access Control | Rate Limiting | 0 | 0/0 | -
Access Control | Local File Inclusion | 0 | 0/840 | -
Authentication | Weak Password | 0 | 0/0 | -
Authentication | Unauthenticated Access | 1 | 23/23 | Critical
Authorization | Broken Function Level Authorization | 0 | 66/66 | -
Authorization | Broken Object Level Authorization | 0 | 31/31 | -
Business Logic | Mass Assignment | 0 | 0/84 | -
Business Logic | Parameter Tampering | 0 | 0/3 | -
Cross Site Scripting | Reflected Cross Site Scripting | 0 | 208/208 | -
Cross Site Scripting | Stored Cross Site Scripting | 0 | 0/0 | -
Data Exposure | Excessive Data Exposure | 0 | 0/17 | -
Improper Asset Management | Default Landing Page | 0 | 0/33 | -
Improper Asset Management | Multiple Versions of API | 0 | 0/0 | -
Insecure Design | XSLT Injection | 0 | 0/0 | -
Insecure Design | HTTPS Content Available via HTTP | 0 | 0/0 | -
Insecure Design | HTTP Redirect | 0 | 0/2 | -
Insecure Design | Anti-CSRF Tokens Check | 0 | 0/0 | -
Insecure Design | Cloud Metadata Potentially Exposed | 0 | 0/49 | -
Insecure Design | GET for POST | 0 | 0/12 | -
Json Web Token | JWT Token Expiry | 0 | 0/0 | -
Json Web Token | JWT Missing Audience Claim | 0 | 0/0 | -
==============================
Scan Evaluation Result
==============================
FAIL
------------------------------
-------- Failed Rules --------
------------------------------
Asset Type | Asset Selection | Vulnerability Selection | Severity | Operator | Threshold | Actual Count
---------- | --------------- | ----------------------- | -------- | -------------| --------- | ------------
ENDPOINT | All | Any | CRITICAL | GREATER_THAN | 0 | 1
Scan URL: https://app.traceable.ai/api-testing/scan/c876ad68-fd97-4d80-a479-cb3f6523bd5a?time=1dThe output shows the Scan Evaluation Result at the end of the scan result. As shown above, the scan evaluation result displays the conditions of the scan evaluation criteria along with the result. In the above case, the scan evaluation result has failed because the threshold was set to 0 with the GREATER_THAN operator. This means that the scan evaluation would fail even if one critical vulnerability is found.