Scan evaluation criteria
- 26 Mar 2024
- 11 Minutes to read
- PDF
Scan evaluation criteria
- Updated on 26 Mar 2024
- 11 Minutes to read
- PDF
Article Summary
Share feedback
Thanks for sharing your feedback!
AST's scan evaluation criteria help you set up rules based on which scans pass or fail. These evaluation criteria help you integrate AST scans with your CI/CD pipeline. Configuring the correct evaluation criteria helps you set up your CI/CD pipeline so that if vulnerabilities are found in a scan, the pipeline may stop proceeding. Scan evaluation criteria are available when you trigger the scans using CLI.
Evaluation criteria are nothing but a set of rules. The evaluation criteria may have one or more than one rule. Each rule in the evaluation criteria is evaluated independently, and then the result of each rule is clubbed together to arrive at the final result. The result of each rule is either true or false. The result of each rule could either be ANDed or ORed. If R1, R2, R3, and R4 are the rules, then they can either be:
R1 & R2 & R3 & R4or
R1 | R2 | R3 | R4Either of the following methods can configure the evaluation criteria:
Using the UI
Configured in
config.yamlfile under thescanconfig
Configure using UI
You can configure the same evaluation criteria using Traceable Platform. Navigate to Testing → Settings → Evaluation Criteria. Traceable provides the following three default criteria, which cannot be edited.
FailOnAny—This defines the criteria for the pipeline to stop if any vulnerability is found in your CI/CD pipeline.
FailOnCritical — This defines the criteria such that if any Critical vulnerability is found in the CI/CD pipeline, then the pipeline will stop.
FailOnHighAndAbove — This defines the criteria that if any High and above (Critical) vulnerability is found in the CI/CD pipeline, then the pipeline will stop.
In addition to the default criteria, you can define your own criteria by choosing a combination of the following. You can add one or more criteria. By default, criteria are ORed. However, you can choose the criteria to be evaluated based on AND condition.
API Endpoints — All or new
Vulnerability — Any or new
Severity — Low, medium, high, or critical
Operators — Greater than, greater than or equal, less than, less than or equals
Threshold — The number of vulnerabilities
For example, in the above screenshot, a custom evaluation criteria is defined where if 3 or more vulnerabilities of medium severity are found on any new API Endpoint, then the CI/CD pipeline would stop.
Configure using config.yaml
Rule components
A rule is made up of two components:
Assets - Defines the types of assets on which you would like to apply the rule. Assets could either be endpoints or services. Assets, for the purpose of running scans, are further classified as
ALLorNEW. ANEWasset is any asset that was not present in the previous scan. For example, if in the first scan, you have two assets, /endpoint1 and /endpoint2, and in the second scan you have three assets, /endpoint1, /endpoint2, and /endpoint3, then /endpoint3 would be aNEWasset.Vulnerability - Defines the type of vulnerability that the assets are exposed to. Vulnerability could be of severity LOW, MEDIUM, HIGH, or CRITICAL. In the rule, when you select the level of severity, Traceable checks for that level of severity and all the levels above it. For example, if you select MEDIUM severity, then HIGH and CRITICAL are also considered. Vulnerability can also be categorized as ANY vulnerability or only
NEWvulnerability. ANEWvulnerability is a vulnerability that was not seen in the previous scan.Threshold - A limit on the number of detected vulnerabilities satisfying the severity criteria of the rule.
Operator - The following set of operators are supported:
ActionScriptActionScript
OPERATOR_GREATER_THAN OPERATOR_GREATER_THAN_OR_EQUALS OPERATOR_LESS_THAN OPERATOR_LESS_THAN_OR_EQUALS
Concept of previous scans - For a scheduled scan, the previous scan is the last scan that ran in this schedule before the current scan. For a one-time scan, the previous scan is the last scan that ran using this policy (the one used in the current scan). For all other types of scans, there is no previous scan, so all the assets are
NEW. For more information on different types of scans, see Dashboard and policies and Schedule scans topics.
Examples
The following are a few examples of how to understand the scan evaluation criteria better. The AND and OR operations are indicated as shown below:
AND operation -
all_evaluate_true: true- Indicates that you need to AND all the rules.OR operation -
all_evaluate_true: false- Indicates that you need to OR all the rules.
Note
If you have not configured any evaluation criteria in the config.yaml, then the default evaluation criteria are considered, which is defined in the scan-manager configuration.
Example 1
evalCriteria:
all_evaluate_true: true
rules:
-
assets:
all: {}
type: ASSET_TYPE_ENDPOINT
vulnerability_scope:
any: {}
operator: OPERATOR_GREATER_THAN
severity: VULNERABILITY_SEVERITY_CRITICAL
threshold: 0In the above example, all the endpoints are selected for the scan. The scan searches for critical vulnerabilities. If the number of critical vulnerabilities exceeds 0, the rule fails; otherwise, it passes.
Example 2
evalCriteria:
all_evaluate_true: true
rules:
-
assets:
new: {}
type: ASSET_TYPE_ENDPOINT
vulnerability_scope:
any: {}
operator: OPERATOR_GREATER_THAN
severity: VULNERABILITY_SEVERITY_HIGH
threshold: 5
-
assets:
all: {}
type: ASSET_TYPE_SERVICE
vulnerability_scope:
new: {}
operator: OPERATOR_GREATER_THAN
severity: VULNERABILITY_SEVERITY_LOW
threshold: 0The above snippet has two rules, one for the endpoint and the other for service.
Endpoint rule — The endpoint rule is set for all the NEW assets with HIGH and CRITICAL vulnerabilities. Since the vulnerability scope is set to high, the scan would search for high and critical vulnerabilities. For the rule to pass, the total number of vulnerabilities should be less than or equal to 5.
Service rule — The service rule is set for all the assets with vulnerabilities that are low to critical. Since the vulnerability scope is set to low, the scan would search for all the vulnerabilities from low to critical. The threshold is set to 0. This means that there should be no NEW vulnerabilities for the rule to pass.
The result of these two rules is ANDed together to arrive at the final result of the scan, whether the scan would pass or fail.
Example of bad rules
Following are a few examples of how rules should not be configured.
Example 1
assets:
all: {}
type: ASSET_TYPE_ENDPOINT
vulnerability_scope:
any: {}
operator: OPERATOR_GREATER_THAN
severity: VULNERABILITY_SEVERITY_HIGH
threshold: -2The above rule is invalid because the threshold should not be negative.
Example 2
assets:
all: {}
type: ASSET_TYPE_ENDPOINT
vulnerability_scope:
any: {}
operator: OPERATOR_LESS_THAN
severity: VULNERABILITY_SEVERITY_CRITICAL
threshold: 6The above rule is not correct because a LESS_THAN operator is used. This means that the rule would fail if the number of vulnerabilities detected is less than 6. In other words, as the number of vulnerabilities decreases from 6, this rule would fail.
Understand the scan evaluation result
Running a scan from CLI gives you a host of information, including the result of scan evaluation criteria. Following is a sample output of a scan result:
Name : cliscan3
ID : c876ad68-fd97-4d80-a479-cb3f6523bd5a
Created at : 2023-06-20 08:47:09
State : Completed
Scan URL : https://app.traceable.ai/api-testing/scan/c876ad68-fd97-4d80-a479-cb3f6523bd5a?time=1d
Summary of Responses Per API Name
API Name | Requests Count | Errors/Timeouts | 200 | 302 | 400 | 403
------------------------------- | -------------- | --------------- | --- | --- | --- | ---
GET /cart | 20 | 0 | 10 | 0 | 0 | 10
POST /config/getVersion | 58 | 0 | 0 | 0 | 0 | 58
GET /product/{product-id} | 201 | 56 | 24 | 0 | 1 | 120
GET /order/{order-id} | 151 | 0 | 10 | 0 | 1 | 140
GET /user/testURI | 2 | 0 | 0 | 0 | 1 | 1
GET / | 17 | 0 | 8 | 0 | 0 | 9
POST / | 70 | 0 | 0 | 14 | 0 | 56
POST /cart | 117 | 0 | 0 | 0 | 2 | 115
POST /config/updateStatus | 58 | 0 | 0 | 0 | 0 | 58
POST /setCurrency | 59 | 0 | 0 | 58 | 0 | 1
GET /pastorders | 231 | 0 | 0 | 0 | 1 | 230
Summary of Responses Per Plugin
Plugin | Requests Count | Errors/Timeouts | 200 | 302 | 400 | 403
-------------------------------- | -------------- | --------------- | --- | --- | --- | ---
referrer_policy_misconfiguration | 31 | 0 | 27 | 0 | 1 | 3
sqli_blind | 550 | 88 | 0 | 44 | 0 | 418
os_command_injection | 192 | 0 | 0 | 0 | 0 | 192
bola | 31 | 2 | 0 | 0 | 0 | 29
sweet32 | 3 | 0 | 2 | 1 | 0 | 0
java_log4shell | 16 | 0 | 0 | 0 | 6 | 10
weak_ciphers | 3 | 0 | 2 | 1 | 0 | 0
bfla | 66 | 6 | 0 | 0 | 0 | 60
nosqli_blind | 400 | 64 | 0 | 32 | 0 | 304
ssrf_blind | 3 | 0 | 0 | 2 | 0 | 1
self_signed_certificate | 3 | 0 | 2 | 1 | 0 | 0
lucky13 | 3 | 0 | 2 | 1 | 0 | 0
revoked_certificate | 3 | 0 | 2 | 1 | 0 | 0
certificate_name_mismatch | 3 | 0 | 2 | 1 | 0 | 0
integer_overflow_error | 6 | 0 | 0 | 0 | 0 | 6
drown | 3 | 0 | 2 | 1 | 0 | 0
broken_certificate_chain | 3 | 0 | 2 | 1 | 0 | 0
unauthenticated_access | 23 | 0 | 0 | 1 | 0 | 22
xss_reflected | 208 | 40 | 0 | 16 | 0 | 152
expired_certificate | 3 | 0 | 2 | 1 | 0 | 0
beast | 3 | 0 | 2 | 1 | 0 | 0
sqli_error_based | 260 | 50 | 0 | 20 | 0 | 190
buffer_overflow | 12 | 0 | 0 | 0 | 0 | 12
crime | 3 | 0 | 2 | 1 | 0 | 0
hsts_header_misconfiguration | 31 | 0 | 27 | 0 | 1 | 3
logjam | 3 | 0 | 2 | 1 | 0 | 0
poodle | 3 | 0 | 2 | 1 | 0 | 0
tls_not_implemented | 3 | 0 | 2 | 1 | 0 | 0
==== VULNERABILITIES ====
Plugin Category | Plugin Subcategory | Vulnerabilities Found | Executed/Generated Tests | Severity
--------------------------- | ------------------------------------------------------------------------------------ | --------------------- | ------------------------ | --------
Access Control | Remote File Inclusion | 0 | 0/674 | -
Access Control | Rate Limiting | 0 | 0/0 | -
Access Control | Local File Inclusion | 0 | 0/840 | -
Authentication | Weak Password | 0 | 0/0 | -
Authentication | Unauthenticated Access | 1 | 23/23 | Critical
Authorization | Broken Function Level Authorization | 0 | 66/66 | -
Authorization | Broken Object Level Authorization | 0 | 31/31 | -
Business Logic | Mass Assignment | 0 | 0/84 | -
Business Logic | Parameter Tampering | 0 | 0/3 | -
Cross Site Scripting | Reflected Cross Site Scripting | 0 | 208/208 | -
Cross Site Scripting | Stored Cross Site Scripting | 0 | 0/0 | -
Data Exposure | Excessive Data Exposure | 0 | 0/17 | -
Improper Asset Management | Default Landing Page | 0 | 0/33 | -
Improper Asset Management | Multiple Versions of API | 0 | 0/0 | -
Insecure Design | XSLT Injection | 0 | 0/0 | -
Insecure Design | HTTPS Content Available via HTTP | 0 | 0/0 | -
Insecure Design | HTTP Redirect | 0 | 0/2 | -
Insecure Design | Anti-CSRF Tokens Check | 0 | 0/0 | -
Insecure Design | Cloud Metadata Potentially Exposed | 0 | 0/49 | -
Insecure Design | GET for POST | 0 | 0/12 | -
Json Web Token | JWT Token Expiry | 0 | 0/0 | -
Json Web Token | JWT Missing Audience Claim | 0 | 0/0 | -
==============================
Scan Evaluation Result
==============================
FAIL
------------------------------
-------- Failed Rules --------
------------------------------
Asset Type | Asset Selection | Vulnerability Selection | Severity | Operator | Threshold | Actual Count
---------- | --------------- | ----------------------- | -------- | -------------| --------- | ------------
ENDPOINT | All | Any | CRITICAL | GREATER_THAN | 0 | 1
Scan URL: https://app.traceable.ai/api-testing/scan/c876ad68-fd97-4d80-a479-cb3f6523bd5a?time=1dThe output shows the Scan Evaluation Result at the end of the scan result. As shown above, the scan evaluation result displays the conditions of the scan evaluation criteria along with the result. In the above case, the scan evaluation result has failed because the threshold was set to 0 with the GREATER_THAN operator. This means that the scan evaluation would fail even if one critical vulnerability is found.
Sample config.yaml
Following is a sample config.yaml file with an evaluation criteria set.
scan:
evalCriteria:
all_evaluate_true : true
rules:
-
assets:
type: ASSET_TYPE_ENDPOINT
all:
vulnerability_scope:
severity: VULNERABILITY_SEVERITY_CRITICAL
operator: OPERATOR_GREATER_THAN
threshold: 0
any:
cli:
max_test_count: 10000
plugin_max_test_count: 500
plugins:
authentication:
unauthenticated_access:
config:
header_exclude_regex: (?i:(postman|host|accept|user-agent))
header_regex: (jwt|session|sessid|token|auth|key)
businesslogic:
mass_assignment:
param_exclude_regex: (time)
param_regex: .*
rce:
java_log4shell:
header_exclude_regex: (?i:(content-length))
header_regex: (?i:(referer|host|x-forwarded))
level: 5
param_exclude_regex: (?i:(time|text|content|html))
param_regex: .*
risk: 1
os_command_injection:
level: 5
param_exclude_regex: (time|text|content|html)
param_regex: .*
risk: 1
sqli:
sqli_blind:
level: 5
param_exclude_regex: (time|text|content|html)
param_regex: .*
risk: 1
sqli_error_based:
level: 5
param_exclude_regex: (time|text|content|html)
param_regex: .*
risk: 1
nosqli:
nosqli_blind:
level: 5
param_exclude_regex: (time|text|content|html)
param_regex: .*
risk: 1
tls:
beast:
broken_certificate_chain:
certificate_name_mismatch:
crime:
drown:
expired_certificate:
logjam:
lucky13:
poodle:
revoked_certificate:
self_signed_certificate:
sweet32:
tls_not_implemented:
weak_ciphers:
timeout:
minutes: 1440Was this article helpful?
