Conformance Analysis

Traceable's Conformance analysis helps you to identify deviations in API specifications as defined by developers compared to the specification that Traceable creates based on the traffic. Traceable provides two options for performing Conformance Analysis, manual and automated. 

Conformance analysis classifies API Endpoints into 3 categories:

• Shadow Endpoints — Shadow Endpoints are those API Endpoints that are identified by Traceable based on the analysis of traffic, but these Endpoints were not part of the specification uploaded by the user. For example, if the API specification that a user has uploaded has two API Endpoints  GET /cart and GET /checkout, however, Traceable also identifies GET /admin as an API Endpoint, then GET /admin would be a shadow Endpoint.
  Note

  In Traceable GET /cart and POST /cart are two different API Endpoints.
• Orphan Endpoints – Orphan Endpoints are those API Endpoints that are part of the API specification uploaded by the user, however, Traceable has either not observed these API Endpoints or they are still being learned. For example, if the API specification that you have uploaded has two API Endpoints GET /cart and GET /checkout, however, Traceable does not observe GET /checkout during analysis, then GET /checkout would be an Orphan Endpoint.
• Matched Endpoints with issues – These are the API Endpoints that are there in the API specification that the user has uploaded and Traceable has also observed these API Endpoints. However, Traceable has observed some discrepancies in parameters, for example, type mismatch, missing parameters, and shadow parameters. An example of type mismatch would be that you have defined a parameter as an integer while Traceable has observed it as a string. 

Manual conformance analysis

Navigate to API Catalog > Conformance Analysis to start Conformance Analysis. To start the Conformance Analysis, you need to upload a valid API specification in either a YAML or JSON format. 

Conformance Analysis

 When you click on Start Analysis button from the above screenshot, a pop-up window is displayed to start the Conformance Analysis. 

Understand the Start Analysis window

Before you start Conformance analysis, you need to understand the options displayed in the pop-up window. 

Start Analysis

•  Select API Documentation- The Conformance Analysis is done against the specification that you choose from this drop-down list. You can select one or all specifications available in the drop-down list. The list of API documentation is populated either from click to upload section or from Administration () > API Discovery and then click on API Documentation.

• Click to upload – The API specifications uploaded here are added to the Select API Documentation drop-down list. You can upload a maximum of 10 API specifications, with a total size not exceeding 20 MB.
  Note

  You can select a specification from the drop-down window and upload a new API specification at the same time. For example, if there are five API specifications available in the drop-down list, you can select all five or lesser and can also add a new specification using the Click to upload section.
• Job name – Provide a meaningful name to the Conformance Analysis job.
• Environment – Select an Environment from the drop-down list against which you want to run the Conformance Analysis job. When you choose an Environment, the API Endpoints scope (the radio buttons) get populated with the number of All Endpoints and External Endpoints. You can decide to either run the Conformance Analysis against all the Endpoints, or decide to run it only against external endpoints.
• Services – Select the service(s) against which you want to run Conformance analysis. You can select one or all the services. The listed services are specific to the Environment you select.

Click on Submit to start the Conformance Analysis job. 

Analysis result

The Conformance Analysis page displays the result when the job is complete. Click on the job to view the details. The Conformance Analysis report summary is displayed in the left-hand side as shown in the screenshot below.

Conformance analysis result

 The details of conformance report have the following sections:

API Endpoints

The API Endpoints sections list the API Endpoints that the uploaded specification has in the Spec section. The Traceable section has the number of API Endpoints observed by Traceable. For example, in the above screenshot, the number of API Endpoints in the Spec are 30 while the number of API Endpoints that Traceable observed is 31. 

Scope

This section displays a summary of the parameters with which the Conformance Analysis job was run. For example, in the above screenshot, the job was run in a Production environment for all the services in the environment against all the API Endpoints. 

Details of API

The Details of API section lists the number of:

•   Shadow Endpoints

  A shadow endpoint is an endpoint that Traceable has observed during traffic analysis but is not listed in your API documentation

  A shadow endpoint is an endpoint that Traceable has observed during traffic analysis but is not listed in your API documentation

  A shadow endpoint is an endpoint that Traceable has observed during traffic analysis but is not listed in your API documentation

  A shadow endpoint is an endpoint that Traceable has observed during traffic analysis but is not listed in your API documentation

•   Orphan Endpoints

  Orphan API Endpoints are those Endpoints that are part of your API documentation, however, Traceable has not observed them as part of API traffic.

  Orphan API Endpoints are those Endpoints that are part of your API documentation, however, Traceable has not observed them as part of API traffic.

  Orphan API Endpoints are those Endpoints that are part of your API documentation, however, Traceable has not observed them as part of API traffic.

  Orphan API Endpoints are those Endpoints that are part of your API documentation, however, Traceable has not observed them as part of API traffic.

• Matched Endpoints with Issues

Click on any of the above three (shadow, orphan, or matched Endpoint with issues) to view more details about each of these. For example, the above screenshot displays all the API Endpoints that were part of both the uploaded specification and Traceable also observed those, but with issues. You can click on the row to display a window that shows the identified issues. For example, when we click on the first row, we see that the matched API Endpoint has 16 shadow parameters. 

Similarly, you can view details about Shadow and Orphan Endpoints.

Create JIRA

You can directly create a JIRA ticket for Shadow and Matched Endpoints with Issues by clicking on the Create JIRA button. The JIRA ticket lists by default list the basic details about the issue. 

Automated conformance analysis

Automated conformance analysis runs on its own at every 12-hours from the time when the last automated job was run. The Automated job runs for those APIs documentation which are enabled for Naming. For example, if 12 API documentations are available out of which 8 are enabled for naming, then the automated conformance analysis job runs for 8 API documentation.

To enable an API documentation for naming, navigate to Administration () > API Discovery. Click on API Documentation tab. To enable naming, click on an existing API specification and click on Enabled for naming button as shown below.

You can also use the API Documentation tab to upload a new specification or delete an existing documentation. Any API specification uploaded from the API Documentation tab is visible in the Select API Documentation drop-down list as displayed in the Understand the Start Analysis window section. In manual conformance analysis, you can choose the environment for which you want to run the conformance analysis, while the automated conformance analysis job runs for all the environments.

The results of the automated conformance analysis are available in the API Catalog > Conformance Analysis > Automated tab. The results of automated conformance analysis are available for 3-days. Since the automated conformance analysis is run against each environment, select the specific environment for which you want to view the automated conformance analysis job results.

Manually triggering automated conformance analysis

Automated conformance analysis jobs, by default, run at 12-hour interval. However, you can also trigger an automated conformance analysis manually from API Catalog > Security Posture page. This triggered job from the Security Posture page is in addition to the regular automated jobs. The Security Posture page displays the summary information about the automated job for that specific environment. For example as shown in the screenshot below, the spec conformance section shows the result of an automated conformance analysis job for the development environment. For more information on Security Posture, see the Security Posture topic.

Security Posture Dashboard