Conformance Analysis
  • 11 Jun 2024
  • 6 Minutes to read
  • PDF

Conformance Analysis

  • PDF

Article summary

Traceable's Conformance analysis helps you to identify deviations in API specifications as defined by developers compared to the specification that Traceable creates based on the traffic. Traceable provides two options for performing Conformance Analysis: manual and automated.

Conformance analysis classifies API Endpoints into 3 categories:

  • Shadow Endpoints — Shadow Endpoints are those API Endpoints that Traceable identifies based on the traffic analysis, but these Endpoints were not part of the specification uploaded by the user. For example, if the API specification that a user has uploaded has two API Endpoints  GET /cart and GET /checkout, however, Traceable also identifies GET /admin as an API Endpoint, then GET /admin would be a shadow Endpoint.

    Note

    In Traceable GET /cart and POST /cart are two different API Endpoints.

  • Orphan Endpoints Orphan Endpoints are those API Endpoints that are part of the API specification uploaded by the user, however, Traceable has either not observed these API Endpoints or are still being learned. For example, if the API specification that you have uploaded has two API Endpoints GET /cart and GET /checkout, however, Traceable does not observe GET /checkout during analysis, then GET /checkout would be an Orphan Endpoint.

  • Matched Endpoints with issues — are the API Endpoints that are there in the API specification that the user has uploaded and Traceable has also observed these API Endpoints. However, Traceable has observed some discrepancies in parameters, for example, type mismatch, missing parameters, and shadow parameters. An example of type mismatch would be that you have defined a parameter as an integer while Traceable has observed it as a string. 


Manual conformance analysis

Navigate to API Catalog → Conformance Analysis to start the Conformance Analysis. You must upload a valid API specification in YAML or JSON format to start the conformance analysis.

Traceable Conformance Analysis
Conformance Analysis

 When you click on Start Analysis button from the above screenshot, a pop-up window is displayed to start the Conformance Analysis. 


Understand the Start Analysis window

Before you start the Conformance analysis, you must understand the options in the pop-up window.

Fill details for conformance analysis
Start Analysis
  • Select API Documentation- The Conformance Analysis is done against the specification you choose from this drop-down list. You can select one or all specifications available in the drop-down list. The list of API documentation is populated either from the click to upload section or from Settings (image-1638268402925) → API Discovery and then click on API Documentation.

  • Click to upload – The API specifications uploaded here are added to the Select API Documentation drop-down list. You can upload a maximum of 10 API specifications, with a total size not exceeding 20 MB.

    Note

    You can select a specification from the drop-down window and upload a new API specification simultaneously. For example, if there are five API specifications available in the drop-down list, you can select all five or fewer and can also add a new specification using the Click to upload section.

  • Job name – Provide a meaningful name to the Conformance Analysis job.

  • Environment – Select an Environment from the drop-down list against which you want to run the Conformance Analysis job. When you choose an Environment, the API Endpoints scope (the radio buttons) gets populated with the number of All Endpoints and External Endpoints. You can decide to run the Conformance Analysis against all the Endpoints or run it only against external endpoints.

  • Services – Select the service(s) against which you want to run the Conformance analysis. You can select one or all the services. The listed services are specific to the Environment you select.

Click on Submit to start the Conformance Analysis job. 

Note

The Conformance Analysis job result is stored for 10 days, after which the job results are automatically deleted.


Analysis result

The Conformance Analysis page displays the result when the job is complete. Click on the job to view the details. The Conformance Analysis report summary is displayed on the left-hand side, as shown in the screenshot below.

Traceable conformance analysis results
Conformance analysis result

The details of the conformance report have the following sections:

API Endpoints

The API Endpoints section lists the API Endpoints in the uploaded Spec section. The Traceable section has the number of API Endpoints observed by Traceable. For example, in the above screenshot, the number of API Endpoints in the Spec is 30, while the number of API Endpoints that Traceable observed is 31.

Scope

This section summarizes the parameters with which the Conformance Analysis job was run. For example, in the above screenshot, the job was run in a Production environment for all the services in the environment against all the API Endpoints. 

Details of API

The Details of API section lists the number of:

  • Shadow Endpoints

  • Orphan Endpoints

  • Matched Endpoints with Issues

Click on any of the above three (shadow, orphan, or matched Endpoint with issues) to view more details about each. For example, the above screenshot displays all the API Endpoints that were part of the uploaded specification and Traceable's observation but with issues. You can click on the row to display a window that shows the identified issues. For example, when we click on the first row, we see that the matched API Endpoint has 16 shadow parameters.

Similarly, you can view details about Shadow and Orphan Endpoints.

Create JIRA

Clicking the Create JIRA button creates a JIRA ticket for Shadow and Matched Endpoints with Issues. The ticket lists the basic details about the issue by default.

Note

The link to job (marked in above screenshot) is valid only for 10-days from the time the results were generated.


Automated conformance analysis

The automated conformance analysis runs independently every 12 hours from when the last automated job was run. The Automated job runs for API documentation that is enabled for Naming. For example, if 12 API documentations are available, out of which 8 are enabled for naming, then the automated conformance analysis job runs for 8 API documentation.

To enable API documentation for naming, navigate to SettingsAPI Discovery. Click on the API Documentation tab. To enable naming, click on an existing API specification and click on the Enabled for naming button, as shown below


You can also use the API Documentation tab to upload a new specification or delete existing documentation. Any API specification uploaded from the API Documentation tab is visible in the Select API Documentation drop-down list as displayed in the Understand the Start Analysis window section. In manual conformance analysis, you can choose the environment for which you want to run the conformance analysis, while the automated conformance analysis job runs for all the environments.

The results of the automated conformance analysis are available in the API Catalog Conformance Analysis Automated tab. The results of the automated conformance analysis are available for 3-days. Since the automated conformance analysis is run against each environment, select the specific environment for which you want to view the automated conformance analysis job results.

Manually triggering automated conformance analysis

Automated conformance analysis jobs, by default, run at 12-hour intervals. However, you can also manually trigger an automated conformance analysis from the API Catalog Security Posture page. This triggered job from the Security Posture page is in addition to the regular automated jobs. The Security Posture page displays the summary of the automated job for that specific environment. For example, as shown in the screenshot below, the spec conformance section shows the result of an automated conformance analysis job for the development environment. For more information on Security Posture, see the Security Posture topic.

Security Posture Dashboard
Security Posture Dashboard

 


Was this article helpful?

ESC

Eddy, a generative AI, facilitating knowledge discovery through conversational intelligence