Traces
  • 19 Dec 2023
  • 9 Minutes to read
  • PDF

Traces

  • PDF

Article summary

This topic explains Traces. You can do a system-wide search on Endpoint traces and spans using Traces. Traces provides various filter options for forensic analysis of traces and spans. Traces are a faster and more efficient way to carry out forensic analysis through traces and spans. In a normal scenario without Traces, you would have to search through every trace from the Traces tab on the API Endpoints details page or the Services page. A user sends a request and creates a trace in the system. The trace may traverse through multiple services to serve the request. A different endpoint may serve each service. In the journey of trace inside your infrastructure, it hops between many services served by different Endpoints. The part of the trace or a sub-segment of a trace that is related to a specific endpoint is called an Endpoint trace in the Traceable platform.

View traces from the Services or API Endpoint page

Click on one of the listed services on the Catalog → Services page. On the Services details page that is displayed, click on the Traces tab to view a list of Traces, as shown in the screenshot below:

  Traces view from the Services page  

Click on an Endpoint, for example, GET/product/{product-id}  to view the traces associated with it. The following screenshot shows the Endpoint trace for the above example.

Trace for an Endpoint

To see the full trace, click on the See Full Trace button. The following screenshot shows the full trace of the request.

You can view the traces similarly from the Traces tab of the API Endpoint details page.


View traces and spans

You can view the same trace from the Endpoint or Services page (as explained above) from the Traces page. The Traces page allows you to visualize the trace data based on various metrics. The Traceable platform receives raw span data from its agent. Traceable's business logic creates the trace and the Endpoint trace (the sub-segment of the trace) from these spans.

The Traces page also gives you the capability to view raw span data. Spans are the building blocks for a trace. Traceable captures all the request and response data; for example, request headers, body, cookies, and so on are captured as individual span attributes. These spans are put together by Traceable to build a trace. The Traces provides both the Endpoint Traces and Spans tab to search through.

The following screenshot shows all the Endpoint traces for the last 1-hour (selected from the top menu). You can filter or apply conditions to narrow down the search. Traceable provides many inbuilt filters. If a filter has an associated operation, it is displayed when you enter a space in the search bar.

You can also auto-fill the filter by clicking on the filter icon next to any value in the columns. For example, from the above screenshot, if you want to see all the Endpoint traces for the HTTP protocol, click on the filter icon next to HTTP.

You can also combine multiple conditions; for example, the screenshot shows the output of an AND operation where protocol = HTTP AND Endpoint Name = GET /v1/data . All the filter conditions produce the result based only on the AND condition.

You can carry out similar searches based on your investigative needs.


Saved queries

Traceable provides you with an option to save your queries for future use. Saved queries can help you save time if you use the same query again. For example, if you want to save the query in the above screenshot, click on the save icon as shown below:

You can view your saved queries by clicking on the filter icon, as marked in the screenshot below.

Traceable provides a few inbuilt queries. The above screenshot shows a query showing the top APIs that see bot traffic with poor IP reputation from the Data Center. Similarly, there are other out-of-the-box queries. You can edit the default queries and save them with another name.

Saved queries with a time range

When you save a query, Traceable gives you three options for the time range you wish to save this query for. These are:

  • Inherit Application Time Range - This is equivalent to not choosing any time range.

  • Last 1 day - The time range displayed as the second option is the same that is chosen from the time-range drop-down at the top. In the example screenshot shown below, it is 1-day. It will be different if you choose a different time range. When you choose this option, the result of the query will be based on the saved time range. This means that if the application time range is 1 week and the saved query time range is 1 day, then the result would be based on the last 1 day's data.

  • Exact time - The last option saves the query result for the time when the query was run. For example, in the screenshot below, the query is run for 17/12/2023, 11:02:39 am - 18/12/2023, 11:02:39 am. When you save the query with this time range, it will always show the result between 17th and 18th December.

    Note

    The exact time range query result is not available for ever. If the chosen time range lies outside of the data retention period, the saved query does not display any result.

View the saved query detail

You can view the saved query details without running the query. Click on the filter icon and hover your pointer on the icon, as shown in the screenshot below.

Note the significance of these icons in the saved query:

  • - This icon means that the query is saved with a time range.

  • traceable_saved_query_time_lapsed_icon - This icon means that the query was saved for a specific time range; however, that time range is now out of the data retention period.


Visualization

You can also create custom visuals to compare against various metrics. One or more metrics can be added for the x and y-axis. The following screenshot gives an example of how you can use Trace visualization.

  Traceable Visualization

In the above example, you are viewing the Average Duration metric that is grouped (in a group of 5) by Endpoint Name over a time interval of 1 minute. When you hover the mouse over the columns, a pop-up window shows the call duration for all five Endpoints. The visualization shows us that at 7:46 PM, the average call duration for POST/app/learnt/QKPmJNWoGI endpoint is 3ms. Similarly, you can create several visualizations to compare various parameters.


Operators

The following table lists the various operators with their descriptions and examples.

Operator

Description

Supported attribute types

Examples

=

This operator tests for an exact match and is case-sensitive. When used against an array, it tests for at least one matching value. It can be used against a particular map entry by including a key value to test against to be included as part of the field declaration, but it cannot be used against a map itself.

String, Number, Boolean, Map Value, String Array

Bot IP = true

User Roles = Developer

Request Headers.x-forwarded-for = 1.2.3.4

!=

This operator is used as a negation of =. For arrays, this checks that there are no matching values.

String, Number, Boolean, Map Value, String Array

Bot IP != true

User Roles != Developer

Request Headers.x-forwarded-for != 1.2.3.4

IN

This operator is like =, except that it accepts a comma-separated list of allowed matches.

String, Number, Boolean, Map Value, String Array

Service Name IN dataservice, loginservice

User Roles IN developer, analyst

Request Headers.x-forwarded-for IN 1.2.3.4, 5.6.7.8

NOT_IN

Negation of IN. For arrays, this checks that there are no matching values.

String, Number, Boolean, Map Value, String Array

Service Name NOT_IN dataservice, loginservice

User Roles NOT_IN developer, analyst

Request Headers.x-forwarded-for NOT_IN 1.2.3.4, 5.6.7.8

~

Evaluates the provided regex for any matches. This operator is case-insensitive.

String, Map Value, String Array

Service Name ~ ^data

Request Headers.x-forwarded-for ~ 10\.10\.10\.\d{1,3}

<

Less than

Number

Duration < 10

>

Greater than

Number

Duration > 10

<=

Less than or equal to

Number

Duration <= 10

>=

Greater than or equal to

Number

Duration >= 10

CONTAINS

This operator checks if the string contains the provided substring. The operator is case-insensitive. For arrays, the operator checks if there are any matching values.

String, String Array

Service Name CONTAINS auth

CONTAINS_KEY

This operator checks if the map contains a key with the provided name. The operator is case-sensitive.

Map

Request Headers CONTAINS_KEY x-forwarded-for

CONTAINS_KEY_LIKE

This operator checks if the map contains a key matching the provided regex. The operator is case-insensitive.

Map

Request Headers CONTAINS_KEY_LIKE forward.*

NOT_CONTAINS_KEY

This operator checks if the map contains no key that exactly matches the provided value. The operator is case-sensitive.

Map

Request Headers NOT_CONTAINS_KEY x-forwarded-for


Filter attributes

The following tables explain a few of the filter attributes for Endpoint traces and spans.

Endpoint Traces

Attribute

Definition

API exit call

Number of outbound calls made by the API Endpoint

Calls

Number of calls to the API Endpoint

Duration

Time spent within the boundary of a service containing the API Endpoint is the time between the entry span into the service and the exit span out from the service.

End time

The end time of the segment of the trace

Endpoint ID

The ID of the API Endpoint this trace segment belongs to

Endpoint Name

Name of the API Endpoint this trace segment belongs to

Endpoint Trace ID

The span ID corresponds to the entry span, which is the starting span of the Endpoint trace segment.

Environment

All the data visible is scoped under the selected environment. But if you select AllEnvironments, this shows the environment corresponding to the trace segment.

Error count

Count of errors in the given API trace

HTTP Method

HTTP Methods, for example, GET/POST/PUT/DELETE, and so on.

IP Address

Client's IP address

Is Endpoint PII

True if the endpoint corresponding to the trace segment has any sensitive information/parameters.

Protocol

HTTP or GRPC

Request Body

Contains the API request for the given API

Request Content-Type

Content-type request header

Service ID

The ID of the Service that owns the API Endpoint trace this segment belongs to

Service Name

Name of the Service that owns the API Endpoint this trace segment belongs to

Session ID

User session ID in which the current operation (that resulted in the trace) was performed

Start time

The start time of the segment of the trace

Trace ID

The trace ID of the trace that this segment is part of

User ID

The username or client_id was extracted from the access token or Authorization header in the inbound request from outside the system.

User Role

The client's actual or assumed role is making the request extracted from the token or application security context.

User Scope

Scopes or granted authorities the client currently possesses were extracted from the token or application security context. The value would come from the scope associated with an OAuth 2.0 Access Token or an attribute value in a SAML 2.0 Assertion.

 SPANS

Attribute

Definition

Duration

The time duration for the span

Endtime

The finish time for the span

Error Count

Count the number of error occurrences in the given span

Parent Span ID

Span ID for the parent span

Protocol

One of HTTP, HTTPS, GRPC, REDIS, MONGO, JDBC, RABBIT_MQ protocol

Request Body

Contains the API request for the given API

Request Content-Type

Content-type request header

Request Params

The request parameters

Request size

The size of the request

Request Headers

The request headers

Response Content Type

Content-type response header

Response Headers

The response headers

Response Size

The size of the response

Service ID

The ID of the service corresponding to the given span

Service Name

Name of the service corresponding to the given span

Span ID

The identifier for the span

Span Name

The name for the span

Start Time

The time when the span event started

Tags

The key-value attributes for the span. It captures various events or metadata about the span

Trace ID

An Identifier for the trace that this span belongs to


Was this article helpful?
ESC

Eddy, a generative AI, facilitating knowledge discovery through conversational intelligence