Istio

  • Updated on Dec 27, 2024
  • Published on Dec 1, 2021
  • 8 minute(s) read

Istio is an open-source service mesh that integrates with existing distributed applications. It also works as a platform with APIs to integrate into logging and telemetry systems. Traceable provides an agent that integrates with Istio to monitor your system and detect anomalies and attacks. Traceable can also block attacks on your system based on the configurations that you do on the platform. For more information on blocking, see Detection and Custom policies. The following is a high-level deployment diagram for Traceable's Istio solution. Note that Traceable attaches to only the Istio ingress gateway.

The deployment steps assume that a stock deployment of Istio is already available.

Traceable module extension (TME)

Traceable module extension (TME) is a type of sidecar container that a Traceable Platform agent can inject into the application pod. It can be considered a miniature Traceable Platform agent.

TME has ext_cap as a component. This synchronously receives requests from applications, such as a gateway. The traceable module extension also houses libtraceable, which has the request-blocking logic. TME is used when native instrumentation or a native plugin is unavailable. In such cases, TME captures data inline and sends it to the Traceable Platform agent asynchronously. TME also blocks the requests inline.

Before you begin

Before you begin deploying Traceable's tracing agent, note that The deployment assumes a stock deployment of Istio is already available.

Support matrix

Component

Version

Istio

  • 1.13.0 and above for Traceable Istio agent 2.x

  • 1.10.0 and above for Traceable Istio agent 1.x

Traceable Platform agent

  • 1.52.0 and above for Traceable Istio agent 2.x

  • 1.36.2 and above for Traceable Istio agent 1.x

Verify your Istio deployment. It is a good practice to have your Istio's deployment diagram with you for reference. If you are installing for a demo environment, the control and ingress are in the Istio-system namespace; however, the namespace may vary. Verify where the control plane and ingress are installed by entering the following command: 

kubectl get svc -n istio-system

Other notable points

  • Modify the Istio networking configuration to use Envoy Filter for authorization

  • Make sure that Traceable's platform agent is already installed. For more information on installing, see Platform agent.

  • Use the sidecar injector to deploy the Traceable module extension (tme). For more information on the Traceable module extension, see Annotations and Labels. The tme receives data from the Istio module and reports it to traceable-agent.

  • Point the Envoy filter to the tme.

  • Ensure that port 5442 is available for Traceable module extension (tme).

Tracing agent at Istio ingress gateway

Configuring the Traceable agent for the Istio ingress gateway consists of the following four steps:

  1. Adding a label to the namespace

  2. Adding an annotation to the spec in your deployment

  3. Enabling Envoy filter

  4. Restart Ingress controller

Step 1 - Add Label

Add traceableai-inject-tme=enabled label to the istio-system namespace. 

kubectl label ns istio-system traceableai-inject-tme=enabled

Step 2 - Add Annotation

Add "tme.traceable.ai/inject:" "true" annotation to the Istio ingress gateway deployment. Enter the following command:

kubectl patch deployment.apps/istio-ingressgateway -p '{"spec": {"template": {"metadata": {"annotations": {"tme.traceable.ai/inject": "true"}}}}}' -n istio-system

Verification of TME Injection:

  • After injecting the TME, check the istio-ingressgateway pod status. If the TME has been successfully injected, the status should change from 1/1 to 2/2.

  • You can further confirm this by running kubectl describe pod <istio-ingressgateway-POD_ID> and verifying that the second container listed is indeed the TME.

To get the POD_ID for a specific pod like istio-ingressgateway, you can use the following kubectl command: 

kubectl get pods -n <namespace>

For example,

kubectl get pods -n istio-system

Look for a pod with a name similar to istio-ingressgateway-<unique_ID>. The <unique_ID> at the end is the POD_ID you are looking for.

Example output: 

NAME                                       READY   STATUS    RESTARTS   AGE
istio-ingressgateway-7f6bb877-6qhzz        2/2     Running   0          53s

In this case, the POD_ID is istio-ingressgateway-7f6bb877-6qhzz.

You can now use this POD_ID to run commands like kubectl describe pod istio-ingressgateway-7f6bb877-6qhzz -n istio-system.

Step 3 - Enable Envoy Filter

Add traceableai-istio=enabled label to the Istio ingress gateway deployment.

If everything has default names, the following command will add the above label to the Istio ingress in istio-system namespace:

kubectl patch deployment.apps/istio-ingressgateway -p '{"spec": {"template": {"metadata": {"labels": {"traceableai-istio": "enabled"}}}}}' -n istio-system

Step 4 - Restart the Ingress controller

Restart the Ingress controller for the tme to attach. Enter the following command:

kubectl rollout restart deployment istio-ingressgateway -n istio-system

Verify the deployment

Enter the following command to verify the deployment. 

kubectl get pods -n istio-system

 Example output

NAME                                  READY   STATUS    RESTARTS   AGE
istio-egressgateway-96cf6b468-87bkf   1/1     Running   0          13m
istio-ingressgateway-7f6bb877-6qhzz   2/2     Running   0          53s
istiod-58c5fdd87b-k9j29               1/1     Running   0          13m

Install Traceable Istio helm chart

Enter the following command to install the Traceable Istio helm chart:

helm install traceableai-istio traceableai/traceableai-istio --namespace istio-system

The above command creates an Envoy Filter object in the Ingress controller.

Helm values

The following tab lists the Helm values for Traceable’s Istio agents 1.x and 2.x. Use the values as per your deployment.

The following table details the various Helm values.

Value

Default value

Description

bodyCaptureSize

131072

Defines the maximum payload size in bytes to capture.

allowFailureMode

true

When set to true, Istio accepts client requests even if the communication with the Traceable Module Extension (tme) fails or the ext_cap service has returned an HTTP 5xx error.

moduleExtensionGRPCPort

5441

Traceable Module Extension (tme) sidecar gRPC port.

moduleExtensionHTTPPort

5442

Traceable Module Extension (tme) sidecar HTTP port.

moduleExtenstionIPAddress

127.0.0.1

Traceable Module Extension (tme) sidecar IP address. By default, it is 127.0.0.1 since it is in the same pod as the Istio gateway.

workloadSelectors

- "traceableai-istio: enabled"

Labels to add to pods in the mesh to enable Traceable Istio module.

allowedContentTypes

{'json', 'x-www-form-urlencoded', 'grpc'}

Capture response bodies whose content-type header contains one of the configured substrings.

serviceName

istio

The service name for spans that are captured by the Traceable Istio module.

connectTimeout

50ms

Timeout to connect to Traceable Module Extension (tme).

requestHandleTimeout

0.2s

Timeout to the ext.authz request filter.

responseHandleTimeout

50

Timeout to the envoy response handler in milliseconds.

webhook.enable

false

Webhook must only be enabled if traceable-agent is running outside the Istio cluster. This Webhook will be used to call into the external traceable-agent to inject the tme sidecar.

webhook.url

-

External traceable-agent URL. It must be HTTPS.

webhook.caBundle

-

A base64 PEM encoded CA bundle that will be used to validate the external traceable-agent’s (which is Webhook's server) certificate.

webhook.failurePolicy

ignore

Defines how to handle webhook failures.

webhook.useNamespaceSelectors.tme

true

Use namespace selector match labels to inject the tme sidecar.

webhook.useNamespaceSelectors.java

true

Use namespace selector match labels to inject the Java sidecar.

ExtProc filter configurations

Helm value

Default

Description

extProc.allowFailureMode

true

Allows Istio to accept client requests even if communication with TME fails or ext_cap returns HTTP 5xx.

extProc.allowModeOverride

false

Enables override of the filter processing mode via TME response.

extProc.messageTimeout

0.2s

Timeout for individual messages on the stream in synchronous mode.

extProc.observabilityMode

false

Enables "Send and Go" mode, sending each HTTP request/response part without pausing.

extProc.requestBodyProcessingMode

BUFFERED

Controls request body processing mode (Allowed values: STREAMED, BUFFERED, BUFFERED_PARTIAL, NONE).

extProc.requestHeaderProcessingMode

SEND

Controls request header handling (Allowed values: SEND, SKIP).

extProc.requestTrailerProcessingMode

SKIP

Controls request trailer handling (Allowed values: SEND, SKIP).

extProc.responseBodyProcessingMode

BUFFERED

Controls response body processing mode (Allowed values: STREAMED, BUFFERED, BUFFERED_PARTIAL, NONE).

extProc.responseHeaderProcessingMode

SEND

Controls response header handling (Allowed values: SEND, SKIP).

extProc.responseTrailerProcessingMode

SKIP

Controls response trailer handling (Allowed values: SEND, SKIP).

Module extension configurations

Helm value

Default

Description

moduleExtension.connectTimeout

50ms

Timeout for connecting to the TME.

moduleExtension.grpcPort

5441

gRPC port for the TME sidecar.

moduleExtension.ipAddress

127.0.0.1

IP address of the TME sidecar (defaults to the same pod as the Istio gateway).

Webhook configurations

Helm value

Default

Description

webhook.caBundle

(None)

Base64 PEM encoded CA bundle for validating webhook server certificates.

webhook.enable

false

Enables webhook if the Traceable agent runs outside the Istio cluster.

webhook.failurePolicy

Ignore

Defines the webhook failure handling policy.

webhook.url

(None)

External Traceable agent URL (must be HTTPS).

webhook.useNamespaceSelectors.tme

true

Uses namespace selector match labels to inject the TME sidecar.

General configurations

Helm value

Default

Description

requestTimeout

0.2s

Timeout for the ext_proc filter.

workloadSelectors

“traceableai-istio:enabled“

Labels to add to pods in the mesh to enable the Traceable Istio module.

serviceName

istiov2

The service name for the Traceable agent.

environment

Default configured TPA environment

The name of the environment in which the Traceable agent is running.

Restart Ingress gateway

Restart the ingress gateway so the changes take effect. Enter the following command:

kubectl rollout restart deployment istio-ingressgateway -n istio-system

Verify the deployment

Enter the following command to verify the deployment:

kubectl get envoyfilters.networking.istio.io -n istio-system                        

Example output:

traceableai-istio           104s

Upgrade

Follow these steps to upgrade the Traceable components within your Istio deployment. This process includes updating the Traceable Platform Agent (TPA), upgrading the Istio filter, and verifying the deployment to ensure a successful upgrade.

Complete the following steps:

  1. Update the Helm Repository. 

    helm repo update

  2. Upgrade the Traceable Platform Agent (TPA) to the latest version to maintain compatibility with the latest Istio filter and features.

  3. Upgrade the Traceable Istio Filter

    1. Search for the Latest Filter Version 

      helm search repo traceableai/traceableai-istio

    2. Upgrade the Istio Filter. This command upgrades the Traceable Istio filter in the istio-system namespace to the latest version.

      helm upgrade traceableai-istio traceableai/traceableai-istio --namespace istio-system

    3. Restart the Istio Ingress Gateway. Restarting the ingress gateway ensures that the updated filter is applied to incoming traffic. 

      kubectl rollout restart deployment istio-ingressgateway -n istio-system

  4. Verify the deployed chart versions.

    1. Check the Traceable Chart Version in the Traceable Namespace. 

      helm ls -n traceableai-system

    2. Check the Traceable Istio Chart Version in the Istio System Namespace. 

      helm ls -n istio-system | grep traceableai-istio

  5. Verify the TME Image Version in the Istio Ingress Gateway. Describe any istio-ingressgateway pod to confirm the TME (Traceable Module Extension) image version:

    kubectl describe pod <istio-ingressgateway-pod-name> -n istio-system

  6. Verify Versions in the Traceable UI

    • Check Platform Agent Version: Confirm that the Platform Agent version reflects the latest release.

    • Check Tracing Agent Version: Ensure the Tracing Agent version matches the latest deployment.

    • Check TME Version in Span Attributes: Inspect the traceableai.module.version attribute in incoming spans to verify that they reflect the latest module version post-upgrade.

(Optional) Custom error message

You can optionally configure a custom error code and message for the blocking function by editing the Traceable Platform agent configuration, as shown below. Configure this inside the injector section.

ext_cap:
  blocking_config:
    response_status_code: 403
    response_message: "Access Forbidden"
injector:
  blocking_config:
    response_status_code: 403
    response_message: "Access Forbidden"

If you are using Helm in values.yaml, then you would need to set:

blockingStatusCode: 
blockingMessage:
injector:
  blockingConfig:
    blockingStatusCode: 
    blockingMessage:

Make sure that the status code is set between 400 and 499. If you add values outside the 4xx range, then 403 (the default status code) is displayed in the Traceable Platform.