Platform agent

1.62.0 — 30^th October

Traceable’s Platform agent 1.62.0 provides the following:

Updates

Use KeyValue Parser as Default

TPA now uses the KeyValue parser as the default mode for data parsing.

OpenTelemetry Collector Upgrade

Upgraded the OpenTelemetry Collector to version v0.136.0.

Optimized libtraceable Instance Handling

Improved performance by using a single libtraceable instance for multi-tenant, multi-environment, or multi-service setups. This ensures a single gRPC client pool is used instead of multiple pools.

install.sh: Directory Ownership Fix

When installing using the install.sh script with --otlp-file-storage-dir and specifying --user and/or --group, TPA now assigns the specified user and group as the directory owner if the directory is created during installation.

Example command:

sudo ./install.sh \
  --otlp-file-storage-dir /var/lib/traceable/otlp \
  --user traceable \
  --group traceable

If the directory /var/lib/traceable/otlp does not exist, it will be created automatically, and ownership will be assigned to the specified user and group.

Mirroring Agent Update

Updated Mirroring Agent version to v1.7.1.

yq Upgrade

Upgraded yq version to v4.48.1.

Java Agent Update

Upgraded Java Agent version to v1.1.23. Environment configuration is now supported by the Java Agent.

1.61.1 — 24^th October

Traceable’s Platform agent 1.61.1 provides the following:

Updates

APIM — Support for Service, Environment, and Token Inputs

The APIM integration now supports three new configuration inputs:

• ta-service-name

• ta-environment-name

• ta-reporting-token

These string parameters allow you to specify the service name, environment name, and token for APIM integrations.

APIM — Use API Name as Service Name

A new option lets APIM automatically use the API name as the service name. This helps group endpoints by service, simplifying configuration in Azure API Management environments.

1.61.0 — 30^th September

Traceable’s Platform agent 1.61.0 provides the following:

Updates

OpenTelemetry Logs Pipeline Added

Introduced support for an OpenTelemetry logs pipeline. This enables TPA and tracing agent logs to be sent to the Traceable Platform. The next step will make these logs available in the Traceable UI.

Helm and Terraform: eBPF Service Naming Support

Added service naming configuration support for eBPF in both Helm and Terraform deployments. This allows users to define how service names are determined for applications using labels, ECS metadata, AWS tags, or environment variables.

Helm Example:

ebpfServiceNaming:
  enabled: true
  k8s:
    labels:
      - label1
      - label2

Terraform Example:

ebpf_service_naming = {
  enabled = true
  k8s = {
    labels = ["label1", "label2"]
  }
}

Helm and Terraform: Expose eBPF perfmap_queue_size

Added support for configuring the perfmap_queue_size value for the eBPF tracer in Helm and Terraform.

Helm Example:

ebpfPerfMapQueueSize: 1024

Terraform Example:

ebpf_perf_map_queue_size = 1024

Systemd VM Installations: Run TPA as Non-Root User

The TPA process can now run as a non-root user during systemd-based VM installations using the install.sh script, improving system security and compatibility.

Helm and Terraform: Configurable Kubernetes Probes

Liveness and readiness probes for Kubernetes are now configurable via Helm and Terraform. The probes now default to using tcpSocket checks instead of httpGet.

Upgraded yq to v4.47.2

TPA now uses yq version 4.47.2, which includes the latest vulnerability fixes and dependency updates.

libtraceable: Initialize Metrics Client Conditionally

The metrics client in libtraceable is now initialized only when metrics are enabled, reducing unnecessary resource usage.

Resolved Issues

Fixed Vulnerability GHSA-2464-8j7c-4cjm

Resolved the security vulnerability identified as GHSA-2464-8j7c-4cjm.

Fixed Istio Crash When Using TME

Resolved a crash issue in Istio environments when using TME. The fix involves dropping WebSocket requests if the handler is not enabled, ensuring stable operation.

AWS API Gateway: Duration Fix for Spans

Fixed an issue where spans in AWS API Gateway occasionally showed incorrect durations (sometimes over 10 minutes). Durations are now correctly set to zero when not explicitly defined.

Fixed Regression in OpenShift SCC Configuration

Addressed a regression where OpenShift SCC incorrectly set securityContext.runAsUser to 0 when not defined in values.yaml. The behavior now aligns with expected configuration handling.

1.60.1 — 12^th September

Traceable Platform agent version 1.60.1 provides the following:

Updates

Configurable Kubernetes Liveness and Readiness Probes

Kubernetes liveness and readiness probes for TPA are now configurable. The probes have also been updated to use httpGet checks instead of tcpCheck. This provides more accurate health monitoring and flexibility to tune probe parameters for different environments.

1.60.0 — 1^st September

Traceable Platform agent version 1.60.0 provides the following:

Updates

Support for Server-Sent Events Parsing on MCP Servers

Added support for parsing Server-Sent Events (SSE) stream bodies on MCP servers. This enables the classification of streaming data formatted as data: value, enhancing visibility into SSE-based event streams.

Envoy ext_proc: Drop Spans for Rate-Limited Requests

Requests resulting in HTTP 408 or 429 due to rate limiting are now excluded from span creation. This prevents noisy spans and reduces processing overhead during DDoS scenarios.

Configure Extra Volumes and Volume Mounts in Kubernetes

You can now specify additional volumes and volumeMounts for the TPA deployment in Kubernetes using Helm. This allows users to mount custom data or configuration paths as needed.

Example:

tpaVolumes:
  - name: tpa-data-volume
    emptyDir: {}
tpaVolumeMounts:
  - name: tpa-data-volume
    mountPath: /data

Resolved Issues

Fixed gRPC Body Capture for ext_proc Filter in Istio

Fixed a bug where request and response bodies for gRPC traffic routed through Envoy in Istio environments were not being captured due to missing support in the ext_proc filter. This issue has been resolved.

Fixed Binding Bug When HTTP Reverse Proxy is Disabled

Fixed a configuration issue where the OTLP receivers were incorrectly binding to 127.0.0.1 instead of 0.0.0.0 when the HTTP reverse proxy was disabled. This caused connectivity issues with the eBPF tracer during upgrades.

1.59.0 — 29^th July

Traceable Platform agent version 1.59.0 provides the following:

Updates

Upgraded OpenTelemetry Collector to v0.128.0

TPA now uses OpenTelemetry Collector version 0.128.0, bringing improvements, bug fixes, and enhanced compatibility with the latest OpenTelemetry ecosystem.

Tracing Agent Authentication Support

Added support for authenticating Tracing Agents when sending data to TPA, improving security in multi-agent and multi-tenant environments.

HSL Enhancements: API Token Authentication and Sender IP Capture

HSL now supports API token-based authentication. Additionally, the sender's IP address is captured and added as an attribute to the HSL message, enhancing trace context and auditability.

Upgraded yq to v4.47.1

The bundled version of yq has been upgraded to v4.47.1, which includes important bug and security fixes.

Resolved Issues

Fixed Vulnerability: GHSA-fv92-fjc5-jj9h

Resolved a high-severity vulnerability identified as GHSA-fv92-fjc5-jj9h.

Protected Spans Now Retained by Span Remover Processor

Fixed an issue where protected spans were incorrectly dropped by TPA. Protected spans are now retained as expected.

1.58.0 — 2^nd July

Updates

Traceable Platform agent version 1.58.0 provides the following:

Collector Servers Now Bind to 127.0.0.1 by Default

All collector endpoint servers (e.g., OTLP receiver, Prometheus metrics) now default to binding on 127.0.0.1 (IPv4 localhost). This enhances security by restricting external exposure.

Breaking change

If you are using ports 4317, 4318, 5441, or 9411 to send traffic to TPA, update your Traceable Agents to use ports 5442 (non-TLS) or 5443 (TLS). This update applies to cases where traffic can be routed through the reverse proxy.

Upgraded Go Runtime to v1.24

TPA now uses Go 1.24, benefiting from the latest performance optimizations and runtime improvements.

Centralized TPA Deployment via Terraform (EC2-backed)

Introduced a new Terraform-based deployment model for centralized TPA installations. This setup supports DNS resolution, auto-scaling, and is backed by EC2 instances, providing a scalable and robust deployment option.

Helm and Terraform Support for extCapTimeoutMs

You can now configure extCapTimeoutMs to set a timeout (in milliseconds) for ext_cap service calls. A value of 0 disables the timeout. This provides a latency cap for blocking integrations.

Configuration Option to Skip Blocking on Client Spans

Added ext_cap.blocking_config.skip_client_spans configuration. When set to true (default), blocking is not executed on client spans.

Improved Performance with re2-based Coraza Integration

TPA now uses Traceable's custom implementation of the RE2 engine within Coraza. This significantly enhances CPU efficiency.

ModSecurity Processor Enabled for Windows

The ModSecurity processor is now enabled for TPA running on Windows.

Improved Error Handling in APIM Templates

Enhanced the error handling logic in APIM templates to ensure failures do not impact existing APIs.

Resolved Issues

Fixed Authorization Bug for otlphttp Header Case Sensitivity

Corrected a case sensitivity issue where requests with a capitalized Traceableai-Agent-Token header were rejected. TPA now accepts headers regardless of casing.

Fixed Uninitialized Blocking Flag in Go Agent

In the Go agent, the blocking flag is now correctly initialized to false within libtraceable, preventing potential unexpected behavior.

Fixed CloudFormation Deployment Failure on Specific AMIs

The AWS CloudFormation deployment script for traffic mirroring has been updated to install the aws-cfn-bootstrap package outside of a virtual environment. This resolves install failures on Amazon Linux 2023 and Ubuntu ARM64 AMIs.

1.57.2 — 11^th June

Updates

Mirroring Agent Update

Improved body capture and service name reporting in the Mirroring Agent to enhance traffic observability across VM, GCP, and AWS environments.

These fixes improve observability and attribution for mirrored traffic across all supported environments (VM, GCP, AWS).

1.57.1 — 6^th June

Updates

Helm and Terraform Support for Custom Environment Variables

You can now define custom environment variables for the TPA and eBPF containers using Helm or Terraform. This provides greater flexibility in advanced deployment scenarios, including debugging, feature toggling, or integrating with external systems.

• Helm configuration:

  tpaEnvironmentVariables:
    - name: MY_ENV_VAR
      value: value1
    - name: ANOTHER_ENV_VAR
      value: value2
  
  ebpfEnvironmentVariables:
    - name: MY_ENV_VAR
      value: value1
    - name: ANOTHER_ENV_VAR
      value: value2

• Terraform configuration:

  tpa_environment_variables = [
    {
      name  = "MY_ENV_VAR"
      value = "value1"
    },
    {
      name  = "ANOTHER_ENV_VAR"
      value = "value2"
    }
  ]
  
  ebpf_environment_variables = [
    {
      name  = "MY_ENV_VAR"
      value = "value1"
    },
    {
      name  = "ANOTHER_ENV_VAR"
      value = "value2"
    }
  ]

1.57.0 — 30^th May

Traceable Platform agent 1.57.0 provides the following:

Updates

Restricted Kubernetes Cluster Roles for Injector

Reduced the permissions required by the injector to only the necessary Kubernetes resources and actions. This enhances security by following the principle of least privilege, ensuring the injector has access only to the resources it needs.

Configurable Timeout for ext_cap Service Calls

Introduced a configurable timeout for external capability (ext_cap) service calls, allowing users to set a maximum time (in milliseconds) for responses. If the timeout is exceeded, the decision-making process bypasses the ext_cap call, improving system resilience and preventing delays from propagating through the system.

Configurable Content Type Capture in Ingress NGINX Controller

Added support to configure the content types (such as JSON, gRPC, and XML) that the Ingress NGINX controller captures for analysis. This can be modified using:

• injector.nginxCpp.config.captureContentTypes (Helm)

• njector.nginx_cpp.config.capture_content_types (Terraform)

Install Script: Added Cleanup for Envoy Proxy During Uninstallation

The install.sh script now includes cleanup for the Envoy proxy component when uninstalling TPA, ensuring no leftover processes or files remain.

Install Script: Preserve Custom package_url When Installing HTTP Proxy

The install.sh script has been enhanced to retain an existing package_url value when installing the HTTP proxy, ensuring user-defined configurations are not overwritten.

AWS Traffic Mirroring CloudFormation Template: Added Tag Prefix Option

Introduced a TagPrefix parameter in the AWS CloudFormation template. When provided, this value will be added as a prefix to all AWS tags generated during deployment.

Helm: Added loadBalancerIP Configuration

Users can now specify the loadBalancerIP in the Helm configuration when deploying TPA as a LoadBalancer service, providing better control over load balancer provisioning.

Helm: Support for Different port and targetPort When Using LoadBalancer Service Type

Added support for configuring different values for port and targetPort in Kubernetes LoadBalancer services. This allows scenarios where the external TLS port differs from the internal agent port (e.g., port: 5443 with targetPort: 5442).

Helm: Support for Base64-Encoded TLS Certificates

Introduced new Helm values to configure base64-encoded TLS certificates directly in the chart:

• tlsPrivateCertificatesAsString.rootCAB64

• tlsPrivateCertificatesAsString.certB64

• tlsPrivateCertificatesAsString.keyB64
  This enables dynamic TLS certificate injection during deployments (e.g., via pipelines), complementing existing support for TLS secrets and files.

Updated Traffic Mirroring for VM, GCP, and AWS to Use Mirroring Agent

The Mirroring Agent has entirely replaced the Suricata-based agent for traffic mirroring in VM, GCP, and AWS environments. This ensures a consistent, streamlined approach for all mirroring use cases.

Do Not Run ModSecurity on Response Headers and Body

To improve performance, ModSecurity will now skip processing response headers and body, focusing analysis solely on request data.

Resolved Issues

Race Condition in AWS VPC Mirroring Provisioning Script

Resolved a race condition in the AWS VPC Mirroring provisioning script that caused package management locks and installation failures.

Incorrect Example Configurations for Deleting in GCP Mirroring

Corrected the example configurations for deleting resources in GCP Mirroring to prevent user errors during cleanup.

Removed imageCredentials.envoyRegistrySuffix Helm Value

The imageCredentials.envoyRegistrySuffix Helm value has been removed. The grpcToHttp.image value should now be used to specify the Envoy image.

1.56.0 — 29^th April

Traceable’s Platform Agent 1.56.0 release provides the following:

Updates

Configurable Injector Webhook Domain

You can now set a custom domain name for the injector's MutatingWebhookConfiguration using the new injectorWebhookDomain Helm parameter.
This enhancement helps users who deploy TPA with private TLS certificates avoid certificate validation errors caused by Kubernetes service DNS resolution.
It removes the need to modify private certificates, making secure deployments easier and more flexible.

Updated NGINX C++ Agent Version to v0.1.91

The NGINX agent has been updated to version 0.1.91, incorporating the latest improvements and stability enhancements.

Updated eBPF Tracer Version to v1.22.0

The eBPF tracer component has been upgraded to version 1.22.0, improving performance and internal fixes.

Added ebpfCustomSpanAttributes Helm Value

You can now configure ebpfCustomSpanAttributes in Helm to define key-value pairs that will be added as span attributes to all spans captured by the eBPF tracer.
This allows for enhanced observability and context tagging in your tracing data.

Kubernetes Compatibility for eBPF Tracer Pods

The node-role.kubernetes.io/master label is now added only for Kubernetes versions earlier than 1.20.
This prevents deprecation warnings and ensures smoother Helm deployments on newer Kubernetes clusters.

Override eBPF Environment with ebpfEnvironment Helm Value

Introduced a new Helm value ebpfEnvironment that allows you to set an environment different from the main TPA instance for the eBPF tracer.
Useful when the tracer should be logically separated or reported under a distinct environment.

Systemd Task Limits Configuration in install.sh

The install script now supports additional flags:

• --set-tasks-accounting: Enables systemd's TasksAccounting for TPA.

• --tpa-max-tasks <MAX_TASKS>: Sets a custom maximum task limit for TPA.
  These options provide greater control over process limits and are especially helpful in resource-constrained VM environments.

Expanded Environment Variable-Based Configuration Override

You can now override boolean and array-type configuration values using environment variables prefixed with TA_OVERRIDE_CONFIG.
This extends the existing support for overriding scalar values and allows for more dynamic deployments.

CloudFormation Support for Base64-Encoded Configuration Override

Added a new parameter TraceableConfigOverrideBase64 in the AWS CloudFormation template.
This allows users to supply a base64-encoded YAML string to override the TPA configuration during stack creation.

Custom Installation Paths in Windows Install Script

The Windows installation script now supports three new options:

• InstallDir

• DataDir

• TempDir
  These allow full control over where the agent is installed and where its data is stored.

Custom Envoy Registry Support for gRPC to HTTP Proxy

Added a new Helm value envoyRegistrySuffix to support pulling the envoy image from a custom registry path.
This enables seamless integration with private or mirrored registries by aligning with enterprise image-pull policies.

Resolved Issues

Fixed TPA Crash Due to Coraza WAF WASM Error

Resolved an issue causing TPA to crash with the error wasm error: invalid table access, seen primarily on RHEL 9.5 and Fedora systems.

TME Containers Now Properly Terminate on Pod Deletion

Fixed an issue where TME containers injected into ingress gateway pods were not being terminated when the gateway pod was deleted.

Terraform Provisioner: Reload Script Now Created Conditionally

Fixed a bug in the AWS traffic mirroring Terraform provisioner where the reload-refresh-token.sh script was being created unconditionally due to a faulty boolean check.

1.55.1 — 2^nd April

Traceable’s Platform Agent 1.55.1 release provides the following:

Update

Removed deprecated Kubernetes label beta.kubernetes.io/os for compatibility with GKE

To ensure successful Helm deployments on Kubernetes versions 1.14 and above (including GKE), support for the deprecated beta.kubernetes.io/os label has been removed.

1.55.0 — 31^st March

Traceable’s Platform Agent 1.55.0 release provides the following:

Updates

Upgraded OpenTelemetry Collector to v0.120.0

Ensures compatibility with the latest OpenTelemetry features and improvements.

Multi-environment support for Apigee Syslog Server

You can now configure the Apigee Syslog integration to support multiple environments.

New Helm configuration options for Kubernetes service customization

• Added support to set service labels and annotations.

• Added support to set externalTrafficPolicy: Local.
  These enhancements make it easier to configure TPA services when using a LoadBalancer service type in cloud environments like EKS or AKS.

Freeze apt and yum upgrades for Traceable packages installed via install.sh

Marking packages on hold prevents accidental overwriting of configurations. Traceable packages will now only update through the install script.

Upgraded Java agent version in injector to v1.1.15

Restart logic optimized for Kubernetes deployments

The agent and tracer deployments will now only restart if:

• The config map changes, or

• TLS or injector is enabled and self-signed certificates are generated during Helm install or upgrade.

Enhanced User Attribution Rules

• Added support for projecting entire JWT claims using UrlProjector.

• Introduced AttributeStringAppend, similar to AttributeArrayAppend, for rule building.

HAProxy improvement

Blocked users will receive an event ID in the response for easier tracking.

Resolved Issues

Fixed: Span ID not shown in response body for blocked requests

Blocked responses now correctly include the trace's spanID.

Security Fixes

• Upgraded expr-lang/expr to v1.17.0 to resolve a high-severity vulnerability (GHSA-93mq-9ffx-83m2).

• Upgraded golang.org/x/net to v0.37.0 to fix a medium-severity vulnerability (GHSA-qxp5-gwg8-xv66).

• Addressed high-severity vulnerabilities in golang-jwt/jwt/v4 and jwt/v5 by upgrading to the latest versions.

• Resolved a medium-severity vulnerability in coraza-waf by upgrading to v3.3.3.

• Fixed OAuth2 library vulnerability (CVE-2025-22868).

1.54.0 — 26^th February

Traceable’s Platform Agent 1.54.0 release provides the following:

Updates

Agent token authentication for TME-based agents

• Introduced the ability to enable authentication using an agent token for TME-based agents.

• The Traceable Platform Agent (TPA) now performs agent token-based authorization, rejecting incoming requests with an invalid traceableai-agent-token header.

• Initial requests may be dropped as authentication occurs asynchronously, a behavior currently implemented only in ext_cap.

Configuration Parameters

• Updated the processing pipeline to use the OpenTelemetry (OTel) connector for managing multiple environments.

Put libtraceable logs under the log directory

• libtraceable logs are now correctly placed under the designated log directory when a custom log path is specified.

• Previously, logs were not being created under the configured directory, leading to issues in environments using mounted log directories for scalability.

Example of the custom install command

./install.sh tpa-only -e ${var.environmentname} -s ${var.servicename} \
--raw-token ${var.token} --no-download --install-dir /tpa/install \
--log-dir /tpa/log --data-dir /tpa/data --otlp-file-storage-dir /tpa/otlp

The --log-dir flag ensures that logs are stored correctly in the designated directory.

Update Agent Attributes Processor

• Implemented new features in the agent attributes processor, enhancing attribute management and processing capabilities.

Security Updates

• Updated dependencies to address Go vulnerabilities (GHSA-29wx-vh33-7x7r), improving the security of the platform agent.

Networking and Load Balancer Improvements

Single Kubernetes Service Mode on Port 5442

• Introduced a single service mode in Kubernetes, allowing Cloud Environments to expose TPA through a single service, reducing excessive port exposure and minimizing the number of listeners on load balancers.

• This setup creates a single listener on the load balancer instead of exposing multiple ports.

Configurable Service Definitions

• The service definition is now configurable, allowing users to specify additional properties like NodePort.

Ability to populate SpanID in blocked messages for req_cap Endpoint in ext_cap

• Added the SpanID in the blocked message for requests blocked by req_cap in ext_cap.

Configuration Update Considerations

• As part of this release, the configuration structure has changed due to the new OTel connector.

• This update is automatically handled when upgrading via Helm, Terraform, or the install script, ensuring a seamless transition.

• Users upgrading manually with an old configuration file may need to adjust their settings to align with the updated structure.

• Direct upgrades using RPM/DPKG packages are discouraged, as they require manual configuration updates to reflect these changes.

1.53.1 — 21^st February

Traceable’s Platform Agent 1.53.1 release provides the following:

Update

• Cluster-wide TPA Naming Enhancement:

  • Introduced a new clusterName value in Helm and a cluster_name variable in Terraform.

  • This configuration ensures that TPA is named as clustername.namespace.deployment, resulting in a unified TPA representation in the UI.

  • Only a single TPA will be shown in the UI for all pods within the same deployment.

This update simplifies the visibility and management of TPAs across deployments.

1.53.0 — 31^st January

Traceable’s Platform Agent 1.53.0 release provides the following:

Updates

• Increased MaxLength for TraceableRefreshToken in CloudFormation Template
  The maximum length for TraceableRefreshToken is now 4096 characters, resolving issues with longer tokens in on-prem installations.

• Pod and Container Security Context Support
  Security context configurations are now supported for both pods and containers, enhancing the security of TPA and sidecar deployments. For more information, see Security Context for Platform agent.

• Topology Spread Constraints, Pod Disruption Budget, and Affinity Enhancements
  New options for pod affinity, anti-affinity, topology spread constraints, and pod disruption budgets provide greater flexibility for Kubernetes deployments. For more information, see Helm and Terraform values.

• Kubernetes Resource Definitions for grpc-to-http Envoy Container
  Added Kubernetes resource configurations for the grpc-to-http envoy container, now available in Helm and Terraform. For more information, see Helm and Terraform values

• Blocked Spans in span_remover Processor
  Blocked spans are no longer sampled, ensuring more efficient span processing.

• Removal of Deprecated Processors
  Simplified the codebase by removing the piifilter and user attribution processors.

• ECS CloudFormation Templates: Configurable Service Discovery
  Made service discovery configurable in ECS CloudFormation templates to prevent conflicts with hosted zones in Route53 during multiple deployments in the same account.