Traffic mirroring for VM
  • 08 Jan 2024
  • 14 Minutes to read
  • PDF

Traffic mirroring for VM

  • PDF

Article summary

Traceable provides a script for setting up traffic mirroring on a virtual machine. The script installs and runs two services. The first service collects and sends the mirrored traffic to the Traceable Platform agent. The second service, the Traceable Platform agent, sends the mirrored traffic to the Traceable Platform. Both these services run using systemctl. Traceable also supports traffic mirroring in an airgapped environment.

systemctl is a command-line utility for controlling the systemd system and service manager on Linux operating systems. It provides several commands for managing systemd units, which are the building blocks of a system-based system. systemd is a system and service manager for Linux operating systems. It provides a standard process for controlling what programs run when a Linux system boots up. For more information on systemctl, see How to use systemctl to manage Linux services.


Download

To download the script, go to Traceable's download site. Navigate to install → traffic-mirroring → linux → latest. Click on the install.sh script to download it, or enter the following command in your terminal:

curl -O https://downloads.traceable.ai/install/traffic-mirroring/linux/latest/install.sh

Provide the execute permission to the script. Enter the following command:

chmod +x install.sh

Installation

The installation script is a utility to install and configure traffic-mirroring components. It can be used as follows:

sudo ./install.sh Subcommand Arguments [Optional-Arguments]

The sub-commands for the script are:

mirror      Uses packet capture for data collection
ebpf        Uses eBPF for data collection
tpa-only    Install Traceable Platform Agent (TPA) only
ebpf-only   Install eBPF only
cleanup     Remove Traceable deployments
help        Prints help

mirror

The mirror sub-command installs the Traceable Platform agent and Suricata and runs them as systemctl services. The service you are mirroring traffic from can be encrypted or unencrypted. For more details of this command (synopsis, example usages, argument descriptions), enter the following:

./install.sh mirror help

The mirror command has the following syntax:

./install.sh mirror -e ENVIRONMENT -s SERVICE_NAME [-r REMOTE_ENDPOINT]
    [-i INTERFACE | --all-interfaces] [-f FILTER_OPTION] [-t] [--no-download]
    [--otlp-exporter-sending-queue-size OTLP_EXPORTER_SENDING_QUEUE_SIZE]
    [--otlp-file-storage-dir OTLP_FILE_STORAGE_DIR] [--tpa-loglevel TPA_LOGLEVEL]
    [--tpa-log-max-size-mb TPA_LOGSIZE] [--tpa-log-max-backups TPA_LOGCOUNT]
    [--tpa-max-memory TPA_MAX_MEM] [--mirror-max-memory MIRROR_MAX_MEM]
    [--packet-capture-max-memory PACKET_CAPTURE_MAX_MEM]
    [--set-memory-accounting] [--remote-cert-path ROOT_CA_CERT]
    [--enable-tls-capture [--run-non-tls-server | --tls-key KEY --tls-cert CERT]
    [--server-port SERVER_PORT]]

The following table explains the usage of the command arguments:

Argument

Mandatory

Description

-s SERVICE_NAME

Yes

The service name for the mirrored traffic.

-e ENVIRONMENT

Yes

Environment name for the mirrored traffic.

-r REMOTE_ENDPOINT

No

Remote endpoint for Traceable Platform Agent. If this is not configured, then it defaults to api.traceable.ai.

-f FILTER_OPTION

No

Custom filter option for mirroring. Defaults to "not host VM_IP"

-t | --token

No

Prompt for Traceable token for authentication. Check token for more details.

--remote-cert-path ROOT_CA_CERT

No

Root CA used to sign certificate for the Traceable Platform Agent.

--no-download

No

Use local packages to install Traceable and its components. For more information, see Airgapped installation.

--otlp-exporter-sending-queue-size SIZE

No

OTLP exporter sending queue size (in batches) for traces and metrics. This config is also used for the sending queue size. The default is 1000.

--otlp-file-storage-dir DIR

No

OTLP file storage extension directory. This is to persist the retry queue for traces and metrics. Default value is /tmp.

--tpa-loglevel LEVEL

No

Logging level for the Traceable service. TPA_LOGLEVEL must be one of info, warn, error, and debug. The default value is info.

--tpa-log-max-size-mb SIZE

No

The maximum size in MB for log files created by the Traceable service. The default value is 10.

--tpa-log-max-backups COUNT

No

The maximum number of backup log files created by the Traceable service. The default value is 10.

--tpa-max-memory MEM

No

Maximum memory limit for traceable-agent service. Takes a memory size in bytes. If the value is suffixed with K, M, G, or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, a percentage value may be specified, which is taken relative to the installed physical memory on the system.

--mirror-max-memory MEM

No

Maximum memory limit for mirror service. Takes a memory size in bytes. If the value is suffixed with K, M, G, or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, a percentage value may be specified, which is taken relative to the installed physical memory on the system.

--packet-capture-max-memory MEM

No

Maximum memory limit for packet capture service. Takes memory size in bytes. If the value is suffixed with K, M, G, or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, a percentage value may be specified, which is taken relative to the installed physical memory of the system.

--set-memory-accounting

No

Set the MemoryAccounting value as true for the services. This is required on VMs where DefaultMemoryAccounting property is not enabled.

Provide exactly one of the following arguments:

Argument

Description

-i INTERFACE

Comma-separated list of interface IDs for the mirrored traffic.

--all-interfaces

Mirror traffic from all the interfaces.

Note

This flag is incompatible with the --enable-tls-capture flag.

Mirroring with SSL/TLS

If your mirrored service is using SSL/TLS protocol, then in addition, use the following arguments:

Argument

Mandatory

Description

--enable-tls-capture

Yes

Capture traffic encrypted with TLS/SSL.

Note

If neither --run-non-tls-server nor --tls-key KEY --tls-cert CERT is provided, then for installation to be successful, the VM should have openssl.

--run-non-tls-server

No

Run Traceable without TLS. By default, the Traceable server uses TLS encryption.

--tls-key KEY --tls-cert CERT

No

KEY is the path to the private key and CERT is the path to the certificate file (X.509 key-cert pair) to be used by the tls-packet-capture module.

--server-port PORT 

No

Port where a TCP server runs when you enable TLS capture. Master keys are sent to this TCP server in a standard format.

mirroring example

Following are a few examples of using the mirror sub-command:

  • If your service is running on the eth0 interface, you can use the following command:

    ActionScriptActionScript

    sudo ./install.sh mirror -i eth0 -e myEnvironment -s myService -t
  • If your service is running with TLS, you can use the following command:

    ActionScriptActionScript

    sudo ./install.sh mirror -i eth0 -e myEnvironment -s myService -t --enable-tls-capture

ebpf

The ebpf sub-command installs the Traceable Platform agent and eBPF and runs them as systemctl services. For more details of this command (synopsis, example usages, argument descriptions), enter the following:

./install.sh ebpf help

The command has the following syntax:

./install.sh ebpf -e ENVIRONMENT -s SERVICE_NAME [-r REMOTE_ENDPOINT]
    [-P PORTS] [-S SSL_DIRS] [-E EXCLUDE_PROCESSES] [--mode MODE]
    [-R MAX_RETURN_PROBES] [--remote-cert-path ROOT_CA_CERT] [-t]
    [--tpa-loglevel TPA_LOGLEVEL] [--tpa-log-max-size-mb TPA_LOGSIZE] 
    [--tpa-log-max-backups TPA_LOGCOUNT] [--ebpf-loglevel EBPF_LOGLEVEL]
    [--ebpf-log-max-size-mb EBPF_LOGSIZE] [--ebpf-log-max-backups EBPF_LOGCOUNT] 
    [--no-download] [--enable-java-tls-capture] [--ebpf-max-memory EBPF_MAX_MEM]
    [--tpa-max-memory TPA_MAX_MEM] [--set-memory-accounting] 
    [--ebpf-request-per-second-limit EBPF_REQUEST_LIMIT] [--ebpf-max-connection EBPF_MAX_CONNECTION] 
    [--ebpf-go-memory-limit EBPF_GO_MEMORY_LIMIT] [--otlp-max-connection-age OTLP_MAX_CONNECTION_AGE]

The following table explains the usage of the command arguments:

Argument

Mandatory

Description

-s SERVICE

Yes

The service name for the mirrored traffic.

-e ENVIRONMENT

Yes

Environment name for the mirrored traffic.

-r REMOTE_ENDPOINT

No

Remote endpoint for Traceable Platform agent.

-P PORTS

No

Comma-separated list of ports that you wish to track.

-S SSL_DIRS

No

Comma-separated list of SSL directories. 

-E EXCLUDE_PROCESSES

No

Comma-separated list of process names that you wish to exclude from tracking.

--mode MODE

No

Traffic type to be captured. MODE can take the following values:

  • ingress: capture all ingress traffic for given PORTS

  • egress: capture all egress traffic for the given PORTS

  • ingress_and_egress: capture all ingress as well as egress traffic for given PORTS.

The default value is ingress.

-R MAX_RETURN_PROBES 

No

Maximum number of return probes.

-t | --token

No

Prompt for Traceable Token for authentication. Check token for more details.

--remote-cert-path PATH

No

Root CA used to sign certificate for the Traceable Platform Agent.

--tpa-loglevel TAP_LOGLEVEL

No

Logging level for the Traceable service. The allowed values are info, warn, error, and debug. The default value is info.

--tpa-log-max-size-mb TPA_LOGSIZE

No

The maximum log file size in MB created by the Traceable service. The default value is 10 MB.

--tpa-log-max-backups TPA_LOGCOUNT

No

Maximum number of backup log files created by the Traceable service. The default value is 10.

--ebpf-loglevel EBPF_LOGLEVEL

No

Logging level for the eBPF service. The allowed values are info, warn, error, debug and trace. The default value is info.

--ebpf-log-max-size-mb EBPF_LOGSIZE

No

Maximum log file size in MB created by the eBPF service. The default value is 10 MB.

--ebpf-log-max-backups EBPF_LOGCOUNT

No

Maximum number of backup log files created by the eBPF service. The default value is 10.

--no-download

No

Use local packages to install Traceable and its components. For more information, see Airgapped installation.

--enable-java-tls-capture

No

Set it to true, if you wish to capture Java TLS traffic. The default value is false.

--ebpf-max-memory EBPF_MAX_MEM

No

Maximum memory limit for the eBPF service. Takes a memory size in bytes. If the value is suffixed with K, M, G, or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, a percentage value may be specified, which is taken relative to the installed physical memory on the system.

--tpa-max-memory TPA_MAX_MEM

No

Maximum memory limit for the Traceable service. Takes memory size in bytes. If the value is suffixed with K, M, G, or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, you can select the value as a percentage, which is taken relative to the installed physical memory on the system.

--set-memory-accounting

No

Set MemoryAccounting to true for the services. This is required on VMs where DefaultMemoryAccounting property is not enabled.

--ebpf-request-per-second-limit

No

Sets the maximum number of requests the eBPF tracer would capture in a second.

--ebpf-max-connection

No

Sets the maximum number of connections the eBPF tracer tracks at any given time.

--ebpf-go-memory-limit

No

This is the substitute for the GOMEMLIMIT environment variable. The argument expects input in k8 units. The default value is 0, which translates to no limit being set.

Examples: 128974848, 129e6, 129M, 128974848000m, 123Mi

--otlp-max-connection-age

No

Sets the keepalive duration for the OTLP server in the Traceable Platform agent. The duration can be expressed as a string. The acceptable units are ns, us, s, m, and h.

Example: 100ms = 100 milliseconds, 120s = 120 seconds,
120m = 120 minutes, 2h = 2 hours. The default value is infinity.

ebpf examples

Following are a few examples of using the ebpf sub-command:

  • If you wish to capture all the traffic, use the following command:

    ActionScriptActionScript

    sudo ./install.sh ebpf -e myEnvironment -s myService -t
  • If you wish to capture traffic from a specific process or a set of processes, use the following command. In the example below, data is captured from ports 8080 and 9001.

    ActionScriptActionScript

    sudo ./install.sh ebpf -e myEnvironment -s myService -P 8080,9001 -t

ebpf-only

The ebpf-only sub-command installs only the eBPF solution and runs them as systemctl services. For more details of this command (synopsis, example usages, argument descriptions), enter the following:

./install.sh ebpf-only help

The command has the following syntax:

./install.sh ebpf-only -s SERVICE_NAME --tpa-endpoint ENDPOINT [-P PORTS]
    [-S SSL_DIRS] [-E EXCLUDE_PROCESSES] [-R MAX_RET_PROBES] [--mode MODE]
    [--tls-root-cert ROOT_CA_CERT] [--reporting-port REPORTING_PORT]
    [--remote-port REMOTE_PORT] [--exporter EXPORTER] [--loglevel LOGLEVEL]
    [--log-max-size-mb SIZE] [--log-max-backups COUNT] 
    [--no-download] [--enable-java-tls-capture] [--max-memory MAX_MEMORY]
    [--set-memory-accounting] [--request-per-second-limit REQUEST_LIMIT] [--max-connection MAX_CONNECTION]
    [--go-memory-limit GO_MEMORY_LIMIT]

The following table explains the usage of the command arguments:

Argument

Mandatory

Description

-s SERVICE_NAME

Yes

The service name for the mirrored traffic.

--tpa-endpoint ENDPOINT

Yes

Endpoint of Traceable Platform Agent. 

-P PORTS

No

Comma-separated list of ports that you wish to track.

-S SSL_DIRS 

No

Comma-separated list of SSL directories. 

-E EXCLUDE_PROCESSES

No

Comma-separated list of process names to be excluded from tracking.

 -R MAX_RETURN_PROBES

No

Maximum number of return probes. The default value is 1.

--mode MODE

No

Traffic type to be captured. MODE can take the following values:

  • ingress: capture all ingress traffic for given PORTS

  • egress: capture all egress traffic for the given PORTS

  • ingress_and_egress: capture all ingress as well as egress traffic for given PORTS.

The default value is ingress.

--tls-root-cert ROOT_CA_CERT

No

Root CA used to sign certificate for Traceable Platform Agent. eBPF uses this to validate the Traceable Platform Agent's certificate. If this value is provided, eBPF uses TLS encryption for data sent to the Traceable Platform Agent.

--reporting-port REPORTING_PORT

No

Reporting endpoint port of Traceable Platform Agent. If --tls-root-cert is provided, the default value is 5443. Otherwise, the default value is 4317.

--remote-port REMOTE_PORT

No

Remote endpoint port of Traceable Platform Agent. If --tls-root-cert is provided, the default value is 5443. Otherwise, the default value is 5441.

--exporter EXPORTER

No

eBPF span exporter. Allowed values are ZIPKIN and OTLP. The default exporter is OTLP.

--loglevel LOGLEVEL

No

Logging level for the eBPF service. The allowed values are info, warn, error, debug and trace. The default value is info.

--log-max-size-mb SIZE

No

Maximum log file size in MB created by the eBPF service. The default value is 10 MB.

--log-max-backups COUNT

No

Maximum number of backup log files created by the eBPF service. The default value is 10.

--no-download

No

Use local packages to install Traceable and its components. For more information, see Airgapped installation.

--enable-java-tls-capture

No

Set it to true, if you wish to capture Java TLS traffic. The default value is false.

--max-memory MAX_MEMORY

No

Maximum memory limit for the Traceable service. Takes memory size in bytes. If the value is suffixed with K, M, G, or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, you can select the value as a percentage, which is taken relative to the installed physical memory on the system.

--set-memory-accounting

No

set MemoryAccounting to true for the eBPF service. This is required on VMs where the DefaultMemoryAccounting property is not enabled.

--request-per-second-limit

No

Sets the maximum number of requests the eBPF tracer would capture in a second.

--max-connection

No

Sets the maximum number of connections the eBPF tracer tracks at any given time.

--go-memory-limit

No

This is the substitute for the GOMEMLIMIT environment variable. The argument expects input in k8 units. The default value is 0, which translates to no limit being set.

Examples: 128974848, 129e6, 129M, 128974848000m, 123Mi

ebpf-only examples

Following are a few examples of using the ebpf-only sub-command:

  • If your Traceable Platform agent is running on an IP address 10.120.0.3, and you wish to install the eBPF solution and send the mirrored traffic to the Traceable Platform agent, use the following command:

    ActionScriptActionScript

    sudo ./install.sh ebpf-only -s myService --tpa-endpoint 10.120.0.3
  • If your Traceable Platform agent is running with the TLS server on 10.120.0.3 and you have your root CA at /path/to/root/ca, then you can enable TLS encryption for the traffic sent from eBPF to the Traceable Platform agent. Enter the following command:

    ActionScriptActionScript

    sudo ./install.sh ebpf-only -s myService --tpa-endpoint 10.120.0.3 --tls-root-cert /path/to/root/ca

tpa-only

The tpa-only sub-command installs the Traceable Platform agent and runs it as a systemctl service. For more details of this command (synopsis, example usages, argument descriptions), enter the following:

./install.sh tpa-only help

The command has the following syntax:

./install.sh tpa-only -e ENVIRONMENT [-s SERVICE_NAME] [-r REMOTE_ENDPOINT]
    [--loglevel LOGLEVEL] [--remote-cert-path ROOT_CA_CERT] [-t]
    [--log-max-size-mb SIZE] [--log-max-backups COUNT] [--no-download]
    [--tls-endpoint ENDPOINT --tls-key KEY --tls-cert CERT --tls-root-cert ROOT_CA_CERT]
    [--max-memory MAX_MEMORY] [--set-memory-accounting] 
    [--otlp-max-connection-age OTLP_MAX_CONNECTION_AGE]

The following table explains the usage of the command arguments:

Argument

Mandatory

Description

-e ENVIRONMENT

Yes

Environment name for the mirrored traffic.

-s SERVICE_NAME

No

The service name for the mirrored traffic.

-r REMOTE_ENDPOINT

No

Remote endpoint for Traceable Platform agent. 

--loglevel LOGLEVEL

No

Logging level for the Traceable service. The allowed values are info, warn, error, and debug. The default value is info.

--remote-cert-path ROOT_CA_CERT

No

Root CA used to sign certificate for the Traceable Platform Agent.

-t | --token

No

Prompt for Traceable token for authentication. Check token for more details.

--log-max-size-mb SIZE

No

Maximum log file size in MB created by the Traceable service. The default value is 10 MB.

--log-max-backups COUNT

No

Maximum number of backup log files created by the eBPF service. The default value is 10.

--no-download

No

Use local packages to install Traceable and its components. For more information, see Airgapped installation.

--max-memory MAX_MEMORY

No

Maximum memory limit for the Traceable service. Takes a memory size in bytes. If the value is suffixed with K, M, G, or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, a percentage value may be specified, which is taken relative to the installed physical memory on the system.

--set-memory-accounting

No

set MemoryAccounting to true for the Traceable service. This is required on VMs where the DefaultMemoryAccounting property is not enabled.

--otlp-max-connection-age

No

Sets the keepalive duration for the OTLP server in the Traceable Platform agent. The duration can be expressed as a string. The acceptable units are ns, us, s, m, and h.

Example: 100ms = 100 milliseconds, 120s = 120 seconds,
120m = 120 minutes, 2h = 2 hours. The default value is infinity.

Traceable Platform agent with TLS

You can choose to enable the TLS server on the Traceable Platform agent. The following arguments enable TLS:

Argument

Mandatory

Description

--tls-endpoint ENDPOINT

Yes

TLS Endpoint of Traceable Platform agent.

--tls-key KEY

Yes

Path to the private key for Traceable Platform agent.

--tls-cert CERT

Yes

Path to the certificate file for Traceable Platform agent.

--tls-root-cert ROOT_CA_CERT

Yes

Path to the root CA file for Traceable Platform agent. 

tpa-only examples

Following are a few examples of using the tpa-only sub-command:

  • If you wish to install Traceable Platform Agent without a TLS server, enter the following command:

    ActionScriptActionScript

    sudo ./install.sh tpa-only -e myEnvironment -t
  • If you wish to set up a TLS server with an endpoint as 0.0.0.0:5443, enter the following command:

    ActionScriptActionScript

    sudo ./install.sh tpa-only -e myEnvironment -t --tls-endpoint 0.0.0.0:5443 --tls-key /path/to/key --tls-cert /path/to/cert --tls-root-cert /path/to/root/ca

Uninstallation

Enter the following command to uninstall. Note that the install script is used for both installation and uninstallation.

sudo ./install.sh cleanup

eBPF with centralized Traceable Platform agent

When you use the ebpf subcommand for installation, the eBPF and Traceable Platform Agent are deployed on the same machine. eBPF starts capturing traffic and sends it to the agent. To streamline the forwarding process, you may run a central agent to which all devices with eBPF can send mirrored traffic. This can be accomplished in two simple steps:

  1. Run the installation script with the tpa-only subcommand on the virtual machine (VM) where you wish to install the central agent.

  2. Run the installation script with the ebpf-only subcommand on all other VMs hosting your services.

If you wish to ensure secure communication between the VM capturing traffic and the central agent, follow these additional steps:

  1. Enable the TLS server during installation with the tpa-only subcommand.

  2. Use the same root CA certificate for all ebpf-only installations.


Airgapped installation

Traceable provides a script for setting up traffic mirroring in an air-gapped environment. An air-gapped system is a computer or network physically isolated from the internet and other unsecured networks. This system protects sensitive data and systems from external threats such as hackers and malware. Airgapped systems are often used in industries where data security is paramount.

Traceable provides two archives for air-gapped installation, one for AMD64 machines and the other for arm64 devices. To download the archive, go to Traceable's download site. Navigate to install → traffic-mirroring → linux → latest. 

Transfer the downloaded archive to your airgapped system. Untar the archive by entering the following command, for example, for an amd64 machine:

tar xvzf traffic-mirroring-amd64.tar.gz

Change directory:

cd traffic-mirroring-amd64/bin/

The bin directory contains install.sh.

To install, run any sub-command mentioned in the Installation section by adding the --no-download option. This option uses the local packages from the tar file to complete the installation instead of depending on the network.


Token

When installing the Traceable Platform Agent, you must provide the platform token to authenticate with the platform. This can be done in multiple ways.

  • Export the token as an environment variable:

    ActionScriptActionScript

    export TA_REFRESH_TOKEN=""
  • Put the token in the default token file:

    ActionScriptActionScript

    echo "" | sudo tee /etc/traceable/agent/token
  • Provide the command-line flag -t or --token and enter the token when prompted.


Was this article helpful?

What's Next
ESC

Eddy, a generative AI, facilitating knowledge discovery through conversational intelligence