Contents x
    Traffic mirroring for VM
    • 08 Jan 2024
    • 14 Minutes to read
    • PDF
    Contents

    Traffic mirroring for VM

    • Updated on 08 Jan 2024
    • 14 Minutes to read
    • PDF
    Article Summary

    Traceable provides a script for setting up traffic mirroring on a virtual machine. The script installs and runs two services. The first service collects and sends the mirrored traffic to the Traceable Platform agent. The second service, the Traceable Platform agent, sends the mirrored traffic to the Traceable Platform. Both these services run using systemctl. Traceable also supports traffic mirroring in an airgapped environment.

    systemctl is a command-line utility for controlling the systemd system and service manager on Linux operating systems. It provides several commands for managing systemd units, which are the building blocks of a system-based system. systemd is a system and service manager for Linux operating systems. It provides a standard process for controlling what programs run when a Linux system boots up. For more information on systemctl, see How to use systemctl to manage Linux services.

    Download

    To download the script, go to Traceable's download site. Navigate to install → traffic-mirroring → linux → latest. Click on the install.sh script to download it, or enter the following command in your terminal:

    curl -O https://downloads.traceable.ai/install/traffic-mirroring/linux/latest/install.sh

    Provide the execute permission to the script. Enter the following command:

    chmod +x install.sh

    Installation

    The installation script is a utility to install and configure traffic-mirroring components. It can be used as follows:

    sudo ./install.sh Subcommand Arguments [Optional-Arguments]

    The sub-commands for the script are:

    mirror      Uses packet capture for data collection
ebpf        Uses eBPF for data collection
tpa-only    Install Traceable Platform Agent (TPA) only
ebpf-only   Install eBPF only
cleanup     Remove Traceable deployments
help        Prints help

    mirror

    The mirror sub-command installs the Traceable Platform agent and Suricata and runs them as systemctl services. The service you are mirroring traffic from can be encrypted or unencrypted. For more details of this command (synopsis, example usages, argument descriptions), enter the following:

    ./install.sh mirror help

    The mirror command has the following syntax:

    ./install.sh mirror -e ENVIRONMENT -s SERVICE_NAME [-r REMOTE_ENDPOINT]
    [-i INTERFACE | --all-interfaces] [-f FILTER_OPTION] [-t] [--no-download]
    [--otlp-exporter-sending-queue-size OTLP_EXPORTER_SENDING_QUEUE_SIZE]
    [--otlp-file-storage-dir OTLP_FILE_STORAGE_DIR] [--tpa-loglevel TPA_LOGLEVEL]
    [--tpa-log-max-size-mb TPA_LOGSIZE] [--tpa-log-max-backups TPA_LOGCOUNT]
    [--tpa-max-memory TPA_MAX_MEM] [--mirror-max-memory MIRROR_MAX_MEM]
    [--packet-capture-max-memory PACKET_CAPTURE_MAX_MEM]
    [--set-memory-accounting] [--remote-cert-path ROOT_CA_CERT]
    [--enable-tls-capture [--run-non-tls-server | --tls-key KEY --tls-cert CERT]
    [--server-port SERVER_PORT]]

    The following table explains the usage of the command arguments:

    Argument

    Mandatory

    Description

    -s SERVICE_NAME

    Yes

    The service name for the mirrored traffic.

    -e ENVIRONMENT

    Yes

    Environment name for the mirrored traffic.

    -r REMOTE_ENDPOINT

    No

    Remote endpoint for Traceable Platform Agent. If this is not configured, then it defaults to api.traceable.ai.

    -f FILTER_OPTION

    No

    Custom filter option for mirroring. Defaults to "not host VM_IP"

    -t | --token

    No

    Prompt for Traceable token for authentication. Check token for more details.

    --remote-cert-path ROOT_CA_CERT

    No

    Root CA used to sign certificate for the Traceable Platform Agent.

    --no-download

    No

    Use local packages to install Traceable and its components. For more information, see Airgapped installation.

    --otlp-exporter-sending-queue-size SIZE

    No

    OTLP exporter sending queue size (in batches) for traces and metrics. This config is also used for the sending queue size. The default is 1000.

    --otlp-file-storage-dir DIR

    No

    OTLP file storage extension directory. This is to persist the retry queue for traces and metrics. Default value is /tmp.

    --tpa-loglevel LEVEL

    No

    Logging level for the Traceable service. TPA_LOGLEVEL must be one of info, warn, error, and debug. The default value is info.

    --tpa-log-max-size-mb SIZE

    No

    The maximum size in MB for log files created by the Traceable service. The default value is 10.

    --tpa-log-max-backups COUNT

    No

    The maximum number of backup log files created by the Traceable service. The default value is 10.

    --tpa-max-memory MEM

    No

    Maximum memory limit for traceable-agent service. Takes a memory size in bytes. If the value is suffixed with K, M, G, or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, a percentage value may be specified, which is taken relative to the installed physical memory on the system.

    --mirror-max-memory MEM

    No

    Maximum memory limit for mirror service. Takes a memory size in bytes. If the value is suffixed with K, M, G, or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, a percentage value may be specified, which is taken relative to the installed physical memory on the system.

    --packet-capture-max-memory MEM

    No

    Maximum memory limit for packet capture service. Takes memory size in bytes. If the value is suffixed with K, M, G, or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, a percentage value may be specified, which is taken relative to the installed physical memory of the system.

    --set-memory-accounting

    No

    Set the MemoryAccounting value as true for the services. This is required on VMs where DefaultMemoryAccounting property is not enabled.

    Provide exactly one of the following arguments:

    Argument

    Description

    -i INTERFACE

    Comma-separated list of interface IDs for the mirrored traffic.

    --all-interfaces

    Mirror traffic from all the interfaces.

    Note

    This flag is incompatible with the --enable-tls-capture flag.

    Mirroring with SSL/TLS

    If your mirrored service is using SSL/TLS protocol, then in addition, use the following arguments:

    Argument

    Mandatory

    Description

    --enable-tls-capture

    Yes

    Capture traffic encrypted with TLS/SSL.

    Note

    If neither --run-non-tls-server nor --tls-key KEY --tls-cert CERT is provided, then for installation to be successful, the VM should have openssl.

    --run-non-tls-server

    No

    Run Traceable without TLS. By default, the Traceable server uses TLS encryption.

    --tls-key KEY --tls-cert CERT

    No

    KEY is the path to the private key and CERT is the path to the certificate file (X.509 key-cert pair) to be used by the tls-packet-capture module.

    --server-port PORT 

    No

    Port where a TCP server runs when you enable TLS capture. Master keys are sent to this TCP server in a standard format.

    mirroring example

    Following are a few examples of using the mirror sub-command:

    • If your service is running on the eth0 interface, you can use the following command:

      ActionScriptActionScript

      sudo ./install.sh mirror -i eth0 -e myEnvironment -s myService -t

    • If your service is running with TLS, you can use the following command:

      ActionScriptActionScript

      sudo ./install.sh mirror -i eth0 -e myEnvironment -s myService -t --enable-tls-capture

    ebpf

    The ebpf sub-command installs the Traceable Platform agent and eBPF and runs them as systemctl services. For more details of this command (synopsis, example usages, argument descriptions), enter the following:

    ./install.sh ebpf help

    The command has the following syntax:

    ./install.sh ebpf -e ENVIRONMENT -s SERVICE_NAME [-r REMOTE_ENDPOINT]
    [-P PORTS] [-S SSL_DIRS] [-E EXCLUDE_PROCESSES] [--mode MODE]
    [-R MAX_RETURN_PROBES] [--remote-cert-path ROOT_CA_CERT] [-t]
    [--tpa-loglevel TPA_LOGLEVEL] [--tpa-log-max-size-mb TPA_LOGSIZE] 
    [--tpa-log-max-backups TPA_LOGCOUNT] [--ebpf-loglevel EBPF_LOGLEVEL]
    [--ebpf-log-max-size-mb EBPF_LOGSIZE] [--ebpf-log-max-backups EBPF_LOGCOUNT] 
    [--no-download] [--enable-java-tls-capture] [--ebpf-max-memory EBPF_MAX_MEM]
    [--tpa-max-memory TPA_MAX_MEM] [--set-memory-accounting] 
    [--ebpf-request-per-second-limit EBPF_REQUEST_LIMIT] [--ebpf-max-connection EBPF_MAX_CONNECTION] 
    [--ebpf-go-memory-limit EBPF_GO_MEMORY_LIMIT] [--otlp-max-connection-age OTLP_MAX_CONNECTION_AGE]

    The following table explains the usage of the command arguments:

    Argument

    Mandatory

    Description

    -s SERVICE

    Yes

    The service name for the mirrored traffic.

    -e ENVIRONMENT

    Yes

    Environment name for the mirrored traffic.

    -r REMOTE_ENDPOINT

    No

    Remote endpoint for Traceable Platform agent.

    -P PORTS

    No

    Comma-separated list of ports that you wish to track.

    -S SSL_DIRS

    No

    Comma-separated list of SSL directories. 

    -E EXCLUDE_PROCESSES

    No

    Comma-separated list of process names that you wish to exclude from tracking.

    --mode MODE

    No

    Traffic type to be captured. MODE can take the following values:

    • ingress: capture all ingress traffic for given PORTS. 

    • egress: capture all egress traffic for the given PORTS. 

    • ingress_and_egress: capture all ingress as well as egress traffic for given PORTS.

    The default value is ingress.

    -R MAX_RETURN_PROBES 

    No

    Maximum number of return probes.

    -t | --token

    No

    Prompt for Traceable Token for authentication. Check token for more details.

    --remote-cert-path PATH

    No

    Root CA used to sign certificate for the Traceable Platform Agent.

    --tpa-loglevel TAP_LOGLEVEL

    No

    Logging level for the Traceable service. The allowed values are info, warn, error, and debug. The default value is info.

    --tpa-log-max-size-mb TPA_LOGSIZE

    No

    The maximum log file size in MB created by the Traceable service. The default value is 10 MB.

    --tpa-log-max-backups TPA_LOGCOUNT

    No

    Maximum number of backup log files created by the Traceable service. The default value is 10.

    --ebpf-loglevel EBPF_LOGLEVEL

    No

    Logging level for the eBPF service. The allowed values are info, warn, error, debug and trace. The default value is info.

    --ebpf-log-max-size-mb EBPF_LOGSIZE

    No

    Maximum log file size in MB created by the eBPF service. The default value is 10 MB.

    --ebpf-log-max-backups EBPF_LOGCOUNT

    No

    Maximum number of backup log files created by the eBPF service. The default value is 10.

    --no-download

    No

    Use local packages to install Traceable and its components. For more information, see Airgapped installation.

    --enable-java-tls-capture

    No

    Set it to true, if you wish to capture Java TLS traffic. The default value is false.

    --ebpf-max-memory EBPF_MAX_MEM

    No

    Maximum memory limit for the eBPF service. Takes a memory size in bytes. If the value is suffixed with K, M, G, or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, a percentage value may be specified, which is taken relative to the installed physical memory on the system.

    --tpa-max-memory TPA_MAX_MEM

    No

    Maximum memory limit for the Traceable service. Takes memory size in bytes. If the value is suffixed with K, M, G, or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, you can select the value as a percentage, which is taken relative to the installed physical memory on the system.

    --set-memory-accounting

    No

    Set MemoryAccounting to true for the services. This is required on VMs where DefaultMemoryAccounting property is not enabled.

    --ebpf-request-per-second-limit

    No

    Sets the maximum number of requests the eBPF tracer would capture in a second.

    --ebpf-max-connection

    No

    Sets the maximum number of connections the eBPF tracer tracks at any given time.

    --ebpf-go-memory-limit

    No

    This is the substitute for the GOMEMLIMIT environment variable. The argument expects input in k8 units. The default value is 0, which translates to no limit being set.

    Examples: 128974848, 129e6, 129M, 128974848000m, 123Mi

    --otlp-max-connection-age

    No

    Sets the keepalive duration for the OTLP server in the Traceable Platform agent. The duration can be expressed as a string. The acceptable units are ns, us, s, m, and h.

    Example: 100ms = 100 milliseconds, 120s = 120 seconds,
    120m = 120 minutes, 2h = 2 hours. The default value is infinity.

    ebpf examples

    Following are a few examples of using the ebpf sub-command:

    • If you wish to capture all the traffic, use the following command:

      ActionScriptActionScript

      sudo ./install.sh ebpf -e myEnvironment -s myService -t

    • If you wish to capture traffic from a specific process or a set of processes, use the following command. In the example below, data is captured from ports 8080 and 9001.

      ActionScriptActionScript

      sudo ./install.sh ebpf -e myEnvironment -s myService -P 8080,9001 -t

    ebpf-only

    The ebpf-only sub-command installs only the eBPF solution and runs them as systemctl services. For more details of this command (synopsis, example usages, argument descriptions), enter the following:

    ./install.sh ebpf-only help

    The command has the following syntax:

    ./install.sh ebpf-only -s SERVICE_NAME --tpa-endpoint ENDPOINT [-P PORTS]
    [-S SSL_DIRS] [-E EXCLUDE_PROCESSES] [-R MAX_RET_PROBES] [--mode MODE]
    [--tls-root-cert ROOT_CA_CERT] [--reporting-port REPORTING_PORT]
    [--remote-port REMOTE_PORT] [--exporter EXPORTER] [--loglevel LOGLEVEL]
    [--log-max-size-mb SIZE] [--log-max-backups COUNT] 
    [--no-download] [--enable-java-tls-capture] [--max-memory MAX_MEMORY]
    [--set-memory-accounting] [--request-per-second-limit REQUEST_LIMIT] [--max-connection MAX_CONNECTION]
    [--go-memory-limit GO_MEMORY_LIMIT]

    The following table explains the usage of the command arguments:

    Argument

    Mandatory

    Description

    -s SERVICE_NAME

    Yes

    The service name for the mirrored traffic.

    --tpa-endpoint ENDPOINT

    Yes

    Endpoint of Traceable Platform Agent. 

    -P PORTS

    No

    Comma-separated list of ports that you wish to track.

    -S SSL_DIRS 

    No

    Comma-separated list of SSL directories. 

    -E EXCLUDE_PROCESSES

    No

    Comma-separated list of process names to be excluded from tracking.

     -R MAX_RETURN_PROBES

    No

    Maximum number of return probes. The default value is 1.

    --mode MODE

    No

    Traffic type to be captured. MODE can take the following values:

    • ingress: capture all ingress traffic for given PORTS. 

    • egress: capture all egress traffic for the given PORTS. 

    • ingress_and_egress: capture all ingress as well as egress traffic for given PORTS.

    The default value is ingress.

    --tls-root-cert ROOT_CA_CERT

    No

    Root CA used to sign certificate for Traceable Platform Agent. eBPF uses this to validate the Traceable Platform Agent's certificate. If this value is provided, eBPF uses TLS encryption for data sent to the Traceable Platform Agent.

    --reporting-port REPORTING_PORT

    No

    Reporting endpoint port of Traceable Platform Agent. If --tls-root-cert is provided, the default value is 5443. Otherwise, the default value is 4317.

    --remote-port REMOTE_PORT

    No

    Remote endpoint port of Traceable Platform Agent. If --tls-root-cert is provided, the default value is 5443. Otherwise, the default value is 5441.

    --exporter EXPORTER

    No

    eBPF span exporter. Allowed values are ZIPKIN and OTLP. The default exporter is OTLP.

    --loglevel LOGLEVEL

    No

    Logging level for the eBPF service. The allowed values are info, warn, error, debug and trace. The default value is info.

    --log-max-size-mb SIZE

    No

    Maximum log file size in MB created by the eBPF service. The default value is 10 MB.

    --log-max-backups COUNT

    No

    Maximum number of backup log files created by the eBPF service. The default value is 10.

    --no-download

    No

    Use local packages to install Traceable and its components. For more information, see Airgapped installation.

    --enable-java-tls-capture

    No

    Set it to true, if you wish to capture Java TLS traffic. The default value is false.

    --max-memory MAX_MEMORY

    No

    Maximum memory limit for the Traceable service. Takes memory size in bytes. If the value is suffixed with K, M, G, or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, you can select the value as a percentage, which is taken relative to the installed physical memory on the system.

    --set-memory-accounting

    No

    set MemoryAccounting to true for the eBPF service. This is required on VMs where the DefaultMemoryAccounting property is not enabled.

    --request-per-second-limit

    No

    Sets the maximum number of requests the eBPF tracer would capture in a second.

    --max-connection

    No

    Sets the maximum number of connections the eBPF tracer tracks at any given time.

    --go-memory-limit

    No

    This is the substitute for the GOMEMLIMIT environment variable. The argument expects input in k8 units. The default value is 0, which translates to no limit being set.

    Examples: 128974848, 129e6, 129M, 128974848000m, 123Mi

    ebpf-only examples

    Following are a few examples of using the ebpf-only sub-command:

    • If your Traceable Platform agent is running on an IP address 10.120.0.3, and you wish to install the eBPF solution and send the mirrored traffic to the Traceable Platform agent, use the following command:

      ActionScriptActionScript

      sudo ./install.sh ebpf-only -s myService --tpa-endpoint 10.120.0.3

    • If your Traceable Platform agent is running with the TLS server on 10.120.0.3 and you have your root CA at /path/to/root/ca, then you can enable TLS encryption for the traffic sent from eBPF to the Traceable Platform agent. Enter the following command:

      ActionScriptActionScript

      sudo ./install.sh ebpf-only -s myService --tpa-endpoint 10.120.0.3 --tls-root-cert /path/to/root/ca

    tpa-only

    The tpa-only sub-command installs the Traceable Platform agent and runs it as a systemctl service. For more details of this command (synopsis, example usages, argument descriptions), enter the following:

    ./install.sh tpa-only help

    The command has the following syntax:

    ./install.sh tpa-only -e ENVIRONMENT [-s SERVICE_NAME] [-r REMOTE_ENDPOINT]
    [--loglevel LOGLEVEL] [--remote-cert-path ROOT_CA_CERT] [-t]
    [--log-max-size-mb SIZE] [--log-max-backups COUNT] [--no-download]
    [--tls-endpoint ENDPOINT --tls-key KEY --tls-cert CERT --tls-root-cert ROOT_CA_CERT]
    [--max-memory MAX_MEMORY] [--set-memory-accounting] 
    [--otlp-max-connection-age OTLP_MAX_CONNECTION_AGE]

    The following table explains the usage of the command arguments:

    Argument

    Mandatory

    Description

    -e ENVIRONMENT

    Yes

    Environment name for the mirrored traffic.

    -s SERVICE_NAME

    No

    The service name for the mirrored traffic.

    -r REMOTE_ENDPOINT

    No

    Remote endpoint for Traceable Platform agent. 

    --loglevel LOGLEVEL

    No

    Logging level for the Traceable service. The allowed values are info, warn, error, and debug. The default value is info.

    --remote-cert-path ROOT_CA_CERT

    No

    Root CA used to sign certificate for the Traceable Platform Agent.

    -t | --token

    No

    Prompt for Traceable token for authentication. Check token for more details.

    --log-max-size-mb SIZE

    No

    Maximum log file size in MB created by the Traceable service. The default value is 10 MB.

    --log-max-backups COUNT

    No

    Maximum number of backup log files created by the eBPF service. The default value is 10.

    --no-download

    No

    Use local packages to install Traceable and its components. For more information, see Airgapped installation.

    --max-memory MAX_MEMORY

    No

    Maximum memory limit for the Traceable service. Takes a memory size in bytes. If the value is suffixed with K, M, G, or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, a percentage value may be specified, which is taken relative to the installed physical memory on the system.

    --set-memory-accounting

    No

    set MemoryAccounting to true for the Traceable service. This is required on VMs where the DefaultMemoryAccounting property is not enabled.

    --otlp-max-connection-age

    No

    Sets the keepalive duration for the OTLP server in the Traceable Platform agent. The duration can be expressed as a string. The acceptable units are ns, us, s, m, and h.

    Example: 100ms = 100 milliseconds, 120s = 120 seconds,
    120m = 120 minutes, 2h = 2 hours. The default value is infinity.

    Traceable Platform agent with TLS

    You can choose to enable the TLS server on the Traceable Platform agent. The following arguments enable TLS:

    Argument

    Mandatory

    Description

    --tls-endpoint ENDPOINT

    Yes

    TLS Endpoint of Traceable Platform agent.

    --tls-key KEY

    Yes

    Path to the private key for Traceable Platform agent.

    --tls-cert CERT

    Yes

    Path to the certificate file for Traceable Platform agent.

    --tls-root-cert ROOT_CA_CERT

    Yes

    Path to the root CA file for Traceable Platform agent. 

    tpa-only examples

    Following are a few examples of using the tpa-only sub-command:

    • If you wish to install Traceable Platform Agent without a TLS server, enter the following command:

      ActionScriptActionScript

      sudo ./install.sh tpa-only -e myEnvironment -t

    • If you wish to set up a TLS server with an endpoint as 0.0.0.0:5443, enter the following command:

      ActionScriptActionScript

      sudo ./install.sh tpa-only -e myEnvironment -t --tls-endpoint 0.0.0.0:5443 --tls-key /path/to/key --tls-cert /path/to/cert --tls-root-cert /path/to/root/ca

    Uninstallation

    Enter the following command to uninstall. Note that the install script is used for both installation and uninstallation.

    sudo ./install.sh cleanup

    eBPF with centralized Traceable Platform agent

    When you use the ebpf subcommand for installation, the eBPF and Traceable Platform Agent are deployed on the same machine. eBPF starts capturing traffic and sends it to the agent. To streamline the forwarding process, you may run a central agent to which all devices with eBPF can send mirrored traffic. This can be accomplished in two simple steps:

    1. Run the installation script with the tpa-only subcommand on the virtual machine (VM) where you wish to install the central agent.

    2. Run the installation script with the ebpf-only subcommand on all other VMs hosting your services.

    If you wish to ensure secure communication between the VM capturing traffic and the central agent, follow these additional steps:

    1. Enable the TLS server during installation with the tpa-only subcommand.

    2. Use the same root CA certificate for all ebpf-only installations.

    Airgapped installation

    Traceable provides a script for setting up traffic mirroring in an air-gapped environment. An air-gapped system is a computer or network physically isolated from the internet and other unsecured networks. This system protects sensitive data and systems from external threats such as hackers and malware. Airgapped systems are often used in industries where data security is paramount.

    Traceable provides two archives for air-gapped installation, one for AMD64 machines and the other for arm64 devices. To download the archive, go to Traceable's download site. Navigate to install → traffic-mirroring → linux → latest. 

    Transfer the downloaded archive to your airgapped system. Untar the archive by entering the following command, for example, for an amd64 machine:

    tar xvzf traffic-mirroring-amd64.tar.gz

    Change directory:

    cd traffic-mirroring-amd64/bin/

    The bin directory contains install.sh.

    To install, run any sub-command mentioned in the Installation section by adding the --no-download option. This option uses the local packages from the tar file to complete the installation instead of depending on the network.

    Token

    When installing the Traceable Platform Agent, you must provide the platform token to authenticate with the platform. This can be done in multiple ways.

    • Export the token as an environment variable:

      ActionScriptActionScript

      export TA_REFRESH_TOKEN=""

    • Put the token in the default token file:

      ActionScriptActionScript

      echo "" | sudo tee /etc/traceable/agent/token

    • Provide the command-line flag -t or --token and enter the token when prompted.

    Was this article helpful?
    What's Next
    ©2023 Traceable Inc
    terms of services
    privacy policy
    oss disclosures