---
title: "Data collection with log analytics"
slug: "data-collection-with-log-analytics"
updated: 2024-07-26T10:53:09Z
published: 2024-07-26T10:53:09Z
---

> ## Documentation Index
> Fetch the complete documentation index at: https://traceabledocs.document360.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Data collection with log analytics

**Log Analytics** in Azure APIM is a tool for gathering and examining detailed logs and metrics from your APIs. It helps you monitor, troubleshoot, and optimize your APIs effectively. Traceable uses Log Analytics to collect data from your APIs, which is sent to the Traceable Platform for further analysis.

This topic outlines the steps for configuring data capture for Azure API Management (APIM) using Log Analytics. The process involves creating a Storage Account, enabling diagnostic logs on APIM, setting up a Log Analytics export rule, and creating an Azure Function to handle the log data.

The following is a high-level data flow diagram for data collection using log analytics.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_deployment(1).png)

- **APIM Gateway**: Collects and sends logs.
- **Diagnostic Logs**: Stores logs before forwarding them to a Storage Account.
- **Storage Account**: Receives logs from Diagnostic Logs and triggers the Azure Function.
- **Azure Function**: Processes logs and sends data to the Traceable Platform agent.
- **Traceable Platform agent**: Consumes the data.
- **Log Analytics Workspace**: Stores and allows analysis of the processed data.

---

## Before you begin

Make a note of the following points before you start with the configuration:

- Make sure that the Traceable Platform agent is deployed. Note the platform agent's IP address. For more information, see [Platform agent](/docs/tpa).
- An Azure API Management gateway is created.
- A Log Analytics workspace is created.

---

## Configuration

### Step 1 — Create a storage account

The Storage Account receives application insight logs, which are processed by an Azure App Function and exported to the Traceable Platform agent. Complete the following steps to create a Storage account:

1. Log into your Azure account and click **Storage accounts**.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_storage_account_1.png)
2. Click **Create** on the Storage accounts page.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_storage_account_2.png)
3. On the **Create a storage account** page, navigate to the **Basics**tab and fill in the values. In the **Redundancy**field, you may choose Locally-redundant storage (LRS).

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_storage_account_3.png)
4. Navigate to the **Networking** tab and configure the following. These configurations limit access to this Storage account so that it is accessible from the Log Analytics workspace and the App Function created in the later steps.
  1. **Network access** — Enable public access from selected virtual networks and IP addresses.
  2. **Virtual networks** — Configure your Virtual network subscription, Virtual network, and Subnets.
  3. **Network routing** — Select Microsoft network routing.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_storage_account_3_networking_tab(1).png)
5. Once the Storage account is configured, configure the retention limits. The integration is event-based, so a short retention period is acceptable. In the Storage account you created above, navigate to Lifecycle Management.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_storage_account_5_networking_tab.png)
6. In the **Details** tab of **Add a rule** page, provide the following:
  1. Rule name
  2. Rule scope — Select **Apply rule to all blob in your storage account**.
  3. Blob type — Select **Block blobs**.
  4. Blob subtype — Select**Base blob**.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_storage_account_6.png)
7. In the **Base**blobs tab, configure the rule conditions. You may select **Delete the blob** in the `then` operation.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_storage_account_7.png)
8. Click on **Add**.

---

### Step 2 — Enable API management diagnostic logs

Complete the following steps:

1. In Azure, navigate to your APIM instance.
2. Go to **Monitoring** → **Diagnostic Settings**.
3. Create a new Diagnostic Setting:
  - Select the **Send to Log Analytics Workspace** checkbox.
  - Ensure the **Destination table** is set to **Resource Specific**.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_enable_apim_diagnostic_logs.png)

---

### Step 3 — Configure diagnostic logging on target APIs

Complete the following steps:

1. Navigate to **APIs** → <select an API> → **Settings**.
2. Enable the **Azure Monitor** setting.
3. Set sampling to 100% to capture all requests.
4. Ensure the **Log Client IP address** option is selected.
5. Under **Additional settings**, enable:
  - Frontend Request
  - Frontend Response
6. Specify any headers to be captured for both the Frontend Request and Response. For body capture to work correctly, ensure you capture content type and Content length.
7. Configure the maximum body bytes to capture. The maximum value is 8192 bytes.
8. Click on **Save**.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_target_APIs(1).png)

---

### Step 4 — Create log analytics export rule

Complete the following steps:

1. Navigate to the Log Analytics workspace.
2. Set up an export rule to send data to the Storage account you created earlier. Navigate to **All** **services**→ **Analytics**→ **Log Analytics workspace**.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_export_rule_2.png)
3. Select your workspace and navigate to **Settings** → **Data export**.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_export_rule_3(3).png)
4. Click **+New export rule** on the Data export page.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_export_rule_4.png)
5. Click **Next** and select the **Source**.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_export_rule_5.png)
6. Select Storage account in the **Destination** tab. Click on **Next** and **Review + Create**.

---

### Step 5 — Create a Function App and a Function

Create an Azure App Function to be triggered when APIM logs are sent to the Storage account. The App Function that you create will need network access to the Storage accounts and Traceable Platform agent. If the Traceable Platform agent is hosted privately, you must use Functions Premium to enable vNet connectivity between the App Function and Traceable Platform agent. Complete the following steps to configure the App Function.

1. Navigate to Function App and click **Create**.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_func_app_1.png)
2. Choose **Functions Premium**as the hosting option.
3. Choose **Node.js** as the Runtime stack and configure the other options as shown. Click **Next**

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_func_app_3.png)
4. In the **Storage**tab, select the storage account that you created earlier.
5. In the **Networking** tab:
  1. Select **Off** in the Enable public access option
  2. Select **On** in the Enable network injection option.
  3. Select the Virtual network.
  4. Inbound access — Select **Off**in the Enable private endpoints option.
  5. Outbound access — Select **On** in the Enable Vnet integration and select the **Outbound subnet** from the drop-down list.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_func_app_5.png)
6. Navigate through the rest of the Monitoring, Deployment, and Tags tabs. Finally, Review and Create the App Function.
7. Navigate to your newly created Function App and click **Create** to create a new function, as shown below.
8. In the **Select template** tab, select **Azure Blob Storage trigger** and click **Next**.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_func_app_7(2).png)
9. In the **Template details** tab, provide the following:
  1. Function name
  2. Path — This must be `am-apimanagementgatewaylogs`.
  3. Storage account connection — Select the previously created Storage account.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_func_app_8.png)
10. Click **Create**to create the Function, and navigate to the **Code**+**Test** tab to configure the function.
11. Navigate to Traceable’s [download](/docs/downloads.traceable.ai) site and copy the function.js from **agent** → **azure-logging** → **latest** → **function.js**.
12. Paste the function you copied in Step 10 in the **Code + Test tab,**as shown in Step 9. Configure the Traceable Platform agent IP address and service name. A default service is already provided.

![](https://cdn.document360.io/24f14f07-13d1-4684-8fae-6d8f811768ee/Images/Documentation/traceable_azure_apim_logAnalytics_func_app_9(1).png)

In Azure Storage, "blob" stands for "Binary Large Object," and is a type of storage designed for storing large amounts of unstructured data. Azure Blob Storage is optimized for storing massive amounts of data.

Block blobs are designed for storing text and binary data, such as documents, media files, and backups. They are composed of blocks of data, each of which is identified by a block ID.

The original blob data stored in Azure Blob Storage.