Use Transformations to fix bad data

What is a Transformation?

With Transformations, you can change data as it flows through Segment to either correct bad data or customize data for a specific destination. Change event and property names to align events with your Tracking Plan, or to conform to a destination-specific requirement. For example, you can create a Transformation to change an event name from completed_order to Order Completed to conform to Segment’s ecommerce spec.

You can also use Segment’s Public API to transform events, properties, and property values for many use cases.

Transformations are very powerful and should be applied with care.

Transformations irrevocably change the event payloads flowing through Segment and immediately affect either all destinations, or a single downstream destination, depending on your settings.

Segment’s goal is to make Transformations a powerful tool that complements a well structured Tracking Plan. Together, these features help your organization scale and achieve high data quality. For that reason Segment recommends that you start your data quality strategy with a clearly defined Tracking Plan. Without this critical component, the risk of creating conflicting or detrimental transformations increases.

Other important notes

• Transformations cannot be applied retroactively: They only apply to data moving forward. However, you can manually extract and re-send (or even Replay) events through a source with an active destination Transformation, which will send the transformed events to your destinations.
• Transformations are available to Protocols customers: If you’re interested in this feature, contact your Account Executive or CSM to learn more about the Protocols package.
• Source-level transformations are irrevocable: When applied at the source, a transformation permanently changes the structure of the event. The original events are not easily recoverable or Replayable. Assume that transformed data cannot be recovered.
• Device-mode destinations are NOT supported: Source scoped transformations will only apply to cloud-mode destinations, warehouses, and S3 destinations. Destination scoped transformations will only apply to cloud-mode destinations.
• Transformation order: Segment processes source-scoped transformations before the events reach destination filters. Segment processes destination-scoped transformations after the events pass through destination filters.
• Multiple instances of the same destination: If multiple instances of a destination are connected to a single source, then the same transformation applies across every instance.
• Transformation scope: Transformations can be applied to event names, properties, and traits. However, it’s important to note that transformations cannot be applied to top-level attributes like userId.

View all Transformations

All Protocols Transformations are listed in the Transformations tab in the Protocols section. The list view supports filtering and sorting to organize transformations by transformation type, source, and destination.

Transformations can be enabled and disabled directly from the list view using the toggle.

Transformations can be deleted and edited by clicking on the overflow menu. When editing a Transformation, only the resulting event or property names, and Transformation name can be edited. If you want to select a different event or source, create a separate Transformation rule.

Create a Transformation

To create a Transformation, navigate to the Transformations tab in Protocols and click New Transformation in the top right. A three-step wizard guides you through creating a transformation.

Step 1: Select the transformation type

To create a Transformation, you first need to select which type of transformation you want to create. For each transformation type, Segment displays a description, use cases, and example payload. Current transformation types available in your Segment workspace include:

Rename track event: Rename track event name at the source or per destination. The events listed in the event names dropdown menu correspond to the events listed on the source schema view.

Edit track event properties: Rename multiple properties and/or change property data structure at the source or per destination

Edit identify or group event traits: Rename multiple traits and/or change trait data structure at the source or per destination

Step 2: Set up the transformation

Depending on the transformation type you selected, relevant drop-down selectors and fields are presented to define how you want to transform the data.

Regardless of the type of transformation selected, first select a source. Each Transformation can only apply to a single source. While this makes it more difficult to apply transformations broadly, it ensures you are only transforming data relevant to the selected source.

After selecting the source, you will need to select a scope. Scope determines where Segment applies the transformation.

• Source scope: Events are transformed in all active Segment cloud-mode destinations, warehouses, and S3 destinations. This scope is best when you want to fix malformed events before sending them to all destinations. These transformations should be treated as a temporary solution to hold you over while your engineering team fixes the root event.

• Destination scope: Events are transformed in ONLY the selected cloud-mode destination. Device-mode destinations, S3, and data warehouses are not currently supported. Use the Destination scope when you want to customize an event to the unique requirements of a destination. These transformations can exist permanently.

Depending on the type of transformation you selected, you will need to enter the relevant event, property, or trait mappings to create the transformation.

• Select an Event: After you select the scope, use the search box to choose the event to transform. You can only select a single track event, identify or group call. If you are renaming the event, simply enter the new name in the provided text field.

• Rename properties or traits: To rename properties or traits within a selected event, click + Add Property. The dropdown that appears contains the properties or traits sent with the selected event. Segment supports JSON Path notation to select nested objects up to four levels deep. For example, order.id selects the id property in the order object. Segment does not support .$. notation to select a property from an array of objects. For example, the following event, which generates products.$.product_id, is unsupported.

analytics.track('Example', {
  products: [{
      product_id: "123"
  }],
})

In this scenario, we do not support the transformation of product_id.

After selecting a property/trait, select JSON Path or Simple String to change the property/trait. Simple string will change the name in-line, while JSON path allows you to move the property/trait in or out of an object.

ingredients.salad → "ingredients": { "salad": "yum" }
ingredients\.salad → "ingredients.salad": "yum"

Step 3: Name the transformation and enable it

Enter a descriptive name to act as a label for the transformation. This label helps you organize your Transformations, and Segment recommends that you make this descriptive and focused on the problem you’re solving. For example Fix misnamed order_completed event for ecommerce spec is much better than Map order_completed.

In this step, you can also choose to keep the Transformation disabled, so you can come back and edit it later. To update, enable, or disable a Transformation, click on the overflow menu and select Edit Transformation.

Use Cases

Here’s a list of Segment Transformations with some use case examples.

• Rename an event: Change an event name from viewed_product to Product Viewed.

• Rename a property or trait: Change the property name revenue to total for a specific destination.

• Update a property value: Use Segment’s Public API to transform the property currency to have the value USD.

• Property Transformations

  • Assigning static values: If you want to create a new property and set a static value, use Segment’s Public API to create new_property: static_value. Segment currently supports setting static values for top-level fields, as well as fields within the context or properties object with propertyValueTransformations. However, Segment doesn’t support changing fields outside the properties or traits object with propertyRenames. You can use propertyValueTransformations on a single object to assign the same value to different fields or on multiple objects to assign a static value to the same field across objects.
  • Casing functions: Use Segment’s Public API to transform property value casing to lowercase, uppercase, snakecase, kebabcase, or titlecase. When transforming data with casing functions, use the fqlDefinedProperties array to define the FQL you want to use and the new or existing propertyName you’d like to transform.
    • Static and dynamic value casing: Use casing functions to transform property values to create uniform tracking data. For example, you can convert usa to USA to keep your downstream data consistent.
      You can transform these properties using static casing functions:
      

      fqlDefinedProperties": [{"fql": "uppercase("United States)", "propertyName": "properties.propertyValue1"}]
      

      or dynamic casing functions:

      fqlDefinedProperties": [{"fql": "lowercase(properties.propertyValue1)", "propertyName": "properties.propertyValue1"}]
      

    • Create a new property with applied casing: Use Segment’s Public API to create a new property and set the value of the new property to the transformed value of an existing property. You can dynamically assign the value of one existing property to another, or assign the value of an existing property to a new property without applying casing functions.
      For example, create a new property (prop2) with a value of lowercase(properties.prop1) by including the following snippet in your payload:
      

      fqlDefinedProperties": [{"fql": "lowercase(properties.prop1)", "propertyName": "properties.prop2"}]
      

  • Note that you can only assign one property to fqlDefinedProperties array.
  • Note that you cannot use fqlDefinedProperties along with event or property rename or property value transformations.