Spec: Common Fields

In the Segment Spec all the API calls have a common structure, and a few common fields.

However, not all destinations accept all fields included in the Spec. Not sure which fields a destination accepts? Refer to the destination’s documentation page, or check out the open-source destination code on GitHub.

Structure

Every API call has the same core structure and fields. These fields describe user identity, timestamping, and mechanical aides like API version.

Here’s an example of these common fields in raw JSON:

{
  "anonymousId": "507f191e810c19729de860ea",
  "context": {
    "active": true,
    "app": {
      "name": "InitechGlobal",
      "version": "545",
      "build": "3.0.1.545",
      "namespace": "com.production.segment"
    },
    "campaign": {
      "name": "TPS Innovation Newsletter",
      "source": "Newsletter",
      "medium": "email",
      "term": "tps reports",
      "content": "image link"
    },
    "device": {
      "id": "B5372DB0-C21E-11E4-8DFC-AA07A5B093DB",
      "advertisingId": "7A3CBEA0-BDF5-11E4-8DFC-AA07A5B093DB",
      "adTrackingEnabled": true,
      "manufacturer": "Apple",
      "model": "iPhone7,2",
      "name": "maguro",
      "type": "ios",
      "token": "ff15bc0c20c4aa6cd50854ff165fd265c838e5405bfeb9571066395b8c9da449"
    },
    "ip": "8.8.8.8",
    "library": {
      "name": "analytics.js",
      "version": "2.11.1"
    },
    "locale": "en-US",
    "network": {
      "bluetooth": false,
      "carrier": "T-Mobile US",
      "cellular": true,
      "wifi": false
    },
    "os": {
      "name": "iPhone OS",
      "version": "8.1.3"
    },
    "page": {
      "path": "/academy/",
      "referrer": "",
      "search": "",
      "title": "Analytics Academy",
      "url": "https://segment.com/academy/"
    },
    "referrer": {
      "id": "ABCD582CDEFFFF01919",
      "type": "dataxu"
    },
    "screen": {
      "width": 320,
      "height": 568,
      "density": 2
    },
    "groupId": "12345",
    "timezone": "Europe/Amsterdam",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
    "userAgentData": {
      "brands": [
        {
          "brand": "Google Chrome",
          "version": "113"
        },
        {
          "brand": "Chromium",
          "version": "113"
        },
        {
          "brand": "Not-A.Brand",
          "version": "24"
        }
      ],
      "mobile": false,
      "platform": "macOS"
    }
  },
  "integrations": {
    "All": true,
    "Mixpanel": false,
    "Salesforce": false
  },
  "event": "Report Submitted",
  "messageId": "022bb90c-bbac-11e4-8dfc-aa07a5b093db",
  "receivedAt": "2015-12-10T04:08:31.909Z",
  "sentAt": "2015-12-10T04:08:31.581Z",
  "timestamp": "2015-12-10T04:08:31.905Z",
  "type": "track",
  "userId": "97980cfea0067",
  "version": 2
}

In more detail these common fields for every API call are:

Beyond this common structure, each API call adds a few specialized top-level fields.

Context

Context is a dictionary of extra information that provides useful context about a datapoint, for example the user’s ip address or locale. You should only use Context fields for their intended meaning.

Context fields automatically collected

Below is a chart that shows you which context variables are populated automatically by the iOS, Android, and analytics.js libraries.

Other libraries only collect context.library, any other context variables must be sent manually.

• IP Address isn’t collected by Segment’s libraries, but is instead filled in by Segment’s servers when it receives a message for client side events only.

• The Android library collects screen.density with this method.

• userAgentData is only collected if the Client Hints API is available on the browser.

• Segment doesn’t collect or append to the context of subsequent calls in the new mobile libraries (Swift, Kotlin, and React Native).

To pass the context variables which are not automatically collected by Segment’s libraries, you must manually include them in the event payload. The following code shows how to pass groupId as the context field of Analytics.js’s .track() event:

analytics.track("Report Submitted", {}, {
  context: {
    groupId: "1234"
  }
});

To add fields to the context object in the new mobile libraries, you must utilize a custom plugin. Documentation for creating plugins for each library can be found here:

• React Native
• Swift
• Kotlin

Integrations

A dictionary of destination names that the message should be sent to. 'All' is a special key that applies when no key for a specific destination is found.

Integrations defaults to the following:

{
  All: true,
  Salesforce: false,
}

This is because Salesforce has strict limits on API calls.

Sending data to the rest of Segment’s destinations is opt-out so if you don’t specify the destination as false in this object, it will be sent to rest of the destinations that can accept it.

Timestamps

Every API call has four timestamps, originalTimestamp, timestamp, sentAt, and receivedAt. They’re used for very different purposes.

All timestamps are ISO-8601 date strings, and are in the UTC timezone. To see the user’s timezone information, check the timezone field that’s automatically collected by client-side libraries.

Timestamp overview

originalTimestamp

The originalTimestamp tells you when call was invoked on the client device or the value of timestamp that you manually passed in.

Note: The originalTimestamp timestamp is not useful for any analysis since it’s not always trustworthy as it can be easily adjusted and affected by clock skew.

sentAt

The sentAt timestamp specifies the clock time for the client’s device when the network request was made to the Segment API. For libraries and systems that send batched requests, there can be a long gap between a datapoint’s timestamp and sentAt. Combined with receivedAt, Segment uses sentAt to correct the original timestamp in situations where a user’s device clock cannot be trusted (mobile phones and browsers). The sentAt and receivedAt timestamps are assumed to occur at the same time (maximum a few hundred milliseconds), and therefore the difference is the user’s device clock skew, which can be applied back to correct the timestamp.

Note: The sentAt timestamp is not useful for any analysis since it’s tainted by user’s clock skew.

receivedAt

The receivedAt timestamp is added to incoming messages as soon as they hit the API. It’s used in combination with sentAt to correct clock skew, and also to aid with debugging libraries and systems that deliver events in batches.

The receivedAt timestamp is most important as the sort key in Segment’s Warehouses product. Use this for max query speed when retrieving data from your Warehouse.

Note: Chronological order of events is not ensured with receivedAt.

timestamp

The timestamp timestamp specifies when the data point occurred, corrected for client-device clock skew. This is the timestamp that is passed to downstream destinations and used for historical replays. It is important to use this timestamp for importing historical data to the API.

If you are using the Segment server Source libraries, or passing calls directly to the HTTP API endpoint, you can manually set the timestamp field. This change updates the originalTimestamp field of the Segment event. If you use a Segment Source in device mode, the library generates timestamp and you cannot manually set one directly in the call payload.

Segment calculates timestamp as timestamp = receivedAt - (sentAt - originalTimeStamp).

FAQ

Why Are Events Received with Timestamps Set in the Past or Future?

If you’re using one of Segment’s client-side libraries, please note that several factors can cause timestamp discrepancies in your event data.

1. Overriding Timestamp Value:
   • When a manual timestamp is set in the payload with a date in the past, it can cause events to appear as if they were sent earlier than they actually were.
2. Analytics.js Source with Retries Enabled:
   • The Retries feature supports offline traffic by queuing events in Analytics.js. These events are sent or retried later when an internet connection is available, keeping the original timestamp intact.
3. Mobile App Backgrounded or Closed:
   • If a user closes the app, events may be queued within the app. These queued events won’t be sent until the app is re-opened, potentially in the future, leading to timestamp discrepancies.
4. Inaccurate Browser/Device Clock Settings:
   • Timestamps can be incorrect if the client’s device time is inaccurate, as the originalTimestamp relies on the client device’s clock, which can be manually adjusted.
5. Traffic from Internet Bots:
   • Internet Bots can sometimes send requests with unusual timestamps, either intentionally or due to incorrect settings, leading to discrepancies.