Setting up the Marketo Observability Data Stream

These instructions describe how to set up and get started using Adobe I/O Events to subscribe to Marketo Observability events.

• Introduction
• Prerequisite Setup
• Setup Adobe I/O
• Developer Guidelines
• Debug

Introduction

The Marketo Observability Data Stream (MODS) provides insights about the data flow into Marketo through the Data Ingestion Service. The stream offers several different event types:

• Metrics - Periodic aggregated metrics showing all data sent to the API and processed into Marketo
• Results - Per API request results for tracking and verification
• Status - Periodic notices providing insight into the remaining API quota and any request processing backlog

Note: MODS (Marketo Observability Data Stream) is currently a Beta Product

Prerequisite Setup

The following are required to be able to subscribe to the data stream:

• Marketo Engage subscription that is IMS-enabled
• Marketo Engage subscription provisioned with a Performance Tier or CET package
• Developer or admin account in the Experience Cloud

Getting Started with Adobe I/O

See the Adobe I/O Events Docs

Basic instructions for this use case, starting from the developer console:

When prompted, click the designated button to proceed

• Select Create new project

  

• Select Add event

  

• Filter by Experience Cloud

• Select Marketo Observability Data Stream

  

• Subscribe to observability events that you're interested in

  

• Set up authentication (OAuth Server-to-Server) to be used for accessing the Journaling API

  

• Set the name and description for your event registration

  

• Finish Event Registration Setup

  

  • Provide a name and description for this event subscription

  • Optionally choose whether to enable Webhook or Runtime action

    • Enable Webhook

      • We recommend batch over single webhooks
      • For Webhook URL a public https endpoint must be provided
      • The endpoint must be able to handle get and post requests
      • The get request must respond with the challenge query if it exists
      • The post request must respond that it received the message or the webhook will re-attempt to send several times before giving up and automatically disabling the webhook sends

    • Enable Runtime action

      • Getting Started with Adobe I/O Runtime
      • Runtime Actions as Webhook

• After Saving

  

  • Verify that the Status is Active
  • If Webhook was selected, verify that it successfully passed the challenge without errors

data-src=marketo-data-streams-developer-guidelines.md

data-src=marketo-data-streams-multiple-instances.md

Event Data Structure Examples

Events are structured in JSON format using the CloudEvents spec

Metrics

{
    "id": "b90382d8-6b23-11ee-b962-0242ac120002",
    "specversion": "1.0",
    "type": "com.adobe.platform.marketo.observability.metrics",
    "source": "urn:data_ingestion_service",
    "time": "2023-08-14T18:00:00Z",
    "datacontenttype": "application/json",
    "data": {
        "munchkinId": "123-ABC-456",
        "windowStart": "2023-08-14T17:00:00Z",
        "requests": {
            "received": 3,
            "processed": 1,
            "rejected": {  
                "606": 1,
                "607": 1
            }
        },
        "records": {
            "person": {
                "high": {
                    "received": 100,
                    "created": 90,
                    "updated": 5,
                    "failed": {
                        "503": 5
                    }
                },
                "normal": {
                    "received": 100,
                    "created": 90,
                    "updated": 5,
                    "failed": {
                        "503": 5
                    }
                }
            },
            "customObject": {
                "high": {
                    "received": 100,
                    "created": 90,
                    "updated": 5,
                    "failed": {
                        "503": 5
                    }
                },
                "normal": {
                    "received": 100,
                    "created": 90,
                    "updated": 5,
                    "failed": {
                        "503": 5
                    }
                }
            }
        }
    }
}

Results

Result events contain an array of processed requests indicating what was processed. These may be complete or partial results depending on the size of the request payload. Request payloads are processed in chunks, and if a request is split into multiple chunks, the processing results could potentially be reported across multiple events.

{
    "id": "b90382d8-6b23-11ee-b962-0242ac120002",
    "specversion": "1.0",
    "type": "com.adobe.platform.marketo.observability.results",
    "source": "urn:data_ingestion_service",
    "time": "2023-08-14T17:30:00Z",
    "datacontenttype": "application/json",
    "data": {
        "munchkinId": "123-ABC-456",
        "requests": [
            {
                "requestId": "cf0a1a20-668e-492a-8ec2-ce8747507068",
                "requestTime": "2023-08-14T17:20:00Z",
                "clientId": "foo@marketo.com",
                "correlationId": "6180bb48-8dc7-4fc5-85ca-a59dd1edb0f3",
                "requestSource": "Adobe Journey Optimizer",
                "objectType": "person",
                "priority": "high",
                "records": {
                    "received": 100,
                    "created": 90,
                    "updated": 10,
                    "failed": {}
                }
            },
            {
                "requestId": "67b28858-f5ee-45ac-aa40-63a04085e6be",
                "requestTime": "2023-08-14T17:20:00Z",
                "clientId": "foo@marketo.com",
                "correlationId": "6180bb48-8dc7-4fc5-85ca-a59dd1edb0f3",
                "requestSource": "Public API",
                "objectType": "customObject",
                "priority": "normal",
                "records": {
                    "received": 100,
                    "created": 10,
                    "updated": 10,
                    "failed": {
                        "503": 10,
                        "404": 10
                    }
                }
            }
        ]
    }
}

Status

{
    "id": "b90382d8-6b23-11ee-b962-0242ac120002",
    "specversion": "1.0",
    "type": "com.adobe.platform.marketo.observability.status",
    "source": "urn:data_ingestion_service",
    "time": "2023-08-14T18:00:00Z",
    "datacontenttype": "application/json",
    "data": {
        "munchkinId": "123-ABC-456",
        "quota": 65432100,
        "queue": {
            "high": {
                "lagSeconds": 15,
                "requestBacklog": 1,
                "recordBacklog": 100
            },
            "normal": {
                "lagSeconds": 135,
                "requestBacklog": 10,
                "recordBacklog": 1000
            }
        },
    }
}

Data Field Definitions

data-src=marketo-data-streams-debug.md