Bulk Data Insertion API
The Bulk Data Insertion API (BDIA) is an Adobe Analytics capability that lets you upload server call data in batches of files instead of using client-side libraries such as AppMeasurement. The server calls in these batch files can be either current (live) data or historical data. For some companies it may offer a simpler way to scale ingest requests than using the Data Insertion API (DIA).
Bulk Data Insertion solves several problems for a variety of use cases. Some use case examples include:
- Ingesting historical data from a previous analytics system
- An internal analytics collection system that makes it unfeasible to use AppMeasurement. You can use Extract-Transform-Load (ETL) processes to put data into batch files then use BDIA to upload them to Adobe Analytics.
- Data collection from devices that have only intermittent connectivity to the internet. These devices store up the interactions until they receive a connection. The device can then upload the data all at once via BDIA.
BDIA vs DIA
Bulk Data Insertion API (BDIA)
Data Insertion API (DIA)
Data submission
CSV file containing multiple events (dozens to tens of thousands), one event per row
One event per API call
Throughput
Optimized for larger files sent less frequently; not intended for more than 1 file per second, nor should files regularly be sent with just a few rows.
No upper bound on request rate
Best fit
Bulk/batch loads, historical data backfills, or sources with intermittent connectivity
High-frequency, near real-time event submission
data-variant=info
data-slots=text
Adobe may add optional request and response members (name/value pairs) to existing API objects at any time and without notice or changes in versioning. Adobe recommends that you refer to the API documentation of any third-party tool you integrate with our APIs so that such additions are ignored in processing if not understood. If implemented properly, such additions are non-breaking changes for your implementation. Adobe will not remove parameters or add required parameters without first providing standard notification through release notes.
Prerequisites
Before using this API, make sure that all of the following are met:
- You can successfully authenticate with the API using OAuth. See Getting started to make sure that you have the correct permissions and have created an API client on Adobe Developer. Note, JWT authentication is deprecated and will only be supported until January 1, 2025. Current integrations using JWT will need to migrate to OAuth Server to Server prior to January 1, 2025.
- The desired report suite is timestamp-enabled or timestamp optional. See Timestamps optional in the Adobe Analytics documentation. All newly created report suites are set to timestamp optional by default.
- Communicate to Adobe the expected volume of ingestion per day. Based on this information, Adobe provisions the appropriate hardware to handle that volume and creates a per-second throttle limit. If enough files are uploaded in a short amount of time to exceed the throttle limit, Adobe ingests uploaded files more slowly. These limits help ensure timely processing and availability of data for reporting. They also help protect the system from becoming overwhelmed before proper capacity is provisioned for a sharp increase in file uploads.
- Follow the established File formatting requirements for each upload.
- If using a Customer Attribute as a seed to automatically generate Experience Cloud Visitor IDs (ECID), provisioning by Adobe is required first. See Use customer ID to identify visitors.
- You are using the correct number of Visitor Groups for your anticipated load. Follow the guidelines on file send frequency limits to avoid having your requests throttled or data processing out of order.
- You do not plan to upload more than 1 file per second, nor send files with very small row counts. If these stipulations are not possible, consider using the Data Insertion API instead.
Once you meet all prerequisites, see File format to prepare your data in a format usable by the API.