Getting Started with Database Storage

IMPORTANT: This documentation describes a product in early-access development and does not reflect all functionality intended for general availability (GA). It cannot be used in production until GA.

Database Storage for App Builder provides document-style database persistence for AIO Runtime Actions. The aio-lib-db library, which is closely modeled on the MongoDB Database Driver for NodeJS, provides the primary programming interface, while the DB Plugin in the AIO CLI provides additional access.

There is a strict one-to-one relationship between an AIO project workspace and a workspace database, and each workspace database is entirely isolated from all other workspace databases.

Provisioning a Workspace Database

Before using Database Storage in an AIO project workspace, a workspace database must be provisioned. This is a self-service operation requiring no special permissions.

Note that there is a strict one-to-one relationship between an AIO project workspace and a workspace database, and each workspace database is entirely isolated from all other workspace databases. Also, each workspace database must reside in one and only one of the following regions:

A workspace database can be provisioned in one of two ways: declaratively in the app.config.yaml application manifest or manually using the db plugin in the AIO CLI.

To provision a database declaratively, add the follow to the runtime manifest of app.config.yaml:

application:
  runtimeManifest:
    database:
      auto-provision: true
      region: emea

When the application is deployed using aio app deploy a database will be provisioned in the specified region unless it is already present.

To provision a database using the AIO CLI, the following command can be used:

aio app db provision [--region <area>]

In addition to provisioning a workspace database in the selected region, running this command will automatically add a database entry to the runtime manifest of app.config.yaml:

application:
  runtimeManifest:
    database:
      auto-provision: false
      region: emea

The db plugin for the AIO CLI will use the database region defined in app.config.yaml. If it is not defined there, it will either use the amer default or whatever is defined in the AIO_DB_REGION environment variable.

In runtime actions, however, aio-lib-db must be initialized in the same region as defined in app.config.yaml. If that region is anything other than the amer default, the region should be set either by passing a {region: "<region>"} argument to the libDb.init() method or by setting the AIO_DB_REGION environment variable.

In case a workspace database is provisioned in the wrong region, it must first be deleted and then provisioned in the correct region. The process is to delete the database using aio app db delete, set the correct region in the app.config.yaml application manifest, and then provision the new workspace database using aio app deploy or aio app db provision.

DB plugin in the AIO CLI

The DB plugin in the AIO CLI is a utility that facilitates provisioning, initializing, querying, and monitoring workspace databases.

The following is only a brief introduction to the DB plugin. For more thorough documentation see aio-cli-plugin-app-storage.

Installation

To install the pre-GA plugin for the AIO CLI:

aio plugins:install @adobe/aio-cli-plugin-app@next
aio plugins:install @adobe/aio-cli-plugin-app-storage@next

Region selection

When using the DB plugin in the AIO CLI, it is important that the region is the same as where the database is provisioned. If not, the connection will fail.

If a database region is present in the app.config.yaml application manifest, that is the region the DB plugin will use.

If no database region is present in the app.config.yaml application manifest, the region may be specified using the --region option or by setting AIO_DB_REGION environment variable and will otherwise default to amer.

Provisioning a workspace database

To provision a workspace database in the current AIO project workspace is as simple as:

aio app db provision

The provisioning status can be retrieved with this:

aio app db status

To check connectivity with the database:

aio app db ping

Additional database commands include:

# Get statistics about your App Builder database
aio app db stats

# Delete the database for your App Builder application (non-production only)
aio app db delete

Collections

Collections do not have to be explicitly created in order to start using them, but if specific fields need to be indexed or documents require schema validation, then creating a collection beforehand makes sense.

To create an empty collection named inventory:

aio app db collection create inventory

To create an empty collection with schema validation:

aio app db collection create inventory --validator '{"type": "object", "required": ["sku", "quantity"]}'

Note: Schema validation is much less common in document-style databases in comparison with relational databases, and not requiring strict schemas is in fact part of the strength of document-style databases. They should be used judiciously if at all for App Builder applications. See Schema Validation in the MongoDB documentation for more information.

Other collection commands:

# List the collections in the database
aio app db collection list

# Rename a collection in the database
aio app db collection rename <CURRENTNAME> <NEWNAME>

# Drop a collection from the database
aio app db collection drop <COLLECTION>

# Get statistics for a collection in the database
aio app db collection stats <COLLECTION>

Indexes

Indexing frequently queried fields is basic to optimizing the performance of a database.

To create a default type index on specific fields:

aio app db index create <COLLECTION> -k sku -k rating

To create a text index using a spec:

aio app db index create <COLLECTION> -s '{"name":"text", "category":"text"}'

Other index commands:

# Drop an index from a collection in the database
aio app db index drop <COLLECTION> <INDEXNAME>

# Get the list of indexes from a collection in the database
aio app db index list <COLLECTION>

The following index types are supported:

See Indexes for Query Optimization in the MongoDB Documentation for more information.

Documents

The DB Plugin for the AIO CLI is useful for inserting documents and making ad hoc queries against collections. It also supports a rich set of update, replace, and delete operations, but those are expected to be used sparingly.

To insert a document into a collection:

aio app db document insert <COLLECTION> '{"name": "John", "age": 30}'

To find a specific document in a collection:

aio app db document find <COLLECTION> '{"name": "John"}'

To insert multiple documents into a collection:

aio app db document insert <COLLECTION> '[{"name": "John", "age": 30}, {"name": "Jane", "age": 25}]'

To find documents in a collection without a filter:

aio app db document find <COLLECTION> '{}'

Note: By default, only the first 20 documents in a collection are returned and only up to a maximum of 100. In order to retrieve all documents in collection larger than 100 documents, aio-lib-db needs to be used.

Other document commands:

# Update documents in a collection
aio app db document update <COLLECTION> <FILTER> <UPDATE>

# Replace a document in a collection
aio app db document replace <COLLECTION> <FILTER> <REPLACEMENT>

# Delete a document from a collection
aio app db document delete <COLLECTION> <FILTER>

# Count documents in a collection
aio app db document count <COLLECTION> <FILTER>

Runtime actions and aio-lib-db

The aio-lib-db package provides the main programming interface for App Builder Database Storage. It is intentionally modeled on the MongoDB Node Driver striving to be a near drop-in replacement for applications developed for MongoDB and/or AWS DocumentDB.

Much of the extensive documentation for the MongoDB Node Driver is valid for aio-lib-db. Rather than duplicating, the following guide provides links to the MongoDB documentation, with notes about any important differences where applicable.

Installation

npm install @adobe/aio-lib-db

Region selection

aio-lib-db must be initialized in the region the workspace database was provisioned. Otherwise, the connection will fail.

To explicitly initialize the library in a specific region, pass the {region: "<region>"} arguement to the libDb.init() method.

Called with no arguments, libDb.init() will initialize the library either in the default amer region or in the region defined in the AIO_DB_REGION environment variable.

Basic usage

The following assumes that a Workspace Database has been provisioned in the AIO Project Workspace using the DB Plugin in the AIO CLI as described above.

Connecting to App Builder Database Storage is where aio-lib-db most differs from the MongoDB Node Driver.

The following is the general pattern for loading and using aio-lib-db:

const libDb = require('@adobe/aio-lib-db')
const { DbError } = require('@adobe/aio-lib-db')

async function main() {
  let client
  try {

    // Implicit region initialization
    const db = await libDb.init()
      
    // Explicit region initialization
    // const db = await libDb.init({region: "emea"})

    // Set up a connection
    client = await db.connect()

    // Select a collection
    const userCollection = await client.collection('users')

    // do stuff with the collection...

  } catch (error) {
    // Errors thrown by the database are reported as such
    if (error instanceof DbError) {
      console.error('Database error:', error.message);
    } else {
      console.error('Unexpected error:', error);
    }
  } finally {
    // Best practice to always close the client connection
    if (client) {
      await client.close()
    }
  }
}

A few things to note in comparison with the MongoDB Node Driver:

Basic CRUD operations

Included in the following are links to the equivalent methods for the MongoDB Node Driver.

Inserting documents

Insert one document:

z
const result = await userCollection.insertOne({
  name: 'Jane Smith',
  email: 'jane@example.com',
  age: 30
})

Insert multiple documents:

const result = await userCollection.insertMany([
  { name: 'Alice', email: 'alice@example.com', age: 27 },
  { name: 'Bob', email: 'bob@example.com', age: 12 }
])

MongoDB Node Driver references:

Finding documents

Find one document:

const user = await userCollection.findOne({ email: 'john@example.com' });

Find all documents matching a filter (returns a cursor - see next section):

const cursor = userCollection.find({ age: { $gte: 18 } })

Find with projection, sort, skip and limit (returns a cursor - see next section):

const cursor = userCollection.find({ age: { $gte: 18 } })
  .project({ name: 1, email: 1 })
  .sort({ name: 1 })
  .skip(2)
  .limit(10)

MongoDB Node Driver references:

Cursor access patterns

Both find and aggregate return cursors.

Using toArray() - loads all results into memory:

const results = await cursor.toArray()

Using iteration - memory efficient:

while (await cursor.hasNext()) {
  const doc = await cursor.next();
  console.log(doc)
}

Using for await...of - most convenient:

for await (const doc of cursor) {
  console.log(doc)
}

Using streams:

const stream = cursor.stream();
stream.on('data', (doc) => {
  console.log(doc)
})

MongoDB Node Driver references:

Updating documents

Update one document:

const result = await userCollection.updateOne(
  { email: 'john@example.com' },
  { $set: { age: 31 } }
)

Replace one document:

const result = await userCollection.replaceOne(
  { email: 'john@example.com' },
  { name: 'Bob', email: 'bob@example.com', age: 12 }
)

Update multiple documents:

const result = await userCollection.updateMany(
  { age: { $lt: 18 } },
  { $set: { category: 'minor' } }
)

Find and update:

const updatedUser = await userCollection.findOneAndUpdate(
  { email: 'john@example.com' },
  { $set: { lastLogin: new Date() } },
  { returnDocument: 'after' }
)

MongoDB Node Driver references:

Deleting documents

Delete one document:

const result = await userCollection.deleteOne({ email: 'john@example.com' })

Delete multiple documents:

const result = await userCollection.deleteMany({ age: { $lt: 0 } })

Find and delete:

const deletedUser = await userCollection.findOneAndDelete({ email: 'john@example.com' })

MongoDB Node Driver references:

Bulk operations

Multiple operations in a single request:

const operations = [
  { insertOne: { document: { name: 'Alice' } } },
  { updateOne: { filter: { name: 'Bob' }, update: { $set: { age: 30 } } } },
  { deleteOne: { filter: { name: 'Charlie' } } }
]

const result = await collection.bulkWrite(operations)

MongoDB Node Driver references:

String and object Representations of the _id field

Every document in DocumentDB has a required _id field that acts as its unique identifier within a collection. Values for the _id field may be specified in the document or generated on the fly by the database server.

When a document with no value specified for the _id field is inserted into a collection, the database service will generate a unique value for the field of type ObjectId and add it to the document. So the following:

const result = await userCollection.insertOne({name: "Jane Smith"})

with a result something like this:

{
   "acknowledged" : true,
   "insertedId" : "56fc40f9d735c28df206d078"
}

When the _id field is represented as a string, for example in an HTTP request or text file, it needs to be converted to an ObjectId before using in a query filter. To retrieve the above document, for example, something like the following is required:

const {ObjectId} = require('bson')
const userDocument = await userCollection.findOne({ _id: new ObjectId("56fc40f9d735c28df206d078")})

with a result something like this:

{
    "name": "Jane Smith",
    "_id": "56fc40f9d735c28df206d078"
}

See the MongoDB docs for more details on the _id field:

Aggregates

Aggregates are a powerful tool for building complex queries.

Simple aggregate pipeline:

const pipeline = [
  { $match: { status: 'active' } },
  { $group: { _id: '$category', count: { $sum: 1 } } },
  { $sort: { count: -1 } }
]

const cursor = collection.aggregate(pipeline)

A geospatial example:

const nearbyStores = await stores.aggregate()
  .geoNear({
    near: {type: 'Point', coordinates: [-122.4194, 37.7749]}, // San Francisco
    distanceField: 'distance',
    maxDistance: 1000, // 1km radius
    spherical: true
  })
  .match({status: 'open'})
  .limit(10)
  .toArray()

MongoDB Node Driver references:

MongoDB and DocumentDB compatibility

The MongoDB 8.0 features supported by App Builder Database Storage (ABDB) are constrained by the AWS DocumentDB with MongoDB compatibility it is built on.

The primary reference for MongoDB compatibility is Supported MongoDB APIs, operations, and data types in Amazon DocumentDB. Note that ABDB uses version 8.0 of the MongoDB API. Some additional functional differences are documented at Functional differences: Amazon DocumentDB and MongoDB.

Beyond those imposed by AWS DocumentDB there are additional constraints imposed by the App Builder Database Storage itself. For example, the App Builder DB exposes far fewer administrative commands than either DocumentDB or MongoDB, because it is a multi-tenant offering.

The following sections highlight the differences between the App Builder Database Storage API and AWS DocumentDB.

Database commands

Database commands are not supported by the App Builder Database Storage.

Administrative and diagnostic features are limited to those provided by the aio-lib-db package and the db plugin for the aio cli.

AWS reference: Database commands.

Query and projection operators

Same as DocumentDB 8.0.

AWS reference: Query and projection operators

Update operators

Same as DocumentDB 8.0.

AWS reference: Update Operators

Geospatial

Same as DocumentDB 8.0.

AWS reference: Geospatial

Cursor methods

Although the general access patterns for cursors with App Builder Database Storage closely follow the DocumentDB/MongoDB model (see the previously mentioned Cursor Access Patterns) only a subset of other methods are supported, and these are only supported when initializing a cursor.

Supported methods when initializing a find cursor:

Supported methods when initializing an aggregate cursor:

Supported methods for reading both find and aggregate cursors:

AWS reference: Cursor Methods

Aggregation pipeline operators

At this time only the following stage operators are supported by App Builder Database:

Note: Support for more stage operators is coming soon.

All other pipeline operators are the same as for DocumentDB 8.0.

AWS reference: Aggregation pipeline operators

Data types

Same as DocumentDB 8.0.

AWS reference: Data types

Indexes and index properties

Same as DocumentDB 8.0.

AWS reference: Indexes and index properties