Getting Started with Database Storage

IMPORTANT: The App Builder Database Storage feature is in beta. It is not recommended for production use.

Database Storage for App Builder provides document-style database persistence for AIO Runtime Actions. The aio-lib-db library, which is closely modeled on the MongoDB Database Driver for NodeJS, provides the primary programming interface, while the DB Plugin in the AIO CLI provides additional access.

There is a strict one-to-one relationship between an AIO project workspace and a workspace database, and each workspace database is entirely isolated from all other workspace databases.

Before use, a workspace database must be explicitly provisioned using either aio-lib-db or the AIO CLI. This is a self-service operation subject to organizational quotas and limits.

Workspace databases are provisioned in one and only one of the following regions: amer, apac, and emea. These acronyms are defined as follows:

DB plugin in the AIO CLI

The DB plugin in the AIO CLI is a utility that facilitates provisioning, initializing, querying, and monitoring workspace databases.

The following is only a brief introduction to the DB plugin. For more thorough documentation see aio-cli-plugin-app-storage.

Installation

To install the pre-GA plugin for the AIO CLI:

aio plugins:install @adobe/aio-cli-plugin-app-storage@next

Region selection

When using the DB plugin in the AIO CLI, it is important that the region is the same as where the database is provisioned. If not, the connection will fail.

The default region is amer, and to set a different one, you can use the --region flag or add the AIO_DB_REGION variable to your .env file. Supported regions include amer, emea, and apac.

Provisioning a workspace database

To provision a workspace database in the current AIO project workspace is as simple as:

aio app db provision

The provisioning status can be retrieved with this:

aio app db status

To check connectivity with the database:

aio app db ping

Additional database commands include:

# Get statistics about your App Builder database
aio app db stats

# Delete the database for your App Builder application (non-production only)
aio app db delete

Collections

Collections do not have to be explicitly created in order to start using them, but if specific fields need to be indexed or documents require schema validation, then creating a collection beforehand makes sense.

To create an empty collection named inventory:

aio app db collection create inventory

To create an empty collection with schema validation:

aio app db collection create inventory --validator '{"type": "object", "required": ["sku", "quantity"]}'

Note: Schema validation is much less common in document-style databases in comparison with relational databases, and not requiring strict schemas is in fact part of the strength of document-style databases. They should be used judiciously if at all for App Builder applications. See Schema Validation in the MongoDB documentation for more information.

Other collection commands:

# List the collections in the database
aio app db collection list

# Rename a collection in the database
aio app db collection rename <CURRENTNAME> <NEWNAME>

# Drop a collection from the database
aio app db collection drop <COLLECTION>

# Get statistics for a collection in the database
aio app db collection stats <COLLECTION>

Indexes

Indexing frequently queried fields is basic to optimizing the performance of a database.

To create a default type index on specific fields:

aio app db index create <COLLECTION> -k sku -k rating

To create a text index using a spec:

aio app db index create <COLLECTION> -s '{"name":"text", "category":"text"}'

Other index commands:

# Drop an index from a collection in the database
aio app db index drop <COLLECTION> <INDEXNAME>

# Get the list of indexes from a collection in the database
aio app db index list <COLLECTION>

The following index types are supported:

See Indexes for Query Optimization in the MongoDB Documentation for more information.

Documents

The DB Plugin for the AIO CLI is useful for inserting documents and making ad hoc queries against collections. It also supports a rich set of update, replace, and delete operations, but those are expected to be used sparingly.

To insert a document into a collection:

aio app db document insert <COLLECTION> '{"name": "John", "age": 30}'

To find a specific document in a collection:

aio app db document find <COLLECTION> '{"name": "John"}'

To insert multiple documents into a collection:

aio app db document insert <COLLECTION> '[{"name": "John", "age": 30}, {"name": "Jane", "age": 25}]'

To find documents in a collection without a filter:

aio app db document find <COLLECTION> '{}'

Note: By default, only the first 20 documents in a collection are returned and only up to a maximum of 100. In order to retrieve all documents in collection larger than 100 documents, aio-lib-db needs to be used.

Other document commands:

# Update documents in a collection
aio app db document update <COLLECTION> <FILTER> <UPDATE>

# Replace a document in a collection
aio app db document replace <COLLECTION> <FILTER> <REPLACEMENT>

# Delete a document from a collection
aio app db document delete <COLLECTION> <FILTER>

# Count documents in a collection
aio app db document count <COLLECTION> <FILTER>

Runtime actions and aio-lib-db

The aio-lib-db package provides the main programming interface for App Builder Database Storage. It is intentionally modeled on the MongoDB Node Driver striving to be a near drop-in replacement for applications developed for MongoDB and/or AWS DocumentDB.

Much of the extensive documentation for the MongoDB Node Driver is valid for aio-lib-db. Rather than duplicating, the following guide provides links to the MongoDB documentation, with notes about any important differences where applicable.

Installation

npm install @adobe/aio-lib-db

Basic usage

The following assumes that a Workspace Database has been provisioned in the AIO Project Workspace using the DB Plugin in the AIO CLI as described above.

Connecting to App Builder Database Storage is where aio-lib-db most differs from the MongoDB Node Driver.

The following is the general pattern for loading and using aio-lib-db:

const libDb = require('@adobe/aio-lib-db')
const { DbError } = require('@adobe/aio-lib-db')

async function main() {
  let client
  try {
    // Initialize the library. Defaults to amer region
    const db = await libDb.init()

    // Region must be specified if the database is not in the amer region
    // const db = await libDb.init({region: 'emea'})

    // Set up a connection
    client = await db.connect()

    // Select a collection
    const userCollection = await client.collection('users')

    // do stuff with the collection...

  } catch (error) {
    // Errors thrown by the database are reported as such
    if (error instanceof DbError) {
      console.error('Database error:', error.message);
    } else {
      console.error('Unexpected error:', error);
    }
  } finally {
    // Best practice to always close the client connection
    if (client) {
      await client.close()
    }
  }
}

A few things to note in comparison with the MongoDB Node Driver:

Basic CRUD operations

Included in the following are links to the equivalent methods for the MongoDB Node Driver.

Inserting documents

Insert one document:

z
const result = await userCollection.insertOne({
  name: 'Jane Smith',
  email: 'jane@example.com',
  age: 30
})

Insert multiple documents:

const result = await userCollection.insertMany([
  { name: 'Alice', email: 'alice@example.com', age: 27 },
  { name: 'Bob', email: 'bob@example.com', age: 12 }
])

MongoDB Node Driver references:

Finding documents

Find one document:

const user = await userCollection.findOne({ email: 'john@example.com' });

Find all documents matching a filter (returns a cursor - see next section):

const cursor = userCollection.find({ age: { $gte: 18 } })

Find with projection, sort, skip and limit (returns a cursor - see next section):

const cursor = userCollection.find({ age: { $gte: 18 } })
  .project({ name: 1, email: 1 })
  .sort({ name: 1 })
  .skip(2)
  .limit(10)

MongoDB Node Driver references:

Cursor access patterns

Both find and aggregate return cursors.

Using toArray() - loads all results into memory:

const results = await cursor.toArray()

Using iteration - memory efficient:

while (await cursor.hasNext()) {
  const doc = await cursor.next();
  console.log(doc)
}

Using for await...of - most convenient:

for await (const doc of cursor) {
  console.log(doc)
}

Using streams:

const stream = cursor.stream();
stream.on('data', (doc) => {
  console.log(doc)
})

MongoDB Node Driver references:

Updating documents

Update one document:

const result = await userCollection.updateOne(
  { email: 'john@example.com' },
  { $set: { age: 31 } }
)

Replace one document:

const result = await userCollection.replaceOne(
  { email: 'john@example.com' },
  { name: 'Bob', email: 'bob@example.com', age: 12 }
)

Update multiple documents:

const result = await userCollection.updateMany(
  { age: { $lt: 18 } },
  { $set: { category: 'minor' } }
)

Find and update:

const updatedUser = await userCollection.findOneAndUpdate(
  { email: 'john@example.com' },
  { $set: { lastLogin: new Date() } },
  { returnDocument: 'after' }
)

MongoDB Node Driver references:

Deleting documents

Delete one document:

const result = await userCollection.deleteOne({ email: 'john@example.com' })

Delete multiple documents:

const result = await userCollection.deleteMany({ age: { $lt: 0 } })

Find and delete:

const deletedUser = await userCollection.findOneAndDelete({ email: 'john@example.com' })

MongoDB Node Driver references:

Bulk operations

Multiple operations in a single request:

const operations = [
  { insertOne: { document: { name: 'Alice' } } },
  { updateOne: { filter: { name: 'Bob' }, update: { $set: { age: 30 } } } },
  { deleteOne: { filter: { name: 'Charlie' } } }
]

const result = await collection.bulkWrite(operations)

MongoDB Node Driver references:

String and object Representations of the _id field

Every document in DocumentDB has a required _id field that acts as its unique identifier within a collection. Values for the _id field may be specified in the document or generated on the fly by the database server.

When a document with no value specified for the _id field is inserted into a collection, the database service will generate a unique value for the field of type ObjectId and add it to the document. So the following:

const result = await userCollection.insertOne({name: "Jane Smith"})

with a result something like this:

{
   "acknowledged" : true,
   "insertedId" : "56fc40f9d735c28df206d078"
}

When the _id field is represented as a string, for example in an HTTP request or text file, it needs to be converted to an ObjectId before using in a query filter. To retrieve the above document, for example, something like the following is required:

const {ObjectId} = require('bson')
const userDocument = await userCollection.findOne({ _id: new ObjectId("56fc40f9d735c28df206d078")})

with a result something like this:

{
    "name": "Jane Smith",
    "_id": "56fc40f9d735c28df206d078"
}

See the MongoDB docs for more details on the _id field:

Aggregates

Aggregates are a powerful tool for building complex queries.

Simple aggregate pipeline:

const pipeline = [
  { $match: { status: 'active' } },
  { $group: { _id: '$category', count: { $sum: 1 } } },
  { $sort: { count: -1 } }
]

const cursor = collection.aggregate(pipeline)

A geospatial example:

const nearbyStores = await stores.aggregate()
  .geoNear({
    near: {type: 'Point', coordinates: [-122.4194, 37.7749]}, // San Francisco
    distanceField: 'distance',
    maxDistance: 1000, // 1km radius
    spherical: true
  })
  .match({status: 'open'})
  .limit(10)
  .toArray()

MongoDB Node Driver references:

MongoDB and DocumentDB compatibility

The MongoDB 8.0 features supported by App Builder Database Storage (ABDB) are constrained by the AWS DocumentDB with MongoDB compatibility it is built on.

The primary reference for MongoDB compatibility is Supported MongoDB APIs, operations, and data types in Amazon DocumentDB. Note that ABDB uses version 8.0 of the MongoDB API. Some additional functional differences are documented at Functional differences: Amazon DocumentDB and MongoDB.

Beyond those imposed by AWS DocumentDB there are additional constraints imposed by the App Builder Database Storage itself. For example, the App Builder DB exposes far fewer administrative commands than either DocumentDB or MongoDB, because it is a multi-tenant offering.

The following sections highlight the differences between the App Builder Database Storage API and AWS DocumentDB.

Database commands

Database commands are not supported by the App Builder Database Storage.

Administrative and diagnostic features are limited to those provided by the aio-lib-db package and the db plugin for the aio cli.

AWS reference: Database commands.

Query and projection operators

Same as DocumentDB 8.0.

AWS reference: Query and projection operators

Update operators

Same as DocumentDB 8.0.

AWS reference: Update Operators

Geospatial

Same as DocumentDB 8.0.

AWS reference: Geospatial

Cursor methods

Although the general access patterns for cursors with App Builder Database Storage closely follow the DocumentDB/MongoDB model (see the previously mentioned Cursor Access Patterns) only a subset of other methods are supported, and these are only supported when initializing a cursor.

Supported methods when initializing a find cursor:

Supported methods when initializing an aggregate cursor:

Supported methods for reading both find and aggregate cursors:

AWS reference: Cursor Methods

Aggregation pipeline operators

At this time only the following stage operators are supported by App Builder Database:

Note: Support for more stage operators is coming soon.

All other pipeline operators are the same as for DocumentDB 8.0.

AWS reference: Aggregation pipeline operators

Data types

Same as DocumentDB 8.0.

AWS reference: Data types

Indexes and index properties

Same as DocumentDB 8.0.

AWS reference: Indexes and index properties

Support and contact information

For any questions, issues, or feedback regarding App Builder Database Storage, reach out to the team using one of the following: