Edit in GitHubLog an issue

Using Style and Structure Image References

Learn how to optionally pass in a source image to use as a style or structure reference for your generated images.

Prerequisites

  • Firefly API credentials. If you don't have them yet, first visit the Firefly Services Getting Started guide to obtain a client_id and client_secret.
  • Node.js installed on your machine and basic familiarity with JavaScript. Note: The code for this guide will make use of the Firefly REST APIs via Node.js, but could be written in any language, or with the SDK.

Working with Reference Images

Before digging in, you'll need to understand how to work with your existing assets as reference images. The APIs discussed in this guide allow you to reference images in two ways:

  • First, you can place your media on cloud storage and generate temporary readable URLs for them. However, these URLs may only be used with S3, Sharepoint, and Dropbox.

  • Secondly, you cna upload via the Firefly Upload API. This API lets you send a source image in either PNG, JPEG, or WebP format, and returns a unique ID that can be used in later calls, like the ones demonstrated below.

Using the Upload API requires a file, as well as the mime type, such as image/jpeg, image/png, image/webp. Below is an example function that demonstrates this. It assumes we already created access token using a CLIENT_ID and CLIENT_SECRET value. See the Create Your First Firefly Application Guide for more help on using your credentials in your code to obtain an access token.

Note: This function is used again in the examples below, and the complete code for this guide is shared at the bottom of this page.

Copied to your clipboard
async function uploadImage(filePath, fileType, id, token) {
    let stream = fs.createReadStream(filePath);
    let stats = fs.statSync(filePath);
    let fileSizeInBytes = stats.size;


    let upload = await fetch('https://firefly-api.adobe.io/v2/storage/image', {
        method:'POST', 
        headers: {
            'Authorization':`Bearer ${token}`, 
            'X-API-Key':id, 
            'Content-Type':fileType, 
            'Content-Length':fileSizeInBytes
        }, 
        duplex:'half', 
        body:stream
    });


    return await upload.json();
}

The result of this call is a JSON object containing the ID of the image:

Copied to your clipboard
{
    "images": [
        {"id": "9035f157-4105-4a63-913b-c120285dd799"}
    ]
}

If you haven't already gone through these tutorials, we recommend you refer to Create your First Firefly Application guide for a step-by-step walkthrough for:

  • Authenticating with getAccessToken()
  • Uploading images for use in uploadImage() calls
  • Downloading the generated results downloadFile()

Using a Style Reference Image

The first example uses a reference image to impact the style of the result. A standard prompt is used in a call to the Generate Images API -- both with and without a style reference image to compare the differences.

First, note the source image used for the style reference. Specifically, notice the color and fire attributes:

Style reference image

Before using this source image as a style reference in the Generate Images API call, you'll need to get an upload ID for it to pass in the style.imageReference.source.uploadId object. An example payload for the Generate Images API is provided below for reference:

Copied to your clipboard
{
    "numVariations":1,
    "prompt":"some prompt",
    "size":{
        "width":1792,
        "height":2304
    }, 
    "style":{
        "imageReference": {
            "source":{
                "uploadId":"The ID value of the uploaded style reference"
            }
        }
    }
}

Note: You could alternatively provide a presigned (temporary) URL from an image in cloud storage in the url property of the style.imageReference.source object. See the Generate Images API for details.

Next, we'll need utility code to get an access token, upload an image (via the Upload API), and download the result. An example is below:

Copied to your clipboard
import fs from 'fs';
import { Readable } from 'stream';
import { finished } from 'stream/promises';


/*
 Set the credentials based on environment variables.
*/
const CLIENT_ID = process.env.CLIENT_ID;
const CLIENT_SECRET = process.env.CLIENT_SECRET;


async function getAccessToken(id, secret) {
    const params = new URLSearchParams();


    params.append('grant_type', 'client_credentials');
    params.append('client_id', id);
    params.append('client_secret', secret);
    params.append('scope', 'openid,AdobeID,firefly_enterprise,firefly_api,ff_apis');
    
    let resp = await fetch('https://ims-na1.adobelogin.com/ims/token/v3', 
        { 
            method: 'POST', 
            body: params
        }
    );


    let data = await resp.json();
    return data.access_token;
}


async function uploadImage(filePath, fileType, id, token) {
    let stream = fs.createReadStream(filePath);
    let stats = fs.statSync(filePath);
    let fileSizeInBytes = stats.size;


    let upload = await fetch('https://firefly-api.adobe.io/v2/storage/image', {
        method:'POST', 
        headers: {
            'Authorization':`Bearer ${token}`, 
            'X-API-Key':id, 
            'Content-Type':fileType, 
            'Content-Length':fileSizeInBytes
        }, 
        duplex:'half', 
        body:stream
    });


    return await upload.json();
}


async function downloadFile(url, filePath) {
    let res = await fetch(url);
    const body = Readable.fromWeb(res.body);
    const download_write_stream = fs.createWriteStream(filePath);
    return await finished(body.pipe(download_write_stream));
}

Now, you'll see an example of a wrapper function for the Generate Images API call that optionally allows you to pass the id of an uploaded style reference image in the uploadId parameter:

Copied to your clipboard
async function generateImage(prompt, id, token, styleReference) {
    let body = {
        numVariations:1,
        prompt,
        size:{
            width:1792,
            height:2304
        }
    }


    if(styleReference) {
        body.style = {
            imageReference: {
                source: { 
                    uploadId: styleReference 
                }
            }
        };
    }


    let req = await fetch('https://firefly-api.adobe.io/v3/images/generate', {
        method:'POST',
        headers: {
            'X-Api-Key':id, 
            'Authorization':`Bearer ${token}`,
            'Content-Type':'application/json'
        }, 
        body: JSON.stringify(body)
    });


    return await req.json();
}

Finally, the demo code which authenticates, uploads the style reference, and then makes two calls using the same prompt, one with a style reference and one without:

Copied to your clipboard
let token = await getAccessToken(CLIENT_ID, CLIENT_SECRET);


let upload = await uploadImage('./styleRef.webp', 'image/webp', CLIENT_ID, token);
let styleReference = upload.images[0].id;


let prompt = 'A long-haired cat majestically riding a flying unicorn. The cat is wielding a rainbow shield and sword, pointing the swords tip outwards.';


// First, no style reference
let result = await generateImage(prompt, CLIENT_ID, token);
let fileName = `./without_style_reference.jpg`;
await downloadFile(result.outputs[0].image.url, fileName);


// Second, with a style reference
result = await generateImage(prompt, CLIENT_ID, token, styleReference);
fileName = `./with_style_reference.jpg`;
await downloadFile(result.outputs[0].image.url, fileName);

Given the prompt, here's the initial result with no style:

Without style reference image

And here's the result with the style reference:

With style reference image

Note the effect that the style reference image has on the generated result. Our new image has the same color scheme, lighting and flame effects that we provided in our reference.

Using a Structure Reference Image

The next feature you'll see is how to use an image as a structure reference. As you can imagine, this tells Firefly to use the source less as a reference on color schemes and styling, but to use the source image as a reference for the image's composition. First, as with the style reference example, once you've uploaded your image using the Firefly Upload API, you can reference it in the your API call:

Copied to your clipboard
{
    "numVariations":1,
    "prompt":"some prompt",
    "size":{
        "width":1792,
        "height":2304
    }, 
    "structure":{
        "imageReference": {
            "source":{
                "uploadId":"The ID value of the uploaded structure reference"
            }
        }
    }
}

Note, that similar to style, you can use cloud storage URLs as well. To demonstrate this, you once again use a simple wrapper for the Generate Images endpoint, but this time optionally accept the ID of an image to use as the structure reference:

Copied to your clipboard
async function generateImage(prompt, id, token, structureReference) {


    let body = {
        numVariations:1,
        prompt,
        size:{
            width:1792,
            height:2304
        }
    }


    if(structureReference) {
        body.structure = {
            imageReference: {
                source: { 
                    uploadId: structureReference 
                }
            }
        };
    }


    let req = await fetch('https://firefly-api.adobe.io/v3/images/generate', {
        method:'POST',
        headers: {
            'X-Api-Key':id, 
            'Authorization':`Bearer ${token}`,
            'Content-Type':'application/json'
        }, 
        body: JSON.stringify(body)
    });


    return await req.json();
}

For example, considering this as a structure reference:

Structure reference image

Note the size and position of the cat, the direction it's facing, and how it props its paws on a table. Now consider this prompt: "picture of a poodle with colorful fur looking majestic"

Without the structure reference, you see:

Without structure reference image

Now, compare it to the following generated image from using the structure reference provided:

With structure reference image

Again, the difference is striking and Firefly generates an image with a layout and composition which is similar to our reference image.

Complete Source Code

The complete source code containing the utilities for authentication, uploading, and downloading is provided below.

Copied to your clipboard
import fs from 'fs';
import { Readable } from 'stream';
import { finished } from 'stream/promises';


/*
  Set the credentials based on environment variables.
*/
const CLIENT_ID = process.env.CLIENT_ID;
const CLIENT_SECRET = process.env.CLIENT_SECRET;


async function getAccessToken(id, secret) {
    const params = new URLSearchParams();


    params.append('grant_type', 'client_credentials');
    params.append('client_id', id);
    params.append('client_secret', secret);
    params.append('scope', 'openid,AdobeID,firefly_enterprise,firefly_api,ff_apis');
    
    let resp = await fetch('https://ims-na1.adobelogin.com/ims/token/v3', 
        { 
            method: 'POST', 
            body: params
        }
    );


    let data = await resp.json();
    return data.access_token;
}


async function uploadImage(filePath, fileType, id, token) {
    let stream = fs.createReadStream(filePath);
    let stats = fs.statSync(filePath);
    let fileSizeInBytes = stats.size;


    let upload = await fetch('https://firefly-api.adobe.io/v2/storage/image', {
        method:'POST', 
        headers: {
            'Authorization':`Bearer ${token}`, 
            'X-API-Key':id, 
            'Content-Type':fileType, 
            'Content-Length':fileSizeInBytes
        }, 
        duplex:'half', 
        body:stream
    });


    return await upload.json();
}


async function downloadFile(url, filePath) {
    let res = await fetch(url);
    const body = Readable.fromWeb(res.body);
    const download_write_stream = fs.createWriteStream(filePath);
    return await finished(body.pipe(download_write_stream));
}


async function generateImage(prompt, id, token, structureReference) {
    let body = {
        numVariations:1,
        prompt,
        size:{
            width:1792,
            height:2304
        }
    }


    if(structureReference) {
        body.structure = {
            imageReference: {
                source: { 
                    uploadId: structureReference 
                }
            }
        };
    }


    let req = await fetch('https://firefly-api.adobe.io/v3/images/generate', {
        method:'POST',
        headers: {
            'X-Api-Key':id, 
            'Authorization':`Bearer ${token}`,
            'Content-Type':'application/json'
        }, 
        body: JSON.stringify(body)
    });


    return await req.json();
}


let token = await getAccessToken(CLIENT_ID, CLIENT_SECRET);


let upload = await uploadImage('./structureRef.webp', 'image/webp', CLIENT_ID, token);
let structureReference = upload.images[0].id;


let prompt = 'picture of a poodle with colorful fur looking majestic';


// First, no structure reference
let result = await generateImage(prompt, CLIENT_ID, token);
let fileName = `./without_structure_reference.jpg`;
await downloadFile(result.outputs[0].image.url, fileName);


// Second, with a structure reference
result = await generateImage(prompt, CLIENT_ID, token, structureReference);
fileName = `./with_structure_reference.jpg`;
await downloadFile(result.outputs[0].image.url, fileName);

Next Steps

While this guide demonstrated two powerful ways to influence Firefly when generating images, there's still more you can learn about to tweak what's generated from your API calls. Check out the other guides in the How-Tos section and the API Reference for more details.

Holly SchinskyNimitha Jalalcfjedimastersushiriot
Last updated 7/22/2024
Was this helpful?

Community

Support

Adobe Developer

  • Privacy
  • Terms of Use
  • Do not sell or share my personal information
  • AdChoices
Copyright © 2024 Adobe. All rights reserved.