SourceAdapterForPull

This interface allows an Adapter to be the source in a Pull Link, fetching records in batches from a source data store.

This is a very commonly-used interface; you will likely also want to consider implementing NamedCredentialAdapter and SchemaAdapter.

Definition

/**
 * Implement this interface if your Adapter knows how to fetch its own records. The most common
 * example of this is an Adapter that calls an external system API to retrieve records.
 */
global interface SourceAdapterForPull extends SourceAdapter {

        /**
         * This method helps you to scale seamlessly to fetch large numbers of records. We do this by splitting requests
         * out into separate execution contexts, if need be.
         *
         * Valence will call planFetch() on your Adapter first, and then start calling fetchRecords(). The number of times
         * fetchRecords() is called depends on what you return from planFetch(). Every call to fetchRecords() will be in
         * its own execution context with a new instance of your Adapter, so you'll lose any state you have in your class.
         *
         * @param context Information about this Link and the current execution of it.
         *
         * @return An instance FetchStrategy that will tell the Valence engine what should happen next
         */
        FetchStrategy planFetch(LinkContext context);

        /**
         * Second, we will call this method sequentially with scopes you gave us in response to planPush(). We give you your
         * scope back so you can use it as needed.
         *
         * If you need to mark errors as failed or warning, use the addError() and addWarning() methods on RecordInFlight.
         *
         * @param context Information about this Link and the current execution of it.
         * @param scope A single scope instance from the List of scopes returned by planFetch()
         *
         * @return All of the records that have been updated since the timestamp passed inside LinkContext.
         */
        List<RecordInFlight> fetchRecords(LinkContext context, Object scope);
}

Tip

Any Adapter that implements this interface should also immediately implement SourceAdapterScopeSerializer. It’s easy to implement and it allows Links that use your source Adapter to run in parallel mode, which is much, much faster.

Behavior

Records are retrieved in a two-step process:

1. Planning

First, there is a planning step. This allows Valence and your Adapter to negotiate the exchange of records that is about to happen. Valence gives you a preview of the request for information so that you can decide how to proceed.

This is used mostly to handle any pagination or batching that needs to take place in order to avoid limits (on either side of the information exchange).

The purpose of planning is to figure out if batches will be needed, how many batches would be needed, and what information is needed by each batch to do its job.

Valence will call planFetch() and hand your Adapter a LinkContext instance. Two values in particular are going to be important to you here:

1. batchSizeLimit - the desired number of records returned by each call to fetchRecords().

2. lastSuccessfulSync - a timestamp for the last time records were successfully retrieved from your adapter. Use this to estimate (or query and count) how many total records you expect to send to Salesforce.

Use these two values, plus what you know about the external system you are connecting to, to plan the retrieval of records.

Planning Result: a FetchStrategy

There are a number of strategies you can use to most effectively retrieve records from the system your adapter is talking to.

The return value for planFetch() lets Valence know how best to interact with your Adapter to get the records we’re interested in. You have tremendous flexibility and control over this process.

Warning

In between execution contexts your Adapter instance is destroyed and any state is lost. If you want to preserve any information (api tokens, offsets, etc), put it inside scope objects that you can give to Valence to hand back to you later.

2. Fetching

After planning is complete, it is time to retrieve records from the external system that your Adapter talks to.

Depending on the FetchStrategy that you returned from planFetch(), different things may happen here. Read up on the FetchStrategy class to understand your options. Also take a look at ChainFetchAdapter.

Your fetchRecords() method is invoked with two arguments:

1. context - the LinkContext instance that contains general information you might need about the current sync that is in progress.

2. scope - A scope object you gave Valence to give back to you later, or null, depending on the circumstances. Use these to craft a request to the external system. Perhaps you stored offsets in your scopes so that you could paginate a large result set.

Note

Each time fetchRecords() is invoked it is called on a new instance of your Adapter class that is inside its own execution context. This means you have a fresh set of Salesforce Governor Limits to work within. To see the limits, look at the asynchronous limits here:

https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_gov_limits.htm

Building Your Query

To decide what your query to the external system should be you can look at:

1. context.linkSourceName - the name of the source table that should be queried (this will match one of the Table names your Adapter returned in getTables()).

2. context.suggestedQueryFields - a filtered list of fields that Valence thinks should be queried for, based on the user-configured mappings and excluding mappings against injected fields (such as those created by Filters).

3. Any details you gave yourself to work with in the scope (such as page number, page size, offset, etc)

Example

List<String> fieldsToSelect = new List<String>();
for(valence.FieldPath fieldPath : context.suggestedQueryFields) {
        fieldsToSelect.add(String.join(fieldPath.getSimplePath(),'.'));
}

String query = String.format('SELECT {0} FROM {1} OFFSET {2} LIMIT {3}', new List<Object>{
        String.join(fieldsToSelect,','),
        context.linkSourceName,
        myScope.offset,
        myScope.limit
});