Creating A Filter

We’re going to walk through how to tackle building a Valence Filter. What to think about, where to start, what’s important.

If you haven’t already read the primer on how Valence works overall, definitely read that first. Also make sure you have read Extensions and Configurability.

Ok, up to speed? Let’s get to work.

The Beginning

A Filter’s job is to take a peek at records as they are traveling from a data source to a data target. The Filter might add, change, or remove some values. A Filter might actually filter and decide that some records should not move on to delivery. There are many variations of what Filters do. If you want to take a peek at some existing Filters as you read this article, you can find them on GitHub.

Any Valence extension Apex class is going to implement various Valence-provided interfaces in order to be a cog in the Valence machine. These interfaces are like a contract: they define what Valence will provide to you, and also what Valence expects from you. It’s always good to start small, and implement just one or two interfaces then build up from there.

Unless you’re building a LinkSplitFilter for Link Splits, your Filter will implement TransformationFilter. This is our base Filter interface and is very simple:

global interface TransformationFilter {

        Boolean validFor(LinkContext context);

        void process(LinkContext context, List<RecordInFlight> records);
}

The first method, validFor() is asking our Filter if it would be appropriate to use for the Link described by the LinkContext. Most Filters return true from this method most of the time, but it’s valuable to have it to cover the exceptions. For example, the Relations Filter that comes with Valence helps to populate Lookup and Master-Detail fields, so that Filter doesn’t make sense to use on a Link going from Salesforce outbound to an external system. In this example, that Filter returns “false” if the target of the Link is not the local Salesforce org. Returning false means Valence will not show your Filter to users as an option when they are configuring that particular Link.

The second method is a Filter’s bread and butter. We are handed a context and some RecordInFlight instances and asked to interact with them. This method has a return type of void because the records parameter we are passed is a live collection. If we make changes or mark up these records, the next Filter in line will see those changes, and so on down the line ending with the target Adapter that will deliver the records.

Filter Order

Filters are selected and arranged in an order by the admin user when configuring a Link, and the selected Filters fire sequentially in that order during a Link run. Each Filter will receive the work product of the previous Filter(s), which means you can put together some sophisticated behaviors where Filters work together to combine functionality.

There is one special Filter that is required to be included for all Links called the Mappings Filter. Most of selecting Filter order is choosing which Filters will run before the Mappings Filter and which ones will run afterwards.

Incidentally, even Valence Filters that come packaged with the app implement the same interfaces you do and abide by the same behaviors. The Mappings Filter implements TransformationFilter and does its magic in its process() method!

Working with RecordInFlight

You’ll want to become very familiar with the methods available on each RecordInFlight instance so that you understand what sort of manipulations are available to you.

The first thing to understand is that a record has two internal maps:

  1. originalProperties - what the record looked like when it was received from the data source

  2. properties - what the record will look like when it is written to the target system

Typically Filters are modifying the properties map, but some Filters actually manipulate the originalProperties map. The one you work with will depend. Sometimes its based on whether your Filter is more familiar with the source system (and its field names) or the target system (and its field names). Other times its based on what other Filters you are trying to influence or support.

Here’s a scenario that will help clarify this idea of Filter collaboration:

The Mappings Filter is responsible for taking values in the originalProperties map and putting them into the properties map. Basically, any mapping that has been defined has a source field that exists in the originalProperties map, and Mappings will take that value and write it into the mapping target field in properties. So if your Filter runs after Mappings already ran, and modifies originalProperties, nothing is going to happen because we’ve already moved on to working with properties. However, if your Filter makes an adjustment to originalProperties before Mappings runs, then you can put stuff in front of Mappings for it to vacuum up and do its thing with.

This is exactly how the Constant Filter works. Admin users need to make sure it runs before Mappings in their ordered Filter list. It creates fields and puts them into originalProperties, and then users define mappings with those fields, and the Mappings Filter actually moves the constant value from the source field to whatever target field the user has chosen in their mapping.

So, in summary, the Constant Filter is complementing and enhancing something another Filter already does (in this case the Mappings Filter).

Separate from working with data, there are some other useful methods on RecordInFlight that Filters often invoke:

  • addError() - attach an error to this record, which will block it from being delivered and also surface the error in the user interface for admins to troubleshoot

  • addWarning() - attach a warning to this record, which does not block delivery, but will also show up in the interface for admins to see

  • ignore() - stop a record from being delivered, but semantically this isn’t so much an error as it is simply a record you know we’re not interested in delivering (also shows up in interface for admins to see…detecting a pattern?)

Participating in Schema Discovery

Valence has a rich understanding of schema, and as part of our obsession with giving admins great context about what they’re looking at, we want to make sure that the schema of their records that they see reflects changes due to the transformations and configurations they set up.

Since Filters can have a big impact on records and what they look like, it’s important that your Filter be able to describe what it does in a programmatic way. This is done by implementing SchemaAwareTransformationFilter.

The interface article has a really clear breakdown of how to properly inform Valence about your impact on schema.

Configuring Your Filter

There are two flavors of how your Filter might be configurable that are described below.

You will also want to read Configurability to learn how configurations are consumed by your Apex class, and how to surface a user interface for admins to do the actual configuring.

Configurable Per Mapping

Many Filters apply a transformation to or interpret the value of a specific field selected by the admin user. For this pattern the ConfigurablePerMappingFilter interface is perfect.

Implementing this interface allows admins to attach a configuration for your Filter to each Mapping they want to apply its logic to.

Examples:

  • Our guide on building a Cutoff Date Filter walks you through setting up a Filter where an admin attaches it to one or more date fields, and the configuration is selecting a date literal to use as a threshold. The Filter evaluates the value in the date field it is attached to, and if that date is before the configured cutoff that record is ignored.

  • The Relationships Filter that comes packaged with Valence is attached to mappings that hold unique identifiers from external systems, and walks admins through how to use that identifier to populate a Master-Detail or Lookup field in Salesforce.

Handling Errors

Every Filter may potentially run into an exception or error when working with records.

If the issue is isolated to a single record and you can proceed, flag that record:

try {
        manipulateRecord(record);
} catch(Exception e) {
        record.addError('Failed to flim flam the jibber jabber', e);
}

If the issue is catastrophic and your Filter cannot continue, throw a valence.FilterException:

throw new valence.FilterException('Things have gone very badly wrong.');

If you have a causing Exception, you can wrap it:

throw new valence.FilterException('Things have gone very badly wrong.', exceptionThatCausedIt);

Test Coverage

When you’re ready to start writing test coverage for your Filter, check out Writing Test Coverage.

Summary

There are a million and one different ways people transform records in their integrations. Rather than try to guess every possible combination and bake them into the engine, we’ve designed and exposed a robust transformation framework that we use internally and is also available for you.

Have a look through our open-source Filters to see if what you need already exists, or to get inspiration and guidance on how to build them.

Start small with TransformationFilter, and then layer in schema and configurability to make your Filter a sophisticated participant in the fabric of Valence!