Transform
@flatfile/plugin-dedupe
Transform
@flatfile/plugin-dedupe
Dedupe records in a sheet via a sheet level custom action.
When embedding Flatfile, this plugin should be deployed in a server-side listener. Learn more
Parameters
jobOperationrequired
string
The jobOperation parameter specifies the name of the job operation to match
on.
opt.on
string
The on parameter specifieds which field key to match on.
opt.keep
'first' | 'last'
The keep option lets you choose whether to keep the first or last duplicate
record.
opt.custom
function
The custom parameter accepts a custom dedupe function. This will override
the keep parameter.
opt.debug
boolean
The debug parameter lets you toggle on/off helpful debugging messages for
development purposes.
API Calls
api.records.getapi.jobs.ackapi.records.deleteapi.jobs.failapi.jobs.complete
Imported NPM Packages
@flatfile/api@1.5.14+@flatfile/hooks@flatfile/listener@0.3.15+remeda@1.14.0+offers a set of utility functions for functional programming and data manipulation in JavaScript, providing a convenient way to work with arrays and objects.ts-pattern@5.0.4+The exhaustive Pattern Matching library for TypeScript with smart type inference.
Usage
An action with the operation name of “dedupe-email” must be configured on a Sheet in order to the plugin to be triggered.
install
npm i @flatfile/plugin-dedupe
import
import { dedupePlugin } from "@flatfile/plugin-dedupe";
// ... inside Workbook configuration
"actions": [
{
"operation": "dedupe-email",
"mode": "background",
"label": "Dedupe emails",
"description": "Remove duplicate emails"
}
]
// ...
// common usage
// Keep the last record encountered (from top to bottom) based on the`email` field key.
// Must have a Sheet level action specified with the operation name `dedupe-email`
listener.use(
dedupePlugin("dedupe-email", {
on: "email",
keep: "last",
})
);
// user specified dedupe function using RemedaJS
// must return a list a record id's for deletion
listener.use(
dedupePlugin("dedupe-email", {
custom: (records) => {
let uniques = new Set();
return R.pipe(
records,
R.reduce((acc, record) => {
const { value } = record.values["email"];
if (uniques.has(value)) {
return [...acc, record.id];
} else {
uniques.add(value);
return acc;
}
}, [] as Array<string>)
);
},
})
);