plugin-dedupe

Dedupe records in a sheet via a sheet level custom action.


plugin-autocast

A plugin for automatically casting values in Flatfile.

plugin-automap

A plugin to provide automapping imported files for headless workflows.

plugin-constraints

A plugin for extending blueprint with external constraints

plugin-convert-currency

A Flatfile plugin for currency conversion using Open Exchange Rates API

plugin-convert-json-schema

A plugin for converting JSON Schema to Flatfile Blueprint and configuring a...

plugin-convert-openapi-schema

A plugin for converting OpenAPI schema to Flatfile Blueprint.

plugin-convert-sql-ddl

A plugin for converting SQL DDL into Flatfile Blueprint.

plugin-convert-translate

A Flatfile Listener plugin for field translation using the Google Translate...

plugin-convert-what3words

A Flatfile plugin for converting What3Words addresses to standard addresses...

plugin-convert-yaml-schema

A plugin for converting YAML Schema definitions to Flatfile Blueprint.

plugin-dedupe

Dedupe records in a sheet via a sheet level custom action.

plugin-delimiter-extractor

A plugin for parsing .delimiter files in Flatfile.

plugin-dxp-configure

A plugin for using DXP class-based configurations.

plugin-enrich-geocode

A Flatfile plugin for geocoding addresses using the Google Maps Geocoding A...

plugin-enrich-gpx

A Flatfile plugin for parsing GPX files and extracting relevant data

plugin-enrich-sentiment

A Flatfile plugin for sentiment analysis of text fields in records

plugin-enrich-summarize

A Flatfile plugin for text summarization and key phrase extraction

plugin-export-delimited-zip

A Flatfile plugin for exporting Workbooks to delimited files and zipping th...

plugin-export-pivot-table

A Flatfile plugin for generating pivot tables from sheet data and saving as...

plugin-export-workbook

A plugin for exporting data in Flatfile to Workbooks.

plugin-extract-html-table

A Flatfile plugin for extracting table data from HTML files

plugin-extract-markdown

A plugin for parsing markdown files in Flatfile.

plugin-import-faker

A Flatfile plugin that generates example records using Faker

plugin-import-llm-records

A Flatfile plugin that generates example records using AI

plugin-import-rss

A Flatfile plugin for importing RSS feed data

plugin-job-handler

A plugin for handling Flatfile Jobs.

plugin-json-extractor

A plugin for parsing json files in Flatfile.

plugin-markdown-extractor

A plugin for parsing markdown files in Flatfile.

plugin-pdf-extractor

A plugin for parsing PDF files in Flatfile.

plugin-record-hook

A plugin for running custom logic on individual data records in Flatfile.

plugin-rollout

A plugin for automatically rolling out new changes to workbooks in flatfile...

plugin-space-configure

A plugin for configuring a Flatfile Space.

plugin-space-configure-from-template

A plugin for configuring a Flatfile Space from a Space Template.

plugin-stored-constraints

A plugin for running stored constraints

plugin-validate-boolean

A Flatfile plugin for boolean validation with multi-language support

plugin-validate-date

A Flatfile plugin for normalizing date formats

plugin-validate-email

A Flatfile Listener plugin for email validation

plugin-validate-isbn

A Flatfile Listener plugin for ISBN validation with configurable options. V...

plugin-validate-number

A Flatfile Listener plugin for number validation

plugin-validate-phone

A validator plugin for phone number formatting on individual data records i...

plugin-validate-string

A Flatfile plugin for string configuration and validation

plugin-view-mapped

A plugin for making the view post mapping show only mapped columns.

plugin-webhook-egress

A plugin for egressing data from a Flatfile Workbook to a webhook.

plugin-xlsx-extractor

A plugin for parsing xlsx files in Flatfile.

plugin-xml-extractor

A plugin for parsing .xml files in Flatfile.

plugin-zip-extractor

A plugin for unzipping zip files and uploading content back in Flatfile.

util-extractor

A library containing common utilities and helpers for extractors.

util-file-buffer

A utility for extracting data from any file and making it available as a bu...

util-response-rejection

This plugin handles response rejections returned from an external source.

Installation


Installnpm i @flatfile/plugin-dedupe
Source: View source
Package:@flatfile/plugin-dedupe 103 installs

This plugin dedupes records in a Sheet via a Sheet-level action.

Event Type: listener.on('job:ready')

When embedding Flatfile, this plugin should be deployed in a server-side listener. Learn more

Parameters

jobOperation - string - (required)

The jobOperation parameter specifies the name of the job operation to match on.

opt.on - string

The on parameter specifies which field key to match on.

opt.keep - 'first' | 'last'

The keep option lets you choose whether to keep the first or last duplicate record.

opt.custom - function()

The custom parameter accepts a custom dedupe function. This will override the keep parameter.

opt.debug - boolean

The debug parameter lets you toggle on/off helpful debugging messages for development purposes.

API Calls

  • api.records.get
  • api.jobs.ack
  • api.records.delete
  • api.jobs.fail
  • api.jobs.complete

Usage

An action with the operation name of "dedupe-email" must be configured on a Sheet for the plugin to be triggered.

npm i @flatfile/plugin-dedupe
import { dedupePlugin } from "@flatfile/plugin-dedupe";
import { Flatfile } from '@flatfile/api'

export const contactsSheet: Flatfile.SheetConfig = {
  name: 'Contacts',
  slug: 'contacts',
  fields: [
    {
      key: 'firstName',
      type: 'string',
      label: 'First Name',
    },
    {
      key: 'lastName',
      type: 'string',
      label: 'Last Name',
    },
    {
      key: 'email',
      type: 'string',
      label: 'Email',
    }
  ],
  // Add a Sheet-level action here for the dedupe plugin
  actions: [
    {
      operation: "dedupe-email",
      mode: "background",
      label: "Dedupe emails",
      description: "Remove duplicate emails"
    }
  ]
}

JavaScript

listener.js

// common usage
// Keep the last record encountered (from top to bottom) based on the`email` field key.
// Must have a Sheet-level action specified with the operation name `dedupe-email`
listener.use(
  dedupePlugin("dedupe-email", {
    on: "email",
    keep: "last",
  })
);

// user specified dedupe function must return a list a record id's for deletion
listener.use(
  dedupePlugin("dedupe-email", {
    custom: (records) => {
      let uniques = new Set();
      let toDelete = [];

      records.forEach(record => {
        const { value } = record.values["email"];
        if (uniques.has(value)) {
          toDelete.push(record.id);
        } else {
          uniques.add(value);
        }
      });

      return toDelete;
    },
  })
);

TypeScript

listener.ts

// common usage
// Keep the last record encountered (from top to bottom) based on the`email` field key.
// Must have a Sheet-level action specified with the operation name `dedupe-email`
listener.use(
  dedupePlugin("dedupe-email", {
    on: "email",
    keep: "last",
  })
);

// user specified dedupe function must return a list a record id's for deletion
listener.use(
  dedupePlugin("dedupe-email", {
    custom: (records: Flatfile.RecordsWithLinks) => {
      let uniques = new Set();
      let toDelete = [];

      records.forEach(record => {
        const { value } = record.values["email"];
        if (uniques.has(value)) {
          toDelete.push(record.id);
        } else {
          uniques.add(value);
        }
      });

      return toDelete;
    },
  })
);