Getting started with Data Hooks®

Flatfile's Data Hooks® are a useful data healing element to re-format, validate and/or correct data automatically during the import without the user having to correct manually. When used properly, they can be used for things like automatically reformatting area codes or country codes, removing special characters, validating emails against external data, and really anything else you can code up. Data Hooks® are the most powerful data healing element to date within Flatfile. There are three hooks that are available, field hooks (AKA column hooks), record hooks (AKA row hooks) and step hooks.

Before beginning, there are a couple considerations to be made when choosing which type of hook to use and how to use it, and those revolve around the order and event flow of the hooks. Below is a helpful diagram that shows the general flow for Flatfile, but it should be noted that registerFieldHook() runs first and only runs once after the 'matching' stage. After these run, registerRecordHook() will run on all records. This hook by default will also then run on individual records as they are updated. You might run into a scenario where you either don't want to run the record hooks on init or on change. This is possible, and you'll see how in the below section on record hooks.

Field Hooks

FlatfileImporter.registerFieldHook(field: string, values =>
  {
    /* function block */
  }
)

Field hooks run validation on a particular field (column) of data at the beginning of the matching stage. These hooks are run before record hooks and will only run once during the import process. These are best used to bulk edit a field or use an outside data source for validation. For example, say you want to verify the email addresses in a file are not already in your database, you could grab all the values in the column within a field hook, send them to your server and validate against your server and send back an error message with any that already exist in your system to display for the user.

In order to use field hooks, you call the field hook with FlatfileImporter.registerFieldHook("field_name", callback => {}) . Each field hook callback function needs to return an array where each item in that array is another array which corresponds to an individual record with the values and errors being the item index of 0 on that array and the original row number of the item being the index item 1 of the array. Here's an example of what the output data will look like and what the hook needs to have as a return value:

Name	Email
John Doe	john@doe.com
Jane Doe	jane@doe.com
Steve Smith	steve@something.com
Wayne Jones	wayne@something.com

In the above scenario, let's say we are sending the values to our server and returning an error for any emails that are already in the database. Let's assume that John and Steve's emails are already in the system

// values passed into the callback function
[
  ["john@doe.come", 1],
  ["jane@doe.com", 2],
  ["steve@something.com", 3],
  ["wayne@something.com", 4],
][
  // what should be passed back in this case
  ([
    {
      value: "john@doe.com", // not required if not changing value
      info: [
        {
          message: "Error message goes here",
          level: "info", // should be 'info', 'warning' or 'error'
        },
      ],
    },
    1,
  ],
  [
    {
      value: "steve@something.com", // not required if not changing value
      info: [
        {
          message: "Error message goes here",
          level: "info", // should be 'info', 'warning' or 'error'
        },
      ],
    },
    3,
  ])
];

With all that in mind, let's visualize the above within the context of making a server call. Quick note: while it's not required to use async/await with Data Hooks®, we recommend using it when working with an outside data source/API call. Note: For visual purposes, we included the value key above, but if the actual value of the data isn't changing, you don't need to pass this back.

import FlatfileImporter from "@flatfile/adapter";

const flatfileConfig = {
  type: "Field Hooks Example",
  fields: [
    {
      label: "Email",
      key: "email",
    },
  ],
};

const importer = new FlatfileImporter("license_key", flatfileConfig);

importer.registerFieldHook("email", async (values) => {
  let returnedEmails = await fetch("https://fake-api-call.com/emails", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(values),
  }).then((response) => response.json());
  // assuming the POST request returns the proper data format
  return returnedEmails;
});

With external data example:

In the below examples, you'll see us add a zero to the beginning of the first value in the data. Please notice in this example that we do not use the async/await syntax, but it is available for use.

import FlatfileImporter from "@flatfile/adapter";

const flatfileConfig = {
  type: "Field Hooks Example",
  fields: [
    {
      label: "Customer ID",
      key: "customer_code",
    },
  ],
};

importer.registerFieldHook("customer_code", (values) => {
  return [
    [
      {
        value: "0" + values[0][0],
        info: [
          {
            message: "padded the start with a 0",
            level: "warning",
          },
        ],
      },
      values[0][1],
    ],
  ];
});

Field Hooks additional notes:

• While we use the value key in each of the above examples, if you aren't changing the original value, this is not required, and we recommend to not include it in your returned values.
• We also use the info array with message and level in all of the examples above to provide a custom error message. This is not required to be given, however, please note that if you choose not to use this, there is still a standard "info" level message letting the user know that the data was automatically formatted.
• You can also call multiple field hooks per import. In order to do this, you would use the .registerFieldHook() method for each field you wish to use a hook.
• If you have registered the field hook and are not seeing the expected results in the import process, please check to make sure that the field name in your config matches the field provided in the .registerFieldHook() method and also that the returned data structure is correct.

Record Hooks

FlatfileImporter.registerRecordHook((record, index, mode) => { // function block})

Record hooks run validation on each record (row) of data and return the record with new data and/or error messaging for the user. These hooks run on init (meaning at the beginning of the "review" step) and then also on change (meaning when a record is updated during the "review" step). In order for this hook to work properly, you will call the function, passing in a callback function with the record (at minimum) as a parameter. Optional other parameters include the index and mode . The record is going to be a specific row of data. You can then use record.fieldName to work with a specific field on each record. You can use these hooks for single-field, multi-field or cross-field validation (examples of each below). You can use index to get the value's index within the data. You can use mode to differentiate between the hook being run on init and also on change .

These hooks can be used in conjunction with other validators (like regex) or can also be used to replace some of the regex validators and pre-format errors instead of having the user do it. They can be used to reformat, replace and validate data accuracy on init and change of a record during the "review" step.

info

Record hooks are processed in batches. This means that if you need to do something sequentially, you should rely on the index as the source of truth for the sequence.

Here are some examples of using the hooks. For context, here is a configuration we can use for all these hooks with the commented out section being where you would put your hooks.

import FlatfileImporter from "@flatfile/adapter";

const flatfileConfig = {
  type: "Contacts",
  fields: [
    {
      label: "Full Name",
      key: "name",
    },
    {
      label: "Email Address",
      key: "email",
    },
    {
      label: "City",
      key: "city",
    },
    {
      label: "State",
      key: "state",
    },
    {
      label: "Zip Code",
      key: "zip",
    },
  ],
};

const importer = new FlatfileImporter("License_Key", flatfileConfig);

importer.setCustomer({
  userId: "12345",
});

// Insert your
// Flatfile Data Hooks®
// here

const launchFlatfile = () => {
  importer.requestDataFromUser().then((results) => {
    importer.displayLoader();
    setTimeout(() => {
      importer.displaySuccess("Success Message!! YAY!");
      console.log(JSON.stringify(results.validData, null, 2));
    }, 1500);
  });
};

Single field validation example - zip code re-formatting (using the above example)

importer.registerRecordHook((record) => {
  let out = {};
  if (record.zip && record.zip.length < 5) {
    out.zip = {
      value: 0 + record.zip,
      info: [
        {
          message: "Padded zip code with a 0",
          level: "info",
        },
      ],
    };
  }
  return out;
});

Multi-field validation example

Cross-field validation example - if city and state aren't present, then zip code is required

importer.registerRecordHook((record) => {
  let out = {};
  if (!record.zip && !record.city && !record.state) {
    out.zip = {
      info: [
        {
          message: "Zip code required if city and state are not specified",
          level: "error",
        },
      ],
    };
    out.city = {
      info: [
        {
          message: "City and State required without Zip code",
          level: "error",
        },
      ],
    };
    out.state = {
      info: [
        {
          message: "City and State required without Zip code",
          level: "error",
        },
      ],
    };
  }
  return out;
});

Filtering event with mode - call mode and do something on "change" only - also do something on init only

importer.registerRecordHook((record, index, mode) => {
  let out = {};
  if (record.email && mode === "change") {
    out.email = {
      info: [
        {
          message: "The email has been changed by the user after upload",
          level: "warning",
        },
      ],
    };
  }
  return out;
});

Step Hooks

FlatfileImporter.registerStepHook("step", (stepObject) => { // function block})

note

The Step Hook feature is a beta access feature. To learn more or enable this feature, please reach out to our support team at support@flatfile.com.

Step Hooks allow you to pause and either proceed or remain on the current step while seeing the current state of the data in process. This Hook allows you to do validation based on headers you see matched in the data and opens up the opportunity to create fields that might not be in upload data set.

There are 4 current steps available for use in the registerStepHook process.

• upload - This is called immediately after a file is selected or dropped into the UI when starting an import.

importer.registerStepHook("upload", (stepObject) => {
  console.log(stepObject)
  return true
})

// expected console
{
  batchId: "string",
  count_columns: number, // number of columns in uploaded file
  count_columns_matched: number, // columns currently matched
  fileName: "string",
  fileSize: number, // size in bits
  fileType: "string",
  headers_matched: [], // array of headers matched
  headers_raw: [], // array of all headers | undefined
  sample: [] // array of up to 100 sample rows of data
}

• match - This runs at the beginning of the matching stage. Depending on if the headers are automatically mapped and detected, this may or may not have the exact same output as the upload step.

importer.registerStepHook("match", (stepObject) => {
  console.log(stepObject)
  return true
})

// expected console
{
  batchId: "string",
  count_columns: number, // number of columns in uploaded file
  count_columns_matched: number, // columns currently matched
  fileName: "string",
  fileSize: number, // size in bits
  fileType: "string",
  headers_matched: [], // array of headers matched
  headers_raw: [], // array of all headers | undefined
  sample: [] // array of up to 100 sample rows of data
}

• match_field - This hook is called anytime a user matches or removes a match from a field with a header from the upload file.

importer.registerStepHook("match_field", (stepObject) => {
  console.log(stepObject)
  return true
})

// expected console
{
  isCustom: boolean,
  sourceIndex: number, // header index in the upload file
  status: number,
  targetKey: "string" | undefined,
  targetType: "string" | undefined
}

• review - This hook would be called at the beginning of the review step.

importer.registerStepHook("review", (stepObject) => {
  console.log(stepObject)
  return true
})

// expected console
{
  batchId: "string",
  count_columns: number, // number of columns in uploaded file
  count_columns_matched: number, // columns currently matched
  fileName: "string",
  fileSize: number, // size in bits
  fileType: "string",
  headers_matched: [], // array of headers matched
  headers_raw: [], // array of all headers | undefined
  sample: [] // array of up to 100 sample rows of data
}