Record Hook Plugin
Execute custom logic on individual data records within a Flatfile Sheet with support for validation, transformation, enrichment, and cleaning.
The Record Hook plugin provides a way to execute custom logic on individual data records within a Flatfile Sheet. It works by attaching a listener to the commit:created
event, which fires when new data is added or existing data is updated.
This plugin is ideal for a variety of use cases, including:
- Data Validation: Implementing complex validation rules that go beyond Flatfile’s built-in validators, such as checking for valid email formats or ensuring conditional field requirements.
- Data Transformation: Modifying data on the fly, like capitalizing names, standardizing formats, or setting default values.
- Data Enrichment: Augmenting records with additional information from external sources.
- Data Cleaning: Removing whitespace, correcting common typos, or normalizing values.
The plugin offers two main functions: recordHook
for processing records one-by-one, and bulkRecordHook
for processing records in batches, which is more efficient for large datasets.
Installation
Install the plugin using npm:
Configuration & Parameters
recordHook
Processes individual records one at a time. The provided callback function is executed for each record in the commit.
Parameters:
sheetSlug
(string, required): The slug of the Sheet you want the hook to apply tocallback
(function, required): A function that contains your custom logic. It receives the record and the event object as argumentsoptions
(object, optional): Configuration optionsconcurrency
(number, default: 10): Controls how many individual record handlers can run in paralleldebug
(boolean, default: false): When set totrue
, enables verbose logging to the console
bulkRecordHook
Processes records in batches (chunks). The provided callback function is executed for each chunk of records.
Parameters:
sheetSlug
(string, required): The slug of the Sheet you want the hook to apply tocallback
(function, required): A function that contains your custom logic. It receives an array of records and the event object as argumentsoptions
(object, optional): Configuration optionschunkSize
(number, default: 10000): Specifies the number of records to process in each batch. Maximum recommended value is 5000parallel
(number, default: 1): Specifies how many chunks of records to process in paralleldebug
(boolean, default: false): When set totrue
, enables verbose logging to the console
Default Behavior:
By default, the plugin processes records for the specified sheetSlug
after a commit is created. bulkRecordHook
processes all records in a single chunk sequentially (parallel: 1
). recordHook
processes up to 10 records concurrently. Debug logging is disabled.
Usage Examples
Basic Single Record Processing
Basic Bulk Record Processing
Configuration Example
Advanced Usage - Email Validation
Troubleshooting
The most effective way to troubleshoot issues is to set the debug: true
option in the plugin configuration. This will print detailed logs to the console, showing the timing and status of each major step: fetching data, running the handler, filtering modified records, and updating records. This can help identify performance bottlenecks or see why records are not being updated as expected.
Notes
Special Considerations
- The plugin operates on the
commit:created
event. This means it runs after Flatfile’s initial parsing and validation but before the data is finalized. - To save changes, your callback function must return the modified record (for
recordHook
) or the array of modified records (forbulkRecordHook
). If you returnnull
,undefined
, or nothing, your changes will not be persisted. - The plugin intelligently detects which records have actually been modified and only sends those records back to the Flatfile API for an update, optimizing performance.
Limitations
- The
chunkSize
forbulkRecordHook
has a recommended maximum of 5000 records to ensure stability and performance.
Error Handling
- The plugin is designed to be robust. It wraps the execution of the user-provided callback in a
try...catch
block. - If an error is thrown inside your callback, the plugin will log the error message to the console with a
[FATAL]
prefix. - Crucially, it will then ensure the commit is marked as complete with the Flatfile API, preventing the import process from stalling due to an error in custom code.