Skip to main content
Version: latest

Configure your engine

Highlights:

What is an engine?

An engine, also called a data engine, is a server-side processor that accepts, validates, and transforms data from your users. You create and configure your engine in code using our SDK. You then deploy your engine to Flatfile's servers, where it listens for incoming data. When it receives data, it processes and validates it according to your rules, then makes it available to fetch via the Flatfile API.

The main components of an engine are:

  1. Sheets, also known as data templates, where you describe the shape of data you want to accept and the validations and transformations you want to perform
  2. Portals, the UI components that accept data from your users
  3. Workbooks, collections of associated Sheets and Portals that you deploy to Flatfile

Sheets

A Sheet is a data template, or a description of one type of data that you want your system to accept.

Sheets contain Fields, which are similar to columns in a database, and represent individual properties of the data record (e.g. "first name", "last name"). In addition to Fields, your Sheet can have template-wide hooks and options, which are defined in the Sheet options object.

// src/index.ts
// ...
// Define Sheets with fields
const Employees = new Sheet(
"Employees",
{
// Define Fields
firstName: TextField({
required: true,
label: "First Name",
}),
lastName: TextField({
required: true,
label: "Last Name",
}),
},
{
// Optional - Sheet options (see https://flatfile.com/docs/guides/workbooks/#sheet-options)
// allowCustomFields: true,
// recordCompute: (record) => {},
// batchRecordsCompute: async (payload: FlatfileRecords<any>) => {},
},
);
// ...

Fields

A Field object represents a column of data in a Sheet. They are similar to columns in a database. Fields have three defining features:

  • Type: the basic type of data stored in this field (e.g. text, number, boolean)
  • Options: additional properties describing the field (e.g. required, unique, default value)
  • Hooks: functions that clean, transform, and validate incoming data
// src/index.ts
// ...
fieldName: TextField({
label: "Name",
description:
"A favorite type of field like: TextField, NumberField, DateField, OptionField, BooleanField ...",
required: true,
unique: false,
default: "Did not awnser :(",
});
// ...

Field types

Field typeDescription
TextFielda string of text
NumberFielda numerical value
DateFielda date
OptionFielda field with a set of pre-defined values
BooleanFielda true / false field
LinkedFielda field that links two sheets together (available with Workspaces)

Field options

Field optionValue typeDescription
labelstringA label that is displayed to user
descriptionstringDescription of the field
primarybooleana primary key for a sheet that must be unique
requiredbooleanRequires a value to be valid.
uniquebooleanMust have a unique value to be valid.
defaultThe default value for this field
castfunctionTakes string or undefined and return either primitive type specified by the field, null, or throw an error.
computefunctionTransform a typed value to typed value
validatefunctionReturns annotations (e.g. warn, error or comments) about a typed value.
annotationsobjectAutomatically filled in messages when default or compute changes a value
stageVisibilityobjectControls what parts of mapping/review/export a field occurs in

See the SDK Object reference for more detail on Field types and options.

Field Hooks

Field Hooks are functions that help to clean up or validate the input data of a field

There are three options: validate, compute and cast

  • Validate takes a typed value and returns annotations. Confirms user-inputted data matches what you're expecting. You can display an error, warning, or comments to the cell informing your user about what needs to be fixed.

  • Compute runs simple logic on the user input value and returns a data value of the same type.

  • Cast takes a string or undefined field and return either the primitive type specified by the field, null, or will throw an error. When the default cast function cannot parse an incoming value, it will throw an error in the UI and retain the original value so users can edit that value. Each of our field types have a default casting, except for the OptionField.

salary: NumberField({
label: "Salary",
description: "Annual Salary in USD",
required: true,
validate: (salary: number) => {
const minSalary = 30_000;
if (salary < minSalary) {
return [new Message(`${salary} is less than minimum wage ${minSalary}`, "warn", "validate")];
}
},
});

Sheet Options

Sheet optionTypeDescription
allowCustomFieldsbooleanCreate additional fields from upload data when incoming column does not match any existing field
recordComputefunctionFunction that receives a row with all required fields fully present and optional fields typed optional?:string. Best used to compute derived values, can also be used to update existing fields.
batchRecordsComputefunctionAsynchronous function that is best for HTTP/API calls. External calls can be made to fill in values from external services. This takes records so it is easier to make bulk calls.

You'll learn more about Data Hooks in another guide.

Portals

A Portal is an embedded UI component for uploading data. Your users will use the Portal to upload their files. View a live example of a Portal here.

To create a Portal give it a name and assign it to a sheet defined in the workbook.

Example:

const EmployeesPortal = new Portal({
name: "EmployeesPortal",
sheet: "Employees",
});

See this guide for instructions on how to embed a Portal into your application.

Workbooks

A Workbook is collection of associated Sheets and Portals. Workbooks wrap up a set of Sheets and Portals into one object, which you will then deploy to Flatfile.

tip

The Quickstart guide shows you how to generate a starter Workbook that comes with examples.

To create a Workbook:

  1. Import dependencies.
  2. Define Sheets with Fields.
  3. Define Portals (optional).
  4. Export the Workbook.

This is what the starting Workbook scaffold looks like in the starter project at src/index.ts:

src/index.ts
// 1. Import dependencies
import { NumberField, Portal, Sheet, TextField, Workbook } from "@flatfile/configure";

/**
* 2. Define Sheets with Fields
* Define your Sheet configuration and Fields here, or import them:
* import { YourSheet } from './path-to-your-sheet/your-sheet.ts'
*/
const MySheet = new Sheet("MySheet", {
firstName: TextField(),
lastName: TextField(),
age: NumberField(),
});

/**
* 3. Define Portals (optional)
* Define your Portals here, or import them:
* import { YourPortal } from './path-to-your-portal/your-portal.ts'
*/
const MyPortal = new Portal({
name: "MyPortal",
sheet: "MySheet",
});

// 4. Export Workbook - Update to reference your Workbook with Sheet(s) and Portal(s)
export default new Workbook({
name: "MyWorkbook",
namespace: "my-workbook",
portals: [MyPortal],
sheets: {
MySheet,
},
});

To deploy your Workbook to Flatfile, export it from your project's src/index.ts file and run the following command:

npx flatfile publish

Alternatively, you can specify a path to any file where a Workbook is exported:

npx flatfile publish ./path-to-your-workbook

To deploy the FullExample Workbook in the starter project from the quickstart guide, you would run:

npx flatfile publish ./examples/workbooks/FullExample.ts

Field examples

TextField

A field that represents a string of text. Note: You don't need to supply all the field options. In this case we want the fullName field to be unique.

fullName: TextField({
required: true,
unique: true,
description: "Given name",
});

NumberField

A field that represtents a number. This example uses a validate function to determine if value is valid for this field. In this example, salary must be greater than or equal to $30,000, and provides a message to be displayed when it's not valid.

salary: NumberField({
label: "Salary",
description: "Annual Salary in USD",
required: true,
validate: (salary: number) => {
const minSalary = 30_000;
if (salary < minSalary) {
return [new Message(`${salary} is less than minimum wage ${minSalary}`, "warn", "validate")];
}
},
});

DateField

A date field converts most string values into a Date object. A warning validation appears if the input value doesn't match to a date.

hireDate: DateField({
required: true,
description: "hire date",
});

OptionField

A field with a set of pre-defined values. Incoming values are matched to the options provided in options based on the strategy provided in matchStrategy. Strategy options are:

fuzzy (default): compares the record to the list of category options and labels to find and recommend approximate matches.

exact: requires a 100% match between the value in the field and the category option in your template. This is case insensitive, so “sales” and “Sales” would still count as a match.

department: OptionField({
label: "Department",
matchStrategy: "exact", // default is: fuzzy
options: {
engineering: "Engineering",
hr: "People Ops",
sales: "Revenue",
},
});

BooleanField

A field that is true or false.

isInactive: BooleanField({
required: false,
description: "Is the employee inactive?",
});

LinkedField

A field that links two sheets together. The sheet attribute specifies which sheet this field links to.

Note

LinkedField can only be implemented in Workspaces. Please contact support@flatfile.io if you are not using Workspaces and would like access to this feature.

Note: You may optionally set upsert: false which disables the default upsert behavior and will display an error on this sheet if a value imported in the LinkedField does not exist on the parent sheet. For more information about our Relational Data feature, visit our Relational Data Guide.

email: LinkedField({
unique: true,
label: "Email",
primary: true,
sheet: BaseSheet,
upsert: false,
});

Here is the sheet linked to in the above field.

Note that the linked sheet must contain at least one field marked as unique.

Note: You must specify a previewFieldKey as an additional sheet option to define which field will be used as a human-readable preview for the linked record.

const BaseSheet = new Sheet(
"BaseSheet",
{
email: TextField({
unique: true,
primary: true,
}),
firstName: TextField("First"),
middleName: TextField("Middle"),
lastName: TextField("Last"),
},
{
previewFieldKey: "email",
},
);