Version: v3.0

Deploy

Flatfile is a hosted solution by default. Optionally, we do provide advanced hosting or self-hosting configurations.

Highlights:

Deploying your Workbook
Hosting regions
Self-hosting

Deploying your Workbook

An easy way to manage and publish changes is by deploying your Workbook using the npm run deploy command demonstrated in CLI Quickstart.

That is likely all you need.

Skip to next guide on embedding a Portal.

Advanced Guides

These next sections are not required for standard implementations.

Hosting regions
Self-hosting

Flatfile hosting regions

Flatfile offers multiple regional environments to support our customer's compliance and regulatory requirements. Data in these environments always remains in the host country.

Current Offerings

Region	AWS Region	Dashboard URL	API Endpoint
UK	eu-west-2	app.uk0.flatfile.com	api.uk0.flatfile.com
Ireland	eu-west-1	app.eu0.flatfile.com	api.eu0.flatfile.com
Germany	eu-central-1	app.de0.flatfile.com	api.de0.flatfile.com
Australia	ap-southeast-2	app.aus0.flatfile.com	api.aus0.flatfile.com
Canada	ca-central-1	app.ca0.flatfile.com	api.ca0.flatfile.com

Configuration

Regional environments require additional configuration in Flatfile developer products.

When initializing a Flatfile session, use the apiUrl parameter to set the correct API endpoint for your region.

See an example in our SDK

For access to our regional environments, please contact our Sales team.

Self-hosting

Delivery paths for the hosted solution
Best Practices
OnPrem Environment Variables
Replicated
Terraform

Delivery paths for the hosted solution

Replicated - the easiest way to get started by installing our application in an existing Kubernetes cluster using KOTS
Elastic Container Registry (ECR) - we provide container images via a private registry for those customers requiring a bespoke implementation

Required Infrastructure

A minimal Flatfile environment requires several components:

Component	Purpose
PostgreSQL	Primary application database for user and upload meta data
Redis	In-memory datastore used to manage asynchronous processing
Object storage	Blob storage for persisting files; we support S3, MinIO, Azure and GCP
MySQL	Database cluster for storing processed upload data

Receiving Updates

Releases are published to our Stable channel in Replicated automatically from source control daily (if there are application changes). Customers can configure their deployment cadence.

For customers using our ECR, updated images are published every 30 days at a minimum. These images are generally tagged as latest or with specific SHAs for beta features. Please reach out to Success for details on access to beta features.

Best Practices

This guide expands on best practices when hosting Flatfile in your own cloud.

Kubernetes

Flatfile recommends using Kubernetes for durability and ease of deployment. The Flatfile application requires 3 deployments: API, worker, and app. The API and worker pods will share the same image.

Kubernetes Recommendations:

Version 1.21 (1.22 not yet supported)
EKS launch template eks_c5_xlarge
API replicas: 2
Worker replicas: 2-6 (depending on size and frequency of uploads, more workers allows faster processing in parallel)
App replicas: 1
Recommended pod memory: 2GB
Recommended pod CPU: 1vcpu

PostgreSQL

When deploying to AWS, it is recommended to use the RDS managed database products. Flatfile strongly recommends using IAM authentication to connect to the database from our API. This requires setting the authentication mechanism database user for the application:

# using the default refinery user:
GRANT rds_iam TO refinery;

AWS Aurora Recommendations

Engine 11.13 or greater
Size db.r5.large

Redis

It is strongly recommended to deploy Redis with encryption and an AUTH token. When constructing the connection string with an auth token, it should take the form:

rediss://:[auth token]@host:6379

The : before the token is quite important - without it, the token is interpreted as a user name.

Elasticache Recommendations

Cluster mode OFF
Size cache.m5.xlarge
Engine 5.0.6
Mutli AZ

MySQL

The MySQL cluster is used to store processed data and as such requires more resources than the PG instance. The MySQL cluster requires a database to be created named ephemeral. Additionally, the application is only compatible with Aurora 3.x versions (MySQL 8.x).

Aurora Recommendations

Aurora 3.x (compatibility with MySQL 8.x)
Size db.r5.xlarge

CORS

CORS policies affect two parts of hosting Flatfile:

Object Storage (files uploaded to the platform are ingressed to S3/object storage)
Flatfile Portal - the embedded Portal product is opened in an iframe by the application it is embedded in

Flatfile images are shipped without explicit CORS policies for Portal; if implementing CORS on your ingress, ensure that all domains that the Portal is served from are whitelisted.

CORS in Object Storage

Object storage buckets should have a CORS policy that allows uploads from domains the Portal is deployed on as well as exposing the ETag header. The ETag header makes it possible to perform multi-part file uploads which is critical for reasonable processing of large files as well as serving customers who may not have access to high speed internet.

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "PUT", "POST", "DELETE"],
    "AllowedOrigins": ["*"],
    "ExposeHeaders": ["ETag"]
  }
]

Queue Observability

Flatfile ships with integrated support for a dashboard to view details on the queue and job processing which may be helpful in debugging. To enable the dashboard, set several environment variables for the API:

ENABLE_BULL_BOARD=true
BULL_BOARD_USERNAME=flatfile
BULL_BOARD_PASSWORD=flatfile

The dashboard will be available at [api host]/admin/bull

Route53 DNS Entries

Once the IRSA config is deployed and the ingress is running, go to Route53 in the console to add 2 records, one for the API and one for the App.

For each, create a new record:

A type record
Add the subdomain (app / api)
Check the “alias” toggle
Select “Alias to Application and Classic Load Balancer”
Choose the appropriate region
Select the load balancer created by TF

Health and Liveness Checks

Flatfile's API implements a health check that includes a database health check.

Health check endpoint: [api host]/health (e.g. https://api.us.flatfile.io/health)
Liveness check endpoint: [api host]/build_info.json (e.g. https://api.us.flatfile.io/build_info.json)

Email Integration

Several environment variables on our API control the sending of emails from our applications. Flatfile sends emails primarily for:

Password resets for our dashboard
Invitations to join a team or workspace as a collaborator
Password-less logins as part of our "login by email" feature

Flatfile supports two specific vendors as well as generic SMTP for sending emails:

AWS Simple Email Service
Sendgrid
Generic SMTP

# for AWS SES:
MAIL_TRANSPORT=ses
# for Sendgrid:
MAIL_TRANSPORT=sendgrid
MAIL_SENDGRID_API_KEY=...
# SMTP:
MAIL_TRANSPORT=smtp
MAIL_SMTP_ENCRYPTION=null
MAIL_SMTP_HOST=smtp.mailtrap.io
MAIL_SMTP_PORT=587
MAIL_SMTP_USERNAME=...
MAIL_SMTP_PASSWORD=...
# All providers:
MAIL_SEND_AS='Flatfile <flatfile@flatfile.com>'

Websockets

Flatfile uses websockets in a number of areas to push real-time events from our API to the SDK. For the frontend application, we use an environment variable to insert the websocket endpoint of the API:

FLATFILE_REFINERY_WS_URL='wss://api.us.flatfile.io' # this should always be your own domain

This variable is automatically configured for self-hosted customers deploying via Replicated. For customers using our image registry and deploying into a bespoke environment, please contact Success to configure this for your environment.

Coming soon

We are implementing a configuration option in the SDK to disable websocket support entirely.

OnPrem Environment Variables

This guide expands on environment variables when hosting Flatfile in your own cloud.

API-WEB / API-WORKER

Access key configuration

ACCESS_KEY_BCRYPT_HASH_ROUNDS: "12"
ACCESS_KEY_EXCHANGE_TIMEBOX: "500"

Monitoring configuration

See the official DataDog Helm chart for additional configuration detail

DD_ENV: ca0-production
DD_LOGS_INJECTION: "true"
DD_PROFILING_ENABLED: "true"
ENABLE_BULL_BOARD=true
BULL_BOARD_USERNAME=flatfile
BULL_BOARD_PASSWORD=flatfile
METRICS_USERNAME=flatfile
METRICS_PASSWORD=flatfile
SENTRY_ENVIRONMENT: ca0-production
SENTRY_TRACES_SAMPLE_RATE: "0.1"

General Configuration

AWS_REGION: ca-central-1
BASE_URL: https://api.ca0.flatfile.com
FLATFILE_DASHBOARD_URL: https://app.ca0.flatfile.com/
FLATFILE_ENV: onprem
FLATFILE_REFINERY_WS_URL: wss://api.ca0.flatfile.com
GITHUB_CB_URL: https://api.ca0.flatfile.com/auth/github/callback
GRAPHQL_SCHEMA_IN_MEMORY: "true"
JWT_EXPIRES_HOURS: "12"
PROJECT_PATH: core/api
REDIRECT_URL: https://app.ca0.flatfile.com

Data Hooks®

LAMBDA_FACTORY_ARN: arn:aws:lambda:ca-central-1:xxxxxxxxxxx:function:lambdaFactory

Email sending

MAIL_SEND_AS: Flatfile Inc. <hello@ca0.flatfile.com>
MAIL_TRANSPORT: ses

Object storage

OBJECT_STORAGE_BUCKET: flatfile-prod-ca0-uploads
OBJECT_STORAGE_PROVIDER: s3
OBJECT_STORAGE_USE_ACCELERATION: "true"

Postgres configuration

PG_AUTH_STYLE: rds_iam
PGHOST: refinery.cluster-abcdefghikj.ca-central-1.rds.amazonaws.com
PGNAME: flatfile
PGPORT: "5432"
POSTGRES_CONNECTION_TIMEOUT: "5000"

SFTP configuration

SFTP_AUTH_TOKEN: ""
SFTP_BUCKET: flatfile-prod-ca0-transfer
SFTP_ROLE: arn:aws:iam::xxxxxxxxxxx:role/transfer-server-iam-role
SFTP_ROOT: sftp

TypeORM configuration

TYPEORM_LOGGER: simple-console
TYPEORM_LOGGING: error,schema,warn
TYPEORM_MAX_QUERY_EXECUTION_TIME: "2000"

Replicated

This guide will guide you on getting started on Replicated.

Prerequisites

To get started on Replicated, your team can schedule time with Flatfile to speak with the Infrastructure team. We will create a customer portal specific to your company and provide the credentials and link.

After recieving your license from the customer portal, we can get started on installing the application via KOTS (Kubernetes Off the Shelf).

Installation

Begin by installing KOTS locally.

# Install the latest version of KOTS:
curl https://kots.io/install | bash

Connect to an existing cluster using kubectl and install the application:

kubectl config use-context [cluster-context]

# Ensure that you have obtained your license file
# The Flatfile team will have forwarded your portal URL and password

# Install the Flatfile application:
kubectl kots install flatfile-kots

# follow the prompts -> you will be asked to provide a namespace and a password to the KOTS admin panel
# When the admin panel is ready you will be asked to upload the license file
# Continue updating the configs with install specific items (e.g. RDS credentials)
# In particular, if you bootstrapped the cluster using our On-Prem TF ensure that:
# 1. RDS IAM is used for Postgres auth
# 2. Use ALB for ingress, providing a VPC ID, certificate ARN, etc
# Verify the deployment in kubectl:
kubectl get all -n flatfile-kots # of whatever namespace has been configured

# returning to the KOTS admin dashboard
kubectl kots admin-console --namespace flatfile-kots

This will install the admin dashboard which you will be able to access locally. Open the dashboard in a browser and upload the license file downloa as well as the load balancing and ingress details.

Terraform

This guide will guide you on getting started on Terraform.

Recommendations

If deploying into a bespoke environment, we recommend using an IaC approach. Flatfile Infra is happy to provide example Terraform for interested customers, including TF that can be used to create infrastructure dependencies for a new Kubernetes cluster.

Support

Please reach out to our Success team for more information or contact Sales if you are not a current customer.

Highlights:​

Deploying your Workbook​

Flatfile hosting regions​

Current Offerings​

Configuration​

Self-hosting​

Delivery paths for the hosted solution​

Required Infrastructure​

Receiving Updates​

Best Practices​

Kubernetes​

PostgreSQL​

AWS Aurora Recommendations​

Redis​

Elasticache Recommendations​

MySQL​

Aurora Recommendations​

CORS​

CORS in Object Storage​

Queue Observability​

Route53 DNS Entries​

Health and Liveness Checks​

Email Integration​

Websockets​

Coming soon​

OnPrem Environment Variables​

API-WEB / API-WORKER​

Access key configuration​

Monitoring configuration​

General Configuration​

Data Hooks®​

Email sending​

Object storage​

Postgres configuration​

SFTP configuration​

TypeORM configuration​

Replicated​

Prerequisites​

Installation​

Terraform​

Recommendations​

Support​