Flatfile is a hosted solution by default. Optionally, we do provide advanced hosting or self-hosting configurations.
Deploying your Workbook
An easy way to manage and publish changes is by deploying your Workbook using the
npm run deploy command demonstrated in CLI Quickstart.
That is likely all you need.
Skip to next guide on embedding a Portal.
These next sections are not required for standard implementations.
Flatfile hosting regions
Flatfile offers multiple regional environments to support our customer's compliance and regulatory requirements. Data in these environments always remains in the host country.
|Region||AWS Region||Dashboard URL||API Endpoint|
Regional environments require additional configuration in Flatfile developer products.
When initializing a Flatfile session, use the
apiUrl parameter to set the correct API endpoint for your region.
See an example in our SDK
For access to our regional environments, please contact our Sales team.
- Delivery paths for the hosted solution
- Best Practices
- OnPrem Environment Variables
Delivery paths for the hosted solution
- Replicated - the easiest way to get started by installing our application in an existing Kubernetes cluster using KOTS
- Elastic Container Registry (ECR) - we provide container images via a private registry for those customers requiring a bespoke implementation
A minimal Flatfile environment requires several components:
|PostgreSQL||Primary application database for user and upload meta data|
|Redis||In-memory datastore used to manage asynchronous processing|
|Object storage||Blob storage for persisting files; we support S3, MinIO, Azure and GCP|
|MySQL||Database cluster for storing processed upload data|
Releases are published to our Stable channel in Replicated automatically from source control daily (if there are application changes). Customers can configure their deployment cadence.
For customers using our ECR, updated images are published every 30 days at a minimum. These images are generally tagged as
latest or with specific SHAs for beta features. Please reach out to Success for details on access to beta features.
This guide expands on best practices when hosting Flatfile in your own cloud.
Flatfile recommends using Kubernetes for durability and ease of deployment. The Flatfile application requires 3 deployments: API, worker, and app. The API and worker pods will share the same image.
- Version 1.21 (1.22 not yet supported)
- EKS launch template
- API replicas: 2
- Worker replicas: 2-6 (depending on size and frequency of uploads, more workers allows faster processing in parallel)
- App replicas: 1
- Recommended pod memory: 2GB
- Recommended pod CPU: 1vcpu
When deploying to AWS, it is recommended to use the RDS managed database products. Flatfile strongly recommends using IAM authentication to connect to the database from our API. This requires setting the authentication mechanism database user for the application:
# using the default refinery user:
GRANT rds_iam TO refinery;
AWS Aurora Recommendations
- Engine 11.13 or greater
- Size db.r5.large
It is strongly recommended to deploy Redis with encryption and an AUTH token. When constructing the connection string with an auth token, it should take the form:
: before the token is quite important - without it, the token is interpreted as a user name.
- Cluster mode OFF
- Size cache.m5.xlarge
- Engine 5.0.6
- Mutli AZ
The MySQL cluster is used to store processed data and as such requires more resources than the PG instance. The MySQL cluster requires a database to be created named
ephemeral. Additionally, the application is only compatible with Aurora 3.x versions (MySQL 8.x).
- Aurora 3.x (compatibility with MySQL 8.x)
- Size db.r5.xlarge
CORS policies affect two parts of hosting Flatfile:
- Object Storage (files uploaded to the platform are ingressed to S3/object storage)
- Flatfile Portal - the embedded Portal product is opened in an iframe by the application it is embedded in
Flatfile images are shipped without explicit CORS policies for Portal; if implementing CORS on your ingress, ensure that all domains that the Portal is served from are whitelisted.
CORS in Object Storage
Object storage buckets should have a CORS policy that allows uploads from domains the Portal is deployed on as well as exposing the
ETag header. The
ETag header makes it possible to perform multi-part file uploads which is critical for reasonable processing of large files as well as serving customers who may not have access to high speed internet.
"AllowedMethods": ["GET", "PUT", "POST", "DELETE"],
Flatfile ships with integrated support for a dashboard to view details on the queue and job processing which may be helpful in debugging. To enable the dashboard, set several environment variables for the API:
The dashboard will be available at
Route53 DNS Entries
Once the IRSA config is deployed and the ingress is running, go to Route53 in the console to add 2 records, one for the API and one for the App.
For each, create a new record:
- A type record
- Add the subdomain (app / api)
- Check the “alias” toggle
- Select “Alias to Application and Classic Load Balancer”
- Choose the appropriate region
- Select the load balancer created by TF
Health and Liveness Checks
Flatfile's API implements a health check that includes a database health check.
- Health check endpoint: [api host]/health (e.g.
- Liveness check endpoint: [api host]/build_info.json (e.g.
Several environment variables on our API control the sending of emails from our applications. Flatfile sends emails primarily for:
- Password resets for our dashboard
- Invitations to join a team or workspace as a collaborator
- Password-less logins as part of our "login by email" feature
Flatfile supports two specific vendors as well as generic SMTP for sending emails:
- AWS Simple Email Service
- Generic SMTP
# for AWS SES:
# for Sendgrid:
# All providers:
Flatfile uses websockets in a number of areas to push real-time events from our API to the SDK. For the frontend application, we use an environment variable to insert the websocket endpoint of the API:
FLATFILE_REFINERY_WS_URL='wss://api.us.flatfile.io' # this should always be your own domain
This variable is automatically configured for self-hosted customers deploying via Replicated. For customers using our image registry and deploying into a bespoke environment, please contact Success to configure this for your environment.
We are implementing a configuration option in the SDK to disable websocket support entirely.
OnPrem Environment Variables
This guide expands on environment variables when hosting Flatfile in your own cloud.
API-WEB / API-WORKER
Access key configuration
See the official DataDog Helm chart for additional configuration detail
MAIL_SEND_AS: Flatfile Inc. <email@example.com>
This guide will guide you on getting started on Replicated.
To get started on Replicated, your team can schedule time with Flatfile to speak with the Infrastructure team. We will create a customer portal specific to your company and provide the credentials and link.
After recieving your license from the customer portal, we can get started on installing the application via KOTS (Kubernetes Off the Shelf).
Begin by installing KOTS locally.
# Install the latest version of KOTS:
curl https://kots.io/install | bash
Connect to an existing cluster using
kubectl and install the application:
kubectl config use-context [cluster-context]
# Ensure that you have obtained your license file
# The Flatfile team will have forwarded your portal URL and password
# Install the Flatfile application:
kubectl kots install flatfile-kots
# follow the prompts -> you will be asked to provide a namespace and a password to the KOTS admin panel
# When the admin panel is ready you will be asked to upload the license file
# Continue updating the configs with install specific items (e.g. RDS credentials)
# In particular, if you bootstrapped the cluster using our On-Prem TF ensure that:
# 1. RDS IAM is used for Postgres auth
# 2. Use ALB for ingress, providing a VPC ID, certificate ARN, etc
# Verify the deployment in kubectl:
kubectl get all -n flatfile-kots # of whatever namespace has been configured
# returning to the KOTS admin dashboard
kubectl kots admin-console --namespace flatfile-kots
This will install the admin dashboard which you will be able to access locally. Open the dashboard in a browser and upload the license file downloa as well as the load balancing and ingress details.
This guide will guide you on getting started on Terraform.
If deploying into a bespoke environment, we recommend using an IaC approach. Flatfile Infra is happy to provide example Terraform for interested customers, including TF that can be used to create infrastructure dependencies for a new Kubernetes cluster.
Please reach out to our Success team for more information or contact Sales if you are not a current customer.