DPF

How to connect your analytics tools to data managed by DPF.

DPF stores your transformed data as Apache Iceberg tables and exposes them through a standardIceberg REST Catalogendpoint. Because Iceberg is an open table format with a standardized catalog API, you can query your data with any compatible engine — without moving it, copying it, or setting up proprietary connectors. The platform is API-first throughout, so the same data and catalog access used here is also what powers ourMCP server for AI agents.

For everything that speaks JDBC or ODBC — BI tools, SQL IDEs like DBeaver, ORMs, and existing database applications — DPF also provides aPostgreSQL wire-protocol gatewayatgateway.dpf-it.com:5432. Clients connect with the stock PostgreSQL driver (no custom driver needed), and your own PostgreSQL database can mount DPF as a linked server viapostgres_fdw. SeePostgreSQL Gatewaybelow.

Catalog federation and shortcuts currently supportread-onlyoperations (SELECT, time travel). Write operations (INSERT, UPDATE, DELETE) are not yet supported through federated access. To modify data, use the DPF API, thePostgreSQL Gateway(which supports readsandwrites), or connect directly via the REST Catalog with an open engine (Spark, Trino, PyIceberg).

curl -X POST https://api.dpf-it.com/connections \ -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "action": "create-connection", "workspaceId": "YOUR_WORKSPACE_ID", "type": "aws_s3", "roleArn": "arn:aws:iam::YOUR_ACCOUNT_ID:role/dpf-ingestion" }' # Response includes: # { # "success": true, # "data": { # "connectionId": "...", # "externalId": "3f9c2e1a-5b6d-4c7e-8f9a-0b1c2d3e4f5a", # "dpfPrincipalArn": "arn:aws:iam::442707444240:role/dpf-aws-connector", # "trustPolicy": { ... }, # "message": "..." # } # }

aws iam create-role \ --role-name dpf-ingestion \ --assume-role-policy-document '{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::442707444240:role/dpf-aws-connector"}, "Action": "sts:AssumeRole", "Condition": {"StringEquals": {"sts:ExternalId": "3f9c2e1a-5b6d-4c7e-8f9a-0b1c2d3e4f5a"}} }] }'

aws iam put-role-policy \ --role-name dpf-ingestion \ --policy-name dpf-s3-read \ --policy-document '{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": ["s3:GetObject", "s3:ListBucket"], "Resource": [ "arn:aws:s3:::YOUR_BUCKET", "arn:aws:s3:::YOUR_BUCKET/" ] }] }' # Narrower still: scope Resource to a prefix instead of the whole bucket, # e.g. "arn:aws:s3:::YOUR_BUCKET/exports/daily/", if DPF only needs one # subfolder — pair with a ListBucket s3:prefix condition to also keep the # bucket-level listing scoped to that same prefix.

curl -X POST https://api.dpf-it.com/connections \ -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "action": "test-connection", "workspaceId": "YOUR_WORKSPACE_ID", "connectionId": "YOUR_CONNECTION_ID" }'

curl -X POST https://api.dpf-it.com/job-triggers \ -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "action": "create-trigger", "workspaceId": "YOUR_WORKSPACE_ID", "type": "aws_s3", "specName": "web-logs", "connectionId": "YOUR_CONNECTION_ID", "s3Bucket": "YOUR_BUCKET", "s3Prefix": "exports/daily/", "frequency": {"unit": "daily", "hourOfDay": 6}, "dedupe": true }'

Because DPF holds no standing credential, you can cut off access at any time from your own AWS account — delete the IAM role, remove its trust statement, or revoke the permissions policy. The next scheduled run will fail and DPF automatically marks the connection as untested, blocking further use until it passestest-connectionagain.

For an SFTP connection, DPF generates an RSA-4096 keypair when the connection is created. Thepublic keyis returned so you can install it on your own SFTP server; theprivate keyis retained by DPF and is never returned by any API call.

- An active DPF workspace with full access
- An SFTP server you control, with a user account DPF will connect as
- Ability to edit that user's~/.ssh/authorized_keysfile on the server
- Create the connectionCallcreate-connectionwithtype: "sftp", the server's hostname, and the username DPF should connect as (defaults tosftpuserif omitted). DPF generates the keypair and returns the public key.

curl -X POST https://api.dpf-it.com/connections \ -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "action": "create-connection", "workspaceId": "YOUR_WORKSPACE_ID", "type": "sftp", "hostname": "sftp.example.com", "username": "sftpuser" }' # Response includes: # { # "success": true, # "data": { # "connectionId": "...", # "publicKey": "ssh-rsa AAAAB3NzaC1yc2EAAA... dpf-446655440000", # "message": "..." # } # }

sudo useradd -m sftpuser sudo -iu sftpuser mkdir -p ~/.ssh chmod 700 ~/.ssh

# As an administrator, become the connection's user (sftpuser here): sudo -iu sftpuser echo 'ssh-rsa AAAAB3NzaC1yc2EAAA... dpf-446655440000' >> ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys

curl -X POST https://api.dpf-it.com/connections \ -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "action": "test-connection", "workspaceId": "YOUR_WORKSPACE_ID", "connectionId": "YOUR_CONNECTION_ID" }'

curl -X POST https://api.dpf-it.com/job-triggers \ -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "action": "create-trigger", "workspaceId": "YOUR_WORKSPACE_ID", "type": "sftp", "specName": "web-logs", "connectionId": "YOUR_CONNECTION_ID", "frequency": {"unit": "daily", "hourOfDay": 6}, "dedupe": true, "preRules": "Process only .csv files under /outbound", "postRules": "Rename each processed file with a .done suffix" }'

jdbc:postgresql://gateway.dpf-it.com:5432/<namespace>?sslmode=require

psql "host=gateway.dpf-it.com port=5432 dbname=<namespace> user=<email> sslmode=require"

Any tool with a PostgreSQL connector works with the gateway. DBeaver is shown step-by-step below; DataGrip, Tableau, and ODBC-based tools follow the same pattern with the connection settings from the previous section.
- Create a new PostgreSQL connectionDatabase → New Database Connection → PostgreSQL. DBeaver will offer to download the official PostgreSQL JDBC driver automatically — accept it (any recent version works).
- Fill in the Main tabFieldValueHostgateway.dpf-it.comPort5432DatabaseYour namespace (e.g.633def9656c1)UsernameYour DPF account emailPasswordYour DPF account password
- Require SSLOn theSSLtab, checkUse SSLand set SSL mode torequire.
- Test and connectClickTest Connection, thenFinish. Open a SQL editor (SQL Editor → New SQL Script) and query your tables directly:

SELECT  FROM customers LIMIT 100; SELECT dpf_filename, COUNT() AS row_count FROM customers GROUP BY dpf_filename ORDER BY row_count DESC;

- DataGrip / JetBrains IDEs:create a PostgreSQL data source with the same values and set SSL to require. No driver property overrides are needed.
- Tableau / Power BI / other BI tools:use the genericPostgreSQLconnector with the same host, port, database, and credentials, with SSL required. Schema browsing and custom SQL both work.
- ODBC (psqlODBC):use the stock PostgreSQL Unicode ODBC driver withSSLmode=require:

Driver={PostgreSQL Unicode};Server=gateway.dpf-it.com;Port=5432;Database=<namespace>;Uid=<email>;Pwd=<password>;SSLmode=require;

CREATE EXTENSION IF NOT EXISTS postgres_fdw;

CREATE SERVER dpf FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host 'gateway.dpf-it.com', port '5432', dbname '633def9656c1');

CREATE USER MAPPING FOR CURRENT_USER SERVER dpf OPTIONS (user 'you@example.com', password 'your-dpf-password');

CREATE FOREIGN TABLE customers ( customer_id text, name text, email text, dpf_filename text, dpf_job text, dpf_ts timestamp ) SERVER dpf OPTIONS (schema_name 'default', table_name 'customers');

-- Read DPF data like any local table SELECT  FROM customers WHERE dpf_ts >= '2026-06-01' LIMIT 100; -- Join DPF data with your application's local tables SELECT c.customer_id, c.name, o.order_id, o.order_total FROM customers c -- foreign table (DPF) JOIN app.orders o -- local table ON o.customer_id = c.customer_id WHERE o.order_total > 1000;

Foreign tables are writable. INSERTs and fully pushed-down (“direct modify”) UPDATE/DELETE statements are forwarded to the DPF engine and committed per statement:

-- Insert into a DPF table from local data INSERT INTO customers (customer_id, name, email) SELECT id, full_name, email FROM app.new_signups; -- Direct-modify update (whole predicate pushed down) UPDATE customers SET email = lower(email) WHERE email  lower(email); -- Direct-modify delete DELETE FROM customers WHERE dpf_filename = 'bad_batch.csv';

UPDATE and DELETE work only when PostgreSQL can push the entire statement down to DPF (“direct modify”): no joins against local tables in the modify statement, and predicates using common operators/functions. Statements that fall back topostgres_fdw's row-by-row mode (which relies on Postgresctids) are rejected — Iceberg tables have noctid. Writes commitper statement: a largeINSERT ... SELECTfans out into batches that each commit independently, so a mid-batch failure leaves earlier rows committed.

postgres_fdwships filters, joins, and aggregates to the remote side when it considers them safe. The DPF engine is highly Postgres-compatible but not identical, so a pushed-down function it doesn't support will surface as a query error. Keep foreign-table predicates to common operators and functions; if a specific expression errors remotely, rewrite it or apply it locally over the fetched rows.

Today foreign tables are declared manually withCREATE FOREIGN TABLE. Support forIMPORT FOREIGN SCHEMA "default" FROM SERVER dpf INTO ...— which auto-generates all table definitions from the DPF catalog — is on the roadmap.

Setting Up the Glue Data Catalog Integration

AWS Glue Data Catalog supportscatalog federationfor remote Iceberg REST catalogs. This feature connects Glue to the DPF catalog endpoint, synchronizing metadata at query time so that AWS analytics engines can discover your tables without any data movement.

Once configured, Athena and Redshift see your DPF tables as if they were native Glue tables — with Lake Formation providing fine-grained access control on top.

- An active DPF workspace with at least one completed data load job
- Your DPF REST Catalog endpoint URL:https://api.dpf-it.com/iceberg/v1
- OAuth2 client credentials for catalog authentication (see Step 1 below to generate)
- AWS account with IAM permissions for Glue, Lake Formation, Athena, and Secrets Manager
- An IAM role with read access to the S3 location where DPF stores your data files

The following steps only need to be performedonce per workspace. After the Glue connection is configured, Athena and Redshift will automatically authenticate using the stored credentials whenever you run a query — no further configuration is needed.

Required: Complete Cross-Account Setup with DPF

Your table data is stored in DPF's AWS account. With Glue catalog federation, Athena and Redshift read those data files directly from Amazon S3 using your own AWS account's credentials — which means your account must be granted cross-account access to your workspace's storage before federated queries can return data. Metadata federation works as soon as you finish the steps below, but data-file reads will fail with an access-denied error until this authorization is in place. After creating your IAM role inStep 3.1(Create IAM role for Glue federation),contact uswith yourAWS account ID,workspace ID, yourclient ID, and theARN of the IAM role you created in Step 3.1. Our team will authorize that role for your specific workspace. This is a one-time step per workspace.

# Generate an API credential (one-time) curl -X POST https://api.dpf-it.com/oauth/clients \ -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "client_name": "Glue Data Catalog Federation" }' # Response contains clientId and clientSecret (save immediately!) # { # "success": true, # "data": { # "clientId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", # "clientSecret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...", # "clientName": "Glue Data Catalog Federation" # } # }

aws secretsmanager create-secret \ --name "dpf/api-credentials" \ --secret-string '{"USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET":"YOUR_CLIENT_SECRET"}' \ --region us-east-1

# 1. Create IAM role for Glue federation aws iam create-role \ --role-name DPFGlueFederationRole \ --assume-role-policy-document '{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"Service": "glue.amazonaws.com"}, "Action": "sts:AssumeRole" }] }' # 2. Attach policy granting access to Secrets Manager and S3 data files aws iam put-role-policy \ --role-name DPFGlueFederationRole \ --policy-name dpf-federation-access \ --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret", "secretsmanager:PutSecretValue" ], "Resource": ["arn:aws:secretsmanager:us-east-1:ACCOUNT:secret:dpf/api-credentials"] }, { "Effect": "Allow", "Action": ["s3:GetObject", "s3:ListBucket"], "Resource": [ "arn:aws:s3:::dpf-storage", "arn:aws:s3:::dpf-storage/" ] }, { "Effect": "Allow", "Action": ["glue:GetDatabase", "glue:GetDatabases", "glue:GetTable", "glue:GetTables"], "Resource": [""] } ] }' # 3. Create the Glue connection (wait ~10s for IAM propagation) aws glue create-connection \ --connection-input '{ "Name": "dpf-catalog-connection", "ConnectionType": "ICEBERGRESTCATALOG", "ConnectionProperties": { "INSTANCE_URL": "https://api.dpf-it.com/iceberg/v1", "ROLE_ARN": "arn:aws:iam::ACCOUNT:role/DPFGlueFederationRole" }, "AuthenticationConfiguration": { "AuthenticationType": "OAUTH2", "OAuth2Properties": { "OAuth2GrantType": "CLIENT_CREDENTIALS", "TokenUrl": "https://api.dpf-it.com/oauth/token", "OAuth2ClientApplication": { "UserManagedClientApplicationClientId": "YOUR_CLIENT_ID" } }, "SecretArn": "arn:aws:secretsmanager:us-east-1:ACCOUNT:secret:dpf/api-credentials" } }' \ --region us-east-1 # 4. Create the federated catalog aws glue create-catalog \ --name "dpf-data" \ --catalog-input '{ "FederatedCatalog": { "ConnectionName": "dpf-catalog-connection", "Identifier": "dpf" }, "CreateDatabaseDefaultPermissions": [], "CreateTableDefaultPermissions": [] }' \ --region us-east-1

aws lakeformation grant-permissions \ --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::ACCOUNT:role/YourAnalyticsRole"}' \ --resource '{"Catalog":{"Id":"dpf-data"}}' \ --permissions "ALL" \ --region us-east-1

aws glue get-databases \ --catalog-id "dpf-data" \ --region us-east-1

SELECT  FROM "dpf-data".<workspace_namespace>.<table_name> LIMIT 100;

SELECT customer_id, name, email, dpf_filename, dpf_job, dpf_ts FROM "dpf-data".my_workspace.customers LIMIT 100;

Time-travel query— view data as it existed at a specific point in time:

SELECT * FROM "dpf-data".my_workspace.customers FOR TIMESTAMP AS OF TIMESTAMP '2026-06-10 12:00:00';