ETLR CLI

Deploy and manage ETLR workflows from your terminal. Perfect for local development, CI/CD pipelines, and team collaboration.

One Command Deploy

Create, update, and start workflows with a single command

Environment Management

Declare env vars in YAML, automatically pulled from your shell

Version Control

Automatic versioning with rollback capability

CI/CD Ready

Built for GitHub Actions, GitLab CI, and automation

Installation

Install the ETLR CLI using pip:

pip install etlr

Authentication

Get Your API Key

First, get your API key from the ETLR dashboard:

https://app.etlr.io/developer

Recommended: Environment Variable

Set your API key once in your shell configuration:

# Add to ~/.zshrc or ~/.bashrc
export ETLR_API_KEY=your_api_key_here

# Reload your shell
source ~/.zshrc  # or ~/.bashrc

Alternative: CLI Flag

Override or specify per-command:

etlr --api-key your_api_key_here list

Priority: CLI flag > ETLR_API_KEY environment variable

Quick Start

1. Create a Workflow

workflow.yaml

workflow:
  name: hello-world
  stage: dev
  input:
    type: http_webhook
  steps:
    - type: print
      message: "Hello from ${input.data}!"

2. Deploy It

etlr deploy workflow.yaml

Output:

Pushing workflow...
✓ Workflow created: hello-world/dev
Deploying workflow...
✓ Workflow deployed and running

3. Check Status

etlr status --name hello-world --stage dev

4. List All Workflows

etlr list

That's it! Your workflow is running.

Core Commands

List Workflows

View all workflows in your organisation:

etlr list

Deploy Workflow

Create/update and start a workflow in one command:

# From file
etlr deploy workflow.yaml

# From workflow.yaml in current directory
etlr deploy

# Override stage
etlr deploy workflow.yaml --stage prod

# With environment variables
etlr deploy workflow.yaml -e API_KEY=xxx -e LOG_LEVEL=debug

# By identifier (if already pushed)
etlr deploy --id <workflow-uuid>
etlr deploy --name my-workflow --stage prod

Get Workflow

View workflow details:

# By ID
etlr get --id <workflow-uuid>

# By name and stage
etlr get --name my-workflow --stage prod

Start Workflow

Start a workflow that was previously stopped:

etlr start --id <workflow-uuid>
etlr start --name my-workflow --stage prod

Stop Workflow

Stop a running workflow:

etlr stop --id <workflow-uuid>
etlr stop --name my-workflow --stage prod

Get Status

Check workflow health with colour-coded status:

etlr status --id <workflow-uuid>
etlr status --name my-workflow --stage prod

Status Indicators:

🟢 Green: Running normally
🟡 Yellow: Paused or warning
🔴 Red: Error or stopped

Delete Workflow

# With confirmation prompt
etlr delete --id <workflow-uuid>
etlr delete --name my-workflow --stage prod

# Skip confirmation (for scripts)
etlr delete --name my-workflow --stage prod --yes

Environment Variables

Declare environment variables in your workflow YAML, and the CLI automatically gathers them from your shell environment.

Basic Usage

1. Declare in workflow.yaml:

workflow:
  name: data-processor
  stage: dev
  
  environment:
    - name: API_KEY
      secret: true
    - name: DATABASE_URL
      secret: true
    - name: LOG_LEVEL
  
  input:
    type: http_webhook
  
  steps:
    - type: http_call
      url: https://api.example.com/data
      headers:
        Authorization: "Bearer ${env:API_KEY}"

2. Set values in your shell:

export API_KEY=sk-abc123...
export DATABASE_URL=postgres://...
export LOG_LEVEL=info

3. Deploy:

etlr deploy workflow.yaml

Output:

Environment variables:
  API_KEY: *** (secret)
  DATABASE_URL: *** (secret)
  LOG_LEVEL: info

Pushing workflow...
✓ Workflow deployed

Secret Masking

Mark sensitive values as secret: true to hide them in CLI output:

environment:
  - name: API_KEY
    secret: true       # Shows as ***
  - name: LOG_LEVEL
    secret: false      # Shows actual value
  - name: TIMEOUT      # Defaults to false

CLI Overrides

Override declared env vars or add new ones with -e flag:

# Override
etlr deploy workflow.yaml -e API_KEY=different-key

# Add new variable
etlr deploy workflow.yaml -e EXTRA_VAR=value

# Multiple overrides
etlr deploy workflow.yaml -e API_KEY=xxx -e LOG_LEVEL=debug

Missing Variables

If required env vars are missing, deployment fails with a helpful error:

$ etlr deploy workflow.yaml
Error: Missing required environment variables:
API_KEY, DATABASE_URL

Set them with:
  export API_KEY=value
  export DATABASE_URL=value

Or use -e flags:
  etlr deploy workflow.yaml -e API_KEY=value -e DATABASE_URL=value

Benefits:

✅ Self-documenting - Anyone reading YAML knows what's needed
✅ Validated - Missing vars caught before deployment
✅ No CLI clutter - No need for multiple -e flags
✅ Version controlled - Variable names (not values) in git
✅ Team friendly - New developers see requirements immediately

Stage Management

Stages (dev, staging, prod) let you run different versions of the same workflow in different environments.

Four Ways to Set Stage

Priority (highest to lowest):

--stage CLI flag (explicit override)
ETLR_STAGE environment variable (session default)
stage field in YAML (file default)
${env:STAGE} in YAML (dynamic from environment)

Method 1: Static in YAML (Simple)

name: my-workflow
stage: dev

etlr deploy workflow.yaml           # Uses 'dev'
etlr deploy workflow.yaml --stage prod  # Override to 'prod'

Use when: Single environment or simple setup

Method 2: ETLR_STAGE Variable (Recommended)

# Set once for your session
export ETLR_STAGE=dev

# Deploy without specifying stage
etlr deploy workflow.yaml           # Uses 'dev'

# Override when needed
etlr deploy workflow.yaml --stage prod

Add to ~/.zshrc for persistence:

export ETLR_STAGE=dev
export ETLR_API_KEY=your_dev_key

Use when: Local development with occasional prod deploys

Method 3: Dynamic in YAML (Flexible)

name: my-workflow
stage: ${env:STAGE}     # or ${env:STAGE, dev} with default

STAGE=dev etlr deploy workflow.yaml
STAGE=prod etlr deploy workflow.yaml

Use when: CI/CD or multiple environments

Method 4: CLI Flag (Most Explicit)

etlr deploy workflow.yaml --stage staging
etlr deploy workflow.yaml --stage prod

Use when: CI/CD pipelines or when you want explicit control

Best Practices by Environment

Local Development:

# ~/.zshrc
export ETLR_STAGE=dev
export ETLR_API_KEY=dev_key

# Just deploy
cd my-project
etlr deploy workflow.yaml

CI/CD:

# .github/workflows/deploy.yml
- name: Deploy to staging
  run: etlr deploy workflow.yaml --stage staging
  env:
    ETLR_API_KEY: ${{ secrets.STAGING_API_KEY }}

- name: Deploy to production
  run: etlr deploy workflow.yaml --stage prod
  env:
    ETLR_API_KEY: ${{ secrets.PROD_API_KEY }}

Team with .env files:

# .env.dev
ETLR_STAGE=dev
ETLR_API_KEY=dev_key

# .env.prod
ETLR_STAGE=prod
ETLR_API_KEY=prod_key

# Deploy
source .env.dev && etlr deploy workflow.yaml
source .env.prod && etlr deploy workflow.yaml

Version Management

ETLR automatically versions workflows on each deploy. You can list, view, and restore previous versions.

List Versions

etlr versions --id <workflow-uuid>

Output:

Versions for workflow abc-123:
  Version 3 (current) - 2025-12-17 14:23:10
  Version 2 - 2025-12-17 12:15:43
  Version 1 - 2025-12-16 09:30:22

Get Specific Version

etlr get-version --id <workflow-uuid> --version 2

Restore Previous Version

# With confirmation
etlr restore --id <workflow-uuid> --version 2

# Skip confirmation
etlr restore --id <workflow-uuid> --version 2 --yes

This creates a new version (e.g., version 4) with the content from version 2.

CI/CD Integration

GitHub Actions

name: Deploy Workflow

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install ETLR CLI
        run: pip install etlr

      - name: Deploy to staging
        if: github.ref == 'refs/heads/main'
        run: etlr deploy workflow.yaml --stage staging
        env:
          ETLR_API_KEY: ${{ secrets.STAGING_API_KEY }}

      - name: Deploy to production
        if: github.ref == 'refs/heads/main' && github.event_name == 'release'
        run: etlr deploy workflow.yaml --stage prod
        env:
          ETLR_API_KEY: ${{ secrets.PROD_API_KEY }}

GitLab CI

stages:
  - deploy

deploy_staging:
  stage: deploy
  script:
    - pip install etlr
    - etlr deploy workflow.yaml --stage staging
  environment:
    name: staging
  variables:
    ETLR_API_KEY: $STAGING_API_KEY

deploy_production:
  stage: deploy
  script:
    - pip install etlr
    - etlr deploy workflow.yaml --stage prod
  environment:
    name: production
  variables:
    ETLR_API_KEY: $PROD_API_KEY
  when: manual

Tips for CI/CD

Store API keys in secret managers (ETLR_API_KEY)
Use --stage flag for explicit environment targeting
Use --yes flag to skip confirmations
Set timeouts for deployment commands
Add status checks after deployment

Tips & Tricks

Per-Project Configuration

Use .env files or direnv for project-specific settings:

# .env
ETLR_API_KEY=project_specific_key
ETLR_STAGE=dev

# Load and deploy
source .env
etlr deploy workflow.yaml

Or use direnv to auto-load when entering directory:

# .envrc
export ETLR_API_KEY=project_key
export ETLR_STAGE=dev

Multiple Accounts

Switch between accounts with CLI flag:

# Production account
etlr --api-key $PROD_KEY deploy workflow.yaml

# Staging account
etlr --api-key $STAGING_KEY deploy workflow.yaml

Workflow Aliases

Create shell aliases for common workflows:

# ~/.zshrc
alias deploy-prod='etlr deploy workflow.yaml --stage prod'
alias deploy-dev='etlr deploy workflow.yaml --stage dev'
alias check-prod='etlr status --name my-workflow --stage prod'

Debugging

Enable verbose output (when available):

etlr --debug deploy workflow.yaml

Or check workflow logs via the web dashboard.

Web Interface vs CLI

Both methods work together seamlessly. Use whichever fits your workflow best:

Feature	Web Interface	CLI
Visual workflow building	Excellent	YAML only
Quick prototyping	Fast	Fast
Version control	Manual export	Native git workflow
CI/CD integration	Not available	Built for it
Batch operations	One at a time	Script multiple deploys
Team collaboration	Good for demos	Excellent with git
Learning curve	Beginner friendly	Requires terminal knowledge