> ## Documentation Index
> Fetch the complete documentation index at: https://code.dcycle.io/llms.txt
> Use this file to discover all available pages before exploring further.

# CI/CD Pipeline Integration

> Automate operational data uploads with GitHub Actions, GitLab CI, or other CI/CD tools

<Note>
  **Early Access** - The Dcycle CLI is currently available for enterprise customers.
  [Contact us](/docs/support) to learn more about access.
</Note>

# CI/CD Pipeline Integration

Integrate Dcycle uploads into your existing CI/CD pipelines for automated, reliable operational data management. This guide covers GitHub Actions, GitLab CI, and general best practices.

## Why CI/CD for Operational Data?

| Manual Process             | CI/CD Automated                |
| -------------------------- | ------------------------------ |
| Remember to upload monthly | Runs automatically on schedule |
| Human errors in data entry | Validated before upload        |
| No audit trail             | Full git history of changes    |
| Single point of failure    | Reliable, repeatable process   |

## GitHub Actions

### Basic Scheduled Upload

```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
# .github/workflows/sustainability-upload.yml
name: Sustainability Data Upload

on:
  schedule:
    # Run every Monday at 6 AM UTC
    - cron: '0 6 * * 1'
  workflow_dispatch:  # Allow manual trigger

env:
  DCYCLE_API_KEY: ${{ secrets.DCYCLE_API_KEY }}
  DCYCLE_ORG_ID: ${{ secrets.DCYCLE_ORG_ID }}

jobs:
  upload:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install Dcycle CLI
        run: brew install dcy

      - name: Upload transport data
        run: |
          dcy logistics upload data/viajes.csv --type requests --yes
          dcy logistics upload data/consumos.csv --type recharges --yes

      - name: Verify upload
        run: |
          echo "Verifying recent uploads..."
          dcy logistics requests list --from $(date -d "7 days ago" +%Y-%m-%d) --format json | jq length
```

### With Data Validation

Add validation before uploading to catch errors early:

```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
# .github/workflows/sustainability-validated.yml
name: Validated Sustainability Upload

on:
  schedule:
    - cron: '0 6 1 * *'  # Monthly on 1st at 6 AM
  pull_request:
    paths:
      - 'data/**'  # Run on PRs that modify data files

env:
  DCYCLE_API_KEY: ${{ secrets.DCYCLE_API_KEY }}
  DCYCLE_ORG_ID: ${{ secrets.DCYCLE_ORG_ID }}

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          brew install dcy

      - name: Validate CSV structure
        run: |
          python scripts/validate_data.py data/viajes.csv --type requests
          python scripts/validate_data.py data/consumos.csv --type recharges

      - name: Check for required columns
        run: |
          # Verify viajes.csv has required columns
          head -1 data/viajes.csv | grep -q "date" || exit 1
          head -1 data/viajes.csv | grep -q "vehicle_plate" || exit 1
          head -1 data/viajes.csv | grep -q "origin" || exit 1
          head -1 data/viajes.csv | grep -q "destination" || exit 1
          echo "✓ All required columns present"

  upload:
    needs: validate
    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install Dcycle CLI
        run: brew install dcy

      - name: Upload data
        run: |
          dcy logistics upload data/viajes.csv --type requests --yes
          dcy logistics upload data/consumos.csv --type recharges --yes

      - name: Create upload summary
        run: |
          echo "## Upload Summary" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY
          echo "- **Date:** $(date)" >> $GITHUB_STEP_SUMMARY
          echo "- **Requests uploaded:** $(wc -l < data/viajes.csv)" >> $GITHUB_STEP_SUMMARY
          echo "- **Recharges uploaded:** $(wc -l < data/consumos.csv)" >> $GITHUB_STEP_SUMMARY
```

### With Slack Notifications

Get notified on success or failure:

```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
# .github/workflows/sustainability-notified.yml
name: Sustainability Upload with Notifications

on:
  schedule:
    - cron: '0 6 * * 1'

env:
  DCYCLE_API_KEY: ${{ secrets.DCYCLE_API_KEY }}
  DCYCLE_ORG_ID: ${{ secrets.DCYCLE_ORG_ID }}

jobs:
  upload:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Install Dcycle CLI
        run: brew install dcy

      - name: Upload data
        id: upload
        run: |
          dcy logistics upload data/viajes.csv --type requests --yes
          dcy logistics upload data/consumos.csv --type recharges --yes

          # Capture counts for notification
          REQUESTS=$(wc -l < data/viajes.csv)
          RECHARGES=$(wc -l < data/consumos.csv)
          echo "requests=$REQUESTS" >> $GITHUB_OUTPUT
          echo "recharges=$RECHARGES" >> $GITHUB_OUTPUT

      - name: Notify success
        if: success()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "✅ *Sustainability data uploaded successfully*"
                  }
                },
                {
                  "type": "section",
                  "fields": [
                    {"type": "mrkdwn", "text": "*Requests:* ${{ steps.upload.outputs.requests }}"},
                    {"type": "mrkdwn", "text": "*Recharges:* ${{ steps.upload.outputs.recharges }}"}
                  ]
                }
              ]
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

      - name: Notify failure
        if: failure()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "❌ Sustainability data upload failed! Check: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}
```

***

## GitLab CI

```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
# .gitlab-ci.yml
stages:
  - validate
  - upload
  - notify

variables:
  DCYCLE_API_KEY: $DCYCLE_API_KEY
  DCYCLE_ORG_ID: $DCYCLE_ORG_ID

validate_data:
  stage: validate
  image: python:3.11
  script:
    - pip install pandas
    - python scripts/validate_data.py data/viajes.csv
    - python scripts/validate_data.py data/consumos.csv
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - if: $CI_PIPELINE_SOURCE == "web"
    - changes:
        - data/**

upload_data:
  stage: upload
  image: python:3.11
  needs: [validate_data]
  script:
    - brew install dcy
    - dcy logistics upload data/viajes.csv --type requests --yes
    - dcy logistics upload data/consumos.csv --type recharges --yes
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - if: $CI_PIPELINE_SOURCE == "web"

notify_success:
  stage: notify
  needs: [upload_data]
  script:
    - |
      curl -X POST "$SLACK_WEBHOOK" \
        -H 'Content-Type: application/json' \
        -d '{"text": "✅ Sustainability data uploaded successfully"}'
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: on_success

notify_failure:
  stage: notify
  needs: [upload_data]
  script:
    - |
      curl -X POST "$SLACK_WEBHOOK" \
        -H 'Content-Type: application/json' \
        -d '{"text": "❌ Sustainability upload failed: '$CI_PIPELINE_URL'"}'
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: on_failure
```

***

## Validation Script

A reusable Python script for data validation:

```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# scripts/validate_data.py
import argparse
import sys
import pandas as pd
from datetime import datetime

SCHEMAS = {
    'requests': {
        'required': ['date', 'vehicle_plate', 'origin', 'destination'],
        'optional': ['distance_km', 'weight_kg', 'toc', 'client'],
        'date_columns': ['date'],
    },
    'recharges': {
        'required': ['date', 'vehicle_plate', 'fuel_type', 'quantity'],
        'optional': ['odometer', 'station'],
        'date_columns': ['date'],
    }
}

def validate_csv(filepath: str, data_type: str) -> list[str]:
    """Validate CSV file against schema"""
    errors = []
    schema = SCHEMAS.get(data_type)

    if not schema:
        return [f"Unknown data type: {data_type}"]

    try:
        df = pd.read_csv(filepath)
    except Exception as e:
        return [f"Failed to read CSV: {e}"]

    # Check required columns
    missing = set(schema['required']) - set(df.columns)
    if missing:
        errors.append(f"Missing required columns: {missing}")

    # Check for empty required fields
    for col in schema['required']:
        if col in df.columns:
            empty_count = df[col].isna().sum()
            if empty_count > 0:
                errors.append(f"Column '{col}' has {empty_count} empty values")

    # Validate date formats
    for col in schema.get('date_columns', []):
        if col in df.columns:
            for idx, val in df[col].items():
                try:
                    datetime.strptime(str(val), '%Y-%m-%d')
                except ValueError:
                    errors.append(f"Row {idx + 2}: Invalid date format '{val}' in {col}")
                    if len(errors) > 10:
                        errors.append("... (truncated)")
                        return errors

    # Check for duplicates
    if df.duplicated().any():
        dup_count = df.duplicated().sum()
        errors.append(f"Found {dup_count} duplicate rows")

    return errors

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('filepath', help='CSV file to validate')
    parser.add_argument('--type', required=True, choices=['requests', 'recharges'])
    args = parser.parse_args()

    errors = validate_csv(args.filepath, args.type)

    if errors:
        print(f"❌ Validation failed for {args.filepath}:")
        for error in errors:
            print(f"  - {error}")
        sys.exit(1)
    else:
        print(f"✓ {args.filepath} is valid")
        sys.exit(0)
```

***

## Setting Up Secrets

### GitHub Actions

1. Go to Repository → Settings → Secrets and variables → Actions
2. Add these secrets:
   * `DCYCLE_API_KEY`: Your Dcycle API key
   * `DCYCLE_ORG_ID`: Your organization ID
   * `SLACK_WEBHOOK` (optional): For notifications

### GitLab CI

1. Go to Settings → CI/CD → Variables
2. Add the same variables (mark as "Masked" for security)

***

## Best Practices

<CardGroup cols={2}>
  <Card title="Use --yes Flag" icon="check">
    Always use `--yes` in CI/CD to skip interactive prompts that would hang the pipeline.
  </Card>

  <Card title="Validate First" icon="shield">
    Run validation in a separate job before upload. Fail fast on bad data.
  </Card>

  <Card title="Idempotent Uploads" icon="repeat">
    Design pipelines to be safely re-runnable. Handle duplicates gracefully.
  </Card>

  <Card title="Monitor & Alert" icon="bell">
    Set up notifications for failures. Don't let broken pipelines go unnoticed.
  </Card>
</CardGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Multi-Org Reporting" icon="sitemap" href="/cli/examples/multi-org-reporting">
    Consolidated reporting across organizations
  </Card>

  <Card title="Logistics Pipeline" icon="truck" href="/cli/examples/logistics-automation">
    Daily logistics data automation
  </Card>
</CardGroup>