Early Access - The Dcycle CLI is currently available for enterprise customers. Contact us to learn more about access.

CI/CD Pipeline Integration

Integrate Dcycle uploads into your existing CI/CD pipelines for automated, reliable operational data management. This guide covers GitHub Actions, GitLab CI, and general best practices.

Why CI/CD for Operational Data?

Manual Process	CI/CD Automated
Remember to upload monthly	Runs automatically on schedule
Human errors in data entry	Validated before upload
No audit trail	Full git history of changes
Single point of failure	Reliable, repeatable process

GitHub Actions

Basic Scheduled Upload

# .github/workflows/sustainability-upload.yml
name: Sustainability Data Upload

on:
  schedule:
    # Run every Monday at 6 AM UTC
    - cron: '0 6 * * 1'
  workflow_dispatch:  # Allow manual trigger

env:
  DCYCLE_API_KEY: ${{ secrets.DCYCLE_API_KEY }}
  DCYCLE_ORG_ID: ${{ secrets.DCYCLE_ORG_ID }}

jobs:
  upload:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install Dcycle CLI
        run: brew install dcy

      - name: Upload transport data
        run: |
          dcy logistics upload data/viajes.csv --type requests --yes
          dcy logistics upload data/consumos.csv --type recharges --yes

      - name: Verify upload
        run: |
          echo "Verifying recent uploads..."
          dcy logistics requests list --from $(date -d "7 days ago" +%Y-%m-%d) --format json | jq length

With Data Validation

Add validation before uploading to catch errors early:

# .github/workflows/sustainability-validated.yml
name: Validated Sustainability Upload

on:
  schedule:
    - cron: '0 6 1 * *'  # Monthly on 1st at 6 AM
  pull_request:
    paths:
      - 'data/**'  # Run on PRs that modify data files

env:
  DCYCLE_API_KEY: ${{ secrets.DCYCLE_API_KEY }}
  DCYCLE_ORG_ID: ${{ secrets.DCYCLE_ORG_ID }}

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          brew install dcy

      - name: Validate CSV structure
        run: |
          python scripts/validate_data.py data/viajes.csv --type requests
          python scripts/validate_data.py data/consumos.csv --type recharges

      - name: Check for required columns
        run: |
          # Verify viajes.csv has required columns
          head -1 data/viajes.csv | grep -q "date" || exit 1
          head -1 data/viajes.csv | grep -q "vehicle_plate" || exit 1
          head -1 data/viajes.csv | grep -q "origin" || exit 1
          head -1 data/viajes.csv | grep -q "destination" || exit 1
          echo "✓ All required columns present"

  upload:
    needs: validate
    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install Dcycle CLI
        run: brew install dcy

      - name: Upload data
        run: |
          dcy logistics upload data/viajes.csv --type requests --yes
          dcy logistics upload data/consumos.csv --type recharges --yes

      - name: Create upload summary
        run: |
          echo "## Upload Summary" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY
          echo "- **Date:** $(date)" >> $GITHUB_STEP_SUMMARY
          echo "- **Requests uploaded:** $(wc -l < data/viajes.csv)" >> $GITHUB_STEP_SUMMARY
          echo "- **Recharges uploaded:** $(wc -l < data/consumos.csv)" >> $GITHUB_STEP_SUMMARY

With Slack Notifications

Get notified on success or failure:

# .github/workflows/sustainability-notified.yml
name: Sustainability Upload with Notifications

on:
  schedule:
    - cron: '0 6 * * 1'

env:
  DCYCLE_API_KEY: ${{ secrets.DCYCLE_API_KEY }}
  DCYCLE_ORG_ID: ${{ secrets.DCYCLE_ORG_ID }}

jobs:
  upload:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Install Dcycle CLI
        run: brew install dcy

      - name: Upload data
        id: upload
        run: |
          dcy logistics upload data/viajes.csv --type requests --yes
          dcy logistics upload data/consumos.csv --type recharges --yes

          # Capture counts for notification
          REQUESTS=$(wc -l < data/viajes.csv)
          RECHARGES=$(wc -l < data/consumos.csv)
          echo "requests=$REQUESTS" >> $GITHUB_OUTPUT
          echo "recharges=$RECHARGES" >> $GITHUB_OUTPUT

      - name: Notify success
        if: success()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "✅ *Sustainability data uploaded successfully*"
                  }
                },
                {
                  "type": "section",
                  "fields": [
                    {"type": "mrkdwn", "text": "*Requests:* ${{ steps.upload.outputs.requests }}"},
                    {"type": "mrkdwn", "text": "*Recharges:* ${{ steps.upload.outputs.recharges }}"}
                  ]
                }
              ]
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

      - name: Notify failure
        if: failure()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "❌ Sustainability data upload failed! Check: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

GitLab CI

# .gitlab-ci.yml
stages:
  - validate
  - upload
  - notify

variables:
  DCYCLE_API_KEY: $DCYCLE_API_KEY
  DCYCLE_ORG_ID: $DCYCLE_ORG_ID

validate_data:
  stage: validate
  image: python:3.11
  script:
    - pip install pandas
    - python scripts/validate_data.py data/viajes.csv
    - python scripts/validate_data.py data/consumos.csv
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - if: $CI_PIPELINE_SOURCE == "web"
    - changes:
        - data/**

upload_data:
  stage: upload
  image: python:3.11
  needs: [validate_data]
  script:
    - brew install dcy
    - dcy logistics upload data/viajes.csv --type requests --yes
    - dcy logistics upload data/consumos.csv --type recharges --yes
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - if: $CI_PIPELINE_SOURCE == "web"

notify_success:
  stage: notify
  needs: [upload_data]
  script:
    - |
      curl -X POST "$SLACK_WEBHOOK" \
        -H 'Content-Type: application/json' \
        -d '{"text": "✅ Sustainability data uploaded successfully"}'
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: on_success

notify_failure:
  stage: notify
  needs: [upload_data]
  script:
    - |
      curl -X POST "$SLACK_WEBHOOK" \
        -H 'Content-Type: application/json' \
        -d '{"text": "❌ Sustainability upload failed: '$CI_PIPELINE_URL'"}'
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: on_failure

Validation Script

A reusable Python script for data validation:

# scripts/validate_data.py
import argparse
import sys
import pandas as pd
from datetime import datetime

SCHEMAS = {
    'requests': {
        'required': ['date', 'vehicle_plate', 'origin', 'destination'],
        'optional': ['distance_km', 'weight_kg', 'toc', 'client'],
        'date_columns': ['date'],
    },
    'recharges': {
        'required': ['date', 'vehicle_plate', 'fuel_type', 'quantity'],
        'optional': ['odometer', 'station'],
        'date_columns': ['date'],
    }
}

def validate_csv(filepath: str, data_type: str) -> list[str]:
    """Validate CSV file against schema"""
    errors = []
    schema = SCHEMAS.get(data_type)

    if not schema:
        return [f"Unknown data type: {data_type}"]

    try:
        df = pd.read_csv(filepath)
    except Exception as e:
        return [f"Failed to read CSV: {e}"]

    # Check required columns
    missing = set(schema['required']) - set(df.columns)
    if missing:
        errors.append(f"Missing required columns: {missing}")

    # Check for empty required fields
    for col in schema['required']:
        if col in df.columns:
            empty_count = df[col].isna().sum()
            if empty_count > 0:
                errors.append(f"Column '{col}' has {empty_count} empty values")

    # Validate date formats
    for col in schema.get('date_columns', []):
        if col in df.columns:
            for idx, val in df[col].items():
                try:
                    datetime.strptime(str(val), '%Y-%m-%d')
                except ValueError:
                    errors.append(f"Row {idx + 2}: Invalid date format '{val}' in {col}")
                    if len(errors) > 10:
                        errors.append("... (truncated)")
                        return errors

    # Check for duplicates
    if df.duplicated().any():
        dup_count = df.duplicated().sum()
        errors.append(f"Found {dup_count} duplicate rows")

    return errors

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('filepath', help='CSV file to validate')
    parser.add_argument('--type', required=True, choices=['requests', 'recharges'])
    args = parser.parse_args()

    errors = validate_csv(args.filepath, args.type)

    if errors:
        print(f"❌ Validation failed for {args.filepath}:")
        for error in errors:
            print(f"  - {error}")
        sys.exit(1)
    else:
        print(f"✓ {args.filepath} is valid")
        sys.exit(0)

Setting Up Secrets

GitHub Actions

Go to Repository → Settings → Secrets and variables → Actions
Add these secrets:
- DCYCLE_API_KEY: Your Dcycle API key
- DCYCLE_ORG_ID: Your organization ID
- SLACK_WEBHOOK (optional): For notifications

GitLab CI

Go to Settings → CI/CD → Variables
Add the same variables (mark as “Masked” for security)

Best Practices

Use --yes Flag

Always use --yes in CI/CD to skip interactive prompts that would hang the pipeline.

Validate First

Run validation in a separate job before upload. Fail fast on bad data.

Idempotent Uploads

Design pipelines to be safely re-runnable. Handle duplicates gracefully.

Monitor & Alert

Set up notifications for failures. Don’t let broken pipelines go unnoticed.

Next Steps

Multi-Org Reporting

Consolidated reporting across organizations

Logistics Pipeline

Daily logistics data automation

Getting Started

Commands

Examples

CI/CD Pipeline Integration

CI/CD Pipeline Integration

Why CI/CD for Operational Data?

GitHub Actions

Basic Scheduled Upload

With Data Validation

With Slack Notifications

GitLab CI

Validation Script

Setting Up Secrets

GitHub Actions

GitLab CI

Best Practices

Use --yes Flag

Validate First

Idempotent Uploads

Monitor & Alert

Next Steps

Multi-Org Reporting

Logistics Pipeline

​CI/CD Pipeline Integration

​Why CI/CD for Operational Data?

​GitHub Actions

​Basic Scheduled Upload

​With Data Validation

​With Slack Notifications

​GitLab CI

​Validation Script

​Setting Up Secrets

​GitHub Actions

​GitLab CI

​Best Practices

Use --yes Flag

Validate First

Idempotent Uploads

Monitor & Alert

​Next Steps

Multi-Org Reporting

Logistics Pipeline

CI/CD Pipeline Integration

Why CI/CD for Operational Data?

GitHub Actions

Basic Scheduled Upload

With Data Validation

With Slack Notifications

GitLab CI

Validation Script

Setting Up Secrets

GitHub Actions

GitLab CI

Best Practices

Next Steps