> ## Documentation Index
> Fetch the complete documentation index at: https://code.dcycle.io/llms.txt
> Use this file to discover all available pages before exploring further.
# How Bulk Ingestion Works
> How every v2 bulk route checks, writes and reports a batch: synchronous and asynchronous routes, all or nothing, dry run, the Idempotency-Key, the perimeter and ingest jobs
The v2 write routes take a batch: up to thousands of records to create, or of ids to delete. Every route handles it the
same way. It checks every record first and writes all of them or none. Then it either answers at once with what it
created (`201 Created`), or answers `202 Accepted` with an **ingest job** that calculates (or deletes) the records in
the background, and you follow that job until it finishes.
**Beta.** Part of the bulk ingest API, in beta. The contract may still change before general availability.
## Synchronous and asynchronous routes
A route is **asynchronous** when its records need work after the request: calculating their emissions, or deleting
them with everything that depends on them. It answers `202 Accepted` with the job, and its `Location` header is
`/v2/ingest-jobs/{id}`. The job processes the records in chunks; see [Following a job](#following-a-job).
A route is **synchronous** when its records need nothing calculated. It answers `201 Created` with the created
resources, in submission order, each with its `source_row_index` and `client_row_id`. It also records a job, born
`completed` with 0 of 0 chunks, which sends no webhook: [List Ingest Jobs](/api-reference/ingest-jobs/list) shows it,
and its [items](/api-reference/ingest-jobs/list-items) map each record to the resource created.
| Route | Answer | `entity_type` | `operation` | Records per request |
| - | - | - | - | - |
| [`POST /v2/wastes`](/api-reference/wastes/create-v2) | `202` | `wastes` | `create` | 1 to 5,000 |
| [`POST /v2/wastes/delete`](/api-reference/wastes/delete-v2) | `202` | `wastes` | `delete` | 1 to 5,000 |
| `POST /v2/invoices/` [`electricity`](/api-reference/invoices/electricity-create-v2), [`fuels`](/api-reference/invoices/fuels-create-v2), [`refrigerants`](/api-reference/invoices/refrigerants-create-v2), [`water`](/api-reference/invoices/water-create-v2), [`district-heating`](/api-reference/invoices/district-heating-create-v2), [`district-cooling`](/api-reference/invoices/district-cooling-create-v2), [`process`](/api-reference/invoices/process-create-v2) | `202` | `invoices` | `create` | 1 to 5,000 |
| [`POST /v2/invoices/delete`](/api-reference/invoices/delete-v2) | `202` | `invoices` | `delete` | 1 to 5,000 |
| [`POST /v2/facilities`](/api-reference/facilities/create-v2) | `201` | `facilities` | `create` | 1 to 1,000 |
| [`POST /v2/facilities/delete`](/api-reference/facilities/delete-v2) | `202` | `facilities` | `delete` | 1 to 1,000 |
| [`POST /v2/purchases`](/api-reference/purchases/create-v2) | `202` | `purchases` | `create` | 1 to 5,000 |
| [`POST /v2/purchases/delete`](/api-reference/purchases/delete-v2) | `202` | `purchases` | `delete` | 1 to 5,000 |
| [`POST /v2/logistics/requests`](/api-reference/logistics/create-requests-v2) | `202` | `logistics` | `create` | 1 to 5,000 |
| [`POST /v2/vehicles`](/api-reference/vehicles/create-v2) | `201` | `vehicles` | `create` | 1 to 1,000 |
| [`POST /v2/vehicles/delete`](/api-reference/vehicles/delete-v2) | `202` | `vehicles` | `delete` | 1 to 1,000 |
| [`POST /v2/vehicle-consumptions`](/api-reference/vehicles/consumptions-create-v2) | `202` | `vehicle_consumptions` | `create` | 1 to 5,000 |
| [`POST /v2/vehicle-consumptions/delete`](/api-reference/vehicles/consumptions-delete-v2) | `202` | `vehicle_consumptions` | `delete` | 1 to 5,000 |
| [`POST /v2/own-workforce/employees`](/api-reference/own-workforce/employees-create-v2) | `201` | `own_workforce` | `create` | 1 to 1,000 |
| [`POST /v2/own-workforce/employees/delete`](/api-reference/own-workforce/employees-delete-v2) | `202` | `own_workforce` | `delete` | 1 to 5,000 |
| [`POST /v2/own-workforce/trainings`](/api-reference/own-workforce/trainings-create-v2) | `201` | `own_workforce_trainings` | `create` | 1 to 1,000 |
| [`POST /v2/own-workforce/trainings/delete`](/api-reference/own-workforce/trainings-delete-v2) | `202` | `own_workforce_trainings` | `delete` | 1 to 5,000 |
| [`POST /v2/own-workforce/absences`](/api-reference/own-workforce/absences-create-v2) | `201` | `own_workforce_accidents_absenteeisms` | `create` | 1 to 1,000 |
| [`POST /v2/own-workforce/absences/delete`](/api-reference/own-workforce/absences-delete-v2) | `202` | `own_workforce_accidents_absenteeisms` | `delete` | 1 to 5,000 |
| [`POST /v2/business-travels`](/api-reference/business-travels/create-v2) | `202` | `business_travels` | `create` | 1 to 5,000 |
| [`POST /v2/business-travels/delete`](/api-reference/business-travels/delete-v2) | `202` | `business_travels` | `delete` | 1 to 5,000 |
| [`POST /v2/transports`](/api-reference/transport/create-v2) | `202` | `transport_routes` | `create` | 1 to 5,000 |
| [`POST /v2/transports/delete`](/api-reference/transport/delete-v2) | `202` | `transport_routes` | `delete` | 1 to 5,000 |
A request over its route's limit is `422 REQUEST_VALIDATION_FAILED`: split a bigger batch into several requests, each
with its own `Idempotency-Key`. Everything else on this page applies to both kinds of route.
## All or nothing
Every record is checked before anything is written: first against the route's schema, then against the rules that
need your data (the ids it references exist and are in the [perimeter](#perimeter), the unit fits, periods do not
overlap…). If one record is invalid, the whole request is rejected with `422 BULK_RECORDS_REJECTED` and **nothing is
written**. Every record is still checked, so one answer lists them all: `rejected` is the number of invalid records and
`errors` lists them (up to 100), in submission order. The checks that weigh records against each other run once every
record passes the rest (the credit-note net of invoices, the contract and remuneration overlaps of own workforce), so
a second `422` can follow a first one you fixed. Each entry has:
* `source_row_index`: the position of the record in `records` (or of the id in `ids`), starting at 0.
* `client_row_id`: the `client_row_id` you sent with the record, or `null` (always `null` in a delete).
* `error_code`: what is wrong, e.g. `SCHEMA_INVALID` or `REFERENCE_NOT_FOUND`.
* `error_params`: the values that explain it, e.g. `{"field": "facility_id", "value": "…"}`.
Fix every listed record and send the whole batch again. A rejected request records nothing, its `Idempotency-Key`
included, so you can resend with the same key or a new one. The codes every route shares are in
[Errors](/api-reference/errors#BULK_RECORDS_REJECTED); each route's page lists its own.
## Dry run
Add `?dry_run=true` to any of these routes to run the record checks and answer without writing anything. A valid batch
answers `200 OK` with `{"dry_run": true, "submitted": 2, "valid": 2}`; an invalid one gets the same
`422 BULK_RECORDS_REJECTED` as the real request. No job is created.
The `Idempotency-Key` header is still required, but a dry run does not check or store it: a key used in the last 24
hours with another body or on another route passes the dry run and can still be rejected on the real request
(`422 IDEMPOTENCY_KEY_REUSED`). Generate the batch's key once and send it with both.
A successful dry run means the records are valid, not that every one will calculate: on an asynchronous route the
calculation runs after the real request, and the [job's items](#following-a-job) give each record's outcome.
## The Idempotency-Key
Every bulk route requires an `Idempotency-Key` header (1 to 255 characters). Use one new key per batch, e.g. a UUID,
and send it again only to retry that same batch:
* **Keys are looked up per organization** (the one in `x-organization-id`), across every v2 bulk route.
* **Within 24 hours, the same key with the same route and body returns the original answer** instead of writing again,
with the header `Idempotent-Replayed: true`: the same job, in its current state, on an asynchronous route; the same
created resources on a synchronous one.
* **The same key with another body, or on another route,** is `422 IDEMPOTENCY_KEY_REUSED`.
* **While the first request with the key is in progress,** a second one is `409 IDEMPOTENCY_KEY_IN_PROGRESS`: retry
with the same key and body after `Retry-After` seconds, and you get the answer of the request that went through.
* **After 24 hours the key is free:** a request with it is a new one, and its records are written again.
The body is compared as JSON: key order, whitespace and how a whole number is written (`1200` or `1200.0`) do not make
it another body. Any other change does, including a record added, removed or moved.
For example, with the key `K`:
| When | Request | Answer |
| - | - | - |
| Day 1, 10:00:00 | `POST /v2/wastes`, key `K`, body A | `202` with job `J1` |
| Day 1, 10:00:01 | The same request, sent again while the first is still being written | `409 IDEMPOTENCY_KEY_IN_PROGRESS`, `Retry-After: 1` |
| Day 1, 10:01 | The same request, retried after a timeout | `202` with job `J1` and `Idempotent-Replayed: true`; nothing is written |
| Day 1, 10:05 | `POST /v2/wastes`, key `K`, body B | `422 IDEMPOTENCY_KEY_REUSED` |
| Day 1, 10:06 | `POST /v2/purchases`, key `K` | `422 IDEMPOTENCY_KEY_REUSED` |
| Day 2, 10:01 | `POST /v2/wastes`, key `K`, body A | `202` with a new job `J2`: the wastes are written again |
## Perimeter
A record is written to the organization in `x-organization-id`, unless the route lets it point to another
organization of the perimeter: a waste or an invoice goes to the organization of its facility, a vehicle consumption to
its vehicle's, a training or an absence to its employee's, and a facility to its `organization_id`. The ids a bulk
delete takes must be in the perimeter too.
The perimeter is the organization in `x-organization-id` and, recursively, the organizations below it. An organization
below is included when:
* its link to its parent organization is accepted and enabled,
* it is enabled itself (a disabled organization is left out, and so is everything below it), and
* its parent is not a group fund. A group fund is part of the perimeter, but the organizations below it are not, so the
perimeter of a fund is the fund alone.
The invoice routes (create and delete) also accept the facilities of every organization where the API key's user (the
member who created the key) is an enabled member. Create keys from a user who is a member only of the organizations
the integration should reach.
An id outside the perimeter answers exactly like an id that does not exist, so ids cannot be probed:
`REFERENCE_NOT_FOUND` for an id a record references, `NOT_FOUND` for an id to delete, and `ORGANIZATION_NOT_FOUND` for
the `organization_id` of a facility.
## Following a job
[`GET /v2/ingest-jobs/{id}`](/api-reference/ingest-jobs/get) returns the job:
| `status` | Meaning |
| - | - |
| `processing` | The records are being calculated or, in a delete job, deleted, chunk by chunk. |
| `completed` | Every chunk ran and every record succeeded. |
| `completed_with_errors` | Every chunk ran and at least one record failed (`counts.failed > 0`). |
`failed` is reserved: treat every status other than `processing` as finished.
* **Progress:** `chunks_done / chunks_total`. `counts` has the records `submitted`, `succeeded` and `failed` so far.
* **End:** `finished_at` is when the last chunk finished (`null` while the job is `processing`).
* **Per record:** [List Ingest Job Items](/api-reference/ingest-jobs/list-items) returns one item per record, in
submission order, with its `source_row_index`, `client_row_id`, the `entity_id` created (or deleted) and its `status`
(`pending`, `succeeded`, `failed`). A failed item says why in `error_code`:
* `CALCULATION_FAILED`: the record is stored, but its emissions could not be calculated.
* `DELETE_FAILED`: in a delete job, the entity was already gone when its chunk ran.
* A route may give the reason with codes of its own instead: logistics legs
([Why a leg failed to calculate](/api-reference/logistics/create-requests-v2#why-a-leg-failed-to-calculate)) and
business travels ([How it works](/api-reference/business-travels/create-v2#how-it-works)).
To know when a job ends, either:
* **Poll** [Get Ingest Job](/api-reference/ingest-jobs/get) every 5 to 10 seconds until `status` is no longer
`processing`, or
* **Receive the [`ingest_job.finished` webhook](/api-reference/webhooks/overview)**, which carries the same job object.
It is sent only for jobs submitted with an API key (`source: api_bulk`), to the webhook endpoints of the organization
in the request's `x-organization-id`, once the last chunk finishes.
`chunks_done` normally moves within minutes. If it stops, see
[A job that does not finish](/api-reference/ingest-jobs/get#a-job-that-does-not-finish).
## Entity types
A job's `entity_type` says what it creates or deletes. Use these values in the `entity_type` filter of
[List Ingest Jobs](/api-reference/ingest-jobs/list); the filter accepts any text, and a value that is not one of these
returns an empty page. The routes of each value are in [the table above](#synchronous-and-asynchronous-routes).
| `entity_type` | What it holds |
| - | - |
| `wastes` | Wastes |
| `invoices` | Invoices of every category |
| `facilities` | Facilities |
| `purchases` | Purchases |
| `logistics` | Logistics legs (requests) |
| `vehicles` | Vehicles |
| `vehicle_consumptions` | Vehicle consumptions |
| `own_workforce` | Own workforce employees, with their contracts and remunerations |
| `own_workforce_trainings` | Own workforce trainings |
| `own_workforce_accidents_absenteeisms` | Own workforce absences and accidents |
| `business_travels` | Business travels |
| `transport_routes` | Freight transport shipments, with their sections |
## Related Endpoints
Status and progress of one job
The jobs of your organization, newest first
Per-record outcomes of a job
Get `ingest_job.finished` instead of polling
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.