How Bulk Ingestion Works

The v2 write routes take a batch: up to thousands of records to create, or of ids to delete. Every route handles it the same way. It checks every record first and writes all of them or none. Then it either answers at once with what it created (201 Created), or answers 202 Accepted with an ingest job that calculates (or deletes) the records in the background, and you follow that job until it finishes.

​

Synchronous and asynchronous routes

A route is asynchronous when its records need work after the request: calculating their emissions, or deleting them with everything that depends on them. It answers 202 Accepted with the job, and its Location header is /v2/ingest-jobs/{id}. The job processes the records in chunks; see Following a job. A route is synchronous when its records need nothing calculated. It answers 201 Created with the created resources, in submission order, each with its source_row_index and client_row_id. It also records a job, born completed with 0 of 0 chunks, which sends no webhook: List Ingest Jobs shows it, and its items map each record to the resource created. A request over its route’s limit is 422 REQUEST_VALIDATION_FAILED: split a bigger batch into several requests, each with its own Idempotency-Key. Everything else on this page applies to both kinds of route.

​

All or nothing

Every record is checked before anything is written: first against the route’s schema, then against the rules that need your data (the ids it references exist and are in the perimeter, the unit fits, periods do not overlap…). If one record is invalid, the whole request is rejected with 422 BULK_RECORDS_REJECTED and nothing is written. Every record is still checked, so one answer lists them all: rejected is the number of invalid records and errors lists them (up to 100), in submission order. The checks that weigh records against each other run once every record passes the rest (the credit-note net of invoices, the contract and remuneration overlaps of own workforce), so a second 422 can follow a first one you fixed. Each entry has:

• source_row_index: the position of the record in records (or of the id in ids), starting at 0.
• client_row_id: the client_row_id you sent with the record, or null (always null in a delete).
• error_code: what is wrong, e.g. SCHEMA_INVALID or REFERENCE_NOT_FOUND.
• error_params: the values that explain it, e.g. {"field": "facility_id", "value": "…"}.

Fix every listed record and send the whole batch again. A rejected request records nothing, its Idempotency-Key included, so you can resend with the same key or a new one. The codes every route shares are in Errors; each route’s page lists its own.

​

Dry run

Add ?dry_run=true to any of these routes to run the record checks and answer without writing anything. A valid batch answers 200 OK with {"dry_run": true, "submitted": 2, "valid": 2}; an invalid one gets the same 422 BULK_RECORDS_REJECTED as the real request. No job is created. The Idempotency-Key header is still required, but a dry run does not check or store it: a key used in the last 24 hours with another body or on another route passes the dry run and can still be rejected on the real request (422 IDEMPOTENCY_KEY_REUSED). Generate the batch’s key once and send it with both. A successful dry run means the records are valid, not that every one will calculate: on an asynchronous route the calculation runs after the real request, and the job’s items give each record’s outcome.

​

The Idempotency-Key

Every bulk route requires an Idempotency-Key header (1 to 255 characters). Use one new key per batch, e.g. a UUID, and send it again only to retry that same batch:

• Keys are looked up per organization (the one in x-organization-id), across every v2 bulk route.
• Within 24 hours, the same key with the same route and body returns the original answer instead of writing again, with the header Idempotent-Replayed: true: the same job, in its current state, on an asynchronous route; the same created resources on a synchronous one.
• The same key with another body, or on another route, is 422 IDEMPOTENCY_KEY_REUSED.
• While the first request with the key is in progress, a second one is 409 IDEMPOTENCY_KEY_IN_PROGRESS: retry with the same key and body after Retry-After seconds, and you get the answer of the request that went through.
• After 24 hours the key is free: a request with it is a new one, and its records are written again.

The body is compared as JSON: key order, whitespace and how a whole number is written (1200 or 1200.0) do not make it another body. Any other change does, including a record added, removed or moved. For example, with the key K:

​

Perimeter

A record is written to the organization in x-organization-id, unless the route lets it point to another organization of the perimeter: a waste or an invoice goes to the organization of its facility, a vehicle consumption to its vehicle’s, a training or an absence to its employee’s, and a facility to its organization_id. The ids a bulk delete takes must be in the perimeter too. The perimeter is the organization in x-organization-id and, recursively, the organizations below it. An organization below is included when:

• its link to its parent organization is accepted and enabled,
• it is enabled itself (a disabled organization is left out, and so is everything below it), and
• its parent is not a group fund. A group fund is part of the perimeter, but the organizations below it are not, so the perimeter of a fund is the fund alone.

The invoice routes (create and delete) also accept the facilities of every organization where the API key’s user (the member who created the key) is an enabled member. Create keys from a user who is a member only of the organizations the integration should reach. An id outside the perimeter answers exactly like an id that does not exist, so ids cannot be probed: REFERENCE_NOT_FOUND for an id a record references, NOT_FOUND for an id to delete, and ORGANIZATION_NOT_FOUND for the organization_id of a facility.

​

Following a job

GET /v2/ingest-jobs/{id} returns the job: failed is reserved: treat every status other than processing as finished.

• Progress: chunks_done / chunks_total. counts has the records submitted, succeeded and failed so far.
• End: finished_at is when the last chunk finished (null while the job is processing).
• Per record: List Ingest Job Items returns one item per record, in submission order, with its source_row_index, client_row_id, the entity_id created (or deleted) and its status (pending, succeeded, failed). A failed item says why in error_code:
  • CALCULATION_FAILED: the record is stored, but its emissions could not be calculated.
  • DELETE_FAILED: in a delete job, the entity was already gone when its chunk ran.
  • A route may give the reason with codes of its own instead: logistics legs (Why a leg failed to calculate) and business travels (How it works).

To know when a job ends, either:

• Poll Get Ingest Job every 5 to 10 seconds until status is no longer processing, or
• Receive the ingest_job.finished webhook, which carries the same job object. It is sent only for jobs submitted with an API key (source: api_bulk), to the webhook endpoints of the organization in the request’s x-organization-id, once the last chunk finishes.

chunks_done normally moves within minutes. If it stops, see A job that does not finish.

​

Entity types

A job’s entity_type says what it creates or deletes. Use these values in the entity_type filter of List Ingest Jobs; the filter accepts any text, and a value that is not one of these returns an empty page. The routes of each value are in the table above.

​

Related Endpoints