Hardcover Ingestion

One of four sources feeding api_warehouse’s personal data platform, and the one that tests the ingestion engine hardest: Hardcover’s API is GraphQL, not REST, while everything else in the warehouse speaks plain HTTP verbs and query params.

A GraphQL source in a REST-shaped engine

Every other endpoint config in the warehouse builds a URL from a path template and query params. Hardcover instead POSTs to a single GraphQL endpoint with a query document — so the endpoint schema gained one extra field, query, that lets a config declare a raw GraphQL string instead of a path. That was the only change needed on the request-building side; response extraction still uses the same dotted response_path mechanism (data.user_books) as every REST source, since GraphQL responses are just nested JSON like anything else.

Auth

Simple bearer-token auth via HARDCOVER_API_TOKEN — no OAuth dance, no refresh flow. The token is normalized to a Bearer {token} header if it isn’t already prefixed that way.

Resolving the user id

Hardcover’s schema scopes user_books by an explicit user_id — there’s no implicit “current user” filter like Spotify’s me/... routes. The pipeline runs a one-off query Me { me { id } } on first use, unwraps Hasura’s array-shaped root field, and caches the id into the API’s persisted config — the same pattern Spotify and Trakt use to persist rotated tokens. Every subsequent request injects that cached user_id as a GraphQL variable.

Endpoints

Three endpoints, distinguished only by a status_id filter in the GraphQL where clause (Hasura’s status convention: 1 = want-to-read, 2 = currently-reading, 3 = read):

• read_books — includes my_rating and the most recent user_book_reads row (read_started_at/read_finished_at).
• currently_reading — includes a start date only.
• want_to_read — just a created-at timestamp, no rating or read dates.

dbt modelling

• Staging — thin passthroughs per status, each extracting book_image ->> 'url' into a plain cover_image_url column and filtering out rows with no title.
• Marts — dim_books unions all three staging models, deduped distinct on (book_id) and ordered to prefer whichever row actually captured a cover image (the same book can appear across all three lists with inconsistent image data depending on when it was fetched); fct_reading_history (finished books only); fct_reading_list (currently-reading and want-to-read unioned into one “in progress or queued” fact with a status/status_at column); fct_reading_stats (yearly aggregates — books read, pages read, average rating, average pages per book).

Engineering notes

This is the clearest proof that the ingestion engine is genuinely generic rather than REST-shaped-and-hoping: a fundamentally different protocol (GraphQL POST + query document vs. REST path + params) only needed one new config field, not a bespoke integration.