# Your types, cast from your database:: Source: https://pgcast.dev/ pgcast reads your Postgres schema and writes the TypeScript types. Its shortcuts produce SQL you can read and predict, and you can always drop down to a plain `sql` template. No ORM, no models, no hidden state. [Get started](https://pgcast.dev/install) [Playground](https://pgcast.dev/playground) ```sh [npm] npm install zapatos@npm:@pgcast/pgcast ``` ```sh [pnpm] pnpm add zapatos@npm:@pgcast/pgcast ``` ```sh [yarn] yarn add zapatos@npm:@pgcast/pgcast@^6 ``` ```sh [bun] bun add zapatos@npm:@pgcast/pgcast@^6 ``` You write: ```ts const films = await db.select('films', db.all, { lateral: { cast: db.select('roles', { film_id: db.parent('id'), }, { order: { by: 'billing', direction: 'ASC' }, }), }, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("films".*) || jsonb_build_object($1::text, "lateral_cast".result) AS result FROM "films" LEFT JOIN LATERAL ( SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("roles".*) AS result FROM "roles" WHERE ("film_id" = "films"."id") ORDER BY "billing" ASC ) AS "sq_roles" ) AS "lateral_cast" ON true ) AS "sq_films" ``` Values: `["cast"]` TypeScript knows: ```ts const films: (films.JSONSelectable & db.LateralResult<{ cast: db.SQLFragment; }>)[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; cast: roles.JSONSelectable[]; }[] ``` ## Postgres is the source Types come from `pg_catalog`, never written by hand. Change a column, regenerate, and the compiler finds every query that broke. ## SQL you can read Each shortcut writes one documented query. Log it, test it, or write it yourself in a `sql` template. ## Values are parameters Values travel as `$1`, `$2`. Names are quoted as Postgres quotes them, so a stray `"` cannot change your query. ## Live playground Write a query in [the playground](https://pgcast.dev/playground) and see its SQL and its type as you type. Press Run, and a real Postgres answers, in your browser. [**Tutorial**: Install pgcast and run your first typed query.](https://pgcast.dev/tutorial) [**How-to guides**: Laterals, pages, batches, retries: one task each.](https://pgcast.dev/guides) [**Reference**: Every export and option, and the SQL it writes.](https://pgcast.dev/reference) [**Explanation**: Why pgcast works as it does.](https://pgcast.dev/explanation) --- # Tutorial Source: https://pgcast.dev/tutorial A lesson: one path from an empty project to a typed query, each step shown and checked. Follow it in order; the why can wait for the [explanations](https://pgcast.dev/explanation). --- # Install Source: https://pgcast.dev/install pgcast keeps Zapatos's module names, so you install it under the `zapatos` alias. Nothing in your imports changes. ## Requirements - Postgres. The test suite runs against 14, 16 and 18. - Node, and `pg` (node-postgres), with `@types/pg` for TypeScript. - TypeScript 5.7 or later. `@types/pg` brings in `@types/node` and `pg-protocol`, whose types now need it; so does [`toBuffer()`](https://pgcast.dev/runtime-config#toBuffer)'s, `Buffer`, as in Zapatos 6.6.1. ### On an older TypeScript pgcast's own types go back to TypeScript 4.1, the first with template literal types. Below 5.7, tell TypeScript not to check the types in `node_modules` (`toBuffer()` then returns `any`): ```json [tsconfig.json] { "compilerOptions": { "skipLibCheck": true } } ``` - From 5.0, that's all. - Below 5.0, TypeScript can't read today's `@types/node`. Install the one made for your version, by its tag, and pin it: here for 4.1, `ts4.9` for 4.9. ```sh npm install --save-dev --save-exact @types/node@ts4.1 ``` > **Stale schema** > > `skipLibCheck` skips your generated `schema.d.ts` too. After an upgrade of pgcast, a schema you haven't regenerated no longer fails on `schemaVersionCanary` ([When the schema changes](https://pgcast.dev/first-query#when-the-schema-changes)): run `npx pgcast` each time. ## Add the package ```sh [npm] npm install zapatos@npm:@pgcast/pgcast pg npm install --save-dev @types/pg ``` ```sh [pnpm] pnpm add zapatos@npm:@pgcast/pgcast pg pnpm add -D @types/pg ``` ```sh [yarn] yarn add zapatos@npm:@pgcast/pgcast@^6 pg yarn add -D @types/pg ``` ```sh [bun] bun add zapatos@npm:@pgcast/pgcast@^6 pg bun add -d @types/pg ``` Yarn 2 and later need a version range in an alias, and Yarn 1 and bun before 1.4 write none without one: hence the `@^6`. > **Why the alias** > > The generated schema and the run-time code talk about `zapatos/db` and `zapatos/schema`. Installed as plain `@pgcast/pgcast`, those imports wouldn't resolve. Native `pgcast/*` names will come later, as an opt-in ([decision 002](https://github.com/AlexandreDemartin/pgcast/blob/master/docs/decisions/002-module-names.md)). Already on Zapatos? Switching is one line of `package.json`, then a look for the few calls whose result changes: [Switch from Zapatos](https://pgcast.dev/switching). That's pgcast in your project. Next, [First query](https://pgcast.dev/first-query) generates the types of a database, then runs a select and an insert with them. Working with an assistant? It can read these docs from [llms.txt](https://pgcast.dev/llms.txt), an index of the pages, or [llms-full.txt](https://pgcast.dev/llms-full.txt), all of them in one file. --- # First query Source: https://pgcast.dev/first-query By the end of this page, you'll have generated the types of a database, asked it for films and added one, with the SQL and the type of each query in front of you. It starts where [Install](https://pgcast.dev/install) stops: pgcast and `pg` in your project. ## Before you start - pgcast installed under the `zapatos` alias, with `pg` and `@types/pg`: [Install](https://pgcast.dev/install). - TypeScript 5.7 or later in your project: `npm install --save-dev typescript`. - Node 22.18 or later, which runs a TypeScript file as it is, stripping its types. - An empty Postgres database you can connect to, its connection string in the `DATABASE_URL` environment variable, and `psql`, Postgres's command-line client. - The films of these docs in it: [`schema.sql`](https://github.com/AlexandreDemartin/pgcast/blob/master/website/schema.sql) holds `studios`, their `films`, the `actors` and the `roles` they play, and a few more tables, with some rows to query. Download it and load it: ```sh curl -O https://raw.githubusercontent.com/AlexandreDemartin/pgcast/master/website/schema.sql psql "$DATABASE_URL" -f schema.sql ``` Every example below runs on that schema. A database of your own works too: swap in your table and column names as you go. ## 1. Set up the project The examples use `import` and a top-level `await`, so your project is an ES module: ```sh npm pkg set type=module ``` Then a `tsconfig.json` that compiles them as Node runs them: ```json [tsconfig.json] { "compilerOptions": { "module": "nodenext", "target": "es2022", "strict": true, "noEmit": true }, "include": ["src"] } ``` - `"module": "nodenext"` makes `import pg from 'pg'` and the top-level `await` compile, as in a `"type": "module"` project. - `strict` turns on `strictNullChecks`, which the generated types need. Without it, a query that names its `columns` (as one does below) gets the type of every column, and a missing row stops showing up as `undefined`. - `noEmit`: `tsc` only checks your code; Node runs the `.ts` file itself. - The generator writes ordinary `.d.ts` files into your source tree, and `zapatos/schema` is declared in one of them, so they must be part of your compilation. With `"include": ["src"]` and the `outDir` below, they are. ## 2. Configure the generator Add `zapatosconfig.json` at the root of your project, where you'll run the generator: ```json [zapatosconfig.json] { "db": { "connectionString": "{{DATABASE_URL}}" }, "outDir": "./src" } ``` - `db` takes what you'd give `new pg.Pool()`. - `outDir` is where the `zapatos` folder goes; without it, the folder you run the generator in. - `{{DATABASE_URL}}` is filled in from the environment, so the connection string stays out of your repository. If the variable isn't set, the generator stops with `Error: If present, zapatosconfig.json must be a valid JSON file, and all referenced environment variables must exist: Environment variable 'DATABASE_URL' is not set`. Every other option, from the schemas to include to per-column settings, is in [Generator](https://pgcast.dev/generator). ## 3. Generate the types Run the generator. It's installed under both names, `pgcast` and `zapatos`: ```sh npx pgcast ``` It prints a line for each placeholder it writes, here for the docs schema's two types that Postgres alone can't describe to TypeScript: ```txt Writing new custom type or domain placeholder file: src/zapatos/custom/PgPoint.d.ts Writing new custom type or domain placeholder file: src/zapatos/custom/PgSeat_label.d.ts ``` And you have: ```txt src/zapatos/ ├── schema.d.ts every table, written again on each run: don't edit it └── custom/ ├── index.d.ts written again on each run too ├── PgPoint.d.ts cinemas.location, a POINT: any, until you say otherwise └── PgSeat_label.d.ts seat_bookings.seat, the seat_label domain: string ``` The placeholders in `custom/` are yours: the generator writes each one once and never overwrites it, while `index.d.ts` is written again on each run. In `schema.d.ts`, each table is a namespace of types, and the shortcuts pick the right one for you: | Type | What it describes | Used by | |---|---|---| | `films.Selectable` | a row, as `pg` returns it | your own `sql` queries | | `films.JSONSelectable` | a row that came back as JSON: here the same, but a `timestamptz`, for one, arrives as a string, not a `Date` ([why](https://pgcast.dev/catalog-types)) | the results of `select()`, `insert()` and the other shortcuts | | `films.Whereable` | a filter: any columns, each with a value | the `where` of `select()`, `selectOne()`, `update()`, `deletes()` | | `films.Insertable` | a new row: the `NOT NULL` columns with no default required, the rest optional | `insert()`, `upsert()` | | `films.Updatable` | changes to a row: every column optional | `update()`, `upsert()` | ## 4. Connect The run-time code never opens a connection of its own: each query runs on the pool or client you give it. Import the run-time code and make a pool: ```ts [src/films.ts] import * as db from 'zapatos/db'; import pg from 'pg'; const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL, allowExitOnIdle: true, // so the script ends once its queries are done }); ``` The examples below carry on in this file. Their top-level `await` needs ESM: that's the `"type": "module"` above. After each step, run it: ```sh node src/films.ts ``` And check its types with `npx tsc`, as your editor does while you type. ## 5. Ask for one film Ask for a film by its title with `selectOne()`, then `.run()` it on the pool: You write: ```ts const film = await db.selectOne('films', { title: 'The Salt Orchard', }).run(pool); ``` Postgres runs: ```sql SELECT to_jsonb("films".*) AS result FROM "films" WHERE ("title" = $1) LIMIT $2 ``` Values: `["The Salt Orchard", 1]` TypeScript knows: ```ts const film: films.JSONSelectable | undefined // that is const film: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; } | undefined ``` The title went to Postgres as a parameter, `("title" = $1)`, never pasted into the SQL. The row comes back as JSON (`to_jsonb`), so its type is `films.JSONSelectable`, and `undefined` if no film has that title. TypeScript won't let you forget it: ```ts console.log(film.release_year); // error TS18048 ``` Check for it instead, and the type narrows to the row (hover `film` to see). Run the file, and it prints `1974`: ```ts if (film) console.log(film.release_year); // 1974 ``` To see the SQL in your own code, call `.compile()` where you'd call `.run()`: it gives the `text` (on one line) and `values` the panel shows, and sends nothing. Add this line to the file: ```ts console.log(db.selectOne('films', { title: 'The Salt Orchard' }).compile()); ``` Run it, and after `1974` it prints: ```txt { text: 'SELECT to_jsonb("films".*) AS result FROM "films" WHERE ("title" = $1) LIMIT $2', values: [ 'The Salt Orchard', 1 ] } ``` ## 6. Ask for several, and only what you need `select()` gives every matching row, in an array. Ask for the films of one studio, oldest first, and name the two columns you want: You write: ```ts const films = await db.select('films', { studio_id: 1000 }, { columns: ['title', 'release_year'], order: { by: 'release_year', direction: 'ASC' }, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "title", $2::text, "release_year") AS result FROM "films" WHERE ("studio_id" = $3) ORDER BY "release_year" ASC ) AS "sq_films" ``` Values: `["title", "release_year", 1000]` TypeScript knows: ```ts const films: db.JSONOnlyColsForTable<"films", ("release_year" | "title")[]>[] // that is const films: { release_year: number; title: string; }[] ``` Each column name shows up twice in `jsonb_build_object($1::text, "title", …)`: as a value, `$1`, the key of the JSON object, and quoted, `"title"`, the column it reads. The type follows the query: two columns asked for, two in the type. Leave `columns` out and you get every column, `films.JSONSelectable[]`. Print them: ```ts console.log(films); ``` Run the file again, and after the year and the SQL it prints: ```txt [ { title: 'The Salt Orchard', release_year: 1974 }, { title: "The Cartographer's Silence", release_year: 2008 } ] ``` With no matching film, the array is empty: `select()` never gives `undefined`. ## 7. Add a film `insert()` takes a `films.Insertable` and gives back the row as Postgres stored it. Type the row first, with the schema's types. Add this import at the top of the file, with the other two: ```ts import type * as s from 'zapatos/schema'; ``` Then the row and the query: You write: ```ts const row: s.films.Insertable = { studio_id: 1002, title: 'The Night Ferry to Orkesund', release_year: 2024, runtime_minutes: 117, }; const newFilm = await db.insert('films', row).run(pool); ``` Postgres runs: ```sql INSERT INTO "films" ("release_year", "runtime_minutes", "studio_id", "title") VALUES ($1, $2, $3, $4) RETURNING to_jsonb("films".*) AS result ``` Values: `[2024, 117, 1002, "The Night Ferry to Orkesund"]` TypeScript knows: ```ts const newFilm: s.films.JSONSelectable // that is const newFilm: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; } ``` `newFilm` is a `films.JSONSelectable`, the row type `selectOne()` gave. The row's `films.Insertable` asks for the `NOT NULL` columns with no default and leaves the others optional; typed on its own, the row gets errors that name the column at fault. Print the new film: ```ts console.log(newFilm); ``` Run the file, and it prints the new row: ```txt { id: 1, title: 'The Night Ferry to Orkesund', genres: [], tagline: null, studio_id: 1002, release_year: 2024, runtime_minutes: 117 } ``` The columns you left out took their defaults: the new `id` from its sequence, `genres` as `[]`, `tagline` as `null`. On a freshly loaded docs schema, the first film you add gets the `id` 1, as the seeded films start at 1000; each run after that adds it again, with the next `id`. A `NOT NULL` column with no default can't be left out. Forget the running time, and TypeScript says so before Postgres can: ```ts const row: s.films.Insertable = { // error TS2741 studio_id: 1002, title: 'The Night Ferry to Orkesund', release_year: 2024, }; ``` > **Opt-in** > > Leave a key out to get its column's default. A key set to `undefined` is sent as `NULL`, as in Zapatos 6.6.1, and pgcast warns once (`PGCAST_DEP002`). With `db.setConfig({ undefinedMeansMissing: true })`, it counts as left out; that's the default from 7.0 ([Versions](https://pgcast.dev/versions)). With the running time back in the row, here is your whole file, the steps of this page in order: ```ts [src/films.ts] import * as db from 'zapatos/db'; import type * as s from 'zapatos/schema'; import pg from 'pg'; const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL, allowExitOnIdle: true, // so the script ends once its queries are done }); const film = await db.selectOne('films', { title: 'The Salt Orchard', }).run(pool); if (film) console.log(film.release_year); // 1974 console.log(db.selectOne('films', { title: 'The Salt Orchard' }).compile()); const films = await db.select('films', { studio_id: 1000 }, { columns: ['title', 'release_year'], order: { by: 'release_year', direction: 'ASC' }, }).run(pool); console.log(films); const row: s.films.Insertable = { studio_id: 1002, title: 'The Night Ferry to Orkesund', release_year: 2024, runtime_minutes: 117, }; const newFilm = await db.insert('films', row).run(pool); console.log(newFilm); ``` ## 8. When the schema changes Run `npx pgcast` again after each migration. `schema.d.ts` is written afresh, and TypeScript checks every query against it. Try it: rename a column, regenerate, and check the file: ```sh psql "$DATABASE_URL" -c 'ALTER TABLE films RENAME runtime_minutes TO minutes' npx pgcast npx tsc ``` The row names `runtime_minutes`, which `films.Insertable` no longer has, so it stops compiling: ```txt src/films.ts:26:3 - error TS2353: Object literal may only specify known properties, and 'runtime_minutes' does not exist in type 'Insertable'. 26 runtime_minutes: 117, ~~~~~~~~~~~~~~~ Found 1 error in src/films.ts:26 ``` Any query that names the column stops compiling too, until you write `minutes` instead. Here, rename it back, so the rest of these docs run as written: ```sh psql "$DATABASE_URL" -c 'ALTER TABLE films RENAME minutes TO runtime_minutes' npx pgcast npx tsc ``` This time `tsc` prints nothing: every query fits the schema again. After you upgrade pgcast, a type error on `schemaVersionCanary` in `schema.d.ts` means the same thing: the generated format has changed, so run the generator again. ## What you've learnt - Point the generator at a database, and find its types in `zapatos/schema`. - Ask for one row with `selectOne()`, several with `select()`, and only some columns with `columns`. - Add a row with `insert()`, and get it back as Postgres stored it. - Read the SQL a query runs with `.compile()`, and its type under your cursor. - Regenerate after a migration, and let TypeScript find the queries that no longer fit. Next: a task of your own in the [how-to guides](https://pgcast.dev/guides), every option in the [reference](https://pgcast.dev/reference), and why it works this way in the [explanations](https://pgcast.dev/explanation). --- # How-to guides Source: https://pgcast.dev/guides Directions for a task you already have in mind: a join, a page of results, a batch, a retry. Each assumes the basics of the [tutorial](https://pgcast.dev/tutorial), and links to the [reference](https://pgcast.dev/reference) for every option. --- # Joins Source: https://pgcast.dev/lateral Get rows with the rows they relate to, in one query and one typed result: a film with its cast, a crew member with their mentor. You nest a shortcut in the [`lateral`](https://pgcast.dev/select#lateral) option of `select()`; it becomes a `LEFT JOIN LATERAL`, and its result a key of each row. You already know `select()` from the [tutorial](https://pgcast.dev/first-query). ## Add the rows that point to it A film's cast is the `roles` whose `film_id` is the film's `id`. Nest a `select()` of `roles` under a key of `lateral`, and name the film's column with `db.parent()`: You write: ```ts const film = await db.selectOne('films', { id: 1000 }, { columns: ['title'], lateral: { cast: db.select('roles', { film_id: db.parent('id') }, { columns: ['character', 'billing'], order: { by: 'billing', direction: 'ASC' }, }), }, }).run(pool); ``` Postgres runs: ```sql SELECT jsonb_build_object($1::text, "title") || jsonb_build_object($2::text, "lateral_cast".result) AS result FROM "films" LEFT JOIN LATERAL ( SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($3::text, "character", $4::text, "billing") AS result FROM "roles" WHERE ("film_id" = "films"."id") ORDER BY "billing" ASC ) AS "sq_roles" ) AS "lateral_cast" ON true WHERE ("id" = $5) LIMIT $6 ``` Values: `["title", "cast", "character", "billing", 1000, 1]` TypeScript knows: ```ts const film: (db.JSONOnlyColsForTable<"films", "title"[]> & db.LateralResult<{ cast: db.SQLFragment[], never>; }>) | undefined // that is const film: { title: string; cast: db.JSONOnlyColsForTable<"roles", ("billing" | "character")[]>[]; } | undefined ``` - `cast` becomes a key of the film, holding an array: `[]` when no role matches. - `db.parent('id')` is `"films"."id"`, a column of the query around it ([`parent()`](https://pgcast.dev/sql-template#parent)). When the two columns share a name, `db.parent()` alone takes the column of its filter. - The subquery takes the options of any `select()`, here `columns` and `order` ([Options](https://pgcast.dev/select#options)). ## Add the row it points to Each film has one studio: `studio_id` is `NOT NULL REFERENCES studios(id)`, so the row is always there. Nest `selectExactlyOne()`, and the key holds that row, not an array: You write: ```ts const film = await db.selectOne('films', { id: 1000 }, { columns: ['title'], lateral: { studio: db.selectExactlyOne('studios', { id: db.parent('studio_id') }, { columns: ['name'] }), cast: db.select('roles', { film_id: db.parent('id') }, { columns: ['character', 'billing'], order: { by: 'billing', direction: 'ASC' }, }), }, }).run(pool); ``` Postgres runs: ```sql SELECT jsonb_build_object($1::text, "title") || jsonb_build_object($2::text, "lateral_cast".result, $3::text, "lateral_studio".result) AS result FROM "films" LEFT JOIN LATERAL ( SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($4::text, "character", $5::text, "billing") AS result FROM "roles" WHERE ("film_id" = "films"."id") ORDER BY "billing" ASC ) AS "sq_roles" ) AS "lateral_cast" ON true LEFT JOIN LATERAL ( SELECT jsonb_build_object($6::text, "name") AS result FROM "studios" WHERE ("id" = "films"."studio_id") LIMIT $7 ) AS "lateral_studio" ON true WHERE ("id" = $8) LIMIT $9 ``` Values: `["title", "cast", "studio", "character", "billing", "name", 1, 1000, 1]` TypeScript knows: ```ts const film: (db.JSONOnlyColsForTable<"films", "title"[]> & db.LateralResult<{ studio: db.SQLFragment, never>; cast: db.SQLFragment[], never>; }>) | undefined // that is const film: { title: string; studio: db.JSONOnlyColsForTable<"studios", "name"[]>; cast: db.JSONOnlyColsForTable<"roles", ("billing" | "character")[]>[]; } | undefined ``` - The joins come in the sorted order of their keys, whatever order you write them in. - On a nullable foreign key, nest [`selectOne()`](https://pgcast.dev/select#selectOne): the key holds the row or `null`, as the mentor below. - In a lateral, `selectExactlyOne()` checks nothing: a missing row gives `null` and no error, though its type has no `null` ([`selectExactlyOne()`](https://pgcast.dev/select#selectExactlyOne)). Keep it for a `NOT NULL` foreign key. ## Go through a join table Films and actors are many-to-many, through `roles`. Nest a lateral in the lateral, to give each role its actor: You write: ```ts const film = await db.selectOne('films', { id: 1000 }, { columns: ['title'], lateral: { cast: db.select('roles', { film_id: db.parent('id') }, { columns: ['character'], order: { by: 'billing', direction: 'ASC' }, lateral: { actor: db.selectExactlyOne('actors', { id: db.parent('actor_id') }, { columns: ['name'] }), }, }), }, }).run(pool); ``` Postgres runs: ```sql SELECT jsonb_build_object($1::text, "title") || jsonb_build_object($2::text, "lateral_cast".result) AS result FROM "films" LEFT JOIN LATERAL ( SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($3::text, "character") || jsonb_build_object($4::text, "lateral_actor".result) AS result FROM "roles" LEFT JOIN LATERAL ( SELECT jsonb_build_object($5::text, "name") AS result FROM "actors" WHERE ("id" = "roles"."actor_id") LIMIT $6 ) AS "lateral_actor" ON true WHERE ("film_id" = "films"."id") ORDER BY "billing" ASC ) AS "sq_roles" ) AS "lateral_cast" ON true WHERE ("id" = $7) LIMIT $8 ``` Values: `["title", "cast", "character", "actor", "name", 1, 1000, 1]` TypeScript knows: ```ts const film: (db.JSONOnlyColsForTable<"films", "title"[]> & db.LateralResult<{ cast: db.SQLFragment<(db.JSONOnlyColsForTable<"roles", "character"[]> & db.LateralResult<{ actor: db.SQLFragment, never>; }>)[], never>; }>) | undefined // that is const film: { title: string; cast: (db.JSONOnlyColsForTable<"roles", "character"[]> & db.LateralResult<{ actor: db.SQLFragment, never>; }>)[]; } | undefined ``` `db.parent()` names a column of the nearest query around it: in `actor`, `"roles"."actor_id"`. Each role's `actor` is `{ name: string }`. Nest as deep as you need; each level adds a `LEFT JOIN LATERAL` to the same query. ## Keep only the far side For the actors alone, without the role's columns, give the inner `lateral` one subquery rather than an object: its result takes the place of each role. You write: ```ts const film = await db.selectOne('films', { id: 1000 }, { columns: ['title'], lateral: { actors: db.select('roles', { film_id: db.parent('id') }, { order: { by: 'billing', direction: 'ASC' }, lateral: db.selectExactlyOne('actors', { id: db.parent('actor_id') }), }), }, }).run(pool); ``` Postgres runs: ```sql SELECT jsonb_build_object($1::text, "title") || jsonb_build_object($2::text, "lateral_actors".result) AS result FROM "films" LEFT JOIN LATERAL ( SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT "lateral_passthru".result AS result FROM "roles" LEFT JOIN LATERAL ( SELECT to_jsonb("actors".*) AS result FROM "actors" WHERE ("id" = "roles"."actor_id") LIMIT $3 ) AS "lateral_passthru" ON true WHERE ("film_id" = "films"."id") ORDER BY "billing" ASC ) AS "sq_roles" ) AS "lateral_actors" ON true WHERE ("id" = $4) LIMIT $5 ``` Values: `["title", "actors", 1, 1000, 1]` TypeScript knows: ```ts const film: (db.JSONOnlyColsForTable<"films", "title"[]> & db.LateralResult<{ actors: db.SQLFragment; }>) | undefined // that is const film: { title: string; actors: actors.JSONSelectable[]; } | undefined ``` The `order` still sorts the roles, so the actors come in billing order. With the role gone, `columns` and `extras` on `roles` would have nothing to add to: the type refuses them, with `strictNullChecks` on. ```ts db.select('roles', { film_id: 1000 }, { columns: ['character'], lateral: db.selectExactlyOne('actors', { id: db.parent('actor_id') }), // error TS2322 }); ``` ## Join a table to itself A crew member's mentor is another row of `crew`. Give the subquery an `alias`, so that `db.parent()` still reaches the outer `crew`: You write: ```ts const crew = await db.select('crew', db.all, { columns: ['name', 'job'], order: { by: 'id', direction: 'ASC' }, lateral: { mentor: db.selectOne('crew', { id: db.parent('mentor_id') }, { alias: 'mentor', columns: ['name'] }), proteges: db.count('crew', { mentor_id: db.parent('id') }, { alias: 'protege' }), }, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "name", $2::text, "job") || jsonb_build_object($3::text, "lateral_mentor".result, $4::text, "lateral_proteges".result) AS result FROM "crew" LEFT JOIN LATERAL ( SELECT jsonb_build_object($5::text, "name") AS result FROM "crew" AS "mentor" WHERE ("id" = "crew"."mentor_id") LIMIT $6 ) AS "lateral_mentor" ON true LEFT JOIN LATERAL ( SELECT count(*) AS result FROM "crew" AS "protege" WHERE ("mentor_id" = "crew"."id") ) AS "lateral_proteges" ON true ORDER BY "id" ASC ) AS "sq_crew" ``` Values: `["name", "job", "mentor", "proteges", "name", 1]` TypeScript knows: ```ts const crew: (db.JSONOnlyColsForTable<"crew", ("name" | "job")[]> & db.LateralResult<{ mentor: db.SQLFragment | undefined, never>; proteges: db.SQLFragment; }>)[] // that is const crew: { name: string; job: crew_job; mentor: db.JSONOnlyColsForTable<"crew", "name"[]> | null; proteges: number; }[] ``` - `mentor` holds the row or `null`: Wilhelmina Strand has no mentor. - [`count()`](https://pgcast.dev/select#count) gives a number, here each member's protégés. - Without the aliases, the subquery's `FROM "crew"` hides the outer one, and `"crew"."mentor_id"` is the subquery's own row. Postgres runs it with no error, and every `mentor` comes back `null`, every count `0`: ```sql SELECT jsonb_build_object($5::text, "name") AS result FROM "crew" WHERE ("id" = "crew"."mentor_id") LIMIT $6 ``` - One alias on the outer query does for both: with `alias: 'member'` there, `db.parent('mentor_id')` is `"member"."mentor_id"` ([`alias`](https://pgcast.dev/select#alias)). ## Order by a computed value Each cinema, and the nearest other one: a self-join again, with the distance as an [`extras`](https://pgcast.dev/select#extras) key and as the `order`. A `sql` fragment in a subquery takes `db.parent()` too: You write: ```ts const cinemas = await db.select('cinemas', db.all, { columns: ['name'], lateral: { nearest: db.selectOne('cinemas', { id: db.conditions.ne(db.parent('id')) }, { alias: 'other', columns: ['name'], extras: { km: db.sql`${'location'} <-> ${db.parent('location')}` }, order: { by: db.sql`${'location'} <-> ${db.parent('location')}`, direction: 'ASC' }, }), }, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "name") || jsonb_build_object($2::text, "lateral_nearest".result) AS result FROM "cinemas" LEFT JOIN LATERAL ( SELECT jsonb_build_object($3::text, "name") || jsonb_build_object($4::text, "location" <-> "cinemas"."location") AS result FROM "cinemas" AS "other" WHERE (("id" <> "cinemas"."id")) ORDER BY "location" <-> "cinemas"."location" ASC LIMIT $5 ) AS "lateral_nearest" ON true ) AS "sq_cinemas" ``` Values: `["name", "nearest", "name", "km", 1]` TypeScript knows: ```ts const cinemas: (db.JSONOnlyColsForTable<"cinemas", "name"[]> & db.LateralResult<{ nearest: db.SQLFragment; }> | undefined, never>; }>)[] // that is const cinemas: { name: string; nearest: (db.JSONOnlyColsForTable<"cinemas", "name"[]> & db.ExtrasResult<"cinemas", { km: db.SQLFragment; }>) | null; }[] ``` - `<->` is Postgres's distance between two points: kilometres here, as `location` is in kilometres east and north of the city centre. - `km` is a `number` because the fragment says so: `db.sql`. - [`ne(db.parent('id'))`](https://pgcast.dev/conditions#conditions.ne) leaves the cinema itself out. - For the three nearest, nest `select()` with [`limit: 3`](https://pgcast.dev/select#order-limit-offset-and-withties): `nearest` becomes an array. ## Watch for - **`strictNullChecks`**: without it (or `strict`), `columns`, `extras` and `lateral` leave the type as the full row, and it refuses nothing ([Strict](https://pgcast.dev/select#options)). - **An unchecked join**: `db.parent()` takes a column of any table, so a join on the wrong column compiles. Postgres refuses a column the parent doesn't have: `db.parent('billing')` under `films` fails with 42703, "column films.billing does not exist". - **[`withTies`](https://pgcast.dev/select#order-limit-offset-and-withties) in a nested `selectOne()`**: it writes `FETCH FIRST $2 ROWS WITH TIES`, which keeps every tied row: under an outer `select()`, the parent row comes back once per tie; under `selectOne()`, with one of them. In a nested `select()`, the ties stay in the array. Unchanged from Zapatos 6.6.1, and open as [#164](https://github.com/AlexandreDemartin/pgcast/issues/164). - **A `lock`**: without `of`, `FOR UPDATE` reaches the subqueries, and Postgres refuses the query (0A000): "FOR UPDATE is not allowed with aggregate functions" under `select()` or `count()`, "FOR UPDATE cannot be applied to the nullable side of an outer join" under `selectOne()`. Name the table ([`lock`](https://pgcast.dev/select#lock)): You write: ```ts const film = await db.selectOne('films', { id: 1000 }, { columns: ['title'], lateral: { cast: db.select('roles', { film_id: db.parent('id') }, { columns: ['character'] }) }, lock: { for: 'UPDATE', of: 'films' }, }).run(pool); ``` Postgres runs: ```sql SELECT jsonb_build_object($1::text, "title") || jsonb_build_object($2::text, "lateral_cast".result) AS result FROM "films" LEFT JOIN LATERAL ( SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($3::text, "character") AS result FROM "roles" WHERE ("film_id" = "films"."id") ) AS "sq_roles" ) AS "lateral_cast" ON true WHERE ("id" = $4) LIMIT $5 FOR UPDATE OF "films" ``` Values: `["title", "cast", "character", 1000, 1]` TypeScript knows: ```ts const film: (db.JSONOnlyColsForTable<"films", "title"[]> & db.LateralResult<{ cast: db.SQLFragment[], never>; }>) | undefined // that is const film: { title: string; cast: db.JSONOnlyColsForTable<"roles", "character"[]>[]; } | undefined ``` `FOR UPDATE OF "films"` locks the film's row, and only it, until the transaction ends: run it on a [`txnClient`](https://pgcast.dev/transaction#transaction). The rows of `roles` are read, not locked. ## See also - [`lateral`](https://pgcast.dev/select#lateral), [`alias`](https://pgcast.dev/select#alias) and [`extras`](https://pgcast.dev/select#extras): each option, the SQL it writes, its type. - [`parent()`](https://pgcast.dev/sql-template#parent): where it works, and what it compiles to. - [Why pgcast](https://pgcast.dev/why): why it works this way. --- # Pagination Source: https://pgcast.dev/paginate Show a long list a page at a time: here the films, newest first, two to a page. The last section puts the steps together in a request handler. It assumes the basics of the [tutorial](https://pgcast.dev/tutorial). **At a glance** - A page is an `order` that never ties, then a `limit`. - To move on, an `offset` for page *n*, or the last row's keys for the next page. - A total, if the page shows one: `count()`, or a window in `extras`. ## Order, then limit Ask for the first page of all the films ([`db.all`](https://pgcast.dev/sql-template#all)): an `order` that ends on a unique column, then a `limit`. You write: ```ts const page = await db.select('films', db.all, { columns: ['id', 'title', 'release_year'], order: [ { by: 'release_year', direction: 'DESC' }, { by: 'id', direction: 'DESC' }, ], limit: 2, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "id", $2::text, "title", $3::text, "release_year") AS result FROM "films" ORDER BY "release_year" DESC, "id" DESC LIMIT $4 ) AS "sq_films" ``` Values: `["id", "title", "release_year", 2]` TypeScript knows: ```ts const page: db.JSONOnlyColsForTable<"films", ("id" | "release_year" | "title")[]>[] // that is const page: { id: number; release_year: number; title: string; }[] ``` - End the [`order`](https://pgcast.dev/select#order-limit-offset-and-withties) on a unique column, `id` here. Two films of the same year can otherwise come back in either order, and from one request to the next a film can turn up on two pages or on none: Postgres promises no order among ties ([LIMIT and OFFSET](https://www.postgresql.org/docs/current/queries-limit.html)). - `limit` is sent as a parameter, `LIMIT $4`. - Name the [`columns`](https://pgcast.dev/select#columns) the page shows, and keep the `order`'s among them if you'll fetch the next page from the last row (below). Then pick how the reader moves on: | | `offset` | The last row's keys | |---|---|---| | Go to page *n* | yes | no: the next page, from the one before | | A deep page | Postgres computes every row it skips | Postgres starts at the last row, with an index that matches the `order` | | A film added or removed meanwhile | shifts every later page by one | the next page still starts after the last row shown | | It needs | an `order` that never ties | an `order` that never ties, and its columns in `columns` | ## Go to page n with offset Skip `(n - 1) × size` rows with [`offset`](https://pgcast.dev/select#order-limit-offset-and-withties): You write: ```ts const page = await db.select('films', db.all, { columns: ['id', 'title', 'release_year'], order: [ { by: 'release_year', direction: 'DESC' }, { by: 'id', direction: 'DESC' }, ], limit: 2, offset: 2, // page 2: (2 - 1) × 2 rows }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "id", $2::text, "title", $3::text, "release_year") AS result FROM "films" ORDER BY "release_year" DESC, "id" DESC LIMIT $4 OFFSET $5 ) AS "sq_films" ``` Values: `["id", "title", "release_year", 2, 2]` TypeScript knows: ```ts const page: db.JSONOnlyColsForTable<"films", ("id" | "release_year" | "title")[]>[] // that is const page: { id: number; release_year: number; title: string; }[] ``` - Past the last page, `select()` gives `[]`. - Postgres still computes the rows it skips, so page 500 costs more than page 1 ([LIMIT and OFFSET](https://www.postgresql.org/docs/current/queries-limit.html)). - A newer film added between two requests pushes the last film of page 1 onto page 2, where the reader sees it twice. For a long list that changes, start from the last row. ## Fetch the next page from the last row This is keyset pagination: ask for the rows that come after the last one you showed, in the same `order`. ### Keep the last row The last row of the page holds its keys, or is `undefined` when the page is empty: then there is no next page. ```ts const last = page.at(-1); if (last === undefined) throw new Error('No next page'); ``` Hover `last` to see its type: a row of the `columns` you asked for, or `undefined`, so the keys must be among them. With `strict` on, the compiler holds you to both: - Without the guard, `last.id` does not compile: `'last' is possibly 'undefined'` (TS18048). - Without `id` in `columns`, it does not compile either (TS2339). With `strict` off, both compile: a missing key goes as `NULL`, and the next page skips the rest of that year. ### Ask for the rows after it Give the `where` as a [`sql`](https://pgcast.dev/sql-template#sql) fragment that compares the `order`'s columns, in its order, with those of `last`, each value through [`db.param()`](https://pgcast.dev/sql-template#param). Here `last` is the one kept above, past its guard, 2008 and 1002: You write: ```ts const nextPage = await db.select('films', db.sql`(${'release_year'}, ${'id'}) < (${db.param(last.release_year)}, ${db.param(last.id)})`, { columns: ['id', 'title', 'release_year'], order: [ { by: 'release_year', direction: 'DESC' }, { by: 'id', direction: 'DESC' }, ], limit: 2, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "id", $2::text, "title", $3::text, "release_year") AS result FROM "films" WHERE ("release_year", "id") < ($4, $5) ORDER BY "release_year" DESC, "id" DESC LIMIT $6 ) AS "sq_films" ``` Values: `["id", "title", "release_year", 2008, 1002, 2]` TypeScript knows: ```ts const nextPage: db.JSONOnlyColsForTable<"films", ("id" | "release_year" | "title")[]>[] // that is const nextPage: { id: number; release_year: number; title: string; }[] ``` - `("release_year", "id") < ($4, $5)` keeps the rows before 2008 and 1002 in the order: an earlier year, or 2008 and a smaller `id` ([row-wise comparison](https://www.postgresql.org/docs/current/functions-comparisons.html#ROW-WISE-COMPARISON)). `<` goes with `DESC`; for an `ASC` order, write `>`. - With an index on `(release_year, id)`, Postgres starts reading at the last row. On 300,000 films, after the last row (1960, 150000), `EXPLAIN (ANALYZE)` shows an `Index Scan Backward` with `Index Cond: (ROW(release_year, id) < ROW(1960, 150000))`: it reads the 2 rows it returns, and none to skip. - Keep the key columns `NOT NULL`, as these are: where the comparison reaches a `NULL`, it gives `NULL`, and that row never shows up. - `s.films.SQL` lets the fragment name only the columns of `films`. ### With a filter To keep a filter beside the comparison, give the comparison in a [Whereable](https://pgcast.dev/sql-template#whereable), as the value of the `order`'s first column, `release_year`: there [`db.self`](https://pgcast.dev/sql-template#self) stands for that column, and the Whereable joins it to the other keys with `AND`. You write: ```ts const nextPage = await db.select('films', { studio_id: 1000, release_year: db.sql`(${db.self}, ${'id'}) < (${db.param(last.release_year)}, ${db.param(last.id)})`, }, { columns: ['id', 'title', 'release_year'], order: [ { by: 'release_year', direction: 'DESC' }, { by: 'id', direction: 'DESC' }, ], limit: 2, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "id", $2::text, "title", $3::text, "release_year") AS result FROM "films" WHERE ((("release_year", "id") < ($4, $5)) AND "studio_id" = $6) ORDER BY "release_year" DESC, "id" DESC LIMIT $7 ) AS "sq_films" ``` Values: `["id", "title", "release_year", 2008, 1002, 1000, 2]` TypeScript knows: ```ts const nextPage: db.JSONOnlyColsForTable<"films", ("id" | "release_year" | "title")[]>[] // that is const nextPage: { id: number; release_year: number; title: string; }[] ``` ### When the directions differ A row comparison goes one way. With `release_year` `DESC` and `id` `ASC`, spell it out with [`conditions.or`](https://pgcast.dev/conditions#conditions.or): an earlier year ([`conditions.lt`](https://pgcast.dev/conditions#conditions.lt)), or the same year and a larger `id` ([`conditions.gt`](https://pgcast.dev/conditions#conditions.gt)). Then bound it on the first key, with [`conditions.and`](https://pgcast.dev/conditions#conditions.and) and [`conditions.lte`](https://pgcast.dev/conditions#conditions.lte): that year or an earlier one. You write: ```ts const nextPage = await db.select('films', db.conditions.and( { release_year: db.conditions.lte(last.release_year) }, db.conditions.or( { release_year: db.conditions.lt(last.release_year) }, { release_year: last.release_year, id: db.conditions.gt(last.id) }, ), ), { columns: ['id', 'title', 'release_year'], order: [ { by: 'release_year', direction: 'DESC' }, { by: 'id', direction: 'ASC' }, ], limit: 2, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "id", $2::text, "title", $3::text, "release_year") AS result FROM "films" WHERE (((("release_year" <= $4))) AND (((("release_year" < $5)) OR (("id" > $6) AND "release_year" = $7)))) ORDER BY "release_year" DESC, "id" ASC LIMIT $8 ) AS "sq_films" ``` Values: `["id", "title", "release_year", 2008, 2008, 1002, 2008, 2]` TypeScript knows: ```ts const nextPage: db.JSONOnlyColsForTable<"films", ("id" | "release_year" | "title")[]>[] // that is const nextPage: { id: number; release_year: number; title: string; }[] ``` - Create the index in the `order`'s directions, `(release_year DESC, id ASC)`. Postgres starts reading at the last row's year: on 300,000 films, after the last row (1960, 150000), `EXPLAIN (ANALYZE)` shows `Index Cond: (release_year <= 1960)`, then a `Filter:` that removes the rest of that year, 1,167 rows. - Without the bound, the `OR` is only a `Filter:`: Postgres reads and throws away every row before the last one, 148,671 here, as for an `offset`. ### Send the cursor to the client A client that asks for the next page sends the last row's keys back. Give them to it as one string, a cursor, and check what comes back before it goes into a query: ```ts type Cursor = { release_year: number; id: number }; function encodeCursor({ release_year, id }: Cursor): string { return Buffer.from(JSON.stringify({ release_year, id })).toString('base64url'); } function decodeCursor(cursor: string): Cursor | undefined { try { const { release_year, id } = JSON.parse(Buffer.from(cursor, 'base64url').toString()); const isKey = (key: number) => Number.isInteger(key) && Math.abs(key) < 2 ** 31; // within Postgres's integer const made = encodeCursor({ release_year, id }) === cursor; // no stray character, no other key if (isKey(release_year) && isKey(id) && made) return { release_year, id }; } catch {} // not base64url JSON return undefined; } ``` - For 2008 and 1002, the cursor is `eyJyZWxlYXNlX3llYXIiOjIwMDgsImlkIjoxMDAyfQ`: JSON in base64url, which a URL takes as it is. - `decodeCursor()` gives `undefined` for a string it did not make: `nonsense`, an empty string, JSON that is not an object, keys that are not integers (`"2008"`, `2008.5`) or out of Postgres's `integer` (`99999999999`, `1e300`). Answer 400, as the [handler](https://pgcast.dev/paginate#a-handler-from-request-to-response) does. - It refuses a key more, standard base64's `==` and a stray `!` too: base64url decoding skips the last two, and destructuring drops a key more, so it encodes the two keys again and compares. - A cursor is encoded, not signed: the client can read it and change it. It holds only where the page starts, and each key goes to Postgres as a parameter: a changed cursor moves the page, nothing more. ## Go back a page Start from the first row you showed, and read the other way: flip the comparison and the `order`'s directions, then put the rows back in the page's order. Here `first` is page 2's first row, A Lantern for Edith (1991, id 1001), kept as `last` was, with `page.at(0)` and its guard: You write: ```ts const previousPage = await db.select('films', db.sql`(${'release_year'}, ${'id'}) > (${db.param(first.release_year)}, ${db.param(first.id)})`, { columns: ['id', 'title', 'release_year'], order: [ { by: 'release_year', direction: 'ASC' }, { by: 'id', direction: 'ASC' }, ], limit: 2, }).run(pool); previousPage.reverse(); // back to newest first ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "id", $2::text, "title", $3::text, "release_year") AS result FROM "films" WHERE ("release_year", "id") > ($4, $5) ORDER BY "release_year" ASC, "id" ASC LIMIT $6 ) AS "sq_films" ``` Values: `["id", "title", "release_year", 1991, 1001, 2]` TypeScript knows: ```ts const previousPage: db.JSONOnlyColsForTable<"films", ("id" | "release_year" | "title")[]>[] // that is const previousPage: { id: number; release_year: number; title: string; }[] ``` - `>` with `ASC` gives the rows just before `first` in the page's order, the nearest first; [`reverse()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/reverse), in place, puts them newest first again. - Fewer rows than `limit`, `[]` included: you reached the start of the list, so show the first page instead. A film added since page 1 was shown can leave one row there, not `[]`. - The index on `(release_year, id)` serves both ways. On 300,000 films, before the first row (1960, 150000), `EXPLAIN (ANALYZE)` shows an `Index Scan`, forwards this time, with `Index Cond: (ROW(release_year, id) > ROW(1960, 150000))`: it reads the 2 rows it returns. - To let a client come back, send `encodeCursor(first)` as a second cursor, `before`, beside `after`. To know whether a page comes before, fetch `size + 1` rows the other way, as the [handler](https://pgcast.dev/paginate#a-handler-from-request-to-response) does for `after`, keep `size` of them and reverse. ## Count the total Two ways, by where you want the number: | | [`count()`](https://pgcast.dev/select#count) | `count(*) OVER ()` in `extras` | |---|---|---| | Queries | one more | none, but it reads every row of the `where`, as `count()` does | | The total | a `number` | a key on each row | | An empty page | still the total | no row, so no total | | With the last row's keys | the list's total, given the list's filter | the rows left: use `count()` | Both count the rows of the `where`. The window's `where` is the page's own: with the last row's comparison in it, it counts the rows left, not the whole list. ### With `count()` You write: ```ts const total = await db.count('films', db.all).run(pool); ``` Postgres runs: ```sql SELECT count(*) AS result FROM "films" ``` Values: `[]` TypeScript knows: ```ts const total: number ``` Give it the list's `where`, and none of the page's options: - `offset` leaves the one row of the count behind, and `.run()` rejects with `TypeError: Cannot read properties of undefined (reading 'result')` ([#163](https://github.com/AlexandreDemartin/pgcast/issues/163)). - `order` by a column: Postgres refuses the query (42803). ### With a window in `extras` A window function in [`extras`](https://pgcast.dev/select#extras) counts the rows before `LIMIT` cuts them, and gives the total on each row: You write: ```ts const page = await db.select('films', db.all, { columns: ['id', 'title', 'release_year'], extras: { total: db.sql`count(*) OVER ()` }, order: [ { by: 'release_year', direction: 'DESC' }, { by: 'id', direction: 'DESC' }, ], limit: 2, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "id", $2::text, "title", $3::text, "release_year") || jsonb_build_object($4::text, count(*) OVER ()) AS result FROM "films" ORDER BY "release_year" DESC, "id" DESC LIMIT $5 ) AS "sq_films" ``` Values: `["id", "title", "release_year", "total", 2]` TypeScript knows: ```ts const page: db.ReturningTypeForTable<"films", ("id" | "release_year" | "title")[], { total: db.SQLFragment; }>[] // that is const page: { id: number; release_year: number; title: string; total: number; }[] ``` - Each row has `total: 4`. The type is the fragment's second type argument, `number`; Postgres's `bigint` arrives as a JSON number. - On an empty page, past the last one, there is no row to carry it: `page[0]?.total` is `undefined`, not the total. - The window makes the page's query read and sort every row of the `where`. On 300,000 films, `EXPLAIN (ANALYZE)` shows a `Seq Scan` and a `WindowAgg` over all of them, where the page alone is an `Index Scan Backward` that stops at its 2 rows. - With `strict` off, `total` is not in the type ([Options](https://pgcast.dev/select#options)), and `page[0].total` does not compile (TS2339). ## A handler, from request to response The steps above in one function: the filter from the query string, the cursor, whether there is a next page, the total, and the JSON reply. `listFilms()` takes the Fetch API's [`Request`](https://developer.mozilla.org/en-US/docs/Web/API/Request) and returns a [`Response`](https://developer.mozilla.org/en-US/docs/Web/API/Response), both global in Node; `encodeCursor()` and `decodeCursor()` are those of [Send the cursor to the client](https://pgcast.dev/paginate#send-the-cursor-to-the-client). ```ts const size = 2; async function listFilms(request: Request): Promise { const params = new URL(request.url).searchParams; const studio = params.get('studio'), after = params.get('after'); const cursor = after === null ? undefined : decodeCursor(after); const badStudio = studio !== null && !/^\d{1,9}$/.test(studio); // 1 to 9 digits: within Postgres's integer const badCursor = after !== null && cursor === undefined; if (badStudio || badCursor) return Response.json({ error: 'Bad studio or cursor' }, { status: 400 }); const filter: s.films.Whereable = studio === null ? {} : { studio_id: Number(studio) }; const rows = await db.select('films', { ...filter, ...(cursor && { release_year: db.sql`(${db.self}, ${'id'}) < (${db.param(cursor.release_year)}, ${db.param(cursor.id)})`, }), }, { columns: ['id', 'title', 'release_year'], order: [ { by: 'release_year', direction: 'DESC' }, { by: 'id', direction: 'DESC' }, ], limit: size + 1, // one more than the page: if it comes back, there is a next page }).run(pool); const films = rows.slice(0, size), last = films.at(-1); const nextCursor = rows.length > size && last !== undefined ? encodeCursor(last) : null; const total = await db.count('films', filter).run(pool); // the list's where, without the cursor return Response.json({ films, nextCursor, total }); } ``` On the docs schema, `GET /films` replies: ```json { "films": [ { "id": 1003, "title": "Small Hours at Calder Bay", "release_year": 2019 }, { "id": 1002, "title": "The Cartographer's Silence", "release_year": 2008 } ], "nextCursor": "eyJyZWxlYXNlX3llYXIiOjIwMDgsImlkIjoxMDAyfQ", "total": 4 } ``` - `limit: size + 1` asks for one row more than the page. If it comes back, there is a next page, and the cursor is the page's last row, not the extra one; `GET /films?after=eyJyZWxlYXNlX3llYXIiOjIwMDgsImlkIjoxMDAyfQ` gives 1001 and 1000, then `"nextCursor": null`. With `limit: size`, a full last page would send a cursor to an empty one. - The page's `where` is [the one with a filter](https://pgcast.dev/paginate#with-a-filter): `filter`, then the comparison when a cursor came. [`count()`](https://pgcast.dev/paginate#with-count) takes `filter` alone, so each page has the list's total: `GET /films?studio=1000` gives 1002 and 1000, and `"total": 2`. Leave it out, and its query, if the page shows no total. - A `studio` that is not 1 to 9 digits, or an `after` that `decodeCursor()` refuses, gets a 400 before any query: `Number()` alone would read `''` as 0, `0x3e8` as 1000, and let `99999999999` reach Postgres, which refuses it. ## See also - [`select()`](https://pgcast.dev/select#select): every option and the SQL it writes; [`count()`](https://pgcast.dev/select#count) and its options. - [SQL template](https://pgcast.dev/sql-template#sql), [`self`](https://pgcast.dev/sql-template#self) and [`param()`](https://pgcast.dev/sql-template#param), for the comparison. - [Why pgcast](https://pgcast.dev/why): why a query stays the SQL you see. --- # Batches Source: https://pgcast.dev/batches Write many rows at once: split them under Postgres's parameter limit, skip the ones already there, add up counts, keep rows with different keys from erasing what's stored, and get back only what you need. It assumes you know [`insert()` and `upsert()`](https://pgcast.dev/insert). ## Insert an array Give [`insert()`](https://pgcast.dev/insert#insert) an array, and every row goes in one query: You write: ```ts const tags = await db.insert('tags', [ { film_id: 1000, tag: 'orchards' }, { film_id: 1000, tag: 'slow cinema' }, { film_id: 1002, tag: 'maps' }, ]).run(pool); ``` Postgres runs: ```sql INSERT INTO "tags" ("film_id", "tag") VALUES ($1, $2), ($3, $4), ($5, $6) RETURNING to_jsonb("tags".*) AS result ``` Values: `[1000, "orchards", 1000, "slow cinema", 1002, "maps"]` TypeScript knows: ```ts const tags: tags.JSONSelectable[] // that is const tags: { film_id: number; tag: string; }[] ``` Each cell is a parameter: three rows of two columns, `VALUES ($1, $2), ($3, $4), ($5, $6) RETURNING`. An empty array sends nothing and resolves with `[]`. ## Split under the parameter limit Postgres takes at most 65,535 parameters in a query. Past that, it refuses the query as a protocol violation, 08P01, with a message that varies with the count and doesn't say why, such as `bind message supplies 0 parameters, but prepared statement "" requires 65536`. A query's parameters, to count before you send it: - one per cell: rows × columns; - none for a cell that is `DEFAULT`, a key missing from its row or [`db.Default`](https://pgcast.dev/sql-template#Default); a `sql` fragment sends its own; - one per column of `returning` and per key of [`extras`](https://pgcast.dev/insert#insert-options), and the parameters of `updateValues`, once. Or ask the query itself: [`.compile()`](https://pgcast.dev/sql-template#compile)`.values.length` of a one-row insert, as below, when every row has the same keys. Then send the rows in chunks of a fixed size, well under the limit. In one transaction, every chunk goes in or none does: ```ts const perRow = db.insert('tags', tags.slice(0, 1)).compile().values.length; // 2: film_id and tag const chunkSize = Math.floor(10_000 / perRow); // 10,000 parameters a query await db.readCommitted(pool, async txnClient => { for (let i = 0; i < tags.length; i += chunkSize) { await db.insert('tags', tags.slice(i, i + chunkSize)).run(txnClient); } }); ``` ```sql START TRANSACTION ISOLATION LEVEL READ COMMITTED -- the insert() of each chunk: 20 of them for 100,000 tags COMMIT ``` - If a chunk fails, [`readCommitted()`](https://pgcast.dev/transaction#readCommitted) sends `ROLLBACK` and rejects with its error: no tag is left in. A deadlock (40P01) runs the chunks again first ([Retries](https://pgcast.dev/retries)). - On the pool, without the transaction, each chunk commits as it goes: the chunks before the one that failed stay. - `upsert()` splits the same way. ## Skip the rows already there Give [`upsert()`](https://pgcast.dev/insert#upsert) the rows and [`updateColumns: db.doNothing`](https://pgcast.dev/insert#doNothing): Postgres inserts the new ones and leaves the others as they are. You write: ```ts const added = await db.upsert('tags', [ { film_id: 1000, tag: 'orchards' }, { film_id: 1003, tag: 'lighthouses' }, ], ['film_id', 'tag'], { updateColumns: db.doNothing }).run(pool); ``` Postgres runs: ```sql INSERT INTO "tags" ("film_id", "tag") VALUES ($1, $2), ($3, $4) ON CONFLICT ("film_id", "tag") DO NOTHING RETURNING to_jsonb("tags".*) || jsonb_build_object('$action', CASE xmax WHEN 0 THEN 'INSERT' ELSE 'UPDATE' END) AS result ``` Values: `[1000, "orchards", 1003, "lighthouses"]` TypeScript knows: ```ts const added: db.UpsertReturnableForTable<"tags", db.ColumnsOption<"tags">, db.ExtrasOption<"tags">, "suppress" | undefined>[] // that is const added: { film_id: number; tag: string; $action: "INSERT" | "UPDATE"; }[] ``` - Only the rows inserted come back ([`doNothing`](https://pgcast.dev/insert#doNothing)): with `orchards` already tagged, `added` holds `lighthouses` alone. - A row given twice in the array is inserted once. ## Add up counts Except with `doNothing`, an upsert refuses an array that holds the same key twice: 21000, `ON CONFLICT DO UPDATE command cannot affect row a second time`. So sum the ballots first, one row per `film_id` and `stars`: ```ts const ballots = [ { film_id: 1002, stars: 4 }, { film_id: 1002, stars: 4 }, { film_id: 1002, stars: 5 }, { film_id: 1000, stars: 5 }, { film_id: 1000, stars: 5 }, { film_id: 1000, stars: 5 }, ]; const counts = new Map(); for (const { film_id, stars } of ballots) { const key = `${film_id}:${stars}`; const row = counts.get(key) ?? { film_id, stars, votes: 0 }; row.votes += 1; counts.set(key, row); } const rows = [...counts.values()]; // the three rows below ``` Then add each row's votes to those stored: set `votes` in [`updateValues`](https://pgcast.dev/insert#upsert-options) from both, the stored row by its table's name, the new one as `EXCLUDED`. You write: ```ts const ratings = await db.upsert('ratings', [ { film_id: 1002, stars: 4, votes: 2 }, { film_id: 1002, stars: 5, votes: 1 }, { film_id: 1000, stars: 5, votes: 3 }, ], ['film_id', 'stars'], { updateValues: { votes: db.sql`${'ratings'}.${'votes'} + EXCLUDED.${'votes'}` }, }).run(pool); ``` Postgres runs: ```sql INSERT INTO "ratings" ("film_id", "stars", "votes") VALUES ($1, $2, $3), ($4, $5, $6), ($7, $8, $9) ON CONFLICT ("film_id", "stars") DO UPDATE SET ("film_id", "stars", "votes") = ROW(EXCLUDED."film_id", EXCLUDED."stars", "ratings"."votes" + EXCLUDED."votes") RETURNING to_jsonb("ratings".*) || jsonb_build_object('$action', CASE xmax WHEN 0 THEN 'INSERT' ELSE 'UPDATE' END) AS result ``` Values: `[1002, 4, 2, 1002, 5, 1, 1000, 5, 3]` TypeScript knows: ```ts const ratings: db.UpsertReturnableForTable<"ratings", db.ColumnsOption<"ratings">, db.ExtrasOption<"ratings">, "suppress" | undefined>[] // that is const ratings: { film_id: number; stars: number; votes: number; $action: "INSERT" | "UPDATE"; }[] ``` - The first run inserts the three rows. The second updates them, to 4, 2 and 6 votes, each with `$action: 'UPDATE'`. - An `undefined` in `updateValues` doesn't leave `votes` alone ([An `undefined` value](https://pgcast.dev/insert#undefined-values), [#168](https://github.com/AlexandreDemartin/pgcast/issues/168)). ## Rows with different keys pgcast inserts a key that some rows lack as `DEFAULT` in those rows. Without `updateColumns`, an upsert then sets every column from `EXCLUDED`, that `DEFAULT` included. Say you correct the names on two bookings, and only one of them has its `booked_at`: - the second row is `(DEFAULT, $5, $6, $7)`, and `"booked_at"` is set from `EXCLUDED."booked_at"`: F8's booking time becomes the time of the upsert, `now()`, the column's default; - pgcast warns once per process, `PGCAST_DEP001`. Name the columns to update instead, with [`updateColumns`](https://pgcast.dev/insert#upsert-options): You write: ```ts const bookings = await db.upsert('seat_bookings', [ { screening_id: 1000, seat: 'F7', customer: 'Ines Calloway', booked_at: '2030-02-20T09:15:00Z' }, { screening_id: 1000, seat: 'F8', customer: 'Barnaby Quill-Hart' }, ], ['screening_id', 'seat'], { updateColumns: 'customer' }).run(pool); ``` Postgres runs: ```sql INSERT INTO "seat_bookings" ("booked_at", "customer", "screening_id", "seat") VALUES ($1, $2, $3, $4), (DEFAULT, $5, $6, $7) ON CONFLICT ("screening_id", "seat") DO UPDATE SET ("customer") = ROW(EXCLUDED."customer") RETURNING to_jsonb("seat_bookings".*) || jsonb_build_object('$action', CASE xmax WHEN 0 THEN 'INSERT' ELSE 'UPDATE' END) AS result ``` Values: `["2030-02-20T09:15:00Z", "Ines Calloway", 1000, "F7", "Barnaby Quill-Hart", 1000, "F8"]` TypeScript knows: ```ts const bookings: db.UpsertReturnableForTable<"seat_bookings", db.ColumnsOption<"seat_bookings">, db.ExtrasOption<"seat_bookings">, "suppress" | undefined>[] // that is const bookings: { booked_at: db.TimestampTzString; customer: string; screening_id: number; seat: PgSeat_label; $action: "INSERT" | "UPDATE"; }[] ``` - The update is `DO UPDATE SET ("customer") = ROW(EXCLUDED."customer")`: F8 keeps its booking time. A new row still takes the `booked_at` given, or the default. - Or give every row the same keys: upsert the rows with a `booked_at` and those without in two calls. - Not every key counts as missing ([Rows with different keys](https://pgcast.dev/insert#upsert-different-keys)). > **Opt-in** > > With `db.setConfig({ upsertRequiresSameKeys: true })` ([`Config.upsertRequiresSameKeys`](https://pgcast.dev/runtime-config#Config.upsertRequiresSameKeys)), such an upsert throws before it sends anything. ### An `undefined` in a row Rows built from input often hold an `undefined`, such as a `booked_at` the form left empty. By default it's sent as `NULL`, as in Zapatos 6.6.1. `booked_at` is `NOT NULL`, so Postgres refuses the whole batch, 23502, `null value in column "booked_at" of relation "seat_bookings" violates not-null constraint`, and pgcast warns once, `PGCAST_DEP002`. > **Opt-in** > > With `db.setConfig({ undefinedMeansMissing: true })` ([`Config.undefinedMeansMissing`](https://pgcast.dev/runtime-config#Config.undefinedMeansMissing)), an `undefined` is a missing key: F8 with `booked_at: undefined` is `(DEFAULT, $5, $6, $7)`, a row with different keys as above ([An `undefined` value](https://pgcast.dev/insert#undefined-values)). ## Choose what comes back Each row comes back whole by default, and from an upsert with its `$action`. For a big batch, ask for less: You write: ```ts const bookings = await db.upsert('seat_bookings', [ { screening_id: 1000, seat: 'F7', customer: 'Ines Calloway', booked_at: '2030-02-20T09:15:00Z' }, { screening_id: 1000, seat: 'F8', customer: 'Barnaby Quill-Hart' }, ], ['screening_id', 'seat'], { updateColumns: 'customer', returning: ['seat'] }).run(pool); const newSeats = bookings.filter(b => b.$action === 'INSERT').map(b => b.seat); ``` Postgres runs: ```sql INSERT INTO "seat_bookings" ("booked_at", "customer", "screening_id", "seat") VALUES ($1, $2, $3, $4), (DEFAULT, $5, $6, $7) ON CONFLICT ("screening_id", "seat") DO UPDATE SET ("customer") = ROW(EXCLUDED."customer") RETURNING jsonb_build_object($8::text, "seat") || jsonb_build_object('$action', CASE xmax WHEN 0 THEN 'INSERT' ELSE 'UPDATE' END) AS result ``` Values: `["2030-02-20T09:15:00Z", "Ines Calloway", 1000, "F7", "Barnaby Quill-Hart", 1000, "F8", "seat"]` TypeScript knows: ```ts const bookings: db.UpsertReturnableForTable<"seat_bookings", "seat"[], db.ExtrasOption<"seat_bookings">, "suppress" | undefined>[] // that is const bookings: { seat: PgSeat_label; $action: "INSERT" | "UPDATE"; }[] ``` - [`returning: ['seat']`](https://pgcast.dev/insert#insert-options) sends `jsonb_build_object($8::text, "seat")` in place of `to_jsonb("seat_bookings".*)`. - `$action` tells the new rows from the updated ones: `newSeats` is `['F8']` when F7 was already booked. - [`reportAction: 'suppress'`](https://pgcast.dev/insert#upsert-options) leaves `$action` out of the SQL and of the type, for a batch that doesn't need it. - Both narrow the type only with `strictNullChecks` on ([Options](https://pgcast.dev/insert#insert-options)). ## See also - [`insert()`](https://pgcast.dev/insert#insert) and [`upsert()`](https://pgcast.dev/insert#upsert): every option, the SQL it adds. - [`transaction()`](https://pgcast.dev/transaction#transaction): its levels, retries and errors. - [Versions](https://pgcast.dev/versions): the opt-ins and their warnings. --- # Retries Source: https://pgcast.dev/retries Book seats at a screening under `SERIALIZABLE`, so that two requests at once can't break a rule, and let pgcast run the transaction again when Postgres refuses one. Then what goes in the callback and what stays out of it, and how to test a retry. It assumes the basics of the [tutorial](https://pgcast.dev/tutorial). ## Put the rule and its writes in one callback The rule: four seats per customer per screening, at most. No constraint can say that, so the code counts, then books, both in one [`serializable()`](https://pgcast.dev/transaction#serializable) transaction: ```ts const bookSeats = (txnClientOrPool: pg.Pool | db.TxnClientForSerializable, screening_id: number, seats: string[], customer: string) => db.serializable(txnClientOrPool, async txnClient => { const held = await db.count('seat_bookings', { screening_id, customer }).run(txnClient); if (held + seats.length > 4) return null; // four each, at most return db.insert('seat_bookings', seats.map(seat => ({ screening_id, seat, customer }))).run(txnClient); }); // the same customer, in two tabs at once const [tabOne, tabTwo] = await Promise.all([ bookSeats(pool, 1000, ['D4', 'D5'], 'Ines Calloway'), bookSeats(pool, 1000, ['E4', 'E5', 'E6'], 'Ines Calloway'), ]); ``` Ines has two tabs open, and books from both at once. Each transaction counts 0 seats held, then books, which would give her five. At `SERIALIZABLE`, Postgres refuses one of the two, at its `INSERT` or at its `COMMIT`, with 40001 (`serialization_failure`); pgcast rolls it back and runs the whole callback again. Here, `tabTwo`'s transaction sends: ```sql -- attempt 1 START TRANSACTION ISOLATION LEVEL SERIALIZABLE -- count(): 0 seats held, then insert() of E4, E5 and E6 COMMIT -- refused with 40001: tabOne has committed D4 and D5 ROLLBACK -- attempt 2, 25 to 250 ms later START TRANSACTION ISOLATION LEVEL SERIALIZABLE -- count(): 2 seats held, and 2 + 3 > 4, so no insert() COMMIT ``` The tab whose transaction commits first gets its rows, `seat_bookings.JSONSelectable[]`; the other gets `null`, here `tabTwo`. The SQL of the callback's queries: [`count()`](https://pgcast.dev/select#count) and [`insert()`](https://pgcast.dev/insert#insert); what pgcast sends around them, and when it retries: [Retries](https://pgcast.dev/transaction#retries). - Run every query of the transaction on `txnClient`. A query on `pool` is not in it: it isn't rolled back, and the next attempt runs it again. - Two requests may still run one after the other, and then neither is retried: the second simply counts the first one's seats. ## Keep side effects out of the callback The callback runs once per attempt, up to five times by default. Postgres rolls back its queries each time; nothing else is undone. So what isn't a query on `txnClient` goes before the transaction or after it: ```ts const screening_id = 1000, seats = ['D4', 'D5'], customer = 'Ines Calloway'; // before: runs once const limit = await fetchSeatLimit(customer); const booked = await db.serializable(pool, async txnClient => { const held = await db.count('seat_bookings', { screening_id, customer }).run(txnClient); if (held + seats.length > limit) return null; return db.insert('seat_bookings', seats.map(seat => ({ screening_id, seat, customer }))).run(txnClient); }); // after: runs once, and only once committed if (booked) await sendConfirmation(customer, booked.map(booking => booking.seat)); ``` | Work | Where | |---|---| | the queries of the rule and the booking | in the callback, on `txnClient` | | a `fetch()`, a file, a price from another service | before | | an email, a payment, a message on a queue | after the `await`, from what the callback returned | | a counter, an array, a log line | from the return value, not changed in the callback | Moved into the callback, a retried booking sends two confirmation emails, and pushes its seats twice onto an array outside it. ## Let the error through pgcast retries only what reaches it. Don't swallow a database error inside the callback: after it, Postgres refuses every query of the transaction with 25P02 (`current transaction is aborted, commands ignored until end of transaction block`), and if the callback returns, pgcast sends `COMMIT`, which Postgres turns into a rollback without an error. The promise then resolves with what the callback returned, nothing is saved, and a 40001 caught this way is never retried; pgcast warns once, `PGCAST_DEP003`. With [`transactionRequiresCommit: true`](https://pgcast.dev/runtime-config#Config.transactionRequiresCommit), the default from 7.0, the promise rejects instead, still without a retry: pgcast never saw the error. If you catch one inside, rethrow it: an `Error`, not `null` or `undefined`, for which pgcast rejects with a `TypeError` of its own instead ([#179](https://github.com/AlexandreDemartin/pgcast/issues/179)). Catch it outside the transaction, once pgcast has given up, with [`isDatabaseError()`](https://pgcast.dev/transaction#isDatabaseError): ```ts const reply = async (seats: string[], customer: string) => { try { const booked = await bookSeats(pool, 1001, seats, customer); return booked ? 'Booked' : 'Four seats each, at most'; } catch (err: any) { if (db.isDatabaseError(err, 'IntegrityConstraintViolation_UniqueViolation')) return 'Seat taken'; if (db.isDatabaseError(err, 'TransactionRollback_SerializationFailure', 'TransactionRollback_DeadlockDetected')) return 'Busy: please try again'; throw err; } }; ``` | What happened | `bookSeats()` | `reply()` | |---|---|---| | booked | resolves with the rows | `'Booked'` | | over the four seats | resolves with `null` | `'Four seats each, at most'` | | a seat already booked by someone else | rejects with 23505, sometimes after a retry | `'Seat taken'` | | a conflict on every attempt | rejects with the last attempt's 40001 or 40P01 | `'Busy: please try again'` | ## Book inside a larger transaction Given the `txnClient` of a transaction already open, `bookSeats()` joins it: `serializable()` sends nothing of its own and calls the callback at once ([Passing a transaction through](https://pgcast.dev/transaction#passing-a-transaction-through)). A couple books two seats, both or neither: ```ts // a couple: both bookings, or neither const couple = await db.serializable(pool, async txnClient => { const wren = await bookSeats(txnClient, 1000, ['G1'], 'Wren Abernathy'); const barnaby = await bookSeats(txnClient, 1000, ['G2'], 'Barnaby Quill'); if (!wren || !barnaby) throw new Error('Both seats, or neither'); // rolls back the other one return [...wren, ...barnaby]; }); ``` One transaction: `START TRANSACTION ISOLATION LEVEL SERIALIZABLE`, the two `count()`s and `insert()`s, `COMMIT`. If either `bookSeats()` resolves with `null`, the `throw` sends `ROLLBACK` in place of `COMMIT`, and the `await` rejects with that `Error`, not retried. On a 40001, it is the outer transaction that rolls back and runs its callback again, both bookings included. Typed `pg.Pool | db.TxnClientForSerializable`, the first parameter refuses the client of a transaction at a lower level, where the rule would no longer hold (2345, with `strict` on or off): ```ts await db.readCommitted(pool, txnClient => bookSeats(txnClient, 1000, ['G1'], 'Wren Abernathy')); // error TS2345 ``` Typed [`db.Queryable`](https://pgcast.dev/sql-template#Queryable), it would take that client, and `bookSeats()` would run at `READ COMMITTED` ([#177](https://github.com/AlexandreDemartin/pgcast/issues/177)). ## Tune and watch the retries Three options of [`setConfig()`](https://pgcast.dev/runtime-config#setConfig), for every transaction of the process: ```ts db.setConfig({ transactionAttemptsMax: 10, transactionRetryDelay: { minMs: 50, maxMs: 500 }, transactionListener: (message, txnId) => console.log(`[txn ${txnId}] ${message}`), }); ``` A retried transaction then logs: ```txt [txn 0] Transaction rollback (code 40001) on attempt 1 of 10, retrying in 292ms [txn 0] Retrying transaction, attempt 2 of 10 ``` - [`transactionAttemptsMax`](https://pgcast.dev/runtime-config#Config.transactionAttemptsMax): the attempts in all. `1` turns the retries off. - [`transactionRetryDelay`](https://pgcast.dev/runtime-config#Config.transactionRetryDelay): the wait before each retry. The client stays checked out of the pool while it waits. - [`transactionListener`](https://pgcast.dev/runtime-config#Config.transactionListener): every message it gets is in [Retries](https://pgcast.dev/transaction#retries). ## Test a retry Two real requests at once may or may not conflict, so a test can't count on it. Throw the failure yourself instead: pgcast retries any error whose `code` is `'40001'`. From an outer transaction, `bookSeats()` joins it, so its rows are rolled back with the failed attempt: ```ts db.setConfig({ transactionAttemptsMax: 2, transactionRetryDelay: { minMs: 0, maxMs: 0 } }); test('a booking runs again after a serialization failure', async () => { await db.deletes('seat_bookings', { screening_id: 1003 }).run(pool); // so that it runs again on the same database let attempts = 0; const booked = await db.serializable(pool, async txnClient => { const seats = await bookSeats(txnClient, 1003, ['H1', 'H2'], 'Tamsin Rowe'); if (++attempts === 1) throw Object.assign(new Error('A serialization failure, for the test'), { code: '40001' }); return seats; }); assert.equal(attempts, 2); assert.deepEqual(booked?.map(booking => booking.seat), ['H1', 'H2']); assert.equal(await db.count('seat_bookings', { screening_id: 1003, customer: 'Tamsin Rowe' }).run(pool), 2); }); test('a booking gives up after its last attempt', async () => { let attempts = 0; await assert.rejects(db.serializable(pool, async () => { attempts++; throw Object.assign(new Error('A serialization failure, for the test'), { code: '40001' }); }), { code: '40001' }); assert.equal(attempts, 2); }); ``` Here with `test` from `node:test` and `assert` from `node:assert/strict`; any test runner does. The first test proves the callback can run twice: the first attempt's two seats are gone, and the second books them once. - The second test gives up: its callback throws on every attempt, so with `transactionAttemptsMax: 2` it runs twice, and the promise rejects with the very error it threw, after `on attempt 2 of 2, giving up`. - `transactionRetryDelay` at `0` keeps the test fast. `setConfig()` holds for the whole process: put back what the other tests expect ([`getConfig()`](https://pgcast.dev/runtime-config#getConfig)). ## See also - [`transaction()`](https://pgcast.dev/transaction#transaction): its isolation levels, the SQL it sends, its retries and its errors. - [`TxnClientForSerializable`](https://pgcast.dev/transaction#TxnClientForSerializable) and [`IsolationSatisfying`](https://pgcast.dev/transaction#IsolationSatisfying): which client a level accepts. - [Errors](https://pgcast.dev/handle-errors): the errors of an app, from Postgres and from pgcast. --- # Custom types Source: https://pgcast.dev/custom-types A type the generator can't map to TypeScript, such as `point`, and each domain come to you as a placeholder in `zapatos/custom`, for you to fill in. Here we fill in `cinemas.location`, a `POINT`, and find the nearest cinema with it, then type the domain `seat_label`, keep both across runs, and fix a placeholder that does not compile. ## Find the placeholder The first time the generator meets such a type, it writes a placeholder for it and says so: ```txt Writing new custom type or domain placeholder file: src/zapatos/custom/PgPoint.d.ts ``` The placeholder gives it `any`: ```ts [src/zapatos/custom/PgPoint.d.ts] declare module 'zapatos/custom' { export type PgPoint = any; // replace with your custom type or interface as desired } ``` In `schema.d.ts`, the column takes that type in all five interfaces: `location: c.PgPoint`, in `cinemas.Selectable` and `JSONSelectable` as in `Whereable`, `Insertable` and `Updatable`. While it is `any`, TypeScript checks nothing: `location: { x: 1.5, y: 2.5 }` compiles, and Postgres refuses it with 22P02, `invalid input syntax for type point: "{"x":1.5,"y":2.5}"`. The file's name, `PgPoint`, comes from [`customTypesTransform`](https://pgcast.dev/generator#generate.customTypesTransform). ## Fill it in The shortcuts return each row as JSON, which writes a point as Postgres prints it: `"(0.4,1.2)"`. Postgres takes that string back as a value too. So in `src/zapatos/custom/PgPoint.d.ts`, replace `any` with that form, a template literal type: ```ts [src/zapatos/custom/PgPoint.d.ts] declare module 'zapatos/custom' { export type PgPoint = `(${number},${number})`; // a point as Postgres writes it: (x,y) } ``` Keep the edit within `declare module 'zapatos/custom'`: the file is the module's declaration. Now a cinema's location is checked as you write it: You write: ```ts const cinema = await db.insert('cinemas', { name: 'The Harbour Lights', location: '(1.5,2.5)', }).run(pool); ``` Postgres runs: ```sql INSERT INTO "cinemas" ("location", "name") VALUES ($1, $2) RETURNING to_jsonb("cinemas".*) AS result ``` Values: `["(1.5,2.5)", "The Harbour Lights"]` TypeScript knows: ```ts const cinema: cinemas.JSONSelectable // that is const cinema: { id: number; location: PgPoint; name: string; } ``` And the object that Postgres refused no longer compiles, here as a row typed `s.cinemas.Insertable`: ```ts const cinema: s.cinemas.Insertable = { name: 'The Harbour Lights', location: { x: 1.5, y: 2.5 } }; // error TS2353 ``` From two numbers, build the point in SQL, each number a [parameter](https://pgcast.dev/sql-template#param): this sends `VALUES (point($1, $2), $3)`, as an `Insertable` value can be any [`sql`](https://pgcast.dev/sql-template#sql) fragment. ```ts await db.insert('cinemas', { name: 'The Harbour Lights', location: db.sql`point(${db.param(x)}, ${db.param(y)})`, }).run(pool); ``` Postgres also takes `'1.5,2.5'`, which the type refuses. The type checks the parentheses and the comma, and lets spaces through: `'(1.5, 2.5)'` compiles, and Postgres takes it. One type serves all five interfaces. But your own query typed `s.cinemas.Selectable` gets its rows from `pg`, which parses a point into `{ x: 0.4, y: 1.2 }`, whatever the type says. Select the point as text there, and the rows fit the type: You write: ```ts type Cinema = Pick; const cinemas = await db.sql`SELECT ${'name'}, ${'location'}::text AS ${'location'} FROM ${'cinemas'}`.run(pool); ``` Postgres runs: ```sql SELECT "name", "location"::text AS "location" FROM "cinemas" ``` Values: `[]` TypeScript knows: ```ts const cinemas: Cinema[] // that is const cinemas: { name: string; location: PgPoint; }[] ``` The Lantern's `location` is then `"(0.4,1.2)"`, as the shortcuts give it. Or type that query's result yourself ([`sql`](https://pgcast.dev/sql-template#sql)). ## Find the nearest cinema `<->` is Postgres's distance between two points, here in kilometres. Order by it, and [`selectOne()`](https://pgcast.dev/select#selectOne) gives the nearest cinema: You write: ```ts const nearest = await db.selectOne('cinemas', db.all, { columns: ['name', 'location'], order: { by: db.sql`${'location'} <-> point(${db.param(0.5)}, ${db.param(1)})`, direction: 'ASC', }, }).run(pool); ``` Postgres runs: ```sql SELECT jsonb_build_object($1::text, "name", $2::text, "location") AS result FROM "cinemas" ORDER BY "location" <-> point($3, $4) ASC LIMIT $5 ``` Values: `["name", "location", 0.5, 1, 1]` TypeScript knows: ```ts const nearest: db.JSONOnlyColsForTable<"cinemas", ("name" | "location")[]> | undefined // that is const nearest: { name: string; location: PgPoint; } | undefined ``` - The Lantern comes back, its `location` the string `"(0.4,1.2)"`, typed `PgPoint`. - For the distance too, add the same fragment to [`extras`](https://pgcast.dev/select#extras): `` km: db.sql`…` ``. - The GiST index on `location` serves this order: on 20,000 cinemas, `EXPLAIN` shows `Index Scan using cinemas_location_idx` and `Order By: (location <-> '(0.5,1)'::point)`, with no sort. To find cinemas at a point, compare with `~=`, Postgres's "same as" for points, in a fragment as the `where`: You write: ```ts const here = await db.select('cinemas', db.sql`${'location'} ~= ${db.param('(0.4,1.2)')}`).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("cinemas".*) AS result FROM "cinemas" WHERE "location" ~= $1 ) AS "sq_cinemas" ``` Values: `["(0.4,1.2)"]` TypeScript knows: ```ts const here: s.cinemas.JSONSelectable[] // that is const here: { id: number; location: PgPoint; name: string; }[] ``` It gives The Lantern. A Whereable `{ location: '(0.4,1.2)' }` compiles but fails in Postgres with 42883, `operator does not exist: point = unknown`: a point has no `=`. To name the type in your own code, import `zapatos/custom` as you import `zapatos/schema`: ```ts import type * as c from 'zapatos/custom'; const here: c.PgPoint = '(0.5,1)'; ``` ## Type a domain A domain gets a placeholder too, which holds its base type's `Selectable` type instead of `any`: the type `pg` returns. That one type serves all five interfaces, `JSONSelectable` included. `seat_label` is a `TEXT` with a `CHECK`, so `src/zapatos/custom/PgSeat_label.d.ts` holds `string`, right in each: ```ts [src/zapatos/custom/PgSeat_label.d.ts] declare module 'zapatos/custom' { export type PgSeat_label = string; // replace with your custom type or interface as desired } ``` And `seat_bookings.seat` is `c.PgSeat_label`. The `CHECK` stays in Postgres: TypeScript lets `seat: 'f7'` through, and Postgres refuses it with 23514, `value for domain seat_label violates check constraint "seat_label_check"`. Catch it with [`isDatabaseError()`](https://pgcast.dev/transaction#isDatabaseError). A template literal type can hold part of the `CHECK`, a capital letter then a number: ```ts [src/zapatos/custom/PgSeat_label.d.ts] declare module 'zapatos/custom' { type PgSeat_labelRow = 'A' | 'B' | 'C' | 'D' | 'E' | 'F' | 'G' | 'H' | 'I' | 'J' | 'K' | 'L' | 'M' | 'N' | 'O' | 'P' | 'Q' | 'R' | 'S' | 'T' | 'U' | 'V' | 'W' | 'X' | 'Y' | 'Z'; export type PgSeat_label = `${PgSeat_labelRow}${number}`; // a row, then a number: F7 } ``` ```ts [src/seats.ts] import * as db from 'zapatos/db'; import type * as c from 'zapatos/custom'; import type pg from 'pg'; function isSeatLabel(seat: string): seat is c.PgSeat_label { return /^[A-Z][0-9]{1,2}$/.test(seat); // the domain's CHECK } export async function bookSeat(pool: pg.Pool, screening_id: number, seat: string, customer: string) { if (!isSeatLabel(seat)) throw new Error(`There is no seat ${seat}`); return db.insert('seat_bookings', { screening_id, seat, customer }).run(pool); } ``` - `seat: 'f7'` then fails to compile, TS2322 in a row typed `s.seat_bookings.Insertable`; `'Z100'` still compiles, and Postgres still refuses it. - Each type in `declare module 'zapatos/custom'` is exported, `PgSeat_labelRow` too: a helper named `Row` would clash with another placeholder's (TS2300). - A `string` from a form no longer fits (TS2322 in a typed row, TS2769 in the call): check it first, with the `CHECK`'s pattern, in a type guard, as `src/seats.ts` does. - These docs keep `PgSeat_label` as generated, `string`. ### A `jsonb` domain A domain is also how to give a `jsonb` column a shape. Say a cinema has a `screen` column, of a domain over `jsonb`: ```sql CREATE DOMAIN projection AS jsonb; ALTER TABLE cinemas ADD COLUMN screen projection; ``` The generator writes `PgProjection.d.ts` with `db.JSONValue` and the import it needs. Replace the type with an interface: ```ts [src/zapatos/custom/PgProjection.d.ts] declare module 'zapatos/custom' { export interface PgProjection { format: '35mm' | '70mm' | 'digital'; seats: number; } } ``` A row typed `s.cinemas.Insertable` with `screen: { format: 'VHS', seats: 120 }` then fails to compile (TS2322), and a cinema's `screen.format` is typed `'35mm' | '70mm' | 'digital'`. Postgres checks none of it, and stores that value all the same: the type is your promise about what the column holds. ## Keep it across runs The generator writes a placeholder once, when it is missing, and never overwrites it. With [`progressListener`](https://pgcast.dev/generator#generate.progressListener), the next runs say: ```txt Custom type or domain declaration file already exists: src/zapatos/custom/PgPoint.d.ts Custom type or domain declaration file already exists: src/zapatos/custom/PgSeat_label.d.ts ``` - Commit `zapatos/custom/` with your code: the placeholders you filled in are yours. - The placeholder does not follow the database. Recreate a domain over another type, and edit its placeholder; or delete the file, and the next run writes a fresh one. - Nothing is deleted: rename or drop a domain, and its old placeholder stays until you delete it. - An array column is a custom type of its own: `point[]` gets `Pg_point.d.ts`, `any`, and `seat_label[]` gets `Pg_seat_label.d.ts`, `any` too. - Two domains of the same name in two schemas share one placeholder ([#42](https://github.com/AlexandreDemartin/pgcast/issues/42)). ## When a placeholder does not compile A domain's placeholder holds its base type's TypeScript name, and imports `zapatos/db` only for `db.JSONValue`. Some placeholders then fail to compile until you add the import, as in Zapatos 6.6.1 ([#184](https://github.com/AlexandreDemartin/pgcast/issues/184)): | Placeholder type | `tsc` says | Add, inside `declare module` | |---|---|---| | `db.Int8String`, for a domain over `bigint`; `numeric`, `time` and the ranges alike | TS2503, `Cannot find namespace 'db'.` | `import type * as db from 'zapatos/db';` | | `crew_job`, for `CREATE DOMAIN lead_job AS crew_job`, over an enum | TS2304, `Cannot find name 'crew_job'.` | `import type { crew_job } from 'zapatos/schema';` | For `CREATE DOMAIN ticket_no AS bigint`, the placeholder holds `db.Int8String`, the string `pg` returns for a `bigint`. But the shortcuts return a `bigint` as a JSON number: a `ticket_no` that [`select()`](https://pgcast.dev/select#select) returns is a `number` typed `db.Int8String`, so `.padStart()` compiles, then throws; and a row for [`insert()`](https://pgcast.dev/insert#insert), typed `Insertable`, refuses `1` (TS2322). With the import, type it as both: ```ts [src/zapatos/custom/PgTicket_no.d.ts] declare module 'zapatos/custom' { import type * as db from 'zapatos/db'; export type PgTicket_no = number | db.Int8String; // pg returns a string, the shortcuts a number } ``` - A domain over `numeric`, `date`, `timestamp`, `timestamptz` or `bytea` is in the same case: its placeholder holds what `pg` returns, not what the shortcuts return ([#194](https://github.com/AlexandreDemartin/pgcast/issues/194)). - Past `Number.MAX_SAFE_INTEGER`, the number loses its last digits, as for any `bigint`: `9007199254740993` comes back as `9007199254740992` ([`customJSONParsingForLargeNumbers`](https://pgcast.dev/generator#generate.customJSONParsingForLargeNumbers)). ## See also - [The files it writes](https://pgcast.dev/generator#the-files-it-writes): each file the generator writes, and when. - [`customTypesTransform`](https://pgcast.dev/generator#generate.customTypesTransform): the names of the placeholders. - [The catalog](https://pgcast.dev/catalog-types): why a type differs between `Selectable` and `JSONSelectable`. --- # Raw SQL Source: https://pgcast.dev/raw-sql When no shortcut writes the query you need (a join, a window function, an `INSERT … SELECT`), write it in the `sql` template, and keep every value that comes from outside out of its text. You already know the shortcuts from the [tutorial](https://pgcast.dev/tutorial); every interpolation is in [SQL template](https://pgcast.dev/sql-template). **At a glance** - Names go between `${…}`, checked against the tables you give `sql`; a value from outside goes in as `db.param()` or in a Whereable. - Syntax a caller chooses, such as a sort direction, is one of your own constants: never give `raw()` a caller's text. - `.compile()` shows the `text` and `values` a query would send: a test that freezes `text` keeps every value from outside in `values`. ## Write the query, its names interpolated Write the SQL as you would in `psql`, but put each table and column name between `${…}` instead of in the text. Give `sql` the tables the query may name, then the type of the rows you expect: You write: ```ts type Tables = s.films.SQL | s.screenings.SQL | s.cinemas.SQL; type Listing = Pick & Pick; function listings(cinema: string) { return db.sql`SELECT ${'films'}.${'title'}, ${'screenings'}.${'starts_at'} FROM ${'screenings'} JOIN ${'films'} ON ${'films'}.${'id'} = ${'screenings'}.${'film_id'} JOIN ${'cinemas'} ON ${'cinemas'}.${'id'} = ${'screenings'}.${'cinema_id'} WHERE ${'cinemas'}.${'name'} = ${db.param(cinema)} ORDER BY ${'screenings'}.${'starts_at'}`; } const listed = await listings('The Lantern').run(pool); ``` Postgres runs: ```sql SELECT "films"."title", "screenings"."starts_at" FROM "screenings" JOIN "films" ON "films"."id" = "screenings"."film_id" JOIN "cinemas" ON "cinemas"."id" = "screenings"."cinema_id" WHERE "cinemas"."name" = $1 ORDER BY "screenings"."starts_at" ``` Values: `["The Lantern"]` TypeScript knows: ```ts const listed: Listing[] // that is const listed: { title: string; starts_at: Date; }[] ``` - An interpolated name is quoted, `"films"`, and checked against the first type argument: here `Tables`, the `SQL` type of each table the query names. Qualify a column as `${'films'}.${'title'}`: the type knows each table's columns, not `'films.title'`. - The second type argument is your word for what comes back: nothing checks it against the SQL. The rows come as `pg` parses them, as a `Selectable` describes them (a `Date` for a `timestamptz`), not as the JSON of the shortcuts' `JSONSelectable`. A `count(*)` is a `bigint`, which `pg` gives as a string: `'4'`. - Keywords stay in the text. SQL you write yourself is safe there; only what comes from outside needs the steps below. A name the type doesn't know fails to compile: a query that names `screenings` needs `s.screenings.SQL` in its first type argument. ```ts db.sql`SELECT * FROM ${'screenings'}`; // error TS2345 ``` A write is written the same way. This `INSERT … SELECT` copies one film's tags to another, and `RETURNING *` gives back the rows it added: You write: ```ts function copyTags(fromFilm: number, toFilm: number) { const columns: s.tags.Column[] = ['film_id', 'tag']; return db.sql`INSERT INTO ${'tags'} (${db.cols(columns)}) SELECT ${db.param(toFilm)}, ${'tag'} FROM ${'tags'} WHERE ${{ film_id: fromFilm }} ON CONFLICT DO NOTHING RETURNING *`; } const copied = await copyTags(1000, 1003).run(pool); ``` Postgres runs: ```sql INSERT INTO "tags" ("film_id", "tag") SELECT $1, "tag" FROM "tags" WHERE ("film_id" = $2) ON CONFLICT DO NOTHING RETURNING * ``` Values: `[1003, 1000]` TypeScript knows: ```ts const copied: s.tags.Selectable[] // that is const copied: { film_id: number; tag: string; }[] ``` - [`db.cols()`](https://pgcast.dev/sql-template#cols) writes the column list, `"film_id", "tag"`. `columns`, typed `s.tags.Column[]`, checks the names: `db.cols()` alone takes any, so `'tagz'` would compile, and Postgres refuse it (42703). - `ON CONFLICT DO NOTHING` skips a tag the film has already. Without it, Postgres refuses the whole `INSERT` with 23505, and adds none. ## Pass each value as a parameter A value from a caller (a form, a URL, a row you read elsewhere) goes in as [`db.param()`](https://pgcast.dev/sql-template#param), or in a [Whereable object](https://pgcast.dev/sql-template#whereable), which makes a parameter of each value: You write: ```ts function studioFilms(studioId: number, genre: string) { return db.sql`SELECT * FROM ${'films'} WHERE ${{ studio_id: studioId }} AND ${db.param(genre)} = ANY(${'genres'})`; } const dramas = await studioFilms(1000, 'drama').run(pool); ``` Postgres runs: ```sql SELECT * FROM "films" WHERE ("studio_id" = $1) AND $2 = ANY("genres") ``` Values: `[1000, "drama"]` TypeScript knows: ```ts const dramas: s.films.Selectable[] // that is const dramas: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` - `${{ studio_id: studioId }}` is `("studio_id" = $1)`: a filter of `films`, as the `where` of [`select()`](https://pgcast.dev/select#select) takes it, so [conditions](https://pgcast.dev/conditions) work in it too: [`gte()`](https://pgcast.dev/conditions#conditions.gte) in `{ release_year: db.conditions.gte(1990) }`. - `${db.param(genre)}` is `$2`, in `$2 = ANY("genres")`: a parameter fits wherever SQL takes a value. - An array is one parameter: `` ${'id'} = ANY(${db.param(ids)}) `` sends `ids` as a Postgres array by default; with `castArrayParamsToJson` on, it does not ([below](https://pgcast.dev/raw-sql#cast-a-parameter-postgres-can-t-type)). - A part of the query that only some calls need is a nested fragment, or `[]` for nothing ([Nested fragments](https://pgcast.dev/sql-template#nested-fragments)). A bare string between `${…}` is a name, not a value, so the type refuses it: ```ts function filmTitled(title: string) { return db.sql`SELECT * FROM ${'films'} WHERE ${'title'} = ${title}`; // error TS2345 } ``` If it got past the type (an `any`, a `@ts-ignore`), it would still be a name, and the caller would pick it: - A real column matches every row: `'title'` compiles to `"title" = "title"`, true wherever `title` is not null, and so do `'films.title'` and `'"title"'`. In a `DELETE`, that deletes them. - An unknown one fails: `'The Salt Orchard'` gives `"title" = "The Salt Orchard"`, a column that does not exist (42703). - It can be any column in scope, a system column such as `xmin` included, and, where a table goes, any relation the role can read, `'schema.table'` included: `` FROM ${'pg_catalog.pg_roles'} `` lists the database's roles. - Any `"` in it is doubled, so it can't close its quotes (in Zapatos 6.6.1 it could, [Identifiers](https://pgcast.dev/sql-template#identifiers)), but that does not make it a value. The keys of an interpolated object are names too. The type refuses an unknown key only in an object literal (TS2353): an object built at run time (a `Record`, `Object.fromEntries()`, a variable with extra keys) passes it with any key. > **Security** > > An interpolated object that is not a `param()`, a fragment or another of pgcast's values compiles as a Whereable, a class instance included. > > A built-in object throws while compiling, before any query: `` db.sql`DELETE FROM ${'ratings'} WHERE ${when}` ``, with `when` a `Date` typed `any`, no longer compiles to `DELETE FROM "ratings" WHERE TRUE`, as in Zapatos 6.6.1, which deleted every rating ([Whereable objects](https://pgcast.dev/sql-template#whereable)). Pass a date as `db.param(when)`. > > Any object with no keys is `TRUE`, a typed Whereable included: `const where: s.ratings.Whereable = {}`, given a `film_id` only `if (filmId)`, compiles with no error to the same `DELETE`. Check for an empty filter before a `DELETE` or an `UPDATE` ([Whereable objects](https://pgcast.dev/sql-template#whereable)). The check below catches `{}`, not a key whose value compiles to `true`, such as `film_id: db.conditions.isNotIn([])`: > > ```ts > function removeRatings(filmId?: number) { > const where: s.ratings.Whereable = {}; > if (filmId) where.film_id = filmId; > if (Object.keys(where).length === 0) throw new Error('No filter: that would delete every rating'); > return db.deletes('ratings', where); > } > ``` > ## Cast a parameter Postgres can't type Postgres gives a parameter a type from where it stands. Compared with a column, it takes the column's type; elsewhere, Postgres may infer another. Give it the type as `param()`'s second argument: You write: ```ts function screeningsWithin(from: Date, hours: number) { return db.sql`SELECT * FROM ${'screenings'} WHERE ${'starts_at'} >= ${db.param(from)} AND ${'starts_at'} < ${db.param(from, 'timestamptz')} + make_interval(hours => ${db.param(hours)})`; } const evening = await screeningsWithin(new Date('2030-03-01T18:00:00Z'), 4).run(pool); ``` Postgres runs: ```sql SELECT * FROM "screenings" WHERE "starts_at" >= $1 AND "starts_at" < CAST($2 AS "timestamptz") + make_interval(hours => $3) ``` Values: `["2030-03-01T18:00:00.000Z", "2030-03-01T18:00:00.000Z", 4]` TypeScript knows: ```ts const evening: s.screenings.Selectable[] // that is const evening: { cinema_id: number; film_id: number; id: number; starts_at: Date; subtitled: boolean | null; }[] ``` - `$1`, compared with `starts_at` in `"starts_at" >= $1`, is a `timestamptz` already. - `$2`, in `CAST($2 AS "timestamptz") + make_interval(hours => $3)`, is added to an `interval`, so without its cast Postgres reads it as one, and fails: `operator does not exist: timestamp with time zone < interval` (42883). - `db.param(from, 'timestamptz')` writes `CAST($2 AS "timestamptz")`; `` ${db.param(from)}::timestamptz `` in the text does the same. - The cast is the type's catalog name, in lower case (`int4`, `timestamptz`, `crew_job`), each part quoted. `'integer'` gives `CAST($1 AS "integer")`, a type that does not exist (42704), as do `'int'` and `'TIMESTAMPTZ'`. - Name a schema as `'public.crew_job'`, which gives `"public"."crew_job"`, as `'public"."crew_job'` does. An array type keeps its bounds outside the quotes: `'int4[]'` gives `"int4"[]`. - With [`castArrayParamsToJson`](https://pgcast.dev/runtime-config#Config.castArrayParamsToJson) on, an array parameter is sent as JSON, `CAST($1 AS "json")`, unless its cast ends with `[]` or `[n]`: `db.param(ids, 'int4[]')` gives `CAST($1 AS "int4"[])` with the array ([#36](https://github.com/AlexandreDemartin/pgcast/issues/36)). Any other string cast keeps the JSON, cast to that type, `'_int4'` included. - With [`castObjectParamsToJson`](https://pgcast.dev/runtime-config#Config.castObjectParamsToJson) on, a plain object is sent as JSON whatever its cast, cast to the type written. Every form of the cast: [`param()`](https://pgcast.dev/sql-template#param). ## Choose SQL at run time from your own constants Some choices are syntax, not values: a sort direction, the column to sort on. A parameter can't stand for them, so map the caller's choice to one you wrote: You write: ```ts function sortedFilms(by: string, direction: string) { const sortable = ['title', 'release_year', 'runtime_minutes'] as const; const column = sortable.find(c => c === by) ?? 'title'; return db.sql`SELECT * FROM ${'films'} ORDER BY ${column} ${db.raw(direction === 'desc' ? 'DESC' : 'ASC')}`; } const longestFirst = await sortedFilms('runtime_minutes', 'desc').run(pool); ``` Postgres runs: ```sql SELECT * FROM "films" ORDER BY "runtime_minutes" DESC ``` Values: `[]` TypeScript knows: ```ts const longestFirst: s.films.Selectable[] // that is const longestFirst: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` - [`db.raw()`](https://pgcast.dev/sql-template#raw) writes its string as it is. Here it only ever gets `'DESC'` or `'ASC'`, whatever the caller sent. - The column is a name: quoted, it can't close its quotes, but the caller's string could name any column in scope, a system column such as `xmin` included, or none (42703, or 42P01 for a table outside the query). `find()` keeps one of yours, typed as the template wants. > **Security** > > Whatever reaches `raw()` runs as SQL, and its type takes any string. Give it a caller's text, and the caller writes your query: > > ```ts > const title = "x' OR TRUE --"; // from the caller > db.sql`SELECT ${'title'} FROM ${'films'} WHERE ${'title'} = ${db.raw(`'${title}'`)}`; > ``` > > ```sql > SELECT "title" FROM "films" WHERE "title" = 'x' OR TRUE --' > ``` > > That returns every film. A value is a `param()`, always: `${db.param(title)}` gives `"title" = $1`, and no film. ## Compare a column with SQL, through `self` A fragment as a Whereable value compares its column with SQL of your own, [`db.self`](https://pgcast.dev/sql-template#self) standing for the column: You write: ```ts function filmsTitled(title: string) { return db.sql`SELECT * FROM ${'films'} WHERE ${{ title: db.sql`lower(${db.self}) = lower(${db.param(title)})` }}`; } const found = await filmsTitled('the salt orchard').run(pool); ``` Postgres runs: ```sql SELECT * FROM "films" WHERE ((lower("title") = lower($1))) ``` Values: `["the salt orchard"]` TypeScript knows: ```ts const found: s.films.Selectable[] // that is const found: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` - `lower(${db.self})` is `lower("title")`, and the caller's text is still a parameter, `lower($1)`. - An equality, unlike [`ilike()`](https://pgcast.dev/conditions#conditions.ilike): a `_` or a `%` in the caller's text matches only itself. - The same object works as the `where` of [`select()`](https://pgcast.dev/select#select) and the other shortcuts. - Where it has no column to stand for, `self` throws "The 'self' column alias has no meaning here", so [`run()`](https://pgcast.dev/sql-template#run) rejects before any query. ## Reach the outer row, through `parent()` A hand-written subquery can go in a shortcut's `lateral`, where [`db.parent()`](https://pgcast.dev/sql-template#parent) names a column of the outer query's table: You write: ```ts const studios = await db.select('studios', db.all, { columns: ['name'], lateral: { titles: db.sql`SELECT string_agg(${'title'}, ', ' ORDER BY ${'release_year'}) AS result FROM ${'films'} WHERE ${'studio_id'} = ${db.parent('id')}`, }, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "name") || jsonb_build_object($2::text, "lateral_titles".result) AS result FROM "studios" LEFT JOIN LATERAL ( SELECT string_agg("title", ', ' ORDER BY "release_year") AS result FROM "films" WHERE "studio_id" = "studios"."id" ) AS "lateral_titles" ON true ) AS "sq_studios" ``` Values: `["name", "titles"]` TypeScript knows: ```ts const studios: (db.JSONOnlyColsForTable<"studios", "name"[]> & db.LateralResult<{ titles: db.SQLFragment; }>)[] // that is const studios: { name: string; titles: string | null; }[] ``` - `${db.parent('id')}` is `"studios"."id"`, the studio of each row. - The subquery gives one column, named `result`: that is what `lateral` reads. Its type argument, `string | null`, is the key's type: `string_agg()` of no rows is `NULL`. - With `strict` off, the key is not in the type ([`lateral`](https://pgcast.dev/select#lateral)). - Outside a `lateral`, `parent()` throws "The 'parent' table alias has no meaning here", so [`run()`](https://pgcast.dev/sql-template#run) rejects before any query. Joins through `lateral`, step by step: [Joins](https://pgcast.dev/lateral). ## Check the SQL before you run it [`.compile()`](https://pgcast.dev/sql-template#compile) gives the `text` and `values` that `run()` would send, and sends nothing. Each value from outside belongs in `values`, with a `$n` in its place in `text`. A test that compiles each query with hostile input, expects it in `values` and freezes `text` keeps it that way, here for `listings()` above: ```ts for (const cinema of [`x' OR TRUE --`, `x" --`]) { test(`listings() keeps ${cinema} a value`, () => { const { text, values } = listings(cinema).compile(); assert.deepEqual(values, [cinema]); assert.equal(text.replace(/\s+/g, ' '), 'SELECT "films"."title", "screenings"."starts_at" FROM "screenings" ' + 'JOIN "films" ON "films"."id" = "screenings"."film_id" JOIN "cinemas" ON "cinemas"."id" = "screenings"."cinema_id" ' + 'WHERE "cinemas"."name" = $1 ORDER BY "screenings"."starts_at"'); }); } ``` - Here with `test` from `node:test` and `assert` from `node:assert/strict`; any test runner does. - `text` keeps the line breaks of the template: the test collapses its whitespace, so that only a change to the SQL fails it. - A search of `text` is not enough: a name compiled from `x" --` is there as `"x"" --"`, its `"` doubled, and a lone `'` matches the query's own literals. ## See also - [SQL template](https://pgcast.dev/sql-template): every interpolation, and what it compiles to. - [Security](https://pgcast.dev/security): why values are parameters, and what pgcast quotes. - [Joins](https://pgcast.dev/lateral): subqueries with `parent()`, nested. --- # Errors Source: https://pgcast.dev/handle-errors Tell what went wrong with a query, and answer it: a seat already booked, a screening that isn't there, an option your own code got wrong. It assumes you know the shortcuts of [First query](https://pgcast.dev/first-query). ## Catch a refusal from Postgres When Postgres refuses a query, `.run()` rejects with node-postgres's `DatabaseError`, whose `code` is the SQLSTATE. Give it to [`isDatabaseError()`](https://pgcast.dev/transaction#isDatabaseError) with the name of the refusal you expect, and let anything else through. Book seat C4 of screening 1002, which someone may have booked already: You write: ```ts const booking = await db.insert('seat_bookings', { screening_id: 1002, seat: 'C4', customer: 'Tamsin Rowe' }) .run(pool) .catch((err): undefined => { if (db.isDatabaseError(err, 'IntegrityConstraintViolation_UniqueViolation')) return undefined; // taken throw err; }); ``` Postgres runs: ```sql INSERT INTO "seat_bookings" ("customer", "screening_id", "seat") VALUES ($1, $2, $3) RETURNING to_jsonb("seat_bookings".*) AS result ``` Values: `["Tamsin Rowe", 1002, "C4"]` TypeScript knows: ```ts const booking: seat_bookings.JSONSelectable | undefined // that is const booking: { booked_at: db.TimestampTzString; customer: string; screening_id: number; seat: PgSeat_label; } | undefined ``` If C4 is taken, Postgres refuses the `INSERT` with 23505, a unique violation, and `booking` is `undefined`; if not, it is the new row. Any other error, from Postgres or not, is thrown again. With `try` and `await`, the `catch` variable is `unknown` under `strict`, which `isDatabaseError()` refuses ([why](https://pgcast.dev/transaction#isDatabaseError)). Narrow it with `instanceof pg.DatabaseError` first (pg 8.6 or later, not `pg.native`, whose errors are plain `Error`s): then TypeScript knows its fields too. ```ts try { await db.insert('seat_bookings', { screening_id: 1002, seat: 'C4', customer: 'Tamsin Rowe' }).run(pool); } catch (err) { if (!(err instanceof pg.DatabaseError && db.isDatabaseError(err, 'IntegrityConstraintViolation_UniqueViolation'))) throw err; console.log(err.detail); // Key (screening_id, seat)=(1002, C4) already exists. } ``` ## Tell the refusals apart Three refusals a booking can meet on the docs schema, each with its `code` and the constraint that `err.constraint` names, which tells two constraints of the same kind apart: | The booking | `code` | `err.constraint` | |---|---|---| | a seat already booked | `23505` | `seat_bookings_pkey` | | the seat `'Z100'`, which the domain `seat_label` refuses | `23514` | `seat_label_check` | | screening 1042, which doesn't exist | `23503` | `seat_bookings_screening_id_fkey` | Each code has its name for `isDatabaseError()`, so one function can give each its answer: ```ts const bookingProblem = (err: pg.DatabaseError) => db.isDatabaseError(err, 'IntegrityConstraintViolation_UniqueViolation') ? 'That seat is taken.' : db.isDatabaseError(err, 'IntegrityConstraintViolation_CheckViolation') ? 'There is no such seat.' : db.isDatabaseError(err, 'IntegrityConstraintViolation_ForeignKeyViolation') ? 'There is no such screening.' : undefined; ``` - A class name, with no `_`, matches every code of its class: `'IntegrityConstraintViolation'` matches all three. Several names match if any does: `db.isDatabaseError(err, 'IntegrityConstraintViolation', 'DataException')`. - The names are Postgres's [error codes](https://www.postgresql.org/docs/current/errcodes-appendix.html) in PascalCase, and the type refuses any other: your editor completes them. - `isDatabaseError(null, …)` throws a `TypeError`, which narrowing first avoids; the other cases, such as the names of class 53: [`isDatabaseError()`](https://pgcast.dev/transaction#isDatabaseError). ## Catch outside a transaction In a transaction, let the error out of the callback and catch it around [`transaction()`](https://pgcast.dev/transaction#transaction): pgcast sends `ROLLBACK`, then rejects with the same error. A customer books seats H1 and H2 of screening 1001, both or neither, and H2 is taken: ```ts const bookPair = (customer: string) => db.serializable(pool, async txnClient => { await db.insert('seat_bookings', { screening_id: 1001, seat: 'H1', customer }).run(txnClient); await db.insert('seat_bookings', { screening_id: 1001, seat: 'H2', customer }).run(txnClient); return true; }); const booked = await bookPair('Wren Abernathy').catch(err => { if (db.isDatabaseError(err, 'IntegrityConstraintViolation_UniqueViolation')) return false; // H1 is not booked either throw err; }); ``` `booked` is `false`, and H1 isn't booked. The transaction sends: ```sql START TRANSACTION ISOLATION LEVEL SERIALIZABLE -- the insert() of H1 -- the insert() of H2, which Postgres refuses with 23505 ROLLBACK ``` The SQL of each `INSERT`: [`insert()`](https://pgcast.dev/insert#insert). A serialisation failure (40001) or a deadlock (40P01) has already been retried by then, up to [`transactionAttemptsMax`](https://pgcast.dev/runtime-config#Config.transactionAttemptsMax) attempts; if the promise still rejects with one, `'TransactionRollback'` recognises both, and the request can be tried again later ([Retries](https://pgcast.dev/retries)). > **Lost writes** > > Don't catch a database error inside the callback and carry on. Postgres has aborted the transaction: each query after it is refused with 25P02 (`InvalidTransactionState_InFailedSqlTransaction`), and if the callback returns, pgcast sends `COMMIT`, which Postgres turns into a rollback without an error. > > The promise resolves, nothing the callback wrote is saved, and pgcast warns once, `PGCAST_DEP003`. With [`transactionRequiresCommit: true`](https://pgcast.dev/runtime-config#Config.transactionRequiresCommit), the default from 7.0, it rejects instead ([`transaction()`, Errors](https://pgcast.dev/transaction#errors)). > > For a seat that may be taken, ask Postgres not to refuse: [`upsert()` with `doNothing`](https://pgcast.dev/insert#doNothing) writes `ON CONFLICT ON CONSTRAINT "seat_bookings_pkey" DO NOTHING` and gives `undefined` for a taken seat, with no error, and the transaction goes on. ## Tell a missing row from the others [`selectExactlyOne()`](https://pgcast.dev/select#selectExactlyOne) is for a row that must be there. With none, `.run()` rejects with a [`NotExactlyOneError`](https://pgcast.dev/select#NotExactlyOneError): screening 1042 doesn't exist. You write: ```ts const screening = await db.selectExactlyOne('screenings', { id: 1042 }).run(pool); ``` Postgres runs: ```sql SELECT to_jsonb("screenings".*) AS result FROM "screenings" WHERE ("id" = $1) LIMIT $2 ``` Values: `[1042, 1]` TypeScript knows: ```ts const screening: screenings.JSONSelectable // that is const screening: { cinema_id: number; film_id: number; id: number; starts_at: db.TimestampTzString; subtitled: boolean | null; } ``` - Recognise it with `err instanceof db.NotExactlyOneError`. It is not a `pg.DatabaseError` and has no `code`, so `isDatabaseError()` gives `false`. - Its message is "One result expected but none returned (hint: check `.query.compile()` on this Error)", and `err.query.compile()` gives the SQL and values of the panel. - It asks for `LIMIT $2`, its value `1`, so two matching rows are not an error: it gives the first, as in Zapatos 6.6.1 ([#32](https://github.com/AlexandreDemartin/pgcast/issues/32)). - In a `lateral`, a missing row is `null`, not an error ([`selectExactlyOne()`](https://pgcast.dev/select#selectExactlyOne)). - When a missing row is ordinary, as in a search, [`selectOne()`](https://pgcast.dev/select#selectOne) gives `undefined` instead, and TypeScript makes you check it. Log its SQL and values where you catch it, then let the error go on: ```ts const screening = await db.selectExactlyOne('screenings', { id: 1042 }).run(pool).catch((err: unknown) => { if (err instanceof db.NotExactlyOneError) console.error(err.query.compile()); throw err; }); ``` ## Fix the errors pgcast throws pgcast throws a plain `Error`, with no `code`, before the query is sent, when a value is not one it accepts: an option outside its type (`'desc'`, `'serializable'`), or a value with no meaning in SQL (`db.Default` in a filter). These are bugs in the calling code, to fix rather than to catch: the messages say how. Where they surface depends on who throws: | Who throws | For example | What you get | |---|---|---| | a shortcut, given an option outside its type | `direction: 'desc'` in an `order` of [`select()`](https://pgcast.dev/select#errors) | the call throws, before `.run()` | | `.compile()`, inside `.run()`, given a value with no meaning in SQL | `db.Default` in a filter ([Whereable objects](https://pgcast.dev/sql-template#whereable)) | `.run()` rejects | | [`transaction()`](https://pgcast.dev/transaction#errors), given a level that is not an `IsolationLevel` | `'serializable'` | its promise rejects | Every one of them, with its page: [Errors pgcast throws before any query](https://pgcast.dev/transaction#errors-pgcast-throws-before-any-query). Zapatos 6.6.1 sent some of these values to Postgres instead: `db.Default` in a filter as the string `'Symbol(DEFAULT)'`. A `.catch()` chained on `.run()` misses what the call throws. The type refuses `'desc'`, but a cast lets it through, and `select()` throws "Direction must be ASC/DESC, not 'desc'": ```ts const direction = request.query.sort as 'ASC' | 'DESC'; db.select('films', db.all, { order: { by: 'title', direction } }) .run(pool) .catch(err => console.error(err)); // never called: select() has thrown already ``` Check the value instead of casting it, so that `direction` is one you wrote: ```ts const direction = request.query.sort === 'desc' ? 'DESC' : 'ASC'; const films = await db.select('films', db.all, { order: { by: 'title', direction } }).run(pool); ``` If a call still throws, inside an `async` function the throw becomes the promise's rejection, so the caller's `.catch()` gets it, and a `try` around the call catches it too. In a `transaction()` callback, `ROLLBACK` follows. ## Answer each kind in one place The error handler of an app can now tell the three kinds apart, and give each its answer: ```ts function statusOf(err: unknown) { if (err instanceof db.NotExactlyOneError) return 404; // no such row if (err instanceof pg.DatabaseError) { if (db.isDatabaseError(err, 'IntegrityConstraintViolation_UniqueViolation')) return 409; // already there if (db.isDatabaseError(err, 'IntegrityConstraintViolation', 'DataException')) return 422; // a value Postgres refuses if (db.isDatabaseError(err, 'TransactionRollback')) return 503; // try again later } return 500; // anything else: a bug, or the database out of reach } ``` ## See also - [`isDatabaseError()`](https://pgcast.dev/transaction#isDatabaseError): the names it takes, and the codes of the docs schema. - [Errors pgcast throws before any query](https://pgcast.dev/transaction#errors-pgcast-throws-before-any-query): each, with what throws it and its messages. - [`transaction()`](https://pgcast.dev/transaction#transaction): its `ROLLBACK`, its retries and its errors. - [Retries](https://pgcast.dev/retries): what the callback must do when it may run again. --- # Testing Source: https://pgcast.dev/testing Three ways to test a function that runs pgcast queries, the most faithful first: against a real Postgres, each test rolled back; on the query it compiles, with no database; on a fake result typed from the schema. It assumes the basics of the [tutorial](https://pgcast.dev/tutorial). ## The function to test `programme()` lists a studio's films, oldest first, as lines for a poster. The examples below test it, or a function like it: ```ts async function programme(queryable: db.Queryable, studio_id: number) { const films = await db.select('films', { studio_id }, { columns: ['title', 'release_year'], order: { by: 'release_year', direction: 'ASC' }, }).run(queryable); return films.map(film => `${film.title} (${film.release_year})`); } ``` It takes a [`Queryable`](https://pgcast.dev/sql-template#Queryable), so a test can hand it any pool or client, or a fake. ## Test against a real database The approach I recommend: the only one of the three that checks the query itself and what Postgres sends back. pgcast's own suite works this way ([`test/helpers/db.ts`](https://github.com/AlexandreDemartin/pgcast/blob/master/test/helpers/db.ts)). ### Copy a database for each test file Load your schema once into a template database, with your migrations, then copy it for each test file and drop the copy after its tests: ```sql CREATE DATABASE films_test_4f1c TEMPLATE films_template; -- this file's tests DROP DATABASE films_test_4f1c; ``` - Postgres copies the template as it is: the migrations don't run again. The role needs `CREATEDB`. - No session may be connected to the template while it is copied: Postgres refuses with 55006 ("source database … is being accessed by other users"). - A random suffix per file lets test files run at once, each on its own database. ### Roll back each test Each test runs on one client, inside `BEGIN`, rolled back afterwards: what it writes is gone before the next test. It seeds its own rows with [`insert()`](https://pgcast.dev/insert#insert): ```ts async function rolledBack(body: (client: pg.PoolClient) => Promise) { const client = await pool.connect(); try { await client.query('BEGIN'); await body(client); } finally { try { await client.query('ROLLBACK'); client.release(); } catch (err) { client.release(err as Error); // a broken connection: destroyed, not reused throw err; } } } test('programme() lists a studio’s films, oldest first', () => rolledBack(async client => { const studio = await db.insert('studios', { name: 'Marrowbank Pictures' }).run(client); await db.insert('films', [ { studio_id: studio.id, title: 'Tidewater', release_year: 2021, runtime_minutes: 95 }, { studio_id: studio.id, title: 'The Quiet Ferry', release_year: 1988, runtime_minutes: 110 }, ]).run(client); assert.deepEqual(await programme(client, studio.id), ['The Quiet Ferry (1988)', 'Tidewater (2021)']); })); ``` - Here with `test` from `node:test` and `assert` from `node:assert/strict`; any test runner does, and `beforeEach` / `afterEach` hooks can hold the same lines. - Read a generated value from what `insert()` returns, as `studio.id` here: never write it into the test (below). - A client whose `ROLLBACK` fails is broken: `release(err)` destroys it rather than handing it to the next test. ### Know what a rollback leaves | What a test did | After its `ROLLBACK` | |---|---| | Rows inserted, updated or deleted; `SET`; a temporary table | undone | | A `SERIAL` or identity value taken | still taken: the next test's `insert()` gets the next id, 2 after 1, though row 1 is gone | | `PREPARE`, or a query run with [`prepared()`](https://pgcast.dev/sql-template#prepared) | kept on the connection, for the next test that gets it from the pool | | A session advisory lock, `pg_advisory_lock()` | still held by that connection | So an id is read, never hard-coded; and a prepared name stands for one text across every test that shares the pool. ### Give code that commits the pool Code that opens its own [`transaction()`](https://pgcast.dev/transaction#transaction) can't join the test's. Given the test's client, inside its `BEGIN`, it sends `START TRANSACTION ISOLATION LEVEL SERIALIZABLE`, which Postgres answers with a warning (25001, "there is already a transaction in progress"), then ends the test's transaction one of two ways ([#233](https://github.com/AlexandreDemartin/pgcast/issues/233)): - If the test has run no query yet, or at the session's own level (`readCommitted()`, on a default server), its `COMMIT` commits the test's transaction, rows and all. - At another level, after any query, the test's seeds included, Postgres refuses with 25001 ("SET TRANSACTION ISOLATION LEVEL must be called before any query"): its `ROLLBACK` undoes the seeds, and the call rejects. Either way, the test's `ROLLBACK` has nothing left to undo (25P01). So give such code the pool, and delete what it commits: ```ts const bookSeat = (pool: pg.Pool, screening_id: number, seat: string, customer: string) => db.serializable(pool, txnClient => db.insert('seat_bookings', { screening_id, seat, customer }).run(txnClient)); test('bookSeat() books a free seat', async () => { try { const booking = await bookSeat(pool, 1002, 'C7', 'Ines Calloway'); assert.equal(booking.seat, 'C7'); assert.equal(await db.count('seat_bookings', { screening_id: 1002 }).run(pool), 1); } finally { await db.deletes('seat_bookings', { screening_id: 1002, customer: 'Ines Calloway' }).run(pool); } }); ``` - The `deletes()` in `finally` runs even when an assertion fails. - Until then, the booking is there for every other connection: a test of the same screening, running at the same moment, can count it. ## Test the query a function builds For a function whose job is to turn its input into a query, no database is needed: [`compile()`](https://pgcast.dev/sql-template#compile) gives the `text` and `values` that `run()` would send. `filmSearch()` builds a search from a query string: ```ts function filmSearch(params: URLSearchParams) { const words = params.get('title'), before = params.get('before'); return db.select('films', { ...(words === null ? {} : { title: db.conditions.ilike(`%${words}%`) }), ...(before === null ? {} : { release_year: db.conditions.lt(Number(before)) }), }, { columns: ['title', 'release_year'], order: { by: 'release_year', direction: 'ASC' } }); } ``` For `?title=lantern&before=2000`: You write: ```ts const films = await filmSearch(new URLSearchParams('title=lantern&before=2000')).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "title", $2::text, "release_year") AS result FROM "films" WHERE (("release_year" < $3) AND ("title" ILIKE $4)) ORDER BY "release_year" ASC ) AS "sq_films" ``` Values: `["title", "release_year", 2000, "%lantern%"]` TypeScript knows: ```ts const films: db.JSONOnlyColsForTable<"films", ("release_year" | "title")[]>[] // that is const films: { release_year: number; title: string; }[] ``` The test compares its compiled query with the one of the call it should make, compiled by the same pgcast: ```ts test('filmSearch() filters on each parameter it is given', () => { assert.deepEqual(filmSearch(new URLSearchParams('title=lantern&before=2000')).compile(), db.select('films', { title: db.conditions.ilike('%lantern%'), release_year: db.conditions.lt(2000), }, { columns: ['title', 'release_year'], order: { by: 'release_year', direction: 'ASC' } }).compile()); }); test('filmSearch() filters on nothing when given nothing', () => { assert.deepEqual(filmSearch(new URLSearchParams()).compile(), db.select('films', {}, { columns: ['title', 'release_year'], order: { by: 'release_year', direction: 'ASC' } }).compile()); }); ``` - It fails when `filmSearch()` makes another call, and not when pgcast writes the same call's SQL another way. - `values` holds more than your input: `columns` sends each name as a parameter (`$1::text`), and a filter's keys go in alphabetical order, `"release_year" < $3` before `"title" ILIKE $4`. A `values` array written by hand ties the test to both. - For SQL you write yourself in a `sql` template, freezing its `text` is fine: the text is yours ([Check the SQL before you run it](https://pgcast.dev/raw-sql#check-the-sql-before-you-run-it)). ### The SQL text is not a stable API pgcast keeps what a query does: the rows it reads or writes, and the type of what `run()` returns. It does not keep the text it compiles: a minor version may change it, a pair of parentheses, the order of clauses, a cast, as long as the result does not. The fix of `not()` is one such change: each condition under `not()` gains a pair of parentheses, and most match the same rows. So a test that freezes the text of a shortcut's query, or a fake that picks its answer by matching that text, can break on an upgrade with nothing wrong in your code. ## Fake a result typed from the schema For the logic around a query, with no database at all: a fake `Queryable` returns the rows you give it, and the rows are typed from the schema, so a fake row stops compiling when its table changes. A few lines you write; pgcast ships no fake. ```ts type Films = db.FullSelectReturnTypeForTable< 'films', ['title', 'release_year'], undefined, undefined, db.SelectResultMode.Many>; const answering = (rows: object[]) => ({ query: async () => ({ rows }) }) as unknown as db.Queryable; test('programme() gives each film its year', async () => { const films: Films = [ { title: 'The Quiet Ferry', release_year: 1988 }, { title: 'Tidewater', release_year: 2021 }, ]; const queryable = answering([{ result: films }]); // select() reads one row, its column result assert.deepEqual(await programme(queryable, 1000), ['The Quiet Ferry (1988)', 'Tidewater (2021)']); }); ``` - `run()` sends the compiled query to `query()` and reads its result from what comes back; the fake ignores the query. - It is neither a `pg.Pool` nor a client, hence `as unknown as`: a plain `as db.Queryable` does not compile (TS2352). - A [`transaction()`](https://pgcast.dev/transaction#transaction) given the fake calls its `connect()`, which it doesn't have: a fake for code that opens one also needs `connect()`, returning itself with a `release()`, and gets `START TRANSACTION …` and `COMMIT` as queries too. Say a migration renames `release_year`: once the generator has run, every fake row that still has it fails to compile. Here, a fake row with a column `films` doesn't have: ```ts const films: Films = [{ title: 'Tidewater', year: 2021 }]; // error TS2353 ``` With `strict` off, `columns` leaves the type at the table's full `JSONSelectable` ([`select()` options](https://pgcast.dev/select#options)): a fake row then needs every column (TS2739), and `null` fits any of them. ### Type each result | Query | One result's type | The fake's `rows` | |---|---|---| | [`select()`](https://pgcast.dev/select#select) | [`FullSelectReturnTypeForTable`](https://pgcast.dev/select#FullSelectReturnTypeForTable)``, `M` the mode | `[{ result }]`, the whole array in one row; `[{ result: [] }]` for none, as Postgres sends it (`[]` gives `undefined`) | | [`selectOne()`](https://pgcast.dev/select#selectOne), [`selectExactlyOne()`](https://pgcast.dev/select#selectExactlyOne) | the same | `[{ result }]`; `[]` for none: `undefined` from `selectOne()`, a [`NotExactlyOneError`](https://pgcast.dev/select#NotExactlyOneError) from `selectExactlyOne()` | | [`count()`](https://pgcast.dev/select#count) and the other aggregates | `number` | `[{ result }]` | | [`insert()`](https://pgcast.dev/insert#insert), [`update()`](https://pgcast.dev/insert#update), [`deletes()`](https://pgcast.dev/insert#deletes) | [`ReturningTypeForTable`](https://pgcast.dev/insert#ReturningTypeForTable)``, for each row | one `{ result }` per row | | [`upsert()`](https://pgcast.dev/insert#upsert) | [`UpsertReturnableForTable`](https://pgcast.dev/insert#UpsertReturnableForTable)``, for each row | one `{ result }` per row | | A [`sql`](https://pgcast.dev/sql-template#sql) template | [`RunResultForSQLFragment`](https://pgcast.dev/sql-template#RunResultForSQLFragment)`` | the rows themselves | - `RunResultForSQLFragment` reads `undefined` as `null`, as a nested query comes back in JSON: for a `selectOne()` you run, `FullSelectReturnTypeForTable` with `SelectResultMode.One` keeps its `undefined`. - The shortcuts read their result from a column named `result`, part of the SQL they compile: a fake that answers in that shape depends on it, as on the text. ### Know what a fake does not check - **The query.** A filter on the wrong column, a value Postgres refuses (`stars: 6` in `ratings`, 23514 from its `CHECK`), SQL it can't parse: the fake answers anyway. - **What Postgres sends back.** Which rows, in which order, which of them `null`: the fake returns what it is given. - A `timestamptz` comes back from a shortcut as a string in the session's time zone: `2030-03-01T19:30:00+00:00` in UTC, `2030-03-01T20:30:00+01:00` in Paris. Its type, `TimestampTzString`, takes `2030-03-01T19:30:00Z` too. From a `sql` template, `pg` gives a `Date`. - A `bigint` comes back from a shortcut as a JSON number, rounded past 253: 9007199254740993 arrives as 9007199254740992 ([`customJSONParsingForLargeNumbers`](https://pgcast.dev/generator#generate.customJSONParsingForLargeNumbers)). From a `sql` template, `pg` gives a string. A real database checks both: keep a test against it for each query, and fakes for the logic around them. ## See also - [Raw SQL](https://pgcast.dev/raw-sql#check-the-sql-before-you-run-it): a test that keeps hostile input in `values`. - [Retries](https://pgcast.dev/retries#test-a-retry): a test that makes a transaction run again. - [`compile()`](https://pgcast.dev/sql-template#compile), [`Queryable`](https://pgcast.dev/sql-template#Queryable), [`transaction()`](https://pgcast.dev/transaction#transaction): what they take and give. - [Why pgcast](https://pgcast.dev/why): why the database is the source of the types. --- # AI assistants Source: https://pgcast.dev/ai-assistant Give an assistant that writes your queries three things: these docs, your generated schema and a short rules file, so that it writes them the pgcast way. Then check its work with the type checker and `.compile()`. It assumes the basics of the [tutorial](https://pgcast.dev/tutorial). ## Give it the docs The site publishes itself for assistants, in two files at its root: ```txt https://pgcast.dev/llms.txt https://pgcast.dev/llms-full.txt ``` - [`llms.txt`](https://pgcast.dev/llms.txt): an index, each page with its URL and its summary. For an assistant that fetches the pages it needs. - [`llms-full.txt`](https://pgcast.dev/llms-full.txt): every page in one Markdown file, each cast panel with the SQL, values and type it shows. For one that reads a whole file at once. Paste a URL into the chat, or let the rules file below name it. ## Give it your schema `npx pgcast` writes `zapatos/schema.d.ts` into your [`outDir`](https://pgcast.dev/generator#generate.outDir): `src/zapatos/schema.d.ts` in the tutorial. It holds what an assistant would otherwise guess: each table's columns, the type of each in a row, a filter, an insert and an update ([The files it writes](https://pgcast.dev/generator#the-files-it-writes)), and a comment on each column with its Postgres type, whether it is `NOT NULL`, and its default. - Tell the assistant to read it before writing a query, and never to edit it: the next run writes it again. The rules file below does both; change its path if your `outDir` differs. - A `COMMENT ON COLUMN` on a table's column, in a migration, reaches that comment too ([`schemaJSDoc`](https://pgcast.dev/generator#generate.schemaJSDoc)): the place to tell the assistant what a column means. A view's, a foreign table's or a partitioned table's does not yet ([#47](https://github.com/AlexandreDemartin/pgcast/issues/47)). - After each migration, run `npx pgcast` again, so that the assistant reads the schema as it is. ## Add the rules file Copy it into your project, and change `src/zapatos/schema.d.ts` if your `outDir` differs. Each rule ends with the page that backs it, for the assistant to read: ```md # Postgres with pgcast This project queries Postgres with pgcast, installed under the `zapatos` alias. The index of its docs, each page with its URL and summary: https://pgcast.dev/llms.txt For an assistant that reads one whole file, every page in one: https://pgcast.dev/llms-full.txt - The database is the source of the types. Before writing a query on a table, read its columns and their types in `src/zapatos/schema.d.ts`; never guess a column. Never edit that file: it is generated. https://pgcast.dev/generator#the-files-it-writes - After a migration, run `npx pgcast` to generate the types again, then fix what no longer compiles. https://pgcast.dev/first-query#when-the-schema-changes - Import the run-time code as `import * as db from 'zapatos/db'` and the types as `import type * as s from 'zapatos/schema'`, never from `pgcast`. https://pgcast.dev/install - Prefer the shortcuts to hand-written SQL: `db.select()`, `db.selectOne()`, `db.count()` and the other aggregates, `db.insert()`, `db.upsert()`, `db.update()`, `db.deletes()`; filters from `db.conditions`; related rows through `lateral`. https://pgcast.dev/select https://pgcast.dev/insert https://pgcast.dev/conditions https://pgcast.dev/lateral - Write what no shortcut covers in a `db.sql` template, typed with the table's names and its rows: `` db.sql`…` ``. https://pgcast.dev/raw-sql - Never concatenate a value into SQL, and never pass one to `db.raw()`. A value goes in a shortcut's object, in a Whereable (`${{ studio_id }}`) or in `db.param()`, and Postgres gets it as a parameter, `$1`. https://pgcast.dev/raw-sql#pass-each-value-as-a-parameter - An interpolated string is a table or column name. Never interpolate a string from outside the code (a request, a user, a file): map it to a name written in the code, and never cast it to make it compile. https://pgcast.dev/raw-sql#choose-sql-at-run-time-from-your-own-constants - A shortcut's object and a Whereable take their keys as column names: build them from fields you name, never pass a request's body or query whole. An empty Whereable is `TRUE`: check that a filter has a key before `db.update()` or `db.deletes()`. https://pgcast.dev/raw-sql#pass-each-value-as-a-parameter - Run each query on the pool or client you are given; inside a transaction, on its `txnClient`. Its callback may run again (40001, 40P01): keep side effects out of it. https://pgcast.dev/transaction#transaction https://pgcast.dev/retries#keep-side-effects-out-of-the-callback - Read the SQL of a new query: `.compile()` in place of `.run()` gives `{ text, values }` and sends nothing. https://pgcast.dev/sql-template#compile - Before calling a change done, run the type check, `npx tsc --noEmit`, and fix each error in the code, never with `any`, a type assertion (`as` a type; `as const` is fine), `@ts-ignore` or `@ts-expect-error`. https://pgcast.dev/ai-assistant#run-the-type-check ``` Put it where your assistant reads its instructions: | As | File | Above the rules | |---|---|---| | `AGENTS.md`, which most assistants read | `AGENTS.md` at the root of the repository, or at the end of yours | nothing | | a Cursor rule | `.cursor/rules/pgcast.mdc` | front matter with a `description` and `alwaysApply: false` | | a Claude skill | `.claude/skills/pgcast/SKILL.md` | front matter with a `description` | - Claude Code reads `AGENTS.md` only when the project has no `CLAUDE.md`, `.claude/CLAUDE.md` or `CLAUDE.local.md`. With one, it reads `AGENTS.md` only if that file imports it, with a line `@AGENTS.md` ([Claude Code's docs](https://code.claude.com/docs/en/memory#agents-md)). - For the last two, the `description` says when the rules apply: the assistant reads it and pulls the rules in when a task needs them. A Cursor rule with `globs` would instead be attached whenever a matching file is in context, whatever its `description` ([Cursor's docs](https://cursor.com/docs/context/rules)). ```md [.cursor/rules/pgcast.mdc] --- description: Querying Postgres with pgcast (zapatos/db). Use when writing or changing a database query. alwaysApply: false --- ``` ```md [.claude/skills/pgcast/SKILL.md] --- description: Querying Postgres with pgcast (zapatos/db). Use when writing or changing a database query. --- ``` ## Check its work ### Run the type check ```sh npx tsc --noEmit ``` Or your project's own script, through `npm run`. A column the assistant made up fails, and so does a string from a request used as a name: ```ts declare const sort: string; // from the request db.select('films', { year: 1974 }); // error TS2353 db.sql`SELECT * FROM ${'films'} ORDER BY ${sort}`; // error TS2345 ``` Fix the code, not the type: with `as any`, or `as s.films.Column`, both compile, and both sort by whatever name the request sends. Map the request to names written in the code instead ([Raw SQL](https://pgcast.dev/raw-sql#choose-sql-at-run-time-from-your-own-constants)), and let a shortcut do the rest, here [`select()`](https://pgcast.dev/select#select) with [`db.all`](https://pgcast.dev/sql-template#all) and [`order`](https://pgcast.dev/select#order-limit-offset-and-withties): ```ts declare const sort: string; // from the request const by = sort === 'length' ? 'runtime_minutes' : 'release_year'; db.select('films', db.all, { order: { by, direction: 'ASC' } }); ``` ### Read the SQL Call [`.compile()`](https://pgcast.dev/sql-template#compile) where the code calls `.run()`, and print it: it gives `{ text, values }`, the SQL and its values. Here, the assistant's answer to "the three longest films of studio 1000", with [`columns`](https://pgcast.dev/select#columns), [`order` and `limit`](https://pgcast.dev/select#order-limit-offset-and-withties), and what Postgres runs: You write: ```ts const longest = await db.select('films', { studio_id: 1000 }, { columns: ['title', 'runtime_minutes'], order: { by: 'runtime_minutes', direction: 'DESC' }, limit: 3, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "title", $2::text, "runtime_minutes") AS result FROM "films" WHERE ("studio_id" = $3) ORDER BY "runtime_minutes" DESC LIMIT $4 ) AS "sq_films" ``` Values: `["title", "runtime_minutes", 1000, 3]` TypeScript knows: ```ts const longest: db.JSONOnlyColsForTable<"films", ("runtime_minutes" | "title")[]>[] // that is const longest: { runtime_minutes: number; title: string; }[] ``` With `.compile()` in place of `.run(pool)`, `console.log()` prints the same SQL as `text`, on one line, and the same values as `values`: - Each value is a `$n` in `text` and an item of `values`: here `1000` and `3`. The column names among the values are the keys of the JSON object each row comes back as ([`select()`](https://pgcast.dev/select#select)). - Each name is in double quotes: `"films"`, `"studio_id"`. - A value written into `text`, with no `$n` and no item in `values` for it, was pasted into the SQL: a `raw()` or a concatenation. Ask for a parameter instead. `.compile()` sends nothing, so it is safe on an `insert()` or a `deletes()` too. > **Coming** > > `npx pgcast --check` will say whether `schema.d.ts` still matches the database, without writing it, once [#52](https://github.com/AlexandreDemartin/pgcast/issues/52) lands: one more check to run before calling a change done. ## See also - [Raw SQL](https://pgcast.dev/raw-sql): the queries no shortcut writes, their names and their values. - [SQL template](https://pgcast.dev/sql-template#sql): what each interpolation compiles to. - [First query](https://pgcast.dev/first-query): the generator, the types it writes, and `.compile()`, step by step. --- # Switch from Zapatos Source: https://pgcast.dev/switching Move a project from Zapatos 6.6.1 to pgcast: change one line of `package.json`, regenerate, then let `tsc`, a look through your code and pgcast's warnings show you the few calls whose result changes. It assumes a project that already runs on Zapatos; for a new one, start at [Install](https://pgcast.dev/install). **At a glance** - One line of `package.json` swaps the package; imports, config and generated files stay as they are. - A few calls give other rows, or throw: [step 4](https://pgcast.dev/switching#find-the-calls-whose-result-changes) lists them, with what to do. - What code may rely on is kept, with a warning, until you turn on its opt-in. ## What stays the same - The module names: `zapatos/db`, `zapatos/schema`, `zapatos/custom` and `zapatos/generate`. No import changes ([decision 002](https://github.com/AlexandreDemartin/pgcast/blob/master/docs/decisions/002-module-names.md)). - The generator: `zapatosconfig.json` and its options, and `npx zapatos`, now also `npx pgcast` ([The CLI](https://pgcast.dev/generator#cli)). - Your placeholders in `zapatos/custom/`: written once, never overwritten. - A `Config` written for Zapatos 6.6.1: the keys pgcast adds are optional ([`Config`](https://pgcast.dev/runtime-config#Config)). - The schema's version canary, so a schema that Zapatos 6.6.1 generated compiles as it is ([`SchemaVersionCanary`](https://pgcast.dev/runtime-config#SchemaVersionCanary)). ## 1. Swap the package Install pgcast under the name Zapatos had: ```sh [npm] npm install zapatos@npm:@pgcast/pgcast ``` ```sh [pnpm] pnpm add zapatos@npm:@pgcast/pgcast ``` ```sh [yarn] yarn add zapatos@npm:@pgcast/pgcast@^6 ``` ```sh [bun] bun add zapatos@npm:@pgcast/pgcast@^6 ``` - `package.json` now reads `"zapatos": "npm:@pgcast/pgcast@^…"`, and `node_modules/zapatos` holds pgcast. - Nothing else to add: pgcast's peer dependencies are Zapatos 6.6.1's, `pg`, `@types/pg` and TypeScript 4.1 or later. - Not as plain `@pgcast/pgcast`: your `zapatos/*` imports would no longer resolve ([Install](https://pgcast.dev/install#add-the-package), § Why the alias). ## 2. Regenerate the schema ```sh npx zapatos ``` pgcast's generator is Zapatos 6.6.1's, with two fixes to what it writes. A `git diff` on your `zapatos/` folder shows nothing, unless: - a column comment runs over several lines: each line now starts with ` * `; - a table has a plain index named like another schema's unique index: its `UniqueIndex` no longer borrows that name. ## 3. Compile ```sh npx tsc ``` pgcast's types refuse a few calls that Zapatos 6.6.1's took. The one you may meet: `db.raw()`, `db.cols()` or `db.vals()` as the value of a filter or a condition. Zapatos 6.6.1 took it as a `Parameter`, and sent it as JSON, `{"value":"initcap('the salt orchard')"}`, so it matched no film. pgcast refuses it (2322, with `strict` on or off): ```ts const films = await db.select('films', { title: db.raw("initcap('the salt orchard')"), // error TS2322 }).run(pool); ``` To compare with SQL, give a `sql` fragment as the value, `db.self` for the column and the value as a [`db.param()`](https://pgcast.dev/sql-template#param): You write: ```ts const films = await db.select('films', { title: db.sql`${db.self} = initcap(${db.param('the salt orchard')})`, }, { columns: ['id', 'title'] }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "id", $2::text, "title") AS result FROM "films" WHERE (("title" = initcap($3))) ) AS "sq_films" ``` Values: `["id", "title", "the salt orchard"]` TypeScript knows: ```ts const films: db.JSONOnlyColsForTable<"films", ("id" | "title")[]>[] // that is const films: { id: number; title: string; }[] ``` - To compare with a plain value, give the value itself, or `db.param(x)`. - In a fragment, a plain string is a name, not a value: `${'the salt orchard'}` would be the column `"the salt orchard"` ([Identifiers](https://pgcast.dev/sql-template#identifiers)). - [`DangerousRawString`](https://pgcast.dev/sql-template#DangerousRawString), [`ColumnNames`](https://pgcast.dev/sql-template#ColumnNames) and [`ColumnValues`](https://pgcast.dev/sql-template#ColumnValues) no longer match a plain `{ value }`, nor each other: `const r: db.DangerousRawString = { value: 'x' }` fails (2741). Make them with `db.raw()`, `db.cols()` and `db.vals()`. - As an inserted or updated value of `insert()`, `update()` or `upsert()` (not in `updateValues`), the type refuses them too, and they throw before any query ([`insert()`](https://pgcast.dev/insert#insert)). ## 4. Find the calls whose result changes pgcast fixes, with no option to turn on, what no code could rely on in Zapatos 6.6.1: a call that Postgres or the type refused, that did the opposite of its name, or that let a string splice SQL. A few such calls did run, and now give other rows, or throw. Look through your code for the first column of each table below. Each change in What to do gives the same result under Zapatos 6.6.1, so you can make it before you switch; only the `jsonb` one finds its rows under pgcast alone. ### Conditions | Your code | Zapatos 6.6.1 | pgcast | What to do | |---|---|---|---| | [`not()`](https://pgcast.dev/conditions#conditions.not) of a fragment with a bare `OR` or `AND`: on `release_year`, `` not(db.sql`${db.self} = 1974 OR ${db.self} = 1991`) `` | `(NOT a) OR b`, so the 1991 film too | `(NOT (a OR b))` | for the old rows, `` db.sql`(NOT ${db.self} = 1974) OR ${db.self} = 1991` `` | | [`and()`](https://pgcast.dev/conditions#conditions.and) of a fragment with a bare `OR` and another operand: `` and(db.sql`${'id'} = 1000 OR ${'id'} = 1001`, { studio_id: 1001 }) `` | `a OR (b AND c)`, so film 1000 too | `((a OR b) AND (c))` | for the old rows, one fragment: `` db.sql`${'id'} = 1000 OR (${'id'} = 1001 AND ${'studio_id'} = ${db.param(1001)})` `` | ### `param()` and `parent()` | Your code | Zapatos 6.6.1 | pgcast | What to do | |---|---|---|---| | [`db.param()`](https://pgcast.dev/sql-template#param) as a filter value, in [`isIn()`](https://pgcast.dev/conditions#conditions.isIn) and [`isNotIn()`](https://pgcast.dev/conditions#conditions.isNotIn), in a plain object of `upsert()` `updateValues`, or in a [`vals([…])`](https://pgcast.dev/sql-template#vals), hand-written or in a fragment value of `insert()`, `update()` or `upsert()` | sent as JSON, `{"value":…}`: no row, every row with `isNotIn()`, other booleans in `updateValues`, `'{"value":…}'` stored, or 22P02 | the value, with its cast | nothing | | `{ data: db.param(x, true) }` on a `jsonb` column | no row | 42883, `jsonb = json` | `{ data: db.param(JSON.stringify(x), 'jsonb') }` | | [`db.parent()`](https://pgcast.dev/sql-template#parent) in `isIn()`, `isNotIn()` or a `vals([…])`: in a `lateral` subquery, or in a fragment value of `insert()`, `update()` or `upsert()` | sent as JSON: no row, every row with `isNotIn()`, `'{"value":…}'` stored, or 22P02 | the parent's column; outside a nested query, a write included, it throws | nothing in a `lateral`; in a write, the value itself | ### `Default`, `self`, `raw()`, `cols()` and `vals()` | Your code | Zapatos 6.6.1 | pgcast | What to do | |---|---|---|---| | [`db.Default`](https://pgcast.dev/sql-template#Default) or [`db.self`](https://pgcast.dev/sql-template#self) as a filter value, a condition's (`ne()`, `add()`), or in a `vals([…])` of a filter or a fragment value: `` insert('films', { …, tagline: db.sql`${db.vals([db.Default])}` }) `` | `'Symbol(DEFAULT)'` or `'Symbol(self)'`, sent or stored | throws while compiling | `db.Default` as the value itself, `{ …, tagline: db.Default }`, in `insert()` or `update()`; in a filter, a value to compare with | | `db.Default` in a `vals([…])` of a query you write: `` db.sql`INSERT INTO films (…) VALUES (${db.vals([…, db.Default])})` `` | `'Symbol(DEFAULT)'` | `DEFAULT` | nothing | | `db.raw()`, `db.cols()` or `db.vals()` as a filter value, a condition's, or in a `vals([…])` of a fragment value: `` { tagline: db.sql`${db.vals([db.raw("'Cy'")])}` } `` | sent as JSON, `{"value":"'Cy'"}` stored | refused by `tsc` as a filter value, and throws while compiling | in a filter, a `sql` fragment value ([Compile](https://pgcast.dev/switching#compile)); in a written value, `raw()` in the fragment itself: `` db.sql`${db.raw("'Cy'")}` `` | | `db.parent()`, `db.raw()`, `db.cols()` or `db.vals()` as a value of `insert()`, `update()` or `upsert()` where the type took it: on a column typed `any`, as a `zapatos/custom` placeholder left as generated is (`{ c: db.parent('id') }`), or in a `vals({…})` of a query you write, `` db.sql`INSERT INTO films (…) VALUES (${db.vals({ …, tagline: db.raw("upper('x')") })})` `` | sent as JSON, `'{"value":…}'` stored | throws before any query | in a write, the value itself, or for SQL a `sql` fragment value, `` db.sql`upper(${db.param('x')})` `` | | `db.parent()` with no column in a `vals({…})` of a `lateral` subquery: `` db.sql`SELECT ${db.vals({ name: db.parent() })}` `` | `'{}'` | the parent's column of its key ([`parent()`](https://pgcast.dev/sql-template#parent)) | nothing | ### `updateValues` and an empty `update()` | Your code | Zapatos 6.6.1 | pgcast | What to do | |---|---|---|---| | A plain value in [`upsert()`](https://pgcast.dev/insert#upsert-options) `updateValues`: a string, `{ tagline: 'films.title' }`, or an object | a column name, `ROW("films"."title")`, or for an object a condition of its keys (`TRUE` with none) | a parameter, the value itself | for the old result, a fragment: `` db.sql`${'films'}.${'title'}` `` | | [`update()`](https://pgcast.dev/insert#update) with no column: `update('films', {}, …)` | 42601 from Postgres | throws when called: "nothing to update: an Updatable needs at least one column" | nothing, unless you catch a `DatabaseError` there | ### Names and keywords | Your code | Zapatos 6.6.1 | pgcast | What to do | |---|---|---|---| | A name (a column, a table, a `lateral` key, an `alias`) or a `param()` cast that splices SQL, or holds a comment between its quoted parts: an [`order`](https://pgcast.dev/select#select) of `{ by: 'release_year" DESC, "title', … }` | `ORDER BY "release_year" DESC, "title" ASC` | `ORDER BY "release_year"" DESC, ""title" ASC`, one name: 42703 | one name per string: `order: [{ by: 'release_year', direction: 'DESC' }, { by: 'title', direction: 'ASC' }]` | | A [`lock`](https://pgcast.dev/select#lock), a [`truncate()`](https://pgcast.dev/insert#truncate) option, an isolation level, or an `aggregate` that is not a function name | written into the SQL | throws before any query | a value the type takes; for the `aggregate`, a function name (`sum`, `my_schema.my_agg`) | ### SQL that changes, rows that don't The text and values of `.compile()` change beyond these rows, where the rows returned don't: | Where | Zapatos 6.6.1 | pgcast | |---|---|---| | every `not()`: a pair of parentheses more | `(NOT "release_year" = $2)` | `(NOT ("release_year" = $2))` | | every operand of `and()`: a pair each | `(a AND b)` | `((a) AND (b))` | | every `param()` filter value, sent as it is: `{ title: db.param('x', 'text') }` | `$2` with `{"value":"x","cast":"text"}` | `CAST($2 AS "text")` with `'x'` | - Tests that snapshot SQL fail at [Read the warnings](https://pgcast.dev/switching#read-the-warnings): check the rows first, then update the snapshots. - Every case of each fix: the [changelog](https://pgcast.dev/changelog), § Fixed and § Security. ## 5. Read the warnings Run your tests, then your app. Where Zapatos 6.6.1 did something that code may rely on, pgcast keeps it, and warns once per process the first time a call does it: ```txt (node:4242) [PGCAST_DEP002] DeprecationWarning: An undefined value in insert(), update() or upsert() is sent as NULL. Set undefinedMeansMissing: true (db.setConfig) to treat it as a missing key instead (DEFAULT in insert(), left out of update() and upsert()); it will be the default in pgcast 7.0. (Use `node --trace-deprecation ...` to show where the warning was created) ``` To find each call: 1. Run with `--trace-deprecation`: Node prints the stack, and the first line below pgcast's own `dist/db/` frames is yours. A test runner's processes get it from `NODE_OPTIONS`: ```sh NODE_OPTIONS=--trace-deprecation npm test ``` 2. Fix the call, or turn on its option (table below). 3. Run again. Each code warns once, so the stack shows only its first call: the next call with the same code warns in turn. | Warning | The call | Fix it, or turn on | |---|---|---| | `PGCAST_DEP001` | a multi-row `upsert()` without [`updateColumns`](https://pgcast.dev/insert#upsert-options), with a key missing from some rows: inserted as `DEFAULT`, it overwrites the existing value ([Rows with different keys](https://pgcast.dev/insert#upsert-different-keys)) | give every row the key (`db.Default` keeps the SQL), or pass `updateColumns`; or [`upsertRequiresSameKeys`](https://pgcast.dev/runtime-config#Config.upsertRequiresSameKeys), and such a call throws | | `PGCAST_DEP002` | an `undefined` value in `insert()`, `update()` or `upsert()`, sent as `NULL` ([An `undefined` value](https://pgcast.dev/insert#undefined-values)) | leave the key out, or give `null` to erase the column; or [`undefinedMeansMissing`](https://pgcast.dev/runtime-config#Config.undefinedMeansMissing), and `undefined` is a missing key | | `PGCAST_DEP003` | a [`transaction()`](https://pgcast.dev/transaction#transaction) whose callback catches a query's error and returns: Postgres answers the `COMMIT` with `ROLLBACK`, and nothing is saved ([Let the error through](https://pgcast.dev/retries#let-the-error-through)) | let the error out of the callback, or `ROLLBACK TO` a `SAVEPOINT` before going on; or [`transactionRequiresCommit`](https://pgcast.dev/runtime-config#Config.transactionRequiresCommit), and such a transaction rejects | ## 6. Turn on the opt-ins Once a run shows no warning, or each one comes from a call you've checked, turn the options on, once, before the first query: ```ts db.setConfig({ upsertRequiresSameKeys: true, undefinedMeansMissing: true, transactionRequiresCommit: true }); ``` All three will be the default in 7.0 ([Versions](https://pgcast.dev/versions)). With them on, an `undefined` value is left out of an `update()`: You write: ```ts db.setConfig({ undefinedMeansMissing: true }); const films = await db.update('films', { title: 'The Salt Orchard', tagline: undefined, // from a form that left it out: the column keeps its value }, { id: 1000 }, { returning: ['id', 'title', 'tagline'] }).run(pool); ``` Postgres runs: ```sql UPDATE "films" SET ("title") = ROW($1) WHERE ("id" = $2) RETURNING jsonb_build_object($3::text, "id", $4::text, "title", $5::text, "tagline") AS result ``` Values: `["The Salt Orchard", 1000, "id", "title", "tagline"]` TypeScript knows: ```ts const films: db.JSONOnlyColsForTable<"films", ("title" | "tagline" | "id")[]>[] // that is const films: { title: string; tagline: string | null; id: number; }[] ``` - `insert()` sends `DEFAULT` for an `undefined` value, and an `update()` with no other key throws "nothing to update: an Updatable needs at least one column". - The `upsert()` of `PGCAST_DEP001` throws when called, before any query: "rows have different keys: pass updateColumns, or give every row the same keys". - A `transaction()` whose `COMMIT` Postgres answers with `ROLLBACK` rejects, without a retry ([`transaction()`, Errors](https://pgcast.dev/transaction#errors)). - No option warns. To keep it that way, `NODE_OPTIONS=--throw-deprecation npm test` fails the run at the first warning. One more opt-in, [`selectExactlyOneChecksForSecondRow`](https://pgcast.dev/runtime-config#Config.selectExactlyOneChecksForSecondRow), has no warning to look for, and stays off in 7.0. With it, a `selectExactlyOne()` run on its own rejects on a second row, not only on none. Turn it on where your tests show no `selectExactlyOne()` matching two rows. The generator has two opt-ins of its own, both to be the default in 7.0. Their warning is a note from `npx zapatos`, not your tests. The note of [`typeEnumsAcrossSchemas`](https://pgcast.dev/generator#generate.typeEnumsAcrossSchemas) names each column whose type is an enum of another schema, and that enum, typed until then as `any` or as the enum of the same name in the relation's schema: 1. Turn it on in `zapatosconfig.json`. 2. Delete the `zapatos/custom/` placeholder of a domain over such an enum: the generator never overwrites one, so the domain would keep the placeholder it already has, typed the old way. The new one gets the labels. 3. Regenerate, and let `tsc` show the values the enum's labels refuse. The note of [`enumOrder`](https://pgcast.dev/generator#generate.enumOrder) names each enum whose labels the generator sorts, though Postgres declares them in another order. Set it to `"declared"` and regenerate: for TypeScript, only `every.` changes, so the code to read is the code that takes a label from it by index or compares it with a tuple. ## Go back ```sh npm install zapatos@6.6.1 ``` - First, remove from `setConfig()` the options Zapatos 6.6.1's `Config` doesn't have (2353): `upsertRequiresSameKeys`, `undefinedMeansMissing`, `transactionRequiresCommit` and `selectExactlyOneChecksForSecondRow`. - Regenerate with Zapatos 6.6.1. It ignores `typeEnumsAcrossSchemas` and `enumOrder` in `zapatosconfig.json`, and writes again the `any` types, the sorted labels, the unprefixed comment lines and any borrowed `UniqueIndex` name. - Each change of [Find the calls whose result changes](https://pgcast.dev/switching#find-the-calls-whose-result-changes) runs as it did, but the `jsonb` one, which finds nothing again. > **Security** > > Code written or left as it is against pgcast's fixes gets Zapatos 6.6.1's results back: each row whose What to do is "nothing", the `not()` row and the `jsonb` row. > > `db.deletes('roles', { character: db.conditions.isNotIn([db.param('The ferryman')]) })` deletes 11 roles under pgcast, and all 12 under Zapatos 6.6.1. > > The holes of [6.6.2's § Security](https://pgcast.dev/changelog#6.6.2-security) are open again: a name, a `lock`, an `aggregate`, a plain value in `updateValues` or a `param()` cast can splice SQL ([Security](https://pgcast.dev/security)). ## See also - [Opt-ins and warnings](https://pgcast.dev/runtime-config#opt-ins-and-warnings): each option, its warning and its message. - [Versions](https://pgcast.dev/versions): why pgcast fixes some things outright and others behind an option. - [Security](https://pgcast.dev/security): why a name or a `raw()` value can no longer splice SQL. --- # Reference Source: https://pgcast.dev/reference Every export of `zapatos/db`, the generated utility types, and every option of the generator: its signature, the SQL it writes, its effect on the type. Dry and complete, for looking things up. --- # SQL template Source: https://pgcast.dev/sql-template `sql` is the tagged template every query in pgcast is built from, the shortcuts included. This page lists what it takes, what each interpolation compiles to, and the `SQLFragment` it returns. ## `sql` ```ts db.sql< Interpolations = db.SQL, RunResult = pg.QueryResult['rows'], Constraint = never, >`…`: SQLFragment ``` Returns a [`SQLFragment`](https://pgcast.dev/sql-template#SQLFragment): the literal parts as you wrote them, each interpolation compiled as below, every value a numbered parameter. Nothing is sent until [`run()`](https://pgcast.dev/sql-template#run). You write: ```ts const films = await db.sql`SELECT * FROM ${'films'} WHERE ${'release_year'} < ${db.param(2000)}`.run(pool); ``` Postgres runs: ```sql SELECT * FROM "films" WHERE "release_year" < $1 ``` Values: `[2000]` TypeScript knows: ```ts const films: s.films.Selectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` ### Type arguments | Argument | Default | What it sets | |---|---|---| | `Interpolations` | [`db.SQL`](https://pgcast.dev/sql-template#SQL): every table and column of the schema | what may be interpolated; `s.films.SQL` allows only the names of `films`, and `cols()` / `vals()` of any value | | `RunResult` | `pg.QueryResult['rows']` (`any[]`) | what `run()` resolves to, and the type of the fragment as a `lateral` or `extras` value | | `Constraint` | `never` | the value type a condition compares with; set by [`conditions`](https://pgcast.dev/conditions#conditions), checked by a Whereable. You rarely give it. | `RunResult` is not checked against the SQL: it is your word for what comes back. The shortcuts set it for you. A name outside `Interpolations` does not compile: ```ts db.sql`SELECT ${'name'} FROM ${'films'}`; // error TS2345 ``` ### Interpolations Each value between `${…}` compiles by what it is. The SQL is the built library's `.compile()`. | Value | Compiles to | Entry | |---|---|---| | a string | a quoted name: `'films'` → `"films"`, `'public.films'` → `"public"."films"` | [Identifiers](https://pgcast.dev/sql-template#identifiers) | | `param(x, cast?)` | `$1`, or `CAST($1 AS "timestamptz")`; `x` in the values | [`param()`](https://pgcast.dev/sql-template#param) | | `raw(s)` | `s`, as it is | [`raw()`](https://pgcast.dev/sql-template#raw) | | `cols(o)` / `cols(a)` | the object's keys, sorted, or the array's items, quoted: `"film_id", "tag"` | [`cols()`](https://pgcast.dev/sql-template#cols) | | `vals(o)` / `vals(a)` | the values, in the order of `cols(o)`, or the array's: `$1, $2` | [`vals()`](https://pgcast.dev/sql-template#vals) | | any other object | a Whereable of its own enumerable keys, plain or not: `("studio_id" = $1)`; `{}` → `TRUE`; a built-in object, such as a `Date`, throws | [Whereable objects](https://pgcast.dev/sql-template#whereable) | | `self` | the current column, in a Whereable's fragment value | [`self`](https://pgcast.dev/sql-template#self) | | `parent(c?)` | the column of the parent query: `"studios"."id"` | [`parent()`](https://pgcast.dev/sql-template#parent) | | `Default` | `DEFAULT` | [`Default`](https://pgcast.dev/sql-template#Default) | | a `sql` fragment | its own SQL, its parameters numbered on | [Nested fragments](https://pgcast.dev/sql-template#nested-fragments) | | an array | each item in turn, nothing between; `[]` → nothing | [Arrays](https://pgcast.dev/sql-template#arrays) | Anything else throws while compiling, so `run()` rejects before any query, and the type refuses it (`null` and `undefined` only with `strictNullChecks`): a number or a boolean (`Alien object while interpolating SQL: 42`), `undefined`, a symbol such as [`all`](https://pgcast.dev/sql-template#all), and `null`: "Interpolating `null` has no meaning in SQL: for a `NULL` value, interpolate `db.param(null)`…". Zapatos 6.6.1 threw a `TypeError` on `null`. The global object, which the type lets through, throws too: "Did you use `self` (the global object) where you meant `db.self` (the Zapatos value)?…". ## Identifiers An interpolated string is a table or column name. It is double-quoted, so its case is kept and a keyword is a name, and split on `.`. It can never close its quotes: | String | Compiles to | | |---|---|---| | `'films'` | `"films"` | | | `'public.films'` | `"public"."films"` | split on `.` | | `'a"b'` | `"a""b"` | a `"` is doubled, as `quote_ident` does | | `'a""b'` | `"a""b"` | already escaped by hand: as it is | | `'"Films"."title"'` | `"Films"."title"` | already a well-formed quoted name: as it is | | `'"films".title."x"'` | `"films".title."x"` | well-formed, ending with a quoted part: as it is | | `'"films".title'` | `"""films"""."title"` | not well-formed, as it ends unquoted: each `"` doubled, split on `.` | | `'"a" \|\| "b"'` | `"""a"" \|\| ""b"""` | not well-formed: one name | - Well-formed: a quoted identifier, then quoted, `U&` quoted or unquoted ones after each `.`, with or without spaces around it, the last one quoted. - The cost: a real column named `a""b` compiles to `"a""b"`, the column `a"b`. - Zapatos 6.6.1 wrapped the string in `"` without escaping, and wrote as it is any string that started and ended with `"`: a name could splice SQL around it. The generated `Table` and `Column` types already refused such names; the why is in [Security](https://pgcast.dev/security). Interpolate names rather than writing them in the literal: they are type-checked, and they follow the schema when you regenerate it. ## `param()` ```ts db.param(value: T, cast?: boolean | string): Parameter ``` A numbered parameter: `$1`, `$2`…, numbered across the whole query, and `value` at that place in the values. The value never enters the SQL text. You write: ```ts const screenings = await db.sql`SELECT * FROM ${'screenings'} WHERE ${'starts_at'} >= ${db.param('2030-03-02T00:00:00Z', 'timestamptz')}`.run(pool); ``` Postgres runs: ```sql SELECT * FROM "screenings" WHERE "starts_at" >= CAST($1 AS "timestamptz") ``` Values: `["2030-03-02T00:00:00Z"]` TypeScript knows: ```ts const screenings: s.screenings.Selectable[] // that is const screenings: { cinema_id: number; film_id: number; id: number; starts_at: Date; subtitled: boolean | null; }[] ``` The last two rows apply when [`castArrayParamsToJson`](https://pgcast.dev/runtime-config#Config.castArrayParamsToJson) or [`castObjectParamsToJson`](https://pgcast.dev/runtime-config#Config.castObjectParamsToJson) is on; both are off by default. | `cast` | Compiles to | Value sent | |---|---|---| | none | `$1` | `value`, as `pg` serialises it | | `'timestamptz'` | `CAST($1 AS "timestamptz")` | `value` | | `'pg_catalog"."text'` | `CAST($1 AS "pg_catalog"."text")` | `value` | | `'public.crew_job'`, `'int4[]'` | `CAST($1 AS "public"."crew_job")`, `CAST($1 AS "int4"[])` | `value` | | `true` | `CAST($1 AS "json")` for an array or a plain object; `$1` otherwise | `JSON.stringify(value)`; otherwise `value` | | `false` | `$1` | `value`, never as JSON | | none, for an array with `castArrayParamsToJson` on, or a plain object with `castObjectParamsToJson` on | `CAST($1 AS "json")` | `JSON.stringify(value)` | | a string, for an array or a plain object under its option | the cast, as above: `CAST($1 AS "jsonb")` | `JSON.stringify(value)`; under `castArrayParamsToJson`, an array whose cast ends with `[]` or `[n]`, `CAST($1 AS "int4"[])`, sends `value` | - A string `cast` without a `"` is split on `.`, each part quoted; array bounds at its end, `[]` or `[n]`, follow the quotes: `'public.crew_job[]'` gives `CAST($1 AS "public"."crew_job"[])`. - With a `"`, it is written as `"cast"` if that is well-formed (quoted parts joined by `.`), or as it is if it is already quoted that way: a type whose own name holds a `.` is `'"crew.job"'`, as `'crew.job'` is type `job` in schema `crew`. - Any other string with a `"` is one quoted name, any `"` doubled, so it cannot close its quotes: `'int4") || (SELECT 1) || ("int4'` gives `CAST($1 AS "int4"") || (SELECT 1) || (""int4")`, a type that does not exist. - Zapatos 6.6.1 wrote any `cast` as it is between quotes, so `'public.crew_job'` and `'int4[]'` named types that do not exist. - Each part stays quoted, so a name only the SQL grammar knows is no type here: `'int'`, `'double precision'`, `'varchar(10)'` fail (42704). Give the type's own name (`'int4'`, `'float8'`, `'timestamptz'`), or write the cast in `sql`: `` db.sql`CAST(${db.param(x)} AS varchar(10))` ``. - Under `castArrayParamsToJson` or `castObjectParamsToJson`, a string cast is the type written, and the value stays the JSON text the option sends (the last row of the table): `param({ a: 1 }, 'jsonb')` reaches `=` and `@>` on a `jsonb` column. - Under `castArrayParamsToJson`, an array whose cast ends with `[]` or `[n]` goes as an array: `'_int4'` and a domain over an array still send JSON, which Postgres refuses as an array (22P02). - Zapatos 6.6.1 dropped a string cast under either option for `CAST($1 AS "json")` ([#36](https://github.com/AlexandreDemartin/pgcast/issues/36)), so a cast that names another type than its target now applies: `param(arr, 'int4[]')` into a `json` column is refused (42804), into a `text` column stores `{1,2}` for `[1,2]`. - Without a cast, `castArrayParamsToJson` makes `` db.sql`${db.param(ids)}::int4[]` `` `CAST($1 AS "json")::int4[]`, which Postgres rejects (42846): write `db.param(ids, 'int4[]')`, or pass `false`, which keeps the value out of JSON under either option: `$1::int4[]`. - As a Whereable value or in `vals([…])` (so in [`isIn()`](https://pgcast.dev/conditions#conditions.isIn)), a `Parameter` compiles as it is, with its cast: `{ title: db.param('x', 'text') }` gives `("title" = CAST($1 AS "text"))`. Zapatos 6.6.1 wrapped it in a second parameter, sent as JSON (`'{"value":"x","cast":"text"}'`). ### `Parameter` ```ts class Parameter { value: T; cast?: boolean | string } ``` What `param()` returns. The generated Whereable, Insertable and Updatable types take a `Parameter` of the column's type wherever they take a `T`. ## `raw()` ```ts db.raw(sql: string): DangerousRawString ``` The string, written into the SQL as it is: for syntax a parameter cannot stand for, such as a keyword. You write: ```ts const films = await db.sql`SELECT * FROM ${'films'} ORDER BY ${'tagline'} ${db.raw('DESC NULLS LAST')}`.run(pool); ``` Postgres runs: ```sql SELECT * FROM "films" ORDER BY "tagline" DESC NULLS LAST ``` Values: `[]` TypeScript knows: ```ts const films: s.films.Selectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` > **Security** > > Whatever reaches `raw()` runs as SQL. Never give it a caller's text; a value is a [`param()`](https://pgcast.dev/sql-template#param). How to keep hand-written SQL safe: [Raw SQL](https://pgcast.dev/raw-sql). As a filter value, `raw()` throws while compiling, and so do `cols()` and `vals()`. As a Whereable value, in `isIn()` / `isNotIn()`, and in `vals([…])` within a filter: > `` `db.raw()`, `db.cols()` and `db.vals()` have no meaning as a value in a Whereable (a filter), nor in `isIn()`: to compare with SQL, use a `sql` fragment value, where `db.self` stands for the column; to compare with a value, give the value itself, or `db.param()`. `` As the value of `eq()`, `add()` and the other [conditions](https://pgcast.dev/conditions): > `` `db.raw()`, `db.cols()` and `db.vals()` have no meaning as the value of `eq()`, `add()` and the other conditions: for SQL, write a `sql` fragment with `db.self` in place of the condition; for a value, give the value itself, or `db.param()`. `` In [`vals({…})`](https://pgcast.dev/sql-template#vals), so as an `insert()`, `upsert()` or `update()` value, and in a `sql` template of your own, in a `lateral` or not: > `` `db.raw()`, `db.cols()` and `db.vals()` have no meaning as a value in `vals({…})`, so as an `insert()`, `upsert()` or `update()` value: to write SQL, use a `sql` fragment value (in `update()`, `db.self` there stands for the column); to write a value, give the value itself, or `db.param()`. `` The type refuses them there and as an `insert()` or `update()` value, on every column it types (a column typed `any`, as a `zapatos/custom` placeholder is, takes anything); it takes them in `vals({…})` of a `sql` template: ```ts db.sql`SELECT * FROM ${'films'} WHERE ${{ title: db.raw("'x'") }}`; // error TS2345 ``` - To compare with SQL: a fragment value, `` { title: db.sql`${db.self} = ${db.raw("'x'")}` } ``. - Zapatos 6.6.1 sent them there as one parameter, as JSON (`'{"value":…}'`): no row on a text or `jsonb` column, every row with `ne()` and `isNotIn()`, an error (22P02) on other types; as a write value, a text or `jsonb` column stored it. - Outside a filter, `vals([db.raw('x')])` is still a parameter, the wrapper as JSON: `'{"value":"x"}'`, which a text column stores as it is; in `upsert()`'s `updateValues`, `raw()` is SQL ([`upsert()`](https://pgcast.dev/insert#upsert)). ### `DangerousRawString` ```ts class DangerousRawString { value: string } ``` What `raw()` returns. A type-only member keeps it from passing as a `Parameter`, and an object of the same shape does not compile in its place (TS2741): ```ts const notRaw: db.DangerousRawString = { value: 'x' }; // error TS2741 ``` ## `cols()` ```ts db.cols(namesOrObject: T): ColumnNames ``` A comma-separated list of quoted names: - an object: its keys, sorted, `cols({ tag, film_id })` → `"film_id", "tag"`; - an array: its items in their order, each an [identifier](https://pgcast.dev/sql-template#identifiers), `cols(['id', 'title'])` → `"id", "title"`. With `vals()` of the same object, an `INSERT` in which keys and values line up: You write: ```ts function insertTag(tag: s.tags.Insertable) { return db.sql`INSERT INTO ${'tags'} (${db.cols(tag)}) VALUES (${db.vals(tag)}) RETURNING *`; } const inserted = await insertTag({ tag: 'coastal', film_id: 1000 }).run(pool); ``` Postgres runs: ```sql INSERT INTO "tags" ("film_id", "tag") VALUES ($1, $2) RETURNING * ``` Values: `[1000, "coastal"]` TypeScript knows: ```ts const inserted: s.tags.Selectable[] // that is const inserted: { film_id: number; tag: string; }[] ``` With arrays, a `SELECT` of some columns and an `IN` list; `s.films.OnlyCols<…>` is the row type of those columns: You write: ```ts const films = await db.sql[]>`SELECT ${db.cols(['id', 'title'])} FROM ${'films'} WHERE ${'id'} IN (${db.vals([1000, 1002])})`.run(pool); ``` Postgres runs: ```sql SELECT "id", "title" FROM "films" WHERE "id" IN ($1, $2) ``` Values: `[1000, 1002]` TypeScript knows: ```ts const films: s.films.OnlyCols<["id", "title"]>[] // that is const films: { id: number; title: string; }[] ``` As a filter value, and as a value in `vals({…})` (so in a write), `cols()` throws, as [`raw()`](https://pgcast.dev/sql-template#raw) does. ### `ColumnNames` ```ts class ColumnNames { value: T } ``` What `cols()` returns. A type-only member keeps it from passing as a `Parameter`. ## `vals()` ```ts db.vals(valuesOrObject: T): ColumnValues ``` A comma-separated list of values. Each value compiles as follows. | Value | In `vals({…})` | In `vals([…])` | |---|---|---| | a plain value | a parameter, `$1` | a parameter, `$1` | | a `Parameter` | as it is, with its cast | as it is, with its cast | | a `sql` fragment | its SQL, with its key as the current column (so [`self`](https://pgcast.dev/sql-template#self) works) | its SQL | | `Default` | `DEFAULT` | `DEFAULT`; throws in a filter | | `parent(c?)` | the [parent column](https://pgcast.dev/sql-template#parent), without `c` the column of its key; throws outside a nested query | the [parent column](https://pgcast.dev/sql-template#parent) | | `raw()`, `cols()`, `vals()` | throw | throw in a filter; a parameter outside one | | `self` | a parameter | throws in a filter; a parameter outside one | - An object's values come in the order of its sorted keys, as `cols()` lists them. - "In a filter" is wherever `vals([…])` has a current column: in `isIn()` and `isNotIn()`, and in a fragment value of a Whereable or of `vals({…})`. There: - `Default` throws "`` `db.Default` has no meaning in a Whereable (a filter)… ``"; - `self` "`` `db.self` has no meaning as a value in a Whereable (a filter), nor in `isIn()`… ``"; - `raw()`, `cols()` and `vals()` as [`raw()`](https://pgcast.dev/sql-template#raw) says. - As a filter value itself, or a value in `vals({…})`, `vals()` throws, as [`raw()`](https://pgcast.dev/sql-template#raw) does. - `insert()`, `upsert()` and `update()` compile their values with `vals({…})`, outside a nested query: `parent()`, `raw()`, `cols()` and `vals()` throw there, before any query. - Zapatos 6.6.1 wrapped a `Parameter`, `Default` and `parent()` in a second parameter in an array: `'{"value":…}'`, `'Symbol(DEFAULT)'`, `'{"value":"id"}'`; and `parent()`, `raw()`, `cols()` and `vals()` in an object, which the writes stored on a text or `jsonb` column. ### `ColumnValues` ```ts class ColumnValues { value: T } ``` What `vals()` returns. A type-only member keeps it from passing as a `Parameter`. ## Whereable objects An object is a Whereable of the table: each own enumerable key a column, each value what it must match. The keys are sorted, each pair compiled as below, joined by `AND`, the whole in parentheses; there is no `WHERE`, which you write. An empty object is `TRUE`. You write: ```ts const films = await db.sql`SELECT * FROM ${'films'} WHERE ${{ studio_id: 1000, runtime_minutes: db.sql`${db.self} > ${db.param(100)}`, }}`.run(pool); ``` Postgres runs: ```sql SELECT * FROM "films" WHERE (("runtime_minutes" > $1) AND "studio_id" = $2) ``` Values: `[100, 1000]` TypeScript knows: ```ts const films: s.films.Selectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` | Value | Compiles to | |---|---| | a plain value, `null` and arrays included | `("title" = $1)` | | a `Parameter` | `("title" = CAST($1 AS "text"))`, as it is | | a `sql` fragment | `(…)`, compiled with the key as the current column: [`self`](https://pgcast.dev/sql-template#self) is the column | | a [condition](https://pgcast.dev/conditions) | the same: a condition is a fragment, `like('The%')` | | `parent(c?)` | `("studio_id" = "studios"."id")`, in a `lateral` subquery | | `Default`, `self` | throws | | `raw()`, `cols()`, `vals()` | throws ([`raw()`](https://pgcast.dev/sql-template#raw)) | - `null` is a parameter too, and `= NULL` matches no row: use [`conditions.isNull`](https://pgcast.dev/conditions#conditions.isNull). - `Default` throws "`` `db.Default` has no meaning in a Whereable (a filter): leave the key out, or compare with a value. `DEFAULT` is for INSERT and UPDATE values. ``"; Zapatos 6.6.1 sent it as `'Symbol(DEFAULT)'`. - The shortcuts take the same object as their `where` ([`select()`](https://pgcast.dev/select#select)). > **Security** > > An interpolated object that is not one of the other [interpolations](https://pgcast.dev/sql-template#interpolations) compiles as a Whereable, plain or not: a class instance `{ title }` gives `("title" = $1)`, and one with no own enumerable key gives `TRUE`, even of a class that extends a built-in (`class Premiere extends Date {}`). Check what reaches a `where` at run time. A built-in object throws while compiling, before any query, even with own keys added (`Object.assign(new Date(0), { id: 1 })`): `Date`, `Map`, `Set`, `WeakMap`, `WeakSet`, `RegExp`, `Promise`, `Error` and its native subtypes (`TypeError`, …), `ArrayBuffer`, a typed array, `Buffer`, `DataView`, and the boxes `Number`, `String` and `Boolean`: > `` A built-in `…` has no meaning as a Whereable (a filter): it would compile to its own enumerable keys, and to `TRUE`, every row, when it has none. To compare with a value, give it as `db.param(…)`; to filter, give a plain object (`{ column: value }`); await a `Promise` first. `` Zapatos 6.6.1 compiled it as a Whereable of its own keys: `TRUE` for most, so `` db.sql`DELETE FROM ${'films'} WHERE ${new Date()}` `` was `DELETE FROM "films" WHERE TRUE` and deleted every row; `("0" = $1 …)` for a typed array. The type refuses most of them (TS2345), not where a column shares one of their members: `db.deletes('actors', new Error('x'))` type-checks, since `actors` has a `name` column. Only the built-ins of the realm that runs pgcast throw, matched by their exact prototype. As in Zapatos 6.6.1: - a `Date` from another realm, such as `vm.runInNewContext('new Date(0)')` (Jest runs each test file in a `vm` context), gives `TRUE`; - an instance of a class that extends a built-in keeps its own keys: `new Frames(1)`, with `class Frames extends Uint8Array {}`, gives `("0" = $1)`. ## `self` ```ts db.self ``` The current column, in a Whereable's fragment value: `` { runtime_minutes: db.sql`${db.self} > ${db.param(100)}` } `` gives `(("runtime_minutes" > $1))`. In a fragment value of `vals({…})`, the current column is its key: `` vals({ tag: db.sql`upper(${db.self})` }) `` gives `upper("tag")`. - Where there is no current column, it throws: "The 'self' column alias has no meaning here". - As a filter value it throws: it is the column, not a value to compare with it. Zapatos 6.6.1 sent it as the string `'Symbol(self)'`. - As a Whereable value, in `isIn()` / `isNotIn()` and in `vals([…])` within a filter: "`` `db.self` has no meaning as a value in a Whereable (a filter), nor in `isIn()`: compare with a value, or use it in a `sql` fragment value, where it stands for the column. ``" - As the value of `eq()`, `add()` and the other conditions: "`` `db.self` has no meaning as the value of `eq()`, `add()` and the other conditions: it is already the column on the left. To compare the column with itself, write a `sql` fragment with `db.self` on both sides. ``" ### `SelfType` ```ts type SelfType = typeof self ``` ## `parent()` ```ts db.parent(x?: T): ParentColumn ``` A column of the parent query's table, in a subquery given to `lateral` ([Joins](https://pgcast.dev/lateral)): `"".""`. You write: ```ts const studios = await db.select('studios', db.all, { lateral: { films: db.select('films', { studio_id: db.parent('id') }, { columns: ['title'] }), }, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("studios".*) || jsonb_build_object($1::text, "lateral_films".result) AS result FROM "studios" LEFT JOIN LATERAL ( SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($2::text, "title") AS result FROM "films" WHERE ("studio_id" = "studios"."id") ) AS "sq_films" ) AS "lateral_films" ON true ) AS "sq_studios" ``` Values: `["films", "title"]` TypeScript knows: ```ts const studios: (s.studios.JSONSelectable & db.LateralResult<{ films: db.SQLFragment[], never>; }>)[] // that is const studios: { id: number; name: string; films: db.JSONOnlyColsForTable<"films", "title"[]>[]; }[] ``` - Without a column, the column of the filter it sits in: `{ film_id: db.parent() }`, in a `lateral` of `roles`, gives `("film_id" = "roles"."film_id")`. With no filter, it throws before writing anything: "`db.parent()` with no column has no meaning here: name the column, `db.parent('id')`…". Zapatos 6.6.1 wrote the table and its `.`, then threw "Alien object while interpolating SQL: undefined". - It works as a Whereable value, in a `sql` fragment, in `eq()` and in `vals([…])`, so in `isIn()`: `{ film_id: isIn([db.parent()]) }` gives `(("film_id" IN ("roles"."film_id")))`. In `vals([…])`, Zapatos 6.6.1 sent it as JSON: `isIn([parent('id')])` compared with `'{"value":"id"}'`. - In `vals({…})` too, where without a column it takes the column of its key: `` db.sql`SELECT ${db.vals({ name: db.parent() })} AS result` ``, in a `lateral` of `studios`, gives `SELECT "studios"."name" AS result`. Zapatos 6.6.1 sent it there as JSON too. - Outside a nested query, it throws "The 'parent' table alias has no meaning here", so `run()` rejects before any query. So does every `insert()`, `upsert()` or `update()` value, where Zapatos 6.6.1 stored `{"value":"id"}` on a text column. ### `ParentColumn` ```ts class ParentColumn { value?: T } ``` What `parent()` returns. A type-only member keeps `cols()` and `vals()` from passing as a `ParentColumn`. ## `Default` ```ts db.Default ``` The keyword `DEFAULT`: a column's default, in an `INSERT` or `UPDATE` value, alone or in `vals()`. You write: ```ts const bookings = await db.sql`INSERT INTO ${'seat_bookings'} (${'screening_id'}, ${'seat'}, ${'customer'}, ${'booked_at'}) VALUES (${db.vals([1000, 'C7', 'Wren Hollis', db.Default])}) RETURNING *`.run(pool); ``` Postgres runs: ```sql INSERT INTO "seat_bookings" ("screening_id", "seat", "customer", "booked_at") VALUES ($1, $2, $3, DEFAULT) RETURNING * ``` Values: `[1000, "C7", "Wren Hollis"]` TypeScript knows: ```ts const bookings: s.seat_bookings.Selectable[] // that is const bookings: { booked_at: Date; customer: string; screening_id: number; seat: PgSeat_label; }[] ``` - In a filter it throws while compiling. - As a Whereable value ([above](https://pgcast.dev/sql-template#whereable)), in `isIn()` / `isNotIn()`, and in `vals([…])` with a current column ([`vals()`](https://pgcast.dev/sql-template#vals)): "`` `db.Default` has no meaning in a Whereable (a filter): leave the key out, or compare with a value. `DEFAULT` is for INSERT and UPDATE values. ``" - As the value of `eq()`, `add()` and the other conditions: "`` `db.Default` has no meaning as the value of `eq()`, `add()` and the other conditions: compare with a value. To set a column to its default in an UPDATE, give `db.Default` as the value itself: `{ n: db.Default }`. ``" - For `add()` and `subtract()`, set the column itself: `{ booked_at: db.Default }`. - Zapatos 6.6.1 sent it as the string `'Symbol(DEFAULT)'` in all of these, and in `vals([…])`, where it is now `DEFAULT`. ### `DefaultType` ```ts type DefaultType = typeof Default ``` ## Nested fragments A `sql` fragment interpolated in another compiles in place. Its parameters are numbered on from the outer ones, and it inherits the outer parent table and current column. A fragment that compiles to nothing, or an empty array, lets a query grow by condition: You write: ```ts function screeningsAt(cinemaId: number, subtitledOnly: boolean) { const subtitled = subtitledOnly ? db.sql` AND ${'subtitled'}` : []; return db.sql`SELECT * FROM ${'screenings'} WHERE ${'cinema_id'} = ${db.param(cinemaId)}${subtitled}`; } const subtitledAtTheLantern = await screeningsAt(1000, true).run(pool); ``` Postgres runs: ```sql SELECT * FROM "screenings" WHERE "cinema_id" = $1 AND "subtitled" ``` Values: `[1000]` TypeScript knows: ```ts const subtitledAtTheLantern: s.screenings.Selectable[] // that is const subtitledAtTheLantern: { cinema_id: number; film_id: number; id: number; starts_at: Date; subtitled: boolean | null; }[] ``` A nested fragment's `RunResult` and `runResultTransform` are not used: only the outer fragment's `run()` counts. ## Arrays An array's items compile one after the other, each as if interpolated alone, with nothing between them: ``[db.sql`a`, ' ', 'b']`` gives `a" ""b"`, the string `' '` being a name. An empty array compiles to nothing. To put something between the items, [`mapWithSeparator()`](https://pgcast.dev/sql-template#mapWithSeparator). ## `mapWithSeparator()` ```ts db.mapWithSeparator( arr: readonly TIn[], separator: TSep, cb: (x: TIn, i: number, a: readonly TIn[]) => TOut, ): (TOut | TSep)[] ``` Maps `arr` with `cb`, `separator` between each two results; `[]` for no items. A plain array, so it compiles as [one](https://pgcast.dev/sql-template#arrays): You write: ```ts const films = await db.sql`SELECT * FROM ${'films'} WHERE ${db.mapWithSeparator( ['The Salt Orchard', 'A Lantern for Edith'], db.sql` OR `, title => ({ title }), )}`.run(pool); ``` Postgres runs: ```sql SELECT * FROM "films" WHERE ("title" = $1) OR ("title" = $2) ``` Values: `["The Salt Orchard", "A Lantern for Edith"]` TypeScript knows: ```ts const films: s.films.Selectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` ## `all` ```ts db.all ``` The `where` of `select()`, `count()` and the other selects that matches every row: `db.select('films', db.all)` writes no `WHERE` ([`select()`](https://pgcast.dev/select#select)). It is not an interpolation: in `sql`, the type refuses it, and it throws. ### `AllType` ```ts type AllType = typeof all ``` ## `SQLFragment` ```ts class SQLFragment ``` What `sql` and every shortcut return: a query, or part of one, not yet compiled. You never construct one with `new`. | Member | What it is | |---|---| | [`run(queryable, force?)`](https://pgcast.dev/sql-template#run) | compiles, runs, and resolves to `RunResult` | | [`compile()`](https://pgcast.dev/sql-template#compile) | `{ text, values }`, the query `run()` sends | | [`runResultTransform`](https://pgcast.dev/sql-template#runResultTransform) | turns `pg`'s result into `RunResult` | | [`prepared(name?)`](https://pgcast.dev/sql-template#prepared) | makes it a prepared statement | | `copy(override?)` | a shallow copy, some properties replaced | | `noop`, `noopResult` | set by `insert()` and `upsert()` given `[]`: `run()` sends nothing and resolves to `noopResult` | | `parentTable` | set on a `lateral` subquery by its parent, for [`parent()`](https://pgcast.dev/sql-template#parent) | | `preparedName` | set by `prepared()` | | `compileExpression()` | used by `compile()`, internal | ### `run()` ```ts fragment.run(queryable: Queryable, force = false): Promise ``` 1. Compiles the fragment ([`compile()`](https://pgcast.dev/sql-template#compile)); an error there rejects, before any query. 2. Calls [`queryListener`](https://pgcast.dev/runtime-config#Config.queryListener) with the query and the transaction's id, if set. 3. Sends the query to `queryable`, and passes `pg`'s result through [`runResultTransform`](https://pgcast.dev/sql-template#runResultTransform). A `noop` fragment sends nothing and gives `noopResult`, unless `force` is `true`. 4. Calls [`resultListener`](https://pgcast.dev/runtime-config#Config.resultListener) with the result, the transaction's id, the time taken and the query, if set. A `noop` fragment still compiles, its SQL behind a comment. `force` sends it, and it inserts nothing: `.run()` gives `[]`. You write: ```ts const forced = await db.insert('tags', []).run(pool, true); ``` Postgres runs: ```sql /* marked no-op: won't hit DB unless forced -> */ INSERT INTO "tags" SELECT null WHERE false ``` Values: `[]` TypeScript knows: ```ts const forced: tags.JSONSelectable[] // that is const forced: { film_id: number; tag: string; }[] ``` ### `compile()` ```ts fragment.compile(): SQLQuery ``` The query as `pg` takes it ([`SQLQuery`](https://pgcast.dev/runtime-config#SQLQuery)): `text`, its parameters numbered from `$1`, and `values`, in their order; `name` too, once [prepared](https://pgcast.dev/sql-template#prepared). It sends nothing, and throws where an interpolation has no meaning. Its arguments are for its own recursion: call it with none. The text is not a stable API: a minor version may change it, as long as the result does not ([Testing](https://pgcast.dev/testing#the-sql-text-is-not-a-stable-api)). You write: ```ts const compiled = db.sql`SELECT * FROM ${'films'} WHERE ${{ studio_id: 1000 }}`.compile(); await pool.query(compiled); // what run() would send ``` Postgres runs: ```sql SELECT * FROM "films" WHERE ("studio_id" = $1) ``` Values: `[1000]` TypeScript knows: ```ts const compiled: db.SQLQuery // that is const compiled: { text: string; values: any[]; name?: string | undefined; } ``` ### `runResultTransform` ```ts fragment.runResultTransform: (qr: pg.QueryResult) => any // default: qr => qr.rows ``` Applied by `run()` to what `pg` returns. The shortcuts set their own (`selectOne()` takes the first row's `result`). Assign one to return something else; `RunResult` must then be the type it returns: You write: ```ts function filmCountQuery() { const query = db.sql`SELECT count(*) FROM ${'films'}`; query.runResultTransform = qr => Number(qr.rows[0].count); return query; } const filmCount = await filmCountQuery().run(pool); ``` Postgres runs: ```sql SELECT count(*) FROM "films" ``` Values: `[]` TypeScript knows: ```ts const filmCount: number ``` It applies only to `run()` on that fragment: not when it is nested in another, nor given to `lateral` or `extras`. ### `prepared()` ```ts fragment.prepared(name = `_zapatos_prepared_${n}`): this ``` Sets the `name` that `compile()` adds to the query, so that Postgres prepares the statement once per connection and runs it by name after ([node-postgres](https://node-postgres.com/features/queries#prepared-statements)). Without a name, `_zapatos_prepared_0`, `_1`… in the order of the calls. The values are parameters, not part of the text, so one name serves every call of a query function whose text does not change; without a name, each call prepares another statement: ```ts const filmById = (id: number) => db.selectOne('films', { id }).prepared('film_by_id'); console.log(filmById(1000).compile().name); // film_by_id const client = await pool.connect(); try { await filmById(1000).run(client); // Postgres prepares the statement on this connection, then runs it await filmById(1001).run(client); // and runs it by name: the same text, other values } finally { client.release(); } ``` A name stands for one text: on a connection that has prepared `'p'`, a fragment of another text named `'p'` makes `pg` reject with "Prepared statements must be unique - 'p' was used for a different statement". ## `RunResultForSQLFragment` ```ts type RunResultForSQLFragment> ``` The `RunResult` of `F`, with `undefined` read as `null`: a fragment nested as a `lateral` or `extras` value comes back as JSON, where a missing row is `null`. So `selectOne()`'s `crew.JSONSelectable | undefined` becomes `crew.JSONSelectable | null` there: ```ts const mentor = db.selectOne('crew', { id: db.parent('mentor_id') }, { alias: 'mentor' }); type Mentor = db.RunResultForSQLFragment; const nameOf = (row: Mentor) => row?.name ?? 'no mentor'; const crew = await db.select('crew', db.all, { lateral: { mentor } }).run(pool); const mentors = crew.map(member => nameOf(member.mentor)); ``` ## `Queryable` ```ts type Queryable = pg.ClientBase | pg.Pool ``` What `run()` takes: a pool, a client, or the client a [`transaction()`](https://pgcast.dev/transaction#transaction) gives its callback. Typed so, a query function runs on any of them: ```ts const titlesOf = (queryable: db.Queryable, studio_id: number) => db.select('films', { studio_id }, { columns: ['title'] }).run(queryable); const larkspur = await titlesOf(pool, 1000); const inTxn = await db.readCommitted(pool, txnClient => titlesOf(txnClient, 1000)); ``` `Queryable` takes a transaction's client whatever its level: a function that needs `SERIALIZABLE` takes `pg.Pool | TxnClientForSerializable` instead ([`TxnClientForSerializable`](https://pgcast.dev/transaction#TxnClientForSerializable), [Book inside a larger transaction](https://pgcast.dev/retries#book-inside-a-larger-transaction)). ## `SQL` ```ts type SQL = SQLExpression | SQLExpression[] ``` The default `Interpolations` of `sql`: an expression, or an array of them. Each table of the generated schema has its own, `s.films.SQL`, which takes the names and Whereables of that table only; `cols()` and `vals()` of any value, as in Zapatos 6.6.1. So `` db.sql`INSERT INTO ${'films'} (${db.cols({ seat: 1 })}) VALUES (${db.vals({ seat: 1 })})` `` type-checks, and fails only in Postgres (42703: `films` has no `seat`). ### `SQLExpression` ```ts type SQLExpression = Table | ColumnNames | ColumnValues | Whereable | Column | ParentColumn | GenericSQLExpression ``` One interpolation: a table or column name of any table, a Whereable, `parent()`, `cols()` / `vals()` of an Updatable, or a [`GenericSQLExpression`](https://pgcast.dev/sql-template#GenericSQLExpression), which takes `cols()` / `vals()` of any value. ### `GenericSQLExpression` ```ts type GenericSQLExpression = SQLFragment | Parameter | DefaultType | DangerousRawString | SelfType | ColumnNames | ColumnValues ``` The interpolations whose type is the same for every table: they fit in any table's `SQL`. `ColumnNames` and `ColumnValues` are among them, so the keys of `cols()` and `vals()` are not checked against the table. --- # Select Source: https://pgcast.dev/select The read shortcuts: `select()`, `selectOne()` and `selectExactlyOne()` for rows, `count()`, `sum()`, `avg()`, `min()` and `max()` for one number. Each option, the SQL it adds and what it does to the type, then the types they share. ## `select()` ```ts db.select(table, where, options?, mode?, aggregate?) ``` Builds one `SELECT` of the rows of `table` that match `where`. It returns a [`SQLFragment`](https://pgcast.dev/sql-template#SQLFragment): `.run(pool)` gives an array of the table's `JSONSelectable` rows, `[]` when none match. - `table`: a table of the generated schema. - `where`: a `Whereable`, an object of columns and their values or [conditions](https://pgcast.dev/conditions), its keys in alphabetical order, joined by `AND` in one pair of parentheses, a condition or `sql` value in its own (`("id" = $1 AND ("studio_id" > $2) AND "title" = $3)`); a [`sql`](https://pgcast.dev/sql-template#sql) fragment, written as it is; or [`db.all`](https://pgcast.dev/sql-template#all), for no `WHERE` at all. - `options`: the [options](https://pgcast.dev/select#options) below, all optional. - `mode` and `aggregate`: what `selectOne()`, `count()` and the others pass; see [`SelectResultMode`](https://pgcast.dev/select#SelectResultMode). You write: ```ts const films = await db.select('films', { studio_id: 1000 }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("films".*) AS result FROM "films" WHERE ("studio_id" = $1) ) AS "sq_films" ``` Values: `[1000]` TypeScript knows: ```ts const films: films.JSONSelectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` ### The SQL it writes - Each row is one `jsonb` value, `to_jsonb("films".*)`, so dates and times come back as strings and `bigint` as a number: the `JSONSelectable` types say so ([The catalog](https://pgcast.dev/catalog-types)). - The rows are gathered into one array, `coalesce(jsonb_agg(result), '[]')`, around a subquery aliased `"sq_"`, so `ORDER BY` and `LIMIT` apply to the rows. One query, one value back. - Inside, the clauses come in this order: `SELECT [DISTINCT …]` · the columns `AS result` · `FROM` the table `[AS alias]` · each `LEFT JOIN LATERAL` · `WHERE` · `GROUP BY` · `HAVING` · `ORDER BY` · `LIMIT` or `FETCH FIRST … WITH TIES` · `OFFSET` · each `FOR …` lock. - Every value is a parameter (`$1`, `$2`); every name is quoted as Postgres quotes it, any `"` in it doubled. Zapatos 6.6.1 let a name that held a `"` close its quotes; pgcast fixed this ([changelog](https://pgcast.dev/changelog#6.6.2-security)). ### Options The SQL of each row is what the built library compiles, on the docs schema; `…` stands for a `sql` fragment. | Option | SQL it adds | Effect on the type | |---|---|---| | `columns: ['title']` | `jsonb_build_object($1::text, "title")` in place of `to_jsonb("films".*)` | only those keys (`JSONOnlyColsForTable`) | | `order: { by: 'title', direction: 'ASC' }` | `ORDER BY "title" ASC` | none | | `limit: 2` | `LIMIT $1` | none | | `offset: 2` | `OFFSET $1` | none | | `withTies: true` | `FETCH FIRST $1 ROWS WITH TIES` in place of `LIMIT $1` | none | | `extras: { cinema: 'name' }` | `\|\| jsonb_build_object($1::text, "name")` | a key per extra (`ExtrasResult`) | | `lateral: { cast: … }` | `LEFT JOIN LATERAL (…) AS "lateral_cast" ON true`, read by `\|\| jsonb_build_object($1::text, "lateral_cast".result)` | a key per subquery (`LateralResult`) | | `lateral: …`, one subquery | `"lateral_passthru".result` in place of the columns, and its `LEFT JOIN LATERAL` | the subquery's result, in place of the row | | `alias: 'mentor'` | `FROM "crew" AS "mentor"`, then `"mentor"` for the table | none | | `distinct: true` | `SELECT DISTINCT` | none | | `distinct: 'cinema_id'` | `SELECT DISTINCT ON ("cinema_id")` | none | | `groupBy: 'film_id'` | `GROUP BY "film_id"` | none | | `having: …` | `HAVING sum("votes") >= $1` | none | | `lock: { for: 'UPDATE' }` | `FOR UPDATE` | none | > **Strict** > > `columns`, `extras` and `lateral` change the type only with `strictNullChecks` (or `strict`) on. Without it, the result stays the full `JSONSelectable` and an extra or a lateral key is not in the type; the type refuses nothing, not even `columns` beside one `lateral` subquery, and one subquery is typed as the table's row while its own result comes back. ### `columns` `columns?: readonly ColumnForTable[]`, a [`ColumnsOption`](https://pgcast.dev/select#ColumnsOption): only these columns, each a key of the result object, in place of the whole row. It does not rename a column: [`extras`](https://pgcast.dev/select#extras) does. A `readonly` array, `['id', 'title'] as const` or a `readonly ColumnForTable[]`, gives the same SQL and the same type as a mutable one. With `strictNullChecks`, Zapatos 6.6.1 typed its rows as `never`, which any type accepted; a type that the query does not return is now refused: ```ts const columns = ['id', 'title'] as const; async function filmCards(queryable: db.Queryable) { const cards: { id: number; title: string }[] = await db.select('films', db.all, { columns }).run(queryable); const withTaglines: { title: string; tagline: string | null }[] = await db.select('films', db.all, { columns }).run(queryable); return [cards, withTaglines]; } ``` ### `order`, `limit`, `offset` and `withTies` - `order`: one `{ by, direction, nulls? }` or an array of them, in that order. `by` is a column or any SQL of the table, its `SQL` type: a `sql` fragment (`` db.sql`array_length(${'genres'}, 1)` ``), [`db.parent()`](https://pgcast.dev/sql-template#parent); `direction` is `'ASC'` or `'DESC'`; `nulls`, `'FIRST'` or `'LAST'`. - `limit` and `offset`: numbers, sent as parameters. - `withTies: true`, with `limit` and `order`: also the rows that tie with the last one on the `order`. Without `limit` it adds nothing; without `order`, Postgres refuses the query (42601). You write: ```ts const page = await db.select('films', db.all, { columns: ['title', 'release_year'], order: [ { by: 'release_year', direction: 'DESC' }, { by: 'title', direction: 'ASC' }, ], limit: 2, offset: 2, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "title", $2::text, "release_year") AS result FROM "films" ORDER BY "release_year" DESC, "title" ASC LIMIT $3 OFFSET $4 ) AS "sq_films" ``` Values: `["title", "release_year", 2, 2]` TypeScript knows: ```ts const page: db.JSONOnlyColsForTable<"films", ("release_year" | "title")[]>[] // that is const page: { release_year: number; title: string; }[] ``` You write: ```ts const leads = await db.select('roles', db.all, { columns: ['film_id', 'character'], order: { by: 'billing', direction: 'ASC' }, limit: 1, withTies: true, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "film_id", $2::text, "character") AS result FROM "roles" ORDER BY "billing" ASC FETCH FIRST $3 ROWS WITH TIES ) AS "sq_roles" ``` Values: `["film_id", "character", 1]` TypeScript knows: ```ts const leads: db.JSONOnlyColsForTable<"roles", ("film_id" | "character")[]>[] // that is const leads: { film_id: number; character: string; }[] ``` Paging through a long list: [Pagination](https://pgcast.dev/paginate). ### `extras` `extras?: { [key: string]: column | SQLFragment }`, an [`ExtrasOption`](https://pgcast.dev/select#ExtrasOption): more keys on each result object. A column gives its value under another key; a `sql` fragment gives the type of its second type parameter, `` db.sql`…` ``, so `number` here. The keys are added to the row with `||`, after `columns`. You write: ```ts const cinemas = await db.select('cinemas', db.all, { columns: ['id'], extras: { cinema: 'name', km: db.sql`${'location'} <-> point(${db.param(0)}, ${db.param(0)})`, }, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "id") || jsonb_build_object($2::text, "name", $3::text, "location" <-> point($4, $5)) AS result FROM "cinemas" ) AS "sq_cinemas" ``` Values: `["id", "cinema", "km", 0, 0]` TypeScript knows: ```ts const cinemas: db.ReturningTypeForTable<"cinemas", "id"[], { cinema: "name"; km: db.SQLFragment; }>[] // that is const cinemas: { id: number; cinema: string; km: number; }[] ``` ### `lateral` `lateral?: { [key: string]: SQLFragment } | SQLFragment`, a [`LateralOption`](https://pgcast.dev/select#LateralOption): other shortcuts, nested, each joined with `LEFT JOIN LATERAL … ON true`. In them, [`db.parent()`](https://pgcast.dev/sql-template#parent) names a column of this query's table. - **An object**: each key becomes a key of the result, holding its subquery's result: an array for `select()`, an object or `null` for `selectOne()`, a number for `count()`. The keys are joined in alphabetical order; each is one identifier, `"lateral_"`. - **One subquery**: its result is passed through in place of each row. Then `columns` and `extras` have nothing to add to, and the type refuses them, with `strictNullChecks` on. Steps, nesting and self-joins: [Joins](https://pgcast.dev/lateral). Since pgcast 6.6.2, a key or an `alias` with a `"` in it stays one identifier; in Zapatos 6.6.1 it could close its quotes ([changelog](https://pgcast.dev/changelog#6.6.2-security)). ### `alias` `alias?: string`: a name for the table in this query, needed to join a table to itself (`crew` and its `mentor_id`). It replaces the table name in the columns, in `"sq_"` and for the `parent()` of each `lateral` subquery, and `lock.of` takes it. Its steps: [Join a table to itself](https://pgcast.dev/lateral#join-a-table-to-itself). ### `distinct` With `true`: `SELECT DISTINCT`, on the whole result object. The select list is that one `result`, so Postgres refuses an `order` on a column with it (42P10). You write: ```ts const cinemaIds = await db.select('screenings', db.all, { columns: ['cinema_id'], distinct: true, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT DISTINCT jsonb_build_object($1::text, "cinema_id") AS result FROM "screenings" ) AS "sq_screenings" ``` Values: `["cinema_id"]` TypeScript knows: ```ts const cinemaIds: db.JSONOnlyColsForTable<"screenings", "cinema_id"[]>[] // that is const cinemaIds: { cinema_id: number; }[] ``` With a column, an array of columns or a `sql` fragment: `SELECT DISTINCT ON (…)`, one row for each value. Give an `order` that starts with the same columns, or which row you get is up to Postgres. You write: ```ts const firstShows = await db.select('screenings', db.all, { columns: ['cinema_id', 'film_id', 'starts_at'], distinct: 'cinema_id', order: [ { by: 'cinema_id', direction: 'ASC' }, { by: 'starts_at', direction: 'ASC' }, ], }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT DISTINCT ON ("cinema_id") jsonb_build_object($1::text, "cinema_id", $2::text, "film_id", $3::text, "starts_at") AS result FROM "screenings" ORDER BY "cinema_id" ASC, "starts_at" ASC ) AS "sq_screenings" ``` Values: `["cinema_id", "film_id", "starts_at"]` TypeScript knows: ```ts const firstShows: db.JSONOnlyColsForTable<"screenings", ("film_id" | "cinema_id" | "starts_at")[]>[] // that is const firstShows: { film_id: number; cinema_id: number; starts_at: db.TimestampTzString; }[] ``` ### `groupBy` and `having` - `groupBy`: a column, an array of columns (`GROUP BY "film_id", "stars"`) or a `sql` fragment. - `having`: a `Whereable`, in parentheses as in `where`, or a `sql` fragment. Postgres returns only the grouped columns, those a grouped primary key determines, and aggregates, and refuses any other (42803): the whole row, without `columns`, runs when `groupBy` holds the primary key. You write: ```ts const popular = await db.select('ratings', db.all, { columns: ['film_id'], extras: { votes: db.sql`sum(${'votes'})` }, groupBy: 'film_id', having: db.sql`sum(${'votes'}) >= ${db.param(10)}`, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "film_id") || jsonb_build_object($2::text, sum("votes")) AS result FROM "ratings" GROUP BY "film_id" HAVING sum("votes") >= $3 ) AS "sq_ratings" ``` Values: `["film_id", "votes", 10]` TypeScript knows: ```ts const popular: db.ReturningTypeForTable<"ratings", "film_id"[], { votes: db.SQLFragment; }>[] // that is const popular: { film_id: number; votes: number; }[] ``` ### `lock` `lock?: SelectLockingOptions | SelectLockingOptions[]`: a locking clause each, in order, written after everything else. | Key | Values | SQL | |---|---|---| | `for` | `'UPDATE'`, `'NO KEY UPDATE'`, `'SHARE'`, `'KEY SHARE'` | `FOR UPDATE` | | `of`, optional | a table name, or the `alias`; or an array of them | `OF "screenings"` | | `wait`, optional | `'NOWAIT'`, `'SKIP LOCKED'` | `NOWAIT` | Postgres refuses a lock with `DISTINCT`, `GROUP BY` or an aggregate (0A000). You write: ```ts const screening = await db.selectOne('screenings', { id: 1000 }, { lock: { for: 'UPDATE', wait: 'NOWAIT' }, }).run(pool); ``` Postgres runs: ```sql SELECT to_jsonb("screenings".*) AS result FROM "screenings" WHERE ("id" = $1) LIMIT $2 FOR UPDATE NOWAIT ``` Values: `[1000, 1]` TypeScript knows: ```ts const screening: screenings.JSONSelectable | undefined // that is const screening: { cinema_id: number; film_id: number; id: number; starts_at: db.TimestampTzString; subtitled: boolean | null; } | undefined ``` ### Errors `select()` throws these as it builds the query, before any is sent: | Message | When | Since | |---|---|---| | `Direction must be ASC/DESC, not '…'` | an `order` with another `direction` | Zapatos | | `Nulls must be FIRST/LAST/undefined, not '…'` | an `order` with another `nulls` | Zapatos | | `Lock strength must be UPDATE/NO KEY UPDATE/SHARE/KEY SHARE, not '…'` | a `lock` with another `for` | pgcast 6.6.2 | | `Lock wait must be NOWAIT/SKIP LOCKED/undefined, not '…'` | a `lock` with another `wait` | pgcast 6.6.2 | | `Aggregate must be a function name (e.g. sum, or my_schema.my_agg), not '…'` | an `aggregate` that is not one, in `SelectResultMode.Numeric` | pgcast 6.6.2 | The type refuses a bad `direction`, `nulls` or `lock`, not a bad `aggregate`, a `string`. Zapatos 6.6.1 wrote a bad `lock` or `aggregate` into the SQL as it was, where it could run a second statement. A value that has no meaning in `where` or `having` (`db.Default`, `db.self`, `db.raw()`) throws as the query compiles: [Conditions](https://pgcast.dev/conditions). ## `selectOne()` ```ts db.selectOne(table, where, options?) ``` As [`select()`](https://pgcast.dev/select#select), with `LIMIT $1`, its value `1`, and no `jsonb_agg`: `.run(pool)` gives the first row, or `undefined` when none matches. Any `limit` you pass is replaced by `1`; `offset` and the other options are kept. In a `lateral`, its key holds the row or `null`. With `withTies` and `order`, it writes `FETCH FIRST $1 ROWS WITH TIES`, here and in `selectExactlyOne()`. `.run()` then gives one of the tied rows; in a `lateral`, the parent row comes back once per tie. You write: ```ts const latest = await db.selectOne('films', { studio_id: 1000 }, { order: { by: 'release_year', direction: 'DESC' }, }).run(pool); ``` Postgres runs: ```sql SELECT to_jsonb("films".*) AS result FROM "films" WHERE ("studio_id" = $1) ORDER BY "release_year" DESC LIMIT $2 ``` Values: `[1000, 1]` TypeScript knows: ```ts const latest: films.JSONSelectable | undefined // that is const latest: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; } | undefined ``` ## `selectExactlyOne()` ```ts db.selectExactlyOne(table, where, options?) ``` As [`selectOne()`](https://pgcast.dev/select#selectOne), but its type has no `undefined`: when no row matches, `.run()` rejects with a [`NotExactlyOneError`](https://pgcast.dev/select#NotExactlyOneError). It checks for at least one row, not exactly one: with `LIMIT $2`, its value `1`, it takes the first. In a `lateral`, a missing row gives `null` and no error, though the type says otherwise. You write: ```ts const studio = await db.selectExactlyOne('studios', { id: 1000 }).run(pool); ``` Postgres runs: ```sql SELECT to_jsonb("studios".*) AS result FROM "studios" WHERE ("id" = $1) LIMIT $2 ``` Values: `[1000, 1]` TypeScript knows: ```ts const studio: studios.JSONSelectable // that is const studio: { id: number; name: string; } ``` With [`selectExactlyOneChecksForSecondRow`](https://pgcast.dev/runtime-config#Config.selectExactlyOneChecksForSecondRow), the query run on its own asks for two rows, `LIMIT $2` with `2`, and `.run()` rejects on a second row too. Nested in another query, in a `lateral` or a `sql` template, it keeps `LIMIT 1`, option or not: a second row there would join the parent row twice. ### `NotExactlyOneError` ```ts class NotExactlyOneError extends Error { query: SQLFragment; } ``` What `selectExactlyOne()`, the mode `ExactlyOne` of `select()`, rejects with when no row matches. Its `name` is `'NotExactlyOneError'`, its `message` "One result expected but none returned (hint: check `.query.compile()` on this Error)"; with [`selectExactlyOneChecksForSecondRow`](https://pgcast.dev/runtime-config#Config.selectExactlyOneChecksForSecondRow), a second row gives "One result expected but more returned (hint: check `.query.compile()` on this Error)". Its `query` is the [`SQLFragment`](https://pgcast.dev/sql-template#SQLFragment) that ran: `err.query.compile()` gives its SQL and values. `err instanceof db.NotExactlyOneError` recognises it; catching it: [Tell a missing row from the others](https://pgcast.dev/handle-errors#tell-a-missing-row-from-the-others). ## `count()` ```ts db.count(table, where, options?) ``` `SELECT count(*) AS result`, with no `jsonb_agg`: `.run(pool)` gives a `number`, `0` when no row matches the `where`. The `bigint` that Postgres returns goes through `Number()`. You write: ```ts const roleCount = await db.count('roles', { actor_id: 1000 }).run(pool); ``` Postgres runs: ```sql SELECT count(*) AS result FROM "roles" WHERE ("actor_id" = $1) ``` Values: `[1000]` TypeScript knows: ```ts const roleCount: number ``` `count()` and the four below take the options of [`select()`](https://pgcast.dev/select#options), with these differences: | Option | SQL it adds | Effect on the type | |---|---|---| | `columns: ['subtitled']` | `count("subtitled")` in place of `count(*)`: the rows where it is not `NULL`; one column, as several (or none, `[]`) throw | none | | `distinct: true` | `SELECT DISTINCT count(*)`, on the one result: not `count(DISTINCT …)`, which needs a `sql` template | none | | `groupBy: 'film_id'` | `GROUP BY "film_id"`; `.run()` gives the first group's value only, and with no group rejects with a `TypeError` | none | | `extras` | nothing | none | | `lateral: { … }` | its `LEFT JOIN LATERAL`s, whose results are not read; a single subquery throws, as there is no row to pass it through to | none | | `having`, `limit`, `offset` | as in `select()`; when they leave no row (a `having` that keeps none, `limit: 0`, `offset: 1`), `.run()` rejects with a `TypeError` | none | `count()` and the four below throw these while compiling, so `.run()` rejects with a plain `Error`, not a `DatabaseError`, before any query, and `.compile()` throws; so does a query that nests one: | Message | When | Since | |---|---|---| | `…() needs one column to aggregate: pass columns: ['column']` | `sum()`, `avg()`, `min()` or `max()` without `columns` | pgcast 6.7.0 | | `…() takes one column in columns, not …` | `columns` with several columns, or none (`[]`) | pgcast 6.7.0 | | `…() cannot pass a lateral query through: nest it in the lateral of a select() instead` | a single subquery as `lateral` | pgcast 6.7.0 | Zapatos 6.6.1 sent the SQL, which Postgres refused: `sum(*)` and `count("id", "mood")` (42883), `count()` (42809), `SELECT AS result` (42601). The type is that of 6.6.1, which lets these calls through, a single subquery as `lateral` when there is no `columns` nor `extras`. Two calls that ran under 6.6.1 now reject too: - An aggregate of your own that takes another number of arguments under a built-in name. After `CREATE AGGREGATE max(int4, int4) (…)`, `db.max('ratings', db.all, { columns: ['stars', 'votes'] })` sent `max("stars", "votes")`. `db.select('ratings', db.all, { columns: ['stars', 'votes'] }, db.SelectResultMode.Numeric, 'max')` sends the same SQL. - A refused aggregate marked `noop`, which resolved to its `noopResult`: `run()` compiles the query before it reads `noop`. With `columns`, it counts two of the four screenings: the other two have `subtitled` `NULL`. You write: ```ts const subtitledKnown = await db.count('screenings', db.all, { columns: ['subtitled'], }).run(pool); ``` Postgres runs: ```sql SELECT count("subtitled") AS result FROM "screenings" ``` Values: `[]` TypeScript knows: ```ts const subtitledKnown: number ``` `count(DISTINCT …)` is a `sql` template. Postgres's `count()` returns a `bigint`, which `pg` gives as a string (`'3'`): `::int` makes it a number, here `3` cinemas. You write: ```ts const distinctCinemas = await db.sql`SELECT count(DISTINCT ${'cinema_id'})::int AS cinemas FROM ${'screenings'}`.run(pool); ``` Postgres runs: ```sql SELECT count(DISTINCT "cinema_id")::int AS cinemas FROM "screenings" ``` Values: `[]` TypeScript knows: ```ts const distinctCinemas: { cinemas: number; }[] ``` `where`, `alias`, `withTies` and `lock` add what they add to `select()`. So does `order`, by a `groupBy` column or a `sql` fragment (`` db.sql`count(*)` ``): by another column, Postgres refuses the query (42803). `.run()` reads the first row of the result. When the query returns none, it rejects with `TypeError: Cannot read properties of undefined (reading 'result')`; so do `sum()`, `avg()`, `min()` and `max()`. ## `sum()` ```ts db.sum(table, where, options?) ``` `SELECT sum("column") AS result`, a `number`; otherwise as [`count()`](https://pgcast.dev/select#count). Give one column in `columns`: without it, `.run()` rejects before any query, where Zapatos 6.6.1 sent `sum(*)`, which Postgres refuses (42883). The type lets that call through, as in 6.6.1. When no row matches the `where`, Postgres returns `NULL`, and `Number(null)` makes it `0`. You write: ```ts const minutes = await db.sum('films', { studio_id: 1000 }, { columns: ['runtime_minutes'], }).run(pool); ``` Postgres runs: ```sql SELECT sum("runtime_minutes") AS result FROM "films" WHERE ("studio_id" = $1) ``` Values: `[1000]` TypeScript knows: ```ts const minutes: number ``` ## `avg()` ```ts db.avg(table, where, options?) ``` `SELECT avg("column") AS result`: the mean, through `Number()`, so `0` when no row matches. Otherwise as [`sum()`](https://pgcast.dev/select#sum). You write: ```ts const meanRuntime = await db.avg('films', db.all, { columns: ['runtime_minutes'], }).run(pool); ``` Postgres runs: ```sql SELECT avg("runtime_minutes") AS result FROM "films" ``` Values: `[]` TypeScript knows: ```ts const meanRuntime: number ``` ## `min()` ```ts db.min(table, where, options?) ``` `SELECT min("column") AS result`, through `Number()`, so `0` when no row matches. The type says `number` whatever the column: - a date or time column (`date`, `timestamp`, `timestamptz`) gives its milliseconds since 1970, as node-postgres reads it as a `Date`; - text or an enum gives `NaN`, unless the text reads as a number: `'9'` gives `9`. Otherwise as [`sum()`](https://pgcast.dev/select#sum). You write: ```ts const earliest = await db.min('films', db.all, { columns: ['release_year'], }).run(pool); ``` Postgres runs: ```sql SELECT min("release_year") AS result FROM "films" ``` Values: `[]` TypeScript knows: ```ts const earliest: number ``` ## `max()` ```ts db.max(table, where, options?) ``` `SELECT max("column") AS result`; as [`min()`](https://pgcast.dev/select#min): `0` when no row matches, a date's milliseconds, and `NaN` on text or an enum unless the text reads as a number. You write: ```ts const longest = await db.max('films', { studio_id: 1000 }, { columns: ['runtime_minutes'], }).run(pool); ``` Postgres runs: ```sql SELECT max("runtime_minutes") AS result FROM "films" WHERE ("studio_id" = $1) ``` Values: `[1000]` TypeScript knows: ```ts const longest: number ``` ## Types What the signatures above are made of, from `zapatos/db`. `T` is a table, `C` the `columns`, `E` the `extras`, `L` the `lateral` and `A` the `alias`. ### `SelectResultMode` ```ts enum SelectResultMode { Many, One, ExactlyOne, Numeric } ``` The fourth argument of `select()`: an array of rows (`select()`), a row or `undefined` (`selectOne()`), a row or an error (`selectExactlyOne()`), a number (`count()` and the others). With `Numeric`, the fifth argument names the aggregate, `'count'` by default; since pgcast 6.6.2 it must be a function name, plain, quoted or `U&` quoted, in up to three dotted parts (`stddev_pop`, `my_schema."MyAgg"`). Any aggregate that returns a number works: You write: ```ts const spread = await db.select('films', db.all, { columns: ['runtime_minutes'], }, db.SelectResultMode.Numeric, 'stddev_pop').run(pool); ``` Postgres runs: ```sql SELECT stddev_pop("runtime_minutes") AS result FROM "films" ``` Values: `[]` TypeScript knows: ```ts const spread: number ``` ### `SelectOptionsForTable` `SelectOptionsForTable`: the `options` object of `select()`, `selectOne()`, `selectExactlyOne()` and the aggregates, every key optional, as in [Options](https://pgcast.dev/select#options). ### `SelectLockingOptions` ```ts interface SelectLockingOptions { for: 'UPDATE' | 'NO KEY UPDATE' | 'SHARE' | 'KEY SHARE'; of?: Unprefixed | A | (Unprefixed
| A)[]; wait?: 'NOWAIT' | 'SKIP LOCKED'; } ``` One locking clause of [`lock`](https://pgcast.dev/select#lock). `of` takes a table name without its schema, or the query's `alias` (`A`). ### `ColumnsOption` `ColumnsOption = readonly ColumnForTable[] | undefined`: the type of `columns`, and of `returning` in [`insert()`](https://pgcast.dev/insert#insert) and the other writes. A `readonly` array gives the same type as a mutable one ([`columns`](https://pgcast.dev/select#columns)). ### `ExtrasOption` `ExtrasOption = SQLFragmentOrColumnMap | undefined`: the type of `extras`, here and in the writes. ### `LateralOption` `LateralOption`: the type of `lateral`. An object of subqueries ([`SQLFragmentMap`](https://pgcast.dev/select#SQLFragmentMap)); or one subquery, only when neither `columns` nor `extras` is given. ### `SQLFragmentMap` `interface SQLFragmentMap { [k: string]: SQLFragment }`: the object form of `lateral`, a key per subquery. ### `SQLFragmentOrColumnMap` `interface SQLFragmentOrColumnMap { [k: string]: SQLFragment | ColumnForTable }`: the form of `extras`, a key per column or fragment. ### `JSONOnlyColsForTable` `JSONOnlyColsForTable = Pick, C[number]>`: a row with only the `columns` (or `returning`) given. It names that row in a signature: ```ts type FilmCard = db.JSONOnlyColsForTable<'films', ['id', 'title']>; function filmCardsOf(queryable: db.Queryable, studio_id: number): Promise { return db.select('films', { studio_id }, { columns: ['id', 'title'] }).run(queryable); } ``` - `C` may be a `readonly` tuple, as `columns` may ([`columns`](https://pgcast.dev/select#columns)). ### `ExtrasResult` `ExtrasResult`: the keys that `extras` adds, each typed as its column in `JSONSelectable`, or as its fragment's result. ### `LateralResult` `LateralResult = { [K in keyof L]: RunResultForSQLFragment }`: the keys that a `lateral` object adds, each typed as its subquery's result, with `undefined` turned into `null` ([`RunResultForSQLFragment`](https://pgcast.dev/sql-template#RunResultForSQLFragment)). ### `FullSelectReturnTypeForTable` `FullSelectReturnTypeForTable`: what `.run()` gives for a mode `M`: the row (the `JSONSelectable`, narrowed by `columns`, with `extras` and `lateral`; or the passed-through subquery's result), as an array, as is, or with `undefined`; or a `number`. The row of the writes is [`ReturningTypeForTable`](https://pgcast.dev/insert#ReturningTypeForTable). ### `SelectSignatures` The call signature of `select()`: `table`, `where`, `options?`, `mode?` (by default `Many`) and `aggregate?`, returning `SQLFragment>`. ### `SelectOneSignatures` The call signature of `selectOne()`: `table`, `where`, `options?`, with the mode `One`. ### `SelectExactlyOneSignatures` The call signature of `selectExactlyOne()`: `table`, `where`, `options?`, with the mode `ExactlyOne`. ### `NumericAggregateSignatures` The call signature of `count()`, `sum()`, `avg()`, `min()` and `max()`: `table`, `where`, `options?`, returning `SQLFragment`. --- # Writes Source: https://pgcast.dev/insert The shortcuts that write: `insert()`, `upsert()`, `update()`, `deletes()` and `truncate()`. For each, its parameters, every option with the SQL it adds and its effect on the type, and what it throws. What every write shares: - It returns a `SQLFragment`: [`.run(pool)`](https://pgcast.dev/sql-template#run) sends it, `.compile()` gives its `text` and `values` without sending anything. - A value is sent as a parameter (`$1`), never written into the SQL; a `sql` fragment is SQL. - Rows come back through `to_jsonb()`, as a table's `JSONSelectable`: a `timestamptz` is a string ([The catalog](https://pgcast.dev/catalog-types)). - An error from Postgres rejects `.run()` with a `DatabaseError` ([`isDatabaseError()`](https://pgcast.dev/transaction#isDatabaseError)). The errors below are thrown by pgcast as it builds the query, before anything is sent. ## `insert()` ```ts db.insert(table, values, options?) ``` Insert one row or several, and return them as Postgres stored them, defaults and serials filled in. - `table`: a table of the schema. - `values`: an `Insertable`, or an array of them. A `sql` fragment is written as SQL (`` db.sql`now()` ``), `db.Default` as `DEFAULT`; an `undefined` value is `NULL` ([An `undefined` value](https://pgcast.dev/insert#undefined-values)). - `raw()`, `cols()` and `vals()` are no values here: they throw while compiling, before any query ([`raw()`](https://pgcast.dev/sql-template#raw) gives the message), and pgcast's type refuses them, where Zapatos 6.6.1's took them. For SQL, write a `sql` fragment; in an array or a `jsonb` object, they are part of the value, sent as JSON. - `db.parent()` throws "The 'parent' table alias has no meaning here": an `INSERT` has no parent query. The type refuses it. - A column typed `any`, as a `zapatos/custom` placeholder is, passes the type, not the run time. Zapatos 6.6.1 sent each as one parameter, as JSON: `{ tagline: db.raw("'x'") }` stored `{"value":"'x'"}`, `db.parent('id')` `{"value":"id"}` ([#259](https://github.com/AlexandreDemartin/pgcast/issues/259)). - `options`: `returning` and `extras` ([Options](https://pgcast.dev/insert#insert-options)). You write: ```ts const films = await db.insert('films', [{ studio_id: 1001, title: 'The Tidewater Ledger', release_year: 2031, runtime_minutes: 117, tagline: 'Every debt comes in with the tide.', }, { studio_id: 1002, title: 'Moths over Harrowgate', release_year: 2031, runtime_minutes: 95, }]).run(pool); ``` Postgres runs: ```sql INSERT INTO "films" ("release_year", "runtime_minutes", "studio_id", "tagline", "title") VALUES ($1, $2, $3, $4, $5), ($6, $7, $8, DEFAULT, $9) RETURNING to_jsonb("films".*) AS result ``` Values: `[2031, 117, 1001, "Every debt comes in with the tide.", "The Tidewater Ledger", 2031, 95, 1002, "Moths over Harrowgate"]` TypeScript knows: ```ts const films: films.JSONSelectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` - One row gives one result; an array gives an array, in the order Postgres returns them, in practice the order given. - The columns are the keys of every row; a key missing from a row is `DEFAULT` in that row, as `tagline` above. - An empty array is a no-op: `.run(pool)` resolves with `[]` and sends nothing, unless forced with `.run(pool, true)`. Its SQL, `INSERT INTO "films" SELECT null WHERE false`, is marked as a no-op. - Postgres takes at most 65,535 parameters in a query: split a bigger insert ([Batches](https://pgcast.dev/batches)). ### Options `returning` and `extras` are the same for `upsert()`, `update()` and `deletes()` ([`ReturningOptionsForTable`](https://pgcast.dev/insert#ReturningOptionsForTable)). You write: ```ts const film = await db.insert('films', { studio_id: 1000, title: 'Winter at Fennick Lock', release_year: 2032, runtime_minutes: 108, }, { returning: ['id'], extras: { name: 'title', shouted: db.sql`upper(${'title'})`, }, }).run(pool); ``` Postgres runs: ```sql INSERT INTO "films" ("release_year", "runtime_minutes", "studio_id", "title") VALUES ($1, $2, $3, $4) RETURNING jsonb_build_object($5::text, "id") || jsonb_build_object($6::text, "title", $7::text, upper("title")) AS result ``` Values: `[2032, 108, 1000, "Winter at Fennick Lock", "id", "name", "shouted"]` TypeScript knows: ```ts const film: db.ReturningTypeForTable<"films", "id"[], { name: "title"; shouted: db.SQLFragment; }> // that is const film: { id: number; name: string; shouted: string; } ``` | Option | SQL it adds | Effect on the type | |---|---|---| | none | `RETURNING to_jsonb("films".*) AS result` | `films.JSONSelectable` | | `returning: ['id']` | `jsonb_build_object($5::text, "id")` in place of `to_jsonb("films".*)` | `Pick` | | `extras: { name: 'title', shouted: … }`, as above | `\|\| jsonb_build_object($6::text, "title", $7::text, upper("title"))` | adds `name`, of the column's type, and `shouted`, of the fragment's `RunResult`: `string`, from `db.sql` | - A `readonly` array in `returning` (`['id'] as const`) gives the same SQL and the same type as a mutable one ([`columns`](https://pgcast.dev/select#columns)). - Both need `strictNullChecks` for their types. ## `upsert()` ```ts db.upsert(table, values, conflictTarget, options?) ``` `INSERT … ON CONFLICT … DO UPDATE`: insert each row, or update the existing row it conflicts with. - `table`, `values`: as for [`insert()`](https://pgcast.dev/insert#insert); `raw()`, `cols()`, `vals()` and `parent()` throw there too, on a conflicting row or a new one. - `conflictTarget`: the arbiter of the conflict: a column, an array of columns, or the name of a unique or primary-key constraint wrapped in [`constraint()`](https://pgcast.dev/insert#constraint). - `options`: `updateColumns`, `noNullUpdateColumns`, `updateValues` and `reportAction` ([Options](https://pgcast.dev/insert#upsert-options)); `returning` and `extras`, as for [`insert()`](https://pgcast.dev/insert#insert-options). You write: ```ts const rating = await db.upsert('ratings', { film_id: 1002, stars: 4, }, ['film_id', 'stars'], { updateValues: { votes: db.sql`${'ratings'}.${'votes'} + 1` }, }).run(pool); ``` Postgres runs: ```sql INSERT INTO "ratings" ("film_id", "stars") VALUES ($1, $2) ON CONFLICT ("film_id", "stars") DO UPDATE SET ("film_id", "stars", "votes") = ROW(EXCLUDED."film_id", EXCLUDED."stars", "ratings"."votes" + 1) RETURNING to_jsonb("ratings".*) || jsonb_build_object('$action', CASE xmax WHEN 0 THEN 'INSERT' ELSE 'UPDATE' END) AS result ``` Values: `[1002, 4]` TypeScript knows: ```ts const rating: db.UpsertReturnableForTable<"ratings", db.ColumnsOption<"ratings">, db.ExtrasOption<"ratings">, "suppress" | undefined> // that is const rating: { film_id: number; stars: number; votes: number; $action: "INSERT" | "UPDATE"; } ``` - The update sets every column inserted, from `EXCLUDED`, and each key of `updateValues`. - Each result has `$action`, `'INSERT'` or `'UPDATE'` ([`UpsertAction`](https://pgcast.dev/insert#UpsertAction)). - One row gives one result; an array gives an array, without the rows that do nothing ([`doNothing`](https://pgcast.dev/insert#doNothing)). - An empty array is a no-op, as for `insert()`. ### Options Each SQL replaces the `DO UPDATE SET …` of the plain upsert, on `ratings` (on `films` for `noNullUpdateColumns`), or adds to it: | Option | SQL it adds | Effect on the type | |---|---|---| | none | `DO UPDATE SET ("film_id", "stars", "votes") = ROW(EXCLUDED."film_id", EXCLUDED."stars", EXCLUDED."votes")` | `ratings.JSONSelectable & UpsertAction` | | `updateColumns: 'votes'` | `DO UPDATE SET ("votes") = ROW(EXCLUDED."votes")` | none | | `updateColumns: db.doNothing` | `DO NOTHING` | `\| undefined`, for one row | | `noNullUpdateColumns: 'tagline'` | `CASE WHEN EXCLUDED."tagline" IS NULL THEN "films"."tagline" ELSE EXCLUDED."tagline" END` in place of `EXCLUDED."tagline"` | none | | `updateValues: { votes: 10 }` | `ROW(EXCLUDED."film_id", EXCLUDED."stars", $3)`: `$3` in place of `EXCLUDED."votes"` | none | | ``updateValues: { votes: db.sql`${'ratings'}.${'votes'} + 1` }`` | `"ratings"."votes" + 1` in place of `EXCLUDED."votes"` | none | | `reportAction: 'suppress'` | no `\|\| jsonb_build_object('$action', CASE xmax WHEN 0 THEN 'INSERT' ELSE 'UPDATE' END)` | no `UpsertAction` | - `updateColumns` takes a column or an array; `noNullUpdateColumns` a column, an array, or `db.all` for every column. [Batches](https://pgcast.dev/batches#rows-with-different-keys) has a panel of `updateColumns`: a booking keeps its time. - `updateValues` is one set of values for every row updated. Its keys are always updated, even when `updateColumns` leaves them out, or is `doNothing` (then the SQL is `DO UPDATE SET`, not `DO NOTHING`). - In `updateValues`, a plain value is a parameter. A value that holds a fragment, `param()`, `raw()`, `cols()`, `vals()`, `parent()`, `Default` or `self`, at any depth of its arrays and plain objects, is compiled as in a `sql` template. - To read a column there, name the table for the existing row, `${'ratings'}.${'votes'}`, or `EXCLUDED` for the new one, `` db.sql`EXCLUDED.${'votes'}` ``. - There, `db.self`, or a fragment or condition that uses it (`` db.sql`${db.self} + 1` ``, [`conditions.add()`](https://pgcast.dev/conditions#conditions.add)), throws "The 'self' column alias has no meaning here", as in Zapatos 6.6.1; `db.parent()`, alone or in a fragment, throws "The 'parent' table alias has no meaning here". > **Security** > > Zapatos 6.6.1 wrote a plain string in `updateValues` as an identifier: `{ tagline: 'happy' }` put `"happy"` in the `ROW(…)`, and a string from a user could read other columns or tables. > > pgcast sends it as a parameter, `$3` in its place, as its type and `update()` always said. To read a column there, write a fragment, as in the table ([Security](https://pgcast.dev/security)). With `noNullUpdateColumns`, a `NULL` leaves the existing value. Film 1002 comes back from a feed with no tagline, and keeps its own: You write: ```ts const film = await db.upsert('films', { id: 1002, studio_id: 1000, title: "The Cartographer's Silence", release_year: 2008, runtime_minutes: 131, tagline: null, }, 'id', { noNullUpdateColumns: 'tagline' }).run(pool); ``` Postgres runs: ```sql INSERT INTO "films" ("id", "release_year", "runtime_minutes", "studio_id", "tagline", "title") VALUES ($1, $2, $3, $4, $5, $6) ON CONFLICT ("id") DO UPDATE SET ("id", "studio_id", "title", "release_year", "runtime_minutes", "tagline") = ROW(EXCLUDED."id", EXCLUDED."studio_id", EXCLUDED."title", EXCLUDED."release_year", EXCLUDED."runtime_minutes", CASE WHEN EXCLUDED."tagline" IS NULL THEN "films"."tagline" ELSE EXCLUDED."tagline" END) RETURNING to_jsonb("films".*) || jsonb_build_object('$action', CASE xmax WHEN 0 THEN 'INSERT' ELSE 'UPDATE' END) AS result ``` Values: `[1002, 2008, 131, 1000, null, "The Cartographer's Silence"]` TypeScript knows: ```ts const film: db.UpsertReturnableForTable<"films", db.ColumnsOption<"films">, db.ExtrasOption<"films">, "suppress" | undefined> // that is const film: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; $action: "INSERT" | "UPDATE"; } ``` - The update still sets every column inserted, `"id"` included; only `"tagline"` keeps the existing value when the new one is `NULL`. - `film` is the updated row: `$action` is `'UPDATE'`, and `tagline` is still `'Every map leaves something out.'`. With `reportAction: 'suppress'`, the result has no `$action`: You write: ```ts const rating = await db.upsert('ratings', { film_id: 1002, stars: 4, votes: 2, }, ['film_id', 'stars'], { reportAction: 'suppress' }).run(pool); ``` Postgres runs: ```sql INSERT INTO "ratings" ("film_id", "stars", "votes") VALUES ($1, $2, $3) ON CONFLICT ("film_id", "stars") DO UPDATE SET ("film_id", "stars", "votes") = ROW(EXCLUDED."film_id", EXCLUDED."stars", EXCLUDED."votes") RETURNING to_jsonb("ratings".*) AS result ``` Values: `[1002, 4, 2]` TypeScript knows: ```ts const rating: ratings.JSONSelectable // that is const rating: { film_id: number; stars: number; votes: number; } ``` With `strictNullChecks` off, the type keeps `$action`, which is then `undefined`. ### Rows with different keys In several rows without `updateColumns`, a key missing from one row is inserted as `DEFAULT`, then set from `EXCLUDED` on conflict: that row's existing value is lost. - `[{ film_id: 1002, stars: 5, votes: 3 }, { film_id: 1002, stars: 4 }]` sends `($4, $5, DEFAULT)`, so an existing count of 4-star votes goes back to 1. - It runs, as in Zapatos 6.6.1, and warns once per process: `PGCAST_DEP001`. - A key of `updateValues`, or a column of an array or string `conflictTarget`, does not count: new rows without their serial `id`, upserted on `'id'`, neither warn nor throw. With `constraint()`, every key counts. - Rows with the same keys, one row, and any `updateColumns` never warn. > **Opt-in** > > With `upsertRequiresSameKeys: true` ([`Config`](https://pgcast.dev/runtime-config#Config.upsertRequiresSameKeys)), such an upsert throws before any query: "rows have different keys: pass updateColumns, or give every row the same keys". Without it, pgcast warns as `PGCAST_DEP001`; it becomes the default in 7.0 ([Versions](https://pgcast.dev/versions)). ## `constraint()` ```ts db.constraint(name) ``` Wrap the name of a unique or primary-key constraint of the table, for the `conflictTarget` of `upsert()`: `ON CONFLICT ON CONSTRAINT "ratings_pkey"`, in place of `ON CONFLICT ("film_id", "stars")`. The names come from the schema, so one the table doesn't have fails to compile: ```ts db.upsert('ratings', { film_id: 1002, stars: 4 }, db.constraint('ratings_pkey')); db.upsert('ratings', { film_id: 1002, stars: 4 }, db.constraint('ratings_stars_key')); // error TS2345 ``` - The type takes every unique index of the table, and Postgres only a constraint. An index made with `CREATE UNIQUE INDEX`, which is no constraint, type-checks, and Postgres refuses it: 42704, `constraint "…" for table "…" does not exist`. - For such an index, pass its columns as the `conflictTarget`: `'code'` for a unique index on `code`. The type does not check that a constraint belongs to the upsert's table, and Postgres refuses another table's: 42704. Only inside the call does `constraint()` take the table's names, as above. Held in a variable, `const k = db.constraint('tags_pkey')` takes any table's name, so `db.upsert('ratings', …, k)` compiles. Give the table, `db.constraint<'ratings'>('ratings_pkey')`, and the type checks the name against it: ```ts const k = db.constraint('tags_pkey'); db.upsert('ratings', { film_id: 1002, stars: 4 }, k); // compiles, then 42704 db.constraint<'ratings'>('tags_pkey'); ``` ## `Constraint` The class that `constraint()` returns, its constraint name in `value`: `new db.Constraint<'ratings'>('ratings_pkey')` is the same. ## `doNothing` An empty array, for the `updateColumns` of `upsert()`: a row that conflicts is left as it is, and returns nothing. You write: ```ts const tag = await db.upsert('tags', { film_id: 1003, tag: 'coastal', }, ['film_id', 'tag'], { updateColumns: db.doNothing, }).run(pool); ``` Postgres runs: ```sql INSERT INTO "tags" ("film_id", "tag") VALUES ($1, $2) ON CONFLICT ("film_id", "tag") DO NOTHING RETURNING to_jsonb("tags".*) || jsonb_build_object('$action', CASE xmax WHEN 0 THEN 'INSERT' ELSE 'UPDATE' END) AS result ``` Values: `[1003, "coastal"]` TypeScript knows: ```ts const tag: db.UpsertReturnableForTable<"tags", db.ColumnsOption<"tags">, db.ExtrasOption<"tags">, "suppress" | undefined> | undefined // that is const tag: { film_id: number; tag: string; $action: "INSERT" | "UPDATE"; } | undefined ``` - One row: `undefined` when it conflicts, and the type says so. - An array: the rows inserted only; `[]` when every row conflicts. ## `UpsertAction` `{ $action: 'INSERT' | 'UPDATE' }`, added to each result of `upsert()`: `'INSERT'` for a new row, `'UPDATE'` for an updated one, read from the row's `xmax`. `reportAction: 'suppress'` leaves it out of the SQL and of the type. ## `UpsertReturnableForTable` `UpsertReturnableForTable`: one row of what `upsert()` returns, the [`ReturningTypeForTable`](https://pgcast.dev/insert#ReturningTypeForTable) with [`UpsertAction`](https://pgcast.dev/insert#UpsertAction), unless `RA`, the `reportAction`, is `'suppress'`: `db.UpsertReturnableForTable<'ratings', undefined, undefined, undefined>` is `ratings.JSONSelectable & UpsertAction`. It does not add the `undefined` of `updateColumns: []` ([`doNothing`](https://pgcast.dev/insert#doNothing)). It needs `strictNullChecks`: without it, it is the whole `JSONSelectable` with `UpsertAction`, whatever `C`, `E` and `RA`. ## `update()` ```ts db.update(table, values, where, options?) ``` Update the rows that match, and return them. The arguments come in the order of the SQL: `SET`, then `WHERE`. - `table`: a table of the schema. - `values`: an `Updatable`, with at least one key. A `sql` fragment is written as SQL, where `db.self` is the column, and so is a condition such as [`conditions.add()`](https://pgcast.dev/conditions#conditions.add). `raw()`, `cols()`, `vals()` and `parent()` throw, as for [`insert()`](https://pgcast.dev/insert#insert). An `undefined` value is `NULL` ([An `undefined` value](https://pgcast.dev/insert#undefined-values)). - `where`: a `Whereable`, or a `sql` fragment ([Conditions](https://pgcast.dev/conditions)). - `options`: `returning` and `extras`, as for [`insert()`](https://pgcast.dev/insert#insert-options). You write: ```ts const screenings = await db.update('screenings', { starts_at: db.sql`${db.self} + interval '15 minutes'`, subtitled: true, }, { id: 1002 }).run(pool); ``` Postgres runs: ```sql UPDATE "screenings" SET ("starts_at", "subtitled") = ROW("starts_at" + interval '15 minutes', $1) WHERE ("id" = $2) RETURNING to_jsonb("screenings".*) AS result ``` Values: `[true, 1002]` TypeScript knows: ```ts const screenings: screenings.JSONSelectable[] // that is const screenings: { cinema_id: number; film_id: number; id: number; starts_at: db.TimestampTzString; subtitled: boolean | null; }[] ``` - Always an array: every row updated, `[]` when none matches. - `SET (…) = ROW(…)`: `ROW()` keeps a single column valid since Postgres 10. - `values` with no key throws: "nothing to update: an Updatable needs at least one column". Zapatos 6.6.1 sent `SET () = ROW()`, which Postgres refused (42601). > **Every row** > > An empty `Whereable`, `{}`, compiles to `WHERE TRUE`: `update()` updates every row of the table, and `deletes()` deletes them all. An empty list in [`and(...filters)` or `not(or(...ids))`](https://pgcast.dev/conditions#conditions.and) reaches every row too. ## `deletes()` ```ts db.deletes(table, where, options?) ``` Delete the rows that match, and return them. `delete` is a reserved word in JavaScript, hence the name. - `table`: a table of the schema. - `where`: a `Whereable`, or a `sql` fragment ([Conditions](https://pgcast.dev/conditions)); `{}` is every row, as for [`update()`](https://pgcast.dev/insert#update). - `options`: `returning` and `extras`, as for [`insert()`](https://pgcast.dev/insert#insert-options). You write: ```ts const removed = await db.deletes('tags', { film_id: 1003, }, { returning: ['tag'] }).run(pool); ``` Postgres runs: ```sql DELETE FROM "tags" WHERE ("film_id" = $1) RETURNING jsonb_build_object($2::text, "tag") AS result ``` Values: `[1003, "tag"]` TypeScript knows: ```ts const removed: db.JSONOnlyColsForTable<"tags", "tag"[]>[] // that is const removed: { tag: string; }[] ``` - Always an array: every row deleted, `[]` when none matches. ## `DeleteSignatures` The interface of `deletes()`'s call signature, `(table, where, options?) => SQLFragment<…[]>`, for a function that wraps it and keeps its types. A wrapper typed with it gets the types of its parameters from it, with no cast, and its calls are checked and typed as those of `deletes()`: ```ts const loggedDeletes: db.DeleteSignatures = (table, where, options) => { const query = db.deletes(table, where, options); console.log(query.compile().text); return query; }; const removed = loggedDeletes('tags', { film_id: 1003 }, { returning: ['tag'] }); ``` ## `truncate()` ```ts db.truncate(table, ...options) ``` Empty one table, or several in one statement. - `table`: a table of the schema, or an array of them. - `options`: `'CONTINUE IDENTITY'` or `'RESTART IDENTITY'`, then `'RESTRICT'` or `'CASCADE'`, each optional. You write: ```ts const emptied = await db.truncate(['seat_bookings', 'ratings', 'tags'], 'RESTART IDENTITY').run(pool); ``` Postgres runs: ```sql TRUNCATE "seat_bookings", "ratings", "tags" RESTART IDENTITY ``` Values: `[]` TypeScript knows: ```ts const emptied: undefined ``` | Option | SQL it adds | Effect on the type | |---|---|---| | none | `TRUNCATE "tags"` | `undefined` | | `'CONTINUE IDENTITY'` | `CONTINUE IDENTITY`: sequences keep their values, as by default | none | | `'RESTART IDENTITY'` | `RESTART IDENTITY`: the tables' sequences restart | none | | `'RESTRICT'` | `RESTRICT`: Postgres refuses if a table not in the list references one of them, as by default | none | | `'CASCADE'` | `CASCADE`: also empties the tables that reference them | none | With `'CASCADE'`, Postgres empties every table that references one of them, at any depth: `films` takes `roles`, `screenings`, `ratings` and `tags` with it, and `seat_bookings` through `screenings`, each named in a notice, `truncate cascades to table "roles"`: You write: ```ts const emptied = await db.truncate('films', 'CASCADE').run(pool); ``` Postgres runs: ```sql TRUNCATE "films" CASCADE ``` Values: `[]` TypeScript knows: ```ts const emptied: undefined ``` - `.run()` resolves with `[]`, where the type says `undefined`. - The type takes the options in that order, one of each kind. pgcast writes them as given: in JavaScript, or past the type with `as any`, `truncate('tags', 'CASCADE', 'RESTART IDENTITY')` sends `TRUNCATE "tags" CASCADE RESTART IDENTITY`, and Postgres refuses it (42601). - Any other option throws: "Truncate option must be CONTINUE IDENTITY/RESTART IDENTITY/RESTRICT/CASCADE, not '…'". Zapatos 6.6.1 wrote it into the SQL as it was. ## An `undefined` value In the values of `insert()`, `update()` and `upsert()`, Zapatos 6.6.1 sends `undefined` as `NULL`: a `NOT NULL DEFAULT` column fails, and `update()` erases the column instead of leaving it. | In | By default, as in 6.6.1 | With `undefinedMeansMissing: true` | |---|---|---| | `insert()` | `NULL` | `DEFAULT`: `VALUES ($1, $2, $3, DEFAULT, $4)` | | `update()` | sets the column to `NULL`, and counts as a key | left out: `SET ("title") = ROW($1)`; with no key left, it throws "nothing to update…" | | `upsert()` | inserts `NULL`, then sets it from `EXCLUDED` | left out: not inserted, nor updated unless `updateColumns` names it (then from `EXCLUDED`, so an existing row takes the column's `DEFAULT`); in several rows, its cell is `DEFAULT` and counts as missing ([Rows with different keys](https://pgcast.dev/insert#upsert-different-keys)) | - By default, each sends the SQL of 6.6.1 and warns once per process: `PGCAST_DEP002`. - An `undefined` in a `where` is unchanged. - An `undefined` in `updateValues` is unchanged too, in both modes: its key is still updated, from `EXCLUDED`. So an existing row takes the value inserted, or the column's `DEFAULT` when the row doesn't insert it, with no warning: `upsert('ratings', { film_id: 1002, stars: 4 }, ['film_id', 'stars'], { updateValues: { votes: undefined } })` sends `EXCLUDED."votes"`, and an existing count goes back to 1. > **Opt-in** > > `undefinedMeansMissing: true` ([`Config`](https://pgcast.dev/runtime-config#Config.undefinedMeansMissing)) reads an `undefined` value as a missing key, as `JSON.stringify()` does. Without it, pgcast warns as `PGCAST_DEP002`; it becomes the default in 7.0 ([Versions](https://pgcast.dev/versions)). ## `ReturningOptionsForTable` `{ returning?, extras? }`: the options of `insert()`, `upsert()`, `update()` and `deletes()` that shape each result ([Options](https://pgcast.dev/insert#insert-options)). ## `ReturningTypeForTable` `ReturningTypeForTable`: one row of what `insert()`, `update()` and `deletes()` return, for the table `T`, the `returning` columns `C` and the `extras` `E` (`undefined` for an option not given): the `JSONSelectable`, narrowed to `C`, with the keys of `E`. `insert()` of one row returns it; `insert()` of an array, `update()` and `deletes()` return an array of it. It types a fake result, in a test, from the schema: ```ts type NewFilm = db.ReturningTypeForTable<'films', ['id', 'title'], undefined>; const inserted: NewFilm = { id: 1000, title: 'The Long Take' }; ``` - Stable: type tests pin it to what `.run()` gives, here for `db.insert('films', film, { returning: ['id', 'title'] })`. - `C` may be a `readonly` tuple, as `returning` may ([Options](https://pgcast.dev/insert#insert-options)). - It needs `strictNullChecks`: without it, it is the whole `JSONSelectable`, whatever `C` and `E`. --- # Conditions Source: https://pgcast.dev/conditions `db.conditions` holds the tests, comparisons and operators you give as the value of a Whereable's key, each a `sql` fragment with that column on its left. This page lists each one, the SQL it writes and the values it takes. ## `conditions` ```ts db.conditions.eq(value) // or: import { conditions as dc } from 'zapatos/db' ``` Each member is a [`sql`](https://pgcast.dev/sql-template#sql) fragment written around [`db.self`](https://pgcast.dev/sql-template#self), or a function that returns one. As the value of a key of a [Whereable](https://pgcast.dev/sql-template#whereable), `self` is that key's column, and the condition sits in its own parentheses inside the Whereable's: `{ release_year: db.conditions.gte(2000) }` gives `(("release_year" >= $1))`. You write: ```ts const films = await db.select('films', { release_year: db.conditions.between(1990, 2010), title: db.conditions.like('The %'), }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("films".*) AS result FROM "films" WHERE (("release_year" BETWEEN ($1) AND ($2)) AND ("title" LIKE $3)) ) AS "sq_films" ``` Values: `[1990, 2010, "The %"]` TypeScript knows: ```ts const films: films.JSONSelectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` - **Where**: as a Whereable value, so in the `where` of [`select()`](https://pgcast.dev/select#select), [`count()`](https://pgcast.dev/select#count), [`update()`](https://pgcast.dev/insert#update) and [`deletes()`](https://pgcast.dev/insert#deletes), the `having` of `select()`, and a Whereable in a `sql` template. Elsewhere `self` has no column: `` db.sql`SELECT ${db.conditions.isNull}` `` throws "The 'self' column alias has no meaning here" as it compiles. - [`and()`](https://pgcast.dev/conditions#conditions.and), [`or()`](https://pgcast.dev/conditions#conditions.or) and [`not()`](https://pgcast.dev/conditions#conditions.not) also take Whereables, so they can be a whole `where`; [`now`](https://pgcast.dev/conditions#conditions.now) and [`fromNow()`](https://pgcast.dev/conditions#conditions.fromNow) hold no `self`, and go anywhere. - **Values**: each value is a parameter, `$1`, except a [`Parameter`](https://pgcast.dev/sql-template#Parameter), kept as it is with its cast; a `sql` fragment, written as its SQL; and [`parent()`](https://pgcast.dev/sql-template#parent), the parent query's column in a `lateral` subquery. `Default`, `self`, `raw()`, `cols()` and `vals()` throw: [Values a condition refuses](https://pgcast.dev/conditions#refused-values). - **Type**: a condition is a `SQLFragment` whose second type argument, `T`, is the type of the value it compares with (its `Constraint`, [`sql`](https://pgcast.dev/sql-template#sql)); the Whereable checks `T` against the column's type. The eight tests, `now` and `fromNow()` have no `T`, and pass on any column. A value of another type does not compile: ```ts db.select('films', { release_year: db.conditions.eq('1991') }); // error TS2322 ``` With `strictNullChecks` (or `strict`) on, the type also refuses `null` as a value, in `eq()`, `isIn()` and the others; off, `eq(null)` compiles to `"col" = $1`, which matches no row. The test for `NULL` is [`isNull`](https://pgcast.dev/conditions#conditions.isNull). ## Tests: null, true, false, unknown Constants, not functions: `{ subtitled: db.conditions.isNull }`, no `()`. Each writes the column, then its test. | Test | Matches a `boolean` that is | |---|---| | `isNull`, `isUnknown` | `NULL` | | `isNotNull`, `isNotUnknown` | `true` or `false` | | `isTrue` | `true` | | `isNotTrue` | `false` or `NULL` | | `isFalse` | `false` | | `isNotFalse` | `true` or `NULL` | You write: ```ts const screenings = await db.select('screenings', { subtitled: db.conditions.isNotTrue, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("screenings".*) AS result FROM "screenings" WHERE (("subtitled" IS NOT TRUE)) ) AS "sq_screenings" ``` Values: `[]` TypeScript knows: ```ts const screenings: screenings.JSONSelectable[] // that is const screenings: { cinema_id: number; film_id: number; id: number; starts_at: db.TimestampTzString; subtitled: boolean | null; }[] ``` `isNull` and `isNotNull` work on any column. The six others are for a `boolean`: on another type Postgres refuses them (42804), though the type lets them through. ### `isNull` `"col" IS NULL`. ### `isNotNull` `"col" IS NOT NULL`. ### `isTrue` `"col" IS TRUE`. ### `isNotTrue` `"col" IS NOT TRUE`. ### `isFalse` `"col" IS FALSE`. ### `isNotFalse` `"col" IS NOT FALSE`. ### `isUnknown` `"col" IS UNKNOWN`: as `isNull`, for a `boolean`. ### `isNotUnknown` `"col" IS NOT UNKNOWN`: as `isNotNull`, for a `boolean`. ## Comparisons Each takes one value of the column's type, `(a: T)`, and writes the column, its operator and the value. You write: ```ts const films = await db.select('films', { release_year: db.conditions.gte(2000), studio_id: db.conditions.ne(1001), }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("films".*) AS result FROM "films" WHERE (("release_year" >= $1) AND ("studio_id" <> $2)) ) AS "sq_films" ``` Values: `[2000, 1001]` TypeScript knows: ```ts const films: films.JSONSelectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` `ne()` and the four orderings match no row whose column is `NULL`; `isDistinctFrom()` does: on `screenings`, `{ subtitled: ne(true) }` finds the `false` row, `{ subtitled: isDistinctFrom(true) }` the `false` row and the two `NULL`s. ### `eq()` `"col" = $1`. ### `ne()` `"col" <> $1`. ### `gt()` `"col" > $1`. [`after()`](https://pgcast.dev/conditions#conditions.after) is the same function. ### `gte()` `"col" >= $1`. ### `lt()` `"col" < $1`. [`before()`](https://pgcast.dev/conditions#conditions.before) is the same function. ### `lte()` `"col" <= $1`. ### `isDistinctFrom()` `"col" IS DISTINCT FROM $1`: as `ne()`, with `NULL` a value like any other, so it is never unknown. ### `isNotDistinctFrom()` `"col" IS NOT DISTINCT FROM $1`: as `eq()`, with `NULL` a value like any other. With `strictNullChecks` on, the type refuses `null` here too: [`isNull`](https://pgcast.dev/conditions#conditions.isNull) is the test for it. ## Ranges Each takes two values of the column's type, `(a: T, b: T)`, each a parameter in its own parentheses. Both bounds belong to the range: `between()` matches them, `notBetween()` does not. You write: ```ts const films = await db.select('films', { release_year: db.conditions.betweenSymmetric(2010, 1990), }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("films".*) AS result FROM "films" WHERE (("release_year" BETWEEN SYMMETRIC ($1) AND ($2))) ) AS "sq_films" ``` Values: `[2010, 1990]` TypeScript knows: ```ts const films: films.JSONSelectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` ### `between()` `"col" BETWEEN ($1) AND ($2)`. `a` is the low bound: `between(2010, 1990)` matches no row. ### `betweenSymmetric()` `"col" BETWEEN SYMMETRIC ($1) AND ($2)`: the bounds in either order. ### `notBetween()` `"col" NOT BETWEEN ($1) AND ($2)`. With the bounds reversed, every row whose column is not `NULL`. ### `notBetweenSymmetric()` `"col" NOT BETWEEN SYMMETRIC ($1) AND ($2)`: the bounds in either order. ## Patterns Each takes one pattern, `(a: T)`, so only on a column whose type is a string: on `release_year`, the type refuses them (TS2322). | Family | Syntax | Matches | |---|---|---| | `like()`, `ilike()` | `%` any text, `_` one character | the whole value | | `similarTo()` | Postgres's SQL regular expressions | the whole value | | `reMatch()`, `reImatch()` | POSIX regular expressions | any part of the value | So `like('Lantern')` matches no title, and `reMatch('Lantern')` matches `A Lantern for Edith`. The `i` forms ignore case. You write: ```ts const films = await db.select('films', { title: db.conditions.reImatch('^(the|a) '), }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("films".*) AS result FROM "films" WHERE (("title" ~* $1)) ) AS "sq_films" ``` Values: `["^(the|a) "]` TypeScript knows: ```ts const films: films.JSONSelectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` ### `like()` `"col" LIKE $1`. ### `notLike()` `"col" NOT LIKE $1`. ### `ilike()` `"col" ILIKE $1`. ### `notIlike()` `"col" NOT ILIKE $1`. ### `similarTo()` `"col" SIMILAR TO $1`. ### `notSimilarTo()` `"col" NOT SIMILAR TO $1`. ### `reMatch()` `"col" ~ $1`. ### `reImatch()` `"col" ~* $1`. ### `notReMatch()` `"col" !~ $1`. ### `notReImatch()` `"col" !~* $1`. ## Lists Each takes an array of values of the column's type, `(a: readonly T[])`, compiled as [`vals()`](https://pgcast.dev/sql-template#vals) of it: one parameter per item. You write: ```ts const films = await db.select('films', { id: db.conditions.isIn([1000, 1002]), }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("films".*) AS result FROM "films" WHERE (("id" IN ($1, $2))) ) AS "sq_films" ``` Values: `[1000, 1002]` TypeScript knows: ```ts const films: films.JSONSelectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` An item compiles as in `vals([…])` within a filter: | Item | Compiles to | |---|---| | a plain value | a parameter, `$1` | | a `Parameter` | as it is, with its cast: `isIn([db.param('x', 'text'), 'y'])` gives `IN (CAST($1 AS "text"), $2)` | | a `sql` fragment | its SQL | | `parent(c?)` | the parent column, in a `lateral` subquery: `IN ("films"."id")` | | `Default`, `self`, `raw()`, `cols()`, `vals()` | throws ([Values a condition refuses](https://pgcast.dev/conditions#refused-values)) | Zapatos 6.6.1 sent a `Parameter` and `parent()` there as one more parameter, as JSON (`'{"value":"x","cast":"text"}'`, `'{"value":"id"}'`), so `isIn()` found nothing and `isNotIn()` matched every row on a text column. ### `isIn()` `"col" IN ($1, $2)`. An empty array gives `false`: no row. ### `isNotIn()` `"col" NOT IN ($1, $2)`. An empty array gives `true`: every row. A `NULL` in the list, as Postgres reads `NOT IN`, makes it match no row; the type refuses `null` there with `strictNullChecks` on. You write: ```ts const films = await db.select('films', { studio_id: db.conditions.isNotIn([1001]), }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("films".*) AS result FROM "films" WHERE (("studio_id" NOT IN ($1))) ) AS "sq_films" ``` Values: `[1001]` TypeScript knows: ```ts const films: films.JSONSelectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` ## Arrays Each takes an array, sent as one parameter, or [`parent()`](https://pgcast.dev/sql-template#parent): `(a: T[] | ParentColumn)`. They compare the elements, whatever the dimensions: `{drama,romance}` overlaps `{{romance,thriller}}`. `T` is the element, as in Zapatos 6.6.1, where the Whereable expects the column's array type: on `genres`, a `text[]`, the type refuses `arrayOverlaps(['romance'])`. ```ts db.select('films', { genres: db.conditions.arrayOverlaps(['romance']) }); ``` It stays that way: every type I tried on the column broke some code that compiles under 6.6.1, either a mock written to its signature or the typed result of `.map(db.conditions.arrayContains)` ([#40](https://github.com/AlexandreDemartin/pgcast/issues/40)). Two ways compile. Given the column's type as `T`, the value is an array of arrays: You write: ```ts const films = await db.select('films', { genres: db.conditions.arrayOverlaps([['romance', 'thriller']]), }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("films".*) AS result FROM "films" WHERE (("genres" && $1)) ) AS "sq_films" ``` Values: `[[["romance","thriller"]]]` TypeScript knows: ```ts const films: films.JSONSelectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` Or write a `sql` fragment value, where `db.self` is the column, and cast its parameter: `` db.sql`${db.self} @> ${db.param(['drama'], 'text[]')}` `` gives `"col" @> CAST($1 AS "text"[])`. You write: ```ts const dramas = await db.select('films', { genres: db.sql`${db.self} @> ${db.param(['drama'], 'text[]')}`, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("films".*) AS result FROM "films" WHERE (("genres" @> CAST($1 AS "text"[]))) ) AS "sq_films" ``` Values: `[["drama"]]` TypeScript knows: ```ts const dramas: films.JSONSelectable[] // that is const dramas: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` With [`castArrayParamsToJson`](https://pgcast.dev/runtime-config#Config.castArrayParamsToJson) on, a condition sends its array as `CAST($1 AS "json")`, an array of arrays too, which Postgres refuses (42883: `text[] @> json`). The condition's type refuses a `Parameter` (TS2345), but in the fragment, the cast given to `param()` wins over the option ([#36](https://github.com/AlexandreDemartin/pgcast/issues/36)): it runs either way. ### `arrayContains()` `"col" @> $1`: the column holds every element of the value. ### `arrayContainedIn()` `"col" <@ $1`: every element of the column is in the value. ### `arrayOverlaps()` `"col" && $1`: they share at least one element. ## Combining: `and()`, `or()`, `not()` Each takes conditions or Whereables, a Whereable compiled with its own keys. As a Whereable value, `self` reaches the conditions inside: `{ release_year: or(lt(1980), gt(2010)) }` gives `((("release_year" < $1 OR "release_year" > $2)))`. As a whole `where`, they take Whereables: You write: ```ts const films = await db.select('films', db.conditions.or( { studio_id: 1002 }, { release_year: db.conditions.lt(1980) }, )).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("films".*) AS result FROM "films" WHERE (("studio_id" = $1) OR (("release_year" < $2))) ) AS "sq_films" ``` Values: `[1002, 1980]` TypeScript knows: ```ts const films: films.JSONSelectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` ### `and()` ```ts db.conditions.and(...conditions: (SQLFragment | Whereable)[]) ``` `((a) AND (b))`: each operand in its own parentheses, joined by `AND`. - A `sql` fragment with a bare `OR` at its top level stays whole: `` and(db.sql`${'id'} = 1000 OR ${'id'} = 1001`, { studio_id: 1001 }) `` gives `(("id" = 1000 OR "id" = 1001) AND (("studio_id" = $1)))`, film 1001 alone. - With no argument, `TRUE`: every row, as a Whereable `{}`. Zapatos 6.6.1 wrote `()`, which Postgres refuses (42601). > **Every row** > > `update()` and `deletes()` given `and(...filters)` or `not(or(...ids))` with an empty list now reach every row: `deletes('films', and())` sends `WHERE TRUE` and deletes them all, `not(or())` gives `(NOT (FALSE))`. Zapatos 6.6.1 threw (42601) before any row changed. Keep the length check where an empty list must not reach every row. Zapatos 6.6.1 wrote the operands bare, `(a OR b AND c)`, which Postgres reads `a OR (b AND c)`: the example above found film 1000 too, whatever its studio, and `deletes()` or `update()` given it deleted or updated that row. Only an `and()` of two operands or more, one with a bare `OR` at its top level, now gives other rows. One whose SQL ends in the next operand: - now fails (42601), `` and(db.sql`${'release_year'} BETWEEN 1970`, db.sql`2000`) ``; - or, when the split falls inside a quoted literal, compares with another value, with no error: `` and(db.sql`${'title'} = 'x`, db.sql`y'`) `` compares `"title"` with `'x) AND (y'`, where it was `'x AND y'`. Any other gains parentheses and gives the same rows. ### `or()` ```ts db.conditions.or(...conditions: (SQLFragment | Whereable)[]) ``` `(a OR b)`: its operands joined by `OR`, as they are: `AND` binds tighter than `OR`, so none reaches across it. With no argument, `FALSE`: no row. Zapatos 6.6.1 wrote `()` here too. ### `not()` ```ts db.conditions.not(condition: SQLFragment | Whereable) ``` `(NOT (…))`: the whole condition negated. You write: ```ts const films = await db.select('films', { studio_id: db.conditions.not(db.conditions.isIn([1000, 1001])), }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("films".*) AS result FROM "films" WHERE (((NOT ("studio_id" IN ($1, $2))))) ) AS "sq_films" ``` Values: `[1000, 1001]` TypeScript knows: ```ts const films: films.JSONSelectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` On `release_year`, `` not(db.sql`${db.self} = 1974 OR ${db.self} = 1991`) `` writes `(NOT ("release_year" = 1974 OR "release_year" = 1991))`. Zapatos 6.6.1 wrote `(NOT "release_year" = 1974 OR "release_year" = 1991)`, which Postgres reads `(NOT "release_year" = 1974) OR "release_year" = 1991`: the 1991 film too. Only a condition with a bare `AND` or `OR` at its top level changes rows; any other gains one pair of parentheses. ## Dates and times [`after()`](https://pgcast.dev/conditions#conditions.after) and [`before()`](https://pgcast.dev/conditions#conditions.before) compare with a value: a `Date`, an ISO 8601 string with its offset (`'2030-03-01T20:00:00Z'`, a `TimestampTzString`; the type refuses another form such as `'2030-03-02'`, TS2322), or one of the two fragments below. `pg` sends a `Date` as a string in the local time zone, with its offset: on a `timestamptz` column, the same instant. You write: ```ts const screenings = await db.select('screenings', { starts_at: db.conditions.after(new Date('2030-03-02T00:00:00Z')), }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("screenings".*) AS result FROM "screenings" WHERE (("starts_at" > $1)) ) AS "sq_screenings" ``` Values: `["2030-03-02T00:00:00.000Z"]` TypeScript knows: ```ts const screenings: screenings.JSONSelectable[] // that is const screenings: { cinema_id: number; film_id: number; id: number; starts_at: db.TimestampTzString; subtitled: boolean | null; }[] ``` ### `after()` `gt()` under another name: `"col" > $1`. ### `before()` `lt()` under another name: `"col" < $1`. ### `now` `now()`: a constant, the start of the current transaction. A value, not a condition: `{ starts_at: after(now) }` gives `(("starts_at" > now()))`, and in `update()` it sets the column. Alone as a Whereable value, `{ starts_at: now }`, Postgres refuses it (42804), though the type lets it through. ### `fromNow()` ```ts db.conditions.fromNow(n: number, unit: IntervalUnit = 'millisecond') ``` `now() + $1`, the value `` `${n} ${unit}` ``, which Postgres reads as an interval: `fromNow(-7, 'days')` sends `'-7 days'`, `fromNow(1.5, 'hours')` `'1.5 hours'`. A value, as [`now`](https://pgcast.dev/conditions#conditions.now). `unit` is one of `microsecond`, `millisecond`, `second`, `minute`, `hour`, `day`, `week`, `month`, `year`, `decade`, each with or without an `s`, then `century`, `centuries`, `millennium`, `millennia`. An `n` whose string Postgres does not read (`NaN`, `1e21`, written `1e+21`) fails with 22007. The screenings of the next seven days, from `now()`, here the time of the query: You write: ```ts const thisWeek = await db.select('screenings', { starts_at: db.conditions.and( db.conditions.after(db.conditions.now), db.conditions.before(db.conditions.fromNow(7, 'days')), ), }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("screenings".*) AS result FROM "screenings" WHERE (((("starts_at" > now()) AND ("starts_at" < now() + $1)))) ) AS "sq_screenings" ``` Values: `["7 days"]` TypeScript knows: ```ts const thisWeek: screenings.JSONSelectable[] // that is const thisWeek: { cinema_id: number; film_id: number; id: number; starts_at: db.TimestampTzString; subtitled: boolean | null; }[] ``` ## Arithmetic: `add()`, `subtract()` ```ts db.conditions.add(a: T) db.conditions.subtract(a: T) ``` Operations, not tests: for an [`update()`](https://pgcast.dev/insert#update) value, where `self` is the column it sets. You write: ```ts const ratings = await db.update('ratings', { votes: db.conditions.add(1), }, { film_id: 1000, stars: 5 }).run(pool); ``` Postgres runs: ```sql UPDATE "ratings" SET ("votes") = ROW("votes" + $1) WHERE ("film_id" = $2 AND "stars" = $3) RETURNING to_jsonb("ratings".*) AS result ``` Values: `[1, 1000, 5]` TypeScript knows: ```ts const ratings: ratings.JSONSelectable[] // that is const ratings: { film_id: number; stars: number; votes: number; }[] ``` - In a `where`, the type lets them through, and Postgres refuses them (42804: "argument of WHERE must be type boolean"). - In `upsert()`'s `updateValues`, there is no current column: they throw "The 'self' column alias has no meaning here" ([`upsert()`](https://pgcast.dev/insert#upsert)). In `insert()`, the column is written, and Postgres refuses it (42703). - They take a number or a `Date`, not a `Parameter` (TS2345): for a cast, write a `sql` fragment value (below). - On a `date` column, the type refuses a number (TS2322), and Postgres could not run it either: `date + unknown` is ambiguous (42725). - On a `date`, `timestamp` or `timestamptz` column, a `Date` type-checks, and Postgres refuses it ([#180](https://github.com/AlexandreDemartin/pgcast/issues/180)). On `date`, `add()` is ambiguous (42725) and `subtract()` gives an integer, not a date (42804); on the other two, `add()` reads it as an interval (22007), `subtract()` gives an interval, not a time (42804). For dates, write a `sql` fragment value, where `db.self` is the column, and cast its parameter. On a `date` column, `` db.sql`${db.self} + ${db.param(3, 'int4')}` `` adds three days: `"col" + CAST($1 AS "int4")`. On a `timestamptz` column, add an interval: a screening moved to the next day. You write: ```ts const postponed = await db.update('screenings', { starts_at: db.sql`${db.self} + ${db.param('1 day', 'interval')}`, }, { id: 1000 }).run(pool); ``` Postgres runs: ```sql UPDATE "screenings" SET ("starts_at") = ROW("starts_at" + CAST($1 AS "interval")) WHERE ("id" = $2) RETURNING to_jsonb("screenings".*) AS result ``` Values: `["1 day", 1000]` TypeScript knows: ```ts const postponed: screenings.JSONSelectable[] // that is const postponed: { cinema_id: number; film_id: number; id: number; starts_at: db.TimestampTzString; subtitled: boolean | null; }[] ``` The type does not check a fragment: an `int4` on a `timestamptz` column type-checks, and Postgres refuses it (42883). ### `add()` `"col" + $1`. ### `subtract()` `"col" - $1`. ## Values a condition refuses `Default`, `self`, `raw()`, `cols()` and `vals()` have no meaning as the value of a condition. The type refuses them (TS2322, or TS2345 where the condition bounds its argument: a pattern, a range's second bound, an array condition, `add()`, `subtract()`). At run time, calling the condition does not throw, compiling it does, so `run()` rejects with a plain `Error` before any query. Zapatos 6.6.1 sent `Default` and `self` as the strings `'Symbol(DEFAULT)'` and `'Symbol(self)'`, and `raw()`, `cols()` and `vals()` as JSON, `'{"value":…}'`: no row on a text column, every row with `ne()` or `isNotIn()`, 22P02 or 22007 on the others. Why a `raw()` value is not SQL: [Security](https://pgcast.dev/security). | Message | When | |---|---| | `` `db.Default` has no meaning as the value of `eq()`, `add()` and the other conditions: compare with a value. To set a column to its default in an UPDATE, give `db.Default` as the value itself: `{ n: db.Default }`. `` | `Default` as the value of a comparison, a range, a pattern, an array condition, `add()` or `subtract()` | | `` `db.self` has no meaning as the value of `eq()`, `add()` and the other conditions: it is already the column on the left. To compare the column with itself, write a `sql` fragment with `db.self` on both sides. `` | `self` there | | `` `db.raw()`, `db.cols()` and `db.vals()` have no meaning as the value of `eq()`, `add()` and the other conditions: for SQL, write a `sql` fragment with `db.self` in place of the condition; for a value, give the value itself, or `db.param()`. `` | `raw()`, `cols()` or `vals()` there | | `` `db.Default` has no meaning in a Whereable (a filter): leave the key out, or compare with a value. `DEFAULT` is for INSERT and UPDATE values. `` | `Default` in `isIn()` or `isNotIn()` | | `` `db.self` has no meaning as a value in a Whereable (a filter), nor in `isIn()`: compare with a value, or use it in a `sql` fragment value, where it stands for the column. `` | `self` in `isIn()` or `isNotIn()` | | `` `db.raw()`, `db.cols()` and `db.vals()` have no meaning as a value in a Whereable (a filter), nor in `isIn()`: to compare with SQL, use a `sql` fragment value, where `db.self` stands for the column; to compare with a value, give the value itself, or `db.param()`. `` | `raw()`, `cols()` or `vals()` in `isIn()` or `isNotIn()` | | `The 'self' column alias has no meaning here` | a condition where `self` has no column: in a `sql` template outside a Whereable, in `upsert()`'s `updateValues` | | `The 'parent' table alias has no meaning here` | `parent()` in a condition outside a `lateral` subquery | A comparison with SQL is a `sql` fragment value, `` { title: db.sql`${db.self} = ${db.raw("'x'")}` } ``: [Raw SQL](https://pgcast.dev/raw-sql). --- # Transactions Source: https://pgcast.dev/transaction `transaction()` and its seven shortcuts, one per isolation level: the SQL they send, their retries and the types of the client they give. Then the errors: `isDatabaseError()` for those from Postgres, and those pgcast throws before any query. **At a glance** - Only the queries you `.run(txnClient)` are in the transaction; one run on the pool is not. - On 40001 or 40P01, the whole callback runs again, five attempts in all by default: write it so it can. - A query error that the callback catches turns `COMMIT` into `ROLLBACK`, and nothing is saved: let the error out. ## `transaction()` ```ts db.transaction(txnClientOrQueryable, isolationLevel, callback) ``` Runs `callback` in one transaction, then commits it, retries it or rolls it back. It returns a promise of what `callback` resolves to. - `txnClientOrQueryable`: a `pg.Pool`, from which it checks out a client, released at the end; a client, `pg.Client`, `pg.native.Client` or a `pg.PoolClient` already checked out, used and left connected; or the client of a transaction already open, [passed through](https://pgcast.dev/transaction#passing-a-transaction-through). - `isolationLevel`: an [`IsolationLevel`](https://pgcast.dev/transaction#IsolationLevel). - `callback`: `async txnClient => …`, given a [`TxnClient`](https://pgcast.dev/transaction#TxnClient) of that level. The queries of the transaction are those `.run(txnClient)`; one run on the pool, or on any other client, is not in it. `db.serializable(pool, callback)` is `db.transaction(pool, db.IsolationLevel.Serializable, callback)`. Two customers book seat F7 of screening 1000 at once: ```ts const bookSeat = (seat: string, customer: string) => db.serializable(pool, async txnClient => { const taken = await db.selectOne('seat_bookings', { screening_id: 1000, seat }).run(txnClient); if (taken) return false; await db.insert('seat_bookings', { screening_id: 1000, seat, customer }).run(txnClient); return true; }); const [ines, barnaby] = await Promise.all([ bookSeat('F7', 'Ines Calloway'), bookSeat('F7', 'Barnaby Quill'), ]); ``` One of `ines` and `barnaby` is `true`, the other `false`. When both read before either commits, Postgres refuses one of the two `INSERT`s with 40001, and pgcast runs that callback again, which finds the seat taken. That transaction sends: ```sql -- attempt 1 START TRANSACTION ISOLATION LEVEL SERIALIZABLE -- the callback's selectOne(), then insert(), which Postgres refuses with 40001 ROLLBACK -- attempt 2, 25 to 250 ms later START TRANSACTION ISOLATION LEVEL SERIALIZABLE -- the callback's selectOne(), which returns the other booking COMMIT ``` The SQL of the callback's queries: [`selectOne()`](https://pgcast.dev/select#selectOne) and [`insert()`](https://pgcast.dev/insert#insert). Given a client you checked out, `transaction()` runs on it and leaves it connected; releasing it is yours: ```ts const client = await pool.connect(); let before: number, after: number; try { before = await db.count('seat_bookings', { screening_id: 1000 }).run(client); await db.readCommitted(client, txnClient => db.insert('seat_bookings', { screening_id: 1000, seat: 'G2', customer: 'Tamsin Rowe' }).run(txnClient)); after = await db.count('seat_bookings', { screening_id: 1000 }).run(client); } finally { client.release(); } ``` The two `count()`s run on `client`, outside the transaction: after the block, `before` is 0, `after` 1. ### The SQL it sends | When | SQL | |---|---| | before the callback | `START TRANSACTION ISOLATION LEVEL SERIALIZABLE`: the level's value, written as it is | | the callback resolves | `COMMIT` | | `START TRANSACTION`, the callback or `COMMIT` rejects | `ROLLBACK`, then a retry or the rejection | - These three run on the transaction's client, as the callback's queries do. Each reaches [`queryListener`](https://pgcast.dev/runtime-config#Config.queryListener) and [`resultListener`](https://pgcast.dev/runtime-config#Config.resultListener) with the transaction's id: a counter from 0, one per transaction opened, kept across retries; a transaction passed through keeps the outer one's. - No `SAVEPOINT`: a transaction inside another is the outer one ([Passing a transaction through](https://pgcast.dev/transaction#passing-a-transaction-through)). - After a query error that the callback catches, Postgres answers the `COMMIT` with `ROLLBACK`, not an error, and nothing is saved. `transaction()` resolves, as in Zapatos 6.6.1, and warns once (`PGCAST_DEP003`); with [`transactionRequiresCommit`](https://pgcast.dev/runtime-config#Config.transactionRequiresCommit), it rejects. ### Retries When `START TRANSACTION`, the callback or `COMMIT` rejects with a `code` of 40001 (`serialization_failure`) or 40P01 (`deadlock_detected`), at any level, `transaction()` sends `ROLLBACK`, waits, then starts again: `START TRANSACTION`, the whole callback, `COMMIT`. | [`Config`](https://pgcast.dev/runtime-config#Config) option | Default | What it sets | |---|---|---| | [`transactionAttemptsMax`](https://pgcast.dev/runtime-config#Config.transactionAttemptsMax) | `5` | the attempts in all, the first included; `0` or `1` gives one | | [`transactionRetryDelay`](https://pgcast.dev/runtime-config#Config.transactionRetryDelay) | `{ minMs: 25, maxMs: 250 }` | the wait before each retry: a whole number of ms, at random between the two | - After the last attempt, it rejects with that attempt's error. - Any other rejection: `ROLLBACK`, then it rejects with it, at once. - A `COMMIT` that Postgres answers with `ROLLBACK` is not retried: pgcast doesn't know which error the callback caught. - [`transactionListener`](https://pgcast.dev/runtime-config#Config.transactionListener) gets each message with the transaction's id: | Message | When | |---|---| | ``Transaction rollback (code ${err.code}) on attempt ${attempt} of ${maxAttempts}, retrying in ${delayBeforeRetry}ms`` | before the wait | | ``Retrying transaction, attempt ${attempt} of ${maxAttempts}`` | before `START TRANSACTION` again | | ``Transaction rollback (code ${err.code}) on attempt ${attempt} of ${maxAttempts}, giving up`` | before it rejects | The callback may run several times: [Retries](https://pgcast.dev/retries). ### Passing a transaction through Given the client of a transaction that `transaction()` has open, it sends nothing: it calls the callback with that client, at once, and returns its promise. No `START TRANSACTION`, `COMMIT`, `ROLLBACK` nor retry of its own; the outer transaction does them. So a function can run in its own transaction, or in the caller's: ```ts const bookSeat = (seat: string, customer: string, txnClientOrPool: typeof pool | db.TxnClientForSerializable) => db.serializable(txnClientOrPool, txnClient => db.insert('seat_bookings', { screening_id: 1001, seat, customer }).run(txnClient)); const pair = await db.serializable(pool, txnClient => Promise.all([ bookSeat('G1', 'Wren Abernathy', txnClient), bookSeat('G2', 'Wren Abernathy', txnClient), ])); ``` One transaction, both seats or neither: `START TRANSACTION ISOLATION LEVEL SERIALIZABLE`, the two `INSERT`s, `COMMIT`. - The callback runs at the outer transaction's level, and that level is not checked, at run time or by the type: every `TxnClient` is a [`Queryable`](https://pgcast.dev/sql-template#Queryable), so `db.serializable(txnClient, …)` type-checks with a client of `db.readCommitted()`, and its callback runs in that `READ COMMITTED` transaction. - A function whose parameter is a `TxnClientFor…` type is checked: [`IsolationSatisfying`](https://pgcast.dev/transaction#IsolationSatisfying). - What marks the client is its own `_zapatos` property ([`TxnClient`](https://pgcast.dev/transaction#TxnClient)). ### Errors | Rejects with | When | Since | |---|---|---| | `Isolation level must be an IsolationLevel value (e.g. IsolationLevel.Serializable), not '…'` | `isolationLevel` is not a value of `IsolationLevel`; thrown before a client is checked out or a query sent | pgcast 6.6.2 | | the callback's error | after `ROLLBACK`; for 40001 and 40P01, after the last attempt | Zapatos | | a `TypeError`, `Cannot destructure property 'code' of 'err' as it is null.` (or `undefined`) | the callback throws `null` or `undefined`, after `ROLLBACK` | Zapatos | | "the transaction was rolled back, not committed: a query in it failed and the callback caught the error, so Postgres answered COMMIT with ROLLBACK; let the error out of the callback, or ROLLBACK TO a SAVEPOINT before going on" | with [`transactionRequiresCommit`](https://pgcast.dev/runtime-config#Config.transactionRequiresCommit), Postgres answered the `COMMIT` with `ROLLBACK`; not retried, and no `ROLLBACK` of its own | pgcast 6.7.0 | The type refuses a level that is not an `IsolationLevel` (2345), a string included: ```ts await db.transaction(pool, 'SERIALIZABLE', async txnClient => true); // error TS2345 ``` Zapatos 6.6.1 wrote any value after `ISOLATION LEVEL` as it was, where it could run a second statement ([changelog](https://pgcast.dev/changelog#6.6.2-security)). A transaction passed through is not checked. ## `serializable()` ```ts db.serializable(txnClientOrQueryable, callback) ``` [`transaction()`](https://pgcast.dev/transaction#transaction) at `IsolationLevel.Serializable`: `START TRANSACTION ISOLATION LEVEL SERIALIZABLE`. Its callback gets a [`TxnClientForSerializable`](https://pgcast.dev/transaction#TxnClientForSerializable). ## `repeatableRead()` ```ts db.repeatableRead(txnClientOrQueryable, callback) ``` [`transaction()`](https://pgcast.dev/transaction#transaction) at `IsolationLevel.RepeatableRead`: `START TRANSACTION ISOLATION LEVEL REPEATABLE READ`. Its callback gets a [`TxnClientForRepeatableRead`](https://pgcast.dev/transaction#TxnClientForRepeatableRead). ## `readCommitted()` ```ts db.readCommitted(txnClientOrQueryable, callback) ``` [`transaction()`](https://pgcast.dev/transaction#transaction) at `IsolationLevel.ReadCommitted`: `START TRANSACTION ISOLATION LEVEL READ COMMITTED`. Its callback gets a [`TxnClientForReadCommitted`](https://pgcast.dev/transaction#TxnClientForReadCommitted). ## `serializableRO()` ```ts db.serializableRO(txnClientOrQueryable, callback) ``` [`transaction()`](https://pgcast.dev/transaction#transaction) at `IsolationLevel.SerializableRO`: `START TRANSACTION ISOLATION LEVEL SERIALIZABLE, READ ONLY`. Its callback gets a [`TxnClientForSerializableRO`](https://pgcast.dev/transaction#TxnClientForSerializableRO). Postgres refuses a write in it (25006). ## `repeatableReadRO()` ```ts db.repeatableReadRO(txnClientOrQueryable, callback) ``` [`transaction()`](https://pgcast.dev/transaction#transaction) at `IsolationLevel.RepeatableReadRO`: `START TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY`. Its callback gets a [`TxnClientForRepeatableReadRO`](https://pgcast.dev/transaction#TxnClientForRepeatableReadRO). Postgres refuses a write in it (25006). ## `readCommittedRO()` ```ts db.readCommittedRO(txnClientOrQueryable, callback) ``` [`transaction()`](https://pgcast.dev/transaction#transaction) at `IsolationLevel.ReadCommittedRO`: `START TRANSACTION ISOLATION LEVEL READ COMMITTED, READ ONLY`. Its callback gets a [`TxnClientForReadCommittedRO`](https://pgcast.dev/transaction#TxnClientForReadCommittedRO). Postgres refuses a write in it (25006). ## `serializableRODeferrable()` ```ts db.serializableRODeferrable(txnClientOrQueryable, callback) ``` [`transaction()`](https://pgcast.dev/transaction#transaction) at `IsolationLevel.SerializableRODeferrable`: `START TRANSACTION ISOLATION LEVEL SERIALIZABLE, READ ONLY, DEFERRABLE`. Its callback gets a [`TxnClientForSerializableRODeferrable`](https://pgcast.dev/transaction#TxnClientForSerializableRODeferrable). Postgres refuses a write in it (25006). With `DEFERRABLE`, Postgres may wait for a snapshot, then runs the transaction without the overhead of a serializable one and with no risk of a serialisation failure (40001) ([SET TRANSACTION](https://www.postgresql.org/docs/current/sql-set-transaction.html)): suited to a long report or a backup. A report whose two counts read the same snapshot: ```ts const report = await db.serializableRODeferrable(pool, async txnClient => ({ films: await db.count('films', db.all).run(txnClient), bookings: await db.count('seat_bookings', db.all).run(txnClient), })); ``` `report` is `{ films: 4, bookings: 0 }` on the docs schema. The transaction sends: ```sql START TRANSACTION ISOLATION LEVEL SERIALIZABLE, READ ONLY, DEFERRABLE -- the callback's two count() queries COMMIT ``` The SQL of each query: [`count()`](https://pgcast.dev/select#count). The other read-only shortcuts send the same, with their level; in [`readCommittedRO()`](https://pgcast.dev/transaction#readCommittedRO), each query reads its own snapshot, so a booking committed between the two counts is in the second. ## Types ### `IsolationLevel` ```ts enum IsolationLevel { Serializable = 'SERIALIZABLE', RepeatableRead = 'REPEATABLE READ', ReadCommitted = 'READ COMMITTED', SerializableRO = 'SERIALIZABLE, READ ONLY', RepeatableReadRO = 'REPEATABLE READ, READ ONLY', ReadCommittedRO = 'READ COMMITTED, READ ONLY', SerializableRODeferrable = 'SERIALIZABLE, READ ONLY, DEFERRABLE', } ``` The second argument of [`transaction()`](https://pgcast.dev/transaction#transaction). Each value is written after `START TRANSACTION ISOLATION LEVEL`; each member has its shortcut: | Member | Shortcut | Its client | |---|---|---| | `Serializable` | [`serializable()`](https://pgcast.dev/transaction#serializable) | [`TxnClientForSerializable`](https://pgcast.dev/transaction#TxnClientForSerializable) | | `RepeatableRead` | [`repeatableRead()`](https://pgcast.dev/transaction#repeatableRead) | [`TxnClientForRepeatableRead`](https://pgcast.dev/transaction#TxnClientForRepeatableRead) | | `ReadCommitted` | [`readCommitted()`](https://pgcast.dev/transaction#readCommitted) | [`TxnClientForReadCommitted`](https://pgcast.dev/transaction#TxnClientForReadCommitted) | | `SerializableRO` | [`serializableRO()`](https://pgcast.dev/transaction#serializableRO) | [`TxnClientForSerializableRO`](https://pgcast.dev/transaction#TxnClientForSerializableRO) | | `RepeatableReadRO` | [`repeatableReadRO()`](https://pgcast.dev/transaction#repeatableReadRO) | [`TxnClientForRepeatableReadRO`](https://pgcast.dev/transaction#TxnClientForRepeatableReadRO) | | `ReadCommittedRO` | [`readCommittedRO()`](https://pgcast.dev/transaction#readCommittedRO) | [`TxnClientForReadCommittedRO`](https://pgcast.dev/transaction#TxnClientForReadCommittedRO) | | `SerializableRODeferrable` | [`serializableRODeferrable()`](https://pgcast.dev/transaction#serializableRODeferrable) | [`TxnClientForSerializableRODeferrable`](https://pgcast.dev/transaction#TxnClientForSerializableRODeferrable) | ### `IsolationSatisfying` ```ts type IsolationSatisfying ``` The levels whose client is accepted where a client of level `T` is asked for: `T` and the stricter levels. A read-write level stands for its read-only one, not the reverse; `SerializableRODeferrable` stands only for itself. | `T` | `IsolationSatisfying` | |---|---| | `Serializable` | `Serializable` | | `RepeatableRead` | `Serializable`, `RepeatableRead` | | `ReadCommitted` | `Serializable`, `RepeatableRead`, `ReadCommitted` | | `SerializableRO` | `Serializable`, `SerializableRO` | | `RepeatableReadRO` | `Serializable`, `RepeatableRead`, `SerializableRO`, `RepeatableReadRO` | | `ReadCommittedRO` | every level but `SerializableRODeferrable` | | `SerializableRODeferrable` | `Serializable`, `SerializableRO`, `SerializableRODeferrable` | A function that takes a [`TxnClientForRepeatableRead`](https://pgcast.dev/transaction#TxnClientForRepeatableRead) can be given the client of `serializable()`, not that of `readCommitted()` (2345, with `strict` on or off): ```ts const countBookings = (txnClient: db.TxnClientForRepeatableRead) => db.count('seat_bookings', { screening_id: 1000 }).run(txnClient); await db.serializable(pool, txnClient => countBookings(txnClient)); await db.readCommitted(pool, txnClient => countBookings(txnClient)); // error TS2345 ``` ### `TxnClient` ```ts interface TxnClient extends pg.PoolClient { _zapatos?: { isolationLevel: T; txnId: number }; } ``` The client a [`transaction()`](https://pgcast.dev/transaction#transaction) callback gets: the one checked out from the pool, or the client given. - `_zapatos` is set while the transaction runs: its `isolationLevel`, and its `txnId`, the id the listeners get. Once the transaction ends, `transaction()` deletes it, and releases a client it checked out. - `T`: the levels the client stands for, an [`IsolationSatisfying`](https://pgcast.dev/transaction#IsolationSatisfying) union in the seven types below. - `_zapatos` is optional in the type, so any `pg.PoolClient` is accepted as a `TxnClient`, with or without a transaction open on it. ### `TxnClientForSerializable` `TxnClient>`: the client of [`serializable()`](https://pgcast.dev/transaction#serializable). ### `TxnClientForRepeatableRead` `TxnClient>`: the client of [`repeatableRead()`](https://pgcast.dev/transaction#repeatableRead); takes that of `serializable()` too. ### `TxnClientForReadCommitted` `TxnClient>`: the client of [`readCommitted()`](https://pgcast.dev/transaction#readCommitted); takes that of `serializable()` and `repeatableRead()` too. ### `TxnClientForSerializableRO` `TxnClient>`: the client of [`serializableRO()`](https://pgcast.dev/transaction#serializableRO); takes that of `serializable()` too. ### `TxnClientForRepeatableReadRO` `TxnClient>`: the client of [`repeatableReadRO()`](https://pgcast.dev/transaction#repeatableReadRO); takes that of `serializable()`, `repeatableRead()` and `serializableRO()` too. ### `TxnClientForReadCommittedRO` `TxnClient>`: the client of [`readCommittedRO()`](https://pgcast.dev/transaction#readCommittedRO); takes that of every shortcut but `serializableRODeferrable()`. ### `TxnClientForSerializableRODeferrable` `TxnClient>`: the client of [`serializableRODeferrable()`](https://pgcast.dev/transaction#serializableRODeferrable); takes that of `serializable()` and `serializableRO()` too. ## `isDatabaseError()` ```ts db.isDatabaseError(err, ...types) ``` `true` when the `code` of `err`, five characters, starts with the code of one of `types`; otherwise `false`, and always with no `types`. - `err`: an object with an optional `code` string; `null` or `undefined` throws a `TypeError`. On an error from Postgres, `.run()` rejects with node-postgres's `DatabaseError`, whose `code` is the SQLSTATE; it also has `constraint`, `detail`, `table` and the other fields Postgres sends. - The type accepts `any`, `pg.DatabaseError` and `{ code?: string }`, and refuses `unknown` and the global `Error` (below); `err instanceof pg.DatabaseError` narrows a `catch` variable to one it accepts. - `types`: names, any number; with none, `false`. Two kinds: | Name | Code | Matches | |---|---|---| | a class: `IntegrityConstraintViolation` | `23` | every code of the class: 23505, 23514, 23503… | | a condition, `Class_Condition`: `IntegrityConstraintViolation_UniqueViolation` | `23505` | that code | The names are those of [Postgres's error codes](https://www.postgresql.org/docs/current/errcodes-appendix.html), in PascalCase: `unique_violation` in the class `integrity_constraint_violation` is `IntegrityConstraintViolation_UniqueViolation`. The type refuses any other name. ```ts const booking = { screening_id: 1002, seat: 'C4', customer: 'Tamsin Rowe' }; await db.insert('seat_bookings', booking).run(pool); try { await db.insert('seat_bookings', booking).run(pool); } catch (err: any) { if (!db.isDatabaseError(err, 'IntegrityConstraintViolation_UniqueViolation')) throw err; console.log(err.constraint); // seat_bookings_pkey } ``` On the docs schema: | Query | `code` | Condition name | |---|---|---| | a second booking of the same seat | `23505` | `IntegrityConstraintViolation_UniqueViolation` | | the seat `'Z100'`, which the domain `seat_label` refuses | `23514` | `IntegrityConstraintViolation_CheckViolation` | | a booking of a screening that does not exist | `23503` | `IntegrityConstraintViolation_ForeignKeyViolation` | | an `insert()` in [`serializableRO()`](https://pgcast.dev/transaction#serializableRO) | `25006` | `InvalidTransactionState_ReadOnlySqlTransaction` | | two concurrent bookings of a seat, at `SERIALIZABLE` (above) | `40001` | `TransactionRollback_SerializationFailure` | - `DiskFull`, `OutOfMemory`, `TooManyConnections` and `ConfigurationLimitExceeded` have the class's code, `53`, as `InsufficientResources` does: each matches every code of the class. `InsufficientResources_DiskFull` matches 53100 only. - The errors pgcast throws itself have no `code`: `false` for each ([below](https://pgcast.dev/transaction#errors-pgcast-throws-before-any-query)). - With `strict` on, a `catch` variable is `unknown`, which the type refuses (2345): ```ts try { await db.insert('seat_bookings', { screening_id: 1002, seat: 'C4', customer: 'Tamsin Rowe' }).run(pool); } catch (err) { db.isDatabaseError(err, 'IntegrityConstraintViolation_UniqueViolation'); // error TS2345 } ``` - A value typed as the global `Error`, as `instanceof Error` narrows it, is refused too, with `strict` on or off (2559): the parameter's type is `{ code?: string }`, and `Error` has no property in common with it. ```ts try { await db.insert('seat_bookings', { screening_id: 1002, seat: 'C4', customer: 'Tamsin Rowe' }).run(pool); } catch (err) { if (err instanceof Error && db.isDatabaseError(err, 'IntegrityConstraintViolation_UniqueViolation')) {} // error TS2559 } ``` Handling them in an app: [Errors](https://pgcast.dev/handle-errors). ## Errors pgcast throws before any query A plain `Error`, with no `code`, before anything is sent. A shortcut throws its own synchronously, from the call itself: `.run()` is never reached, so a `.catch()` chained on it does not get them. The others are thrown as `.compile()` compiles the query, so `.run()` rejects with them; `transaction()` rejects with its own. Each is listed with what throws it: | Thrown by | How | Messages | |---|---|---| | `select()`, `selectOne()`, `selectExactlyOne()` and the aggregates, given a bad `order`, `lock` or `aggregate` | the call throws | [`select()`, Errors](https://pgcast.dev/select#errors) | | `upsert()` of rows with different keys, with `upsertRequiresSameKeys` | the call throws | [Rows with different keys](https://pgcast.dev/insert#upsert-different-keys) | | `update()` with no column to set | the call throws | [`update()`](https://pgcast.dev/insert#update) | | `truncate()` with another option | the call throws | [`truncate()`](https://pgcast.dev/insert#truncate) | | an interpolation that has no meaning in SQL: a number, `null`, `db.all`, the global object | `.run()` rejects | [Interpolations](https://pgcast.dev/sql-template#interpolations) | | `db.self` with no column; `db.parent()` with no parent table, so as an `insert()`, `upsert()` or `update()` value | `.run()` rejects | [`self`](https://pgcast.dev/sql-template#self), [`parent()`](https://pgcast.dev/sql-template#parent) | | `db.Default`, `db.self`, `db.raw()`, `db.cols()` or `db.vals()` as a filter value; `db.raw()`, `db.cols()` or `db.vals()` as an `insert()`, `upsert()` or `update()` value, or in any `vals({…})` | `.run()` rejects | [Whereable objects](https://pgcast.dev/sql-template#whereable), [Conditions](https://pgcast.dev/conditions), [`insert()`](https://pgcast.dev/insert#insert), [`vals()`](https://pgcast.dev/sql-template#vals) | | `transaction()` with another isolation level | its promise rejects | [`transaction()`, Errors](https://pgcast.dev/transaction#errors) | After a query, pgcast rejects in three cases of its own: `selectExactlyOne()` with no row, or a second one under [`selectExactlyOneChecksForSecondRow`](https://pgcast.dev/runtime-config#Config.selectExactlyOneChecksForSecondRow), with a [`NotExactlyOneError`](https://pgcast.dev/select#NotExactlyOneError); `count()` and the other aggregates with no row to read, with a `TypeError` ([`count()`](https://pgcast.dev/select#count)); `transaction()` whose `COMMIT` Postgres answers with `ROLLBACK`, with [`transactionRequiresCommit`](https://pgcast.dev/runtime-config#Config.transactionRequiresCommit) ([Errors](https://pgcast.dev/transaction#errors)). --- # Configuration Source: https://pgcast.dev/runtime-config What `zapatos/db` lets you set while your code runs: `setConfig()` and each option of `Config`, the listeners, the opt-ins and their warning codes. Then the types the generated schema gives values that JSON or `pg` hand back as strings, and the helpers that convert them. ## `setConfig()` ```ts db.setConfig(newConfig: NewConfig): Config ``` Sets the keys of `newConfig` on the configuration, and leaves the others as they are. There is one configuration per process, shared by `import` and `require`. ```ts db.setConfig({ transactionAttemptsMax: 3, transactionRetryDelay: { minMs: 50, maxMs: 500 }, undefinedMeansMissing: true, }); ``` - The merge is one level deep: `transactionRetryDelay` is replaced whole, so the type needs both `minMs` and `maxMs`. - A key that has a default and that you give as `undefined` (or `null`) goes back to its default: `setConfig({ transactionRetryDelay: undefined })` sets `{ minMs: 25, maxMs: 250 }`. A listener has none, so `undefined` removes it. Giving `undefined` compiles, except under `exactOptionalPropertyTypes`. - Zapatos 6.6.1 set the key to `undefined`. After `transactionAttemptsMax: undefined`, `transaction()` gave up after its first attempt ("… on attempt 1 of undefined, giving up"); after `transactionRetryDelay: undefined`, every `transaction()` rejected with "TypeError: Cannot destructure property 'minMs' of 'config.transactionRetryDelay' as it is undefined." and never released the client it took from a pool ([#246](https://github.com/AlexandreDemartin/pgcast/issues/246)). - It returns a copy of the new configuration, as `getConfig()` does: changing that object changes nothing. Zapatos 6.6.1 returned the configuration itself ([#245](https://github.com/AlexandreDemartin/pgcast/issues/245)). - It keeps the `transactionRetryDelay` object you give it, not a copy: changing its `minMs` or `maxMs` afterwards changes the configuration. - An unknown key does not compile (TS2353), and neither does a value of the wrong type (TS2322): ```ts db.setConfig({ retries: 3 }); // error TS2353 ``` ## `getConfig()` ```ts db.getConfig(): Config ``` A copy of the configuration, `transactionRetryDelay` included: `getConfig().transactionRetryDelay.minMs = 1` changes nothing. The listeners are the functions you set. Zapatos 6.6.1 shared `transactionRetryDelay`, so that line changed the configuration ([#245](https://github.com/AlexandreDemartin/pgcast/issues/245)). To change the configuration for a while, as tests do, keep the copy and give it back in a `finally`, so that a throw in between, a failing test, still restores it: ```ts const saved = db.getConfig(); db.setConfig({ transactionRetryDelay: { minMs: 0, maxMs: 0 } }); try { // … the transactions that should retry at once } finally { db.setConfig(saved); } ``` - It puts back each key `saved` holds. - A key `saved` doesn't hold stays as set: a listener set in between keeps listening. Give it as `undefined` to remove it: `db.setConfig({ queryListener: undefined })`. ## `Config` ```ts interface Config { transactionAttemptsMax: number; transactionRetryDelay: { minMs: number; maxMs: number }; castArrayParamsToJson: boolean; castObjectParamsToJson: boolean; upsertRequiresSameKeys?: boolean; undefinedMeansMissing?: boolean; transactionRequiresCommit?: boolean; selectExactlyOneChecksForSecondRow?: boolean; queryListener?(query: SQLQuery, txnId?: number): void; resultListener?(result: any, txnId?: number, elapsedMs?: number, query?: SQLQuery): void; transactionListener?(message: string, txnId?: number): void; } ``` The keys pgcast adds are optional, so a `Config` written for Zapatos 6.6.1 still compiles; `getConfig()` types them `boolean | undefined`, though it always gives a value. No option changes a type. | Option | Default | What it changes | Read | |---|---|---|---| | [`transactionAttemptsMax`](https://pgcast.dev/runtime-config#Config.transactionAttemptsMax) | `5` | how many times `transaction()` runs its callback | at each `transaction()` | | [`transactionRetryDelay`](https://pgcast.dev/runtime-config#Config.transactionRetryDelay) | `{ minMs: 25, maxMs: 250 }` | the wait before each retry | at each `transaction()` | | [`castArrayParamsToJson`](https://pgcast.dev/runtime-config#Config.castArrayParamsToJson) | `false` | an array parameter: `CAST($1 AS "json")` in place of `$1` | at each `compile()` | | [`castObjectParamsToJson`](https://pgcast.dev/runtime-config#Config.castObjectParamsToJson) | `false` | a plain object parameter: `CAST($1 AS "json")` in place of `$1` | at each `compile()` | | [`upsertRequiresSameKeys`](https://pgcast.dev/runtime-config#Config.upsertRequiresSameKeys) | `false` | a multi-row `upsert()` with a key missing from some row, still set from `EXCLUDED`: throw, not warn | at each `upsert()` | | [`undefinedMeansMissing`](https://pgcast.dev/runtime-config#Config.undefinedMeansMissing) | `false` | an `undefined` value in `insert()`, `update()` and `upsert()`: a missing key, not `NULL` | at each call of those | | [`transactionRequiresCommit`](https://pgcast.dev/runtime-config#Config.transactionRequiresCommit) | `false` | a `COMMIT` that Postgres answers with `ROLLBACK`: reject, not resolve | at each `transaction()` | | [`selectExactlyOneChecksForSecondRow`](https://pgcast.dev/runtime-config#Config.selectExactlyOneChecksForSecondRow) | `false` | a `selectExactlyOne()` run on its own: `LIMIT 2`, and a second row rejects | at each `selectExactlyOne()` | | [`queryListener`](https://pgcast.dev/runtime-config#Config.queryListener) | none | called with each query `run()` sends | at each `run()` | | [`resultListener`](https://pgcast.dev/runtime-config#Config.resultListener) | none | called with each result `run()` gives | at each `run()` | | [`transactionListener`](https://pgcast.dev/runtime-config#Config.transactionListener) | none | called when an attempt of `transaction()` fails with `40001` or `40P01`, and at each retry | at each `transaction()` | ### `transactionAttemptsMax` ```ts transactionAttemptsMax: number // default: 5 ``` How many times [`transaction()`](https://pgcast.dev/transaction#transaction) runs its callback. After a serialisation failure (`40001`) or a deadlock (`40P01`), it rolls back and runs the callback again, until this many attempts have failed; then it throws the last error. Any other error is rolled back and thrown at once. - `1`, `0` or less: one attempt, no retry. With `0`, the message reads "… on attempt 1 of 0, giving up" ([#179](https://github.com/AlexandreDemartin/pgcast/issues/179)). - Read when `transaction()` starts, so a change affects the next transaction, not the one running. - How to write a callback that can run twice: [Retries](https://pgcast.dev/retries). ### `transactionRetryDelay` ```ts transactionRetryDelay: { minMs: number; maxMs: number } // default: { minMs: 25, maxMs: 250 } ``` Before each retry, `transaction()` waits a whole number of milliseconds from `minMs` to `maxMs`, rounded from a uniform draw between them: `Math.round(minMs + (maxMs - minMs) * Math.random())`. So `minMs` and `maxMs` come up half as often as each value between them. [`transactionListener`](https://pgcast.dev/runtime-config#Config.transactionListener) is told the wait. ### `castArrayParamsToJson` ```ts castArrayParamsToJson: boolean // default: false ``` With `true`, every array sent as a parameter is sent as JSON: `CAST($1 AS "json")`, its value `JSON.stringify()`'d. That is `param(x)`, and the values of the shortcuts and Whereables, which are parameters too. Off, `pg` sends an array as a Postgres array. A string cast wins ([#36](https://github.com/AlexandreDemartin/pgcast/issues/36)); Zapatos 6.6.1 dropped it for `"json"`: - An array whose cast ends with `[]` or `[n]` is sent as an array: `param([1, 2], 'int4[]')` gives `CAST($1 AS "int4"[])`. - Any other array keeps the JSON, `'_int4'` and a domain over an array included: `param([1, 2], 'jsonb')` gives `CAST($1 AS "jsonb")`. You write: ```ts db.setConfig({ castArrayParamsToJson: true }); const lengths = await db.sql`SELECT json_array_length(${db.param(['mystery', 'drama'])}) AS n`.run(pool); ``` Postgres runs: ```sql SELECT json_array_length(CAST($1 AS "json")) AS n ``` Values: `["[\"mystery\",\"drama\"]"]` TypeScript knows: ```ts const lengths: { n: number; }[] ``` - Off, the same query fails: `json_array_length($1)` gives 22P02 "invalid input syntax for type json". `param(x, true)` sends the array as JSON under either setting. - On a Postgres array column, Postgres refuses an array sent as JSON: 42804 "column "genres" is of type text[] but expression is of type json" in an `insert()` or `update()`, 42883 "operator does not exist: text[] = json" in a Whereable, and "… text[] && json" with [`arrayOverlaps()`](https://pgcast.dev/conditions#conditions.arrayOverlaps). - In an `insert()`, an `update()` or a Whereable, `db.param(genres, 'text[]')` sends it as an array. The array conditions take no `Parameter` ([#40](https://github.com/AlexandreDemartin/pgcast/issues/40)): compare in a [`sql` fragment](https://pgcast.dev/conditions#arrays) there. - `param(x, false)` sends the array as it is, under either setting: `{ genres: db.param(['mystery', 'drama'], false) }` gives `SET ("genres") = ROW($1)`. - The items of `isIn()` are parameters one by one, so an array of numbers there is unchanged: `"id" IN ($1, $2)`. - How `param()` compiles each `cast`: [`param()`](https://pgcast.dev/sql-template#param). ### `castObjectParamsToJson` ```ts castObjectParamsToJson: boolean // default: false ``` With `true`, every plain object sent as a parameter is sent as `CAST($1 AS "json")`, its value `JSON.stringify()`'d. Off, it is `$1`, and `pg` sends the same JSON text, with no type: Postgres then infers one from where `$1` stands. A string cast wins, the value still JSON text: `param({ a: 1 }, 'jsonb')` gives `CAST($1 AS "jsonb")`, which `=` and `@>` on a `jsonb` column take ([#36](https://github.com/AlexandreDemartin/pgcast/issues/36)); Zapatos 6.6.1 dropped it for `"json"`, which they refuse (42883). You write: ```ts db.setConfig({ castObjectParamsToJson: true }); const pitches = await db.sql`SELECT ${db.param({ title: 'The Lighthouse Wager', runtime_minutes: 101 })} ->> 'title' AS title`.run(pool); ``` Postgres runs: ```sql SELECT CAST($1 AS "json") ->> 'title' AS title ``` Values: `["{\"title\":\"The Lighthouse Wager\",\"runtime_minutes\":101}"]` TypeScript knows: ```ts const pitches: { title: string; }[] ``` - Off, the same query fails: `$1 ->> 'title'` gives 42725 "operator is not unique: unknown ->> unknown". A cast in the SQL, `$1::jsonb`, works either way. - A plain object is an object literal, `{…}`: a `Date`, a `Buffer` or a class instance is sent as `$1`, unchanged. - `param(x, false)` gives `$1` under either setting; `param(x, true)` gives `CAST($1 AS "json")`. ### `upsertRequiresSameKeys` ```ts upsertRequiresSameKeys?: boolean // default: false; the default in 7.0 ``` With `true`, a multi-row [`upsert()`](https://pgcast.dev/insert#upsert) without `updateColumns`, where a key missing from some row is still set from `EXCLUDED`, throws when called, before any query: "rows have different keys: pass updateColumns, or give every row the same keys". Off, it runs as in Zapatos 6.6.1 and warns once per process: `PGCAST_DEP001`. Which keys count, and the SQL: [Rows with different keys](https://pgcast.dev/insert#upsert-different-keys). ### `undefinedMeansMissing` ```ts undefinedMeansMissing?: boolean // default: false; the default in 7.0 ``` With `true`, an `undefined` value in [`insert()`](https://pgcast.dev/insert#insert), [`update()`](https://pgcast.dev/insert#update) and [`upsert()`](https://pgcast.dev/insert#upsert) is a missing key: `DEFAULT` in `insert()`, left out of `update()` and `upsert()`. Off, it is sent as `NULL`, as in Zapatos 6.6.1, and warns once per process: `PGCAST_DEP002`. The SQL of each: [An `undefined` value](https://pgcast.dev/insert#undefined-values). ### `transactionRequiresCommit` ```ts transactionRequiresCommit?: boolean // default: false; the default in 7.0 ``` When the callback of [`transaction()`](https://pgcast.dev/transaction#transaction) catches a query's error and returns, Postgres has aborted the transaction: it answers the `COMMIT` with the command tag `ROLLBACK`, not an error, and nothing is saved. With `true`, `transaction()` then rejects, without a retry, with "the transaction was rolled back, not committed: a query in it failed and the callback caught the error, so Postgres answered COMMIT with ROLLBACK; let the error out of the callback, or ROLLBACK TO a SAVEPOINT before going on". Off, it resolves with what the callback returned, as in Zapatos 6.6.1, and warns once per process: `PGCAST_DEP003`. What to do instead: [Let the error through](https://pgcast.dev/retries#let-the-error-through). - Not covered: a callback that sends `COMMIT` or `ROLLBACK` itself. - Not tested with `pg-native`. ### `selectExactlyOneChecksForSecondRow` ```ts selectExactlyOneChecksForSecondRow?: boolean // default: false, in 7.0 too ``` With `true`, a [`selectExactlyOne()`](https://pgcast.dev/select#selectExactlyOne) run on its own asks for two rows, `LIMIT $2` with `2`, and `.run()` rejects with a [`NotExactlyOneError`](https://pgcast.dev/select#NotExactlyOneError) when a second row comes: "One result expected but more returned (hint: check `.query.compile()` on this Error)". Off, it asks for one, `LIMIT 1`, and gives the first row that matches, as in Zapatos 6.6.1. - Nested in another query, in a `lateral` or a `sql` template, it keeps `LIMIT 1`, option or not: a second row there would join the parent row twice. "On its own" is how the query is compiled: `.run()` and `.compile()` compile it alone, a parent query compiles it as part of its own SQL. - Off, it never warns, and it stays off in 7.0: "at least one row" is how Zapatos reads `selectExactlyOne()`, nested in a `lateral` on a `NOT NULL` foreign key ([decision 003](https://github.com/AlexandreDemartin/pgcast/blob/master/docs/decisions/003-fixes-and-deprecation.md)). - Read when `selectExactlyOne()` is called, not at `.run()`. ### `queryListener` ```ts queryListener?(query: SQLQuery, txnId?: number): void ``` Called by [`run()`](https://pgcast.dev/sql-template#run) once the query is compiled, before it is sent. - `query`: the [`SQLQuery`](https://pgcast.dev/runtime-config#SQLQuery) `run()` sends, `{ text, values }`, and `name` once [prepared](https://pgcast.dev/sql-template#prepared). - `txnId`: in a [transaction](https://pgcast.dev/runtime-config#transaction-ids), its id; otherwise `undefined`. - It is called for a no-op fragment too, which sends nothing: `insert('tags', [])` gives `/* marked no-op: won't hit DB unless forced -> */ INSERT INTO "tags" SELECT null WHERE false`. - It is not called when compiling throws. If it throws, `run()` rejects, and the query is not sent. ```ts db.setConfig({ queryListener: (query, txnId) => console.log(`(${txnId ?? '-'})`, query.text, query.values), resultListener: (result, txnId, elapsedMs) => console.log(`(${txnId ?? '-'}, ${elapsedMs?.toFixed(1)}ms)`, result), transactionListener: (message, txnId) => console.log(`(${txnId})`, message), }); ``` ### `resultListener` ```ts resultListener?(result: any, txnId?: number, elapsedMs?: number, query?: SQLQuery): void ``` Called by `run()` with what it resolves to, once [`runResultTransform`](https://pgcast.dev/sql-template#runResultTransform) has run. - `result`: `{ title: 'The Salt Orchard' }` for `selectOne('films', { id: 1000 }, { columns: ['title'] })`, a number for `count()`, `[]` for a no-op. - `txnId`: as for [`queryListener`](https://pgcast.dev/runtime-config#Config.queryListener). - `elapsedMs`: from just before the query is sent to just after the transform, from `performance.now()`, in fractions of a millisecond. - `query`: the query, as `queryListener` got it. - It is not called when the query rejects, nor when the transform throws (a `selectExactlyOne()` that finds no row). ### `transactionListener` ```ts transactionListener?(message: string, txnId?: number): void ``` Called by `transaction()` when an attempt fails with `40001` or `40P01`, and again when the next attempt starts, with one of these messages and the transaction's id: | When | Message | |---|---| | an attempt failed, another is left | `` `Transaction rollback (code ${err.code}) on attempt ${attempt} of ${maxAttempts}, retrying in ${delayBeforeRetry}ms` `` | | the next attempt starts | `` `Retrying transaction, attempt ${attempt} of ${maxAttempts}` `` | | the last attempt failed | `` `Transaction rollback (code ${err.code}) on attempt ${attempt} of ${maxAttempts}, giving up` `` | With `transactionAttemptsMax: 3` and `transactionRetryDelay: { minMs: 1, maxMs: 2 }`, a callback that fails with `40001` each time gives `Transaction rollback (code 40001) on attempt 1 of 3, retrying in 1ms`, `Retrying transaction, attempt 2 of 3`, then the same for attempt 2, and `Transaction rollback (code 40001) on attempt 3 of 3, giving up`. It is not called for a transaction that commits on its first attempt, nor for any other error. ### Transaction ids The `txnId` the listeners get. - Each `transaction()` call that opens a transaction takes the next id: `0` for the first in the process, then `1`… - Every query `run()` sends on its client gets that id, through every retry: yours, and `transaction()`'s own `START TRANSACTION ISOLATION LEVEL SERIALIZABLE`, `COMMIT` and `ROLLBACK`, which go through both query listeners like any other. - A `transaction()` given that client opens no transaction, and its queries keep the outer id. - Outside a transaction, and on the client once it has ended, the id is `undefined`. ## `NewConfig` ```ts type NewConfig = Partial ``` What [`setConfig()`](https://pgcast.dev/runtime-config#setConfig) takes: any keys of `Config`, each one whole. ## `SQLQuery` ```ts interface SQLQuery { text: string; values: any[]; name?: string } ``` A compiled query, as `pg`'s `query()` takes it: what [`compile()`](https://pgcast.dev/sql-template#compile) returns and the listeners get. `name` is set by [`prepared()`](https://pgcast.dev/sql-template#prepared). ## Opt-ins and warnings A behaviour of Zapatos 6.6.1 that code may rely on, but that pgcast would change, stays the default until 7.0: an option turns the new one on, and the old one warns. Why this way: [Versions](https://pgcast.dev/versions). | Option | Warning | It warns when | Default from | |---|---|---|---| | [`upsertRequiresSameKeys: true`](https://pgcast.dev/runtime-config#Config.upsertRequiresSameKeys) | `PGCAST_DEP001` | a multi-row `upsert()` without `updateColumns` has a key missing from some row that is still set from `EXCLUDED` ([Rows with different keys](https://pgcast.dev/insert#upsert-different-keys)) | 7.0 | | [`undefinedMeansMissing: true`](https://pgcast.dev/runtime-config#Config.undefinedMeansMissing) | `PGCAST_DEP002` | a value of `insert()`, `update()` or `upsert()` is `undefined` ([An `undefined` value](https://pgcast.dev/insert#undefined-values)) | 7.0 | | [`transactionRequiresCommit: true`](https://pgcast.dev/runtime-config#Config.transactionRequiresCommit) | `PGCAST_DEP003` | Postgres answers the `COMMIT` of `transaction()` with `ROLLBACK`, after a query error the callback caught ([Let the error through](https://pgcast.dev/retries#let-the-error-through)) | 7.0 | One opt-in has no warning and no default in 7.0: [`selectExactlyOneChecksForSecondRow`](https://pgcast.dev/runtime-config#Config.selectExactlyOneChecksForSecondRow), since 6.6.1's reading of `selectExactlyOne()`, at least one row, is one Zapatos documents. The generator has its own: [`typeEnumsAcrossSchemas`](https://pgcast.dev/generator#generate.typeEnumsAcrossSchemas) and [`enumOrder`](https://pgcast.dev/generator#generate.enumOrder), whose old paths are notes through [`warningListener`](https://pgcast.dev/generator#generate.warningListener), once per run, rather than a `DeprecationWarning`; they will be the defaults in 7.0 too. Each warning is a Node.js `DeprecationWarning`, emitted once per process for each code, when the shortcut is called (`PGCAST_DEP003`: when `transaction()` resolves), and only when the old behaviour is used. Its message names the option. Node prints it to `stderr`: ```txt (node:4242) [PGCAST_DEP002] DeprecationWarning: An undefined value in insert(), update() or upsert() is sent as NULL. Set undefinedMeansMissing: true (db.setConfig) to treat it as a missing key instead (DEFAULT in insert(), left out of update() and upsert()); it will be the default in pgcast 7.0. ``` - `process.on('warning', …)` gets it, with `name` `'DeprecationWarning'` and `code` `'PGCAST_DEP002'`. - `node --no-deprecation` hides it, from that hook too; `node --no-warnings` hides it from `stderr` only. `node --throw-deprecation` throws it, uncaught, so the process ends. - With the option on, the call never warns. To pick out pgcast's warnings in that hook, to log them your way: `@types/node` types the warning `Error`, which has no `code`, so say it may have one. Node still prints each warning to `stderr`; run with `node --no-warnings` to leave the hook's as the only output. ```ts process.on('warning', (warning: Error & { code?: string }) => { if (warning.code?.startsWith('PGCAST_')) console.warn(warning.code, warning.message); }); ``` ## Value types `pg` and JSON give some Postgres values as strings. The generated schema types them with the template string types below, by where the value is ([The catalog](https://pgcast.dev/catalog-types)): | Postgres type | `Selectable` | `JSONSelectable` | `Insertable`, `Updatable`, `Whereable` | |---|---|---|---| | `int8` | [`Int8String`](https://pgcast.dev/runtime-config#Int8String) | `number` | `number \| Int8String \| bigint` | | `numeric` | [`NumericString`](https://pgcast.dev/runtime-config#NumericString) | `number` | `number \| NumericString` | | `bytea` | `Buffer` | [`ByteArrayString`](https://pgcast.dev/runtime-config#ByteArrayString) | `ByteArrayString \| Buffer` | | `date` | `Date` | [`DateString`](https://pgcast.dev/runtime-config#DateString) | `DateString \| Date` | | `timestamp` | `Date` | [`TimestampString`](https://pgcast.dev/runtime-config#TimestampString) | `TimestampString \| Date` | | `timestamptz` | `Date` | [`TimestampTzString`](https://pgcast.dev/runtime-config#TimestampTzString) | `TimestampTzString \| Date` | | `time` | [`TimeString`](https://pgcast.dev/runtime-config#TimeString) | `TimeString` | `TimeString` | | `timetz` | [`TimeTzString`](https://pgcast.dev/runtime-config#TimeTzString) | `TimeTzString` | `TimeTzString` | | `int4range`, `int8range`, `numrange` | [`NumberRangeString`](https://pgcast.dev/runtime-config#NumberRangeString) | `NumberRangeString` | `NumberRangeString` | | `tsrange`, `tstzrange`, `daterange` | [`DateRangeString`](https://pgcast.dev/runtime-config#DateRangeString) | `DateRangeString` | `DateRangeString` | | `json`, `jsonb` | [`JSONValue`](https://pgcast.dev/runtime-config#JSONValue) | `JSONValue` | `JSONValue` | - An array column takes the array of its item's type: `int8[]` gives `Int8String[]` in a `Selectable`. - With the generator's [`customJSONParsingForLargeNumbers`](https://pgcast.dev/generator#generate.customJSONParsingForLargeNumbers), `int8` and `numeric` are `number | Int8String` and `number | NumericString` in a `JSONSelectable` ([`enableCustomJSONParsingForLargeNumbers()`](https://pgcast.dev/runtime-config#enableCustomJSONParsingForLargeNumbers)). - The shortcuts return `JSONSelectable`s. A `Selectable` is what `pg` returns for a plain `SELECT` of the row. Each type checks the shape of a string, not its value: `'2030-13-45'` is a `DateString`. ```ts const opening: db.DateString = '2030-03-01'; const premiere: db.TimestampTzString = '2030-03-01T19:30:00'; // error TS2322 ``` ## JSON ### `JSONValue` ```ts type JSONValue = null | boolean | number | string | JSONObject | JSONArray ``` A value JSON can hold: the type of a `json` or `jsonb` column. `undefined`, a `Date` and a function are not one. ### `JSONObject` ```ts type JSONObject = { [k: string]: JSONValue } ``` ### `JSONArray` ```ts type JSONArray = JSONValue[] ``` ## Large numbers ### `Int8String` ```ts type Int8String = `${number}` ``` An `int8` as `pg` returns it, a string: `'9007199254740993'`, which no `number` holds. ### `NumericString` ```ts type NumericString = `${number}` ``` A `numeric` as `pg` returns it, a string that keeps its digits: `'1.10'`. ### `enableCustomJSONParsingForLargeNumbers()` ```ts db.enableCustomJSONParsingForLargeNumbers(pg: typeof import('pg')): void ``` Makes `pg` parse `json` and `jsonb` values with [json-custom-numbers](https://www.npmjs.com/package/json-custom-numbers), which gives a number as a string when a `number` would change it. Each shortcut except the aggregates returns `jsonb`, so it applies to their results. ```ts db.enableCustomJSONParsingForLargeNumbers(pg); ``` | A number in JSON | `JSON.parse()`, the default | Once enabled | |---|---|---| | `12`, `0.1` | `12`, `0.1` | `12`, `0.1` | | `9007199254740993` | `9007199254740992` | `'9007199254740993'` | | `9007199254740992`, an integer past `Number.MAX_SAFE_INTEGER` | `9007199254740992` | `'9007199254740992'` | | `1.00000000000000000001` | `1` | `'1.00000000000000000001'` | | a `numeric` past `Number.MAX_VALUE` | `Infinity` | its digits, as a string | - It does nothing for `count()`, `sum()`, `avg()`, `min()` and `max()`: their result is not JSON, and goes through `Number()` ([`count()`](https://pgcast.dev/select#count), [#163](https://github.com/AlexandreDemartin/pgcast/issues/163)). A `sum()` of `9007199254740993` still gives `9007199254740992`. - It sets the type parsers of the `pg` module you give, so every pool and client of that module, from then on, pgcast's queries or not. - It does not change the types: the generator's [`customJSONParsingForLargeNumbers`](https://pgcast.dev/generator#generate.customJSONParsingForLargeNumbers) does. - An `int8` or `numeric` column outside JSON is a string already, by `pg`'s own parser. ## Ranges ### `RangeString` ```ts type RangeString = `${'[' | '('}${Bound},${Bound}${']' | ')'}` ``` A range as Postgres writes it: a bracket, two bounds, a bracket. It does not cover an empty range, which Postgres writes `empty`: `to_jsonb('empty'::int4range)` is `"empty"`, and `const r: db.NumberRangeString = 'empty'` does not compile (TS2322, [#185](https://github.com/AlexandreDemartin/pgcast/issues/185)). ### `NumberRangeString` ```ts type NumberRangeString = RangeString ``` An `int4range`, `int8range` or `numrange`: `'[1,5)'`, or `'[1.5,)'` with no upper bound. ### `DateRangeString` ```ts type DateRangeString = RangeString ``` A `tsrange`, `tstzrange` or `daterange`. Its bounds follow the session's `DateStyle`, so they are typed `string`: `daterange('2030-03-01', '2030-03-08')` is `'[2030-03-01,2030-03-08)'` with the `ISO` style, and `'[01/03/2030,08/03/2030)'` with `SQL, DMY`. The bounds of a `tsrange` or `tstzrange` come quoted, since they hold a space: `'["2030-03-01 19:30:00","2030-03-02 00:00:00")'`. ## Bytes ### `ByteArrayString` ```ts type ByteArrayString = `\\x${string}` ``` A `bytea` in JSON, as hex after `\x`: `'\\x4c61'` in TypeScript source. ### `toBuffer()` ```ts db.toBuffer(d: T): T extends ByteArrayString ? Buffer | Exclude : T ``` Decodes the hex of a `ByteArrayString` into a `Buffer`, typed `Buffer`. `null` gives `null`, and the type keeps it; `undefined` throws, and the type refuses it with `strictNullChecks` on ([`strict()`](https://pgcast.dev/runtime-config#strict)). It decodes the whole value in memory. ```ts const bytes = db.toBuffer('\\x4c61726b73707572'); const word = bytes.toString(); // 'Larkspur' ``` ## Dates and times A date or time in a `JSONSelectable` is the string `to_jsonb()` writes, ISO 8601. A `timestamptz` carries the session's `TimeZone` offset: `'2030-03-01T19:30:00+00:00'` in `UTC`, `'2030-03-02T01:00:00+05:30'` in `Asia/Kolkata`. You write: ```ts const screening = await db.selectExactlyOne( 'screenings', { id: 1000 }, { columns: ['starts_at'] }, ).run(pool); const startsAt = db.toDate(screening.starts_at); ``` Postgres runs: ```sql SELECT jsonb_build_object($1::text, "starts_at") AS result FROM "screenings" WHERE ("id" = $2) LIMIT $3 ``` Values: `["starts_at", 1000, 1]` TypeScript knows: ```ts const screening: db.JSONOnlyColsForTable<"screenings", "starts_at"[]> // that is const screening: { starts_at: db.TimestampTzString; } ``` | Type | Is | For example | |---|---|---| | [`DateString`](https://pgcast.dev/runtime-config#DateString) | `${number}-${number}-${number}` | `'2030-03-01'` | | [`TimeString`](https://pgcast.dev/runtime-config#TimeString) | `${number}:${number}`, then an optional `:${number}` | `'19:30'`, `'19:30:00'` | | [`TzSuffix`](https://pgcast.dev/runtime-config#TzSuffix) | `'Z'`, or `+` or `-` and a number, then an optional `:${number}` | `'Z'`, `'+01'`, `'+05:30'` | | [`TimeTzString`](https://pgcast.dev/runtime-config#TimeTzString) | `${TimeString}${TzSuffix}` | `'19:30:00+01'` | | [`TimestampString`](https://pgcast.dev/runtime-config#TimestampString) | `${DateString}T${TimeString}` | `'2030-03-01T19:30:00'` | | [`TimestampTzString`](https://pgcast.dev/runtime-config#TimestampTzString) | `${TimestampString}${TzSuffix}` | `'2030-03-01T19:30:00+00:00'` | ### `DateString` ```ts type DateString = `${number}-${number}-${number}` ``` A `date`. ### `TimeString` ```ts type TimeString = `${number}:${number}${'' | `:${number}`}` ``` A `time`, with or without its seconds. ### `TzSuffix` ```ts type TzSuffix = 'Z' | `${'+' | '-'}${number}${'' | `:${number}`}` ``` The time zone at the end of a `TimeTzString` or a `TimestampTzString`. ### `TimeTzString` ```ts type TimeTzString = `${TimeString}${TzSuffix}` ``` A `timetz`. ### `TimestampString` ```ts type TimestampString = `${DateString}T${TimeString}` ``` A `timestamp`: no time zone. ### `TimestampTzString` ```ts type TimestampTzString = `${TimestampString}${TzSuffix}` ``` A `timestamptz`. ### `toDate()` ```ts db.toDate(d: D, tzInterpretation?: undefined): D extends null ? null : Date db.toDate(d: D, tzInterpretation: 'UTC' | 'local'): D extends null ? null : Date ``` Makes a `Date` of a string from Postgres. A `TimestampTzString` has its time zone. A `TimestampString` or a `DateString` has none, so you say which to read it in: `'UTC'`, or `'local'`, the time zone of the JavaScript process. ```ts const curtain = db.toDate('2030-03-01T19:30:00+00:00'); const rehearsal = db.toDate('2030-03-01T15:00:00', 'UTC'); const opening = db.toDate('2030-03-01', 'local'); ``` - `toDate('2030-03-01', 'UTC')` is midnight UTC; `'local'` is local midnight. - `null` gives `null`; the type keeps it, so `TimestampTzString | null` gives `Date | null`. - A `Date` keeps milliseconds: the microseconds of a Postgres value are dropped. - It does not check the string: one a `Date` cannot read gives an Invalid Date. A `TimestampString` or a `DateString` without `tzInterpretation` does not compile (TS2345). Nor does a `TimestampTzString` with one, since the first overload takes `tzInterpretation` only as `undefined` (TS2769): ```ts db.toDate('2030-03-01T15:00:00'); // error TS2345 db.toDate('2030-03-01T19:30:00+00:00', 'UTC'); // error TS2769 ``` ### `toString()` ```ts db.toString( d: D, stringTypeTz: T, ): D extends null ? null : { timestamptz: TimestampTzString; 'timestamp:UTC': TimestampString; … }[T] ``` Makes a string Postgres reads of a `Date`, typed by `stringTypeTz`; `null` gives `null`. ```ts const curtain = new Date('2030-03-01T19:30:00Z'); const day: db.DateString = db.toString(curtain, 'date:UTC'); // '2030-03-01' const stamp: db.TimestampTzString = db.toString(curtain, 'timestamptz'); // '2030-03-01T19:30:00.000Z' ``` | `stringTypeTz` | Gives | For 19:30 UTC on 1 March 2030, in a process with `TZ=Asia/Kolkata` | |---|---|---| | `'timestamptz'` | `TimestampTzString`, from `toISOString()` | `'2030-03-01T19:30:00.000Z'` | | `'timestamp:UTC'` | `TimestampString` | `'2030-03-01T19:30:00.000'` | | `'timestamp:local'` | `TimestampString`, in the process's time zone | `'2030-03-02T01:00:00.000'` | | `'date:UTC'` | `DateString` | `'2030-03-01'` | | `'date:local'` | `DateString`, in the process's time zone | `'2030-03-02'` | ## `strict()` ```ts db.strict(fn: (x: FnIn) => FnOut): (d: T) => T extends FnIn ? Exclude | FnOut : T ``` Wraps a one-argument function so that `null` gives `null` without calling it, as a Postgres function declared `STRICT` does. The type follows: `FnIn` gives `FnOut`, `FnIn | null` gives `FnOut | null`, `null` gives `null`. [`toBuffer()`](https://pgcast.dev/runtime-config#toBuffer) is `strict()` of a `ByteArrayString` decoder. ```ts const shout = db.strict((s: string) => s.toUpperCase()); const loud = shout(tagline); ``` - Only `null` passes through, as only `NULL` does for a `STRICT` function in Postgres: `undefined` reaches `fn`. A function that takes it handles it: `db.strict((title: string | undefined) => title ?? 'untitled')(undefined)` gives `'untitled'`. - With `strictNullChecks` on, the type refuses `undefined` unless `fn` takes it: `shout(undefined)` and `db.toBuffer(undefined)` do not compile (TS2345). - With `strictNullChecks` off, a type has no `null` nor `undefined` in it, and none of these types say anything of them: `toDate(null)` is typed `Date`, `shout(null)` `any`. `db.toBuffer(undefined)` then compiles, and throws "TypeError: Cannot read properties of undefined (reading 'slice')". ## `SchemaVersionCanary` ```ts interface SchemaVersionCanary { version: 104 } ``` The version of the generated schema's format that this `zapatos/db` reads. The generated `zapatos/schema` declares `interface schemaVersionCanary extends db.SchemaVersionCanary { version: 104 }`, so a schema written for another format does not compile: ```txt error TS2430: Interface 'schemaVersionCanary' incorrectly extends interface 'SchemaVersionCanary'. Types of property 'version' are incompatible. ``` - pgcast's version is Zapatos 6.6.1's, `104`, so the canary of a schema that Zapatos 6.6.1 generated passes. - Regenerating the schema clears the error ([Generator](https://pgcast.dev/generator)). --- # Generator Source: https://pgcast.dev/generator The generator reads your database's catalog and writes the types of `zapatos/schema`. This page lists how to run it, from the command line or from code, every option of `zapatosconfig.json` and `generate()`, and the files it writes. ## The CLI ```sh npx pgcast [config] npx zapatos [config] ``` The same program under two names: `pgcast`, and `zapatos`, as in Zapatos 6.6.1. It reads the config, then runs [`generate()`](https://pgcast.dev/generator#generate) with it. - `zapatosconfig.json`: read from the folder you run it in, if it is there; never from a parent folder. Without it, the config is `{}`. - `config`: one optional argument, a JSON object, merged over the file key by key: `npx pgcast '{"outDir": "./generated"}'` keeps the file's `db` and replaces its `outDir`. - A key given here replaces the file's whole value, so `'{"db": {"application_name": "codegen"}}'` drops the file's `connectionString`. - The file's variables are filled in before the merge: one it names must be set even when the argument replaces its key, or the CLI stops with `Environment variable 'NAME' is not set`. - Each string in both, at any depth, has its environment variables filled in ([Environment variables](https://pgcast.dev/generator#environment-variables)). - There are no flags: `--help` is read as the JSON argument, and fails. Any argument after the first is ignored. - Paths are relative to the folder you run it in: `outDir`, and so the `zapatos` folder. ```json [zapatosconfig.json] { "db": { "connectionString": "{{DATABASE_URL}}" }, "outDir": "./src", "schemas": { "public": { "include": "*", "exclude": ["seat_bookings"] } } } ``` A config it can't read stops it before any query, with code 1, Node printing the error and its stack. The error starts with what it was reading: - the file: `If present, zapatosconfig.json must be a valid JSON file, and all referenced environment variables must exist:` - the argument: `If present, the argument to Zapatos must be valid JSON, and all referenced environment variables must exist:` Then it ends with the cause: | Cause | Ending | |---|---| | not valid JSON | `JSON.parse`'s message, which depends on the input and on Node: on Node 22, `Unexpected end of JSON input` for an empty file, `No number after minus sign in JSON at position 1 (line 1 column 2)` for `--help`, `Expected double-quoted property name in JSON at position 15 (line 1 column 16)` for a trailing comma | | a variable is not set | `Environment variable 'DATABASE_URL' is not set` | A query that fails, the connection included, also ends it with code 1, as [`generate()`](https://pgcast.dev/generator#generate) says. ### Environment variables `{{NAME}}` in a string of the config is replaced by `process.env.NAME`, before anything else runs. - Anywhere in a string, several per string: `"postgresql://{{PGUSER}}@localhost/films"`. - Spaces inside the braces are allowed: `{{ DATABASE_URL }}`. - In values only, at any depth of objects and arrays; never in keys: `"schemas": { "{{SCHEMA}}": … }` is a schema named `{{SCHEMA}}`. Numbers and booleans pass through. - A variable set to the empty string counts as set, and gives `""`. - Only the CLI does this, for the file and for its argument. [`generate()`](https://pgcast.dev/generator#generate) takes its config as it is: read `process.env` yourself. ## `generate()` ```ts import { generate, type Config } from 'zapatos/generate'; generate(config: Config): Promise ``` Reads the catalog of `config.db` and writes the [files](https://pgcast.dev/generator#the-files-it-writes) under `config.outDir`. `config` takes the [options](https://pgcast.dev/generator#options) of `zapatosconfig.json`, and also functions where the JSON takes `true` or a name: the three listeners and `customTypesTransform`. ```ts await generate({ db: { connectionString: process.env.DATABASE_URL }, outDir: './src', progressListener: line => console.log(`[types] ${line}`), customTypesTransform: name => `Db_${name}`, }); ``` - It opens a `pg.Pool` of its own on `config.db`, and ends it before resolving. - It sends only `SELECT`s, on `information_schema` and `pg_catalog`: one for the relations of each schema, one for its enums, then two per relation, its columns and its unique indexes, and one for the enums of each other schema a column's type comes from (`pg_catalog` included). - A query that fails, the connection included, prints `*** error ***` and the error with `console.log`, then ends the process with `process.exit(1)`, as in Zapatos 6.6.1 ([#183](https://github.com/AlexandreDemartin/pgcast/issues/183)). On a failed query the promise never settles, so a `try`/`catch` around it does not run. - Other errors reject it: a write that fails (`ENOTDIR: not a directory, mkdir 'afile/zapatos'` when `outDir` is a file), a malformed [rule](https://pgcast.dev/generator#generate.schemas). - At run time, a key it doesn't know is ignored: an old `srcMode`, a misspelt `outdir`. In TypeScript, an object literal with one fails to compile (TS2353, or TS2561 with a suggestion for a near-miss such as `outdir`). - `zapatos/generate` also exports `Config` and the functions `generate()` is built from (`tsForConfig()`, `definitionForRelationInSchema()` and others), with no entry here. ## Options Every option is optional. Each one given replaces its default whole: `schemas` without `public` leaves `public` out. | Option | Default | What it sets | |---|---|---| | [`db`](https://pgcast.dev/generator#generate.db) | none: `pg`'s own defaults | the connection | | [`outDir`](https://pgcast.dev/generator#generate.outDir) | `"."` | where the `zapatos` folder goes | | [`outExt`](https://pgcast.dev/generator#generate.outExt) | `".d.ts"` | the extension of every file written | | [`schemas`](https://pgcast.dev/generator#generate.schemas) | `{ "public": { "include": "*", "exclude": [] } }` | the schemas, and the relations of each | | [`unprefixedSchema`](https://pgcast.dev/generator#generate.unprefixedSchema) | `"public"` | the schema whose names need no prefix | | [`columnOptions`](https://pgcast.dev/generator#generate.columnOptions) | `{}` | columns left out of `Insertable` and `Updatable`, or optional | | [`customTypesTransform`](https://pgcast.dev/generator#generate.customTypesTransform) | `"PgMy_type"` | the names of custom types | | [`schemaJSDoc`](https://pgcast.dev/generator#generate.schemaJSDoc) | `true` | a JSDoc comment on each relation and column | | [`customJSONParsingForLargeNumbers`](https://pgcast.dev/generator#generate.customJSONParsingForLargeNumbers) | `false` | the `JSONSelectable` type of `bigint` and `numeric` | | [`typeEnumsAcrossSchemas`](https://pgcast.dev/generator#generate.typeEnumsAcrossSchemas) | `false` | the type of an enum of another schema | | [`enumOrder`](https://pgcast.dev/generator#generate.enumOrder) | `"alphabetical"` | the order of an enum's labels | | [`progressListener`](https://pgcast.dev/generator#generate.progressListener) | `false` | where each step is reported | | [`warningListener`](https://pgcast.dev/generator#generate.warningListener) | `true` | where warnings go | | [`debugListener`](https://pgcast.dev/generator#generate.debugListener) | `false` | where each query and its result go | ### `db` `pg.ClientConfig`: what you'd give `new pg.Pool()`, a `connectionString` or `host`, `user`, `database` and the rest. Without it, `pg` takes its defaults and the `PGHOST`, `PGUSER`, `PGDATABASE`… environment variables. ### `outDir` `string`, default `"."`. The folder that receives the `zapatos` folder, created if needed: `"./src"` writes `src/zapatos/schema.d.ts`. Relative to the current folder. ### `outExt` `string`, default `".d.ts"`. The extension of every file written, the placeholders included: `".ts"` writes `zapatos/schema.ts`, `zapatos/custom/PgPoint.ts`. Only the names change; the content is the same `declare module` text. With `".ts"`, the generator skips the check for the [old layout](https://pgcast.dev/generator#the-old-layout-warning). ### `schemas` `{ [schema]: { include, exclude } }`, each of `include` and `exclude` `"*"` or an array of relation names. The schemas to generate, and which relations of each: tables, views, materialised views and foreign tables. Both keys are required. In `generate()`, a rule without one fails to compile (TS2741); in the JSON, the CLI stops with `TypeError: Cannot read properties of undefined (reading 'indexOf')` ([#301](https://github.com/AlexandreDemartin/pgcast/issues/301)). | Rule | Generates | |---|---| | `{ "include": "*", "exclude": [] }` | every relation of the schema | | `{ "include": ["films", "studios"], "exclude": [] }` | `films` and `studios` | | `{ "include": "*", "exclude": ["seat_bookings"] }` | all but `seat_bookings` | | `{ "include": ["films", "studios"], "exclude": ["studios"] }` | `films`: `exclude` wins | | `{ "include": "*", "exclude": "*" }` | no relation; the schema's enums, still | - Names match exactly, case included. A name of `include` that the schema doesn't have generates nothing, and the generator warns, through [`warningListener`](https://pgcast.dev/generator#generate.warningListener); a schema the database doesn't have gives an empty namespace, and a warning for each name its `include` lists. With `"exclude": "*"`, nothing is checked. - The relations are those the database role can see: a table or view it has no privilege on is left out, as if it did not exist; a materialised view is always there. - The schema's enums are always generated, whatever `include` and `exclude` say. - The relations of a schema other than [`unprefixedSchema`](https://pgcast.dev/generator#generate.unprefixedSchema) are in its namespace: `archive.screenings`, its `Table` type `'archive.screenings'`. - A schema, relation or enum whose name is not a TypeScript identifier (`"box office"`) is written as it is, and `schema.d.ts` does not compile (TS1005) ([#48](https://github.com/AlexandreDemartin/pgcast/issues/48)). A column's name is quoted, and works. With a second schema, `archive`, that holds a `screenings` table of its own: ```json [zapatosconfig.json] { "db": { "connectionString": "{{DATABASE_URL}}" }, "schemas": { "public": { "include": "*", "exclude": [] }, "archive": { "include": "*", "exclude": [] } } } ``` A query names its table with the schema, and its rows are `s.archive.screenings.JSONSelectable[]`: ```ts const past = await db.select('archive.screenings', { film_id: 1000 }).run(pool); ``` Its SQL is [`select()`](https://pgcast.dev/select#select)'s, with the name quoted part by part: `to_jsonb("archive"."screenings".*)`, `FROM "archive"."screenings"`, and the alias `"sq_archive.screenings"`. ### `unprefixedSchema` `string | null`, default `"public"`. The schema whose relations are named without their schema: `films`, its `Table` type `'films'`, so `db.select('films', …)` and `"films"` in the SQL, which Postgres finds through its `search_path`. - `null`: every relation is prefixed, `public.films` its namespace and `'public.films'` its `Table` type. A query names it so, `db.select('public.films', db.all)`, and the SQL too: `FROM "public"."films"`. `db.select('films', db.all)` no longer compiles. - Set it to `null` when a schema has the name of a relation of `public`: a schema `films` beside the table `films` makes two `films` namespaces, which TypeScript merges, and `schema.d.ts` fails with TS2300, `Duplicate identifier 'Table'`. - A schema missing from [`schemas`](https://pgcast.dev/generator#generate.schemas) is allowed: every schema is then prefixed. ### `columnOptions` `{ [table]: { [column]: { insert?, update? } } }`. Changes the `Insertable` and `Updatable` types of a column, as a trigger that fills it would need; nothing else changes, at run time neither. ```json [zapatosconfig.json] { "columnOptions": { "*": { "booked_at": { "insert": "excluded", "update": "excluded" } }, "films": { "tagline": { "insert": "excluded", "update": "excluded" } } } } ``` | Option | Effect on the type | |---|---| | `"insert": "auto"`, `"update": "auto"` | none: what the catalog says, the defaults | | `"insert": "excluded"` | the key leaves `Insertable`: `db.insert('films', { …, tagline: 'x' })` fails to compile (TS2769) | | `"insert": "optional"` | the key is optional in `Insertable`, its type unchanged: `title?: string \| db.Parameter \| db.SQLFragment` | | `"update": "excluded"` | the key leaves `Updatable`: `db.update('seat_bookings', { booked_at: … }, …)` fails to compile (TS2353) | - The table is named as its `Table` type names it: `films`, `archive.screenings`, and `public.films` when [`unprefixedSchema`](https://pgcast.dev/generator#generate.unprefixedSchema) is `null`. `"films"` applies to `public.films` only. - `"*"` is every table of every schema. It is the whole key only, never part of a name, and not for columns. - A table's own entry for a column replaces the `"*"` entry for that column, as a whole: with `"*": { "booked_at": { "insert": "excluded", "update": "excluded" } }` and `"seat_bookings": { "booked_at": { "update": "excluded" } }`, `booked_at` is back in `seat_bookings.Insertable`. - A table with every column excluded gets `[key: string]: never`, as a view that cannot be written to. - Another value, or a misspelt key, is ignored: the column stays `auto`. ### `customTypesTransform` `"PgMy_type" | "my_type" | "PgMyType"`, or in `generate()` a function `(name: string) => string`; default `"PgMy_type"`. The TypeScript name of each type the generator leaves to you: a domain, or a type it can't map, such as `point`. Each one gets a [placeholder](https://pgcast.dev/generator#the-files-it-writes), and the column the type `c.`. | Value | `seat_label` | `point` | `point[]` | |---|---|---|---| | `"PgMy_type"` | `PgSeat_label` | `PgPoint` | `Pg_point` | | `"my_type"` | `seat_label` | `point` | `_point` | | `"PgMyType"` | `PgSeatLabel` | `PgPoint` | `Pg_Point` | | `name => 'Db_' + name` | `Db_seat_label` | `Db_point` | `Db__point` | - The function gets the domain's name, or Postgres's name for the type, `_point` for an array of `point`. - An array of a domain or of such a type is a custom type of its own. - The names are not qualified by schema: the `c` namespace is flat across schemas, so a domain `notes` in `public` and one in `archive` are both `PgNotes`, with one placeholder for the two. - The generator says so through [`warningListener`](https://pgcast.dev/generator#generate.warningListener), as for two names that one transform makes one (`seat-label` and `seat_label`), and keeps the name, which an edited placeholder depends on. ### `schemaJSDoc` `boolean`, default `true`. A JSDoc comment on each relation (its name and kind) and on each column of every interface: its name, its `COMMENT ON COLUMN` if it has one, its Postgres type, then a last line on how it is written. Your editor shows it on hover: ```ts /** * **roles.billing** * * Position in the credits: 1 is top billing * - `int4` in database * - `NOT NULL`, no default */ billing: number; ``` - The type line names a domain with its base type: `` `seat_label` (base type: `text`) in database ``. - The last line is `NOT NULL` or `Nullable` and the default (`` default: `nextval(…)` ``, `no default`, or `identity column` for `GENERATED BY DEFAULT AS IDENTITY`); `Generated column` for a generated or `GENERATED ALWAYS AS IDENTITY` column; `Materialized view column` for one of a materialised view. - A comment over several lines puts each of them behind a `*`, whether it breaks them with `\n`, `\r\n` or `\r`. - Each relation shows its own comments, whatever its kind: Postgres copies none to a partition, an inheriting table or a view, so a view over `roles` shows no comment on `billing`. - On a table or a materialised view, a comment holding `*/` ends the JSDoc there, so `schema.d.ts` usually does not compile (TS1131), until [#48](https://github.com/AlexandreDemartin/pgcast/issues/48) escapes it. Its lines after the first are written as Zapatos 6.6.1 writes them, with no `*`, so one that compiles under 6.6.1, by opening a comment after its `*/`, still does. On a view, a foreign table or a partitioned table, whose column comments Zapatos 6.6.1 never wrote, such a comment is left out ([#48](https://github.com/AlexandreDemartin/pgcast/issues/48)). - A comment, or one of its lines, starting with a JSDoc tag (`@deprecated use lead`) becomes that tag in the hover, on every kind of relation, and swallows the generated lines after it ([#309](https://github.com/AlexandreDemartin/pgcast/issues/309)). `false` writes none, so `schema.d.ts` is shorter. ### `customJSONParsingForLargeNumbers` `boolean`, default `false`. The `JSONSelectable` type of a `bigint` (`int8`) or `numeric` column, for a project that parses JSON with [`enableCustomJSONParsingForLargeNumbers()`](https://pgcast.dev/runtime-config#enableCustomJSONParsingForLargeNumbers): | Column | `false` | `true` | |---|---|---| | `int8` | `number` | `(number \| db.Int8String)` | | `numeric` | `number` | `(number \| db.NumericString)` | - The other interfaces don't change: `Selectable` is `db.Int8String` or `db.NumericString` either way. - `true` also silences the warning such a column gives: "Note: this database has bigint/int8 and/or numeric/decimal columns, for which JSON.parse may lose precision…". ### `typeEnumsAcrossSchemas` `boolean`, default `false`; an opt-in, `true` by default from 7.0 ([Versions](https://pgcast.dev/versions)). The type of a column whose type is an enum of another schema. `false`, as in Zapatos 6.6.1, looks the enum up by name in the schema of the column's relation; `true` looks it up in its own schema, and writes its labels: ```sql CREATE SCHEMA archive; CREATE TYPE archive.print_format AS ENUM ('nitrate', 'safety', 'digital'); CREATE TYPE archive.crew_job AS ENUM ('projectionist', 'archivist'); -- public has a crew_job too CREATE TABLE prints (id int PRIMARY KEY, format archive.print_format NOT NULL, formats archive.print_format[], keeper archive.crew_job); ``` | Column of `prints` | `false` | `true` | |---|---|---| | `format` | `c.PgPrint_format`, an `any` placeholder | `('digital' \| 'nitrate' \| 'safety')` | | `formats` | `c.Pg_print_format \| null`, an `any` placeholder | `('digital' \| 'nitrate' \| 'safety')[] \| null` | | `keeper` | `crew_job \| null`: the labels of `public.crew_job` | `('archivist' \| 'projectionist') \| null` | - The labels are written out, not named: the enum's schema needn't be in [`schemas`](https://pgcast.dev/generator#generate.schemas). In the order of [`enumOrder`](https://pgcast.dev/generator#generate.enumOrder), as for every enum. - A domain over such an enum stays a custom type, `c.`; its new placeholder gets the labels rather than `any`. An existing placeholder is never overwritten. - `false` notes, through [`warningListener`](https://pgcast.dev/generator#generate.warningListener), the columns `true` would change, each with its type: for a domain, the type under it. ### `enumOrder` `"alphabetical"` or `"declared"`, default `"alphabetical"`; an opt-in, `"declared"` by default from 7.0 ([Versions](https://pgcast.dev/versions)). The order of an enum's labels. `"alphabetical"`, as in Zapatos 6.6.1, sorts them; `"declared"` keeps the order the enum declares them in, by which Postgres sorts and compares its values (`ORDER BY`, `<`, `enum_range()`): ```sql CREATE TYPE crew_job AS ENUM ('director', 'writer', 'editor', 'cinematographer', 'composer'); ``` | Type | `"alphabetical"` | `"declared"` | |---|---|---| | `crew_job` | `'cinematographer' \| 'composer' \| 'director' \| 'editor' \| 'writer'` | `'director' \| 'writer' \| 'editor' \| 'cinematographer' \| 'composer'` | | `every.crew_job` | `['cinematographer', 'composer', 'director', 'editor', 'writer']` | `['director', 'writer', 'editor', 'cinematographer', 'composer']` | - A label added by `ALTER TYPE … ADD VALUE … BEFORE` or `AFTER` takes its place in that order, as `enum_range()` gives it. - For TypeScript, only `every.` changes: a union is the same type in any order. With [`typeEnumsAcrossSchemas`](https://pgcast.dev/generator#generate.typeEnumsAcrossSchemas), the labels written out for an enum of another schema, and the new placeholder of a domain over one, take the same order. - Another value stops the generator before any query, and writes nothing: `enumOrder must be 'alphabetical' (the default) or 'declared', not 'declard'`. In TypeScript, it fails to compile (TS2322, or TS2820 with a suggestion for a near-miss such as `"declard"`). - `"alphabetical"` notes, through [`warningListener`](https://pgcast.dev/generator#generate.warningListener), each generated enum whose declared order is not alphabetical. ### `progressListener` `boolean`, or in `generate()` a function `(s: string) => void`; default `false`. `true` prints each step with `console.log`; a function gets each line. ```txt (Re)creating schema folder: src/zapatos/schema.d.ts Writing generated schema: src/zapatos/schema.d.ts Custom type or domain declaration file already exists: src/zapatos/custom/PgPoint.d.ts Custom type or domain declaration file already exists: src/zapatos/custom/PgSeat_label.d.ts Writing custom types file: src/zapatos/custom/index.d.ts ``` ### `warningListener` `boolean`, or in `generate()` a function `(s: string) => void`; default `true`, with `console.log`. `false` silences: - `Writing new custom type or domain placeholder file: src/zapatos/custom/PgPoint.d.ts`, once per new placeholder; - `Note: custom types "archive"."notes" and "public"."notes" are both named PgNotes in TypeScript, so they share one type and one placeholder file, src/zapatos/custom/PgNotes.d.ts. To type them apart, rename one of them in the database.` - once per run for each name that two Postgres types share ([`customTypesTransform`](https://pgcast.dev/generator#generate.customTypesTransform)); when their names differ, it adds that a `customTypesTransform` function can keep them apart; - `Note: the include entry "Films" of schema "public" matches no relation there, so it generates nothing. Check its spelling and case, that the relation is in schema "public", and that the database role has a privilege on it: include names tables, views, materialised views and foreign tables.` - once per run for each entry of an [`include`](https://pgcast.dev/generator#generate.schemas) list that its schema doesn't have; - the note on `bigint` and `numeric` columns, once per process ([`customJSONParsingForLargeNumbers`](https://pgcast.dev/generator#generate.customJSONParsingForLargeNumbers)); - `` Note: public.prints.format (archive.print_format), public.prints.formats (archive.print_format[]) and public.prints.keeper (archive.crew_job) have a type of another schema, which is looked up by name in the schema of their relation: an enum becomes an `any` custom type, or the enum of the same name there. Set typeEnumsAcrossSchemas: true to look it up in its own schema; it will be the default in pgcast 7.0. `` - once per run, without [`typeEnumsAcrossSchemas`](https://pgcast.dev/generator#generate.typeEnumsAcrossSchemas), when a column it would change is generated; - `` Note: every.crew_job lists the labels of enum public.crew_job in alphabetical order, not in their declared order, by which Postgres sorts and compares them. Set enumOrder: 'declared' to keep that order; it will be the default in pgcast 7.0. `` - once per run, with [`enumOrder`](https://pgcast.dev/generator#generate.enumOrder) `"alphabetical"`, when a schema of [`schemas`](https://pgcast.dev/generator#generate.schemas) has an enum whose declared order is not alphabetical; several as `every. lists the labels of enums archive.print_format and public.crew_job`; - the [old layout warning](https://pgcast.dev/generator#the-old-layout-warning). ### `debugListener` `boolean`, or in `generate()` a function `(s: string) => void`; default `false`. Gets each query the generator sends, `>>> query >>>`, its text and its values, then its result as `pg` returns it, as JSON, `<<< result <<<`. The relations are read concurrently: with several, the results come back interleaved, and a query's number can change from one run to the next, so `` is what pairs a result with its query. The unique indexes of `tags`, with `"schemas": { "public": { "include": ["tags"], "exclude": [] } }`, cut after its rows: ```txt >>> query 3 >>> SELECT DISTINCT i.indexname FROM pg_catalog.pg_indexes i JOIN pg_catalog.pg_class c ON c.relname = i.indexname JOIN pg_catalog.pg_index idx ON idx.indexrelid = c.oid AND idx.indisunique WHERE i.tablename = $1 AND i.schemaname = $2 ORDER BY i.indexname + ["tags","public"] <<< result 3 <<< { "command": "SELECT", "rowCount": 1, "oid": null, "rows": [ { "indexname": "tags_pkey" } ], … ``` The result is `pg`'s whole `Result` object: after `rows` come `fields`, its column types, and about seventy lines of `pg`'s own type parsers, the same for every query. ## The files it writes ```txt src/zapatos/ ├── schema.d.ts written again on each run └── custom/ ├── index.d.ts written again on each run ├── PgPoint.d.ts written once, if it is not there └── PgSeat_label.d.ts written once, if it is not there ``` | File | Written | Holds | |---|---|---| | `schema.d.ts` | on each run | `declare module 'zapatos/schema'`: a namespace per relation, its `Table`, `Selectable`, `JSONSelectable`, `Whereable`, `Insertable`, `Updatable`, `UniqueIndex`, `Column`, `OnlyCols`, `SQLExpression` and `SQL`; each enum, and its tuple `every.`; `Schema` and the tuple `AllSchemas`; per schema and over all of them, the unions (`Table`, `Selectable`…) and the tuples `AllBaseTables`, `AllForeignTables`, `AllViews`, `AllMaterializedViews` and `AllTablesAndViews`; the `*ForTable` lookups; the version canary | | `custom/index.d.ts` | on each run, when there is a custom type | an empty `declare module 'zapatos/custom'` | | `custom/.d.ts` | once: never overwritten | a placeholder, `export type PgSeat_label = string;`, yours to edit | - A placeholder holds the `Selectable` type of the domain's base type: `string` for `seat_label`, a `text`; `Date` for a `date` domain; `db.JSONValue` for a `json` or `jsonb` one, with its `import type * as db from 'zapatos/db'`. A type the generator can't map (`point`) gets `any`. How to fill one in: [Custom types](https://pgcast.dev/custom-types). - A base type mapped to another `db.*` type (`int8`, `numeric`, `time`, ranges…) gives a placeholder that does not compile until you edit it: `export type PgTicket_no = db.Int8String;` comes with no import, and `tsc` fails with TS2503, `Cannot find namespace 'db'` ([#184](https://github.com/AlexandreDemartin/pgcast/issues/184)). Add `import type * as db from 'zapatos/db';` above it. - A domain over an enum is the same: `export type PgGenre_d = genre;` fails with TS2304 until you add `import type { genre } from 'zapatos/schema';`. - Nothing is deleted: a placeholder whose type is gone stays, and so does `custom/` once there is no custom type left. - `schema.d.ts` declares `schemaVersionCanary`, `version: 104`. A `zapatos/db` that expects another version fails to compile there: generate again ([`SchemaVersionCanary`](https://pgcast.dev/runtime-config#SchemaVersionCanary)). - Each file is a `declare module` of the `zapatos/*` names, so your `tsconfig.json` must include it: `"include": ["src"]`, with `outDir` `"./src"`. ### The old layout warning Zapatos before 3.0 copied its source into yours. After writing, when `outExt` is not `".ts"`, the generator looks for that layout: `zapatos/schema.ts`, a `zapatos/src` folder, or a file under `zapatos/custom` that does not end in `.d.ts`. If it finds one, it warns, through [`warningListener`](https://pgcast.dev/generator#generate.warningListener), with the steps to convert: ```txt *** IMPORTANT: ZAPATOS NO LONGER COPIES ITS SOURCE TO YOUR SOURCE TREE *** ``` So moving from `outExt: ".ts"` back to `".d.ts"` warns too, until you delete the `.ts` files. Why the types look as they do, a `timestamptz` a `Date` in `Selectable` and a string in `JSONSelectable`: [The catalog](https://pgcast.dev/catalog-types). --- # Explanation Source: https://pgcast.dev/explanation Why pgcast works as it does: the ideas behind it, where its types come from, its versions and its security model. For reading away from the keyboard. --- # Why pgcast Source: https://pgcast.dev/why Why pgcast works as it does: the idea it keeps from Zapatos, what the fork adds to it, and what it will never do. The steps are in the [guides](https://pgcast.dev/guides), every option in the [reference](https://pgcast.dev/reference). ## Just your database, with type safety pgcast is Zapatos, kept going. Zapatos's docs open with its aim: > Postgres and TypeScript are each, individually, fabulous. Zapatos aims to make them work beautifully together. No abstractions, no distractions: just your database, with type safety. Its case is against ORMs. "It is a truth universally acknowledged that ORMs aren't very good," its author writes, and "abstractions that obscure the underlying SQL, or that prioritise ease of switching to another database tomorrow over effective use of this database today, are a source of misery." VS Code's type checking and autocomplete "speed development, prevent bugs, and simplify refactoring", but "talking to the database is a place where they really don't just happen." So Zapatos makes three choices, and pgcast keeps all three: the types come from the database, the SQL stays in sight, and nothing happens that you didn't ask for. ## Types cast from the database Postgres already knows your schema: each column, its type, whether it can be null, whether it has a default, the comment you gave it. The generator asks the catalog (`pg_catalog` and `information_schema`) and writes `zapatos/schema.d.ts`. That file holds the types of each table, view, materialised view and foreign table, a union of strings for each enum, and a placeholder for each domain or type it can't map on its own ([Custom types](https://pgcast.dev/custom-types)). No row type is written by hand, so none can disagree with its table. Hence the name. In Postgres, a cast (`::`) gives a value its type; in English, a cast is also a mould. pgcast's types are cast from your schema, and these docs' examples, fittingly, are about a film's cast. A column the table doesn't have is refused before any query runs: ```ts const films = await db.select('films', { relase_year: 1974 }).run(pool); // error TS2353 ``` The types are as fresh as the last run of the generator. Run it again after each migration, and the compiler points at every query that no longer fits. How each Postgres type becomes a TypeScript type, and why a row as JSON differs from a row from `pg`: [The catalog](https://pgcast.dev/catalog-types). ## The SQL you read is the SQL that runs Each shortcut writes one query, and its reference shows that SQL. Zapatos's docs promise "everyday CRUD queries with no fuss and no surprises, fully and automatically typed". Here, a studio with its films: You write: ```ts const studio = await db.selectOne('studios', { id: 1000 }, { columns: ['name'], lateral: { films: db.select('films', { studio_id: db.parent('id') }, { columns: ['title', 'release_year'], order: { by: 'release_year', direction: 'ASC' }, }), }, }).run(pool); ``` Postgres runs: ```sql SELECT jsonb_build_object($1::text, "name") || jsonb_build_object($2::text, "lateral_films".result) AS result FROM "studios" LEFT JOIN LATERAL ( SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($3::text, "title", $4::text, "release_year") AS result FROM "films" WHERE ("studio_id" = "studios"."id") ORDER BY "release_year" ASC ) AS "sq_films" ) AS "lateral_films" ON true WHERE ("id" = $5) LIMIT $6 ``` Values: `["name", "films", "title", "release_year", 1000, 1]` TypeScript knows: ```ts const studio: (db.JSONOnlyColsForTable<"studios", "name"[]> & db.LateralResult<{ films: db.SQLFragment[], never>; }>) | undefined // that is const studio: { name: string; films: db.JSONOnlyColsForTable<"films", ("release_year" | "title")[]>[]; } | undefined ``` The panel is what you get in your own code: [`.compile()`](https://pgcast.dev/sql-template#compile) gives the same `text` and `values`, and sends nothing. Every value is a parameter, the JSON keys included; every table and column name is quoted. The nesting is Postgres's own work, `jsonb_agg` in a `LEFT JOIN LATERAL`: the studio and its films come back as one JSON value, in one round trip, with no rows to stitch together in JavaScript. TypeScript knows the one column asked for, the films beneath it, and `undefined` when no studio has that `id`: that takes `strict`, as [First query](https://pgcast.dev/first-query#set-up-the-project) sets it. When a shortcut doesn't fit, the [`sql`](https://pgcast.dev/sql-template#sql) template is always there. You write the SQL, it interpolates names, values as parameters, and other fragments, and you type the result with the generated types. The shortcuts are built on it. ## Explicit, never hidden There are no models. A query gives back plain objects, typed: "just (fully-typed) POJOs", in the words of Zapatos's docs. There's no class to extend, no relationship to declare, nothing loaded lazily behind a property, and no state for a later save to write back. Nor is there a default connection: each `.run()` takes the pool or client to run on, as "Zapatos is always explicit about what client or pool is running your query". A transaction starts only when you call [`transaction()`](https://pgcast.dev/transaction#transaction). It sends `START TRANSACTION` at the isolation level you chose, then `COMMIT` or `ROLLBACK`, and runs your callback again after a serialisation failure or a deadlock ([Retries](https://pgcast.dev/retries)). The same goes for your own functions. Zapatos's docs advise against giving a function's client argument the pool as its default: "I find that way it's too easy to accidentally issue queries outside of transactions." So a function of yours takes its client with no default, and each caller passes the pool or its transaction's client. One that only runs queries takes a [`Queryable`](https://pgcast.dev/sql-template#Queryable). One that opens a transaction takes the pool or the client of its level, `pg.Pool | db.TxnClientForSerializable` for [`serializable()`](https://pgcast.dev/transaction#serializable). A `Queryable` would also take the client of a transaction at a lower level, and the function would run inside it, at that level ([Book inside a larger transaction](https://pgcast.dev/retries#book-inside-a-larger-transaction)). ## What pgcast adds pgcast starts from Zapatos 6.6.1. Before changing it, the fork pinned what 6.6.1 does, with tests, and that [characterisation](https://github.com/AlexandreDemartin/pgcast/blob/master/docs/CHARACTERISATION-REPORT.md) found dozens of surprises, 7 of them about security, all fixed in 6.6.2. A finding pgcast acts on takes one of three paths ([decision 003](https://github.com/AlexandreDemartin/pgcast/blob/master/docs/decisions/003-fixes-and-deprecation.md)): | Path | When | What changes for you | Example | |---|---|---|---| | Fix | a security hole; or Postgres or the type already refused the call; or the result contradicts the function's name, type or docs | the call does what it says, or throws an error that says what to do, before any query | [`update()`](https://pgcast.dev/insert#update) with no columns throws `nothing to update: an Updatable needs at least one column`, where Postgres refused `SET () = ROW()` | | Opt-in | the result is valid, and code may rely on it | nothing until you turn the option on; the old path warns once per process | [`undefinedMeansMissing`](https://pgcast.dev/runtime-config#Config.undefinedMeansMissing): an `undefined` value is a missing key, not `NULL`; warning `PGCAST_DEP002` | | Document | Postgres works that way, or it was meant | nothing: the docs say so | [`isNotIn()`](https://pgcast.dev/conditions#conditions.isNotIn) with a `NULL` in its list matches no row, as Postgres reads `NOT IN` | The security fixes came first, in 6.6.2 ([Security](https://pgcast.dev/security)): - no name (a table, a column, a key, an alias) and no cast of `param()` can close its quotes; - a plain value in [`upsert()`](https://pgcast.dev/insert#upsert)'s `updateValues` is a parameter; - the keywords of a lock, `truncate()` or an isolation level, and the name of an aggregate, are checked before any query. The opt-ins that warn, [`undefinedMeansMissing`](https://pgcast.dev/runtime-config#Config.undefinedMeansMissing), [`upsertRequiresSameKeys`](https://pgcast.dev/runtime-config#Config.upsertRequiresSameKeys) and [`transactionRequiresCommit`](https://pgcast.dev/runtime-config#Config.transactionRequiresCommit), and the generator's [`typeEnumsAcrossSchemas`](https://pgcast.dev/generator#generate.typeEnumsAcrossSchemas) and [`enumOrder`](https://pgcast.dev/generator#generate.enumOrder), become the defaults in 7.0, and their old paths go in 8.0. [`selectExactlyOneChecksForSecondRow`](https://pgcast.dev/runtime-config#Config.selectExactlyOneChecksForSecondRow), which never warns, stays off. Until 7.0, 6.x is a drop-in for Zapatos 6.6.1 ([Versions](https://pgcast.dev/versions)). Zapatos 6.6.1 shipped no test suite of its own (its first consumer's tests exercised "most of the code base at least a little"); its CI linted and built. Its roadmap called one "kind of indispensable", one that tests "not just returned values but also inferred types". pgcast's tests check the SQL each call compiles, run it on Postgres 14, 16 and 18, check the types, accepted and refused, and measure what they cost the compiler. These docs are the latest addition. Every example compiles against the library at build: its SQL comes from its own `.compile()`, its type from TypeScript, and an example that no longer fits fails the build. What stays is everything you import. The module names are still `zapatos/db` and `zapatos/schema`, the API is Zapatos's, and so is the behaviour, but for the fixes: installed under the `zapatos` alias, pgcast needs no change to your imports ([Switch from Zapatos](https://pgcast.dev/switching)). > **Coming** > > The aim is Zapatos going the last mile: foreign-key joins, the conditions Postgres offers, CTEs, functions and raw SQL, each with a typed form, so that real code never falls back on `db.raw`, a hand-written row type or a cast. Each comes under the same rules, and the [roadmap](https://github.com/AlexandreDemartin/pgcast#roadmap) gives their order. ## What pgcast will not do Some changes are refused whatever they'd gain, because each would take away something above: | pgcast won't add | Because | |---|---| | Models, entity classes, relationships declared in TypeScript | a result is "just (fully-typed) POJOs"; the relationships are in the database | | Lazy loading, an identity map, change tracking, a query cache | each is hidden state: a query would run, or not, without your asking | | A method chain that builds a query (`.where().join()`) | it hides the SQL; a shortcut writes one query, and the `sql` template yours | | An implicit, default or global client; a transaction you didn't open | "Zapatos is always explicit about what client or pool is running your query" | | A connection pool, or migrations | `pg` makes the pool easy, and "Zapatos doesn't handle schema migrations": your migration tool does | | A layer to switch databases | it costs "effective use of this database today" | | A type the database doesn't state | a guessed type drifts from the schema | | A value pasted into the SQL | values are parameters; only [`raw()`](https://pgcast.dev/sql-template#raw) writes a string into the SQL as it is, and it returns a `DangerousRawString` | | SQL that depends on state, or changes with the values, undocumented | you could no longer read the query from the call | | Complexity in the core for a niche case, or a gain lost in network latency | "Small is beautiful": it goes to a separate package, or behind an opt-in | ## Further reading - The entries this page talks about: [`select()`](https://pgcast.dev/select#select), [`selectOne()`](https://pgcast.dev/select#selectOne), the [`sql`](https://pgcast.dev/sql-template#sql) template and [`.compile()`](https://pgcast.dev/sql-template#compile), [`transaction()`](https://pgcast.dev/transaction#transaction), [`Config.undefinedMeansMissing`](https://pgcast.dev/runtime-config#Config.undefinedMeansMissing), [`Config.upsertRequiresSameKeys`](https://pgcast.dev/runtime-config#Config.upsertRequiresSameKeys), [`Config.transactionRequiresCommit`](https://pgcast.dev/runtime-config#Config.transactionRequiresCommit) and [`Config.selectExactlyOneChecksForSecondRow`](https://pgcast.dev/runtime-config#Config.selectExactlyOneChecksForSecondRow). - The guides: [Joins](https://pgcast.dev/lateral), [Retries](https://pgcast.dev/retries), [Switch from Zapatos](https://pgcast.dev/switching). - Zapatos's own [docs](https://jawj.github.io/zapatos/), for 6.6.1; what pgcast changes, in the [changelog](https://pgcast.dev/changelog). --- # The catalog Source: https://pgcast.dev/catalog-types Every type pgcast gives you is read from your database; none is written by hand. This page is about how that works: what the generator asks Postgres, why each table comes out as five types rather than one, and how the type of a query's result follows the options you give it. **At a glance** - The generator reads your database's catalog and writes `zapatos/schema.d.ts`; your code is checked against that file alone. - Each table comes out five times, once per context a value travels in: `Selectable`, `JSONSelectable`, `Whereable`, `Insertable`, `Updatable`. - A shortcut's result is JSON that Postgres builds, and its type is worked out from the call, never declared. ## Postgres describes itself A Postgres database knows its own shape, and keeps it in its catalog: the system tables of `pg_catalog`, and the standard views of `information_schema` built over them. When you run `npx pgcast`, the generator asks those tables a handful of questions for each schema, and writes the answers down as TypeScript in `zapatos/schema.d.ts`. It doesn't connect again: from then on, your code is checked against that file alone. | It reads | From | It becomes | |---|---|---| | each relation, its kind, and whether it takes an `INSERT` | `information_schema.tables`, `pg_matviews` | a namespace per table, view, foreign table or materialised view | | each column's type, or its domain and the domain's base type | `information_schema.columns`; `pg_attribute` and `pg_type` for a materialised view | the column's TypeScript type, once per context ([below](https://pgcast.dev/catalog-types#one-column-five-types)) | | whether the column may be `NULL` | `is_nullable`; `attnotnull` for a materialised view | `\| null`, and optional in `Insertable` | | its default, identity or generated value | `column_default`, `identity_generation`, `is_generated` | optional in `Insertable`; a `GENERATED ALWAYS` column is left out of `Insertable` and `Updatable` | | its `COMMENT ON COLUMN` | `pg_description` | the JSDoc your editor shows for the column, with its Postgres type and default | | each enum and its labels | `pg_enum` | a union of strings, and the tuple `every.` | | each unique index | `pg_indexes`, `pg_namespace`, `pg_class`, `pg_index` | `UniqueIndex`, the names [`constraint()`](https://pgcast.dev/insert#constraint) takes | To watch the queries themselves, set [`debugListener`](https://pgcast.dev/generator#generate.debugListener). A domain, such as `seat_label`, and a type the generator has no mapping for, such as `point`, each get a placeholder in `zapatos/custom/`, yours to fill in ([Custom types](https://pgcast.dev/custom-types)). And what the catalog can't see, such as a column a trigger fills in, you say in the config: [`columnOptions`](https://pgcast.dev/generator#generate.columnOptions) leaves it out of `Insertable` or `Updatable`, or makes it optional in `Insertable`. ### What the generator doesn't read The generator reads names, types, nullability and defaults. Postgres also keeps each foreign key and `CHECK` constraint, in `pg_constraint`, but Zapatos 6.6.1 never read that table, nor the SQL inside views and functions; pgcast doesn't yet either. So `ratings.stars`, `CHECK (stars BETWEEN 1 AND 5)`, is a `number`; a `seat_label` matching `'^[A-Z][0-9]{1,2}$'` is a `string`; and nothing tells TypeScript that `films.studio_id` points at a studio. A view is typed as Postgres reports it, and Postgres reports every column of a view as nullable, whatever the tables underneath say, unless its type is a `NOT NULL` domain. A view of the `id` and `title` of `films` types both as `| null`, and gets an `Insertable` with every column optional, since Postgres can write through it. A view over a join, which it can't write through, gets an `Insertable` and an `Updatable` that take nothing (`[key: string]: never`). ```sql CREATE VIEW film_titles AS SELECT id, title FROM films; ``` ```ts // zapatos/schema.d.ts, its JSDoc left out export namespace film_titles { export type Table = 'film_titles'; export interface Selectable { id: number | null; title: string | null; } … } ``` A materialised view keeps no `NOT NULL` at all, not even a domain's, so its columns are nullable too. For views, pgcast keeps to what Postgres states, as Zapatos did: a nullability the database doesn't state would be a guess, and a guess is the one thing a generated type mustn't be. Some gaps in what the generator reads are known, and described here as they are today: - Enum labels come out in alphabetical order: `crew_job` is `'cinematographer' | 'composer' | 'director' | 'editor' | 'writer'`, not the order of its `CREATE TYPE`, unless [`enumOrder`](https://pgcast.dev/generator#generate.enumOrder) is `"declared"`. - A column comment holding `*/` ends the JSDoc early on a table or a materialised view; on a view, a foreign table or a partitioned table, it is left out ([#48](https://github.com/AlexandreDemartin/pgcast/issues/48)). - An enum of another schema becomes an `any` placeholder, or takes the labels of the enum of the same name in the relation's schema, unless [`typeEnumsAcrossSchemas`](https://pgcast.dev/generator#generate.typeEnumsAcrossSchemas) is on. > **Coming** > > Reading more of the catalog, each as an opt-in: foreign keys, for laterals that know their join ([#54](https://github.com/AlexandreDemartin/pgcast/issues/54)); a domain's `CHECK (VALUE = ANY (ARRAY[…]))` as a union of its values ([#55](https://github.com/AlexandreDemartin/pgcast/issues/55)); and, as a separate tool, the result types of hand-written SQL ([#62](https://github.com/AlexandreDemartin/pgcast/issues/62)). ## One column, five types A value isn't the same thing everywhere it goes. A `timestamptz` read by a plain `SELECT` reaches you as a `Date`, because `pg` parses it; read through JSON, it's a string; in a filter it may be either, or some SQL; in a new row, it may be required, or left to its default. So the generator writes each table five times, once per context, each column typed for that context. Here are two columns of `screenings`, from the generated schema: | Type | What it describes | `starts_at`, `timestamptz NOT NULL` | `subtitled`, a nullable `boolean` | |---|---|---|---| | `Selectable` | a row as `pg` returns it, from your own `sql` query | `Date` | `boolean \| null` | | `JSONSelectable` | a row as the shortcuts return it | `TimestampTzString` | `boolean \| null` | | `Whereable` | a filter: every key optional | `TimestampTzString \| Date` | `boolean`, never `null` | | `Insertable` | a new row | required: `TimestampTzString \| Date` | optional: `boolean`, `null` or `DEFAULT` | | `Updatable` | changes to a row: every key optional | `TimestampTzString \| Date` | `boolean`, `null` or `DEFAULT` | Wherever a value goes in, it may also be a [`param()`](https://pgcast.dev/sql-template#param) or a [`sql`](https://pgcast.dev/sql-template#sql) fragment, and in a filter a [`parent()`](https://pgcast.dev/sql-template#parent) column. A `Whereable` refuses `null` because `"subtitled" = NULL` is never true in SQL. Given one anyway (the type lets it through when `strictNullChecks` is off), the filter compiles to `"subtitled" = $1`, with `null` as `$1`, and finds nothing. [`conditions.isNull`](https://pgcast.dev/conditions#conditions.isNull) writes the `IS NULL` you meant. The first does not compile (TS2322, with `strict` on); the second is the filter you meant: ```ts db.select('screenings', { subtitled: null }); // error TS2322 db.select('screenings', { subtitled: db.conditions.isNull }); ``` A domain or custom type is the exception: its placeholder is one type, used in all five contexts. That suits `seat_label`, a string everywhere, but not a domain over `int8`, `numeric`, `bytea` or a date, whose JSON differs from what `pg` returns ([#194](https://github.com/AlexandreDemartin/pgcast/issues/194)). The same row, asked for twice, shows the first two side by side: ```ts const [row] = await db.sql` SELECT * FROM ${'screenings'} WHERE ${{ id: 1000 }}`.run(pool); const json = await db.selectExactlyOne('screenings', { id: 1000 }).run(pool); row.starts_at; // a Date: pg parsed the timestamptz json.starts_at; // a string: to_jsonb() wrote it ``` In a session whose `TimeZone` is `UTC`, the first is the `Date` `2030-03-01T19:30:00.000Z`, the second the string `'2030-03-01T19:30:00+00:00'`. In the `sql` query, `s.screenings.Selectable[]` is your word for what `SELECT *` returns; the shortcut works out its own. ## Why the shortcuts return JSON Each shortcut is one query, and Postgres builds its result itself: each row as one JSON value, with `to_jsonb()` or `jsonb_build_object()`, each `lateral` nested inside it, and for `select()` the rows gathered into one array with `jsonb_agg()`. That's how `select()` brings back films with their cast already in place, in one round trip, with nothing to assemble in JavaScript afterwards. The price is that every value travels as JSON, and JSON has no dates, no bytes and one kind of number. `JSONSelectable` says what arrives instead: - a date or a time, as the ISO 8601 string `to_jsonb()` writes; a `timestamptz` with the session's offset, `+00:00` in `UTC` and `+05:30` in `Asia/Kolkata`; - a `bytea`, as its hex string (with the default `bytea_output`), `'\xdeadbeef'`, where `pg` gives a `Buffer`; - an `int8` or a `numeric`, as a JSON number, where `pg` gives a string. A large one loses precision as `JSON.parse()` reads it: the `int8` `9007199254740993` arrives as `9007199254740992`. The generator warns when a schema has such columns, and [`customJSONParsingForLargeNumbers`](https://pgcast.dev/generator#generate.customJSONParsingForLargeNumbers) types them for a parser that keeps them whole. Zapatos returned JSON from every shortcut that returns rows, even those that could do without it, "in the interests of consistency", as its docs put it: one shortcut's row is the same type as another's. And it typed the strings rather than converting them behind your back: [`TimestampTzString`](https://pgcast.dev/runtime-config#TimestampTzString) and its siblings check a string's shape, and [`toDate()`](https://pgcast.dev/runtime-config#toDate) turns one into a `Date` when you choose to, saying in which time zone to read a string that carries none ([Dates and times](https://pgcast.dev/runtime-config#dates-and-times)). ## How a result type follows the query A shortcut's result type is never declared: TypeScript works it out from the call. - The table gives its `JSONSelectable`. - `columns` keeps only those keys, as `returning` does for [`insert()` and the other writes](https://pgcast.dev/insert#insert-options). - Each `extras` key adds the type of its column, or of its `sql` fragment. - Each `lateral` key adds the result type of its subquery, worked out the same way, one level down. A `lateral` given as [one subquery](https://pgcast.dev/lateral#keep-only-the-far-side) replaces the row with its result instead. Then the shortcut sets the shape: an array for `select()`, the row or `undefined` for `selectOne()`, the row for `selectExactlyOne()`, a number for `count()`. Hover `films` to see what came of this one: You write: ```ts const films = await db.select('films', { studio_id: 1000 }, { columns: ['title'], extras: { year: 'release_year', hours: db.sql`round(${'runtime_minutes'} / 60.0, 1)`, }, lateral: { studio: db.selectOne('studios', { id: db.parent('studio_id') }, { columns: ['name'] }), roles: db.count('roles', { film_id: db.parent('id') }), }, }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT jsonb_build_object($1::text, "title") || jsonb_build_object($2::text, "release_year", $3::text, round("runtime_minutes" / 60.0, 1)) || jsonb_build_object($4::text, "lateral_roles".result, $5::text, "lateral_studio".result) AS result FROM "films" LEFT JOIN LATERAL ( SELECT count(*) AS result FROM "roles" WHERE ("film_id" = "films"."id") ) AS "lateral_roles" ON true LEFT JOIN LATERAL ( SELECT jsonb_build_object($6::text, "name") AS result FROM "studios" WHERE ("id" = "films"."studio_id") LIMIT $7 ) AS "lateral_studio" ON true WHERE ("studio_id" = $8) ) AS "sq_films" ``` Values: `["title", "year", "hours", "roles", "studio", "name", 1, 1000]` TypeScript knows: ```ts const films: (db.JSONOnlyColsForTable<"films", "title"[]> & db.ExtrasResult<"films", { year: "release_year"; hours: db.SQLFragment; }> & db.LateralResult<{ studio: db.SQLFragment | undefined, never>; roles: db.SQLFragment; }>)[] // that is const films: { title: string; year: number; hours: number; studio: db.JSONOnlyColsForTable<"studios", "name"[]> | null; roles: number; }[] ``` Three things in that type are worth a closer look: - **`studio` may be `null`, not `undefined`.** `selectOne()` can find nothing, and in a nested query nothing is JSON's `null`, so the type turns the `undefined` into `null`. - **`hours` is a `number` because you said so.** A `sql` fragment's type is its second type argument, here `number`, and pgcast doesn't parse your SQL to check it ([Type arguments](https://pgcast.dev/sql-template#type-arguments)). `year`, a column, takes its type from the schema. - **`roles` is a `number`**, the type of `count()`, though Postgres counts in `int8`: like Zapatos, pgcast assumes no count comes near `Number.MAX_SAFE_INTEGER`. Here `studio` is never missing, as `films.studio_id` is `NOT NULL REFERENCES studios`; but the generator doesn't read foreign keys, so say so yourself with `selectExactlyOne()`, and `studio` loses its `| null`. Zapatos's docs list this among their limitations, and [#54](https://github.com/AlexandreDemartin/pgcast/issues/54) plans to read it from the catalog. All of this needs `strictNullChecks`, or `strict`. Without it, `columns`, `extras`, `lateral` and `returning` leave the type at the table's full `JSONSelectable`, and `films[0].studio` doesn't compile ([`select()` options](https://pgcast.dev/select#options)). ## What pgcast kept, and what it changes The generator is Zapatos 6.6.1's, its version canary still at 104 ([`SchemaVersionCanary`](https://pgcast.dev/runtime-config#SchemaVersionCanary)). Its fixes change `schema.d.ts` where 6.6.1's was wrong: a table's `UniqueIndex` lists only its own unique indexes, and each line of a multi-line column comment gets its `*` ([changelog](https://pgcast.dev/changelog)). What pgcast changes on the way in is in the run-time library, not in the generated types. [`raw()`](https://pgcast.dev/sql-template#raw), [`cols()`](https://pgcast.dev/sql-template#cols) and [`vals()`](https://pgcast.dev/sql-template#vals) no longer type-check as a value of a `Whereable`, `Insertable` or `Updatable`, where 6.6.1 took them for a `Parameter`, an object of the same shape (`{ value }`). And an `undefined` value can mean a missing key, with [`undefinedMeansMissing`](https://pgcast.dev/runtime-config#Config.undefinedMeansMissing) ([An `undefined` value](https://pgcast.dev/insert#undefined-values)). Changes to the output itself come the same way, each as a fix or an opt-in of its own: while [`typeEnumsAcrossSchemas`](https://pgcast.dev/generator#generate.typeEnumsAcrossSchemas) and [`enumOrder`](https://pgcast.dev/generator#generate.enumOrder) are off, the rest of `schema.d.ts` is 6.6.1's ([Versions](https://pgcast.dev/versions)). What pgcast refuses is a second source of truth: a model class, a schema written in TypeScript, a type declared beside the table it describes. The database has the last word, and the cost is a habit: run the generator again after each migration, and let TypeScript point at the queries that no longer fit ([First query](https://pgcast.dev/first-query#when-the-schema-changes)). ## Further reading - The entries this page talks about: [`select()` options](https://pgcast.dev/select#options), [`JSONOnlyColsForTable`](https://pgcast.dev/select#JSONOnlyColsForTable), [`ExtrasResult`](https://pgcast.dev/select#ExtrasResult), [`LateralResult`](https://pgcast.dev/select#LateralResult), [`RunResultForSQLFragment`](https://pgcast.dev/sql-template#RunResultForSQLFragment), the [value types](https://pgcast.dev/runtime-config#value-types). - The generator, its options and the files it writes: [Generator](https://pgcast.dev/generator). - Filling in a placeholder: [Custom types](https://pgcast.dev/custom-types). --- # Versions Source: https://pgcast.dev/versions Why pgcast's version numbers carry on from Zapatos's, how it decides whether a fix may change what your code gets back, and what a `PGCAST_DEP00n` warning is asking of you. ## Carrying on from 6.6.1 pgcast starts where Zapatos 6.6.1 stopped, so its first version is 6.6.2, not 1.0. A number that carries on says exactly that: this is the next version of the library you already have. The module names are still `zapatos/db` and `zapatos/schema`, and the generated schema keeps 6.6.1's [canary](https://pgcast.dev/runtime-config#SchemaVersionCanary), so a project switches by changing its dependency, not its code ([Switch from Zapatos](https://pgcast.dev/switching)). The numbers follow [SemVer](https://semver.org/), read from the side of a project on 6.6.1: - A patch fixes: 6.6.2 closes the security holes that pgcast's characterisation of 6.6.1 found. - A minor adds: 6.7.0 brings the opt-ins below, with fixes and new features. - Only a major may change a result that working code can rely on, so a range such as `^6.6.2` never brings one of those. Zapatos broke things in its majors too, each at once: 1.0 renamed the transaction types, 2.0 made the last argument of `upsert()` an options object, 6.0 put the schema in the names of tables and enums outside `public`. Each came with notes on what to rewrite. pgcast spreads a break over two majors instead, and warns you ahead of the first. ## Four answers to a surprise Characterising Zapatos 6.6.1 turned up dozens of surprises ([the report](https://github.com/AlexandreDemartin/pgcast/blob/master/docs/CHARACTERISATION-REPORT.md)): SQL that Postgres refuses, results that contradict a function's name, holes in the escaping. About half of them can't be fixed without changing a result that is valid today ([its Summary](https://github.com/AlexandreDemartin/pgcast/blob/master/docs/CHARACTERISATION-REPORT.md#summary) keeps the count), and [decision 003](https://github.com/AlexandreDemartin/pgcast/blob/master/docs/decisions/003-fixes-and-deprecation.md) sorts those by a single question: could working code depend on what it does now? The report leaves a few as they are, each with its reason. | Class | When | What changes for you | Example | |---|---|---|---| | Fix | Postgres or the type already refused it, or it contradicts the function's name; every security hole | at once; the changelog lists each result that changes | [`not()`](https://pgcast.dev/conditions#conditions.not) negates its whole condition: `` not(sql`a OR b`) `` gave `(NOT a OR b)`, which Postgres reads `(NOT a) OR b` | | Opt-in | the result is valid, and code may rely on it | nothing until you opt in; the old behaviour warns | an `undefined` value sent as `NULL`, until [`undefinedMeansMissing`](https://pgcast.dev/runtime-config#Config.undefinedMeansMissing) makes it a missing key | | Document | Postgres does it, or it is meant | nothing; the docs say so | [`isNotIn()`](https://pgcast.dev/conditions#conditions.isNotIn) with a `NULL` in its list matches no row: that is how Postgres reads `NOT IN` | | Leave | [the report](https://github.com/AlexandreDemartin/pgcast/blob/master/docs/CHARACTERISATION-REPORT.md) gives the reason: the type protects, the case is rare, or a fix would need a guess | nothing | an `int[][]` column is typed `number[]`: Postgres doesn't enforce the dimensions, and stores one, two or three in it alike | A fix can still stop a call that only worked through the bug. 6.6.2's escaping fixes are an example: a column name such as `id" AS "x` used to close its quotes and add an alias, and now names a column that doesn't exist. The changelog lists every such call, with what to write instead. One opt-in will stay off for good. [`selectExactlyOne()`](https://pgcast.dev/select#selectExactlyOne) asks Postgres for one row only, so it means "at least one", and Zapatos's own docs nest it in a `lateral`, where looking for a second row would duplicate the parent row. Its option, [`selectExactlyOneChecksForSecondRow`](https://pgcast.dev/runtime-config#Config.selectExactlyOneChecksForSecondRow), looks for a second row only in a query run on its own, never in one nested in another, never warns, and never becomes the default. ## Why an opt-in, not a breaking change Take an `update()` with an `undefined` value: ```ts await db.update('films', { tagline: undefined }, { id: 1002 }).run(pool); ``` Zapatos 6.6.1 sends `SET ("tagline") = ROW($1)` with `null`, and the film loses its tagline. That may be a bug, a key copied from a form field that was never filled in; it may also be how a project clears a column. pgcast can't tell which, and neither can your compiler: had 6.7 simply left the column out, that code would still compile, still run, and quietly stop clearing taglines. So the new reading arrives as an option, off by default, and the old one warns when it runs. You switch when you're ready, one option at a time, with your tests to show what moved. The pattern is borrowed: Node's own deprecation cycle (documented, then a warning at run time, then removed in a major), React Router's future flags, and TypeScript's `--strict` family, one option per safer behaviour. The cost is a few more keys in [`Config`](https://pgcast.dev/runtime-config#Config), and two paths in the code until 8.0. Each key is optional, so a `Config` written for Zapatos 6.6.1 still compiles. ## Reading a warning Call that `update()` on 6.7 and Node prints, once: ```txt (node:4242) [PGCAST_DEP002] DeprecationWarning: An undefined value in insert(), update() or upsert() is sent as NULL. Set undefinedMeansMissing: true (db.setConfig) to treat it as a missing key instead (DEFAULT in insert(), left out of update() and upsert()); it will be the default in pgcast 7.0. (Use `node --trace-deprecation ...` to show where the warning was created) ``` In brackets is the code, one per deprecated behaviour of the runtime, there to search for: [Opt-ins and warnings](https://pgcast.dev/runtime-config#opt-ins-and-warnings) lists them, with what sets each off. Then the message says what still happens, which option changes it, and that 7.0 will make it the default. Nothing else changes: the query runs as it did in 6.6.1, unless Node runs with `--throw-deprecation`, which ends the process ([Opt-ins and warnings](https://pgcast.dev/runtime-config#opt-ins-and-warnings)). The warning comes when you call the shortcut, before `.run()`, and only if the old behaviour is actually used: an `update()` with no `undefined` in it never warns. It comes once per process for each code, so it tells you that the old path ran, not where nor how often. `node --trace-deprecation` adds the stack, down to the line that called `update()`; for the rest, turn the option on and let your tests find what changes. `PGCAST_DEP003` comes later: once `transaction()` has sent `COMMIT` and Postgres has answered `ROLLBACK`, just before the promise resolves. Where `undefined` meant "clear it", `null` says so under either setting, and never warns. ## 7.0 and 8.0 7.0 turns on by default the opt-ins that decision 003 lists: `upsertRequiresSameKeys`, `undefinedMeansMissing`, `transactionRequiresCommit` and those 6.7 still brings, except `selectExactlyOneChecksForSecondRow`, which stays off. Decision 003 makes that the whole of 7.0's break, and the warnings announce it, so a project that has turned every option on, and sees no `PGCAST_DEP` warning nor generator note, has nothing to change. The old behaviours go in 8.0. The generator gets opt-ins of its own in 6.7: [`typeEnumsAcrossSchemas`](https://pgcast.dev/generator#generate.typeEnumsAcrossSchemas), which types an enum of another schema by its labels, and [`enumOrder`](https://pgcast.dev/generator#generate.enumOrder), which keeps an enum's labels in their declared order. Their warnings are notes through [`warningListener`](https://pgcast.dev/generator#generate.warningListener), not a `DeprecationWarning`, and come only when your schema has something they would change, once per run: `typeEnumsAcrossSchemas`'s names each column, `enumOrder`'s each enum. > **Coming** > > pgcast isn't on npm yet. 6.6.2 is written; 6.7.0, which brings the opt-ins, is in progress: [its milestone](https://github.com/AlexandreDemartin/pgcast/milestone/1). ## Further reading - The entries this page talks about: [`Config.upsertRequiresSameKeys`](https://pgcast.dev/runtime-config#Config.upsertRequiresSameKeys), [`Config.undefinedMeansMissing`](https://pgcast.dev/runtime-config#Config.undefinedMeansMissing), [`Config.transactionRequiresCommit`](https://pgcast.dev/runtime-config#Config.transactionRequiresCommit), [`Config.selectExactlyOneChecksForSecondRow`](https://pgcast.dev/runtime-config#Config.selectExactlyOneChecksForSecondRow), [Opt-ins and warnings](https://pgcast.dev/runtime-config#opt-ins-and-warnings), [`typeEnumsAcrossSchemas`](https://pgcast.dev/generator#generate.typeEnumsAcrossSchemas), [`enumOrder`](https://pgcast.dev/generator#generate.enumOrder), [An `undefined` value](https://pgcast.dev/insert#undefined-values), [`conditions.not()`](https://pgcast.dev/conditions#conditions.not), [`conditions.isNotIn()`](https://pgcast.dev/conditions#conditions.isNotIn), [`selectExactlyOne()`](https://pgcast.dev/select#selectExactlyOne), [`SchemaVersionCanary`](https://pgcast.dev/runtime-config#SchemaVersionCanary). - The decision behind it: [decision 003](https://github.com/AlexandreDemartin/pgcast/blob/master/docs/decisions/003-fixes-and-deprecation.md), and every change by version in the [changelog](https://pgcast.dev/changelog). --- # Security Source: https://pgcast.dev/security What keeps a caller's input from changing the SQL pgcast sends, what it can't guard against, and how to report a flaw. The steps are in [Raw SQL](https://pgcast.dev/raw-sql); this page is the why. **At a glance** - Values travel as parameters, beside the SQL text; names are quoted; keywords come from a closed list. - `raw()` is the one door from a string to code: give it only your own constants. - Valid SQL can still do the wrong thing: a caller's string as a name, an empty filter, an object that is not a filter. ## Values never enter the SQL Every value pgcast sends, from a filter, an `insert()` or a [`param()`](https://pgcast.dev/sql-template#param), becomes a numbered parameter, `$1`, and travels beside the SQL text rather than inside it. Postgres parses the text first and binds the values after: by then, a value can only be data. pgcast never escapes a value, so there is no escaping to get wrong. This is Zapatos's foundation, kept as it was. You write: ```ts function filmsTitled(title: string) { // the title comes from the caller return db.select('films', { title }); } const films = await filmsTitled("x' OR 'a' = 'a").run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("films".*) AS result FROM "films" WHERE ("title" = $1) ) AS "sq_films" ``` Values: `["x' OR 'a' = 'a"]` TypeScript knows: ```ts const films: films.JSONSelectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` Written into the text, that title would close its string and match every film: `("title" = 'x' OR 'a' = 'a')`. As `("title" = $1)`, it is a title no film has, and the query returns none. The split has a second effect. node-postgres sends a query with values through Postgres's extended protocol, where a query is one statement: `SELECT $1::int; SELECT 2` is refused with "cannot insert multiple commands into a prepared statement" (42601). A query with no values, and no `.prepared()` name, goes as plain text, where a `;` can start a second statement. That is why the holes below were at their worst in a query without parameters. ## Names are quoted, not parameters A parameter stands for a value, never for a table or a column: `SELECT * FROM $1` is a syntax error (42601). So names go into the text, and pgcast guards them twice: 1. **The type**: `Table` and `Column` are unions of the names the generator read from the catalog, so a name you write is one the database has. 2. **The quoting**, because the type is gone at run time and names can arrive without passing it: the keys of a request's JSON body, given to `select()` as its filter, are column names. Each string pgcast writes as a name is double-quoted and split on `.`, any `"` in it doubled unless it is already a well-formed quoted name (`'"title"'` is `"title"`, `'a""b'` stays `"a""b"`). So it can never close its quotes: You write: ```ts function filmsMatching(filter: any) { // a request's JSON body, as it came return db.select('films', filter); } const films = await filmsMatching({ 'title" = "title" OR TRUE OR "title': 'x' }).run(pool); ``` Postgres runs: ```sql SELECT coalesce(jsonb_agg(result), '[]') AS result FROM ( SELECT to_jsonb("films".*) AS result FROM "films" WHERE ("title"" = ""title"" OR TRUE OR ""title" = $1) ) AS "sq_films" ``` Values: `["x"]` TypeScript knows: ```ts const films: films.JSONSelectable[] // that is const films: { genres: string[]; id: number; release_year: number; runtime_minutes: number; studio_id: number; tagline: string | null; title: string; }[] ``` The key stays one name, a column that does not exist (42703). Zapatos 6.6.1 trusted the type here (its docs warned never to override it), and wrote that key as `("title" = "title" OR TRUE OR "title" = $1)`, which returns every film; without parameters, such a name could run a second statement. The type refuses such a key in an object literal (TS2353). An object built at run time (a `Record`, `Object.fromEntries()`, a variable with extra keys) passes it, as `any` and plain JavaScript do. Every rule of the quoting, a name already quoted by hand included: [Identifiers](https://pgcast.dev/sql-template#identifiers). ## Keywords are checked before any query Some SQL is neither a value nor a name: `DESC`, `FOR UPDATE`, `SERIALIZABLE`, `RESTART IDENTITY`, the name of an aggregate. pgcast writes these into the text as they are, so it takes them only from a closed list, or, for an aggregate, only a function name. The type says which; at run time pgcast checks again and throws before any query, because a cast, an `any` or a JSON body can hand it anything: a `lock` of `{ for: 'UPDATE; DROP TABLE ratings; --' }` throws "Lock strength must be UPDATE/NO KEY UPDATE/SHARE/KEY SHARE, not '…'". Values with no meaning where they stand are refused the same way: `db.Default`, `db.self` or `db.raw()` as a filter value throws while compiling, so `.run()` rejects before anything reaches Postgres. Zapatos 6.6.1 sent them as strings or as JSON: no row matched on a text column, every row under `ne()` and `isNotIn()`, and other types failed. Such a value is a bug in the calling code, and a bug should not become a query. Each of these errors, with what throws it: [Errors pgcast throws before any query](https://pgcast.dev/transaction#errors-pgcast-throws-before-any-query). Telling them from Postgres's own: [Errors](https://pgcast.dev/handle-errors#fix-the-errors-pgcast-throws). ## `raw()` is the one door [`raw()`](https://pgcast.dev/sql-template#raw) writes its string into the SQL as it is: the one interpolation where a string becomes code, and its type takes any string. That is its job, for syntax no parameter can stand for, chosen by your own code. > **Security** > > Given a caller's text, `raw()` lets the caller write your query. So it only ever gets your constants, and a caller's choice is mapped to one of them ([Choose SQL at run time from your own constants](https://pgcast.dev/raw-sql#choose-sql-at-run-time-from-your-own-constants)). pgcast can't check what you put there, which is why `SECURITY.md` leaves SQL written in `raw()` out of its scope. As a filter value, `raw()` throws. Zapatos 6.6.1 sent it there as a parameter by accident, its wrapper as JSON; compiling it as SQL instead would have turned every `{ title: db.raw(input) }` into an injection. So it is neither: to compare a column with SQL, write a fragment, where `db.self` stands for the column. ## What changed from Zapatos 6.6.1 The characterisation of Zapatos 6.6.1 found seven ways for a caller's string to reach the SQL as code. pgcast closed each, and refuses the values a filter can't use: | What reaches pgcast | Zapatos 6.6.1 | pgcast | |---|---|---| | a `"` in a name: a table, a column, a filter key, `cols()`, `order.by` | closes its quotes | doubled: `'a"b'` is `"a""b"` | | a string that starts and ends with `"` | written as it is | as it is only if it is a well-formed quoted name | | the type of `param(x, cast)` | written between quotes as it is | as it is if well-formed once quoted (`pg_catalog"."int4`); otherwise one quoted name, any `"` doubled | | a `lateral` key, or the `alias` or table name of `select()` | spliced into `"lateral_…"` and `"sq_…"` | one name there too | | `lock.for`, `lock.wait`, the options of `truncate()`, the level of `transaction()` | written as it is | checked: anything else throws | | the `aggregate` of `select()` | written as it is | a function name, or it throws | | a string in the `updateValues` of `upsert()` | a column name | a parameter | | `db.Default`, `db.self`, `db.raw()`, `db.cols()` or `db.vals()` as a filter value | a string, or JSON | throws | Zapatos 6.6.1 already checked `order.direction` and `order.nulls`. Each hole, what it let through and the test that pins it: the [characterisation report](https://github.com/AlexandreDemartin/pgcast/blob/master/docs/CHARACTERISATION-REPORT.md#security); the SQL before and after: [the changelog](https://pgcast.dev/changelog#6.6.2-security). ## What pgcast can't guard against pgcast keeps a caller's input from becoming code; it can't tell whether the SQL does what you meant. In the three cases below the SQL is valid, and does the wrong thing. ### A caller's string as a name Quoted, a name is still the one the caller chose: - As a table name, `'schema.table'` included, it is any relation the role can read: given `'pg_catalog.pg_roles'`, `select()` lists the database's roles. - As a column name, it is any column in scope, a system column such as `xmin` included, or none (42703). `` ${'title'} = ${title} ``, with `title` typed `any` and set to `'title'`, compiles to `"title" = "title"`, true for every film. Map a caller's choice to a name from your own list, and remember that the keys of a request's body, given as a filter, are names too. ### An empty filter A filter is its conditions joined by `AND`, and no condition at all is `TRUE`, as in Zapatos. So a filter built from optional fields that are all missing matches every row: You write: ```ts function removeRatings(filmId?: number) { const where: s.ratings.Whereable = {}; if (filmId) where.film_id = filmId; return db.deletes('ratings', where); } const removed = await removeRatings().run(pool); ``` Postgres runs: ```sql DELETE FROM "ratings" WHERE TRUE RETURNING to_jsonb("ratings".*) AS result ``` Values: `[]` TypeScript knows: ```ts const removed: s.ratings.JSONSelectable[] // that is const removed: { film_id: number; stars: number; votes: number; }[] ``` > **Security** > > In `deletes()` or `update()`, an empty filter is every row: `removeRatings()` deletes them all. Check that a filter built at run time has a key, and no condition that is `TRUE` whatever the row, as `isNotIn([])` is, before you write with it. ### An object that is not a filter An interpolated object that is not one of pgcast's values compiles as a filter of its own enumerable keys: a class instance gives its fields, and `TRUE` when it has none. A built-in object of the realm that runs pgcast (`Date`, `Map`, `Promise`, a typed array, …) throws before any query; one from another realm, such as a `vm` context, does not. In Zapatos 6.6.1, `` db.sql`DELETE FROM ${'ratings'} WHERE ${when}` ``, with `when` a `Date` typed `any`, compiled to `DELETE FROM "ratings" WHERE TRUE` ([Whereable objects](https://pgcast.dev/sql-template#whereable)). Pass a date as `db.param(when)`. ## Report a flaw If you think you have found a flaw in pgcast (the SQL it compiles from a caller's input, or the code the generator writes), please report it privately: [report a vulnerability](https://github.com/AlexandreDemartin/pgcast/security/advisories/new) from the repository's Security tab, never in a public issue. Give the call, the SQL its `.compile()` gives, and what an attacker controls; a flaw pgcast shares with Zapatos is reported here too. You'll hear back within a week. The fix ships in a patch release, with a GitHub security advisory and its CVE, published on a date agreed with you, and credit in it and in the CHANGELOG unless you'd rather not be named. What is in scope, and which versions are supported: [SECURITY.md](https://github.com/AlexandreDemartin/pgcast/blob/master/SECURITY.md). ## Further reading - [Raw SQL](https://pgcast.dev/raw-sql): the steps, from parameters to `raw()` with your own constants. - [SQL template](https://pgcast.dev/sql-template): every interpolation and what it compiles to, [Identifiers](https://pgcast.dev/sql-template#identifiers) and [Whereable objects](https://pgcast.dev/sql-template#whereable) among them. - [Errors](https://pgcast.dev/handle-errors): pgcast's own errors, and Postgres's. - [`SECURITY.md`](https://github.com/AlexandreDemartin/pgcast/blob/master/SECURITY.md): scope, threat model and supported versions.