Data formats and migrations

Every part of reportd's data records its format, and reportd upgrades older data one step at a time before it opens its stores.

Part	Where its format is	Current format
SQLite template store	PRAGMA user_version	3
File template store	<templateStore.path>/_format.json	2
File report store	<reportStore.path>/_format.json	1
dataDir (auth, audit log, approvals, licence history, template sources)	<dataDir>/_format.json	8
Git repository	git's own format	—

A folder that holds data but has no _format.json is format 1: it was written before formats were recorded. A part that doesn't exist yet is created in the current format.

At start

When reportd starts it reads every part's format:

• All current: it starts.
• Older: with migrations.auto (the default), it first writes a backup of the data as it is to dataDir/backups/pre-migrate-<time>.tar.gz (the same archive reportd backup writes), then runs each step and logs it. Each step records the new format as soon as it finishes, so a stop part-way resumes at the next step.
• Newer than this reportd reads (data written by a later version): it refuses to start. Data is never downgraded; to go back to an older reportd, restore a backup taken before the upgrade.

With migrations.auto: false, reportd refuses to start while steps are pending and names them. Run them with reportd migrate while reportd is stopped:

reportd migrate --dry-run   # list the steps
reportd migrate             # back up, then run them

RFML documents

Report documents carry their own language version ("rfml"), and the engine migrates an older one when it loads it (see the RFML spec, §24.1). reportd stores the migrated document as a draft revision of its own (edit source migrate, message RFML <from> → <to>) before the next save of that draft. To migrate every draft at once, for instance before retiring an engine that still reads an old version:

reportd migrate --documents --dry-run   # list the drafts of an older version
reportd migrate --documents             # store each one migrated, through the engine

--documents runs the store steps first, then starts the engine (as reportd does) to migrate the drafts. Published versions aren't rewritten: renders migrate them in memory, and the next publish commits the migrated form.

The steps so far

Part	To	Step
SQLite template store	2	Add the assistant's proposals and conversations and the job queue tables. Databases from before had gained them on every open; they are now part of the schema.
SQLite template store	3	Add the trash (trash.md): templates.trashed_at and trash_item, and the trash_items table. Every template stays as it was, out of the trash.
File template store	2	Nothing changes: .trash/ is made when the first template goes there. An older reportd, which would list nothing in it and let a new template take a trashed one's id, refuses the store.
dataDir	2	Move the presets from presets/presets/*.json up to presets/*.json.
dataDir	3	Keep only a hash of each session token in auth.json: sessions (by raw token) becomes sessionHashes (by the token's SHA-256), so everyone stays signed in and the file signs no one in.
dataDir	4	Nothing changes. auth.json may now hold MFA data, so an older reportd, which would sign those users in with a password alone, refuses the folder.
dataDir	5	Nothing changes: presets/.trash/ is made when the first preset goes there. An older reportd, which would seed the presets again once all of them are in the trash, refuses the folder.
dataDir	6	Record the 60-day transition to licences in licence-history, when the install has users (in auth.json, or imported into Postgres) or templates, live or in the trash (licence.md). A new install has an empty dataDir, so never runs this and is Community from its first start.
dataDir	7	Turn the presets in presets/*.json into libraries, each released as 1.0.0 in every environment (below), then remove presets/, its trash included. An environment that already holds the same release is left alone, so a stop part-way, or a restored backup from before, runs it again safely. A library of the same name and version with other content stops the step with an error, so reportd doesn't start, and the presets stay as they are. With no libraries to put them in (sourceControl.driver isn't git, or setup hasn't made the environment branches yet), the presets are left where they are: nothing reads them, and the folder can be deleted.

| dataDir | 8 | Add template-sources.json with no sources (template-sources.md), unless it is there. An older reportd, which would ignore the sources, refuses the folder. |

A SQLite database with tables but no format at all is from before RFML; it has no upgrade path and is refused.

The presets become libraries

The studio's presets were host-wide files in dataDir/presets/<id>.json of four kinds: page, layout, style and color. They are now library items (designer/libraries.md): page templates, themes and theme tokens, versioned and released like components. Each preset maps to one of them:

Preset	Becomes	Mapping
page {width, height} in points	A page template of the library presets	Named after the preset (US Letter → us-letter). Its size is the named RFML size within a point of it (A3, A4, A5, B5, letter, legal or tabloid). A named size turned on its side is that size with orientation: landscape. Any other size is {width: "<w>pt", height: "<h>pt"}. It has no margin.
layout {margins: {top, right, bottom, left}} in points	An A4 page template of presets	Named a4-<name> (Compact → a4-compact), size: "A4", and margin as the CSS shorthand in points (28pt, 36pt 72pt, or all four).
color {hex}	A token of the presets theme	The token is named after the preset (Harbour → harbour), and its value is the hex as it was.
style {locale, currency, fontFamily, spacing, colors}	A theme library of its own, style-<id>, after the preset's id (style-larkmere → style-larkmere, corporate → style-corporate)	colors become the theme's tokens, each named after its key (primary, ink …). spacing becomes the token space (8 → 8pt). fontFamily becomes defaults.document.font-family. A theme holds styles only, so locale and currency go in the library's manifest, as x-locale and x-currency, and in its description: "The Forge style, from the studio's presets (Helvetica, en-US, USD)".

A library has one theme, so each style preset is a library of its own. A report imports one with theme: true to build on it, or copies it in with "Copy the theme into this report".

Names are lower case, with words joined by hyphens. A name that is taken gets -2, -3 …. page is never used as a page template's name, since the studio lays a library's components out on a page of that name. Each library is released as 1.0.0 in every environment, with the notes "Migrated from the studio's presets (dataDir/presets)." and the author reportd migrate. With the seeded presets, the result is:

• presets: the page templates a4, a5, tabloid, us-legal and us-letter, then a4-comfortable (46pt), a4-compact (28pt) and a4-generous (72pt), and a theme with the tokens copper, harbour, ink, night and paper;
• style-forge, style-plain and style-larkmere.

The releases are commits on the local environment branches. With a sourceControl.remote, they reach it with the branch's next push (pushOnPublish pushes at the next publish).

Presets in the trash (presets/.trash/) were deleted, and aren't moved. The backup taken before the migration keeps them, with every other preset.

Writing a step

A step belongs to the package that owns the data (MigrationPart in internal/store/sqlite, internal/store/file and internal/reportstore; migrate.DataPart for dataDir). Add it to the end of the part's steps with the next format number, raise the format the store's Open expects, and update backup.Formats (a test fails until they agree). A step must be safe to run again after a stop part-way. Add a test that migrates data in the previous format and reads it back.