Template sources

Designers start a new report by copying a template from a template source. The example gallery, Larkmere Water's examples, is one; an organisation adds its own: a git repository of its letterhead, invoice and statement, a folder on the server, or a published gallery on its intranet. A copy is upgraded to the RFML version this installation writes, and a template that needs a newer Reportforge is listed but can't be copied. This page is for whoever sets up the sources. Copying a template is in designer/examples.md.

Who sets them up

Managing sources needs the templates.sources permission. Only the administrator role has it, a custom role can be given it, and an agent key never has it, whatever its roles, as with fonts.sources: a source decides what reportd fetches, and what designers copy. Everyone who can create reports (template.write) sees the templates of every source that is on.

Sources of an organisation's own need a licence with the template-sources feature (licence.md). Community has the example gallery, examples.gallery (one folder or published gallery, as before), and import. When a licence lapses, the sources stay as they were set up, but are off, and come back on with a licence that grants the feature. Reports already copied from them are untouched.

Admin's Template sources page

Admin › Template sources, beside Font sources, is shown to those with templates.sources. It lists the sources, the example gallery first, each with its kind, where it reads from (a token only as its secret: reference), who changed it last, and what its last read found: how many templates, the commit, when, or the error of a read that failed.

• Add a source opens a form that changes with the kind: a git repository's address, ref, folder, token reference and cap; a folder's path on the server; a gallery's address. A git repository and a gallery can be allowed internal addresses. The id follows the name until you change it, and can't be changed later; nor can the kind. The token field takes only a secret: reference, and says how to set one.
• Edit changes a source. reportd checks every field, and a refusal names the field it is about.
• The On switch turns a source on or off. Designers see the templates of the sources that are on; a source that is off isn't read.
• Refresh now reads a source at once, instead of waiting for the hour.
• Remove, after asking, removes a source. Reports already copied from it stay as they are.

The example gallery can be turned off, but not edited or removed. Without a licence that has template-sources, the page shows the example gallery, which can still be turned on or off, and says that a licence adds your organisation's own sources; there is no Add. Sources kept from a licence that lapsed show as off, with that reason, and can't be changed until a licence grants the feature again.

The routes behind the page, all under templates.sources:

Route	Does
GET /v1/template-sources	Lists the sources, each with its state: when it was last read, the commit, how many templates, and the error of a read that failed
PUT /v1/template-sources/{id}	Adds a source, or changes it, or turns it on or off
DELETE /v1/template-sources/{id}	Removes a source
POST /v1/template-sources/{id}/refresh	Reads a source again now

Each change is audited as templates.source.add, .change or .remove, with the source's id.

The example gallery

The source examples comes with Reportforge: the Larkmere Water gallery, at the address examples.gallery sets (designer/examples.md). It is on unless examples.gallery is off. It can be turned off with its switch on the Template sources page, or PUT /v1/template-sources/examples with {"enabled": false}, which is the same as examples.gallery: off was. With it off, an empty catalog offers your organisation's templates instead of the examples. It can't be removed, and its address is the setting's, so a machine without internet access keeps its folder of examples as before.

Adding a source

A source has an id (a lower-case letter, then lower-case letters, digits and hyphens, at most 48), a name designers see as its templates' heading, a kind, and the fields its kind needs:

Kind	Fields
Git repository (git)	url: the repository, https:// only. ref: a branch, tag or full commit hash; the repository's default branch when empty. path: the folder that holds the templates; the root when empty. token: for a private repository. allowPrivate: for a server inside the network. maxBytes: the clone's cap.
Folder (folder)	folder: an absolute path on the server, which reportd reads each time.
Published gallery (gallery)	url: an https:// folder with an index.json and a .zip per template, as cargo xtask gallery builds. allowPrivate: for one inside the network.

curl -X PUT https://reports.example.com/v1/template-sources/finance \
  -H "Authorization: Bearer $REPORTD_KEY" -H "Content-Type: application/json" \
  -d '{"name": "Finance templates", "kind": "git", "enabled": true,
       "url": "https://github.com/acme/report-templates.git", "ref": "main",
       "path": "templates", "token": "secret:env:TEMPLATES_TOKEN"}'

A token is only ever a secret: reference, such as secret:env:TEMPLATES_TOKEN or secret:file:templates-token (a file in secrets.dir), which reportd resolves through the providers secrets.providers turns on (env, file). It is never stored or shown as the token itself. reportd sends it as HTTP basic auth, which every host below takes. The secret's value is the token, sent with the user x-access-token, or user:token for a host that needs a particular user.

There is no SSH: reportd reads repositories over https only.

Setting up a repository, host by host

Give reportd a token that can only read the one repository. reportd never writes to a source: people change it through their own git workflow.

Host	Repository URL	Token	Its scope
GitHub	https://github.com/<owner>/<repo>.git	A fine-grained personal access token, or a GitHub App's installation token, limited to the repository	Repository permissions: Contents: Read-only (Metadata: Read-only comes with it)
GitLab	https://gitlab.com/<group>/<project>.git, or your GitLab's address	A project access token with the Reporter role, or a deploy token	read_repository
Azure DevOps	https://dev.azure.com/<organisation>/<project>/_git/<repo>	A personal access token	Code: Read
Bitbucket Cloud	https://bitbucket.org/<workspace>/<repo>.git	A repository access token; the secret's value is x-token-auth:<token>	Repositories: Read
Self-hosted (Gitea, Forgejo, GitLab, Bitbucket Data Center…)	The server's https address for the repository	The server's read-only access token, or user:token	Read access to the repository only

A public repository needs no token. A server inside your network needs allowPrivate (see the address guard), and a certificate the machine running reportd trusts.

How a repository is laid out

reportd accepts three layouts, and checks for them in this order in the folder path names:

1. A published gallery: an index.json and a .zip per template, each with its size and SHA-256, as cargo xtask gallery builds.
2. A source folder: a gallery.json listing the templates' ids, in order, each a folder named by its id that holds a bundle whose document has that id, as the repository's examples/ is.
3. A plain folder of templates: otherwise, every folder that holds a template.rfml.json is a template, in path order. Folders whose names start with a dot are skipped, and so is everything inside a template's own folder. Each template is named by its path: invoices/standard is invoices-standard.

A minimal repository in the third layout:

report-templates/
  README.md
  templates/
    invoices/
      standard/
        template.rfml.json
        samples/main.json
        schemas/main.json
    letters/
      welcome/
        template.rfml.json
        samples/main.json

With path: templates it lists two templates, invoices-standard and letters-welcome. Each one's title and description are its document's meta.title and meta.description. A template's folder is a bundle: its document, and the files a bundle may hold (samples/, schemas/, tests/, assets/); anything else beside them, such as a README, isn't copied. A document that can't be read is still listed, as a version this Reportforge can't read, so its author sees it.

Versions

Every listed template has a status, worked out from its document's rfml and the versions this installation's engine reads. The version this installation writes is the engine's current version, which renderd reports in GET /v1/info as current (RFML 0.2 today; the engine also reads 0.1).

The template's rfml	Status	On copy or import
The version this installation writes	available	Copied as it is
An older version the engine reads	available	Copied as the engine reads it. When the engine migrates that version, the first draft is the migrated document, and its history says so: "…, RFML 0.1 → 0.2". RFML 0.1 is read as it is (0.2 only adds features), so a 0.1 template stays 0.1.
Newer than this installation writes	too-new: "Needs Reportforge x or later (RFML y)"	Refused: 409 rfml-too-new, with the version and, when this Reportforge knows it, the release that writes it
Older than anything the engine reads, or not a version at all	unsupported: "Made for an RFML version this Reportforge can't read"	Refused: 422 rfml-unsupported

A newer minor version isn't copied even when the engine could read every feature it uses: a starting point that works only until someone edits it is no starting point. The same test applies to a report imported from a file (POST /v1/templates/import).

Reading and refreshing

• Git. A repository is cloned shallow, one commit, into dataDir/cache/template-sources/<id>/. reportd reads the ref again at most once an hour, and fetches only when it has moved. A clone is capped at 100 MB by default (maxBytes, up to 1 GB), and each template's bundle at 16 MB.
• Failures. A read that fails is kept for 5 minutes before reportd tries again. A repository read before keeps listing what it read then, and its state shows the error. GET /v1/examples lists the templates of the sources that can be read, and says which one can't be reached; designers see one line for it, "Finance templates can't be reached right now.", in its place.
• Refresh. Refresh now on the Template sources page, or POST /v1/template-sources/{id}/refresh, reads a source at once, and answers with its new state.
• The commit. Each listing records the commit it was read at, and a copy's first edit names it: "from Finance templates: invoices/standard @ 1a2b3c4". The copy isn't linked to its source: later changes to the source don't reach it.
• Folders are read on every request, so edits show at once. Published galleries are read as the example gallery is: the index once an hour, a failure kept for 5 minutes.

The clones are a cache: deleting dataDir/cache/template-sources/ only makes reportd clone again. Backups leave it out.

The address guard

Every read of a repository or a published gallery goes through the same guard as remote assets, HTTP data sources and the example gallery. It refuses loopback, private and link-local addresses, checked after DNS, refuses a redirect from https to http or past 5 hops, and uses no proxy, so the check means what it says. A source on a server inside your network sets allowPrivate, which lets it, and only it, reach internal addresses.

Where sources are kept

Sources are data, not config. On state.driver: file they are in dataDir/template-sources.json; on postgres, in the schema's template_sources table, so every node offers the same (state.md). They go into backups (backup.md), and reportd state import and export move them with the other administration data. The file holds tokens only as secret: references.