Refreshing library pins
Every library whose types the toolchain ships is pinned to one upstream ref in one
of five lane registries: the LuaLS, script_api and
authored registries, with the fixtures their entries name, sit in the newest
data folder, packages/library-types/data/<version>/, and the markdown and OpenAPI registries
at the root of packages/library-types/ in a toolchain checkout. Upstream keeps moving;
the pins do not. This page is what to do when one of them has fallen behind.
Authoring LuaLS library types and Authoring forked library types cover adding a library. This page covers moving one that is already there, which is a different job: the generation commands are the easy half, and the couplings that move with the pin are the half that is not discoverable from the sync scripts.
This page is for maintainers of the library data in the toolchain repo, changed by pull requests there. Consuming a library in your own project is
resolve, and your project's declared dependency version is unrelated to the pin here.The
buncommands on this page run from the toolchain root unless a step names another folder.
The library update workflow
The Library update workflow (the toolchain's
.github/workflows/library-update.yml) runs
library update
over packages/library-types/data/ with the latest CLI, then opens one pull request
per library it wrote, from the branch library-update/<library>. It runs daily,
and when dispatched:
gh workflow run library-update.ymlA PR that moves a pin is titled update <library> to <ref>; one that only adds
commits to a review is titled record <library> commits. A later run force-pushes
the same branch and retitles its open PR, so a library never has two. A library that
cannot be read upstream is named in the run's log, and fails the run once every
other library has its PR. A run dispatched while another is going waits for it to
finish.
What merges without a human
The workflow turns on auto-merge, so the PR merges once ci-done passes, only when
both hold:
- the
changeiskeptoradded, or the PR only records commits; - every file the PR touches is the library's
reviewed.jsonor under itssource/, and CI scopes it as library data.
The second is read from the commit, not from what library update reported, so a
file anything else wrote keeps the PR for a human. Every other PR stays open. A run
turns off an open PR's auto-merge before it replaces the branch, and turns it on
again only for the commit it read.
Reading a PR left open
The PR body names the change and the members upstream added, removed and
changed. A changed PR whose three lists are all empty had no single review of
the pinned commit to compare with, so read the diff under source/ instead.
The library-check run on the PR says whether the library still fits: a module below
its floor is named with the value it measured, and a usage/ proof that stops
compiling is named by file and line. Fix the fork, floor or proof as the couplings a re-pin moves with
it describe, on a branch of your own: the
next run force-pushes library-update/<library> and drops any commit pushed to it.
Once yours merges, the next run finds the library current and closes the bot's PR.
follows
follows in a library's library.json says what library update watches: tags
in the pin's series, the default branch, or none. With none set, a library
pinned to a commit follows its default branch and any other follows its tags.
none holds a library, and no request is made for it. library-defold-persist and
library-defold-rendy follow none because their whiteboxdev repos answer 404,
which would fail every run; they hold no review, so resolve reports them
unrecorded.
How a stale pin is noticed
The library update workflow runs daily, so a library
that moved upstream gets one PR from library-update/<library>, refreshed on each
run. The bot closes that PR, with a comment, once a run finds the library current or
held, or no longer lists the library, as when a new data folder renames it.
The bot files an issue titled library update cannot land <library> only when it
cannot land a library: library update failed for it, or its PR's checks were red
when the run started. The issue names the failure or links the PR, a later run edits
it rather than filing a second, and the bot closes it, with a comment, once neither
holds. A run reads the checks before it refreshes any PR, so an issue opens or
closes up to one run after the checks finish.
bun run upstream:library-check is a manual report that no workflow runs. It scans
all five registries and reports, per upstream repo, whether the pin has fallen
behind:
bun run upstream:library-check # prose report
bun run upstream:library-check --json # one object, for toolingThree things about the report are worth knowing before you act on it:
- The unit of work is the repo, not the module.
britzl/defold-inputbacks ten authored modules andbritzl/defold-orthographicis consumed by two lanes. One bump answers every dependent, so the check groups byrepo+ref, counts the dependents, and lists them under--json. - A commit-pinned repo is only actionable when the delta touches a path this
repo reads. The registries already record which upstream files each target
consumes, so a README-only commit on a pinned repo's default branch reports as
behind-headwithout asking you to do anything. - A pin that feeds only the fidelity comparison is marked internal. When
every dependent is a markdown or OpenAPI target whose
decisionis notgo, no user receives its output, so the report setsinternaland the prose line ends in(internal); the bump needs no changelog bullet. It is still actionable: a stale spec hides upstream movement from the go/no-go comparison. The library update bot does not update these targets, so this report is the only place their drift shows.
A repo the check could not read at all is reported as unknown and exits the
command non-zero. That is deliberate: a renamed or deleted upstream must not look
like "current".
The check reads GitHub and Codeberg; a repo on any other host reports unknown.
When an upstream moves, search the author's old repos for a pointer to the new
home before withdrawing the library. Re-pointing means a new repo and a new
ref together, because a moved repo often has rewritten history and the old
commit no longer exists there.
Survey the Asset Portal
The registries only see libraries someone already added.
<data folder>/asset-portal.json records every
Defold Asset Portal entry, pinned to a
defold/asset-portal commit, with the GitHub repo each entry names classified
by what a dependency on it installs:
GITHUB_TOKEN="$(gh auth token)" bun run --cwd packages/library-types asset-portal:syncThe sync refuses to start without GITHUB_TOKEN or GH_TOKEN, because it makes
about five API calls per entry. It follows renames, so repo is the canonical
URL (the portal's britzl/extension-imgui is recorded as
defold/extension-imgui). Each repo is read at its latest release, else its
newest tag, else its default-branch head.
Files count only inside the repo's game.project [library] include_dirs,
since nothing else reaches a project that adds the dependency. Each entry's
kind is the first row that matches:
kind | The repo at its pin |
|---|---|
no-source | is not readable on GitHub; reason is no-github-url or not-found |
not-a-dependency | declares no include_dirs: a tool, sample or editor integration |
script-api | ships a .script_api, which resolve already types |
native-untyped | ships an ext.manifest without a .script_api |
pure-lua-annotated | ships Lua, and one of up to four sampled files carries ---@ annotations |
pure-lua | ships Lua with no annotated sample |
no-lua-api | ships no Lua: shaders, materials, sounds or editor scripts only |
bun run asset-portal:report joins the manifest against every lane registry
and the defold org listing in defold-extensions.json, offline, and prints
four answers:
- Tracked: entries whose repo a registry already pins, counted per lane. The
portallane counts thescript-apientries that have reference pages. A tracked entry files here whatever itskind. - Candidates: untracked entries of kind
script-api,native-untyped,pure-lua-annotatedorpure-luathat are not archived, not a fork, were pushed within 730 days of the sync, and carry no verdict. They are grouped by kind and ranked by stars, which only order the list and say nothing about quality. - Declined: untracked entries with a recorded verdict, each with its reason.
- Nothing to type: untracked
no-lua-api,not-a-dependencyandno-sourceentries.
A line between them counts the untracked entries with a Lua API that fail the maintenance rule.
After a manifest sync, give every script-api candidate its reference pages:
GITHUB_TOKEN="$(gh auth token)" bun run asset-portal:script-apiIt converts each .script_api under the entry's include_dirs at the pin
asset-portal.json recorded, writes the page list to
<data folder>/asset-portal-script-api.json and one doc per page under
asset-portal-script-api/api-doc/, and rebuilds that directory on every run.
Those entries then count as tracked on the portal lane. A doc that fails to
convert, a page key another page already holds, or a page key
that is not one safe route and file name (ASCII letters, digits, _, . and
-, starting and ending with a letter, digit or _, and not a Windows device
name such as CON or NUL before its first .) stops the sync before it
changes any file, instead of dropping the page. A page key names only the page's
api-doc file: its address comes from the library's origin repo, as
Library page addresses
describes.
A library that moved forge keeps a stale portal entry: the portal's
Klaleus/library-defold-checkpoint is gone from GitHub, and the authored lane
pins codeberg.org/Klaleus/defold-checkpoint. Map such an entry to the tracked
repo in packages/library-types/asset-portal-aliases.json. The report refuses
to run when an alias names an entry the manifest no longer lists, or a repo no
registry pins.
An entry whose pinned tree holds no runtime module a project would require,
such as an editor script, an import tool or a project template, stays a
candidate until someone looks. Record the decision in
packages/library-types/asset-portal-verdicts.json: a map from entry id to one
short reason naming what the tree holds instead. A verdict takes the entry off
the candidate list for good, whatever its maintenance state. The report refuses
to run when a verdict names an entry the manifest no longer lists, or an entry a
registry pins directly or through an alias, since either verdict is stale.
The weekly Library upstream check workflow (the toolchain's
.github/workflows/library-upstream-check.yml) compares the portal head's entry ids
with the manifest. When ids were added or removed, it opens one
issue, titled Asset portal entries differ from asset-portal.json, that lists
them and names the re-sync command. The title never changes, so later
runs stay quiet while that issue is open. A changed entry under an unchanged id
is not reported, because the portal rewrites release metadata every six hours.
Refresh the lane
Four lanes fetch and regenerate. Run their commands from the toolchain root, in
order, after editing the pinned ref:
| Lane | Pin | Commands |
|---|---|---|
| LuaLS | the library folder's reviewed.json | bun run --cwd packages/library-types luals:fetch, then luals:fidelity, api-doc |
| Markdown | markdown-targets.json | bun packages/cli/src/library/sync-markdown-types.ts --fetch --fidelity --api-doc |
| OpenAPI | openapi-targets.json | bun packages/cli/src/library/sync-openapi-types.ts --fetch --fidelity --api-doc |
| script_api | the library folder's reviewed.json | bun packages/cli/src/library/sync-script-api-types.ts --fetch --fidelity --data packages/library-types/data, then bun run --cwd packages/library-types api-doc |
A library folder sits in the newest version-named folder of packages/library-types/data/, named for its repository: its
<host>/<owner>/<repo>, lowercased, without .git.
--fetch is the only step that touches the network; everything after it reads
the committed fixtures, so a regeneration is reproducible offline. No lane
writes a declaration: resolve prints each target's types from its fixtures, so
check them by printing them with bun test packages/cli/src/library from the
toolchain root.
Set GITHUB_TOKEN before fetching. The fetchers read it (sync-luals-types.ts
and sync-library-types.ts both send it as a bearer token) and GitHub degrades
unauthenticated API traffic — the git/trees calls behind --fetch and
upstream:library-check can return 504 Gateway Timeout on every attempt while
the rate-limit budget still shows requests remaining, so a 504 here is not
evidence that the pin or the repo is wrong.
GITHUB_TOKEN="$(gh auth token)" bun run --cwd packages/library-types luals:fetchThe authored lane is the exception, and it is the one that surprises people.
A library's forks have no --fetch and no fidelity pass. Their
source/upstream/ snapshots are vendored by hand and their .d.ts is
hand-written, so bun run --cwd packages/library-types api-doc is the only command they take.
Refreshing an authored pin means:
- edit the
refin the library folder'sreviewed.json; - copy the upstream files named by
upstreamLuainto the library folder'ssource/upstream/…by hand (the path under it mirrors the upstream path); - update the hand-written
.d.tsfor whatever the new upstream added or changed; - re-run
bun run --cwd packages/library-types parityandbun run --cwd packages/library-types api-doc.
The couplings a re-pin moves with it
None of these are visible from the sync scripts, and each of them reds ci if you
skip it.
- The committed floor.
floorsin the library'slibrary.jsonholds a per-module measurement the gates andlibrary checkassert against. New upstream surface changes the measurement, so the floor entry moves with the pin — raise a generated module's ratio withbun run --cwd packages/library-types fidelity:floor, and edit a fork's or native extension's{ callable, field }pair by hand. - The parity reports.
authored-parity-floor.test.tschecks each committed report underpackages/library-types/fidelity/authored/andfidelity/native/against a fresh measurement of the vendored copy. After re-vendoring, re-runbun run --cwd packages/library-types parity, review the coverage change, and commit the reports in the same commit. An upstream change that moves no number leaves the reports as they are. - The provenance assertions. The toolchain's
api-surface-loader.test.tsreads the pin back out of the emitted API-doc metadata and asserts it exactly — for druid,meta.commitis therefindruid/reviewed.jsonandmeta.sourceUrlishttps://github.com/Insality/druid/tree/<ref>. A bump moves both, and they are in a different package from the pin you edited. - The pin kind.
refKindin the library'slibrary.jsonsays whether the pin is a release, a tag or a commit, and the docs page's adopt step follows it. It moves with the pin:api-surface-loader.test.tsreds when the asset-portal register records another kind at the same ref. - The pin-refresh record.
packages/library-types/pin-refresh.jsonstates which members a bump added and removed, one entry per refresh, each tagged with thelaneit belongs to (luals,script-api, oropenapi).packages/cli/src/library/pin-refresh.test.tschecks every name it states against the surface that lane's own parser yields for the committed fixtures —buildTargetModel,committedScriptApiDoc, orparseOpenApi. Regenerating the fixtures without updating the record redsci, so derive the new entry by diffing the old and new member names rather than from upstream release notes. A second key,deliberatelyUnmoved, records a pin you examined and chose to leave alone, and is guarded on itsrefalone — so a note left standing after that pin later moves also reds. The guard deliberately does not check that the record is complete, nor itschangedMembersandnotes, which are prose. - The compile-only surface proofs. The
.test-d.tsfiles under each library folder'susage/assert the emitted.d.tsby assigning from it, and pin the absence of non-public members with@ts-expect-error. A bump breaks them in both directions: a member that left the public surface fails the assignment, and a member upstream stopped marking@field privateor@localmakes its expect-error directive unused (TS2578). The second is the easy one to misread — check the old and new fixture annotations before assuming the emitter regressed, because dropping aprivatemarker upstream is a real surface change and the declaration is then correct to emit. - The generated docs artifacts.
packages/docs/llms.txt,packages/docs/llms-full.txt, and the docs-site search and symbol indexes are committed, not built on demand. Regenerate them withbun run --cwd packages/docs-site build-indexeswhenever the API-doc model changed.
What a re-pin does not touch
The corpus pin and a consumer's declared dependency are different things, and conflating them is the first mistake a repo-wide find-and-replace makes.
Several CLI tests hardcode archive URLs and extracted-entry paths, and some of
them pin exactly the corpus string. packages/cli/src/library-match.test.ts,
packages/cli/src/resolve.test.ts and packages/cli/src/druid-resolve-e2e.test.ts
all carry a druid version literal in consumer archive URLs, and it can equal the
ref the registry entry holds. That collision is the reason a repo-wide
find-and-replace is unsafe: one copy of the version must move with the bump (the
registry entry, and the provenance assertions named above), and
another must not.
normalizeSourceId (packages/cli/src/library-match.ts) strips
/archive/<ref>.zip and matches on the repo name alone, so a consumer fixture is
version-independent either way — which cuts both ways. Rewriting its version
leaves the suite green, so a green bun test is not evidence the rewrite was
correct. The 1.2.3 fixtures in packages/cli/src/library-scene-materialize.test.ts
and the later resolve.test.ts cases are consumer fixtures on the same footing
and equally out of scope.
Leave them alone. A corpus bump changes the data folder, the floors,
the emitted artifacts and the docs indexes. It does not change what a test project
declares in its game.project.
Check the work
bun run typecheck
bun run lint
bun testThen re-run bun run upstream:library-check — the repo you bumped should now
report as current, and the issue the workflow opened can be closed.
Retiring a curated target
A curated target is only needed while its upstream ships no types resolve can
use. Once an upstream release ships a types/ folder that resolve accepts,
every project on that release gets those files in place of the curated copy,
and the curated copy only costs a pin, a parity floor and a re-pin per release.
bun run upstream:shipped-types-check reads the newest release of every repo
behind a curated target, across the authored, LuaLS, script_api and native
lanes, and runs its types/ folder through the same selection and compile
check resolve applies:
bun run upstream:shipped-types-check # prose report
bun run upstream:shipped-types-check --json # one object, for toolingThe newest release is the newest tag by commit date, or the default branch head
for a repo with no tags. Each repo reports accepted, rejected with the
check's first error, or none. A repo it could not read reports unknown and
exits the command non-zero.
The Library upstream check workflow runs the same sweep weekly and opens
one issue per accepted repo, titled
retire curated types for <slug>: <ref> ships types. The issue lists the
shipped files and every curated entry the repo backs.
The issue retires nothing. Retirement is a reviewed change, as pixels was
(toolchain commit a32a1972), that removes each listed entry's registry entry, the
input the issue lists for it, its /api page, vendored upstream source and
parity floor, along with the tests that named it. Before making it, weigh the projects left
behind: a project still on a release before the one that ships types has no
types/ folder to fall back on once the curated copy is gone.
An upstream maintainer gets the curated files to commit with export-types, as
Shipping types with a library
describes.