scholarr/SPRINT.md

Scholarr product sprint

Approval and authority

Justin authorized preparation, issue triage, and a new baseline commit. He has not authorized launch. Review this document as the single product and execution reference for the proposed 24-hour run. Approval tracking: justinzeus/scholarr#9. Runner preparation: justinzeus/nightly#92; reset supervisor: justinzeus/nightly#93.

An explicit owner go for the concrete plan approves the proposed defaults below and replaces the old, separate owner-freeze gates for this sprint. The agents may then choose implementation details and record their rationale here without stopping for another spec ceremony. They must preserve the locked requirements. A requirement conflict, missing credential needed for a real feature, or unsafe provider operation remains a blocker; agents must report it rather than invent a fallback or claim completion. Preparation alone does not activate this authority.

The reset preserves Git history, the frozen design, and prior decision evidence. It supplies one new starting commit on an isolated branch. It does not erase the legacy repository, dirty local files, old decision issues, or production data.

Intended result

Build a self-hosted author watchlist and publication tracker that Justin's dad can use daily and other households can install. A reader follows researchers, sees their publications, tracks read state, resolves ambiguous identities, and finds legal open-access copies. An administrator can understand provider health, recover access, audit changes, and back up or restore the installation.

The target after the run is one integrated, runnable product branch and one reviewed PR, with executable acceptance evidence and a usable installation path. A pile of independent PRs, a static UI demonstration, mock-only provider clients, or a service with omitted authentication modes does not meet the target. Twenty-four hours is a delivery target, not a promise that unfinished work is done.

The run does not deploy, merge into main, publish a release, import the real household database, or replace the existing workflow. Justin still performs manual UI approval and the real-data shadow period before a production cutover.

Locked requirements

• Go backend, Vue 3 frontend, SQLite only. Use a serialized writer, foreign keys, WAL, numbered migrations, and restart-safe persistence.
• A source-agnostic FollowedAuthor is the author root. Names are display/search labels, never identity keys. Preserve external source identities and provenance.
• Share deduplicated authors and publications globally. Keep follows, read state, visibility, and user preferences private to their user.
• Support internal username/password, generic OIDC, and trusted-header auth. Internal auth remains the zero-dependency default and recovery path.
• Use one validated declarative configuration, environment-variable references for secrets, safe first-admin setup, and a straightforward first-run flow.
• Use the frozen design/DESIGN.md, design/tokens.css, and bundled assets. Implement dashboard, publication list/detail, author list/detail, review, author search/add, bulk import, sources, activity, settings, login, onboarding, and not-found behavior. Keep loading, empty, populated, and degraded states honest, with responsive navigation, keyboard access, and light/dark themes.
• Serve system fonts and bundled assets locally. The design proof's support.js and synthetic example records do not ship as product behavior.
• Show source freshness and confidence words rather than invented certainty or raw match percentages. Keep new-publication counts scoped to the current user.
• Preview bulk imports and destructive admin actions before applying them. Audit consequential writes and support conflict-checked undo.
• Implement documented clients for OpenAlex, Crossref, arXiv, Unpaywall, and ORCID. Missing keys disable that client's network lane with an actionable health state; they do not authorize anonymous scraping or a substitute API.
• Zero application-initiated Google contact. Scholar URLs/IDs are inert import metadata or human-clickable links. Parse them locally; never dereference them. No scraping, CAPTCHA solving, anti-bot evasion, or Semantic Scholar v1 expansion.

The data-model draft supplies detailed table contracts and its 22 falsifiable acceptance cases. The frozen visual design supplies the appearance contract. This document controls sprint scope and defaults where the old process differs.

Proposed decisions approved together at launch

Decision	Sprint default
D1: public identity	Internal integer joins; immutable UUIDv7 public IDs only on externally referenced or merge-aliased rows.
D2a: unfollow	Hide records without an active follow path; retain read/favorite state and restore it on refollow.
D2b: legacy favorites	Retain migrated favorite state for lossless import/export; do not add a favorite UI outside the frozen design.
D2c: legacy reading	Treat a publication as read if any legacy link for that user says read.
D2d: NEW	Derive at read time from first incremental discovery within 14 days; baseline, manual, and legacy imports are not NEW.
D3: attachment	Reuse exact identities, explicit provider crosswalks, and completed high-confidence calibration evidence. Review weaker or conflicting evidence; names alone never attach or merge.
D4: undo	No arbitrary expiry while recorded row-version preconditions hold. A conflict prevents the entire undo.
D5: orphans	Retain automatically. Admin garbage collection requires a backup, dry-run preview, reference checks, and audit.
D6: shared identity	Only administrators merge shared identity or write global negative-identity evidence. Reader decisions remain per-user.
D7: purge	Hard purge with no resurrection tombstone. A later sanctioned strong-ID ingest can create a new canonical row; legacy-only data cannot recreate it automatically.
Distribution	A self-contained Go binary with embedded built Vue assets, plus a non-root container image. Compose is an installation example, not a live deployment.
License	Propose MIT with Justin's personal copyright identity. Add the license only after the owner approves this packet.
Acquisition	OpenAlex is primary. Crossref/arXiv add work metadata; ORCID anchors identity; Unpaywall resolves legal OA locations. Persist cursors, deduplicate, and retain provenance.
Sync	One active run per user, shared provider budgets, idempotent resume, additive baseline import, scheduled and manual sync through the same queue.
Matching	Exact strong identifiers first; weak/ambiguous candidates go to review. Keep recorded calibration expectations, using public-safe fixtures only.
OA	Store location, license, and provenance. Expose a legal human-clickable location; do not fetch PDFs or arbitrary publisher URLs during the sprint.
Config changes	Validate before applying. Document restart requirements; do not add hot reload for settings whose safe transition is unclear.

Security details follow maintained Go libraries: memory-hard password hashing, secure expiring sessions, CSRF defenses, bounded login attempts, OIDC issuer and signature validation with state/nonce and PKCE, and trusted headers accepted only from explicitly trusted proxies. Reject direct-header spoofing. Isolate user queries and authorize admin operations on the server, not only in the UI.

Provider safety and the sprint's network boundary

Spend Codex quota on coding, testing, and reviewing, not on harvesting papers. Automated tests and the repeatable integration smoke use deterministic local HTTP fixtures. Production clients must be real. Justin additionally authorized a bounded live-provider acceptance check, not bulk acquisition.

After fixture tests prove throttling and circuit breaking, an operator-owned host check may issue at most two requests each to OpenAlex and Crossref, one each to arXiv and Unpaywall, and zero to ORCID until its credentials and non-commercial eligibility are confirmed. Six requests total, including retries; a failure does not replenish the allowance. Use public example works, the operator's existing contact identity, and the escrowed OpenAlex key. Keep credentials, private calibration caches, and the household dump out of writer sandboxes. Do not perform these checks during preparation.

Record actual endpoint, status, timestamp, and remaining allowance without keys or full private payloads. Skip a provider whose prerequisite is missing and report it as unverified, not passed. A 429/auth/ban response stops that provider immediately. Honor the conservative spacing below even for this small check. The application defaults acquisition to disabled until the operator explicitly enables configured providers. Installation never starts a historical crawl; confirmation queues limited work and never bypasses the scheduler.

The shipped provider scheduler must enforce:

• One outstanding request per provider and no burst allowance. Share the limit across users, search, sync, OA lookups, retries, and manual refreshes.
• Persist the next-request time, cooldown, daily allowance consumption, and in-progress work before issuing requests. Restart never clears a cooldown or resets a daily budget. A daily allowance uses the provider's documented reset; if none exists, enforce a rolling 24-hour window.
• Every attempted request, including retries, consumes the budget. Cache by strong identifiers/query parameters; coalesce duplicate queued work. Expose queued, cooling, failed, and disabled states without retry loops in the UI.
• A 429 immediately opens the shared provider circuit. Honor a valid Retry-After date/duration, then permit at most one rate-limited probe. Missing or invalid Retry-After requires an administrator-set cooldown; do not guess a short retry interval. A second 429 in a rolling 24-hour period requires manual re-arm after the administrator reviews the cause.
• On 401/403, a ban/challenge response, or an unexpected redirect, stop the lane and require operator attention. Do not rotate keys, IPs, accounts, or user agents.
• Treat 503 as a provider cooldown, since some APIs use it for burst limiting. Other transient failures get at most two retries with exponential backoff and jitter under the same budgets. Malformed responses stop the work item; they do not trigger an endpoint retry loop. Retain resumable work and surface errors.
• Use honest identification and configured contact details where the provider requires them. An absent required identity or key fails closed.
• Restrict production acquisition to the provider's approved HTTPS host and path. Reject arbitrary base URLs, Google hosts, private/link-local targets, and unapproved redirects. Local fixture endpoints exist only in test configuration.
• Persist budgets per provider credential/IP scope; an application limiter cannot account for other services using tank's IP. Before live enablement, the owner must allocate that shared-IP allowance. Use the smaller of published limits, this product's conservative default, and the allocated operator allowance.

No rate limit can guarantee that a provider will never block an IP. The zero-live sprint budget removes this run's data-API traffic; conservative client limits and circuit breakers constrain a later enabled installation.

Conservative live-enable defaults

These are local engineering limits for a later owner-approved connection check, not promised provider entitlements or the sprint's budget. The sprint budget remains zero. One outstanding request and no burst apply to each row. Official documentation was checked on 2026-10-02.

Provider	Local check allowance	Access and policy constraints
OpenAlex	1 request per 10 seconds; 20 attempts/day; $0.05/day spend cap	Current docs conflict about keyless access. Require an owner-managed API key and policy clarification before live enablement. Price requests by operation; request count alone is not the daily budget. Exclude content/PDF endpoints. Authentication, costs, deprecations.
Crossref	1 request per 10 seconds; 80 attempts/day	Use polite mailto and an identifying User-Agent. Honor rate-interval and concurrency response headers; do not interpret pool counts as a fixed RPS entitlement. No published daily cap was established. Access and authentication.
arXiv	1 request per 6 seconds; 80 attempts/day	Published terms require at least 3 seconds between calls and one connection across all machines under your control. Use the sanctioned query route, not paper scraping. Validate the approved TLS endpoint before live enablement. Terms, manual.
Unpaywall	1 request per 10 seconds; 80 attempts/day	DOI lookup requires a configured email. Docs ask for at most 100,000 calls/day but state no RPS or quota-accounting key. The search route retired on 2026-09-18; do not implement it. API.
ORCID	Disabled; conditional check: 1 request per 10 seconds, 80 reads/day	Require owner confirmation of non-commercial eligibility and an individually registered public client. Implement client-credentials OAuth and fixtures; do not automate account registration. Anonymous/public limits and 503 burst responses are not an invitation to use their burst allowance. Limits, registration, public-client terms.

OpenAlex's published 100-RPS ceiling and daily dollar budgets, and ORCID's published anonymous/public read quotas, do not supersede these smaller local allowances. ORCID Public API terms prohibit use connected with a revenue-generating product/service. Public release must document this constraint and leave that lane disabled unless the operator meets the terms or has an approved Member API path. No paid provider use is authorized by this sprint.

Build order and the acceptance ledger

Use one integration issue, with one verifiable behavior per acceptance box. Nightly works its boxes sequentially on the same branch; each later item inherits the previous verified commit. Both review roles run on each candidate. Keep the specification in this document and evidence/ticks in Forgejo, rather than copying another plan into each role prompt.

Work in this dependency order:

1. Runnable Go process, validated config, SQLite lifecycle, health/readiness, and a default gate that runs offline. The first real feature supplies the Go module; do not spend a separate item on an empty scaffold.
2. Numbered schema migrations, database invariants, public identity, and the data-model's independent acceptance cases. Exercise migrations up/down and serialized concurrent writes before acquisition code uses them.
3. Safe first-admin creation, internal authentication/session recovery, tenant isolation, generic OIDC, and trusted-proxy authentication.
4. The shared persisted provider limiter and block/cooldown behavior, then one documented provider adapter at a time against local fixtures.
5. Exact identity matching, conservative candidate review, author follow/unfollow, publication dedup/provenance, per-user reading, and NEW projection.
6. Restart-safe queued baseline/incremental sync, partial failure, cancellation, dry-run import, audited merge/undo, legal OA resolution, and export.
7. Vue's real API integration and the frozen screens, each a separate checked behavior. No production fixture data and no missing buttons hidden as polish.
8. Safe installation, license, embedded assets/container packaging, offline demo, backup/restore, operational documentation, and the final full-flow smoke.

Every implementation item states its check. Keep tests focused on observable behavior, invariant violations, failure transitions, or a plausible regression. Use a fake clock for cooldowns, not real sleeps. Use the race detector and parser fuzz smoke in the final gate. Synthetic fixtures carry no real household identity. Do not invent a claim that the private calibration corpus passed if the run did not exercise a sanitized copy; list that validation as owner-side acceptance.

Execution envelope

Setting	Proposed value
Project	Scholarr only; base is the preserved sprint-baseline branch.
Work	One integrated product issue, ordered by the build ledger.
Provider/model	openai-codex/gpt-6.1-sol in scope, writer, escalation, spec review, and standards review.
Thinking	xhigh in all five slots. Never change this sprint to max.
Codex budget/floor	100 window points / 0% remaining reserve; the available subscription balance is the practical bound.
Night/issue wall	1,440 active minutes each; subscription pauses do not consume the issue wall, but do consume the night wall.
Issue token guard	100 million non-cached tokens; finite secondary guard, not a target.
Role session	Existing 120-minute cap and idle detection.
Concurrency	One project issue; the two independent review roles may run together.
Repairs/refusals/resources	Existing caps, sandbox isolation, admission reserves, and broker protections.
Data-provider requests	Local fixtures in writer/review sandboxes; separate operator acceptance allowance of six real requests, never a crawl.
End	All acceptance complete, quota exhaustion, wall cutoff, or a reported safety/stuck condition. No filler work to burn remaining quota.

Nightly checks the night wall between role sessions. A role or verification already running may finish after the 24-hour scheduler cutoff; this plan does not claim an exact process-kill deadline. Only reviewed, verified commits count as output. Exhausting quota may interrupt an item and discard its uncommitted work; completed items remain on the integration branch.

Operator-owned ceiling configuration permits the larger wall/token/budget values; ordinary presets keep their existing defaults. The Scholarr preset requests the larger values explicitly. Freeze the runner settings in the saved draft and keep no model fallback. Approval builds/checks the Go and Node image without calling a model. Launch spends subscription quota and requires Justin's explicit go on the resolved plan, not merely permission to prepare it.

Expiring Codex reset credit

Read-only account inspection on 2026-10-02 found 98% of the weekly window remaining, an ordinary weekly reset at 2026-10-09T21:10:30Z, and three saved reset credits. The earliest credit expires at 2026-10-04T05:32:19.368242Z. This saved credit, not the weekly rollover, is the urgent deadline. Re-read live usage and credit details before approval.

Preparation does not redeem a credit. A separate explicit owner go may arm a host-only, single-use supervisor for the soonest-expiring available credit. Use the official installed Codex app-server account/reset-credit protocol, not cache invalidation or guessed private HTTP calls. Pin the expected account and credit; persist one idempotency UUID before redemption. Never expose OAuth credentials to runner images, prompts, logs, or repository files.

The proposed trigger is a quiescent run with at least 99% weekly usage, or the expiry guard at 2026-10-04T05:17:19.368242Z after safely stopping work. The expiry guard explicitly trades remaining old-window quota for preserving the expiring reset. Confirm backend eligibility first; a rejected or ambiguous redemption must not select another credit or create a new UUID. Record and reconcile the result using the same request ID. Redeem at most one credit; never automatically launch a second night or spend the newly reset window.

This policy maximizes useful work, not artificial token burn. Subscription throttling, an early finished product, or an unmet backend reset prerequisite can prevent complete exhaustion; do not promise either exhaustion or redemption.

Final product gate

Before marking the integration issue complete, prove:

• The built binary serves the built Vue assets and survives a SQLite reopen.
• Two users have isolated follows/read state over shared deduplicated works.
• First admin, login/logout, recovery, OIDC, and trusted-header modes behave under both valid and hostile inputs using local identity-provider fixtures.
• An offline seeded smoke drives onboarding, search/import preview, confirmation, baseline sync, restart/resume, incremental sync, review, undo, reading, and export through the real HTTP interface. Tests do not make public-provider calls.
• Injected 429, banned/auth responses, daily exhaustion, redirect rejection, and restart during cooldown prove the provider safeguards with a controlled clock.
• Migrations, race tests, parser fuzz smoke, frontend type/build gates, and a backup/restore into a fresh data directory pass.
• README states exact setup, disabled-provider defaults, key/contact requirements, real-provider enablement, backup/restore, and limits. The shipped app does not contain the design tool runtime, private records, tokens, or placeholder features.

Report unmet boxes as unmet. The bounded live-provider check is part of the sprint acceptance allowance above. Justin's visual review, private-corpus/import validation, and the existing 2-3 week household shadow run remain acceptance before production use; the shadow run cannot finish within this sprint.

Owner go

The launch packet must name the pushed baseline commit, runner commit/release, actual saved plan ID, full resolved role table and caps, image checks, and the integration issue. It must show any failed preflight rather than hiding it.

A launch go approves this product scope and bundled defaults, the six-request public-provider allowance, use of the available Codex subscription at xhigh, and launch of that exact plan. Reset-credit redemption needs explicit approval of the separate single-use trigger policy. Neither go authorizes a main merge, publication, production deployment, real-data import, bulk provider crawl, or a second night. Start launch/monitoring in a fresh session from the approval issue after the matching runner/plugin and reset supervisor preflights pass.