justinzeus/scholarr
1
0
Fork 0
Code Issues 2 Pull requests Projects Releases Packages Wiki Activity Actions

Define the integrated Scholarr product sprint

Browse source
This commit is contained in:
justinzeus 2026-10-03 00:02:22 +02:00
parent 41e8714ca3
commit 0e22839ae4
7 changed files with 361 additions and 145 deletions
Download patch file Download diff file Expand all files Collapse all files

314
SPRINT.md Normal file
View file

@ -0,0 +1,314 @@
# Scholarr product sprint
## Approval and authority
Justin authorized preparation, issue triage, and a new baseline commit. He has
not authorized launch. Review this document as the single product and execution
reference for the proposed 24-hour run. Approval tracking: justinzeus/scholarr#9.
Runner preparation: justinzeus/nightly#92; reset supervisor: justinzeus/nightly#93.
An explicit owner go for the concrete plan approves the proposed defaults below
and replaces the old, separate owner-freeze gates for this sprint. The agents
may then choose implementation details and record their rationale here without
stopping for another spec ceremony. They must preserve the locked requirements.
A requirement conflict, missing credential needed for a real feature, or unsafe
provider operation remains a blocker; agents must report it rather than invent
a fallback or claim completion. Preparation alone does not activate this authority.
The reset preserves Git history, the frozen design, and prior decision evidence.
It supplies one new starting commit on an isolated branch. It does not erase the
legacy repository, dirty local files, old decision issues, or production data.
## Intended result
Build a self-hosted author watchlist and publication tracker that Justin's dad
can use daily and other households can install. A reader follows researchers,
sees their publications, tracks read state, resolves ambiguous identities, and
finds legal open-access copies. An administrator can understand provider health,
recover access, audit changes, and back up or restore the installation.
The target after the run is one integrated, runnable product branch and one
reviewed PR, with executable acceptance evidence and a usable installation path.
A pile of independent PRs, a static UI demonstration, mock-only provider clients,
or a service with omitted authentication modes does not meet the target.
Twenty-four hours is a delivery target, not a promise that unfinished work is done.
The run does not deploy, merge into main, publish a release, import the real
household database, or replace the existing workflow. Justin still performs
manual UI approval and the real-data shadow period before a production cutover.
## Locked requirements
- Go backend, Vue 3 frontend, SQLite only. Use a serialized writer, foreign keys,
WAL, numbered migrations, and restart-safe persistence.
- A source-agnostic `FollowedAuthor` is the author root. Names are display/search
labels, never identity keys. Preserve external source identities and provenance.
- Share deduplicated authors and publications globally. Keep follows, read state,
visibility, and user preferences private to their user.
- Support internal username/password, generic OIDC, and trusted-header auth.
Internal auth remains the zero-dependency default and recovery path.
- Use one validated declarative configuration, environment-variable references
for secrets, safe first-admin setup, and a straightforward first-run flow.
- Use the frozen `design/DESIGN.md`, `design/tokens.css`, and bundled assets.
Implement dashboard, publication list/detail, author list/detail, review,
author search/add, bulk import, sources, activity, settings, login, onboarding,
and not-found behavior. Keep loading, empty, populated, and degraded states
honest, with responsive navigation, keyboard access, and light/dark themes.
- Serve system fonts and bundled assets locally. The design proof's
`support.js` and synthetic example records do not ship as product behavior.
- Show source freshness and confidence words rather than invented certainty or
raw match percentages. Keep new-publication counts scoped to the current user.
- Preview bulk imports and destructive admin actions before applying them.
Audit consequential writes and support conflict-checked undo.
- Implement documented clients for OpenAlex, Crossref, arXiv, Unpaywall, and
ORCID. Missing keys disable that client's network lane with an actionable
health state; they do not authorize anonymous scraping or a substitute API.
- Zero application-initiated Google contact. Scholar URLs/IDs are inert import
metadata or human-clickable links. Parse them locally; never dereference them.
No scraping, CAPTCHA solving, anti-bot evasion, or Semantic Scholar v1 expansion.
The data-model draft supplies detailed table contracts and its 22 falsifiable
acceptance cases. The frozen visual design supplies the appearance contract.
This document controls sprint scope and defaults where the old process differs.
## Proposed decisions approved together at launch
| Decision | Sprint default |
|---|---|
| D1: public identity | Internal integer joins; immutable UUIDv7 public IDs only on externally referenced or merge-aliased rows. |
| D2a: unfollow | Hide records without an active follow path; retain read/favorite state and restore it on refollow. |
| D2b: legacy favorites | Retain migrated favorite state for lossless import/export; do not add a favorite UI outside the frozen design. |
| D2c: legacy reading | Treat a publication as read if any legacy link for that user says read. |
| D2d: NEW | Derive at read time from first incremental discovery within 14 days; baseline, manual, and legacy imports are not NEW. |
| D3: attachment | Reuse exact identities, explicit provider crosswalks, and completed high-confidence calibration evidence. Review weaker or conflicting evidence; names alone never attach or merge. |
| D4: undo | No arbitrary expiry while recorded row-version preconditions hold. A conflict prevents the entire undo. |
| D5: orphans | Retain automatically. Admin garbage collection requires a backup, dry-run preview, reference checks, and audit. |
| D6: shared identity | Only administrators merge shared identity or write global negative-identity evidence. Reader decisions remain per-user. |
| D7: purge | Hard purge with no resurrection tombstone. A later sanctioned strong-ID ingest can create a new canonical row; legacy-only data cannot recreate it automatically. |
| Distribution | A self-contained Go binary with embedded built Vue assets, plus a non-root container image. Compose is an installation example, not a live deployment. |
| License | Propose MIT with Justin's personal copyright identity. Add the license only after the owner approves this packet. |
| Acquisition | OpenAlex is primary. Crossref/arXiv add work metadata; ORCID anchors identity; Unpaywall resolves legal OA locations. Persist cursors, deduplicate, and retain provenance. |
| Sync | One active run per user, shared provider budgets, idempotent resume, additive baseline import, scheduled and manual sync through the same queue. |
| Matching | Exact strong identifiers first; weak/ambiguous candidates go to review. Keep recorded calibration expectations, using public-safe fixtures only. |
| OA | Store location, license, and provenance. Expose a legal human-clickable location; do not fetch PDFs or arbitrary publisher URLs during the sprint. |
| Config changes | Validate before applying. Document restart requirements; do not add hot reload for settings whose safe transition is unclear. |
Security details follow maintained Go libraries: memory-hard password hashing,
secure expiring sessions, CSRF defenses, bounded login attempts, OIDC issuer and
signature validation with state/nonce and PKCE, and trusted headers accepted only
from explicitly trusted proxies. Reject direct-header spoofing. Isolate user
queries and authorize admin operations on the server, not only in the UI.
## Provider safety and the sprint's network boundary
Spend Codex quota on coding, testing, and reviewing, not on harvesting papers.
Automated tests and the repeatable integration smoke use deterministic local
HTTP fixtures. Production clients must be real. Justin additionally authorized
a bounded live-provider acceptance check, not bulk acquisition.
After fixture tests prove throttling and circuit breaking, an operator-owned
host check may issue at most **two requests each** to OpenAlex and Crossref,
**one each** to arXiv and Unpaywall, and **zero** to ORCID until its credentials
and non-commercial eligibility are confirmed. Six requests total, including
retries; a failure does not replenish the allowance. Use public example works,
the operator's existing contact identity, and the escrowed OpenAlex key.
Keep credentials, private calibration caches, and the household dump out of
writer sandboxes. Do not perform these checks during preparation.
Record actual endpoint, status, timestamp, and remaining allowance without keys
or full private payloads. Skip a provider whose prerequisite is missing and
report it as unverified, not passed. A 429/auth/ban response stops that provider
immediately. Honor the conservative spacing below even for this small check.
The application defaults acquisition to disabled until the operator explicitly
enables configured providers. Installation never starts a historical crawl;
confirmation queues limited work and never bypasses the scheduler.
The shipped provider scheduler must enforce:
- One outstanding request per provider and no burst allowance. Share the limit
across users, search, sync, OA lookups, retries, and manual refreshes.
- Persist the next-request time, cooldown, daily allowance consumption, and
in-progress work before issuing requests. Restart never clears a cooldown or
resets a daily budget. A daily allowance uses the provider's documented reset;
if none exists, enforce a rolling 24-hour window.
- Every attempted request, including retries, consumes the budget. Cache by
strong identifiers/query parameters; coalesce duplicate queued work. Expose
queued, cooling, failed, and disabled states without retry loops in the UI.
- A 429 immediately opens the shared provider circuit. Honor a valid
`Retry-After` date/duration, then permit at most one rate-limited probe.
Missing or invalid `Retry-After` requires an administrator-set cooldown;
do not guess a short retry interval. A second 429 in a rolling 24-hour period
requires manual re-arm after the administrator reviews the cause.
- On 401/403, a ban/challenge response, or an unexpected redirect, stop the lane
and require operator attention. Do not rotate keys, IPs, accounts, or user agents.
- Treat 503 as a provider cooldown, since some APIs use it for burst limiting.
Other transient failures get at most two retries with exponential backoff and
jitter under the same budgets. Malformed responses stop the work item; they
do not trigger an endpoint retry loop. Retain resumable work and surface errors.
- Use honest identification and configured contact details where the provider
requires them. An absent required identity or key fails closed.
- Restrict production acquisition to the provider's approved HTTPS host and path.
Reject arbitrary base URLs, Google hosts, private/link-local targets, and
unapproved redirects. Local fixture endpoints exist only in test configuration.
- Persist budgets per provider credential/IP scope; an application limiter cannot
account for other services using tank's IP. Before live enablement, the owner
must allocate that shared-IP allowance. Use the smaller of published limits,
this product's conservative default, and the allocated operator allowance.
No rate limit can guarantee that a provider will never block an IP. The zero-live
sprint budget removes this run's data-API traffic; conservative client limits and
circuit breakers constrain a later enabled installation.
### Conservative live-enable defaults
These are local engineering limits for a later owner-approved connection check,
not promised provider entitlements or the sprint's budget. The sprint budget
remains zero. One outstanding request and no burst apply to each row.
Official documentation was checked on 2026-10-02.
| Provider | Local check allowance | Access and policy constraints |
|---|---|---|
| OpenAlex | 1 request per 10 seconds; 20 attempts/day; $0.05/day spend cap | Current docs conflict about keyless access. Require an owner-managed API key and policy clarification before live enablement. Price requests by operation; request count alone is not the daily budget. Exclude content/PDF endpoints. [Authentication](https://help.openalex.org/api/authentication/), [costs](https://help.openalex.org/access/example-costs/), [deprecations](https://help.openalex.org/api/deprecations/). |
| Crossref | 1 request per 10 seconds; 80 attempts/day | Use polite `mailto` and an identifying User-Agent. Honor rate-interval and concurrency response headers; do not interpret pool counts as a fixed RPS entitlement. No published daily cap was established. [Access and authentication](https://www.crossref.org/documentation/retrieve-metadata/rest-api/access-and-authentication/). |
| arXiv | 1 request per 6 seconds; 80 attempts/day | Published terms require at least 3 seconds between calls and one connection across all machines under your control. Use the sanctioned query route, not paper scraping. Validate the approved TLS endpoint before live enablement. [Terms](https://info.arxiv.org/help/api/tou.html), [manual](https://info.arxiv.org/help/api/user-manual.html). |
| Unpaywall | 1 request per 10 seconds; 80 attempts/day | DOI lookup requires a configured email. Docs ask for at most 100,000 calls/day but state no RPS or quota-accounting key. The search route retired on 2026-09-18; do not implement it. [API](https://unpaywall.org/products/api). |
| ORCID | Disabled; conditional check: 1 request per 10 seconds, 80 reads/day | Require owner confirmation of non-commercial eligibility and an individually registered public client. Implement client-credentials OAuth and fixtures; do not automate account registration. Anonymous/public limits and 503 burst responses are not an invitation to use their burst allowance. [Limits](https://info.orcid.org/ufaqs/what-are-the-api-limits/), [registration](https://info.orcid.org/documentation/integration-guide/registering-a-public-api-client/), [public-client terms](https://info.orcid.org/public-client-terms-of-service/). |
OpenAlex's published 100-RPS ceiling and daily dollar budgets, and ORCID's
published anonymous/public read quotas, do not supersede these smaller local
allowances. ORCID Public API terms prohibit use connected with a
revenue-generating product/service. Public release must document this constraint
and leave that lane disabled unless the operator meets the terms or has an
approved Member API path. No paid provider use is authorized by this sprint.
## Build order and the acceptance ledger
Use **one integration issue**, with one verifiable behavior per acceptance box.
Nightly works its boxes sequentially on the same branch; each later item inherits
the previous verified commit. Both review roles run on each candidate. Keep the
specification in this document and evidence/ticks in Forgejo, rather than copying
another plan into each role prompt.
Work in this dependency order:
1. Runnable Go process, validated config, SQLite lifecycle, health/readiness, and
a default gate that runs offline. The first real feature supplies the Go module;
do not spend a separate item on an empty scaffold.
2. Numbered schema migrations, database invariants, public identity, and the
data-model's independent acceptance cases. Exercise migrations up/down and
serialized concurrent writes before acquisition code uses them.
3. Safe first-admin creation, internal authentication/session recovery, tenant
isolation, generic OIDC, and trusted-proxy authentication.
4. The shared persisted provider limiter and block/cooldown behavior, then one
documented provider adapter at a time against local fixtures.
5. Exact identity matching, conservative candidate review, author follow/unfollow,
publication dedup/provenance, per-user reading, and NEW projection.
6. Restart-safe queued baseline/incremental sync, partial failure, cancellation,
dry-run import, audited merge/undo, legal OA resolution, and export.
7. Vue's real API integration and the frozen screens, each a separate checked
behavior. No production fixture data and no missing buttons hidden as polish.
8. Safe installation, license, embedded assets/container packaging, offline demo,
backup/restore, operational documentation, and the final full-flow smoke.
Every implementation item states its check. Keep tests focused on observable
behavior, invariant violations, failure transitions, or a plausible regression.
Use a fake clock for cooldowns, not real sleeps. Use the race detector and parser
fuzz smoke in the final gate. Synthetic fixtures carry no real household identity.
Do not invent a claim that the private calibration corpus passed if the run did
not exercise a sanitized copy; list that validation as owner-side acceptance.
## Execution envelope
| Setting | Proposed value |
|---|---|
| Project | Scholarr only; base is the preserved sprint-baseline branch. |
| Work | One integrated product issue, ordered by the build ledger. |
| Provider/model | `openai-codex/gpt-6.1-sol` in scope, writer, escalation, spec review, and standards review. |
| Thinking | `xhigh` in all five slots. Never change this sprint to `max`. |
| Codex budget/floor | 100 window points / 0% remaining reserve; the available subscription balance is the practical bound. |
| Night/issue wall | 1,440 active minutes each; subscription pauses do not consume the issue wall, but do consume the night wall. |
| Issue token guard | 100 million non-cached tokens; finite secondary guard, not a target. |
| Role session | Existing 120-minute cap and idle detection. |
| Concurrency | One project issue; the two independent review roles may run together. |
| Repairs/refusals/resources | Existing caps, sandbox isolation, admission reserves, and broker protections. |
| Data-provider requests | Local fixtures in writer/review sandboxes; separate operator acceptance allowance of six real requests, never a crawl. |
| End | All acceptance complete, quota exhaustion, wall cutoff, or a reported safety/stuck condition. No filler work to burn remaining quota. |
Nightly checks the night wall between role sessions. A role or verification
already running may finish after the 24-hour scheduler cutoff; this plan does not
claim an exact process-kill deadline. Only reviewed, verified commits count as
output. Exhausting quota may interrupt an item and discard its uncommitted work;
completed items remain on the integration branch.
Operator-owned ceiling configuration permits the larger wall/token/budget values;
ordinary presets keep their existing defaults. The Scholarr preset requests the
larger values explicitly. Freeze the runner settings in the saved draft and keep
no model fallback. Approval builds/checks the Go and Node image without calling a
model. Launch spends subscription quota and requires Justin's explicit go on the
resolved plan, not merely permission to prepare it.
### Expiring Codex reset credit
Read-only account inspection on 2026-10-02 found 98% of the weekly window
remaining, an ordinary weekly reset at **2026-10-09T21:10:30Z**, and three saved
reset credits. The earliest credit expires at
**2026-10-04T05:32:19.368242Z**. This saved credit, not the weekly rollover, is
the urgent deadline. Re-read live usage and credit details before approval.
Preparation does not redeem a credit. A separate explicit owner go may arm a
host-only, single-use supervisor for the soonest-expiring available credit.
Use the official installed Codex app-server account/reset-credit protocol, not
cache invalidation or guessed private HTTP calls. Pin the expected account and
credit; persist one idempotency UUID before redemption. Never expose OAuth
credentials to runner images, prompts, logs, or repository files.
The proposed trigger is a quiescent run with at least 99% weekly usage, or the
expiry guard at **2026-10-04T05:17:19.368242Z** after safely stopping work.
The expiry guard explicitly trades remaining old-window quota for preserving
the expiring reset. Confirm backend eligibility first; a rejected or ambiguous
redemption must not select another credit or create a new UUID. Record and
reconcile the result using the same request ID. Redeem at most one credit;
never automatically launch a second night or spend the newly reset window.
This policy maximizes useful work, not artificial token burn. Subscription
throttling, an early finished product, or an unmet backend reset prerequisite
can prevent complete exhaustion; do not promise either exhaustion or redemption.
## Final product gate
Before marking the integration issue complete, prove:
- The built binary serves the built Vue assets and survives a SQLite reopen.
- Two users have isolated follows/read state over shared deduplicated works.
- First admin, login/logout, recovery, OIDC, and trusted-header modes behave under
both valid and hostile inputs using local identity-provider fixtures.
- An offline seeded smoke drives onboarding, search/import preview, confirmation,
baseline sync, restart/resume, incremental sync, review, undo, reading, and export
through the real HTTP interface. Tests do not make public-provider calls.
- Injected 429, banned/auth responses, daily exhaustion, redirect rejection, and
restart during cooldown prove the provider safeguards with a controlled clock.
- Migrations, race tests, parser fuzz smoke, frontend type/build gates, and a
backup/restore into a fresh data directory pass.
- README states exact setup, disabled-provider defaults, key/contact requirements,
real-provider enablement, backup/restore, and limits. The shipped app does not
contain the design tool runtime, private records, tokens, or placeholder features.
Report unmet boxes as unmet. The bounded live-provider check is part of the
sprint acceptance allowance above. Justin's visual review, private-corpus/import
validation, and the existing 2-3 week household shadow run remain acceptance
before production use; the shadow run cannot finish within this sprint.
## Owner go
The launch packet must name the pushed baseline commit, runner commit/release,
actual saved plan ID, full resolved role table and caps, image checks, and the
integration issue. It must show any failed preflight rather than hiding it.
A launch go approves this product scope and bundled defaults, the six-request
public-provider allowance, use of the available Codex subscription at `xhigh`,
and launch of that exact plan. Reset-credit redemption needs explicit approval
of the separate single-use trigger policy. Neither go authorizes a main merge,
publication, production deployment, real-data import, bulk provider crawl,
or a second night. Start launch/monitoring in a fresh session from the approval
issue after the matching runner/plugin and reset supervisor preflights pass.