karakeep-migration

Migrate one user's data from Linkwarden to Karakeep via API→API, preserving what the built-in importers drop: the collection hierarchy (→ nested Karakeep lists), tags, original creation dates, and descriptions (→ notes).

Part of the TrueNAS-Admin Linkwarden→Karakeep migration (docs/karakeep-migration-roadmap.md in that project). Both apps run on the homelab TrueNAS; Linkwarden stays running untouched as a read-only archive, so rollback is trivial.

What it does

1. Reads Linkwarden GET /api/v1/collections and rebuilds the tree in Karakeep as nested lists (POST /api/v1/lists with parentId), depth-first.
2. Reads all Linkwarden links (GET /api/v1/links?cursor=…, paginated) and for each:
   • POST /api/v1/bookmarks (type=link, url, title←name, note←description, createdAt←original),
   • attaches tags (POST /api/v1/bookmarks/{id}/tags),
   • adds it to its collection's list (PUT /api/v1/lists/{listId}/bookmarks/{bmId}).
3. Records every linkwardenId → karakeepId mapping in a JSONL manifest so re-runs are safe.

Each created bookmark triggers an asynchronous crawl + full-page archive on the NAS — that, not the API writes, is the heavy part. Run the big import overnight.

Requirements

• Python 3 (stdlib only — no pip installs; runs unmodified on the TrueNAS host).

• Credentials in the environment (never committed):

  Var	Meaning
  LINKWARDEN_URL	default https://linkwarden.riera.co.uk
  LINKWARDEN_ACCESS_TOKEN	the user's Linkwarden access token (Settings → Access Tokens)
  KARAKEEP_URL	default https://karakeep.riera.co.uk
  KARAKEEP_KEY	the user's Karakeep API key (Settings → API Keys)

  Load them with: set -a; source ~/.tokens; set +a

Usage

# 1. Dry-run (DEFAULT): counts, tag census, tree preview — writes nothing
python3 migrate.py --user marc

# 2. Perform the migration (idempotent; safe to re-run after an interruption)
python3 migrate.py --user marc --run

# 3. After the crawls settle (hours later), list URLs whose crawl failed
python3 migrate.py --user marc --report-dead

Run the real migration detached on the NAS so it survives a disconnect:

set -a; source ~/.tokens; set +a
tmux new -s kkmig 'python3 migrate.py --user marc --run 2>&1 | tee manifests/run-marc.log'

Options: --rate (req/s, default 3), --manifest-dir (default manifests/, gitignored), --report-dir (default reports/, committed).

Idempotency & safety

• Manifest (manifests/manifest-<user>.jsonl) maps Linkwarden ids → Karakeep ids for both lists and bookmarks. On restart, already-migrated items are skipped. A bookmark record is written only after its tags + list membership also succeed, so an interrupted item is redone cleanly.
• Karakeep dedupes by URL: re-creating a URL returns the existing bookmark with alreadyExists=true, which is treated as success.
• Linkwarden is never written to — read-only source.
• The manifest can contain your data (id maps, not secrets) and is git-ignored; .tokens is never read from or written to the repo. Only code + reports/dead-links-*.txt are committed.

Verification (per user)

After --run:

• Compare counts: Linkwarden links vs Karakeep bookmarks, collections vs lists, tag census.
• Spot-check ~20 random manifest mappings (open both sides).
• Meilisearch search sanity checks in the Karakeep UI.
• --report-dead explains the delta from dead URLs; dupes are logged as alreadyExists.