searxng-research-proxy

Two-instance SearXNG search for the self-hosted deep-research pipeline on truenas (192.168.0.5). Recovers the CAPTCHA-prone search engines (google/brave/startpage) that the home IP loses, without coupling the working direct engines to the VPN.

LDR / research-queue ─► searxng-merge :30055 ─┬─► SearXNG-A :30053  (direct / home IP)
                        (union + dedupe by URL) └─► SearXNG-B :30054  (via gluetun-B, Surfshark VPN)
                                                       gluetun-B exit IP ROTATES hourly

Components

merge.py — the merge proxy (app searxng-merge, host :30055)

Stdlib-only HTTP service. Fans a /search request out to N SearXNG instances (env UPSTREAMS, default A :30053 + B :30054) concurrently and unions the results. It is HTML-native because LDR (local-deep-research) consumes SearXNG by scraping the HTML results page, not the JSON API:

• serves / with 200 so LDR's availability check passes;
• for format=html (LDR default): extracts each instance's <article> result blocks, dedupes by result URL, and interleaves them (A0,B0,A1,B1,…) so B's VPN-recovered results survive LDR's max_results cap even when A also returned a full page;
• for format=json: unions results deduped by URL (for JSON consumers);
• tolerates any upstream being down.

Deployed as a TrueNAS custom app running python /app/merge.py, bind-mounting this file at /app.

rotate.sh — hourly VPN IP rotation (app searxng-vpn)

SearXNG-B shares gluetun-B's network namespace (network_mode: service:gluetun), so a container restart would drop its network. Instead this cycles the tunnel in place via gluetun's control API (PUT /v1/vpn/status stopped→running on :8000), which reconnects to a new server from the SERVER_COUNTRIES pool = a fresh exit IP. Driven by a TrueNAS cronjob (hourly). Resolves the container name dynamically so it survives redeploys.

Deploy / update

Both apps are TrueNAS custom apps managed via the middleware API (never hand-edit compose). To update code: edit here → scp to the app dir on truenas → midclt call app.redeploy <app>. Operational details live in the search-stack skill in the TrueNAS-Admin repo.

Related

• search-stack skill — SearXNG engine management + this architecture.
• deep-research-stack skill — the research engines/daemon/queue that consume this.
• marc/research-queue — the daemon that queues research and reads the merge via LDR.