wiki:MoveOffTrac

Version 30 (modified by btlogy, at 2025-08-22T16:00:29Z) (diff)

--

Move Off Trac

The goal of this page is to cover the phases of a project aiming at moving some critical features from Trac to an other solution (or combination of).

More information about the start of this project can be found in ticket #4095.

Discussions also happened in Nuts&Bolts meetings: see from WeeklyMeeting#April22024.

In addition, and as mentioned in the deliverables, more details can be found in this dedicated ​MoveOffTrac repository.

Scope

Goals and Requirements

  1. MUST replace Trac as currently used for Ticket and Wiki by some alternative(s):
    • MUST look better: current UI looks old which makes the Tahoe-LAFS project looking dead
    • MUST be better maintained: Trac is dying? max 2 contributors per month - ​https://openhub.net/p/trac
    • MUST be easier to maintain: Trac is difficult to update (current = v1.0.13/2016-09-11, latest = v1.6.0/2023-09-23)?
    • MUST allow Self-Registration (Trac requires manual registration via email)
    • COULD support OAuth2 with Github?
    • MUST be (F)OSS
    • MUST be self-hostable
      • note "able": MUST be a story for getting to a self-hosted instance if we want
      • ..but doesn't have to be self-hosted right away
  2. MUST replace the current landing page (start/home page from the Trac/Wiki?):
  3. MUST replace the current binary repository for Tahoe-LAFS releases (​https://tahoe-lafs.org/downloads)
    • by providing at least a similar way to transfer files via ssh
  4. COULD be used to replace:
    • Github code hosting and review (pull request), keeping only a mirrored clone
    • Github Actions (to avoid leaving secrets in environment variables)
    • Circle CI (to avoid giving them too many permissions)

Inclusions

  • Trac users of the Tahoe-LAFS project
  • Trac issues of the Tahoe-LAFS project
  • Trac wiki pages of the Tahoe-LAFS project
  • Trac HTML home page of the Tahoe-LAFS project
  • Hall of Fame HTML page of the Tahoe-LAFS project
  • The related DNS records (mostly: tahoe-lafs.org)

Exclusions

  • Other Trac projects (see DevInfra)
  • Buildbot master instances for some other Trac projects
  • DARCS SCM for some of the other Trac projects
  • Any other services provided by the current server and not yet documented in DevInfra

Deliverables

  1. A VPS (hosted by Hetzner) providing the following features powered by NixOS and Gitea or Forgejo:
    • a tracking system provisioned with the issues migrated from Trac (same numbers)
    • a Wiki system provisioned with the relevant pages migrated from Trac (same names)
    • a static website replacing the landing page from Trac with the code required for CI and CD
    • a blog post for the Hall of Fame page (if sensible - fallback = static page)
    • a (Git) repository defining the VPS it-self and its configuration as code (including the secrets using sops)
    • optionally including the ToFu code allowing to manage the related DNS records (if Gandi supports it)
  2. A detailed migration plan to handle the transition (DNS changes and/or redirections)
    • documented in a this wiki page
    • covering the possible manual steps related to DNS records, HTTP redirections or URL rewriting
    • with a (Git) repository including/referring to the tools used to actually migrate
  3. An high-level migration plan for the features hosted on the VPS described to be later hosted on CodeBerg SaaS (assuming it is possible)
    • documented in this wiki page too

Deliverable 1 - self-hosted server

  1. DONE - Testing/PoC (retired):
  2. COMPLETE - Production VPS with configured Forgejo only (details in #4161)
    • ✅ Infrastructure as Code with CI/CD for ToFu and Nix in ​GitHub/ Tahoe-LAFS/infrastructure repository
    • ✅ Hetzner VPS managed from ToFu code
    • ✅ NixOS server managed from Nix code (webforge)
    • ⛔ DNS records managed from ToFu code in tahoe-lafs.org domain (no proper access - see #4162)
    • ✅ WORKAROUND: DNS records managed from ToFu code in of.tahoe-lafs.org subdomain
    • ✅ Nightly local and remote backups in place with rsnapshot and BorgBase
    • ✅ Credentials shared via ​sops and ​pass repositories
    • ✅ Forgejo with basic its requirements (DB and mailer) managed from Nix code
    • ✅ Forgejo configured with OAuth2 SSO with CodeBerg, GitHub and GitLab, a ​Tahoe-LAFS org and some well-known users
  3. IN PROGRESS - Website only with redirect/proxy rules to legacy resources
    • ✅ web-landing-page source code with CI/CD up and running on ​Tahoe-LAFS/web-landing-page
    • ✅ web-landing-page with redirect/proxy rules hosted on webforge and managed from Nix code - ​https://home.of.tahoe-lafs.org/
    • ⛔ Manual update of DNS records to expose the web-landing-page (see #4183)
    • ⏹️ Migrate the Hall of Fame as a post or a page (not worked on)
  4. PENDING - Issues and wiki hosted by Forgejo
    • ⛔ Extract the Trac data and reconfiguration in RO via direct access to the legacy server
    • ⏸️ WORKAROUND: asynchronous extraction of Trac data via Meejah and avoid POST via proxy rules
    • ⏸️ Rework the redirect/proxy rules from Nix code to expose Forgejo instead of Trac
    • ⏸️ Execute the migration plan as described in Deliverable 2
    • ⏸️ Rework Forgejo configuration to allow new user registration
  5. TODO - Other resources from the Linode
    • ⏹️ Replace the current binary repository for Tahoe-LAFS releases (not worked on)

Deliverable 2 - detailed migration plan from Trac to self-hosted Forgejo

  • Requirements:
    • Code: ​MoveOffTrac and ​trac2gitea sub-module
    • Credentials:
      • SSH key to access Trac data on Linode server
      • API token to write content under the Forgejo org/owner
  • Steps:
    1. On Trac server (the export is scripted in helpers/trac_dump.sh in ​MoveOffTrac):
      1. Activate the Python environment if needed
      2. Launch a trac-admin <path/to/project> shell and run the hotcopy <dump_dir> command
      3. Purge session from the hotcopy with trac-admin <dump_dir> and session purge ...
      4. Purge db backup from rm -rf <dump_dir>/db/trac-backup*
      5. Purge log files from the hotcopy rm -f <dump_dir>/log/*
      6. Compact the DB with sqlite3 <dump_dir>/db/trac.db 'VACUUM;'
      7. Copy trac.htdigest to <dump_dir>
      8. Copy trac public_html to <dump_dir>
      9. Archive the Trac data tar -czvf <dump_file> <dump_dir>
    2. On a workstation:
      1. Checkout ​MoveOffTrac and ​trac2giteam sub-module (e.g. in ~/MoveOffTrac)
      2. Restore the dump with tar -C trac/project -xzvf <dump_file> --strip-components=1
      3. Adapt or patch the configuration with patch trac/project/conf/trac.ini < helpers/trac.ini.patch if needed
      4. Launch a docker-compose run --rm trac bash -c "trac-admin project" session and run permission add <username_here> admin if needed
      5. Start the server with docker-compose run --rm --service-ports trac
      6. Open a browser on ​http://localhost:8000/project/ and verify the service (including the login if needed)
      7. Edit the WikiStart page to replace link: /~trac/ -> ../chrome/site/ or transform them as attachment if needed
    3. On the Forgejo server:
      1. Checkout ​MoveOffTrac and ​trac2giteam sub-module (e.g. in ~/MoveOffTrac)
      2. Create an empty repository and (e.g. trac) and disable unwanted features (e.g. Unit/Code?)
      3. Create an empty Wiki home page in the repository prepared above
      4. Build trac2gitea using the instructions provided (in ​trac2gitea)
      5. Transfer the Trac project directory from the workstation (previous sextion) (e.g.: in ~/MoveOffTrac/trac/project)
      6. Migrate the Trac project to Forgejo: 
          export TOKEN=<token>
  export OWNER=tahoe-lafs
  export REPO=trac
  sudo ./trac2gitea/trac2gitea --app-ini /var/lib/forgejo/custom/conf/app.ini --db-only trac/project /var/lib/forgejo $OWNER $REPO
  sudo ./trac2gitea/trac2gitea --app-ini /var/lib/forgejo/custom/conf/app.ini --wiki-only --wiki-token=$TOKEN --wiki-dir=/tmp/$REPO trac/project /var/lib/forgejo $OWNER $REPO
      7. Restart Forgejo: systemctl start forgejo.service
  • Caveats:
    • tables are not always correctly converted to MD (e.g. #1170 - possibly wrongly formatted)
    • some original links are still not converted and points to tahoe-lafs.org (e.g. ​http://!?)
    • tickets authors need to be created as user before the migration to link contributions to their profile on Forgejo
    • changes history of wiki pages will be not be linked to their authors profile on Forgejo (just their name from Trac)

Deliverable 3 - high-level migration plan from self-hosted to an SaaS provider

  1. COMPLETE - Migration plan for the Forgejo repository/ies:
  2. IN PROGRESS - Migration plan for the static website
    • ▶️ Prospects:
      1. ​CodeBerg Pages (or GitHub Pages)
        • will break legacy links (e.g. from GitHub), unless we keep a minimal NixOS VPS with redirect/proxy rules
      2. ​StaticHost (to be tested)
        • might support redirect/proxy rules and avoid broken links
        • could replace the CI/CD too!?
    • ⏹️ Steps for X
    • ⏹️ PoC (optional)