= Move Off Trac =

The goal of this page is to cover the phases of a project aiming at moving some critical features from Trac to an other solution (or combination of).

More information about the start of this project can be found in ticket #4095.

Discussions also happened in Nuts&Bolts meetings: see from WeeklyMeeting#April22024.

In addition, and as mentioned in the deliverables, more details can be found in this dedicated [https://github.com/tahoe-lafs/MoveOffTrac/tree/main/docs MoveOffTrac] repository.

== Scope ==

=== Goals and Requirements ===

1. MUST replace Trac as currently used for Ticket and Wiki by some alternative(s):
   * MUST look better: current UI looks old which makes the Tahoe-LAFS project looking dead
   * MUST be better maintained: Trac is dying? max 2 contributors per month - https://openhub.net/p/trac
   * MUST be easier to maintain: Trac is difficult to update (current = v1.0.13/2016-09-11, latest = v1.6.0/2023-09-23)?
   * MUST allow Self-Registration (Trac requires manual registration via email)
   * COULD support OAuth2 with Github? 
   * MUST be (F)OSS
   * MUST be self-hostable
       * note "able": MUST be a story for getting to a self-hosted instance if we want
       * ..but doesn't have to be self-hosted right away
2. MUST replace the current landing page (start/home page from the !Trac/Wiki):
   * preferably with a static blog/page generator updated from a Git repo (e.g.: (Go)Jekyll, Hugo)
   * possibly with a LAMP stack as discussed: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3892
3. MUST replace the current binary repository for Tahoe-LAFS releases (https://tahoe-lafs.org/downloads)
   * by providing at least a similar way to transfer files via ssh
4. COULD be used to replace:
   * Github code hosting and review (pull request), keeping only a mirrored clone
   * Github Actions (to avoid leaving secrets in environment variables)
   * Circle CI (to avoid giving them too many permissions)

=== Inclusions ===

* Trac users of the Tahoe-LAFS project
* Trac issues of the Tahoe-LAFS project
* Trac wiki pages of the Tahoe-LAFS project
* Trac HTML home page of the Tahoe-LAFS project
* Hall of Fame HTML page of the Tahoe-LAFS project
* The related DNS records (mostly: tahoe-lafs.org)

=== Exclusions ===

* Other Trac projects (see DevInfra)
* Buildbot master instances for some other Trac projects
* DARCS SCM for some of the other Trac projects
* Any other services provided by the current server and not yet documented in DevInfra

=== Deliverables ===

1. A VPS (hosted by Hetzner) providing the following features powered by NixOS and Gitea or Forgejo:
   * a tracking system provisioned with the issues migrated from Trac (same numbers)
   * a Wiki system provisioned with the relevant pages migrated from Trac (same names)
   * a static website replacing the landing page from Trac with the code required for CI and CD
   * a blog post for the Hall of Fame page (if sensible - fallback = static page)
   * a (Git) repository defining the VPS it-self and its configuration as code (including the secrets using sops)
   * optionally including the !ToFu code allowing to manage the related DNS records (if Gandi supports it)
2. A detailed migration plan to handle the transition (DNS changes and/or redirections)
   * documented in a this wiki page
   * covering the possible manual steps related to DNS records, HTTP redirections or URL rewriting
   * with a (Git) repository including/referring to the tools used to actually migrate
3. An high-level migration plan for the features hosted on the VPS described to be later hosted on !CodeBerg SaaS (assuming it is possible)
   * documented in this wiki page too

== Deliverable 1 - self-hosted server ==

1. **DONE** - Testing/PoC (retired):
   * ✅ VPS provided by Hetzner, mostly defined using !ToFu code and deployed by pull request
   * ✅ OS based on NixOS, defined using Nix code and deployed by pull request
   * ✅ Redirections from legacy to new URLs: provided by Nginx, defined using Nix code and deployed by pull request
   * ✅ Issues tracking and Wiki pages: provided by Gitea or Forgejo:
     * https://code.lafs.eval.latfa.net/tahoe-lafs/trac-2024-07-25 (retired)
     * https://forge.lafs.eval.latfa.net/tahoe-lafs/trac (retired)
   * ✅ Static website: provided by Nginx, build from Jekyll code and deployed by pull request
     * source: https://code.lafs.eval.latfa.net/tahoe-lafs/web-landing-page (retired)
     * preview: https://www.lafs.eval.latfa.net/ (retired)
     * live: https://www.lafs.eval.latfa.net/ (retired)
2. **COMPLETE** - Production VPS with configured Forgejo only (details in #4161)
   * ✅ Infrastructure as Code with CI/CD for !ToFu and Nix in [https://github.com/tahoe-lafs/infrastructure GitHub/ Tahoe-LAFS/infrastructure] repository
   * ✅ Hetzner VPS managed from !ToFu code
   * ✅ NixOS server managed from Nix code (`webforge`)
   * ⛔ DNS records managed from !ToFu code in `tahoe-lafs.org` domain (no proper access - see #4162)
   * ✅ WORKAROUND: DNS records managed from !ToFu code in `of.tahoe-lafs.org` subdomain
   * ✅ Nightly local and remote backups in place with rsnapshot and !BorgBase
   * ✅ Credentials shared via [https://github.com/tahoe-lafs/infrastructure sops] and [https://github.com/tahoe-lafs/infrastructure-pass pass] repositories
   * ✅ Forgejo with basic its requirements (DB and mailer) managed from Nix code
   * ✅ Forgejo configured with OAuth2 SSO with !CodeBerg, !GitHub and !GitLab, a [https://forge.of.tahoe-lafs.org/ Tahoe-LAFS] org and some well-known users
3. **IMPEDED** - Website only with redirect/proxy rules to legacy resources
   * ✅ web-landing-page source code with CI/CD up and running on [https://forge.of.tahoe-lafs.org/Tahoe-LAFS/web-landing-page Tahoe-LAFS/web-landing-page]
   * ✅ web-landing-page with redirect/proxy rules hosted on `webforge` and managed from Nix code - https://home.of.tahoe-lafs.org/
   * ⛔ Manual update of DNS records to expose the web-landing-page (see #4183)
   * ⏹️ Migrate the Hall of Fame as a post or a page (not worked on)
4. **PENDING** - Issues and wiki hosted by Forgejo
   * ⛔ Extract the Trac data and reconfiguration in RO via direct access to the legacy server
   * ⏸️ WORKAROUND: asynchronous extraction of Trac data via Meejah and avoid POST via proxy rules
   * ⏸️ Rework the redirect/proxy rules from Nix code to expose Forgejo instead of Trac
   * ⏸️ Execute the migration plan as described in Deliverable 2
   * ⏸️ Rework Forgejo configuration to allow new user registration
5. **TODO** - Other resources from the Linode
   * ⏹️ Replace the current binary repository for Tahoe-LAFS releases (not worked on)

== Deliverable 2 - detailed migration plan from Trac to self-hosted Forgejo ==

* Requirements:
  * Code: [https://github.com/tahoe-lafs/MoveOffTrac MoveOffTrac] and [https://github.com/tahoe-lafs/trac2gitea trac2gitea] sub-module
  * Credentials:
    * SSH key to access Trac data on Linode server
    * API token to write content under the Forgejo org/owner
* Steps:
  1. On Trac server (the export is scripted in `helpers/trac_dump.sh` in [https://github.com/tahoe-lafs/MoveOffTrac MoveOffTrac]):
    1. Activate the  Python environment if needed
    2. Launch a `trac-admin <path/to/project>` shell and run the `hotcopy <dump_dir>` command
    3. Purge session from the hotcopy with `trac-admin <dump_dir>` and `session purge ...`
    4. Purge db backup from `rm -rf <dump_dir>/db/trac-backup*`
    5. Purge log files from the hotcopy `rm -f <dump_dir>/log/*`
    6. Compact the DB with `sqlite3 <dump_dir>/db/trac.db 'VACUUM;'`
    7. Copy trac.htdigest to `<dump_dir>`
    8. Copy trac public_html to `<dump_dir>`
    9. Archive the Trac data `tar -czvf <dump_file> <dump_dir>`
  2. On a workstation:
    1. Checkout [https://github.com/tahoe-lafs/MoveOffTrac MoveOffTrac] and [https://github.com/tahoe-lafs/trac2gitea trac2giteam] sub-module (e.g. in `~/MoveOffTrac`)
    1. Restore the dump with `tar -C trac/project -xzvf <dump_file> --strip-components=1`
    2. Adapt or patch the configuration with `patch trac/project/conf/trac.ini < helpers/trac.ini.patch` if needed
    3. Launch a `docker-compose run --rm trac bash -c "trac-admin project"` session and run `permission add <username_here> admin` if needed
    4. Start the server with `docker-compose run --rm --service-ports trac`
    5. Open a browser on http://localhost:8000/project/ and verify the service (including the login if needed)
    6. Edit the WikiStart page to replace link: `/~trac/` -> `../chrome/site/` or transform them as attachment if needed
  3. On the Forgejo server:
    1. Checkout [https://github.com/tahoe-lafs/MoveOffTrac MoveOffTrac] and [https://github.com/tahoe-lafs/trac2gitea trac2giteam] sub-module (e.g. in `~/MoveOffTrac`)
    2. Create an empty repository and (e.g. `trac`) and disable unwanted features (e.g. Unit/Code)
    3. Create an empty Wiki home page in the repository prepared above
    4. Build trac2gitea using the instructions provided (in [https://github.com/tahoe-lafs/trac2gitea trac2gitea])
    5. Transfer the Trac project directory from the workstation (previous sextion) (e.g.: in `~/MoveOffTrac/trac/project`)
    6. Migrate the Trac project to Forgejo:
       {{{
         export TOKEN=<token>
         export OWNER=tahoe-lafs
         export REPO=trac
         sudo ./trac2gitea/trac2gitea --app-ini /var/lib/forgejo/custom/conf/app.ini --db-only trac/project /var/lib/forgejo $OWNER $REPO
         sudo ./trac2gitea/trac2gitea --app-ini /var/lib/forgejo/custom/conf/app.ini --wiki-only --wiki-token=$TOKEN --wiki-dir=/tmp/$REPO trac/project /var/lib/forgejo $OWNER $REPO
       }}}
    7. Restart Forgejo: `systemctl start forgejo.service`
* Caveats:
  * tables are not always correctly converted to MD (e.g. #1170 - possibly wrongly formatted)
  * some original links are still not converted and points to tahoe-lafs.org (e.g. http://!?)
  * tickets authors need to be created as user before the migration to link contributions to their profile on Forgejo
  * changes history of wiki pages will be not be linked to their authors profile on Forgejo (just their name from Trac)

== Deliverable 3 - high-level migration plan from self-hosted to an SaaS provider ==

1. **COMPLETE** - Migration plan for the Forgejo repository/ies:
   * ✅ Prospects:
     a. [https://codeberg.org/ CodeBerg]
     b. [https://gitea.com/ Gitea]
     c. [https://hostedgitea.com/ Hosted Gitea]
     d. [https://www.stellarhosted.com/gitea/ StellarHosted]
   * ✅ Steps for !CodeBerg:
     1. Sign-in to [https://forge.of.tahoe-lafs.org Forgejo] and create an API token on Forgejo with Read access on repositories own by the [https://forge.of.tahoe-lafs.org/Tahoe-LAFS Tahoe-LAFS org]
     2. Sign-in to !CodeBerg as an owner of the [https://codeberg.org/Tahoe-LAFS Tahoe-LAFS] org
     3. Use the [https://codeberg.org/repo/migrate migration tool] on !CodeBerg to migrate the repository/ies from [https://forge.of.tahoe-lafs.org Forgejo]
     4. Archive the migrated repository/ies on [https://forge.of.tahoe-lafs.org Forgejo]
   * ✅ PoC (optional):
     * Issues and Wiki pages originally hosted on Trac (archived)
       * Outcome: https://codeberg.org/Tahoe-LAFS/trac-2024-07-25
       * Caveats:
         * all links to contributors are broken (likely because !CodeBerg will not allow this)
         * all links to comments are broken (likely because they use the ID from the source!)
         * all the link to attachment are broken (likely because the migration tool does not cover them - yet)
     * Code, issues, pull requests and CI/CD of the new static website (archived)
       * Outcome: https://codeberg.org/Tahoe-LAFS/web-landing-page
       * Caveats:
         * CI runner does not support (yet) docker (work around possible with local installation of ruby+gems)
2. **IN PROGRESS** - Migration plan for the static website
   * ▶️ Prospects:
     a. [https://docs.codeberg.org/codeberg-pages/ CodeBerg Pages] (or !GitHub Pages)
        * will break legacy links (e.g. from !GitHub), unless we keep a minimal NixOS VPS with redirect/proxy rules
     b. [https://www.statichost.eu/docs/ssg-guides/#jekyll StaticHost] (to be tested)
        * might support redirect/proxy rules and avoid broken links
        * could replace the CI/CD too!?
   * ⏹️ Steps for X
   * ⏹️ PoC (optional)