= Move Off Trac =

The goal of this page is to cover the phases of a project aiming at moving some critical features from Trac to an other solution (or combination of).

More information about the start of this project can be found in ticket #4095.

Discussions also happened in Nuts&Bolts meetings: see from WeeklyMeeting#April22024.

In addition, and as mentioned in the deliverables, more details can be found in this dedicated [https://github.com/tahoe-lafs/MoveOffTrac/tree/main/docs MoveOffTrac] repository.

== Scope ==

=== Goals and Requirements ===

1. MUST replace Trac as currently used for Ticket and Wiki by some alternative(s):
   * MUST look better: current UI looks old which makes the Tahoe-LAFS project looking dead
   * MUST be better maintained: Trac is dying? max 2 contributors per month - https://openhub.net/p/trac
   * MUST be easier to maintain: Trac is difficult to update (current = v1.0.13/2016-09-11, latest = v1.6.0/2023-09-23)?
   * MUST allow Self-Registration (Trac requires manual registration via email)
   * COULD support OAuth2 with Github? 
   * MUST be (F)OSS
   * MUST be self-hostable
       * note "able": MUST be a story for getting to a self-hosted instance if we want
       * ..but doesn't have to be self-hosted right away
2. MUST replace the current landing page (start/home page from the Trac/Wiki):
   * preferably with a static blog/page generator updated from a Git repo (e.g.: (Go)Jekyll, Hugo)
   * possibly with a LAMP stack as discussed: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3892
3. MUST replace the current binary repository for Tahoe-LAFS releases (https://tahoe-lafs.org/downloads)
   * by providing at least a similar way to transfer files via ssh
4. COULD be used to replace:
   * Github code hosting and review (pull request), keeping only a mirrored clone
   * Github Actions (to avoid leaving secrets in environment variables)
   * Circle CI (to avoid giving them too many permissions)

=== Inclusions ===

* Trac users of the Tahoe-LAFS project
* Trac issues of the Tahoe-LAFS project
* Trac wiki pages of the Tahoe-LAFS project
* Trac HTML home page of the Tahoe-LAFS project
* Hall of Fame HTML page of the Tahoe-LAFS project
* The related DNS records (mostly: tahoe-lafs.org)

=== Exclusions ===

* Other Trac projects (see DevInfra)
* Buildbot master instances for some other Trac projects
* DARCS SCM for some of the other Trac projects
* Any other services provided by the current server and not yet documented in DevInfra

=== Deliverables ===

1. A VPS (hosted by Hetzner) providing the following features powered by NixOS and Gitea or Forgejo:
   * a tracking system provisioned with the issues migrated from Trac (same numbers)
   * a Wiki system provisioned with the relevant pages migrated from Trac (same names)
   * a static website replacing the landing page from Trac with the code required for CI and CD
   * a blog post for the Hall of Fame page (if sensible - fallback = static page)
   * a (Git) repository defining the VPS it-self and its configuration as code (including the secrets using sops)
   * optionally including the !ToFu code allowing to manage the related DNS records (if Gandi supports it)
2. A detailed migration plan to handle the transition (DNS changes and/or redirections)
   * documented in a this wiki page
   * covering the possible manual steps related to DNS records, HTTP redirections or URL rewriting
   * with a (Git) repository including/referring to the tools used to actually migrate
3. An high-level migration plan for the features hosted on the VPS described to be later hosted on !CodeBerg SaaS (assuming it is possible)
   * documented in this wiki page too

== Deliverable 1 - self-hosted server ==

1. **DONE** - Testing/PoC (retired):
   * ✅ VPS provided by Hetzner, mostly defined using !ToFu code and deployed by pull request
   * ✅ OS based on NixOS, defined using Nix code and deployed by pull request
   * ✅ Redirections from legacy to new URLs: provided by Nginx, defined using Nix code and deployed by pull request
   * ✅ Issues tracking and Wiki pages: provided by Gitea or Forgejo:
     * https://code.lafs.eval.latfa.net/tahoe-lafs/trac-2024-07-25
     * https://forge.lafs.eval.latfa.net/tahoe-lafs/trac
   * ✅ Static website: provided by Nginx, build from Jekyll code and deployed by pull request
     * source: https://code.lafs.eval.latfa.net/tahoe-lafs/web-landing-page
     * preview: https://www.lafs.eval.latfa.net/
     * live: https://www.lafs.eval.latfa.net/
2. **IN PROGRESS** - Production (details in #4161):
   1. **PENDING** - Website only: VPS + OS + NGinx + website + redirect/proxy rules to legacy resources
      * ✅ Hetzner VPS is up and running on NixOS (webforge.of.tahoe-lafs.org)
      * ✅ Nightly local and remote backups in place with rsnaphot and !BorgBase
      * ✅ Nix and !ToFu source code with CI/CD up and running on [https://github.com/tahoe-lafs/infrastructure GitHub/Tahoe-LAFS/infrastructure]
      * ✅ Forgejo is up and running with 6 well-known users and 3 SSO providers on https://forge.of.tahoe-lafs.org/
      * ✅ web-landing-page source code with CI/CD up and running on [https://forge.of.tahoe-lafs.org/Tahoe-LAFS/web-landing-page Forgejo/Tahoe-LAFS/web-landing-page]
      * ✅ web-landing-page up and running on https://home.of.tahoe-lafs.org/
      * ✅ Credentials are shared via [https://github.com/tahoe-lafs/infrastructure sops] and [https://github.com/tahoe-lafs/infrastructure-pass pass] repositories
      * ⛔ Waiting for the DNS records to be changed in #4183
      * ➕ the Hall of Fame pages till needs to be "migrated"
   2. **PENDING** - All features from the PoC above: issues and wiki hosted by Forgejo
      * ⛔ Extract the data and reconfigure Trac in RO via root access to the legacy server
      * ⏸️ OR asynchronous workaround with Meejah
      * ⏸️ Rework the redirect/proxy rules to use Forgejo instead of Trac 

== Deliverable 2 - migration plan from Trac to self-hosted Gitea or Forgejo ==

* Requirements:
  * Code: [https://github.com/tahoe-lafs/MoveOffTrac MoveOffTrac] and [https://github.com/tahoe-lafs/trac2gitea trac2gitea] submodule
  * Credentials:
    * SSH key to access Trac data on Linode server
    * API token to write content under the Forgejo org/owner
* Steps:
  1 On Trac server (the export is scripted in `helpers/trac_dump.sh` in [https://github.com/tahoe-lafs/MoveOffTrac MoveOffTrac]):
    1. Activate the  Python environment if needed
    2. Launch a `trac-admin <path/to/project>` shell and run the `hotcopy <dump_dir>` command
    3. Purge session from the hotcopy with `trac-admin <dump_dir>` and `session purge ...`
    4. Purge db backup from `rm -rf <dump_dir>/db/trac-backup*`
    5. Purge log files from the hotcopy `rm -f <dump_dir>/log/*`
    6. Compact the DB with `sqlite3 <dump_dir>/db/trac.db 'VACUUM;'`
    7. Copy trac.htdigest to <dump_dir>
    8. Copy trac public_html to <dump_dir>
    9. Archive the Trac data `tar -czvf <dump_file> <dump_dir>`
  2. On a workstation:
    1. Checkout [https://github.com/tahoe-lafs/MoveOffTrac MoveOffTrac] and [https://github.com/tahoe-lafs/trac2gitea trac2giteam] submodule (e.g. in `~/MoveOffTrac`)
    1. Restore the dump with `tar -C trac/project -xzvf <dump_file> --strip-components=1`
    2. Adapt or patch the configuration with `patch trac/project/conf/trac.ini < helpers/trac.ini.patch` if needed
    3. Launch a `docker-compose run --rm trac bash -c "trac-admin project"` session and run `permission add <username_here> admin` if needed
    4. Start the server with `docker-compose run --rm --service-ports trac`
    5. Open a browser on http://localhost:8000/project/ and verify the service (including the login if needed)
    6. Edit the WikiStart page to replace link: `/~trac/` -> `../chrome/site/` or transform them as attachment if needed
  3. On the Forgejo server:
    1. Checkout [https://github.com/tahoe-lafs/MoveOffTrac MoveOffTrac] and [https://github.com/tahoe-lafs/trac2gitea trac2giteam] submodule (e.g. in `~/MoveOffTrac`)
    2. Create an empty repository and (e.g. `trac`) and disable unwanted features (e.g. Unit/Code)
    3. Create an empty Wiki home page in the repository prepared above
    4. Build trac2gitea using the instructions provided (in [https://github.com/tahoe-lafs/trac2gitea trac2gitea])
    5. Transfer the Trac project directory from the workstation (previous sextion) (e.g.: in `~/MoveOffTrac/trac/project`)
    6. Remote: migrate the Trac project to Forgejo:
       {{{
         export TOKEN=<token>
         export OWNER=tahoe-lafs
         export REPO=trac
         sudo ./trac2gitea/trac2gitea --app-ini /var/lib/forgejo/custom/conf/app.ini --db-only trac/project /var/lib/forgejo $OWNER $REPO
         sudo ./trac2gitea/trac2gitea --app-ini /var/lib/forgejo/custom/conf/app.ini --wiki-only --wiki-token=$TOKEN --wiki-dir=/tmp/$REPO trac/project /var/lib/forgejo $OWNER $REPO
       }}}
    7. Remote: `systemctl start forgejo.service`
* Caveats:
  * migration works only with sqlite DB!
  * tables are not always correctly converted to MD (e.g. #1170 - possibly wrongly formatted)
  * some original links are still not converted and points to tahoe-lafs.org (e.g. http://!?)
  * tickets authors need to be created as user before the migration to link contributions to their profile
  * changes history of wiki pages will be not be linked to their authors

== Deliverable 3 - high-level migration plan from self-hosted to an SaaS provider ==

1. **DONE** - Migration of the Forgejo repository/ies:
   * ✅ Plan:
     1. Sign-in to [https://forge.of.tahoe-lafs.org Forgejo] and create an API token on Forgejo with Read access on repositories own by the [https://forge.of.tahoe-lafs.org/Tahoe-LAFS Tahoe-LAFS org]
     2. Sign-in to !CodeBerg as a member of the [https://codeberg.org/Tahoe-LAFS Tahoe-LAFS] org
     3. Use the [https://codeberg.org/repo/migrate migration tool] on !CodeBerg to migrate the repository/ies from [https://forge.of.tahoe-lafs.org Forgejo]
     4. Archive the migrated repository/ies on [https://forge.of.tahoe-lafs.org Forgejo]
   * ✅ PoC (examples):
     * Issues and Wiki pages originally hosted on Trac (archived)
       * Outcome: https://codeberg.org/Tahoe-LAFS/trac-2024-07-25
       * Caveats:
         * all links to contributors are broken (likely because !CodeBerg will not allow this)
         * all links to comments are broken (likely because they use the ID from the source!)
         * all the link to attachment are broken (likely because the migration tool does not cover them - yet)
     * Code, issues, pull requests and CI/CD of the new static website (archived)
       * Outcome: https://codeberg.org/Tahoe-LAFS/web-landing-page
       * Caveats:
         * CI runner does not support (yet) docker (work around possible with local installation of ruby+gems)
2. **IN PROGRESS** - Migration of the static website
   * ▶️ Prospects:
     a. [https://docs.codeberg.org/codeberg-pages/ CodeBerg Pages], which will break legacy links (e.g. from !GitHub)
     b. [https://www.statichost.eu/docs/ssg-guides/#jekyll StaticHost], which might support redirect/proxy rules and avoid broken links.