Impact on other sites

From Rangjung Yeshe Wiki Texts
Revision as of 23:02, 21 November 2026 by Eric (talk | contribs) (Report part 2: after 01:49 UTC only content changed; database dumps in ~/lcp/backup)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Part of User:Eric/Report 2026-11-22. Written mainly for Mingyur: what the work of 21–22 November 2026 means for the other sites on mirage-aws, and what to know before the next deploy. The detailed log is on User:Eric/Report 2026-11-22/Changes. Times are UTC.

What each site went through

Site Containers What changed for it Downtime on 22 Nov
rywikitexts.tsadra.org rywikitexts-mediawiki, -mariadb, -redis, -elastic-master Everything in the Changes log: accounts, Apache/PHP/LocalSettings hardening, new image (MW 1.43.12, PHP 8.3.37), 21 submodules updated, Redis password and memory, JobQueueDB, code read-only, Elasticsearch flag, database clean-stop fix, Longchenpa import, Vector.css. 29 s (deploy), ~10 s (database fix), search ~4½ min, 1–2 min (docker upgrade and reboot)
tsadra.org and www.tsadra.org tsadra-wp, tsadra-mariadb, tsadra-redis Database clean-stop fix only: start script backup/entrypoint/start-script-db.sh (last lines), new docker-compose.override.yml, database container recreated with the same image. No change to WordPress, its files, its database content or its configuration. ~10 s without database at 01:12:00–01:12:14; restarts during the docker upgrade (~01:20) and the reboot (01:22:34–~01:24)
dharmacloud.tsadra.org dharmacloud-wp, dharmacloud-mariadb, dharmacloud-redis Same as tsadra.org (database clean-stop fix only). ~10 s at 01:12:14–01:12:23; restarts as above
bodhicitta.tsadra.org (bca) bca-mediawiki, bca-mariadb, bca-redis, bca-elastic-master Database clean-stop fix only: start script init/start-script-db.sh (wiki35 repository), new docker-compose.override.yml, database container recreated with the same image (10.11.19). No change to MediaWiki, LocalSettings, extensions or content. ~10 s at 01:11:47–01:11:57; restarts as above; after the reboot its update.php took 40 s before the site answered
conference.tsadra.org and translationandtransmission.org conference-static Nothing, apart from the restarts. Its database and WordPress containers were already stopped and were not touched. restarts only
Shared containers proxy-web-auto (nginx), docker-gen-auto, letsencrypt-auto, varnish, netdata nginx: 2 new files used only by the rywikitexts site (see below). docker-gen-auto was restarted once to regenerate the configuration; the result for the other sites was the same as before. Nothing else. restarts only

After the reboot every site was checked through nginx on the server itself: all 20 containers were running, the 4 databases started cleanly with no crash recovery, and all 7 host names answered (200 or their usual redirect).

Server-level changes that affect every site

  • Operating system: Amazon Linux release 2023.12.20261120, kernel 6.18.62, docker 25.0.18, containerd 2.3.7, tailscale 1.104.7. Before this, the release was frozen at 2023.12.20260706 and the nightly security update could not install anything. The root crontab line is unchanged; see the follow-ups below.
  • SSH: /etc/ssh/sshd_config.d/00-tsadra-hardening.conf sets PermitRootLogin no and X11Forwarding no. root had no keys, so no access was lost. The Borg backups go out from root and are not affected. Tailscale SSH is separate.
  • MariaDB stop behaviour: on the 4 running databases, docker stop, docker compose down/restart and server shutdowns now stop MariaDB cleanly in 1–2 s. A plain mariadb-admin shutdown now makes the container exit, and its restart policy starts it again.
  • Not changed: the AWS security group, the Cloudflare settings, the Bitbucket repositories, the backup jobs (database dumps at 00:00, Borg at 03:00), and the netdata and varnish containers.

Before the next deploy: things to know

  1. rywikitexts runs a new pinned image. Deploy MediaWiki with all three files:
    docker compose -f docker-compose.yml -f docker-compose.override.yml -f backup/pin-sec-20261121.yml up -d --no-deps mediawiki
    A plain docker compose up -d without the pin would start tsadra/mediawiki:1.43, which is still the old 1.43.9 / PHP 8.3.36 image. The previous pin (backup/pin-oauth-20261116-160811.yml) also points to the old image.
  2. New docker-compose.override.yml files in ~/docker/rywikitexts, bca, tsadra and dharmacloud.
    • Docker compose loads them on its own when no -f is given. When you pass -f files, add the override, or the database (and on rywikitexts, Redis) start without their new settings.
    • rywikitexts' override has two blocks: mariadb (clean stop) and redis (snapshot interval; the password itself is in config/redis.conf).
  3. Local changes in git working trees. These files now differ from the repository:
    • init/start-script-db.sh in rywikitexts and bca, and backup/entrypoint/start-script-db.sh in tsadra and dharmacloud;
    • in rywikitexts also config/apache2.conf, config/php.ini, config/LocalSettings-production.php (two appended blocks, no secrets added), config/redis.conf and config/elastic-search/jvm.options;
    • .env has a new REDIS_PASSWORD.
    A git pull that touches one of these files will fail. A git reset --hard would silently undo the security changes and break Redis (MediaWiki would send a password the old Redis configuration doesn't have), so please don't reset these working trees. Instead, commit the non-secret parts upstream; the backups next to each file show exactly what changed.
  4. init/init-perms-production.sh undoes step B5. It puts group 33 (www-data) and group write back on all extension and skin code, and scripts/upgrade-code-and-deploy.sh runs it on every deploy. A proposed change for the repository is in the kit (init-perms-production.patch).
  5. Submodules: 20 bundled extensions and Vector are at their REL1_43 heads, the same state the deploy script would produce.
  6. Redis on rywikitexts needs the password for any manual check. Use docker exec -e REDISCLI_AUTH="$(sed -n 's/^REDIS_PASSWORD=//p' .env)" rywikitexts-redis redis-cli …, which keeps the password off the command line.
  7. Job queue (rywikitexts): jobs are in the job database table now. showJobs counts can be about 30 s old. The cron runJobs every 10 minutes is unchanged.
  8. MariaDB image on rywikitexts is now the current tsadra/mariadb:10.11 tag (same MariaDB 10.11.19). The old one is still tagged tsadra/mariadb:10.11-rywikitexts-keep.
  9. Upstream for the database fix (all Tsadra wiki and WordPress repositories): end the start script with exec /usr/local/bin/docker-entrypoint.sh mariadbd and use CMD ["/bin/bash", "/start-script-db.sh"] in exec form. After that, the override blocks are no longer needed.
  10. Cloudflare caches load.php for up to 24 hours (s-maxage=86400, from $wgResourceLoaderMaxage['unversioned'] in LocalSettings). Changes to site CSS/JS and module updates reach visitors with that delay. MediaWiki's default is 5 minutes.
  11. TsadraWikiStyles in Vector (scripts/tws.js): the Tools links are added as <div> instead of <li>, which also creates a duplicate id="n-randompage", and Font Awesome icons are added next to Echo's. Both are covered by CSS in MediaWiki:Vector.css for now. The real fix is a skin check in tws.js.

After 01:49: content only

The later work of the night (see Changes) touched only pages of rywikitexts and its two style pages (MediaWiki:Common.css, MediaWiki:Vector.css). No server setting, container or other site was changed.

  • Each import ran from a script in a screen session, after a full dump of the rywikitexts database. The dumps are in /home/ec2-user/lcp/backup/: 5 files, about 1.5 GB in all, readable only by ec2-user. They can be deleted once Eric has checked the pages; the disk has 38 GB free.
  • The imports ran MediaWiki's maintenance scripts directly (link refresh, SMW data, search index, purge), not through the job queue, so they added no jobs.

Rollback

Each step can be undone on its own from /home/ec2-user/sec-2026-11-21/kit/. Rolling back a step brings back the problem it fixed.

Step Command
Database clean stop (per site) bash b1_stopfix.sh rollback ~/docker/<site>, delete the override if the script created it, then docker compose up -d <db service>
New MediaWiki image bash b3_mw_image.sh rollback
Redis / job queue bash b4_redis_jobqueue.sh rollback, then recreate redis and mediawiki
Code read-only bash b5_code_readonly.sh rollback
Elasticsearch flag / SSH bash b6_es_sshd.sh es-rollback / ssh-rollback
Web hardening bash a2_web_hardening.sh rollback
Cloudflare logging bash a3_cloudflare_only.sh rollback
Accounts bash a1_accounts.sh rollback
Longchenpa import see User:Eric/Report 2026-11-22/Longchenpa Project

The operating-system update can't be rolled back as a whole. Older kernels stay installed and can be chosen at boot.

Open follow-ups

These are named here in general terms only. The details are in the private runbook (ask Eric).

  • Rotate the credentials that are stored in the git repositories. This has to go through the repositories (read secrets from the environment, untrack .env), or a later git reset would bring the old values back.
  • Phase 2 of the Cloudflare-only access for rywikitexts, after the log from step A3 has been read.
  • Two-factor authentication for administrators; switch automated logins to bot passwords.
  • Review the grants of the "Tsadra MCP" OAuth consumer.
  • Restrict the monitoring and cache ports on the private network.
  • Decide on Marcus's rights and on the remote script loaded by MediaWiki:Common.js.
  • Make the nightly security update effective again (root crontab), keeping docker and kernel updates for planned windows.
  • Move old evidence directories and old configuration backups off the server, to an encrypted archive.
  • Make the changes above permanent in the repositories (see "Before the next deploy").