Impact on other sites
Part of User:Eric/Report 2026-11-22. Written mainly for Mingyur: what the work of 21–22 November 2026 means for the other sites on mirage-aws, and what to know before the next deploy. The detailed log is on User:Eric/Report 2026-11-22/Changes. Times are UTC.
What each site went through
| Site | Containers | What changed for it | Downtime on 22 Nov |
|---|---|---|---|
| rywikitexts.tsadra.org | rywikitexts-mediawiki, -mariadb, -redis, -elastic-master | Everything in the Changes log: accounts, Apache/PHP/LocalSettings hardening, new image (MW 1.43.12, PHP 8.3.37), 21 submodules updated, Redis password and memory, JobQueueDB, code read-only, Elasticsearch flag, database clean-stop fix, Longchenpa import, Vector.css. | 29 s (deploy), ~10 s (database fix), search ~4½ min, 1–2 min (docker upgrade and reboot) |
| tsadra.org and www.tsadra.org | tsadra-wp, tsadra-mariadb, tsadra-redis | Database clean-stop fix only: start script backup/entrypoint/start-script-db.sh (last lines), new docker-compose.override.yml, database container recreated with the same image. No change to WordPress, its files, its database content or its configuration. |
~10 s without database at 01:12:00–01:12:14; restarts during the docker upgrade (~01:20) and the reboot (01:22:34–~01:24) |
| dharmacloud.tsadra.org | dharmacloud-wp, dharmacloud-mariadb, dharmacloud-redis | Same as tsadra.org (database clean-stop fix only). | ~10 s at 01:12:14–01:12:23; restarts as above |
| bodhicitta.tsadra.org (bca) | bca-mediawiki, bca-mariadb, bca-redis, bca-elastic-master | Database clean-stop fix only: start script init/start-script-db.sh (wiki35 repository), new docker-compose.override.yml, database container recreated with the same image (10.11.19). No change to MediaWiki, LocalSettings, extensions or content. |
~10 s at 01:11:47–01:11:57; restarts as above; after the reboot its update.php took 40 s before the site answered |
| conference.tsadra.org and translationandtransmission.org | conference-static | Nothing, apart from the restarts. Its database and WordPress containers were already stopped and were not touched. | restarts only |
| Shared containers | proxy-web-auto (nginx), docker-gen-auto, letsencrypt-auto, varnish, netdata | nginx: 2 new files used only by the rywikitexts site (see below). docker-gen-auto was restarted once to regenerate the configuration; the result for the other sites was the same as before. Nothing else. | restarts only |
After the reboot every site was checked through nginx on the server itself: all 20 containers were running, the 4 databases started cleanly with no crash recovery, and all 7 host names answered (200 or their usual redirect).
Server-level changes that affect every site
- Operating system: Amazon Linux release 2023.12.20261120, kernel 6.18.62, docker 25.0.18, containerd 2.3.7, tailscale 1.104.7. Before this, the release was frozen at 2023.12.20260706 and the nightly security update could not install anything. The root crontab line is unchanged; see the follow-ups below.
- SSH:
/etc/ssh/sshd_config.d/00-tsadra-hardening.confsetsPermitRootLogin noandX11Forwarding no. root had no keys, so no access was lost. The Borg backups go out from root and are not affected. Tailscale SSH is separate. - MariaDB stop behaviour: on the 4 running databases,
docker stop,docker compose down/restartand server shutdowns now stop MariaDB cleanly in 1–2 s. A plainmariadb-admin shutdownnow makes the container exit, and its restart policy starts it again. - Not changed: the AWS security group, the Cloudflare settings, the Bitbucket repositories, the backup jobs (database dumps at 00:00, Borg at 03:00), and the netdata and varnish containers.
Before the next deploy: things to know
- rywikitexts runs a new pinned image. Deploy MediaWiki with all three files:
docker compose -f docker-compose.yml -f docker-compose.override.yml -f backup/pin-sec-20261121.yml up -d --no-deps mediawiki
A plaindocker compose up -dwithout the pin would starttsadra/mediawiki:1.43, which is still the old 1.43.9 / PHP 8.3.36 image. The previous pin (backup/pin-oauth-20261116-160811.yml) also points to the old image. - New
docker-compose.override.ymlfiles in~/docker/rywikitexts,bca,tsadraanddharmacloud.- Docker compose loads them on its own when no
-fis given. When you pass-ffiles, add the override, or the database (and on rywikitexts, Redis) start without their new settings. - rywikitexts' override has two blocks:
mariadb(clean stop) andredis(snapshot interval; the password itself is inconfig/redis.conf).
- Docker compose loads them on its own when no
- Local changes in git working trees. These files now differ from the repository:
init/start-script-db.shin rywikitexts and bca, andbackup/entrypoint/start-script-db.shin tsadra and dharmacloud;- in rywikitexts also
config/apache2.conf,config/php.ini,config/LocalSettings-production.php(two appended blocks, no secrets added),config/redis.confandconfig/elastic-search/jvm.options; .envhas a newREDIS_PASSWORD.
- A
git pullthat touches one of these files will fail. Agit reset --hardwould silently undo the security changes and break Redis (MediaWiki would send a password the old Redis configuration doesn't have), so please don't reset these working trees. Instead, commit the non-secret parts upstream; the backups next to each file show exactly what changed.
init/init-perms-production.shundoes step B5. It puts group 33 (www-data) and group write back on all extension and skin code, andscripts/upgrade-code-and-deploy.shruns it on every deploy. A proposed change for the repository is in the kit (init-perms-production.patch).- Submodules: 20 bundled extensions and Vector are at their
REL1_43heads, the same state the deploy script would produce. - Redis on rywikitexts needs the password for any manual check. Use
docker exec -e REDISCLI_AUTH="$(sed -n 's/^REDIS_PASSWORD=//p' .env)" rywikitexts-redis redis-cli …, which keeps the password off the command line. - Job queue (rywikitexts): jobs are in the
jobdatabase table now.showJobscounts can be about 30 s old. The cronrunJobsevery 10 minutes is unchanged. - MariaDB image on rywikitexts is now the current
tsadra/mariadb:10.11tag (same MariaDB 10.11.19). The old one is still taggedtsadra/mariadb:10.11-rywikitexts-keep. - Upstream for the database fix (all Tsadra wiki and WordPress repositories): end the start script with
exec /usr/local/bin/docker-entrypoint.sh mariadbdand useCMD ["/bin/bash", "/start-script-db.sh"]in exec form. After that, the override blocks are no longer needed. - Cloudflare caches
load.phpfor up to 24 hours (s-maxage=86400, from$wgResourceLoaderMaxage['unversioned']in LocalSettings). Changes to site CSS/JS and module updates reach visitors with that delay. MediaWiki's default is 5 minutes. - TsadraWikiStyles in Vector (
scripts/tws.js): the Tools links are added as<div>instead of<li>, which also creates a duplicateid="n-randompage", and Font Awesome icons are added next to Echo's. Both are covered by CSS in MediaWiki:Vector.css for now. The real fix is a skin check in tws.js. - LocalSettings on rywikitexts: glossary off, no jobs in page views (22 Nov 17:35). The repository copy still loads Lingo and Semantic Glossary and has no
$wgJobRunRate. A deploy from it would bring back the timeouts on the large Gyubum and Longchenpa pages, so please put the same three lines into the repository.
After 01:49: content only
The later work of the night (see Changes) touched only pages of rywikitexts and its two style pages (MediaWiki:Common.css, MediaWiki:Vector.css). No server setting, container or other site was changed.
- Each import ran from a script in a
screensession, after a full dump of the rywikitexts database. The dumps are in/home/ec2-user/lcp/backup/: 5 files, about 1.5 GB in all, readable only by ec2-user. They can be deleted once Eric has checked the pages; the disk has 38 GB free. - The imports ran MediaWiki's maintenance scripts directly (link refresh, SMW data, search index, purge), not through the job queue, so they added no jobs.
17:33–17:51: content audit fixes
Only rywikitexts was touched. The one change outside the wiki itself is a scheduled task on the server (below); no other site or container was changed.
- LocalSettings (
config/LocalSettings-production.php):wfLoadExtension( 'SemanticGlossary' );andwfLoadExtension( 'Lingo' );are commented out, and$wgJobRunRate = 0;is appended (block marked 2026-11-22 content audit A-01). The wiki had no glossary, and Lingo made large pages take up to 159 s to display, past PHP's 30 s limit. Jobs no longer run at the end of page views. Backup:config/LocalSettings-production.php.bak-lingo-20261122. - Jobs now run only from the command line:
- the container's own cron,
/run-jobs.shevery 10 minutes (from the image, unchanged); - a runner added on the server,
/home/ec2-user/bin/rywikitexts-runjobs.sh, every 5 minutes from ec2-user's crontab (status in/home/ec2-user/logs/rywikitexts-runjobs.status). The two can run side by side; the server one is not needed now that page views no longer take jobs, and can be removed.
- the container's own cron,
- Outage: a first attempt turned off only Lingo. Semantic Glossary requires Lingo, so rywikitexts answered with errors for about 90 seconds (17:33–17:34) until the change was rolled back. It was redone with both extensions and a check that loads the new settings before they go live.
- Content: 1,449 pages got a new revision and 1,043 pages were deleted, after a dump of the rywikitexts database (
/home/ec2-user/lcp/backup/db_before_content_fix_2026-11-22_1740.sql.gz, 341 MB). Link refresh, purge and search index ran from maintenance scripts. Details: Changes.
Rollback
Each step can be undone on its own from /home/ec2-user/sec-2026-11-21/kit/. Rolling back a step brings back the problem it fixed.
| Step | Command |
|---|---|
| Database clean stop (per site) | bash b1_stopfix.sh rollback ~/docker/<site>, delete the override if the script created it, then docker compose up -d <db service>
|
| New MediaWiki image | bash b3_mw_image.sh rollback
|
| Redis / job queue | bash b4_redis_jobqueue.sh rollback, then recreate redis and mediawiki
|
| Code read-only | bash b5_code_readonly.sh rollback
|
| Elasticsearch flag / SSH | bash b6_es_sshd.sh es-rollback / ssh-rollback
|
| Web hardening | bash a2_web_hardening.sh rollback
|
| Cloudflare logging | bash a3_cloudflare_only.sh rollback
|
| Accounts | bash a1_accounts.sh rollback
|
| Longchenpa import | see User:Eric/Report 2026-11-22/Longchenpa Project |
| Glossary off, job runner (17:35) | bash c1_lingo_jobrunner.sh rollback
|
| Content audit fixes (17:40) | restore the previous revisions or the dump above; the deleted pages from Special:Undelete |
The operating-system update can't be rolled back as a whole. Older kernels stay installed and can be chosen at boot.
Open follow-ups
These are named here in general terms only. The details are in the private runbook (ask Eric).
- Rotate the credentials that are stored in the git repositories. This has to go through the repositories (read secrets from the environment, untrack
.env), or a latergit resetwould bring the old values back. - Phase 2 of the Cloudflare-only access for rywikitexts, after the log from step A3 has been read.
- Two-factor authentication for administrators; switch automated logins to bot passwords.
- Review the grants of the "Tsadra MCP" OAuth consumer.
- Restrict the monitoring and cache ports on the private network.
- Decide on Marcus's rights and on the remote script loaded by MediaWiki:Common.js.
- Make the nightly security update effective again (root crontab), keeping docker and kernel updates for planned windows.
- Move old evidence directories and old configuration backups off the server, to an encrypted archive.
- Make the changes above permanent in the repositories (see "Before the next deploy"), including the 22 November LocalSettings change (glossary off,
$wgJobRunRate = 0).