<?xml version="1.0"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
	<id>https://rywikitexts.tsadra.org/index.php?action=history&amp;feed=atom&amp;title=User%3AEric%2FReport_2026-11-22%2FImpact_on_other_sites</id>
	<title>User:Eric/Report 2026-11-22/Impact on other sites - Revision history</title>
	<link rel="self" type="application/atom+xml" href="https://rywikitexts.tsadra.org/index.php?action=history&amp;feed=atom&amp;title=User%3AEric%2FReport_2026-11-22%2FImpact_on_other_sites"/>
	<link rel="alternate" type="text/html" href="https://rywikitexts.tsadra.org/index.php?title=User:Eric/Report_2026-11-22/Impact_on_other_sites&amp;action=history"/>
	<updated>2026-11-22T03:07:14Z</updated>
	<subtitle>Revision history for this page on the wiki</subtitle>
	<generator>MediaWiki 1.43.12</generator>
	<entry>
		<id>https://rywikitexts.tsadra.org/index.php?title=User:Eric/Report_2026-11-22/Impact_on_other_sites&amp;diff=16675&amp;oldid=prev</id>
		<title>Eric: Report part 2: impact on the other mirage-aws sites, notes for deploys, follow-ups</title>
		<link rel="alternate" type="text/html" href="https://rywikitexts.tsadra.org/index.php?title=User:Eric/Report_2026-11-22/Impact_on_other_sites&amp;diff=16675&amp;oldid=prev"/>
		<updated>2026-11-22T01:49:03Z</updated>

		<summary type="html">&lt;p&gt;Report part 2: impact on the other mirage-aws sites, notes for deploys, follow-ups&lt;/p&gt;
&lt;p&gt;&lt;b&gt;New page&lt;/b&gt;&lt;/p&gt;&lt;div&gt;Part of [[User:Eric/Report 2026-11-22]]. Written mainly for &amp;#039;&amp;#039;&amp;#039;Mingyur&amp;#039;&amp;#039;&amp;#039;: what the work of 21–22 November 2026 means for the other sites on mirage-aws, and what to know before the next deploy. The detailed log is on [[User:Eric/Report 2026-11-22/Changes]]. Times are UTC.&lt;br /&gt;
&lt;br /&gt;
== What each site went through ==&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
! Site !! Containers !! What changed for it !! Downtime on 22 Nov&lt;br /&gt;
|-&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;rywikitexts.tsadra.org&amp;#039;&amp;#039;&amp;#039; || rywikitexts-mediawiki, -mariadb, -redis, -elastic-master || Everything in the Changes log: accounts, Apache/PHP/LocalSettings hardening, new image (MW 1.43.12, PHP 8.3.37), 21 submodules updated, Redis password and memory, JobQueueDB, code read-only, Elasticsearch flag, database clean-stop fix, Longchenpa import, Vector.css. || 29 s (deploy), ~10 s (database fix), search ~4½ min, 1–2 min (docker upgrade and reboot)&lt;br /&gt;
|-&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;tsadra.org&amp;#039;&amp;#039;&amp;#039; and &amp;#039;&amp;#039;&amp;#039;www.tsadra.org&amp;#039;&amp;#039;&amp;#039; || tsadra-wp, tsadra-mariadb, tsadra-redis || &amp;#039;&amp;#039;&amp;#039;Database clean-stop fix only:&amp;#039;&amp;#039;&amp;#039; start script &amp;lt;code&amp;gt;backup/entrypoint/start-script-db.sh&amp;lt;/code&amp;gt; (last lines), new &amp;lt;code&amp;gt;docker-compose.override.yml&amp;lt;/code&amp;gt;, database container recreated with the same image. No change to WordPress, its files, its database content or its configuration. || ~10 s without database at 01:12:00–01:12:14; restarts during the docker upgrade (~01:20) and the reboot (01:22:34–~01:24)&lt;br /&gt;
|-&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;dharmacloud.tsadra.org&amp;#039;&amp;#039;&amp;#039; || dharmacloud-wp, dharmacloud-mariadb, dharmacloud-redis || Same as tsadra.org (database clean-stop fix only). || ~10 s at 01:12:14–01:12:23; restarts as above&lt;br /&gt;
|-&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;bodhicitta.tsadra.org&amp;#039;&amp;#039;&amp;#039; (bca) || bca-mediawiki, bca-mariadb, bca-redis, bca-elastic-master || Database clean-stop fix only: start script &amp;lt;code&amp;gt;init/start-script-db.sh&amp;lt;/code&amp;gt; (wiki35 repository), new &amp;lt;code&amp;gt;docker-compose.override.yml&amp;lt;/code&amp;gt;, database container recreated with the same image (10.11.19). No change to MediaWiki, LocalSettings, extensions or content. || ~10 s at 01:11:47–01:11:57; restarts as above; after the reboot its update.php took 40 s before the site answered&lt;br /&gt;
|-&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;conference.tsadra.org&amp;#039;&amp;#039;&amp;#039; and &amp;#039;&amp;#039;&amp;#039;translationandtransmission.org&amp;#039;&amp;#039;&amp;#039; || conference-static || Nothing, apart from the restarts. Its database and WordPress containers were already stopped and were not touched. || restarts only&lt;br /&gt;
|-&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;Shared containers&amp;#039;&amp;#039;&amp;#039; || proxy-web-auto (nginx), docker-gen-auto, letsencrypt-auto, varnish, netdata || nginx: 2 new files used only by the rywikitexts site (see below). docker-gen-auto was restarted once to regenerate the configuration; the result for the other sites was the same as before. Nothing else. || restarts only&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
After the reboot every site was checked through nginx on the server itself: all 20 containers were running, the 4 databases started cleanly with no crash recovery, and all 7 host names answered (200 or their usual redirect).&lt;br /&gt;
&lt;br /&gt;
== Server-level changes that affect every site ==&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;Operating system&amp;#039;&amp;#039;&amp;#039;: Amazon Linux release 2023.12.20261120, kernel 6.18.62, docker 25.0.18, containerd 2.3.7, tailscale 1.104.7. Before this, the release was frozen at 2023.12.20260706 and the nightly security update could not install anything. The root crontab line is unchanged; see the follow-ups below.&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;SSH&amp;#039;&amp;#039;&amp;#039;: &amp;lt;code&amp;gt;/etc/ssh/sshd_config.d/00-tsadra-hardening.conf&amp;lt;/code&amp;gt; sets &amp;lt;code&amp;gt;PermitRootLogin no&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;X11Forwarding no&amp;lt;/code&amp;gt;. root had no keys, so no access was lost. The Borg backups go out from root and are not affected. Tailscale SSH is separate.&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;MariaDB stop behaviour&amp;#039;&amp;#039;&amp;#039;: on the 4 running databases, &amp;lt;code&amp;gt;docker stop&amp;lt;/code&amp;gt;, &amp;lt;code&amp;gt;docker compose down/restart&amp;lt;/code&amp;gt; and server shutdowns now stop MariaDB cleanly in 1–2 s. A plain &amp;lt;code&amp;gt;mariadb-admin shutdown&amp;lt;/code&amp;gt; now makes the container exit, and its restart policy starts it again.&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;Not changed&amp;#039;&amp;#039;&amp;#039;: the AWS security group, the Cloudflare settings, the Bitbucket repositories, the backup jobs (database dumps at 00:00, Borg at 03:00), and the netdata and varnish containers.&lt;br /&gt;
&lt;br /&gt;
== Before the next deploy: things to know ==&lt;br /&gt;
# &amp;#039;&amp;#039;&amp;#039;rywikitexts runs a new pinned image.&amp;#039;&amp;#039;&amp;#039; Deploy MediaWiki with all three files:&amp;lt;br/&amp;gt;&amp;lt;code&amp;gt;docker compose -f docker-compose.yml -f docker-compose.override.yml -f backup/pin-sec-20261121.yml up -d --no-deps mediawiki&amp;lt;/code&amp;gt;&amp;lt;br/&amp;gt;A plain &amp;lt;code&amp;gt;docker compose up -d&amp;lt;/code&amp;gt; without the pin would start &amp;lt;code&amp;gt;tsadra/mediawiki:1.43&amp;lt;/code&amp;gt;, which is still the old 1.43.9 / PHP 8.3.36 image. The previous pin (&amp;lt;code&amp;gt;backup/pin-oauth-20261116-160811.yml&amp;lt;/code&amp;gt;) also points to the old image.&lt;br /&gt;
# &amp;#039;&amp;#039;&amp;#039;New &amp;lt;code&amp;gt;docker-compose.override.yml&amp;lt;/code&amp;gt; files&amp;#039;&amp;#039;&amp;#039; in &amp;lt;code&amp;gt;~/docker/rywikitexts&amp;lt;/code&amp;gt;, &amp;lt;code&amp;gt;bca&amp;lt;/code&amp;gt;, &amp;lt;code&amp;gt;tsadra&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;dharmacloud&amp;lt;/code&amp;gt;.&lt;br /&gt;
#* Docker compose loads them on its own when no &amp;lt;code&amp;gt;-f&amp;lt;/code&amp;gt; is given. When you pass &amp;lt;code&amp;gt;-f&amp;lt;/code&amp;gt; files, add the override, or the database (and on rywikitexts, Redis) start without their new settings.&lt;br /&gt;
#* rywikitexts&amp;#039; override has two blocks: &amp;lt;code&amp;gt;mariadb&amp;lt;/code&amp;gt; (clean stop) and &amp;lt;code&amp;gt;redis&amp;lt;/code&amp;gt; (snapshot interval; the password itself is in &amp;lt;code&amp;gt;config/redis.conf&amp;lt;/code&amp;gt;).&lt;br /&gt;
# &amp;#039;&amp;#039;&amp;#039;Local changes in git working trees.&amp;#039;&amp;#039;&amp;#039; These files now differ from the repository:&lt;br /&gt;
#* &amp;lt;code&amp;gt;init/start-script-db.sh&amp;lt;/code&amp;gt; in rywikitexts and bca, and &amp;lt;code&amp;gt;backup/entrypoint/start-script-db.sh&amp;lt;/code&amp;gt; in tsadra and dharmacloud;&lt;br /&gt;
#* in rywikitexts also &amp;lt;code&amp;gt;config/apache2.conf&amp;lt;/code&amp;gt;, &amp;lt;code&amp;gt;config/php.ini&amp;lt;/code&amp;gt;, &amp;lt;code&amp;gt;config/LocalSettings-production.php&amp;lt;/code&amp;gt; (two appended blocks, no secrets added), &amp;lt;code&amp;gt;config/redis.conf&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;config/elastic-search/jvm.options&amp;lt;/code&amp;gt;;&lt;br /&gt;
#* &amp;lt;code&amp;gt;.env&amp;lt;/code&amp;gt; has a new &amp;lt;code&amp;gt;REDIS_PASSWORD&amp;lt;/code&amp;gt;.&lt;br /&gt;
#:A &amp;lt;code&amp;gt;git pull&amp;lt;/code&amp;gt; that touches one of these files will fail. A &amp;lt;code&amp;gt;git reset --hard&amp;lt;/code&amp;gt; would silently undo the security changes and break Redis (MediaWiki would send a password the old Redis configuration doesn&amp;#039;t have), so please don&amp;#039;t reset these working trees. Instead, commit the non-secret parts upstream; the backups next to each file show exactly what changed.&lt;br /&gt;
# &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;init/init-perms-production.sh&amp;lt;/code&amp;gt; undoes step B5.&amp;#039;&amp;#039;&amp;#039; It puts group 33 (www-data) and group write back on all extension and skin code, and &amp;lt;code&amp;gt;scripts/upgrade-code-and-deploy.sh&amp;lt;/code&amp;gt; runs it on every deploy. A proposed change for the repository is in the kit (&amp;lt;code&amp;gt;init-perms-production.patch&amp;lt;/code&amp;gt;).&lt;br /&gt;
# &amp;#039;&amp;#039;&amp;#039;Submodules&amp;#039;&amp;#039;&amp;#039;: 20 bundled extensions and Vector are at their &amp;lt;code&amp;gt;REL1_43&amp;lt;/code&amp;gt; heads, the same state the deploy script would produce.&lt;br /&gt;
# &amp;#039;&amp;#039;&amp;#039;Redis on rywikitexts needs the password&amp;#039;&amp;#039;&amp;#039; for any manual check. Use &amp;lt;code&amp;gt;docker exec -e REDISCLI_AUTH=&amp;quot;$(sed -n &amp;#039;s/^REDIS_PASSWORD=//p&amp;#039; .env)&amp;quot; rywikitexts-redis redis-cli …&amp;lt;/code&amp;gt;, which keeps the password off the command line.&lt;br /&gt;
# &amp;#039;&amp;#039;&amp;#039;Job queue&amp;#039;&amp;#039;&amp;#039; (rywikitexts): jobs are in the &amp;lt;code&amp;gt;job&amp;lt;/code&amp;gt; database table now. &amp;lt;code&amp;gt;showJobs&amp;lt;/code&amp;gt; counts can be about 30 s old. The cron &amp;lt;code&amp;gt;runJobs&amp;lt;/code&amp;gt; every 10 minutes is unchanged.&lt;br /&gt;
# &amp;#039;&amp;#039;&amp;#039;MariaDB image on rywikitexts&amp;#039;&amp;#039;&amp;#039; is now the current &amp;lt;code&amp;gt;tsadra/mariadb:10.11&amp;lt;/code&amp;gt; tag (same MariaDB 10.11.19). The old one is still tagged &amp;lt;code&amp;gt;tsadra/mariadb:10.11-rywikitexts-keep&amp;lt;/code&amp;gt;.&lt;br /&gt;
# &amp;#039;&amp;#039;&amp;#039;Upstream for the database fix&amp;#039;&amp;#039;&amp;#039; (all Tsadra wiki and WordPress repositories): end the start script with &amp;lt;code&amp;gt;exec /usr/local/bin/docker-entrypoint.sh mariadbd&amp;lt;/code&amp;gt; and use &amp;lt;code&amp;gt;CMD [&amp;quot;/bin/bash&amp;quot;, &amp;quot;/start-script-db.sh&amp;quot;]&amp;lt;/code&amp;gt; in exec form. After that, the override blocks are no longer needed.&lt;br /&gt;
# &amp;#039;&amp;#039;&amp;#039;Cloudflare caches &amp;lt;code&amp;gt;load.php&amp;lt;/code&amp;gt; for up to 24 hours&amp;#039;&amp;#039;&amp;#039; (&amp;lt;code&amp;gt;s-maxage=86400&amp;lt;/code&amp;gt;, from &amp;lt;code&amp;gt;$wgResourceLoaderMaxage[&amp;#039;unversioned&amp;#039;]&amp;lt;/code&amp;gt; in LocalSettings). Changes to site CSS/JS and module updates reach visitors with that delay. MediaWiki&amp;#039;s default is 5 minutes.&lt;br /&gt;
# &amp;#039;&amp;#039;&amp;#039;TsadraWikiStyles in Vector&amp;#039;&amp;#039;&amp;#039; (&amp;lt;code&amp;gt;scripts/tws.js&amp;lt;/code&amp;gt;): the Tools links are added as &amp;lt;code&amp;gt;&amp;amp;lt;div&amp;amp;gt;&amp;lt;/code&amp;gt; instead of &amp;lt;code&amp;gt;&amp;amp;lt;li&amp;amp;gt;&amp;lt;/code&amp;gt;, which also creates a duplicate &amp;lt;code&amp;gt;id=&amp;quot;n-randompage&amp;quot;&amp;lt;/code&amp;gt;, and Font Awesome icons are added next to Echo&amp;#039;s. Both are covered by CSS in [[MediaWiki:Vector.css]] for now. The real fix is a skin check in tws.js.&lt;br /&gt;
&lt;br /&gt;
== Rollback ==&lt;br /&gt;
Each step can be undone on its own from &amp;lt;code&amp;gt;/home/ec2-user/sec-2026-11-21/kit/&amp;lt;/code&amp;gt;. Rolling back a step brings back the problem it fixed.&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
! Step !! Command&lt;br /&gt;
|-&lt;br /&gt;
| Database clean stop (per site) || &amp;lt;code&amp;gt;bash b1_stopfix.sh rollback ~/docker/&amp;amp;lt;site&amp;amp;gt;&amp;lt;/code&amp;gt;, delete the override if the script created it, then &amp;lt;code&amp;gt;docker compose up -d &amp;amp;lt;db service&amp;amp;gt;&amp;lt;/code&amp;gt;&lt;br /&gt;
|-&lt;br /&gt;
| New MediaWiki image || &amp;lt;code&amp;gt;bash b3_mw_image.sh rollback&amp;lt;/code&amp;gt;&lt;br /&gt;
|-&lt;br /&gt;
| Redis / job queue || &amp;lt;code&amp;gt;bash b4_redis_jobqueue.sh rollback&amp;lt;/code&amp;gt;, then recreate redis and mediawiki&lt;br /&gt;
|-&lt;br /&gt;
| Code read-only || &amp;lt;code&amp;gt;bash b5_code_readonly.sh rollback&amp;lt;/code&amp;gt;&lt;br /&gt;
|-&lt;br /&gt;
| Elasticsearch flag / SSH || &amp;lt;code&amp;gt;bash b6_es_sshd.sh es-rollback&amp;lt;/code&amp;gt; / &amp;lt;code&amp;gt;ssh-rollback&amp;lt;/code&amp;gt;&lt;br /&gt;
|-&lt;br /&gt;
| Web hardening || &amp;lt;code&amp;gt;bash a2_web_hardening.sh rollback&amp;lt;/code&amp;gt;&lt;br /&gt;
|-&lt;br /&gt;
| Cloudflare logging || &amp;lt;code&amp;gt;bash a3_cloudflare_only.sh rollback&amp;lt;/code&amp;gt;&lt;br /&gt;
|-&lt;br /&gt;
| Accounts || &amp;lt;code&amp;gt;bash a1_accounts.sh rollback&amp;lt;/code&amp;gt;&lt;br /&gt;
|-&lt;br /&gt;
| Longchenpa import || see [[User:Eric/Report 2026-11-22/Longchenpa Project]]&lt;br /&gt;
|}&lt;br /&gt;
The operating-system update can&amp;#039;t be rolled back as a whole. Older kernels stay installed and can be chosen at boot.&lt;br /&gt;
&lt;br /&gt;
== Open follow-ups ==&lt;br /&gt;
These are named here in general terms only. The details are in the private runbook (ask Eric).&lt;br /&gt;
* Rotate the credentials that are stored in the git repositories. This has to go through the repositories (read secrets from the environment, untrack &amp;lt;code&amp;gt;.env&amp;lt;/code&amp;gt;), or a later &amp;lt;code&amp;gt;git reset&amp;lt;/code&amp;gt; would bring the old values back.&lt;br /&gt;
* Phase 2 of the Cloudflare-only access for rywikitexts, after the log from step A3 has been read.&lt;br /&gt;
* Two-factor authentication for administrators; switch automated logins to bot passwords.&lt;br /&gt;
* Review the grants of the &amp;quot;Tsadra MCP&amp;quot; OAuth consumer.&lt;br /&gt;
* Restrict the monitoring and cache ports on the private network.&lt;br /&gt;
* Decide on Marcus&amp;#039;s rights and on the remote script loaded by [[MediaWiki:Common.js]].&lt;br /&gt;
* Make the nightly security update effective again (root crontab), keeping docker and kernel updates for planned windows.&lt;br /&gt;
* Move old evidence directories and old configuration backups off the server, to an encrypted archive.&lt;br /&gt;
* Make the changes above permanent in the repositories (see &amp;quot;Before the next deploy&amp;quot;).&lt;/div&gt;</summary>
		<author><name>Eric</name></author>
	</entry>
</feed>