Changes
Part of User:Eric/Report 2026-11-22. Every change made on 21–22 November 2026 on the mirage-aws server and on rywikitexts, in the order it was done. Times are UTC. Each step was approved by Eric before it ran and checked afterwards; the step letters (A1, B3…) are the ones used in the runbook and in the scripts.
Where things are on the server:
- The scripts used:
/home/ec2-user/sec-2026-11-21/kit/. Each has acheckaction, which only reads, plusapply,verifyandrollback. - Rollback data:
/home/ec2-user/sec-2026-11-21/state/(mode 700). - Every file that was edited was first copied next to itself as
*.bak-sec-20261121, or*.bak-stopfix-20261121for the database start scripts. - Configuration files that are mounted into a container were edited in place (same inode), so the running container saw the change. Afterwards the host copy and the container copy were compared (md5).
21 November: audit and preparation (nothing changed)
- Audit of rywikitexts: a read-only audit of the server, security and content. It used read-only commands on the server, outside checks at a low rate without logging in, and the public API.
- Longchenpa pages: built and tested on Eric's laptop, with render tests through the wiki's own parser that save nothing.
- Scripts and runbook: written, and every script's read-only
checkwas run against the server.
Step A: no downtime (22 Nov 01:03–01:05)
A1. Dormant administrator accounts
RyWikiTextsSysop,RyTxtWikiSysopandRyWikiTextsAdminwere shared administrator accounts, unused for years. Each:- lost all its groups (bureaucrat, sysop, smwadministrator);
- got a random password, which also replaced an old-style password hash on one of them;
- lost its open sessions;
- was blocked indefinitely.
Rangjungwas blocked indefinitely: it uploaded spam PDFs in February 2026, and its one-week block had expired.- All of this is in Special:Log under Eric (3 user-rights changes, 4 blocks).
Marcuswas not changed; that waits for Eric's decision.- Undo:
a1_accounts.sh rollbackgives the groups back and lifts the blocks. The random passwords stay.
A2. Web server hardening (rywikitexts only)
config/apache2.confgot a block at the end that answers 403 to:- the web installer
/mw-config/; - the code directories (
maintenance,includes,tests,cache,serialized,docs,languages,vendor); - top-level developer files (
*.yml,*.md,*.lock,*.json,README,RELEASE-NOTES-*…); - the LocalSettings copies and
CloudflareAuthCookie.php.
- the web installer
- This was needed because
AllowOverride Nonemeans the.htaccessfiles MediaWiki ships are ignored. Over the previous 7 days, only scanners had asked for these paths.
config/php.ini:display_errors = Offandexpose_php = Off. TheX-Powered-By: PHP/…header is gone.config/LocalSettings-production.phpgot a block appended at the end:$wgCdnServersNoPurge = [ '172.16.0.0/12' ];: MediaWiki now sees the visitor's real IP address instead of the proxy's, so blocks, rate limits, the login throttle and LoginNotify work per visitor. Checked: through Cloudflare, the API reports the real address.$wgCookieSameSite = 'Lax';- sysops no longer have
editwidgetsoreditaccount;editaccountmoved to bureaucrats. All current sysops are bureaucrats. - uploads of svg, ps, doc, xls, ppt, mpp and flv are switched off. None of these exist on the wiki, and the 2 docx files are not affected. UploadWizard gets the same list.
- File modes:
.envand its old copies are 600, andbackup/dbis no longer readable by other users. - Apache was reloaded gracefully, with no downtime. The script would have restored all three files on its own if any check had failed; all passed.
- Undo:
a2_web_hardening.sh rollback.
- Two new files in the shared proxy configuration (
/mnt/miragedata/docker/proxy/data/):conf.d/00-rywikitexts-cfonly.conf: ageomap and a log format;vhost.d/rywikitexts.tsadra.org_location: it logs such requests to/var/log/nginx/rywikitexts_direct.loginsideproxy-web-auto.
- Nothing is blocked. Blocking is phase 2, after the log has been read (planned after 23 Nov 01:05) and with its own approval.
docker-gen-autowas restarted to regenerate the nginx configuration. The generated configuration for the other sites came out the same.- The first attempt failed nginx's own test (
nginx -t) because the log format was in a file loaded afterdefault.conf. The script removed its files at once, and nginx kept running on its old configuration. The file was renamed with a00-prefix and the second attempt passed. - Undo:
a3_cloudflare_only.sh rollback.
New MediaWiki image (22 Nov 01:07–01:09, no downtime)
tsadra/mediawiki:1.43-sec-20261121was built from the repository's ownDockerfile-mediawiki, run from a temporary copy so the repository is unchanged. Three differences:- Docker Hub's
mediawiki:1.43was still 1.43.11 (already with PHP 8.3.37), so core was brought to 1.43.12 with the official patch. Its GPG signature (MediaWiki release key) was checked, it was dry-run first, and the version was checked after. - No copy of LocalSettings is baked into the image any more. The old images carried one, with passwords in it; the live file is mounted anyway.
- The web root
/var/www/htmlis 755 root instead of 1777.
- Docker Hub's
- Smoke tests on throwaway containers all passed: PHP 8.3.37, MW 1.43.12, all 10 PHP extensions, Apache configuration test with the live
apache2.conf. - No existing image tag was moved.
Step B: maintenance window (22 Nov 01:11–01:27)
B1. Database clean-stop fix: rywikitexts, bca, tsadra, dharmacloud (01:11–01:12)
Why. In all Tsadra MariaDB containers, process 1 is a /bin/sh wrapper that ignores the stop signal. So every docker stop, docker compose down/restart or server shutdown waited 10 s and then killed MariaDB. On the next start InnoDB ran crash recovery: 4 times on rywikitexts since 1 October. A reboot without this fix would have done the same to all four databases. The fix was tested on ryzen on 15 October.
What changed, per project (~/docker/rywikitexts, bca, tsadra, dharmacloud):
- The start script's last lines now
execthe official MariaDB entrypoint, somariadbdbecomes process 1 and shuts down cleanly. The files:init/start-script-db.shfor rywikitexts and bca (wiki35 repository);backup/entrypoint/start-script-db.shfor tsadra and dharmacloud (wordpress repository).
- A new
docker-compose.override.yml("Local to this server, not in git") runs the script in exec form, mounts the fixed script read-only (wiki35 layout), and setsstop_grace_period: 1m. - Each database container was recreated after a clean manual stop, then tested with a stop and start. Each time MariaDB logged "Shutdown complete" and started with no crash recovery.
- Downtime: about 8 s per database, plus 2 s for the test.
- The image stayed the same, except rywikitexts: it moved from the old build to the current
tsadra/mariadb:10.11tag, which has the same MariaDB 10.11.19 and the same script, and which bca has used since October.
- conference's database was already stopped and was not touched.
- Undo:
b1_stopfix.sh rollback <dir>, thendocker compose up -d <db service>. That brings back the hard kill on every stop.
B4. Redis and the job queue (rywikitexts, 01:12)
- Redis now needs a password (
REDIS_PASSWORDin.env;requirepassinconfig/redis.conf, mode 640 for the redis group). - Its memory went from 1 GB to 2 GB. It had been full, and had evicted 856,000 keys in 35 days.
- It saves a snapshot every 15 minutes instead of every minute (
--save 900 1in aredisblock of the override file). - LocalSettings reads the password from the environment.
- The MediaWiki job queue moved from Redis to the database (
JobQueueDB), as on the rangjung wikis since October. Without the jobchron service, Redis never released delayed jobs or retried failed ones. Redis is still the cache. The Redis queue was empty when it was switched.
B3. New MediaWiki deployed (rywikitexts, 01:12–01:13)
- 20 bundled extensions and the Vector skin (git submodules under
data/) moved to the head of theirREL1_43branch. These are the ones the 1.43.10–1.43.12 security releases patched: AbuseFilter, CategoryTree, Cite, CiteThisPage, ConfirmEdit, Echo, Gadgets, InputBox, LoginNotify, MultimediaViewer, Nuke, OATHAuth, PdfHandler, ReplaceText, Scribunto, SpamBlacklist, TemplateData, Thanks, TitleBlacklist, VisualEditor and Vector. SyntaxHighlight and WikiEditor were already current. The previous commits are recorded for rollback. - New pin file
backup/pin-sec-20261121.yml. Redis and MediaWiki were recreated. - rywikitexts was down for 29 s. update.php ran at start.
- Now: MediaWiki 1.43.12, PHP 8.3.37. Pages, Special:Version, search, SMW queries and the API answer; there are no PHP fatal errors; jobs run from the database.
- Undo:
b3_mw_image.sh rollbackputs back the previous pin (backup/pin-oauth-20261116-160811.yml) and the recorded submodule commits.
B5. Extension and skin code read-only for the web server (01:13)
data/extensionsanddata/skinsnow belong toec2-user:ec2-userwithout group write. The web server (www-data) can still read everything. Before this, it could write 11,006 PHP files, so any file-write bug could have become a permanent backdoor.- Two kinds of directory stay writable, with the sticky bit so files can't be swapped: the Tibetan converters'
Lingua-BO-Wylie/bin(they write temporary files there) andWidgets/compiled_templates. - Checked: the converter (
<wytotib>) still works, and every file and font is still readable (569 font files tested). - Side effect: for about one second between the two permission commands, the converter could not write its temporary file (17 warnings). One page view in that second, an old revision of User:Eric, was affected; it was purged. There have been no errors since.
- Undo:
b5_code_readonly.sh rollback, from a full snapshot of modes and owners.
B6. Elasticsearch and SSH (01:15–01:20)
- Elasticsearch 7.10.2 (rywikitexts) bundles an old log4j.
-Dlog4j2.formatMsgNoLookups=truewas added toconfig/elastic-search/jvm.options.- Incident: the original file has no newline at the end, so the first attempt glued the new line onto the last option. Elasticsearch could not start, and search was down from about 01:15 to 01:19:30. The file was restored and the line added correctly: the cluster is green, the option is active, and the index counts are unchanged (697 search hits for "Longchenpa").
- SSH: new file
/etc/ssh/sshd_config.d/00-tsadra-hardening.confwithPermitRootLogin noandX11Forwarding no. root had no SSH keys, so nobody lost access. A new ec2-user session and sudo were tested.
B2. Operating-system update and reboot (01:19–01:26)
- Amazon Linux was locked to release 2023.12.20260706 (July), so the nightly
dnf update --securityinstalled nothing: 142 security notices were pending, 3 of them Critical. dnf upgrade --releasever=latestmoved it to release 2023.12.20261120:- kernel 6.18.62 (was 6.18.36);
- docker 25.0.18 and containerd 2.3.7;
- tailscale 1.104.7;
- plus the rest of the pending updates.
- Upgrading the docker package restarted every container on the server (about 01:20). Thanks to B1 all databases stopped cleanly.
- During that restart, rywikitexts' update.php lost its database connection for a moment, while MariaDB was restarting too. After the reboot it ran cleanly.
- Reboot at 01:22:34; the server was back at 01:23:07.
- After the reboot:
- all 20 containers were running again;
- all 4 databases started cleanly;
- all 7 sites answered;
- docker and tailscale were up.
- rywikitexts answered from 01:24:12. bodhicitta (bca) answered once its own update.php had finished, 40 s later.
Longchenpa Project import (22 Nov 01:32–01:42)
See User:Eric/Report 2026-11-22/Longchenpa Project. A fresh database dump was taken first and is kept as the rollback point. 723 new pages were imported, and the 10-byte page Longchenpa got its content. All 724 pages were checked against the package and are identical.
Style fixes in MediaWiki:Vector.css (22 Nov 01:38 and 01:40)
These were reported by Eric. Neither was caused by the upgrade: the upgrade changed no style file in core, Vector or any updated extension. Both come from the TsadraWikiStyles extension (scripts/tws.js), whose code is written for the Chameleon skin:
- In Vector, "Recent changes" and "Random page" in the Tools box came out larger (16px) than the other items (12px), because tws.js inserts them as
<div><a class="nav-link">instead of list items. A CSS rule now gives them the same size and spacing. - tws.js also adds Font Awesome bell and inbox icons next to Echo's own notification icons, so Vector showed two rows of icons. The extra ones are now hidden in Vector.
- Cloudflare keeps the wiki's style files for up to 24 hours (
s-maxage=86400), so visitors may see these fixes only after that.