Changes

From Rangjung Yeshe Wiki Texts
Revision as of 23:02, 21 November 2026 by Eric (talk | contribs) (Report part 1: changes after 01:49 UTC (content only))
Jump to navigation Jump to search

Part of User:Eric/Report 2026-11-22. Every change made on 21–22 November 2026 on the mirage-aws server and on rywikitexts, in the order it was done. Times are UTC. Each step was approved by Eric before it ran and checked afterwards; the step letters (A1, B3…) are the ones used in the runbook and in the scripts.

Where things are on the server:

  • The scripts used: /home/ec2-user/sec-2026-11-21/kit/. Each has a check action, which only reads, plus apply, verify and rollback.
  • Rollback data: /home/ec2-user/sec-2026-11-21/state/ (mode 700).
  • Every file that was edited was first copied next to itself as *.bak-sec-20261121, or *.bak-stopfix-20261121 for the database start scripts.
  • Configuration files that are mounted into a container were edited in place (same inode), so the running container saw the change. Afterwards the host copy and the container copy were compared (md5).

21 November: audit and preparation (nothing changed)

  • Audit of rywikitexts: a read-only audit of the server, security and content. It used read-only commands on the server, outside checks at a low rate without logging in, and the public API.
  • Longchenpa pages: built and tested on Eric's laptop, with render tests through the wiki's own parser that save nothing.
  • Scripts and runbook: written, and every script's read-only check was run against the server.

Step A: no downtime (22 Nov 01:03–01:05)

A1. Dormant administrator accounts

  • RyWikiTextsSysop, RyTxtWikiSysop and RyWikiTextsAdmin were shared administrator accounts, unused for years. Each:
    • lost all its groups (bureaucrat, sysop, smwadministrator);
    • got a random password, which also replaced an old-style password hash on one of them;
    • lost its open sessions;
    • was blocked indefinitely.
  • Rangjung was blocked indefinitely: it uploaded spam PDFs in February 2026, and its one-week block had expired.
  • All of this is in Special:Log under Eric (3 user-rights changes, 4 blocks).
  • Marcus was not changed; that waits for Eric's decision.
  • Undo: a1_accounts.sh rollback gives the groups back and lifts the blocks. The random passwords stay.

A2. Web server hardening (rywikitexts only)

  • config/apache2.conf got a block at the end that answers 403 to:
    • the web installer /mw-config/;
    • the code directories (maintenance, includes, tests, cache, serialized, docs, languages, vendor);
    • top-level developer files (*.yml, *.md, *.lock, *.json, README, RELEASE-NOTES-*…);
    • the LocalSettings copies and CloudflareAuthCookie.php.
This was needed because AllowOverride None means the .htaccess files MediaWiki ships are ignored. Over the previous 7 days, only scanners had asked for these paths.
  • config/php.ini: display_errors = Off and expose_php = Off. The X-Powered-By: PHP/… header is gone.
  • config/LocalSettings-production.php got a block appended at the end:
    • $wgCdnServersNoPurge = [ '172.16.0.0/12' ];: MediaWiki now sees the visitor's real IP address instead of the proxy's, so blocks, rate limits, the login throttle and LoginNotify work per visitor. Checked: through Cloudflare, the API reports the real address.
    • $wgCookieSameSite = 'Lax';
    • sysops no longer have editwidgets or editaccount; editaccount moved to bureaucrats. All current sysops are bureaucrats.
    • uploads of svg, ps, doc, xls, ppt, mpp and flv are switched off. None of these exist on the wiki, and the 2 docx files are not affected. UploadWizard gets the same list.
  • File modes: .env and its old copies are 600, and backup/db is no longer readable by other users.
  • Apache was reloaded gracefully, with no downtime. The script would have restored all three files on its own if any check had failed; all passed.
  • Undo: a2_web_hardening.sh rollback.

A3. Logging of requests that do not come through Cloudflare (shared proxy, rywikitexts only)

  • Two new files in the shared proxy configuration (/mnt/miragedata/docker/proxy/data/):
    • conf.d/00-rywikitexts-cfonly.conf: a geo map and a log format;
    • vhost.d/rywikitexts.tsadra.org_location: it logs such requests to /var/log/nginx/rywikitexts_direct.log inside proxy-web-auto.
  • Nothing is blocked. Blocking is phase 2, after the log has been read (planned after 23 Nov 01:05) and with its own approval.
  • docker-gen-auto was restarted to regenerate the nginx configuration. The generated configuration for the other sites came out the same.
  • The first attempt failed nginx's own test (nginx -t) because the log format was in a file loaded after default.conf. The script removed its files at once, and nginx kept running on its old configuration. The file was renamed with a 00- prefix and the second attempt passed.
  • Undo: a3_cloudflare_only.sh rollback.

New MediaWiki image (22 Nov 01:07–01:09, no downtime)

  • tsadra/mediawiki:1.43-sec-20261121 was built from the repository's own Dockerfile-mediawiki, run from a temporary copy so the repository is unchanged. Three differences:
    1. Docker Hub's mediawiki:1.43 was still 1.43.11 (already with PHP 8.3.37), so core was brought to 1.43.12 with the official patch. Its GPG signature (MediaWiki release key) was checked, it was dry-run first, and the version was checked after.
    2. No copy of LocalSettings is baked into the image any more. The old images carried one, with passwords in it; the live file is mounted anyway.
    3. The web root /var/www/html is 755 root instead of 1777.
  • Smoke tests on throwaway containers all passed: PHP 8.3.37, MW 1.43.12, all 10 PHP extensions, Apache configuration test with the live apache2.conf.
  • No existing image tag was moved.

Step B: maintenance window (22 Nov 01:11–01:27)

B1. Database clean-stop fix: rywikitexts, bca, tsadra, dharmacloud (01:11–01:12)

Why. In all Tsadra MariaDB containers, process 1 is a /bin/sh wrapper that ignores the stop signal. So every docker stop, docker compose down/restart or server shutdown waited 10 s and then killed MariaDB. On the next start InnoDB ran crash recovery: 4 times on rywikitexts since 1 October. A reboot without this fix would have done the same to all four databases. The fix was tested on ryzen on 15 October.

What changed, per project (~/docker/rywikitexts, bca, tsadra, dharmacloud):

  • The start script's last lines now exec the official MariaDB entrypoint, so mariadbd becomes process 1 and shuts down cleanly. The files:
    • init/start-script-db.sh for rywikitexts and bca (wiki35 repository);
    • backup/entrypoint/start-script-db.sh for tsadra and dharmacloud (wordpress repository).
  • A new docker-compose.override.yml ("Local to this server, not in git") runs the script in exec form, mounts the fixed script read-only (wiki35 layout), and sets stop_grace_period: 1m.
  • Each database container was recreated after a clean manual stop, then tested with a stop and start. Each time MariaDB logged "Shutdown complete" and started with no crash recovery.
    • Downtime: about 8 s per database, plus 2 s for the test.
    • The image stayed the same, except rywikitexts: it moved from the old build to the current tsadra/mariadb:10.11 tag, which has the same MariaDB 10.11.19 and the same script, and which bca has used since October.
  • conference's database was already stopped and was not touched.
  • Undo: b1_stopfix.sh rollback <dir>, then docker compose up -d <db service>. That brings back the hard kill on every stop.

B4. Redis and the job queue (rywikitexts, 01:12)

  • Redis now needs a password (REDIS_PASSWORD in .env; requirepass in config/redis.conf, mode 640 for the redis group).
  • Its memory went from 1 GB to 2 GB. It had been full, and had evicted 856,000 keys in 35 days.
  • It saves a snapshot every 15 minutes instead of every minute (--save 900 1 in a redis block of the override file).
  • LocalSettings reads the password from the environment.
  • The MediaWiki job queue moved from Redis to the database (JobQueueDB), as on the rangjung wikis since October. Without the jobchron service, Redis never released delayed jobs or retried failed ones. Redis is still the cache. The Redis queue was empty when it was switched.

B3. New MediaWiki deployed (rywikitexts, 01:12–01:13)

  • 20 bundled extensions and the Vector skin (git submodules under data/) moved to the head of their REL1_43 branch. These are the ones the 1.43.10–1.43.12 security releases patched: AbuseFilter, CategoryTree, Cite, CiteThisPage, ConfirmEdit, Echo, Gadgets, InputBox, LoginNotify, MultimediaViewer, Nuke, OATHAuth, PdfHandler, ReplaceText, Scribunto, SpamBlacklist, TemplateData, Thanks, TitleBlacklist, VisualEditor and Vector. SyntaxHighlight and WikiEditor were already current. The previous commits are recorded for rollback.
  • New pin file backup/pin-sec-20261121.yml. Redis and MediaWiki were recreated.
  • rywikitexts was down for 29 s. update.php ran at start.
  • Now: MediaWiki 1.43.12, PHP 8.3.37. Pages, Special:Version, search, SMW queries and the API answer; there are no PHP fatal errors; jobs run from the database.
  • Undo: b3_mw_image.sh rollback puts back the previous pin (backup/pin-oauth-20261116-160811.yml) and the recorded submodule commits.

B5. Extension and skin code read-only for the web server (01:13)

  • data/extensions and data/skins now belong to ec2-user:ec2-user without group write. The web server (www-data) can still read everything. Before this, it could write 11,006 PHP files, so any file-write bug could have become a permanent backdoor.
  • Two kinds of directory stay writable, with the sticky bit so files can't be swapped: the Tibetan converters' Lingua-BO-Wylie/bin (they write temporary files there) and Widgets/compiled_templates.
  • Checked: the converter (<wytotib>) still works, and every file and font is still readable (569 font files tested).
  • Side effect: for about one second between the two permission commands, the converter could not write its temporary file (17 warnings). One page view in that second, an old revision of User:Eric, was affected; it was purged. There have been no errors since.
  • Undo: b5_code_readonly.sh rollback, from a full snapshot of modes and owners.

B6. Elasticsearch and SSH (01:15–01:20)

  • Elasticsearch 7.10.2 (rywikitexts) bundles an old log4j. -Dlog4j2.formatMsgNoLookups=true was added to config/elastic-search/jvm.options.
    • Incident: the original file has no newline at the end, so the first attempt glued the new line onto the last option. Elasticsearch could not start, and search was down from about 01:15 to 01:19:30. The file was restored and the line added correctly: the cluster is green, the option is active, and the index counts are unchanged (697 search hits for "Longchenpa").
  • SSH: new file /etc/ssh/sshd_config.d/00-tsadra-hardening.conf with PermitRootLogin no and X11Forwarding no. root had no SSH keys, so nobody lost access. A new ec2-user session and sudo were tested.

B2. Operating-system update and reboot (01:19–01:26)

  • Amazon Linux was locked to release 2023.12.20260706 (July), so the nightly dnf update --security installed nothing: 142 security notices were pending, 3 of them Critical.
  • dnf upgrade --releasever=latest moved it to release 2023.12.20261120:
    • kernel 6.18.62 (was 6.18.36);
    • docker 25.0.18 and containerd 2.3.7;
    • tailscale 1.104.7;
    • plus the rest of the pending updates.
  • Upgrading the docker package restarted every container on the server (about 01:20). Thanks to B1 all databases stopped cleanly.
    • During that restart, rywikitexts' update.php lost its database connection for a moment, while MariaDB was restarting too. After the reboot it ran cleanly.
  • Reboot at 01:22:34; the server was back at 01:23:07.
  • After the reboot:
    • all 20 containers were running again;
    • all 4 databases started cleanly;
    • all 7 sites answered;
    • docker and tailscale were up.
  • rywikitexts answered from 01:24:12. bodhicitta (bca) answered once its own update.php had finished, 40 s later.

Longchenpa Project import (22 Nov 01:32–01:42)

See User:Eric/Report 2026-11-22/Longchenpa Project. A fresh database dump was taken first and is kept as the rollback point. 723 new pages were imported, and the 10-byte page Longchenpa got its content. All 724 pages were checked against the package and are identical.

Style fixes in MediaWiki:Vector.css (22 Nov 01:38 and 01:40)

These were reported by Eric. Neither was caused by the upgrade: the upgrade changed no style file in core, Vector or any updated extension. Both come from the TsadraWikiStyles extension (scripts/tws.js), whose code is written for the Chameleon skin:

  • In Vector, "Recent changes" and "Random page" in the Tools box came out larger (16px) than the other items (12px), because tws.js inserts them as <div><a class="nav-link"> instead of list items. A CSS rule now gives them the same size and spacing.
  • tws.js also adds Font Awesome bell and inbox icons next to Echo's own notification icons, so Vector showed two rows of icons. The extra ones are now hidden in Vector.
  • Cloudflare keeps the wiki's style files for up to 24 hours (s-maxage=86400), so visitors may see these fixes only after that.

Cloudflare purge of the style file (22 Nov 02:05)

Eric still saw the old sidebar in Chrome. Cloudflare was serving a copy of the wiki's style file from before the fix (8 hours old). That one address (load.php?…modules=site.styles…) was purged through the Cloudflare API, with the zone and token that MediaWiki already uses for its own purges; the token was read on the server and never shown. Cloudflare then served the fixed file. A browser that had kept the old copy needs one reload.

Tibetan font of the Longchenpa pages (22 Nov 02:18–02:22)

  • Eric found the Tibetan of the new pages less clear than on the older pages (for example Chos dbyings rin po che'i mdzod/Introduction/Commentary). The new pages used classes that name a font installed on few computers, so most browsers fell back to a system font. They now use the wiki's own web font TibetanMachineUni, like the rest of the wiki.
  • In the Pe cin e-texts the marks « » enclose the small-print notes of the print. These notes are now shown in small print and the marks are hidden (two classes, lcp-note and lcp-mark, added to MediaWiki:Common.css at 02:18).
  • 516 Longchenpa pages and their templates got a new revision (02:19–02:22), after a database dump (~/lcp/backup/db_before_longchenpa_v2_…). All 724 pages were checked against the package.

Theg mchog mdzod e-text converted to Unicode (22 Nov 02:56–03:08)

  • The pages Klong chen rab byams gsung 'bum/volume 017 441 and Klong chen rab byams gsung 'bum/volume 018 450 (the Pe cin text of the Theg mchog mdzod, input in 2020) were stored in an old, pre-Unicode font encoding: the Tibetan letters were typed as Latin symbols, so the pages showed gibberish.
  • They were converted to Unicode (two edits by Eric at 02:56). The table of symbols was worked out by aligning the text with a corrected Unicode text of the same work (BDRC W22920): after conversion 99.2 % of the syllables are identical to it. Codes left by drawings in the Word file became invisible comments.
  • The 165 Longchenpa pages built from them were updated (02:59–03:08), after a database dump.
  • To undo: restore the previous revision of the two pages.

A damaged ngöndro page repaired (22 Nov 03:01)

Theg mchog mdzod study page (22 Nov 03:11–03:29)

  • New: Theg mchog mdzod (Longchenpa) and 53 subpages (one per chapter, long chapters in parts of at most 40 pages, plus a glossary), after a database dump.
  • Each part has one tab per edition: the Adzom Chögar edition first (the standard edition), then the BDRC W22920 e-text, the Pe cin and Karma Lekshey Ling e-texts, and six other editions as OCR of their BDRC scans. Then a critical edition (the corrected base text with the variants of the other editions under each page), the English translation (a machine translation, labelled as such, not yet reviewed by a translator) and a bilingual version.
  • It is linked from Seven Treasuries (mdzod bdun) and Longchenpa Collected Works.

Adzom edition and Tibetan sizes (22 Nov 03:29–04:01)

  • Eric asked for the Adzom Chögar edition, the standard edition of Longchenpa's writings, to have its own tab wherever possible.
    • New: མཛོད་བདུན། ༼ཨ་འཛོམ་པར་མ།༽, 7 volume pages and 19 text pages (OCR of the BDRC scans MW1PD8, not proofread). The texts were found in the OCR by aligning it with the Karma Lekshey Ling e-texts.
    • The Pe cin, sde dge and Karma Lekshey Ling pages of the same works (44 pages) got an Adzom edition tab. It shows the Adzom text itself where it is short enough (24 pages), otherwise a link to the Adzom page (20 pages).
    • Seven Treasuries (mdzod bdun) has an Adzom column, first.
  • Eric found the Tibetan sometimes too big beside the English, and out of line. The Tibetan beside English is now sized relative to the English: 1.3 times in a line of English, 1.45 times for titles, 1.85 times for the title at the top of a page. Before, it had fixed sizes of 16–28 points (21–37 px) next to 14 px English.
    • The classes are in MediaWiki:Common.css (edit at 03:29, then a Cloudflare purge of the style file). The page texts keep the size Eric chose.
    • This wiki shows a page's display title in every link to it, so the old 24-point display titles made links such as Same work in very large. The display titles now use a class that is 1.3 times the English in links and normal size in the page heading (second edit of MediaWiki:Common.css at 03:34). The stored display titles of all project pages were refreshed (03:34–04:01) and the pages purged.
    • Empty rows of the info tables no longer leave a blank line.
    • In the study page, the variants now sit under each page in English-size type.
  • 85 Longchenpa pages (28 new, 57 changed) and the 54 study pages got a new revision, after a database dump (~/lcp/backup/db_before_longchenpa_v4_…).
  • To undo: restore the previous revisions, or the dump. The new pages can be deleted.