Changes
Part of User:Eric/Report 2026-11-22. Every change made on 21–22 November 2026 on the mirage-aws server and on rywikitexts, in the order it was done. Times are UTC. Each step was approved by Eric before it ran and checked afterwards; the step letters (A1, B3…) are the ones used in the runbook and in the scripts.
Where things are on the server:
- The scripts used:
/home/ec2-user/sec-2026-11-21/kit/. Each has acheckaction, which only reads, plusapply,verifyandrollback. - Rollback data:
/home/ec2-user/sec-2026-11-21/state/(mode 700). - Every file that was edited was first copied next to itself as
*.bak-sec-20261121, or*.bak-stopfix-20261121for the database start scripts. - Configuration files that are mounted into a container were edited in place (same inode), so the running container saw the change. Afterwards the host copy and the container copy were compared (md5).
21 November: audit and preparation (nothing changed)
- Audit of rywikitexts: a read-only audit of the server, security and content. It used read-only commands on the server, outside checks at a low rate without logging in, and the public API.
- Longchenpa pages: built and tested on Eric's laptop, with render tests through the wiki's own parser that save nothing.
- Scripts and runbook: written, and every script's read-only
checkwas run against the server.
Step A: no downtime (22 Nov 01:03–01:05)
A1. Dormant administrator accounts
RyWikiTextsSysop,RyTxtWikiSysopandRyWikiTextsAdminwere shared administrator accounts, unused for years. Each:- lost all its groups (bureaucrat, sysop, smwadministrator);
- got a random password, which also replaced an old-style password hash on one of them;
- lost its open sessions;
- was blocked indefinitely.
Rangjungwas blocked indefinitely: it uploaded spam PDFs in February 2026, and its one-week block had expired.- All of this is in Special:Log under Eric (3 user-rights changes, 4 blocks).
Marcuswas not changed; that waits for Eric's decision.- Undo:
a1_accounts.sh rollbackgives the groups back and lifts the blocks. The random passwords stay.
A2. Web server hardening (rywikitexts only)
config/apache2.confgot a block at the end that answers 403 to:- the web installer
/mw-config/; - the code directories (
maintenance,includes,tests,cache,serialized,docs,languages,vendor); - top-level developer files (
*.yml,*.md,*.lock,*.json,README,RELEASE-NOTES-*…); - the LocalSettings copies and
CloudflareAuthCookie.php.
- the web installer
- This was needed because
AllowOverride Nonemeans the.htaccessfiles MediaWiki ships are ignored. Over the previous 7 days, only scanners had asked for these paths.
config/php.ini:display_errors = Offandexpose_php = Off. TheX-Powered-By: PHP/…header is gone.config/LocalSettings-production.phpgot a block appended at the end:$wgCdnServersNoPurge = [ '172.16.0.0/12' ];: MediaWiki now sees the visitor's real IP address instead of the proxy's, so blocks, rate limits, the login throttle and LoginNotify work per visitor. Checked: through Cloudflare, the API reports the real address.$wgCookieSameSite = 'Lax';- sysops no longer have
editwidgetsoreditaccount;editaccountmoved to bureaucrats. All current sysops are bureaucrats. - uploads of svg, ps, doc, xls, ppt, mpp and flv are switched off. None of these exist on the wiki, and the 2 docx files are not affected. UploadWizard gets the same list.
- File modes:
.envand its old copies are 600, andbackup/dbis no longer readable by other users. - Apache was reloaded gracefully, with no downtime. The script would have restored all three files on its own if any check had failed; all passed.
- Undo:
a2_web_hardening.sh rollback.
- Two new files in the shared proxy configuration (
/mnt/miragedata/docker/proxy/data/):conf.d/00-rywikitexts-cfonly.conf: ageomap and a log format;vhost.d/rywikitexts.tsadra.org_location: it logs such requests to/var/log/nginx/rywikitexts_direct.loginsideproxy-web-auto.
- Nothing is blocked. Blocking is phase 2, after the log has been read (planned after 23 Nov 01:05) and with its own approval.
docker-gen-autowas restarted to regenerate the nginx configuration. The generated configuration for the other sites came out the same.- The first attempt failed nginx's own test (
nginx -t) because the log format was in a file loaded afterdefault.conf. The script removed its files at once, and nginx kept running on its old configuration. The file was renamed with a00-prefix and the second attempt passed. - Undo:
a3_cloudflare_only.sh rollback.
New MediaWiki image (22 Nov 01:07–01:09, no downtime)
tsadra/mediawiki:1.43-sec-20261121was built from the repository's ownDockerfile-mediawiki, run from a temporary copy so the repository is unchanged. Three differences:- Docker Hub's
mediawiki:1.43was still 1.43.11 (already with PHP 8.3.37), so core was brought to 1.43.12 with the official patch. Its GPG signature (MediaWiki release key) was checked, it was dry-run first, and the version was checked after. - No copy of LocalSettings is baked into the image any more. The old images carried one, with passwords in it; the live file is mounted anyway.
- The web root
/var/www/htmlis 755 root instead of 1777.
- Docker Hub's
- Smoke tests on throwaway containers all passed: PHP 8.3.37, MW 1.43.12, all 10 PHP extensions, Apache configuration test with the live
apache2.conf. - No existing image tag was moved.
Step B: maintenance window (22 Nov 01:11–01:27)
B1. Database clean-stop fix: rywikitexts, bca, tsadra, dharmacloud (01:11–01:12)
Why. In all Tsadra MariaDB containers, process 1 is a /bin/sh wrapper that ignores the stop signal. So every docker stop, docker compose down/restart or server shutdown waited 10 s and then killed MariaDB. On the next start InnoDB ran crash recovery: 4 times on rywikitexts since 1 October. A reboot without this fix would have done the same to all four databases. The fix was tested on ryzen on 15 October.
What changed, per project (~/docker/rywikitexts, bca, tsadra, dharmacloud):
- The start script's last lines now
execthe official MariaDB entrypoint, somariadbdbecomes process 1 and shuts down cleanly. The files:init/start-script-db.shfor rywikitexts and bca (wiki35 repository);backup/entrypoint/start-script-db.shfor tsadra and dharmacloud (wordpress repository).
- A new
docker-compose.override.yml("Local to this server, not in git") runs the script in exec form, mounts the fixed script read-only (wiki35 layout), and setsstop_grace_period: 1m. - Each database container was recreated after a clean manual stop, then tested with a stop and start. Each time MariaDB logged "Shutdown complete" and started with no crash recovery.
- Downtime: about 8 s per database, plus 2 s for the test.
- The image stayed the same, except rywikitexts: it moved from the old build to the current
tsadra/mariadb:10.11tag, which has the same MariaDB 10.11.19 and the same script, and which bca has used since October.
- conference's database was already stopped and was not touched.
- Undo:
b1_stopfix.sh rollback <dir>, thendocker compose up -d <db service>. That brings back the hard kill on every stop.
B4. Redis and the job queue (rywikitexts, 01:12)
- Redis now needs a password (
REDIS_PASSWORDin.env;requirepassinconfig/redis.conf, mode 640 for the redis group). - Its memory went from 1 GB to 2 GB. It had been full, and had evicted 856,000 keys in 35 days.
- It saves a snapshot every 15 minutes instead of every minute (
--save 900 1in aredisblock of the override file). - LocalSettings reads the password from the environment.
- The MediaWiki job queue moved from Redis to the database (
JobQueueDB), as on the rangjung wikis since October. Without the jobchron service, Redis never released delayed jobs or retried failed ones. Redis is still the cache. The Redis queue was empty when it was switched.
B3. New MediaWiki deployed (rywikitexts, 01:12–01:13)
- 20 bundled extensions and the Vector skin (git submodules under
data/) moved to the head of theirREL1_43branch. These are the ones the 1.43.10–1.43.12 security releases patched: AbuseFilter, CategoryTree, Cite, CiteThisPage, ConfirmEdit, Echo, Gadgets, InputBox, LoginNotify, MultimediaViewer, Nuke, OATHAuth, PdfHandler, ReplaceText, Scribunto, SpamBlacklist, TemplateData, Thanks, TitleBlacklist, VisualEditor and Vector. SyntaxHighlight and WikiEditor were already current. The previous commits are recorded for rollback. - New pin file
backup/pin-sec-20261121.yml. Redis and MediaWiki were recreated. - rywikitexts was down for 29 s. update.php ran at start.
- Now: MediaWiki 1.43.12, PHP 8.3.37. Pages, Special:Version, search, SMW queries and the API answer; there are no PHP fatal errors; jobs run from the database.
- Undo:
b3_mw_image.sh rollbackputs back the previous pin (backup/pin-oauth-20261116-160811.yml) and the recorded submodule commits.
B5. Extension and skin code read-only for the web server (01:13)
data/extensionsanddata/skinsnow belong toec2-user:ec2-userwithout group write. The web server (www-data) can still read everything. Before this, it could write 11,006 PHP files, so any file-write bug could have become a permanent backdoor.- Two kinds of directory stay writable, with the sticky bit so files can't be swapped: the Tibetan converters'
Lingua-BO-Wylie/bin(they write temporary files there) andWidgets/compiled_templates. - Checked: the converter (
<wytotib>) still works, and every file and font is still readable (569 font files tested). - Side effect: for about one second between the two permission commands, the converter could not write its temporary file (17 warnings). One page view in that second, an old revision of User:Eric, was affected; it was purged. There have been no errors since.
- Undo:
b5_code_readonly.sh rollback, from a full snapshot of modes and owners.
B6. Elasticsearch and SSH (01:15–01:20)
- Elasticsearch 7.10.2 (rywikitexts) bundles an old log4j.
-Dlog4j2.formatMsgNoLookups=truewas added toconfig/elastic-search/jvm.options.- Incident: the original file has no newline at the end, so the first attempt glued the new line onto the last option. Elasticsearch could not start, and search was down from about 01:15 to 01:19:30. The file was restored and the line added correctly: the cluster is green, the option is active, and the index counts are unchanged (697 search hits for "Longchenpa").
- SSH: new file
/etc/ssh/sshd_config.d/00-tsadra-hardening.confwithPermitRootLogin noandX11Forwarding no. root had no SSH keys, so nobody lost access. A new ec2-user session and sudo were tested.
B2. Operating-system update and reboot (01:19–01:26)
- Amazon Linux was locked to release 2023.12.20260706 (July), so the nightly
dnf update --securityinstalled nothing: 142 security notices were pending, 3 of them Critical. dnf upgrade --releasever=latestmoved it to release 2023.12.20261120:- kernel 6.18.62 (was 6.18.36);
- docker 25.0.18 and containerd 2.3.7;
- tailscale 1.104.7;
- plus the rest of the pending updates.
- Upgrading the docker package restarted every container on the server (about 01:20). Thanks to B1 all databases stopped cleanly.
- During that restart, rywikitexts' update.php lost its database connection for a moment, while MariaDB was restarting too. After the reboot it ran cleanly.
- Reboot at 01:22:34; the server was back at 01:23:07.
- After the reboot:
- all 20 containers were running again;
- all 4 databases started cleanly;
- all 7 sites answered;
- docker and tailscale were up.
- rywikitexts answered from 01:24:12. bodhicitta (bca) answered once its own update.php had finished, 40 s later.
Longchenpa Project import (22 Nov 01:32–01:42)
See User:Eric/Report 2026-11-22/Longchenpa Project. A fresh database dump was taken first and is kept as the rollback point. 723 new pages were imported, and the 10-byte page Longchenpa got its content. All 724 pages were checked against the package and are identical.
Style fixes in MediaWiki:Vector.css (22 Nov 01:38 and 01:40)
These were reported by Eric. Neither was caused by the upgrade: the upgrade changed no style file in core, Vector or any updated extension. Both come from the TsadraWikiStyles extension (scripts/tws.js), whose code is written for the Chameleon skin:
- In Vector, "Recent changes" and "Random page" in the Tools box came out larger (16px) than the other items (12px), because tws.js inserts them as
<div><a class="nav-link">instead of list items. A CSS rule now gives them the same size and spacing. - tws.js also adds Font Awesome bell and inbox icons next to Echo's own notification icons, so Vector showed two rows of icons. The extra ones are now hidden in Vector.
- Cloudflare keeps the wiki's style files for up to 24 hours (
s-maxage=86400), so visitors may see these fixes only after that.
Cloudflare purge of the style file (22 Nov 02:05)
Eric still saw the old sidebar in Chrome. Cloudflare was serving a copy of the wiki's style file from before the fix (8 hours old). That one address (load.php?…modules=site.styles…) was purged through the Cloudflare API, with the zone and token that MediaWiki already uses for its own purges; the token was read on the server and never shown. Cloudflare then served the fixed file. A browser that had kept the old copy needs one reload.
Tibetan font of the Longchenpa pages (22 Nov 02:18–02:22)
- Eric found the Tibetan of the new pages less clear than on the older pages (for example Chos dbyings rin po che'i mdzod/Introduction/Commentary). The new pages used classes that name a font installed on few computers, so most browsers fell back to a system font. They now use the wiki's own web font TibetanMachineUni, like the rest of the wiki.
- In the Pe cin e-texts the marks « » enclose the small-print notes of the print. These notes are now shown in small print and the marks are hidden (two classes,
lcp-noteandlcp-mark, added to MediaWiki:Common.css at 02:18). - 516 Longchenpa pages and their templates got a new revision (02:19–02:22), after a database dump (
~/lcp/backup/db_before_longchenpa_v2_…). All 724 pages were checked against the package.
Theg mchog mdzod e-text converted to Unicode (22 Nov 02:56–03:08)
- The pages Klong chen rab byams gsung 'bum/volume 017 441 and Klong chen rab byams gsung 'bum/volume 018 450 (the Pe cin text of the Theg mchog mdzod, input in 2020) were stored in an old, pre-Unicode font encoding: the Tibetan letters were typed as Latin symbols, so the pages showed gibberish.
- They were converted to Unicode (two edits by Eric at 02:56). The table of symbols was worked out by aligning the text with a corrected Unicode text of the same work (BDRC W22920): after conversion 99.2 % of the syllables are identical to it. Codes left by drawings in the Word file became invisible comments.
- The 165 Longchenpa pages built from them were updated (02:59–03:08), after a database dump.
- To undo: restore the previous revision of the two pages.
A damaged ngöndro page repaired (22 Nov 03:01)
- Looking for other pages with the same problem, the scan found one page damaged in another way: རྡོ་རྗེ་ཐེག་པའི་ཐུན་མོང་གི་སྔོན་འགྲོ་སྤྱི་ལ་སྦྱོར་ཆོག་པའི་ཁྲིད་ཀྱི་རྒྱབ་ཡིག་་་. An earlier, faulty conversion had put subjoined letters before vowels, dropped ya btags and ra btags, and used the wrong code for ཉ.
- 37,897 syllables were repaired, only where the repair is unique or clearly dominant on the wiki. The share of syllables not found elsewhere on the wiki went from 25.3 % to 2.5 %; the doubtful ones were left as they were.
- To undo: restore the previous revision.
Theg mchog mdzod study page (22 Nov 03:11–03:29)
- New: Theg mchog mdzod (Longchenpa) and 53 subpages (one per chapter, long chapters in parts of at most 40 pages, plus a glossary), after a database dump.
- Each part has one tab per edition: the Adzom Chögar edition first (the standard edition), then the BDRC W22920 e-text, the Pe cin and Karma Lekshey Ling e-texts, and six other editions as OCR of their BDRC scans. Then a critical edition (the corrected base text with the variants of the other editions under each page), the English translation (a machine translation, labelled as such, not yet reviewed by a translator) and a bilingual version.
- It is linked from Seven Treasuries (Longchenpa) and Longchenpa Collected Works.
Adzom edition and Tibetan sizes (22 Nov 03:29–04:01)
- Eric asked for the Adzom Chögar edition, the standard edition of Longchenpa's writings, to have its own tab wherever possible.
- New: Mdzod bdun (a 'dzom par ma), 7 volume pages and 19 text pages (OCR of the BDRC scans MW1PD8, not proofread). The texts were found in the OCR by aligning it with the Karma Lekshey Ling e-texts.
- The Pe cin, sde dge and Karma Lekshey Ling pages of the same works (44 pages) got an Adzom edition tab. It shows the Adzom text itself where it is short enough (24 pages), otherwise a link to the Adzom page (20 pages).
- Seven Treasuries (Longchenpa) has an Adzom column, first.
- Eric found the Tibetan sometimes too big beside the English, and out of line. The Tibetan beside English is now sized relative to the English: 1.3 times in a line of English, 1.45 times for titles, 1.85 times for the title at the top of a page. Before, it had fixed sizes of 16–28 points (21–37 px) next to 14 px English.
- The classes are in MediaWiki:Common.css (edit at 03:29, then a Cloudflare purge of the style file). The e-texts kept the size Eric chose until the next morning (see the next section).
- This wiki shows a page's display title in every link to it, so the old 24-point display titles made links such as Same work in very large. The display titles now use a class that is 1.3 times the English in links and normal size in the page heading (second edit of MediaWiki:Common.css at 03:34). The stored display titles of all project pages were refreshed (03:34–04:01) and the pages purged.
- Empty rows of the info tables no longer leave a blank line.
- In the study page, the variants now sit under each page in English-size type.
- 85 Longchenpa pages (28 new, 57 changed) and the 54 study pages got a new revision, after a database dump (
~/lcp/backup/db_before_longchenpa_v4_…). - To undo: restore the previous revisions, or the dump. The new pages can be deleted.
E-text size (22 Nov 15:28–15:30)
- In the morning Eric asked for the e-texts to be made smaller too. They had a fixed size (16 points, about 1.5 times the English). They are now 1.3 times the English with wider line spacing, the same as the Tibetan paragraphs of the bilingual version: class lcp-etext in MediaWiki:Common.css (edit at 15:30, then a Cloudflare purge of the style file).
- The 54 study pages got a new revision so that their Tibetan text blocks carry the same class; nothing else changed in them.
- The two Pe cin Theg mchog mdzod text pages showed the marker "figure in the print (Word object)" as text in 107 places. In the source pages it is a hidden note, so it is now left out. Longchenpa Collected Works/About was updated to match.
- 57 pages got a new revision, after a database dump (
~/lcp/backup/db_before_longchenpa_v5_…). - To undo: remove the line
div.lcp-etextfrom MediaWiki:Common.css (the size), or restore the previous revisions of the pages.
Content audit fixes (22 Nov 17:33–17:51)
In the afternoon Eric asked for a new, in-depth audit of the content of this wiki, then for most of its findings to be fixed. Everything below concerns rywikitexts only.
- Glossary extension. Lingo and its companion Semantic Glossary were loaded but held no glossary at all. Lingo still scanned every page as it was displayed, which made large pages take up to 159 seconds to display (the limit is 30), so some visits ended in an error and search updates failed. Both are now off; the same 2.65 MB page now displays in 1.5 seconds. Background jobs (search updates, link tables) now run every 5 minutes from a scheduled runner on the server instead of at the end of page views (
$wgJobRunRate = 0).- A first attempt turned off only Lingo; Semantic Glossary needs it, so the wiki answered with errors for about 90 seconds (17:33–17:34) until the change was rolled back. It was then redone with both extensions and a check that loads the new settings before they go live.
- Nyingma Gyubum: page boundaries. The 2020 import had cut the volumes into pages that overlap: most pages ended with the title of the next text, and the next page began with the end of the previous one. 552 overlaps were removed, so each text now begins with its own title (rgya gar skad du …) and ends with its own colophon. No text was lost: across the whole collection the new pages hold exactly the old text with each overlapping passage kept once. Pages that contain several later texts in full (e.g. Ngb-pt-0727) were left as they are, because part of their text is found nowhere else; they need the source edition.
- Gyubum volume pages. The ten pages རྙིང་མ་རྒྱུད་འབུམ།/Vol 01 … Vol 10 now list every text of the volume, with its number and Tibetan title, linked. Until now the 975 texts were reachable only through import debris.
- Import debris. 1,043 pages were deleted: redirects named after server paths and .docx.wiki files from the 2020 import, two broken redirects, and one test page. They can be restored from Special:Undelete. Double redirects were then fixed.
- Pe cin e-texts (the 26-volume gsung 'bum): 615 pages cleaned of Word-conversion residue (1,558 page-number codes, 457 running heads, 175 embedded-object codes, dot leaders and private-use characters). The «…» small-print notes were kept.
- Tibetan Unicode. 513 vowel signs that had become separated from their letter (after a tsheg, shad or space) were put back, and 61 doubled e/o vowel signs were written as the single ai/au signs they stand for. Cases that need a reader's judgement were left.
- Other fixes. Template:TibFontClass12 no longer prints a literal {{{1}}} on about 230 Lekshey Ling pages. The title གནས་ལུགས་རིན་པོ་ཆེའི་མཛོད།, which held a copy of the Lung gi gter mdzod, is now a redirect to the root text of the Gnas lugs mdzod. Dead external links were replaced by Internet Archive copies (or marked as no longer online), a few addresses moved to their new site or to https, and an unused Google Analytics snippet was removed from MediaWiki:Common.js.
- 1,449 pages got a new revision, after a database dump. To undo: restore the previous revisions or the dump; undelete the deleted pages; turn the two extensions back on in LocalSettings.
Theg mchog mdzod: outline, quotations, bilingual colours, tooltips (22 Nov 18:06–18:10)
- Eric asked for a table of contents and the detailed outline of the Treasury of the Supreme Vehicle, a translation and summary of each outline entry, the sources of all the quotations, a colour-coded bilingual tab, and the translator's notes as tooltips, as on the Jonang wiki's bilingual texts.
- Theg mchog mdzod (Longchenpa) has a new Contents tab (every chapter with its summary and the first two levels of its outline) and a Quoted works tab (163 works). Each part has a new Outline tab (2,273 divisions in all, each with an English heading and a summary) and a new Quotations tab (1,339 quotations, each with the source as the text names it, the work identified and, where found, the text on this wiki that contains the quoted words).
- The Translation tab shows the outline headings where each section begins, the source after each quotation, and the notes in tooltips (point at a number or tap it). The Bilingual tab follows the Jonang wiki's layout: blue outline, gold quotations, red words of a quotation being explained, verse line by line.
- Outline, summaries and sources were made by AI (Claude Opus 5.5) from the Tibetan and the translation and placed in the text by program; they have not been checked by a scholar, and the pages say so.
- The parts now have at most 12 pages (before, up to 40), so each page stays near half a megabyte: 139 parts. The former single-part titles (Ch. 01, 03, 07, 08, 10, 11, 15, 16, 21, 24 and Appendix 1) redirect to part 1.
- The styles are the classes sted-* in MediaWiki:Common.css; the tooltips need no script.
- 154 pages got a new revision or were created, after a database dump (
~/lcp/backup/db_before_longchenpa_v6_…). - To undo: restore the previous revisions of the study pages (or the dump), and remove the sted-* block from MediaWiki:Common.css.