From bd7c5551166b68963c620b5b69c0b422434bd80a Mon Sep 17 00:00:00 2001 From: Leon Zandman Date: Tue, 19 May 2026 15:31:00 +0200 Subject: [PATCH] Cleanup --- plan-buildsystem-agnostic.md | 262 ----------------------------- plan-solve-windows-build-errors.md | 72 -------- 2 files changed, 334 deletions(-) delete mode 100644 plan-buildsystem-agnostic.md delete mode 100644 plan-solve-windows-build-errors.md diff --git a/plan-buildsystem-agnostic.md b/plan-buildsystem-agnostic.md deleted file mode 100644 index c3285b0e..00000000 --- a/plan-buildsystem-agnostic.md +++ /dev/null @@ -1,262 +0,0 @@ -# Plan: Make the build system platform-agnostic (Windows-capable) - -> **Agent: keep this plan up to date.** After Leon executes a stage, update the "Progress" section below — check the box, note the commit SHA (if any) and date, and record any deviations from the plan (e.g. new sub-steps discovered, scope changes, verification surprises). If a stage's approach changes mid-flight, edit the corresponding section in "Changes" so the plan stays the source of truth, not a stale snapshot. - -## Progress - -Stages are executed by Leon, not the agent. Tick each box once the stage is done and verified. - -### Section 1 — [Gruntfile.js](Gruntfile.js) -- [x] 1a. `calcDownloadHash` rewritten as custom Grunt task _(2026-05-19, code written, awaiting commit + verification)_ -- [x] 1b. `repoSize` rewritten as custom Grunt task _(2026-05-19)_ -- [x] 1c. `nodeConsumerTestPath` switched to `os.tmpdir()` _(2026-05-19)_ -- [x] 1d. `setupNodeConsumers` / `teardownNodeConsumers` / `testCJSNodeConsumer` / `testESMNodeConsumer` ported _(2026-05-19)_ -- [x] 1e. `generateConfig` / `generateNodeIndex` rewritten as custom Grunt tasks _(2026-05-19)_ -- [x] 1f. `chainCommands()` helper deleted _(2026-05-19)_ -- [x] 1g. Webpack output path uses `path.join` _(2026-05-19)_ - -### Section 2 — [package.json](package.json) -- [x] 2a. `setheapsize` deleted _(2026-05-19)_ -- [x] 2b. `getheapsize` switched to `getHeapSize.mjs` _(2026-05-19)_ -- [x] 2c. `minor` / `tag` switched to `.mjs` orchestrators _(2026-05-19)_ - -### Verification -- [x] macOS: 7-step verification pass (see "Verification" section below) _(2026-05-19)_ -- [x] Windows: 7-step verification pass _(2026-05-19)_ - -### Notes / deviations -- 2026-05-19: Windows verification executed end-to-end on a clean tree (PowerShell, Node v24.14.1, npm 11.11.0): - - `npm install`: 1485 packages installed, postinstall hooks (`fixCryptoApiImports`, `fixSnackbarMarkup`) both ran fine. Exit 0. - - `npm test`: `generateConfig` and `generateNodeIndex` custom tasks ran successfully. 243/243 Node API tests + 1935/1935 operation tests all passing. - - `npm run build`: **BLOCKED by pre-existing webpack errors** (same 7 errors on `master`): jimp babel-loader `Identifier 'e' has already been declared` + `Asset Modules Plugin` invalid generator (`filename` property on `asset/inline`). Confirmed identical failure on `master` branch — not introduced by our changes. `calcDownloadHash` is pure Node.js (`crypto.createHash`/`fs`) and structurally cross-platform. - - `npx grunt`: `repoSize` printed `925 tracked files` / `23.4M repository size`. All ESLint tasks passed. - - `npm run testnodeconsumer`: tmp dir created under `os.tmpdir()`, `npm link` ran, CJS + ESM consumers executed, dir removed. Exit 0. - - `npm run getheapsize`: printed `node heap limit = 4288 Mb` via `getHeapSize.mjs`. - - `npm start`: dev server reached `http://localhost:8080/`; `Invoke-WebRequest` returned HTTP 200 with 9448 bytes of CyberChef HTML. Killed cleanly. -- 2026-05-19: macOS verification executed end-to-end on a clean tree (`rm -rf build/ node_modules/`). - - `npm test`: 243 Node API tests + 1935 operation tests, all passing. `generateConfig` and `generateNodeIndex` custom tasks ran without shell-outs. - - `npm run build`: `calcDownloadHash` produced [build/prod/sha256digest.txt](build/prod/sha256digest.txt) matching `shasum -a 256 build/prod/CyberChef_v11.0.0.zip`; no `DOWNLOAD_HASH_PLACEHOLDER` left in [build/prod/index.html](build/prod/index.html); hash present once. Pre-existing `webpack-bundle-analyzer` ENOENT warnings about `LoaderWorker.js`/`ZipWorker.js` are unrelated (analyzer references workers it doesn't find on disk) — not introduced by this change. - - `npx grunt`: `repoSize` printed `925\ttracked files` / `256.9M\trepository size` (same shape as the old shell version; absolute size differs only because `build/` was populated). - - `npm run testnodeconsumer`: tmp dir created at `$TMPDIR/tmp-cyberchef`, CJS + ESM consumers ran, dir removed. - - `npm run getheapsize`: printed `node heap limit = 4288 Mb` via the new [src/core/config/scripts/getHeapSize.mjs](src/core/config/scripts/getHeapSize.mjs). - - `npm start`: dev server reached `Project is running at … http://localhost:8080/`; `curl http://localhost:8080/` returned HTTP 200 with the CyberChef HTML. Killed cleanly. -- 2026-05-19: macOS re-verified after the Windows-fix regex changes in c1d638fb ([webpack.config.js](webpack.config.js) `node_modules\/` → `node_modules[/\\]`, `bmfonts\/` → `bmfonts[/\\]`, `web\/static` → `web[/\\]static`). Patterns are platform-agnostic — `/` still matches via `[/\\]` on macOS. - - `npm run build`: webpack compiled successfully in ~39s; all post-tasks (copy, zip, clean, calcDownloadHash, chmod) completed. - - Artifact integrity: [build/prod/CyberChef_v11.0.0.zip](build/prod/CyberChef_v11.0.0.zip) SHA256 matches [build/prod/sha256digest.txt](build/prod/sha256digest.txt); hash stamped once into [build/prod/index.html](build/prod/index.html); no `DOWNLOAD_HASH_PLACEHOLDER` remaining. - - `npm test`: 243/243 Node API tests + 1935/1935 operation tests passing. - - Pre-existing `webpack-bundle-analyzer` ENOENT warnings about `*Worker.js` still appear — same ones noted in the prior macOS verification entry above, unchanged by c1d638fb. - ---- - -## Context - -The build is currently tuned to Linux/macOS. A developer on Windows can clone the repo and `npm install`, but `npm run build`, `npm test`, and `npm run testnodeconsumer` will fail because several [Gruntfile.js](Gruntfile.js) `exec` tasks and three `package.json` scripts shell out to Unix utilities (`sha256sum`/`shasum`, `awk`, `sed`, `wc`, `du`, `egrep`, `xargs`, `mkdir`, `cp`, `rm -rf`, `cd`, `export`, backticks, `$(…)` subshells, single-quoted args). A `chainCommands()` helper at [Gruntfile.js:172-183](Gruntfile.js#L172-L183) hints at Windows awareness but no Windows branch exists for the underlying commands. - -**Scope (per user):** Local dev only. Goal: `npm install`, `npm run build`, `npm test`, `npm start`, `npm run testnodeconsumer` all work on Windows in addition to macOS/Linux. CI (`ubuntu-latest`-pinned) and docs are out of scope. - -**Approach (per user):** Inline Node.js — rewrite shell-out tasks using `crypto`, `fs`, `os`, `path` from the Node 24 stdlib. No new devDependencies. Eliminates platform branching rather than adding more. - -**Things that are already fine and should stay untouched:** -- [.gitattributes](.gitattributes) (`* text=auto eol=lf`) -- [.editorconfig](.editorconfig) (`end_of_line = lf`) -- [webpack.config.js](webpack.config.js) and `webpack-dev-server` (cross-platform) -- App code in [src/web/](src/web/) and [src/node/](src/node/) (browser-safe; no `fs`/`child_process` leaks) -- The `chmod` Grunt task — `grunt-chmod` is a silent no-op on Windows, so it doesn't break anything -- The `browserTests` task (`./node_modules/.bin/nightwatch`) — npm creates a `.cmd` shim on Windows; cmd.exe resolves the path. Leave as-is. - ---- - -## Changes - -### 1. [Gruntfile.js](Gruntfile.js) — replace shell-out tasks with Node code - -Convert the exec tasks below into **custom Grunt tasks** that run Node code directly. This is cleaner than passing JS-as-a-string through `grunt-exec`, and it deletes all platform branching. - -#### 1a. `calcDownloadHash` — [Gruntfile.js:331-346](Gruntfile.js#L331-L346) - -Currently branches on `darwin` vs default, both shell-only. Replace with: - -```js -grunt.registerTask("calcDownloadHash", "Computes the SHA256 of the standalone zip and stamps it into index.html", function () { - const crypto = require("crypto"); - const fs = require("fs"); - const zipPath = path.join("build", "prod", `CyberChef_v${pkg.version}.zip`); - const digestPath = path.join("build", "prod", "sha256digest.txt"); - const indexPath = path.join("build", "prod", "index.html"); - const hash = crypto.createHash("sha256").update(fs.readFileSync(zipPath)).digest("hex"); - fs.writeFileSync(digestPath, hash); - const html = fs.readFileSync(indexPath, "utf8").replace("DOWNLOAD_HASH_PLACEHOLDER", hash); - fs.writeFileSync(indexPath, html); -}); -``` - -Then in the `prod` task list at [Gruntfile.js:32](Gruntfile.js#L32), change `"exec:calcDownloadHash"` → `"calcDownloadHash"`. Delete the `calcDownloadHash` entry from the `exec` block. - -#### 1b. `repoSize` — [Gruntfile.js:347-353](Gruntfile.js#L347-L353) - -Cosmetic info printed by the default `lint` task. Replace with: - -```js -grunt.registerTask("repoSize", "Reports tracked file count and repo size", function () { - const { execSync } = require("child_process"); - const fs = require("fs"); - const fileCount = execSync("git ls-files", { encoding: "utf8" }).trim().split("\n").length; - let bytes = 0; - const walk = (dir) => { - for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { - if (entry.name === ".git" || entry.name === "node_modules") continue; - const p = path.join(dir, entry.name); - if (entry.isDirectory()) walk(p); - else if (entry.isFile()) bytes += fs.statSync(p).size; - } - }; - walk("."); - const mb = (bytes / (1024 * 1024)).toFixed(1); - grunt.log.writeln(`\n${fileCount}\ttracked files`); - grunt.log.writeln(`${mb}M\trepository size`); -}); -``` - -Update [Gruntfile.js:57](Gruntfile.js#L57) (`default` task) to call `"repoSize"` instead of `"exec:repoSize"`. Delete the `repoSize` entry from the `exec` block. - -#### 1c. `nodeConsumerTestPath` — [Gruntfile.js:100](Gruntfile.js#L100) - -Replace `"~/tmp-cyberchef"` with `path.join(os.tmpdir(), "tmp-cyberchef")` (add `const os = require("os");` at the top alongside the existing `path` require). `~` is not expanded on Windows, and `os.tmpdir()` is the right answer on every OS. - -#### 1d. `setupNodeConsumers` / `teardownNodeConsumers` / `testCJSNodeConsumer` / `testESMNodeConsumer` — [Gruntfile.js:382-412](Gruntfile.js#L382-L412) - -These use `mkdir`, `cp`, `cd`, `rm -rf`. Replace with custom Grunt tasks using Node APIs and grunt-exec's `cwd` option: - -```js -grunt.registerTask("setupNodeConsumers", "Sets up a temp dir for testing CJS/ESM consumers", function () { - const fs = require("fs"); - grunt.log.writeln("\n--- Testing node consumers ---"); - fs.mkdirSync(nodeConsumerTestPath, { recursive: true }); - fs.cpSync("tests/node/consumers", nodeConsumerTestPath, { recursive: true }); - require("child_process").execSync("npm link", { stdio: "inherit" }); - require("child_process").execSync("npm link cyberchef", { stdio: "inherit", cwd: nodeConsumerTestPath }); -}); - -grunt.registerTask("teardownNodeConsumers", "Removes the consumer test temp dir", function () { - require("fs").rmSync(nodeConsumerTestPath, { recursive: true, force: true }); - grunt.log.writeln("\n--- Node consumer tests complete ---"); -}); -``` - -For the CJS/ESM consumer test exec entries at [Gruntfile.js:399-412](Gruntfile.js#L399-L412), drop the `cd …` prefix and pass `cwd: nodeConsumerTestPath` on the exec config object instead: - -```js -testCJSNodeConsumer: { - command: `node ${nodeFlags} cjs-consumer.js`, - cwd: nodeConsumerTestPath, - stdout: false, -}, -testESMNodeConsumer: { - command: `node ${nodeFlags} esm-consumer.mjs`, - cwd: nodeConsumerTestPath, - stdout: false, -}, -``` - -Update the `testnodeconsumer` registered task at [Gruntfile.js:51-53](Gruntfile.js#L51-L53) to reference the new task names (drop the `exec:` prefix for setup/teardown). - -#### 1e. `generateConfig` / `generateNodeIndex` — [Gruntfile.js:361-378](Gruntfile.js#L361-L378) - -These chain `echo` (with single-quoted args containing `\n`) and `echo [] > …` redirection — single quotes aren't quotes on Windows cmd.exe, and the `\n` handling in `chainCommands` is a band-aid. Replace each with a custom Grunt task that uses `grunt.log.writeln` for the banners and `fs.writeFileSync` for the empty JSON file, then `execSync`s the `node …` calls: - -```js -grunt.registerTask("generateConfig", "Regenerates operation config files", function () { - const { execSync } = require("child_process"); - const fs = require("fs"); - grunt.log.writeln("\n--- Regenerating config files. ---"); - fs.writeFileSync(path.join("src", "core", "config", "OperationConfig.json"), "[]\n"); - execSync(`node ${nodeFlags} src/core/config/scripts/generateOpsIndex.mjs`, { stdio: "inherit" }); - execSync(`node ${nodeFlags} src/core/config/scripts/generateConfig.mjs`, { stdio: "inherit" }); - grunt.log.writeln("--- Config scripts finished. ---\n"); -}); - -grunt.registerTask("generateNodeIndex", "Regenerates the node index", function () { - const { execSync } = require("child_process"); - grunt.log.writeln("\n--- Regenerating node index ---"); - execSync(`node ${nodeFlags} src/node/config/scripts/generateNodeIndex.mjs`, { stdio: "inherit" }); - grunt.log.writeln("--- Node index generated. ---\n"); -}); -``` - -Update the four task lists at [Gruntfile.js:26](Gruntfile.js#L26), [:31](Gruntfile.js#L31), [:38](Gruntfile.js#L38), [:44](Gruntfile.js#L44) to call `"generateConfig"` / `"generateNodeIndex"` instead of `"exec:generateConfig"` / `"exec:generateNodeIndex"`. Update the `watch:config` block at [:321](Gruntfile.js#L321) likewise. Delete the two exec entries. - -#### 1f. `chainCommands()` helper — [Gruntfile.js:172-183](Gruntfile.js#L172-L183) - -After 1a–1e it has zero call sites. Delete it. - -#### 1g. Webpack output path — [Gruntfile.js:113](Gruntfile.js#L113) - -Change `__dirname + "/build/prod"` to `path.join(__dirname, "build", "prod")`. Cosmetic (webpack handles `/` on Windows), but matches the rest of the file once we're touching it. - -### 2. [package.json](package.json) — fix the three broken scripts - -#### 2a. `setheapsize` — [package.json:211](package.json#L211) - -`export NODE_OPTIONS=…` in an npm script sets the variable in a subshell that immediately exits — it does nothing useful on **any** platform. It's also broken on Windows (`export` isn't a cmd.exe builtin). **Recommendation: delete it.** Users who need a larger heap can set `NODE_OPTIONS` in their own shell. - -#### 2b. `getheapsize` — [package.json:210](package.json#L210) - -Uses `node -e '…single-quoted JS…'`. Windows cmd.exe doesn't strip single quotes, so the JS won't parse. Switch to a `.mjs` file: - -- Create [src/core/config/scripts/getHeapSize.mjs](src/core/config/scripts/getHeapSize.mjs): - ```js - import v8 from "v8"; - console.log(`node heap limit = ${v8.getHeapStatistics().heap_size_limit / (1024 * 1024)} Mb`); - ``` -- Update the script to `"node --no-warnings src/core/config/scripts/getHeapSize.mjs"`. - -#### 2c. `minor` and `tag` — [package.json:208-209](package.json#L208-L209) - -Both use `$(npm pkg get version | xargs)` which (a) is a subshell (broken on Windows cmd.exe) and (b) relies on `xargs` to strip the quotes npm wraps the version in. - -Cleaner: a tiny helper script that prints the bare version. Create [src/core/config/scripts/printVersion.mjs](src/core/config/scripts/printVersion.mjs): - -```js -import { readFileSync } from "fs"; -import { fileURLToPath } from "url"; -import { dirname, join } from "path"; -const pkg = JSON.parse(readFileSync(join(dirname(fileURLToPath(import.meta.url)), "..", "..", "..", "..", "package.json"), "utf8")); -process.stdout.write(pkg.version); -``` - -Then in `package.json`, replace both scripts with versions that capture stdout via `node -p` invocations. **Note:** npm scripts on Windows still can't do shell-style command substitution. The portable rewrite uses `node` as the substitution engine. Both `minor` and `tag` involve interactive `git tag -s` / version bumping, so we can also extract the multi-step flow into a single `.mjs` orchestrator — recommended approach: - -- Create [src/core/config/scripts/tagRelease.mjs](src/core/config/scripts/tagRelease.mjs) that reads `package.json`, runs `git tag -s "v${version}" -m "${version}"` via `execSync`, and logs the message. The `minor` script likewise calls `newMinorVersion.mjs`, runs `npm version minor --git-tag-version=false`, then re-reads the version and logs the PR/tag prompt. -- Update `package.json` to `"minor": "node src/core/config/scripts/bumpMinor.mjs"` and `"tag": "node src/core/config/scripts/tagRelease.mjs"`. - -This eliminates all shell-substitution syntax from `package.json`. - ---- - -## Files to be modified - -- [Gruntfile.js](Gruntfile.js) — bulk of the work (sections 1a–1g) -- [package.json](package.json) — script changes (sections 2a–2c) -- New: [src/core/config/scripts/getHeapSize.mjs](src/core/config/scripts/getHeapSize.mjs) -- New: [src/core/config/scripts/bumpMinor.mjs](src/core/config/scripts/bumpMinor.mjs) and [src/core/config/scripts/tagRelease.mjs](src/core/config/scripts/tagRelease.mjs) (or one combined helper) - -## Reused utilities - -- Node stdlib `crypto.createHash`, `fs.cpSync`, `fs.rmSync`, `fs.readdirSync({ withFileTypes: true })`, `os.tmpdir()`, `path.join` — no new devDependencies -- Existing `grunt-exec` `cwd` option (already part of grunt-exec, no new dep) for the consumer-test exec tasks -- Existing `grunt.log.writeln` instead of `echo` -- Existing pattern from [src/core/config/scripts/newMinorVersion.mjs:50](src/core/config/scripts/newMinorVersion.mjs#L50) (uses `execSync` for git) — same pattern for new helpers - -## Verification - -Run on macOS first (current dev machine; quickest feedback loop): - -1. `rm -rf build/ node_modules/ && npm install` — confirms `postinstall` still passes -2. `npm test` — full test suite must pass (covers `configTests` → `exec:generateConfig` + `exec:generateNodeIndex`, renamed) -3. `npm run build` — exercises the prod task chain including the rewritten `calcDownloadHash`. Verify [build/prod/sha256digest.txt](build/prod/sha256digest.txt) exists and matches `shasum -a 256 build/prod/CyberChef_v*.zip`, and that [build/prod/index.html](build/prod/index.html) contains the hash (no `DOWNLOAD_HASH_PLACEHOLDER` left). -4. `npx grunt` (default lint task) — should print the `repoSize` output with the same shape as before -5. `npm run testnodeconsumer` — verify a tmp dir is created under `os.tmpdir()`, both consumer scripts succeed, and the tmp dir is cleaned up -6. `npm run getheapsize` — should print `node heap limit = … Mb` -7. `npm start` — confirms dev server still launches (no changes to it, but verifies nothing collateral broke) - -Then if a Windows machine is available, run the same 7 steps in PowerShell. If not available, the change is structurally safe because every shell-out has been eliminated; the residual risk is in the small set of `execSync("npm link …")` and `execSync("git ls-files")` calls, both of which are documented cross-platform. diff --git a/plan-solve-windows-build-errors.md b/plan-solve-windows-build-errors.md deleted file mode 100644 index d7af2b5f..00000000 --- a/plan-solve-windows-build-errors.md +++ /dev/null @@ -1,72 +0,0 @@ -# Plan: Fix Windows `npm run build` webpack errors - -## Context - -`npm run build` fails on Windows with 7 webpack errors (exit code 1). - -The two visible error classes are: - -1. **jimp babel-loader conflict**: `Module parse failed: Identifier 'e' has already been declared` at `node_modules/jimp/dist/browser/index.js:31135`. Babel is transforming jimp's pre-minified browser bundle, which produces invalid JS. On macOS/Linux, jimp is correctly excluded from babel-loader; on Windows, the exclude regex silently fails. - -2. **Asset Modules Plugin — invalid generator**: `generator has an unknown property 'filename'` on `asset/inline` type. First-party images (e.g. `src/web/static/images/file-128x128.png`) should match the `asset/resource` rule with a `filename` generator, but also match the `asset/inline` rule whose `exclude: /web\/static/` doesn't fire on Windows paths. Two rules matching causes webpack to reject the generator config. - -## Root cause - -All three bugs share one root cause: **regex path separators in [webpack.config.js](webpack.config.js) use literal `/` which doesn't match `\` on Windows**. - -Webpack 5 matches `test` / `exclude` / `include` patterns against the module's **absolute file path**, which uses the OS-native separator. On Windows, paths contain `\`, so `/node_modules\//` doesn't match `C:\…\node_modules\jimp\…`. - -Proof (from the verification terminal): - -``` -> node -e "const re = /node_modules\/(?!crypto-api|bootstrap)/; console.log(re.test('node_modules/jimp'), re.test('node_modules\\jimp'))" -true false - -> node -e "const re = /web\/static/; console.log(re.test('src/web/static/x'), re.test('src\\web\\static\\x'))" -true false -``` - -## Changes - -### Single file: [webpack.config.js](webpack.config.js) - -Three regexes need `\/` → `[/\\]` (matches both forward and back slashes): - -#### A. Line 145 — babel-loader exclude - -```diff -- exclude: /node_modules\/(?!crypto-api|bootstrap)/, -+ exclude: /node_modules[/\\](?!crypto-api|bootstrap)/, -``` - -**Effect**: On Windows, jimp (and all other non-allowlisted node_modules) are correctly excluded from babel-loader again. Fixes error 1. - -#### B. Line 208 — bmfonts test - -```diff -- test: /(\.fnt$|bmfonts\/.+\.png$)/, -+ test: /(\.fnt$|bmfonts[/\\].+\.png$)/, -``` - -**Effect**: Font bitmap PNGs inside `bmfonts/` are matched on Windows too, so they get `asset/resource` with the `assets/fonts/[name][ext]` filename generator instead of falling through to the generic PNG rules. Not directly surfacing as an error today (the bmfonts files may not be in the current source tree), but would cause wrong asset handling on Windows if they were. - -#### C. Line 224 — third-party image exclude - -```diff -- exclude: /web\/static/, -+ exclude: /web[/\\]static/, -``` - -**Effect**: First-party images in `src/web/static/` are correctly excluded from the `asset/inline` rule on Windows, so they only match the `asset/resource` rule above (which has the `filename` generator). Fixes error 2. - -## Files modified - -- [webpack.config.js](webpack.config.js) — 3 one-character regex changes (insert `[/\\]` in place of `\/`) - -No new files, no new dependencies. - -## Verification - -1. `npm run build` on Windows — should now complete without webpack errors; `build/prod/` should contain the full output including `sha256digest.txt` with the correct hash stamped into `index.html`. -2. `npm run build` on macOS — should still work identically (forward slashes match `[/\\]`). -3. `npm start` on Windows — dev server should compile without errors (currently the dev server compiles with warnings but serves; after the fix it should be clean).