diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..a5bc34b3 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,54 @@ +# AGENTS.md + +Working agreement for AI coding agents in this repo. Task-specific plans (e.g. [plan-buildsystem-agnostic.md](plan-buildsystem-agnostic.md)) own step-by-step changes; this file owns durable conventions that should outlast any single plan. + +When a `plan-*.md` file exists in the repo root, treat it as the source of truth for the in-flight task — read it before proposing changes to the same area. + +## Project + +CyberChef — client-side web app for encoding, encryption, compression, and data analysis. The same operation source ([src/core/operations/](src/core/operations/)) ships as both a browser bundle and a Node consumer package. Build orchestration is Grunt ([Gruntfile.js](Gruntfile.js)); bundling is webpack ([webpack.config.js](webpack.config.js)). + +## Toolchain + +- Node **24** (`engines: ">=24 <25"` in [package.json](package.json)). Don't reach for features outside that range or assume newer. +- Package manager: npm. +- **No new devDependencies** without explicit approval. Prefer the Node stdlib (`crypto`, `fs`, `os`, `path`, `child_process`, `v8`). + +## Common commands + +| Task | Command | +| --- | --- | +| Install (runs `postinstall` Grunt fixups) | `npm install` | +| Dev server | `npm start` | +| Production build | `npm run build` | +| Full test suite | `npm test` | +| Node-consumer tests | `npm run testnodeconsumer` | +| UI tests (Nightwatch) | `npm run testui` | +| Lint | `npm run lint` | + +## Build-system conventions + +- The build must work on **macOS, Linux, and Windows**. Don't introduce Unix-only shell-outs in [Gruntfile.js](Gruntfile.js) or [package.json](package.json) `scripts` — that means no `sed`, `awk`, `xargs`, `wc`, `du`, `egrep`, `sha256sum`/`shasum`, `$(…)` command substitution, single-quoted JS passed to `node -e`, `~` path expansion, or `export FOO=…` in an npm script. Reach for the Node stdlib first; if a shell command is genuinely needed, confirm it works in both `cmd.exe` and POSIX shells. +- Prefer custom Grunt tasks (`grunt.registerTask`) over `grunt-exec` entries when the work is Node code — they're easier to read, easier to test, and platform-neutral. +- One-off helper scripts live in [src/core/config/scripts/](src/core/config/scripts/) as `.mjs` files. Follow the existing pattern in [newMinorVersion.mjs](src/core/config/scripts/newMinorVersion.mjs) (ESM, top-level `execSync` for git ops, no external deps beyond stdlib). + +## Coding conventions + +From [CONTRIBUTING.md](CONTRIBUTING.md): + +- 4-space indentation, LF line endings, UTF-8 without BOM, trailing newline on every file. +- `CamelCase` for namespaces/objects, `camelCase` for functions/variables, `UNDERSCORE_UPPER_CASE` for constants. +- Vanilla JS preferred. Don't add UI frameworks; jQuery is already vendored but should be avoided for new code. +- Operations must be client-side wherever possible (design principle: works on closed networks / offline). + +## Repo layout (orientation) + +- [src/core/](src/core/) — operations, config, generation scripts (browser-safe; no `fs`/`child_process` here) +- [src/web/](src/web/) — browser UI +- [src/node/](src/node/) — Node consumer entry points +- [tests/](tests/) — `operations/`, `node/`, and UI (`browser/`) test suites + +## Workflow + +- **Leon authors all git commits and pushes himself.** Stop at "ready to commit" — do not run `git commit`, `git push`, `gh pr create`, or any branch-moving command without an explicit ask. Suggesting a commit message in chat is fine; executing it is not. +- For multi-step changes, write or update a `plan-*.md` in the repo root before editing. Keep step-by-step task detail in the plan, not in this file. diff --git a/plan-buildsystem-agnostic.md b/plan-buildsystem-agnostic.md new file mode 100644 index 00000000..7762aea3 --- /dev/null +++ b/plan-buildsystem-agnostic.md @@ -0,0 +1,243 @@ +# Plan: Make the build system platform-agnostic (Windows-capable) + +> **Agent: keep this plan up to date.** After Leon executes a stage, update the "Progress" section below — check the box, note the commit SHA (if any) and date, and record any deviations from the plan (e.g. new sub-steps discovered, scope changes, verification surprises). If a stage's approach changes mid-flight, edit the corresponding section in "Changes" so the plan stays the source of truth, not a stale snapshot. + +## Progress + +Stages are executed by Leon, not the agent. Tick each box once the stage is done and verified. + +### Section 1 — [Gruntfile.js](Gruntfile.js) +- [ ] 1a. `calcDownloadHash` rewritten as custom Grunt task +- [ ] 1b. `repoSize` rewritten as custom Grunt task +- [ ] 1c. `nodeConsumerTestPath` switched to `os.tmpdir()` +- [ ] 1d. `setupNodeConsumers` / `teardownNodeConsumers` / `testCJSNodeConsumer` / `testESMNodeConsumer` ported +- [ ] 1e. `generateConfig` / `generateNodeIndex` rewritten as custom Grunt tasks +- [ ] 1f. `chainCommands()` helper deleted +- [ ] 1g. Webpack output path uses `path.join` + +### Section 2 — [package.json](package.json) +- [ ] 2a. `setheapsize` deleted +- [ ] 2b. `getheapsize` switched to `getHeapSize.mjs` +- [ ] 2c. `minor` / `tag` switched to `.mjs` orchestrators + +### Verification +- [ ] macOS: 7-step verification pass (see "Verification" section below) +- [ ] Windows: 7-step verification pass (if a Windows machine is available) + +### Notes / deviations +_(Agent: log any changes from the original plan here, with date.)_ + +--- + +## Context + +The build is currently tuned to Linux/macOS. A developer on Windows can clone the repo and `npm install`, but `npm run build`, `npm test`, and `npm run testnodeconsumer` will fail because several [Gruntfile.js](Gruntfile.js) `exec` tasks and three `package.json` scripts shell out to Unix utilities (`sha256sum`/`shasum`, `awk`, `sed`, `wc`, `du`, `egrep`, `xargs`, `mkdir`, `cp`, `rm -rf`, `cd`, `export`, backticks, `$(…)` subshells, single-quoted args). A `chainCommands()` helper at [Gruntfile.js:172-183](Gruntfile.js#L172-L183) hints at Windows awareness but no Windows branch exists for the underlying commands. + +**Scope (per user):** Local dev only. Goal: `npm install`, `npm run build`, `npm test`, `npm start`, `npm run testnodeconsumer` all work on Windows in addition to macOS/Linux. CI (`ubuntu-latest`-pinned) and docs are out of scope. + +**Approach (per user):** Inline Node.js — rewrite shell-out tasks using `crypto`, `fs`, `os`, `path` from the Node 24 stdlib. No new devDependencies. Eliminates platform branching rather than adding more. + +**Things that are already fine and should stay untouched:** +- [.gitattributes](.gitattributes) (`* text=auto eol=lf`) +- [.editorconfig](.editorconfig) (`end_of_line = lf`) +- [webpack.config.js](webpack.config.js) and `webpack-dev-server` (cross-platform) +- App code in [src/web/](src/web/) and [src/node/](src/node/) (browser-safe; no `fs`/`child_process` leaks) +- The `chmod` Grunt task — `grunt-chmod` is a silent no-op on Windows, so it doesn't break anything +- The `browserTests` task (`./node_modules/.bin/nightwatch`) — npm creates a `.cmd` shim on Windows; cmd.exe resolves the path. Leave as-is. + +--- + +## Changes + +### 1. [Gruntfile.js](Gruntfile.js) — replace shell-out tasks with Node code + +Convert the exec tasks below into **custom Grunt tasks** that run Node code directly. This is cleaner than passing JS-as-a-string through `grunt-exec`, and it deletes all platform branching. + +#### 1a. `calcDownloadHash` — [Gruntfile.js:331-346](Gruntfile.js#L331-L346) + +Currently branches on `darwin` vs default, both shell-only. Replace with: + +```js +grunt.registerTask("calcDownloadHash", "Computes the SHA256 of the standalone zip and stamps it into index.html", function () { + const crypto = require("crypto"); + const fs = require("fs"); + const zipPath = path.join("build", "prod", `CyberChef_v${pkg.version}.zip`); + const digestPath = path.join("build", "prod", "sha256digest.txt"); + const indexPath = path.join("build", "prod", "index.html"); + const hash = crypto.createHash("sha256").update(fs.readFileSync(zipPath)).digest("hex"); + fs.writeFileSync(digestPath, hash); + const html = fs.readFileSync(indexPath, "utf8").replace("DOWNLOAD_HASH_PLACEHOLDER", hash); + fs.writeFileSync(indexPath, html); +}); +``` + +Then in the `prod` task list at [Gruntfile.js:32](Gruntfile.js#L32), change `"exec:calcDownloadHash"` → `"calcDownloadHash"`. Delete the `calcDownloadHash` entry from the `exec` block. + +#### 1b. `repoSize` — [Gruntfile.js:347-353](Gruntfile.js#L347-L353) + +Cosmetic info printed by the default `lint` task. Replace with: + +```js +grunt.registerTask("repoSize", "Reports tracked file count and repo size", function () { + const { execSync } = require("child_process"); + const fs = require("fs"); + const fileCount = execSync("git ls-files", { encoding: "utf8" }).trim().split("\n").length; + let bytes = 0; + const walk = (dir) => { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + if (entry.name === ".git" || entry.name === "node_modules") continue; + const p = path.join(dir, entry.name); + if (entry.isDirectory()) walk(p); + else if (entry.isFile()) bytes += fs.statSync(p).size; + } + }; + walk("."); + const mb = (bytes / (1024 * 1024)).toFixed(1); + grunt.log.writeln(`\n${fileCount}\ttracked files`); + grunt.log.writeln(`${mb}M\trepository size`); +}); +``` + +Update [Gruntfile.js:57](Gruntfile.js#L57) (`default` task) to call `"repoSize"` instead of `"exec:repoSize"`. Delete the `repoSize` entry from the `exec` block. + +#### 1c. `nodeConsumerTestPath` — [Gruntfile.js:100](Gruntfile.js#L100) + +Replace `"~/tmp-cyberchef"` with `path.join(os.tmpdir(), "tmp-cyberchef")` (add `const os = require("os");` at the top alongside the existing `path` require). `~` is not expanded on Windows, and `os.tmpdir()` is the right answer on every OS. + +#### 1d. `setupNodeConsumers` / `teardownNodeConsumers` / `testCJSNodeConsumer` / `testESMNodeConsumer` — [Gruntfile.js:382-412](Gruntfile.js#L382-L412) + +These use `mkdir`, `cp`, `cd`, `rm -rf`. Replace with custom Grunt tasks using Node APIs and grunt-exec's `cwd` option: + +```js +grunt.registerTask("setupNodeConsumers", "Sets up a temp dir for testing CJS/ESM consumers", function () { + const fs = require("fs"); + grunt.log.writeln("\n--- Testing node consumers ---"); + fs.mkdirSync(nodeConsumerTestPath, { recursive: true }); + fs.cpSync("tests/node/consumers", nodeConsumerTestPath, { recursive: true }); + require("child_process").execSync("npm link", { stdio: "inherit" }); + require("child_process").execSync("npm link cyberchef", { stdio: "inherit", cwd: nodeConsumerTestPath }); +}); + +grunt.registerTask("teardownNodeConsumers", "Removes the consumer test temp dir", function () { + require("fs").rmSync(nodeConsumerTestPath, { recursive: true, force: true }); + grunt.log.writeln("\n--- Node consumer tests complete ---"); +}); +``` + +For the CJS/ESM consumer test exec entries at [Gruntfile.js:399-412](Gruntfile.js#L399-L412), drop the `cd …` prefix and pass `cwd: nodeConsumerTestPath` on the exec config object instead: + +```js +testCJSNodeConsumer: { + command: `node ${nodeFlags} cjs-consumer.js`, + cwd: nodeConsumerTestPath, + stdout: false, +}, +testESMNodeConsumer: { + command: `node ${nodeFlags} esm-consumer.mjs`, + cwd: nodeConsumerTestPath, + stdout: false, +}, +``` + +Update the `testnodeconsumer` registered task at [Gruntfile.js:51-53](Gruntfile.js#L51-L53) to reference the new task names (drop the `exec:` prefix for setup/teardown). + +#### 1e. `generateConfig` / `generateNodeIndex` — [Gruntfile.js:361-378](Gruntfile.js#L361-L378) + +These chain `echo` (with single-quoted args containing `\n`) and `echo [] > …` redirection — single quotes aren't quotes on Windows cmd.exe, and the `\n` handling in `chainCommands` is a band-aid. Replace each with a custom Grunt task that uses `grunt.log.writeln` for the banners and `fs.writeFileSync` for the empty JSON file, then `execSync`s the `node …` calls: + +```js +grunt.registerTask("generateConfig", "Regenerates operation config files", function () { + const { execSync } = require("child_process"); + const fs = require("fs"); + grunt.log.writeln("\n--- Regenerating config files. ---"); + fs.writeFileSync(path.join("src", "core", "config", "OperationConfig.json"), "[]\n"); + execSync(`node ${nodeFlags} src/core/config/scripts/generateOpsIndex.mjs`, { stdio: "inherit" }); + execSync(`node ${nodeFlags} src/core/config/scripts/generateConfig.mjs`, { stdio: "inherit" }); + grunt.log.writeln("--- Config scripts finished. ---\n"); +}); + +grunt.registerTask("generateNodeIndex", "Regenerates the node index", function () { + const { execSync } = require("child_process"); + grunt.log.writeln("\n--- Regenerating node index ---"); + execSync(`node ${nodeFlags} src/node/config/scripts/generateNodeIndex.mjs`, { stdio: "inherit" }); + grunt.log.writeln("--- Node index generated. ---\n"); +}); +``` + +Update the four task lists at [Gruntfile.js:26](Gruntfile.js#L26), [:31](Gruntfile.js#L31), [:38](Gruntfile.js#L38), [:44](Gruntfile.js#L44) to call `"generateConfig"` / `"generateNodeIndex"` instead of `"exec:generateConfig"` / `"exec:generateNodeIndex"`. Update the `watch:config` block at [:321](Gruntfile.js#L321) likewise. Delete the two exec entries. + +#### 1f. `chainCommands()` helper — [Gruntfile.js:172-183](Gruntfile.js#L172-L183) + +After 1a–1e it has zero call sites. Delete it. + +#### 1g. Webpack output path — [Gruntfile.js:113](Gruntfile.js#L113) + +Change `__dirname + "/build/prod"` to `path.join(__dirname, "build", "prod")`. Cosmetic (webpack handles `/` on Windows), but matches the rest of the file once we're touching it. + +### 2. [package.json](package.json) — fix the three broken scripts + +#### 2a. `setheapsize` — [package.json:211](package.json#L211) + +`export NODE_OPTIONS=…` in an npm script sets the variable in a subshell that immediately exits — it does nothing useful on **any** platform. It's also broken on Windows (`export` isn't a cmd.exe builtin). **Recommendation: delete it.** Users who need a larger heap can set `NODE_OPTIONS` in their own shell. + +#### 2b. `getheapsize` — [package.json:210](package.json#L210) + +Uses `node -e '…single-quoted JS…'`. Windows cmd.exe doesn't strip single quotes, so the JS won't parse. Switch to a `.mjs` file: + +- Create [src/core/config/scripts/getHeapSize.mjs](src/core/config/scripts/getHeapSize.mjs): + ```js + import v8 from "v8"; + console.log(`node heap limit = ${v8.getHeapStatistics().heap_size_limit / (1024 * 1024)} Mb`); + ``` +- Update the script to `"node --no-warnings src/core/config/scripts/getHeapSize.mjs"`. + +#### 2c. `minor` and `tag` — [package.json:208-209](package.json#L208-L209) + +Both use `$(npm pkg get version | xargs)` which (a) is a subshell (broken on Windows cmd.exe) and (b) relies on `xargs` to strip the quotes npm wraps the version in. + +Cleaner: a tiny helper script that prints the bare version. Create [src/core/config/scripts/printVersion.mjs](src/core/config/scripts/printVersion.mjs): + +```js +import { readFileSync } from "fs"; +import { fileURLToPath } from "url"; +import { dirname, join } from "path"; +const pkg = JSON.parse(readFileSync(join(dirname(fileURLToPath(import.meta.url)), "..", "..", "..", "..", "package.json"), "utf8")); +process.stdout.write(pkg.version); +``` + +Then in `package.json`, replace both scripts with versions that capture stdout via `node -p` invocations. **Note:** npm scripts on Windows still can't do shell-style command substitution. The portable rewrite uses `node` as the substitution engine. Both `minor` and `tag` involve interactive `git tag -s` / version bumping, so we can also extract the multi-step flow into a single `.mjs` orchestrator — recommended approach: + +- Create [src/core/config/scripts/tagRelease.mjs](src/core/config/scripts/tagRelease.mjs) that reads `package.json`, runs `git tag -s "v${version}" -m "${version}"` via `execSync`, and logs the message. The `minor` script likewise calls `newMinorVersion.mjs`, runs `npm version minor --git-tag-version=false`, then re-reads the version and logs the PR/tag prompt. +- Update `package.json` to `"minor": "node src/core/config/scripts/bumpMinor.mjs"` and `"tag": "node src/core/config/scripts/tagRelease.mjs"`. + +This eliminates all shell-substitution syntax from `package.json`. + +--- + +## Files to be modified + +- [Gruntfile.js](Gruntfile.js) — bulk of the work (sections 1a–1g) +- [package.json](package.json) — script changes (sections 2a–2c) +- New: [src/core/config/scripts/getHeapSize.mjs](src/core/config/scripts/getHeapSize.mjs) +- New: [src/core/config/scripts/bumpMinor.mjs](src/core/config/scripts/bumpMinor.mjs) and [src/core/config/scripts/tagRelease.mjs](src/core/config/scripts/tagRelease.mjs) (or one combined helper) + +## Reused utilities + +- Node stdlib `crypto.createHash`, `fs.cpSync`, `fs.rmSync`, `fs.readdirSync({ withFileTypes: true })`, `os.tmpdir()`, `path.join` — no new devDependencies +- Existing `grunt-exec` `cwd` option (already part of grunt-exec, no new dep) for the consumer-test exec tasks +- Existing `grunt.log.writeln` instead of `echo` +- Existing pattern from [src/core/config/scripts/newMinorVersion.mjs:50](src/core/config/scripts/newMinorVersion.mjs#L50) (uses `execSync` for git) — same pattern for new helpers + +## Verification + +Run on macOS first (current dev machine; quickest feedback loop): + +1. `rm -rf build/ node_modules/ && npm install` — confirms `postinstall` still passes +2. `npm test` — full test suite must pass (covers `configTests` → `exec:generateConfig` + `exec:generateNodeIndex`, renamed) +3. `npm run build` — exercises the prod task chain including the rewritten `calcDownloadHash`. Verify [build/prod/sha256digest.txt](build/prod/sha256digest.txt) exists and matches `shasum -a 256 build/prod/CyberChef_v*.zip`, and that [build/prod/index.html](build/prod/index.html) contains the hash (no `DOWNLOAD_HASH_PLACEHOLDER` left). +4. `npx grunt` (default lint task) — should print the `repoSize` output with the same shape as before +5. `npm run testnodeconsumer` — verify a tmp dir is created under `os.tmpdir()`, both consumer scripts succeed, and the tmp dir is cleaned up +6. `npm run getheapsize` — should print `node heap limit = … Mb` +7. `npm start` — confirms dev server still launches (no changes to it, but verifies nothing collateral broke) + +Then if a Windows machine is available, run the same 7 steps in PowerShell. If not available, the change is structurally safe because every shell-out has been eliminated; the residual risk is in the small set of `execSync("npm link …")` and `execSync("git ls-files")` calls, both of which are documented cross-platform.