feautre complete
This commit is contained in:
Andy L WSL
parent
f37e215c22
commit
cb896736a1
3
.env.template
Normal file
3
.env.template
Normal file
@ -0,0 +1,3 @@
|
||||
# CyberChef Cloud Operations Tests
|
||||
# Copy this file to `.env` and fill in your valid GCP API Key if you want to run the Nightwatch browser tests against live Cloud APIs.
|
||||
CYBERCHEF_GCP_TEST_KEY="YOUR_API_KEY_HERE"
|
||||
1
.gitignore
vendored
1
.gitignore
vendored
@ -13,3 +13,4 @@ src/node/index.mjs
|
||||
**/*.DS_Store
|
||||
tests/browser/output/*
|
||||
.node-version
|
||||
.env
|
||||
|
||||
@ -53,3 +53,35 @@ To ensure the CyberChef engine builds cleanly and passes its internal checks wit
|
||||
```bash
|
||||
npm run test
|
||||
```
|
||||
|
||||
## End-to-End Browser Testing (Nightwatch)
|
||||
|
||||
CyberChef is equipped with Nightwatch for end-to-end testing of features running in a real Chromium browser. This is extremely useful for Cloud Operations that cannot be easily tested within the headless NodeJS environment.
|
||||
|
||||
### Setting up API Keys
|
||||
|
||||
To prevent live API keys from being leaked in CI/CD, the Cloud API browser tests rely on local environment variables:
|
||||
1. Copy the `.env.template` file to `.env`:
|
||||
```bash
|
||||
cp .env.template .env
|
||||
```
|
||||
2. Open `.env` and replace `YOUR_API_KEY_HERE` with your actual Google Cloud API key for the `CYBERCHEF_GCP_TEST_KEY` variable. Ensure the API Key restrictions at console.cloud.google.com are set up to accept requests from `http://localhost:8080/*`.
|
||||
|
||||
### Running Nightwatch Tests
|
||||
|
||||
Once the `.env` file is prepared, you can trigger the entire browser suite:
|
||||
```bash
|
||||
npm run test:browser
|
||||
```
|
||||
|
||||
Or just the Cloud Operations specifically:
|
||||
```bash
|
||||
npx nightwatch tests/browser/03_cloud_ops.js
|
||||
```
|
||||
*Note: Make sure your local CyberChef dev server (`npm run start`) is currently running on `localhost:8080`, as Nightwatch tests require a live application target!*
|
||||
|
||||
#### Troubleshooting (WSL/Linux Environments)
|
||||
If your `nightwatch` tests immediately crash with ChromeDriver status `127` or `1`, your system may be missing Chromium's required graphical libraries. You can view exactly what is missing in `tests/browser/output/*_chromedriver.log`. You will typically need to install these packages on Ubuntu/Debian:
|
||||
```bash
|
||||
sudo apt update && sudo apt install -y libnss3 libatk1.0-0 libatk-bridge2.0-0 libcups2 libxkbcommon0 libxcomposite1 libxdamage1 libxrandr2 libgbm1 libpangocairo-1.0-0 libasound2
|
||||
```
|
||||
|
||||
40
docs/how to test with an LLM.md
Normal file
40
docs/how to test with an LLM.md
Normal file
@ -0,0 +1,40 @@
|
||||
# How to Test with an LLM
|
||||
|
||||
When developing agentic AI integrations—especially those interacting with Complex UIs and Cloud APIs—automated testing is the only way for the LLM to verify that its own code modifications are successful.
|
||||
|
||||
This document summarizes critical lessons learned during the implementation and verification of the Cloud API (Google Translate) integration using Nightwatch.js as the E2E verification framework. Following these guidelines ensures an LLM can quickly and reliably verify application outputs autonomously.
|
||||
|
||||
## 1. Headless Browser Environments
|
||||
Agents typically operate inside containers, WSL (Windows Subsystem for Linux), or headless virtual machines that lack graphical display servers (X11/Wayland).
|
||||
* **Actionable Advice**: Webdrivers (like ChromeDriver) must be configured explicitly to run without a GUI. Always inject the following Chrome options into the `nightwatch.json` configuration:
|
||||
* `--headless`
|
||||
* `--no-sandbox` (Critical for running as root in Docker/WSL)
|
||||
* `--disable-gpu`
|
||||
* `--disable-dev-shm-usage` (Prevents memory crashes in containerized `/dev/shm`)
|
||||
|
||||
## 2. Secure Credential Management for Agents
|
||||
E2E testing against live Cloud APIs requires real credentials, but these must never be committed to source control or logged in the agent's chat history.
|
||||
* **Actionable Advice**: Use the `dotenv` package.
|
||||
* Provide an `.env.template` so the human operator knows what variables to supply (e.g., `CYBERCHEF_GCP_TEST_KEY=YOUR_API_KEY_HERE`).
|
||||
* Ensure `.env` is strongly `.gitignore`d.
|
||||
* **Graceful Skips**: If the script detects the API key is missing or is exactly the template string, the test block should `return;` or gracefully skip rather than throw a hard failure. This allows public CI pipelines to continue passing.
|
||||
|
||||
## 3. UI Obfuscation vs Data Structures
|
||||
CyberChef provides UI arguments like `toggleString` which are specifically designed to mask sensitive inputs (like API keys turning into `****` on screen). However, changing a UI parameter type fundamentally changes the shape of the data passed to the backend functions.
|
||||
* **Actionable Advice**: An LLM might assume an argument is a plain string. If a UI masking attribute is introduced, the test framework *must* be updated to pass the correct object payload (e.g., passing `{ option: "UTF8", string: "AIzaSy..." }` instead of `"AIzaSy..."`), and the backend worker code must be updated to unpack `authStringObj.string` prior to network fetching.
|
||||
* *Failure to map the data object correctly results in Web Worker crashes or infinite hangs that the test framework cannot concisely diagnose.*
|
||||
|
||||
## 4. Bypassing Legacy Test Utilities
|
||||
Test framework abstractions (such as wrapper polling functions in `browserUtils.js`) can easily break or mask the root cause of an error. In our case, `expectOutput` attempted to check the length of an unresolved promise state, throwing a "Timeout Error" even though the UI had populated the data perfectly.
|
||||
* **Actionable Advice**: When an LLM is getting unhelpful "Timeout" errors despite visually verifying the UI is acting correctly, bypass the utility framework. Write raw Javascript to evaluate state inside the browser context, and return it to the Node runner for standard validation:
|
||||
```javascript
|
||||
browser.execute(function () {
|
||||
return window.app.manager.output.outputEditorView.state.doc.toString();
|
||||
}, [], function ({ value }) {
|
||||
browser.assert.equal(value, "Expected String");
|
||||
});
|
||||
```
|
||||
|
||||
## 5. Giving the LLM "Eyes"
|
||||
When an E2E test fails, the LLM only sees a terminal stack trace. It cannot see the state of the UI (e.g., did an error snackbar pop up? Did a preloader hang?).
|
||||
* **Actionable Advice**: Instruct the LLM to aggressively implement screenshot captures during test runs (`browser.saveScreenshot("tests/browser/output/debug.png");`), particularly right before an assertion is about to be checked. The LLM can then visually parse the exact state of the UI grid, inputs, outputs, and overlays, bridging the gap between a generic "TimeoutError" and a specific visual bug.
|
||||
@ -1,10 +1,12 @@
|
||||
{
|
||||
"src_folders": ["tests/browser"],
|
||||
"exclude": ["tests/browser/browserUtils.js"],
|
||||
"src_folders": [
|
||||
"tests/browser"
|
||||
],
|
||||
"exclude": [
|
||||
"tests/browser/browserUtils.js"
|
||||
],
|
||||
"output_folder": "tests/browser/output",
|
||||
|
||||
"test_settings": {
|
||||
|
||||
"default": {
|
||||
"launch_url": "http://localhost:8080",
|
||||
"webdriver": {
|
||||
@ -14,19 +16,23 @@
|
||||
"log_path": "tests/browser/output"
|
||||
},
|
||||
"desiredCapabilities": {
|
||||
"browserName": "chrome"
|
||||
"browserName": "chrome",
|
||||
"chromeOptions": {
|
||||
"args": [
|
||||
"--no-sandbox",
|
||||
"--headless",
|
||||
"--disable-gpu",
|
||||
"--disable-dev-shm-usage"
|
||||
]
|
||||
}
|
||||
},
|
||||
"enable_fail_fast": true
|
||||
},
|
||||
|
||||
"dev": {
|
||||
"launch_url": "http://localhost:8080"
|
||||
},
|
||||
|
||||
"prod": {
|
||||
"launch_url": "http://localhost:8000/index.html"
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
43
package-lock.json
generated
43
package-lock.json
generated
@ -123,7 +123,7 @@
|
||||
"babel-plugin-dynamic-import-node": "^2.3.3",
|
||||
"babel-plugin-transform-builtin-extend": "1.1.2",
|
||||
"base64-loader": "^1.0.0",
|
||||
"chromedriver": "^130.0.4",
|
||||
"chromedriver": "^145.0.6",
|
||||
"cli-progress": "^3.12.0",
|
||||
"colors": "^1.4.0",
|
||||
"compression-webpack-plugin": "^11.1.0",
|
||||
@ -131,6 +131,7 @@
|
||||
"core-js": "^3.48.0",
|
||||
"cspell": "^8.19.4",
|
||||
"css-loader": "7.1.4",
|
||||
"dotenv": "^17.3.1",
|
||||
"eslint": "^9.39.3",
|
||||
"eslint-plugin-jsdoc": "^48.11.0",
|
||||
"globals": "^15.15.0",
|
||||
@ -6377,28 +6378,35 @@
|
||||
}
|
||||
},
|
||||
"node_modules/chromedriver": {
|
||||
"version": "130.0.4",
|
||||
"resolved": "https://registry.npmjs.org/chromedriver/-/chromedriver-130.0.4.tgz",
|
||||
"integrity": "sha512-lpR+PWXszij1k4Ig3t338Zvll9HtCTiwoLM7n4pCCswALHxzmgwaaIFBh3rt9+5wRk9D07oFblrazrBxwaYYAQ==",
|
||||
"version": "145.0.6",
|
||||
"resolved": "https://registry.npmjs.org/chromedriver/-/chromedriver-145.0.6.tgz",
|
||||
"integrity": "sha512-qobFdfjk7G7U9GKB6RYGBuqQ8L0QG1M30p90sNIWLKdpeobhsedfBhVxRqT4m/nWAtM0PhNb9GDD9qzDwSSGlA==",
|
||||
"dev": true,
|
||||
"hasInstallScript": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@testim/chrome-version": "^1.1.4",
|
||||
"axios": "^1.7.4",
|
||||
"axios": "^1.13.5",
|
||||
"compare-versions": "^6.1.0",
|
||||
"extract-zip": "^2.0.1",
|
||||
"proxy-agent": "^6.4.0",
|
||||
"proxy-from-env": "^1.1.0",
|
||||
"proxy-from-env": "^2.0.0",
|
||||
"tcp-port-used": "^1.0.2"
|
||||
},
|
||||
"bin": {
|
||||
"chromedriver": "bin/chromedriver"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
"node": ">=20"
|
||||
}
|
||||
},
|
||||
"node_modules/chromedriver/node_modules/proxy-from-env": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/proxy-from-env/-/proxy-from-env-2.0.0.tgz",
|
||||
"integrity": "sha512-h2lD3OfRraP3R51rNFKIE8nX+qoLr1mE74X91YhVxtDbt+OD6ntoNZv56+JgI4RCdtwQ5eexsOk1KdOQDfvPCQ==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/ci-info": {
|
||||
"version": "3.3.0",
|
||||
"resolved": "https://registry.npmjs.org/ci-info/-/ci-info-3.3.0.tgz",
|
||||
@ -8436,16 +8444,16 @@
|
||||
}
|
||||
},
|
||||
"node_modules/dotenv": {
|
||||
"version": "16.3.1",
|
||||
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.3.1.tgz",
|
||||
"integrity": "sha512-IPzF4w4/Rd94bA9imS68tZBaYyBWSCE47V1RGuMrB94iyTOIEwRmVL2x/4An+6mETpLrKJ5hQkB8W4kFAadeIQ==",
|
||||
"version": "17.3.1",
|
||||
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-17.3.1.tgz",
|
||||
"integrity": "sha512-IO8C/dzEb6O3F9/twg6ZLXz164a2fhTnEWb95H23Dm4OuN+92NmEAlTrupP9VW6Jm3sO26tQlqyvyi4CsnY9GA==",
|
||||
"dev": true,
|
||||
"license": "BSD-2-Clause",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/motdotla/dotenv?sponsor=1"
|
||||
"url": "https://dotenvx.com"
|
||||
}
|
||||
},
|
||||
"node_modules/dunder-proto": {
|
||||
@ -13658,6 +13666,19 @@
|
||||
"url": "https://github.com/chalk/chalk?sponsor=1"
|
||||
}
|
||||
},
|
||||
"node_modules/nightwatch/node_modules/dotenv": {
|
||||
"version": "16.3.1",
|
||||
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.3.1.tgz",
|
||||
"integrity": "sha512-IPzF4w4/Rd94bA9imS68tZBaYyBWSCE47V1RGuMrB94iyTOIEwRmVL2x/4An+6mETpLrKJ5hQkB8W4kFAadeIQ==",
|
||||
"dev": true,
|
||||
"license": "BSD-2-Clause",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/motdotla/dotenv?sponsor=1"
|
||||
}
|
||||
},
|
||||
"node_modules/nightwatch/node_modules/glob": {
|
||||
"version": "7.2.3",
|
||||
"resolved": "https://registry.npmjs.org/glob/-/glob-7.2.3.tgz",
|
||||
|
||||
@ -55,7 +55,7 @@
|
||||
"babel-plugin-dynamic-import-node": "^2.3.3",
|
||||
"babel-plugin-transform-builtin-extend": "1.1.2",
|
||||
"base64-loader": "^1.0.0",
|
||||
"chromedriver": "^130.0.4",
|
||||
"chromedriver": "^145.0.6",
|
||||
"cli-progress": "^3.12.0",
|
||||
"colors": "^1.4.0",
|
||||
"compression-webpack-plugin": "^11.1.0",
|
||||
@ -63,6 +63,7 @@
|
||||
"core-js": "^3.48.0",
|
||||
"cspell": "^8.19.4",
|
||||
"css-loader": "7.1.4",
|
||||
"dotenv": "^17.3.1",
|
||||
"eslint": "^9.39.3",
|
||||
"eslint-plugin-jsdoc": "^48.11.0",
|
||||
"globals": "^15.15.0",
|
||||
|
||||
@ -48,8 +48,9 @@ class GoogleTranslate extends Operation {
|
||||
},
|
||||
{
|
||||
"name": "GCP Auth String",
|
||||
"type": "string",
|
||||
"value": ""
|
||||
"type": "toggleString",
|
||||
"value": "",
|
||||
"toggleValues": ["UTF8", "Latin1", "Base64", "Hex"]
|
||||
}
|
||||
];
|
||||
}
|
||||
@ -60,7 +61,8 @@ class GoogleTranslate extends Operation {
|
||||
* @returns {string}
|
||||
*/
|
||||
async run(input, args) {
|
||||
const [sourceLanguage, targetLanguage, authType, authString] = args;
|
||||
const [sourceLanguage, targetLanguage, authType, authStringObj] = args;
|
||||
const authString = typeof authStringObj === "string" ? authStringObj : (authStringObj.string || "");
|
||||
|
||||
if (input.length === 0) return "";
|
||||
if (!authString) throw new OperationError("Error: Please provide a valid GCP Auth String (API Key or OAuth Token).");
|
||||
|
||||
73
tests/browser/03_cloud_ops.js
Normal file
73
tests/browser/03_cloud_ops.js
Normal file
@ -0,0 +1,73 @@
|
||||
/**
|
||||
* End-to-end tests for Cloud Operations via Nightwatch.
|
||||
*
|
||||
* NOTE: Tests that execute real API calls will be skipped if the required API keys
|
||||
* are not found in the environment variables (e.g. running in a public CI pipeline).
|
||||
*
|
||||
* @author CyberChefCloud
|
||||
* @copyright Crown Copyright 2026
|
||||
* @license Apache-2.0
|
||||
*/
|
||||
|
||||
const browserUtils = require("./browserUtils.js");
|
||||
require('dotenv').config();
|
||||
|
||||
module.exports = {
|
||||
|
||||
before: browser => {
|
||||
browser
|
||||
.resizeWindow(1280, 800)
|
||||
.url(browser.launchUrl)
|
||||
.useCss()
|
||||
.waitForElementNotPresent("#preloader", 10000)
|
||||
.click("#auto-bake-label");
|
||||
},
|
||||
|
||||
"Google Translate: Missing Key Validation": function (browser) {
|
||||
browserUtils.loadRecipe(browser, "Google Translate", "Hello World", [
|
||||
"en",
|
||||
"es",
|
||||
"API Key",
|
||||
{ option: "UTF8", string: "" }
|
||||
]);
|
||||
|
||||
browser.waitForElementNotVisible("#snackbar-container", 6000);
|
||||
browserUtils.bake(browser);
|
||||
browser.pause(2000);
|
||||
browser.execute(function () {
|
||||
return window.app.manager.output.outputEditorView.state.doc.toString();
|
||||
}, [], function ({ value }) {
|
||||
browser.assert.ok(value.includes("Please provide a valid GCP Auth String"));
|
||||
});
|
||||
},
|
||||
|
||||
"Google Translate: Successful API Key Translation": function (browser) {
|
||||
const testKey = process.env.CYBERCHEF_GCP_TEST_KEY;
|
||||
|
||||
if (!testKey || testKey === "YOUR_API_KEY_HERE") {
|
||||
console.log("No valid CYBERCHEF_GCP_TEST_KEY found in environment variables. Skipping live API test.");
|
||||
return;
|
||||
}
|
||||
|
||||
browserUtils.loadRecipe(browser, "Google Translate", "Hello", [
|
||||
"en",
|
||||
"es",
|
||||
"API Key",
|
||||
{ option: "UTF8", string: testKey }
|
||||
]);
|
||||
|
||||
browser.waitForElementNotVisible("#snackbar-container", 6000);
|
||||
browserUtils.bake(browser);
|
||||
browser.pause(2000);
|
||||
browser.saveScreenshot("tests/browser/output/success_debug.png");
|
||||
browser.execute(function () {
|
||||
return window.app.manager.output.outputEditorView.state.doc.toString();
|
||||
}, [], function ({ value }) {
|
||||
browser.assert.equal(value, "Hola");
|
||||
});
|
||||
},
|
||||
|
||||
after: function (browser) {
|
||||
browser.end();
|
||||
}
|
||||
};
|
||||
@ -185,10 +185,11 @@ function loadRecipe(browser, opName, input, args) {
|
||||
function expectOutput(browser, expected, waitNotNull = false, waitWindow = 1000) {
|
||||
if (waitNotNull && expected !== "") {
|
||||
browser.waitUntil(async function () {
|
||||
const output = await this.execute(function() {
|
||||
const result = await this.execute(function () {
|
||||
return window.app.manager.output.outputEditorView.state.doc.toString();
|
||||
});
|
||||
return output.length;
|
||||
const output = result && result.value !== undefined ? result.value : result;
|
||||
return typeof output === "string" && output.length > 0;
|
||||
}, waitWindow);
|
||||
}
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user