exlim/cyberchef
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'master' of https://github.com/gchq/CyberChef.wiki into node-lib

d98762625 2019-05-03 14:03:16 +01:00
commit 50dc2317fa
7 changed files with 468 additions and 103 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

121
Adding-a-new-operation.md

@ -1,113 +1,31 @@
## How to add an operation
1. Create a new file in the `src/core/operations` directory and name it using CamelCase. e.g. `MyOperation.js`
2. In this file, create a namespace with the same name and populate it with a single function looking like this (all function and variable names should be written in camelCase):
```javascript
const MyOperation = {
runMyOperation: function (input, args) {
return input;
}
};
export default MyOperation;
```
- `input` will be the input data passed on from the previous operation (or the data entered by the user if yours is the first operation). Its data type is specified in the next step by `inputType`.
- `args` will be an array of the arguments for your operation. They are specified in the next step by `args`.
- Make sure that you return the output data in the format specified in the next step by `outputType`.
3. Choose which module to add it to. This decision should be based on how much extra code your operation will add to the app, including any dependencies it imports. If it doesn't require any dependencies, add it to the 'Default' module in `src/core/config/modules/Default.js`. Import it at the top of the file:
```javascript
import MyOperation from "../../operations/MyOperation.js";
```
and then add it to the operation list like so:
```javascript
"My Operation": MyOperation.runMyOperation, // a reference to the function that runs your operation
```
If it imports the same dependencies as other operations, add it to the relevant existing module. If it imports entirely new dependencies that are not related to other operations in any way, create a new module using an existing module as a template and then import this new module into the `src/core/config/modules/OpModules.js` file.
4. In `src/core/config/OperationConfig.js`, import your operation at the top of the file:
```javascript
import MyOperation from "../operations/MyOperation.js";
```
Then create a new entry:
```javascript
"My Operation": {
module: "Module name",
description: "A short description if necessary, optionally containing HTML code (e.g. lists and paragraphs)",
inputType: "byteArray", // the input type for your operation, see the next section for valid types
outputType: "byteArray", // the output type for your operation, see the next section for valid types
highlight: true, // [optional] true if the operation does not change the position of bytes in the output (so that highlighting can be calculated)
highlightReverse: true, // [optional] same as above but for the reverse of the operation (output to input highlighting)
manualBake: false, // [optional] true if auto-bake should be disabled when this operation is added to the recipe
args: [ // A list of the arguments that the user will be presented with
{
name: "Argument name",
type: "string", // the argument data type, see the next section for valid types
value: MyOperation.DEFAULT_VALUE // the default value of the argument
}
]
}
```
The easiest way to create a new operation is to use the provided quickstart script. This can be run using the command `npm run newop`. This script will walk you through the configuration process and create your operation file in the `src/core/operations` directory.
For example:
```javascript
"XOR": {
module: "Default",
description: "XOR the input with the given key, provided as either a hex or ASCII string.<br>e.g. fe023da5<br><br><b>Options</b><br><u>Null preserving:</u> If the current byte is 0x00 or the same as the key, skip it.<br><br><u>Differential:</u> Set the key to the value of the previously decoded byte.",
inputType: "byteArray",
outputType: "byteArray",
args: [
{
name: "Key",
type: "binaryString",
value: ""
},
{
name: "Key format",
type: "option",
value: BitwiseOp.KEY_FORMAT
},
{
name: "Null preserving",
type: "boolean",
value: BitwiseOp.XOR_PRESERVE_NULLS
},
{
name: "Differential",
type: "boolean",
value: BitwiseOp.XOR_DIFFERENTIAL
}
]
}
```
Once this file has been created, add your operation to the [`src/core/config/Categories.json`](https://github.com/gchq/CyberChef/blob/master/src/core/config/Categories.json) file. This determines which menu it will appear in. You can add it to multiple menus if you feel it is appropriate.
5. In `src/core/config/Categories.js`, add your operation name to an appropriate list. This determines which menu it will appear in. You can add it to multiple menus if you feel it is appropriate.
6. Finally, run `grunt dev` if you haven't already. If it's already running, it should automatically build a development version when you save the files.
7. You should now be able to view your operation on the site by browsing to [`localhost:8080`](http://localhost:8080).
8. You can write whatever code you like as long as it is encapsulated within the namespace you created (`MyOperation`). Take a look at `src/core/operations/Entropy.js` for a good example.
9. You may find it useful to use some helper functions which have been written in `src/core/Utils.js`. These are available in the `Utils` object (e.g. `Utils.strToByteArray("Hello")` returns `[72,101,108,108,111]`).
Finally, run `grunt dev` if you haven't already. If it's already running, it should automatically build a development version when you save the files. You should now be able to view your operation on the site by browsing to [`localhost:8080`](http://localhost:8080).
You can write whatever code you like as long as it is encapsulated within the object you created. Take a look at [`src/core/operations/Entropy.mjs`](https://github.com/gchq/CyberChef/blob/master/src/core/operations/Entropy.mjs) for a good example.
You may find it useful to use some helper functions which have been written in [`src/core/Utils.mjs`](https://github.com/gchq/CyberChef/blob/master/src/core/Utils.mjs) (e.g. `Utils.strToByteArray("Hello")` returns `[72,101,108,108,111]`).
## Data types
**Input and Output**
Five data types are supported for the input and output of operations:
Nine data types are supported for the input and output of operations:
1. `string` - e.g. `"hello"`
2. `byteArray` - e.g. `[104,101,108,108,111]`
3. `number` - e.g. `562` or `3.14159265`
4. `html` - e.g. `"<p>hello</p>"`
5. [`ArrayBuffer`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer) - e.g. `new Uint8Array([104,101,108,108,111]).buffer`
6. `BigNumber` - e.g. `12345678901234567890`
7. `JSON` - e.g. `[{"a":1,"b":2}]`
8. [`File`](https://developer.mozilla.org/en-US/docs/Web/API/File) - e.g. `new File()`
9. `List<File>` - e.g. `[new File(), new File()]`
Each operation can define any of these data types as their input or output. The data will be automatically converted to the specified type before running the operation.
@ -132,14 +50,27 @@ Operation arguments (ingredients) can be set to any of the following types:
- User is presented with a checkbox, operation receives `true` or `false`.
7. `option`
- Given an array of strings, the user is presented with a dropdown selection box with each of those strings as an option. The selected string is sent to the operation.
- You can use the `defaultIndex` property to define which index should be selected by default, if required.
7. `populateOption`
- Given an array of `{name: "", value: ""}` objects, the user is presented with a dropdown selection box with the names as options. The corresponding value will be assigned to whichever argument index the `target` parameter is set to.
- See the *Regular expression* configuration in `src/core/config/OperationConfig.js` for an example of how this works.
8. `editableOption`
8. `editableOption` or `editableOptionShort`
- Given an array of `{name: "", value: ""}` objects, the user is presented with an editable dropdown menu. The items in the dropdown are labelled with `name` and set the argument to `value` when selected.
- You can use the `defaultIndex` property to define which index should be selected by default, if required.
9. `toggleString`
- User is presented with a string input box with a toggleable dropdown attached.
- Populate the dropdown using the `toggleValues` property.
- Operation receives an object with two properties: `option` containing the user's dropdown selection, and `string` containing the input box contents.
- Particularly useful for arguments that can be specified in various different formats.
- See the *XOR* configuration in `src/core/config/OperationConfig.js` for an example of how this works.
## Presenting complex data
The output of your operation will be passed on to the next operation in the recipe, or to the Output field if it is the final operation. If your operation has a complex output, it should be presented to the user in a friendly format, perhaps using HTML markup, however this format should not be sent to follow-on operations, as it would make onward processing unnecessarily complex.
In these situations, the `present` function should be used. This function is called if your operation is the final operation in the recipe. It is passed the output of your `run` function which you can then manipulate into a suitable format for displaying to the user. This allows you to return a sensible format which can be easily processed from your `run` function.
The data type for your present function should be specified in the operation constructor using `this.presentType`.
A good example of this can be found in [`src/core/operations/Unzip.mjs`](https://github.com/gchq/CyberChef/blob/master/src/core/operations/Unzip.mjs).

71
Automatic-detection-of-encoded-data-using-CyberChef-Magic.md Normal file

@ -0,0 +1,71 @@
CyberChef v8 introduces the ['Magic'](https://gchq.github.io/CyberChef/#recipe=Magic()) operation, designed to automatically detect how your data is encoded and which operations can be used to decode it. A number of methods are used to achieve this.
### Pattern matching
Many common data encoding schemes, such as Base64, Hexadecimal and Gzip, have predictable structures that can be detected using pattern matching techniques. Regular expressions have been written for all operations where this is the case. They are each run over the data and any matches are recorded. In some cases, multiple regular expressions are written for the same operation where different arguments can be applied, for example Base64 using a non-standard alphabet.
Example regular expression for Base64 data using the standard alphabet:
```regex
/^(?:[A-Z\\d+/]{4})+(?:[A-Z\\d+/]{2}==|[A-Z\\d+/]{3}=)?$/i
```
Example regular expression for Base64 data using the y64 alphabet:
```regex
/^(?:[A-Z\\d._]{4}){5,}(?:[A-Z\\d._]{2}--|[A-Z\\d._]{3}-)?$/i
```
### Speculative execution
For every pattern that matches, the corresponding operation is speculatively executed to determine what the output looks like. Various metrics are collected for each of these possible branches to determine whether they look like valid data or not. Each branch is also checked for further pattern matches, meaning that data under multiple levels of encoding can be unwrapped recursively. The maximum number of levels of recursion is controlled by the 'Depth' argument.
The methods used to detect how "valid" the data looks are as follows, ranging from simple to more complex techniques:
#### Magic byte detection
In many file formats, a [magic number](https://en.wikipedia.org/wiki/List_of_file_signatures) is included to allow trivial detection of the file type. If a magic byte sequence is found in a branch, it increases the likelihood that the correct decoding sequence has been found.
![Gzip detection](https://user-images.githubusercontent.com/22770796/43699009-b52eae5a-9944-11e8-9c60-a648be5e1788.png)
#### UTF-8 detection
UTF-8 data has a well-defined structure which can be easily tested for. The presence of valid UTF-8 data may suggest that a valid decoding sequence has been found.
#### Entropy measurement
[Shannon Entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory)), in the context of information theory, is a measure of the rate at which information is produced by a source of data. It can be used, in a broad sense, to detect whether data is likely to be structured or unstructured. If a branch results in data with high entropy, it is possible that it has simply output unstructured, random garbage. Branches resulting in lower entropy data are ranked higher as they are more likely to contain repeating structures.
#### Byte frequency analysis
On average, the English language contains more "e"s than any other letter. In fact, given a long enough sample, the relative frequency of each character is very predictable, to the extent that we can consider any text not roughly matching these frequencies as unlikely to be English.
![English letter frequencies](https://user-images.githubusercontent.com/22770796/43697173-2253822c-993a-11e8-9ced-b567b5eea61a.png)
This set of frequencies can be expanded to include all possible bytes, incorporating punctuation, numbers, symbols and other formatting characters. To generate a set of accurate "truth data", the [English language Wikipedia dump](https://dumps.wikimedia.org/enwiki/) was downloaded, wiki syntax was stripped out, then the byte frequencies were calculated. The resulting values assume a character encoding of UTF-8.
For every branch created by the Magic operation, the byte frequencies for the output are calculated and then compared to this truth data using [Pearson's chi-squared goodness of fit test](https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test). This process tells us how closely the branch's output matches the English language and therefore hopefully gives us an idea of how likely it is that we have found correctly decoded data, if that data includes a reasonably high proportion of English text.
Truth data was also generated for all other languages supported by Wikipedia. By default, only the top 38 languages are checked (based on the most popular languages used on the Internet, as listed on [W3 Techs](https://w3techs.com/technologies/overview/content_language/all)), however if 'Extensive language support' is selected, all 245 languages are supported.
![Georgian language detection](https://user-images.githubusercontent.com/22770796/43700059-24afe498-9949-11e8-8ce4-ce4c79863f49.png)
### Intensive mode
The above methods have been optimised to run reasonably quickly over most types of input, however there are some methods which take considerably longer to run due to their high branching factor. These can be turned on by enabling the 'Intensive mode' argument.
#### Character encoding brute forcing
For each branch, the data is converted into a number of different character encodings. If this conversion results in different data, a new branch is created and metrics are calculated as described above. This can help to detect the correct character encodings for [mojibake](https://en.wikipedia.org/wiki/Mojibake) (garbled data represented in the wrong encoding). Over 40 character encodings are currently supported.
![Mojibake detection](https://user-images.githubusercontent.com/22770796/43699979-e84e070a-9948-11e8-82fd-3bbaf98be75b.png)
#### Arithmetic logic brute forcing
Single byte XORs are carried out over every branch, creating 255 further branches to analyse. Bit rotates are also calculated, resulting in another 7 branches.
![Single byte XOR detection](https://user-images.githubusercontent.com/22770796/43699531-0f2fb3de-9947-11e8-90be-16778a29973c.png)
### Automated background magic
As well as being available as a standalone operation, CyberChef runs the 'Magic' operation automatically in a background thread whenever the Output is changed. If it manages to find an operation or set of operations that can help decode the data, the magic icon will be displayed in the Output pane. Hovering over this icon shows which operations are most likely to help and a snippet of what they will produce. Clicking the icon will append those operations to your recipe.
![Automated magic](https://user-images.githubusercontent.com/22770796/43699791-19b77f2a-9948-11e8-87d6-8d822d528615.png)

2
Contributing.md

@ -26,7 +26,7 @@ Before your contributions can be accepted, you must:
4. Use Vanilla JS if at all possible to reduce the number of libraries required and relied upon. Frameworks like jQuery, although included, should not be used unless absolutely necessary.
With these principals in mind, any changes or additions to CyberChef should keep it:
With these principles in mind, any changes or additions to CyberChef should keep it:
- Standalone
- Efficient

350
Enigma,-the-Bombe,-and-Typex.md Normal file

@ -0,0 +1,350 @@
## How to guides
### How to encrypt/decrypt with Enigma
We'll start with a step-by-step guide to decrypting a known message. You can see the result of
these steps in CyberChef
[here](https://gchq.github.io/CyberChef/#recipe=Enigma('3-rotor','','','','BDFHJLCPRTXVZNYEIWGAKMUSQO<W','K','O','AJDKSIRUXBLHWTMCQGZNPYFVOE<F','N','P','ESOVPZJAYQUIRHXLNFTGKDCMWB<K','G','M','AY%20BR%20CU%20DH%20EQ%20FS%20GL%20IP%20JX%20KN%20MO%20TZ%20VW','AH%20CO%20DE%20GZ%20IJ%20KM%20LQ%20NY%20PS%20TW',true)&input=WFRTWU4gV0FFVUcgRVpBTFkgTlJRSU0gQU1MWlggTUZVT0QgQVdYTFkgTFpDVVogUU9RQlEgSkxDUEsgTkREUlcgRg).
Let's say that our message is as follows:
`XTSYN WAEUG EZALY NRQIM AMLZX MFUOD AWXLY LZCUZ QOQBQ JLCPK NDDRW F`
And that we've been told that a German service Enigma is in use with the following settings:
Rotors `III`, `II`, and `IV`, reflector `B`, ring settings (Ringstellung in German) `KNG`,
plugboard (Steckerbrett)`AH CO DE GZ IJ KM LQ NY PS TW`, and finally the rotors are set to `OPM`.
Enigma settings are generally given left-to-right. Therefore, you should ensure the 3-rotor Enigma
is selected in the first dropdown menu, and then use the dropdown menus to put rotor
`III` in the 1st rotor slot, `II` in the 2nd, and `IV` in the 3rd, and pick
`B` in the reflector slot. In the ring setting and initial value boxes for the 1st rotor, put `K`
and `O` respectively, `N` and `P` in the 2nd, and `G` and `M` in the 3rd. Copy the plugboard
settings `AH CO DE GZ IJ KM LQ NY PS TW` into the plugboard box. Finally, paste the message into
the input window.
The output window will now read as follows:
`HELLO CYBER CHEFU SERST HISIS ATEST MESSA GEFOR THEDO CUMEN TATIO N`
The Enigma machine doesn't support any special characters, so there's no support for spaces, and
by default unsupported characters are removed and output is put into the traditional five-character
groups. (You can turn this off by disabling "strict input".) In some messages you may see X used to
represent space.
Encrypting with Enigma is exactly the same as decrypting - if you copy the decrypted message back
into the input box with the same recipe, you'll get the original ciphertext back.
#### Plugboard, rotor and reflector specifications
The plugboard exchanges pairs of letters, and is specified as a space-separated list of those
pairs. For example, with the plugboard `AB CD`, `A` will be exchanged for `B` and vice versa, `C`
for `D`, and so forth. Letters that aren't specified are not exchanged, but you can also specify,
for example, `AA` to note that `A` is not exchanged. A letter cannot be exchanged more than once.
In standard late-war German military operating practice, ten pairs were used.
You can enter your own components, rather than using the standard ones. A rotor is an arbitrary
mapping between letters - the rotor specification used here is the letters the rotor maps A through
Z to, so for example with the rotor `ESOVPZJAYQUIRHXLNFTGKDCMWB`, `A` maps to `E`, `B` to `S`, and
so forth. Each letter must appear exactly once. Additionally, rotors have a defined step point (the
point or points in the rotor's rotation at which the neighbouring rotor is stepped) - these are
specified using a `<` followed by the letters at which the step happens.
Reflectors, like the plugboard, exchange pairs of letters, so they are entered the same way.
However, letters cannot map to themselves.
### How to encrypt/decrypt with Typex
The Typex machine is very similar to Enigma. There are a few important differences from a user
perspective:
* Five rotors are used.
* Rotor wirings cores can be inserted into the rotors backwards.
* The input plugboard (on models which had one) is more complicated, allowing arbitrary letter
mappings, which means it functions like, and is entered like, a rotor.
* There was an additional plugboard which allowed rewiring of the reflector: this is supported by
simply editing the specified reflector.
Like Enigma, Typex only supports enciphering/deciphering the letters A-Z. However, the keyboard was
marked up with a standardised way of representing numbers and symbols using only the letters. You
can enable emulation of these keyboard modes in the operation configuration. Note that this needs
to know whether the message is being encrypted or decrypted.
### How to attack Enigma using the Bombe
Let's take the message from the first example, and try and decrypt it without knowing the settings
in advance. Here's the message again:
`XTSYN WAEUG EZALY NRQIM AMLZX MFUOD AWXLY LZCUZ QOQBQ JLCPK NDDRW F`
Let's assume to start with that we know the rotors used were `III`, `II`, and `IV`, and reflector
`B`, but that we know no other settings. Put the ciphertext in the input window and the Bombe
operation in your recipe, and choose the correct rotors and reflector. We need one additional piece
of information to attack the message: a "crib". This is a section of known plaintext for the
message. If we know something about what the message is likely to contain, we can guess possible
cribs.
We can also eliminate some cribs by using the property that Enigma cannot encipher a letter as
itself. For example, let's say our first guess for a crib is that the message begins with "Hello
world". If we enter `HELLO WORLD` into the crib box, it will inform us that the crib is invalid,
as the `W` in `HELLO WORLD` corresponds to a `W` in the ciphertext. (Note that spaces in the input
and crib are ignored - they're included here for readability.) You can see this in CyberChef
[here](https://gchq.github.io/CyberChef/#recipe=Bombe('3-rotor','LEYJVCNIXWPBQMDRTAKZGFUHOS','BDFHJLCPRTXVZNYEIWGAKMUSQO<W','AJDKSIRUXBLHWTMCQGZNPYFVOE<F','ESOVPZJAYQUIRHXLNFTGKDCMWB<K','AY%20BR%20CU%20DH%20EQ%20FS%20GL%20IP%20JX%20KN%20MO%20TZ%20VW','HELLO%20WORLD',0,true)&input=WFRTWU4gV0FFVUcgRVpBTFkgTlJRSU0gQU1MWlggTUZVT0QgQVdYTFkgTFpDVVogUU9RQlEgSkxDUEsgTkREUlcgRg)
Let's try "Hello CyberChef" as a crib instead. If we enter `HELLO CYBER CHEF`, the operation
will run and we'll be presented with some information about the run, followed by a list of stops.
You can see this
[here](https://gchq.github.io/CyberChef/#recipe=Bombe('3-rotor','LEYJVCNIXWPBQMDRTAKZGFUHOS','BDFHJLCPRTXVZNYEIWGAKMUSQO<W','AJDKSIRUXBLHWTMCQGZNPYFVOE<F','ESOVPZJAYQUIRHXLNFTGKDCMWB<K','AY%20BR%20CU%20DH%20EQ%20FS%20GL%20IP%20JX%20KN%20MO%20TZ%20VW','HELLO%20CYBER%20CHEF',0,true)&input=WFRTWU4gV0FFVUcgRVpBTFkgTlJRSU0gQU1MWlggTUZVT0QgQVdYTFkgTFpDVVogUU9RQlEgSkxDUEsgTkREUlcgRg).
Here you'll notice that it says `Bombe run on menu with 0 loops (2+ desirable).`, and there are a
large number of stops listed. The menu is built from the crib you've entered, and is a web linking
ciphertext and plaintext letters. (If you're maths inclined, this is a graph where letters - plain
or ciphertext - are nodes and states of the Enigma machine are edges.) The machine performs better
on menus which have loops in them - a letter maps to another to another and eventually returns to
the first - and additionally on longer menus. However, menus that are too long risk failing because
the Bombe doesn't simulate the middle rotor stepping, and the longer the menu the more likely this
is to have happened. Getting a good menu is a mixture of art and luck, and you may have to try a
number of possible cribs before you get one that will produce useful results.
<img src="https://user-images.githubusercontent.com/22770796/54355102-285ca900-4650-11e9-823a-c001cd2cf99c.png" width=500 alt="Bombe menu diagram">
In this case, if we extend our crib by a single character to `HELLO CYBER CHEFU`, we get a loop in
the menu (that `U` maps to a `Y` in the ciphertext, the `Y` in the second cipher block maps to
`A`, the `A` in the third ciphertext block maps to `E`, and the `E` in the second crib block maps
back to `U`). We immediately get a manageable number of results. You can see this
[here](https://gchq.github.io/CyberChef/#recipe=Bombe('3-rotor','LEYJVCNIXWPBQMDRTAKZGFUHOS','BDFHJLCPRTXVZNYEIWGAKMUSQO<W','AJDKSIRUXBLHWTMCQGZNPYFVOE<F','ESOVPZJAYQUIRHXLNFTGKDCMWB<K','AY%20BR%20CU%20DH%20EQ%20FS%20GL%20IP%20JX%20KN%20MO%20TZ%20VW','HELLO%20CYBER%20CHEFU',0,true)&input=WFRTWU4gV0FFVUcgRVpBTFkgTlJRSU0gQU1MWlggTUZVT0QgQVdYTFkgTFpDVVogUU9RQlEgSkxDUEsgTkREUlcgRg).
Each result gives a set of rotor initial values and a set of identified plugboard wirings.
Extending the crib further to `HELLO CYBER CHEFU SER` produces a single result, and it has also
recovered eight of the ten plugboard wires and identified four of the six letters which are not
wired. You can see this
[here](https://gchq.github.io/CyberChef/#recipe=Bombe('3-rotor','LEYJVCNIXWPBQMDRTAKZGFUHOS','BDFHJLCPRTXVZNYEIWGAKMUSQO<W','AJDKSIRUXBLHWTMCQGZNPYFVOE<F','ESOVPZJAYQUIRHXLNFTGKDCMWB<K','AY%20BR%20CU%20DH%20EQ%20FS%20GL%20IP%20JX%20KN%20MO%20TZ%20VW','HELLO%20CYBER%20CHEFU%20SER',0,true)&input=WFRTWU4gV0FFVUcgRVpBTFkgTlJRSU0gQU1MWlggTUZVT0QgQVdYTFkgTFpDVVogUU9RQlEgSkxDUEsgTkREUlcgRg).
We now have two things left to do:
1. Recover the remaining plugboard settings.
1. Recover the ring settings.
This will need to be done manually.
Set up an Enigma operation with these settings. Leave the ring positions set to `A` for the moment,
so from top to bottom we have rotor `III` at initial value `E`, rotor `II` at `C`, and rotor `IV`
at `G`, reflector `B`, and plugboard `DE AH BB CO FF GZ LQ NY PS RR TW UU`.
You can see this
[here](https://gchq.github.io/CyberChef/#recipe=Enigma('3-rotor','LEYJVCNIXWPBQMDRTAKZGFUHOS','A','A','BDFHJLCPRTXVZNYEIWGAKMUSQO<W','A','E','AJDKSIRUXBLHWTMCQGZNPYFVOE<F','A','C','ESOVPZJAYQUIRHXLNFTGKDCMWB<K','A','G','AY%20BR%20CU%20DH%20EQ%20FS%20GL%20IP%20JX%20KN%20MO%20TZ%20VW','DE%20AH%20BB%20CO%20FF%20GZ%20LQ%20NY%20PS%20RR%20TW%20UU',true)&input=WFRTWU4gV0FFVUcgRVpBTFkgTlJRSU0gQU1MWlggTUZVT0QgQVdYTFkgTFpDVVogUU9RQlEgSkxDUEsgTkREUlcgRg).
You will immediately notice that the output is not the same as the decryption preview from the
Bombe operation! Only the first three characters - `HEL` - decrypt correctly. This is because the
middle rotor stepping was ignored by the Bombe. You can correct this by adjusting the ring position
and initial value on the right-hand rotor in sync. They are currently `A` and `G` respectively.
Advance both by one to `B` and `H`, and you'll find that now only the first two characters decrypt
correctly.
Keep trying settings until most of the message is legible. You won't be able to get the whole
message correct, but for example at `F` and `L`, which you can see
[here](https://gchq.github.io/CyberChef/#recipe=Enigma('3-rotor','LEYJVCNIXWPBQMDRTAKZGFUHOS','A','A','BDFHJLCPRTXVZNYEIWGAKMUSQO<W','A','E','AJDKSIRUXBLHWTMCQGZNPYFVOE<F','A','C','ESOVPZJAYQUIRHXLNFTGKDCMWB<K','F','L','AY%20BR%20CU%20DH%20EQ%20FS%20GL%20IP%20JX%20KN%20MO%20TZ%20VW','DE%20AH%20BB%20CO%20FF%20GZ%20LQ%20NY%20PS%20RR%20TW%20UU',true)&input=WFRTWU4gV0FFVUcgRVpBTFkgTlJRSU0gQU1MWlggTUZVT0QgQVdYTFkgTFpDVVogUU9RQlEgSkxDUEsgTkREUlcgRg),
our message now looks like:
`HELLO CYBER CHEFU SERTC HVSJS QTEST KESSA GEFOR THEDO VUKEB TKMZM T`
At this point we can recover the remaining plugboard settings. The only letters which are not known
in the plugboard are `J K V X M I`, of which two will be unconnected and two pairs connected. By
inspecting the ciphertext and partially decrypted plaintext and trying pairs, we find that
connecting `IJ` and `KM` results, as you can see
[here](https://gchq.github.io/CyberChef/#recipe=Enigma('3-rotor','LEYJVCNIXWPBQMDRTAKZGFUHOS','A','A','BDFHJLCPRTXVZNYEIWGAKMUSQO<W','A','E','AJDKSIRUXBLHWTMCQGZNPYFVOE<F','A','C','ESOVPZJAYQUIRHXLNFTGKDCMWB<K','F','L','AY%20BR%20CU%20DH%20EQ%20FS%20GL%20IP%20JX%20KN%20MO%20TZ%20VW','DE%20AH%20BB%20CO%20FF%20GZ%20LQ%20NY%20PS%20RR%20TW%20UU%20IJ%20KM',true)&input=WFRTWU4gV0FFVUcgRVpBTFkgTlJRSU0gQU1MWlggTUZVT0QgQVdYTFkgTFpDVVogUU9RQlEgSkxDUEsgTkREUlcgRg),
in:
`HELLO CYBER CHEFU SERST HISIS ATEST MESSA GEFOR THEDO CUMEO TMKZK T`
This is looking pretty good. We can now fine tune our ring settings. Adjusting the right-hand rotor to
`G` and `M` gives, as you can see
[here](https://gchq.github.io/CyberChef/#recipe=Enigma('3-rotor','LEYJVCNIXWPBQMDRTAKZGFUHOS','A','A','BDFHJLCPRTXVZNYEIWGAKMUSQO<W','A','E','AJDKSIRUXBLHWTMCQGZNPYFVOE<F','A','C','ESOVPZJAYQUIRHXLNFTGKDCMWB<K','G','M','AY%20BR%20CU%20DH%20EQ%20FS%20GL%20IP%20JX%20KN%20MO%20TZ%20VW','DE%20AH%20BB%20CO%20FF%20GZ%20LQ%20NY%20PS%20RR%20TW%20UU%20IJ%20KM',true)&input=WFRTWU4gV0FFVUcgRVpBTFkgTlJRSU0gQU1MWlggTUZVT0QgQVdYTFkgTFpDVVogUU9RQlEgSkxDUEsgTkREUlcgRg),
`HELLO CYBER CHEFU SERST HISIS ATEST MESSA GEFOR THEDO CUMEN WMKZK T`
which is the best we can get with only adjustments to the first rotor. You now need to adjust the
second rotor. Here, you'll find that anything from `D` and `F` to `Z` and `B` gives the correct
decryption, for example
[here](https://gchq.github.io/CyberChef/#recipe=Enigma('3-rotor','LEYJVCNIXWPBQMDRTAKZGFUHOS','A','A','BDFHJLCPRTXVZNYEIWGAKMUSQO<W','A','E','AJDKSIRUXBLHWTMCQGZNPYFVOE<F','D','F','ESOVPZJAYQUIRHXLNFTGKDCMWB<K','G','M','AY%20BR%20CU%20DH%20EQ%20FS%20GL%20IP%20JX%20KN%20MO%20TZ%20VW','DE%20AH%20BB%20CO%20FF%20GZ%20LQ%20NY%20PS%20RR%20TW%20UU%20IJ%20KM',true)&input=WFRTWU4gV0FFVUcgRVpBTFkgTlJRSU0gQU1MWlggTUZVT0QgQVdYTFkgTFpDVVogUU9RQlEgSkxDUEsgTkREUlcgRg).
It's not possible to determine the exact original settings from only this message. In practice,
for the real Enigma and real Bombe, this step was achieved via methods that exploited the Enigma
network operating procedures, but this is beyond the scope of this document.
#### What if I don't know the rotors?
You'll need the "Multiple Bombe" operation for this. You can define a set of rotors to choose
from - the standard WW2 German military Enigma configurations are provided or you can define your
own - and it'll run the Bombe against every possible combination. This will take up to a few hours
for an attack against every possible configuration of the four-rotor Naval Enigma! You should run
a single Bombe first to make sure your menu is good before attempting a multi-Bombe run.
You can see an example of using the Multiple Bombe operation to attack the above example message
without knowing the rotor order in advance
[here](https://gchq.github.io/CyberChef/#recipe=Multiple_Bombe('German%20Service%20Enigma%20(First%20-%203%20rotor)','EKMFLGDQVZNTOWYHXUSPAIBRCJ<R%5CnAJDKSIRUXBLHWTMCQGZNPYFVOE<F%5CnBDFHJLCPRTXVZNYEIWGAKMUSQO<W%5CnESOVPZJAYQUIRHXLNFTGKDCMWB<K%5CnVZBRGITYUPSDNHLXAWMJQOFECK<A','','AY%20BR%20CU%20DH%20EQ%20FS%20GL%20IP%20JX%20KN%20MO%20TZ%20VW','HELLO%20CYBER%20CHEFU%20SER',0,true)&input=WFRTWU4gV0FFVUcgRVpBTFkgTlJRSU0gQU1MWlggTUZVT0QgQVdYTFkgTFpDVVogUU9RQlEgSkxDUEsgTkREUlcgRg).
#### What if I get far too many stops?
Use a longer or different crib. Try to find one that produces loops in the menu.
#### What if I get no stops, or only incorrect stops?
Are you sure your crib is correct? Try alternative cribs.
#### What if I know my crib is right, but I still don't get any stops?
The middle rotor has probably stepped during the encipherment of your crib. Try a shorter or
different crib.
## How things work
### How Enigma works
We won't go into the full history of Enigma and all its variants here, but as a brief overview of
how the machine works:
Enigma uses a series of letter-\>letter conversions to produce ciphertext from plaintext. It
is symmetric, such that the same series of operations on the ciphertext recovers the original
plaintext.
The bulk of the conversions are implemented in "rotors", which are just an arbitrary mapping from
the letters A-Z to the same letters in a different order. Additionally, to enforce the symmetry, a
reflector is used, which is a symmetric paired mapping of letters (that is, if a given reflector
maps X to Y, the converse is also true). These are combined such that a letter is mapped through
three different rotors, the reflector, and then back through the same three rotors in reverse.
To avoid Enigma being a simple [Caesar cipher](https://wikipedia.org/wiki/Caesar_cipher), the
rotors rotate (or "step") between enciphering letters, changing the effective mappings. The right
rotor steps on every letter, and additionally defines a letter (or
later, letters) at which the adjacent (middle) rotor will be stepped. Likewise, the middle rotor
defines a point at which the left rotor steps. (A mechanical issue known as the
double-stepping anomaly means that the middle rotor actually steps twice when the left hand rotor
steps.)
The German military Enigma adds a plugboard, which is a configurable pair mapping of letters
(similar to the reflector, but not requiring that every letter is exchanged) applied before the
first rotor (and thus also after passing through all the rotors and the reflector).
It also adds a ring setting, which allows the stepping point to be adjusted.
Later in the war, the Naval Enigma added a fourth rotor. This rotor does not step during
operation. (The fourth rotor is thinner than the others, and fits alongside a thin reflector,
meaning this rotor is not interchangeable with the others on a real Enigma.)
There were a number of other variants and additions to Enigma which are not currently supported
here, as well as different Enigma networks using the same basic hardware but different rotors
(which are supported by supplying your own rotor configurations).
### How Typex works
Typex is a clone of Enigma, with a few changes implemented to improve security. It uses five rotors
rather than three, and the _rightmost_ two are static. Each rotor has more stepping points.
Additionally, the rotor design is slightly different: the wiring for each rotor is in a removable
core, which sits in a rotor housing that has the ring setting and stepping notches. This means each
rotor has the same stepping points, and the rotor cores can be inserted backwards, effectively
doubling the number of rotor choices.
Later models (from the Mark 22, which is the variant we simulate here) added two plugboards: an
input plugboard, which allowed arbitrary letter mappings
(rather than just pair switches) and thus functioned similarly to a configurable extra static
rotor, and a reflector plugboard, which allowed rewiring the reflector.
### How the Bombe works
The Bombe is a mechanism for efficiently testing and discarding possible rotor positions, given
some ciphertext and known plaintext. It exploits the symmetry of Enigma and the reciprocal
(pairwise) nature of the plugboard to do this regardless of the plugboard settings. Effectively,
the machine makes a series of guesses about the rotor positions and plugboard settings and for
each guess it checks to see if there are any contradictions (e.g. if it finds that, with its
guessed settings, the letter `A` would need to be connected to both `B` and `C` on the plugboard,
that's impossible, and these settings cannot be right). This is implemented via careful connection
of electrical wires through a group of simulated Enigma machines.
A full explanation of the Bombe's operation is beyond the scope of this document - you can read
the source code, and the authors also recommend Graham Ellsbury's
[Bombe explanation](http://www.ellsbury.com/bombe1.htm), which is very clearly diagrammed.
## Implementation in CyberChef
### Enigma/Typex
Enigma and Typex were implemented from documentation of their functionality.
Enigma rotor and reflector settings are from GCHQ's documentation of known Enigma wirings. We
currently simulate all basic versions of the German Service Enigma; most other versions should be
possible by manually entering the rotor wirings. There are a few models of Enigma, or attachments
for the Service Enigma, which we don't currently simulate. The operation was tested against some
of GCHQ's working examples of Enigma machines. Output should be letter-for-letter identical to a
real German Service Enigma. Note that some Enigma models used numbered rather than lettered
rotors - we've chosen to stick with the easier-to-use lettered rotors.
There were a number of different Typex versions over the years. We implement the Mark 22, which is
backwards compatible with some (but not completely with all, as some early variants supported case
sensitivity) older Typex models. GCHQ also has a partially working Mark 22 Typex. This was used to
test the plugboards and mechanics of the machine. Typex rotor settings were changed regularly, and
none have ever been published, so a test against real rotors was not possible. An example set of
rotors have been randomly generated for use in the Typex operation. Some additional information on
the internal functionality was provided by the Bombe Rebuild Project.
### The Bombe
The Bombe was likewise implemented on the basis of documentation of the attack and the machine. The
Bombe Rebuild Project at the National Museum of Computing answered a number of technical questions
about the machine and its operating procedures, and helped test our results against their working
hardware Bombe, for which the authors would like to extend our thanks.
Constructing menus from cribs in a manner that most efficiently used the Bombe hardware was another
difficult step of operating the real Bombes. We have chosen to generate the menu automatically from
the provided crib, ignore some hardware constraints of the real Bombe (e.g. making best use of the
number of available Enigmas in the Bombe hardware; we simply simulate as many as are necessary),
and accept that occasionally the menu selected automatically may not always be the optimal choice.
This should be rare, and we felt that manual menu creation would be hard to build an interface for,
and would add extra barriers to users experimenting with the Bombe.
The output of the real Bombe is optimised for manual verification using the checking machine, and
additionally has some quirks (the rotor wirings are rotated by, depending on the rotor, between one
and three steps compared to the Enigma rotors). Therefore, the output given is the _ring position_,
and a correction depending on the rotor needs to be applied to the _initial value_, setting it to
`W` for rotor V, `X` for rotor IV, and `Y` for all other rotors. We felt that this would require
too much explanation in CyberChef, so the output of CyberChef's Bombe operation is the initial
value for each rotor, with the ring positions set to `A`, required to decrypt the ciphertext starting
at the beginning of the crib. The actual stops are the same. This would not have caused problems at
Bletchley Park, as operators working with the Bombe would never have dealt with a real or simulated
Enigma, and vice versa.
By default the checking machine is run automatically and stops which fail silently discarded. This
can be disabled in the operation configuration, which will cause it to output all stops from the
actual Bombe hardware instead. (In this case you only get one stecker pair, rather than the set
identified by the checking machine.)
#### Optimisation
A three-rotor Bombe run (which tests 17,576 rotor positions and takes about 15-20 minutes on
original Turing Bombe hardware) completes in about a fifth of a second in our tests. A four-rotor
Bombe run takes about 5 seconds to try all 456,976 states. This also took about 20 minutes on the
four-rotor US Navy Bombe (which rotates about 30 times faster than the Turing Bombe!). CyberChef
operations run single-threaded in browser JavaScript.
We have tried to remain fairly faithful to the implementation of the real Bombe, rather than
a from-scratch implementation of the underlying attack. There is one small deviation from "correct"
behaviour: the real Bombe spins the slow rotor on a real Enigma fastest. We instead spin the fast
rotor on an Enigma fastest. This means that all the other rotors in the entire Bombe are in the
same state for the 26 steps of the fast rotor and then step forward: this means we can compute
the 13 possible routes through the lower two/three rotors and reflector (symmetry means there are
only 13 routes) once every 26 ticks and then save them. This does not affect where the machine stops,
but it does affect the order in which those stops are generated.
The fast rotors repeat each others' states: in the 26 steps of the fast rotor between steps of the
middle rotor, each of the scramblers in the complete Bombe will occupy each state once. This means
we can once again store each state when we hit them and reuse them when the other scramblers rotate
through the same states.
Note also that it is not necessary to complete the energisation of all wires: as soon as 26 wires
in the test register are lit, the state is invalid and processing can be aborted.
The above simplifications reduce the runtime of the simulation by an order of magnitude.
If you have a large attack to run on a multiprocessor system - for example, the complete M4 Naval
Enigma, which features 1344 possible choices of rotor and reflector configuration, each of which
takes about 5 seconds - you can open multiple CyberChef tabs and have each run a subset of the
work. For example, on a system with four or more processors, open four tabs with identical Multiple
Bombe recipes, and set each tab to a different combination of 4th rotor and reflector (as there are
two options for each). Leave the full set of eight primary rotors in each tab. This should complete
the entire run in about half an hour on a sufficiently powerful system.

16
Getting-started.md

@ -11,18 +11,19 @@ CyberChef uses the Grunt build system, so it's very easy to install. You'll need
npm will then install all the dependencies needed by Grunt.
_Consider adding `export NODE_OPTIONS=--max_old_space_size=2048` to your `~/.bashrc` file. If you attempt to build a production version of CyberChef, you may get a "JavaScript heap out of memory" error if you do not set this environment variable._
## Compiling
Grunt has been configured with several tasks to aid in the development process:
```
grunt dev
```
> Use this when developing new functionality. It will launch a web server on port 8080 hosting an uncompressed, development version of CyberChef, accessible by browsing to [`localhost:8080`](http://localhost:8080). Whenever a source file is modified, the development version will be rebuilt automatically.
> Note: This task will initially result in an error relating to the `MetaConfig.js` file but will quickly rebuild and should complete successfully. This is due to the `MetaConfig.js` file being built at the same time as the rest of the app and therefore not being available for compilation immediately.
```
grunt prod
@ -69,9 +70,11 @@ grunt docs
- `src/`
- `core/` - Core CyberChef files that make up the heart of the application
- `config/` - Files specifying the operation configurations
- `modules/` - Modules containing the run functions for each operation
- `lib/` - Libraries that we can't currently import through npm
- `modules/` - Automatically generated module references
- `lib/` - Libraries containing shared code for multiple operations
- `errors/` - Custom error types
- `operations/` - Operation objects
- `vendor/` - Libraries that cannot currently be imported through npm
- `node/` - Wrappers for the NodeJS version of CyberChef
- `web/` - The code which makes up the CyberChef web app
- `css/`
@ -85,10 +88,11 @@ grunt docs
- `test/`
- `tests/` - Configuration for tests on operations and recipes
- `.babelrc` - Babel transpilation configuration
- `.travid.yml` - Travis CI build process configuration
- `.editorconfig` - Text editor conventions stored in a cross-compatible format
- `.travis.yml` - Travis CI build process configuration
- `Gruntfile.js` - Grunt build process configuration
- `webpack.config.js` - Webpack configuration
- `postcss.config.js` - PostCSS configuration
- `LICENSE` - The Apache 2.0 licence information
- `package.json` - npm configuration and a list of all the dependencies
- `README.md` - An introduction to CyberChef
- `README.md` - An introduction to CyberChef

2
Home.md

@ -7,7 +7,7 @@ Welcome to the CyberChef wiki pages. Here you can find guides for installing and
- [[Repository structure|Getting-started#repository-structure]]
2. [[Contributing]]
- [[Coding conventions|Contributing#coding-conventions]]
- [[Design principals|Contributing#design-principals]]
- [[Design principles|Contributing#design-principles]]
3. [[Adding a new operation]]
- [[How to add an operation|Adding-a-new-operation#how-to-add-an-operation]]
- [[Data types|Adding-a-new-operation#data-types]]

9
Troubleshooting.md Normal file

@ -0,0 +1,9 @@
## Script error when running `cyberchef.htm` from a local file in Safari
Sometimes Safari throws a scriptError or syntaxError when you run `cyberchef.htm` from a file. The workaround for this is to run an instance of [`http-server`](https://www.npmjs.com/package/http-server) with compression enabled. You can do this with the following commands:
```
cd <path to directory containing cyberchef.htm>
npm install -g http-server
http-server -g .
```