magdev/boxsim-fileformat-rev
1
Fork 0
Code Issues Pull Requests Actions Releases Activity

Add boxsim CLI: pure Python codec, netlist extraction, KiCad project generation

Browse Source 
Replaces the bpj2ini.py prototype with a packaged tool. The A2F codec is now
reimplemented in Python, so neither Boxsim.exe nor a CPU emulator is needed to
decode a container; the emulator survives as a development-only oracle that the
codec is differentially tested against.

Verified:
- 134 containers decode byte-identically to Boxsim's own decoder, 270 pre-2.0
  files rejected by both implementations (tools/difftest.py --self-test)
- Default.BPJ: 22 components, 14 nets, no dangling terminal
- 22 of 22 parts resolve to library symbols; one series inferred from a
  sibling, two DC resistance deviations reported
- ERC clean and the netlist KiCad extracts from the generated sheet is an
  identical partition to the one derived from the Boxsim grid
- container build byte-identical to the host build
- 32 tests passing, ruff clean

Decode time went 69 s -> 14.5 s (binding emulator hooks to trap addresses)
-> 1.6 s (pure Python).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
magdevandClaude Fable 5 committed 2026-09-29 11:10:43 +02:00
1 parent eb81ca61cb
commit 4737a5bae8
28 files changed
+4459 -122
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+130 -33
View File
@@ -1,63 +1,160 @@
# Boxsim .bpj / .bch file format (reverse engineered)
# boxsim
Boxsim (Visaton, Delphi, codec and container "MyIniFile" by UG 2004) stores projects (`.bpj`) and driver files (`.bch`) in a three-layer container. Reverse engineered 2026-09-28 from Boxsim 2.1 DE (`Boxsim.exe`, no prior public documentation existed).
Read Boxsim loudspeaker projects and turn their crossovers into KiCad projects.
## Container layout
[Boxsim](https://www.visaton.de/) is Visaton's free loudspeaker simulation program. Its project files (`.bpj`) and driver files (`.bch`) use an undocumented container, reverse engineered here. `boxsim` decodes them and generates a KiCad 10 project whose schematic mirrors the crossover you drew in Boxsim, with every part bound to a symbol and footprint from [kicad-visaton-lib](https://src.bundespruefstelle.ch/boxdev/kicad-visaton-lib).
## Install
```bash
pip install .
```
Nothing is required at runtime beyond Python 3.11+. KiCad 10 is a hard dependency of the verification gate (`boxsim build --verify`) only, and lives on the host.
Point the tool at the Visaton library and the Boxsim parts catalogue:
```bash
export BOXSIM_KICAD_LIB=/path/to/kicad-visaton-lib
export BOXSIM_CATALOG="/path/to/BoxSim 2.1 DE/Weichenbauteile/Visatonteile.txt"
```
The symbol library must be in the current KiCad format. Convert an older one with `kicad-cli sym upgrade --force symbols/Visaton.kicad_sym`.
## Use
```bash
boxsim decode project.bpj -o project.ini # raw INI payload
boxsim extract project.bpj -o project.json # structured model, symbols resolved
boxsim netlist project.bpj # crossover netlist
boxsim build project.bpj out/ --verify # a KiCad 10 project
```
`build` writes `<name>.kicad_sch`, `<name>.kicad_pro` and project-local `sym-lib-table` / `fp-lib-table`. No board is generated: use KiCad's *Update PCB from Schematic*, which works because every Visaton symbol carries a `Footprint` property.
### Docker
```bash
./tools/build-image.sh
docker run --rm -v "$PWD:/work" boxsim:0.1.0 build project.bpj out/
```
The image bundles the tool, the Visaton library and the catalogue. It contains no KiCad and no `Boxsim.exe`, so `decode`, `extract`, `netlist` and `build` work inside it while `--verify` does not. Run the gate on the host.
## The file format
| Layer | Content |
| ----- | ------- |
| 1 | 8-byte plaintext magic `MyIni2CK` |
| 2 | Everything after the magic is XOR-encrypted with a position-based PRNG keystream, seeds (123, 456), identical for every file |
| 3 | The decrypted body is a custom LZ + canonical-Huffman stream ("A2F" codec, `CompressA2F`/`ExpandA2F` at VA `0xa1013c`/`0xa11880`) |
| 4 | The decompressed payload is plain INI-style text: `key=<TAB>value` lines, CRLF, `#Section` headers, German decimal commas, arrays as `;`-separated values |
| 2 | XOR against a keystream from a table-shuffled three-LCG generator, seeded (123, 456) for every file ever written |
| 3 | A custom LZ plus canonical-Huffman codec ("A2F") |
| 4 | INI-style text: `#Section` headers, `key<TAB>value`, CRLF, German decimal commas, `;`-separated arrays |
## Layer 2: the Krypt PRNG
Files written by Boxsim 1.2 use an older container without the magic and are rejected.
Table-shuffled combination of three LCGs (found at VA `0xa12be4` init, `0xa12c9c` next byte). All arithmetic mod 2^32:
### The Krypt keystream
Three linear congruential generators feeding a 256-entry shuffle table (Boxsim 2.1: init at VA `0xa12be4`, next byte at `0xa12c9c`). All arithmetic mod 2^32:
```text
init(s1=123, s2=456):
s3 = (s1 mod 0xFFFE) * (s2 mod 0x7F23)
repeat 256 times: step(); table[i] = s1 + s2
256 times: step(); table[i] = s1 + s2
step():
s1 = s1 * 0x10B95 + 1
s2 = s2 * 0x10BA9 + 1 (reset to 0 if >= 0xFFFFFFFF)
s3 = s3 * 0x1155D + 1
step(): s1 = s1*0x10B95 + 1
s2 = s2*0x10BA9 + 1 (reset to 0 if >= 0xFFFFFFFF)
s3 = s3*0x1155D + 1
next_byte():
step()
idx = s3 and 0xFF
out = table[idx]; table[idx] = s1 + s2
return out mod 256
next(): step(); i = s3 and 0xFF
out = table[i]; table[i] = s1 + s2
return out mod 256
```
`plaintext[i] = filebyte[8+i] XOR next_byte()`.
### The A2F codec
## Layer 3: the A2F codec
A bit stream read MSB first through a 31-bit accumulator, with symbols from a canonical Huffman code over 356 symbols decoded via a 65536-entry direct lookup table. All code lengths start at 9.
Sliding-window LZ (1 MB window, flush keeps the last 0x3FFF8 bytes) with a 356-symbol canonical Huffman alphabet: 0..255 literals, 0x100 end-of-stream, 0x101 rebuild code-length tables (adaptive), 0x102 special, rest match lengths/distances. Decoding uses a 65536-entry direct lookup table; initial state gives every symbol a 9-bit code. Not re-implemented: `bpj2ini.py` runs the original `ExpandA2F` machine code under Unicorn instead, which is exact by construction.
| Symbol | Meaning |
| ------ | ------- |
| `0x000`-`0x0FF` | literal byte |
| `0x100` | end of stream |
| `0x101` | new code lengths follow, rebuild the tables |
| `0x102` | define N entries of the 64-entry digram dictionary |
| `0x103`-`0x10A` | match, length 3, distance in k+3 bits |
| `0x10B`-`0x114` | match, length 4, distance in k+3 bits |
| `0x115`-`0x123` | match, unary-coded length class, distance in k+3 bits |
| `0x124`-`0x163` | emit the two bytes of digram dictionary entry k |
## Crossover encoding (`#Weiche<n>` section)
Distances take `k+3` raw bits biased by `2**(k+3) - 7`, so the classes tile 1.. with no gaps. Code lengths are coded as two zero bits meaning "same as previous", otherwise a nibble plus one, where the value 5 escapes to a second nibble.
The crossover editor grid serializes as `_Part<N>_*` keys. Only non-empty slots are written:
Boxsim decodes from a staging buffer and reads past the end of the input for the final symbol's lookup window, so the decoder tolerates a bounded number of zero padding bytes. Beyond that a truncated file fails loudly.
### The crossover grid
The crossover editor is an 11 x 18 node grid. A slot number encodes position and orientation:
| Range | Meaning |
| ----- | ------- |
| 1-180 | horizontal, 10 per row, joins node(r,c) to node(r,c+1) |
| 181-367 | vertical, 11 per row, joins node(r,c) to node(r+1,c) |
| 368-375 | free link, endpoints given by `_nod1` and `_nod2` |
Nodes are numbered row-major and zero based: `node = row * 11 + col`.
Keys in a `#Weiche<n>` section:
| Key | Meaning |
| --- | ------- |
| `_Part<N>_typ` | 0 empty, 1 wire, 2 resistor, 3 capacitor, 4 inductor, 5 driver, 6 amplifier input |
| `_Part<N>_value` | Component value in SI base units (Ohm, Farad, Henry) |
| `_Part<N>_Rvalue` | Series resistance of an inductor in Ohm |
| `_Part<N>_bem` | Part label, e.g. `MKT-A`, `SP 0,6mm`, `Keramik 10W` |
| `_Part<N>_chassisnr` | Driver index (0-based) for typ 5 |
| `_Part<N>_verpolt` | Polarity flag (-1 / 1) for typ 5 and 6 |
| `_Part368..375_nod1/nod2` | Extra node-to-node connections |
| `_Part<N>_value` | SI units: Ohm, Farad, Henry |
| `_Part<N>_Rvalue` | inductor DC resistance in Ohm |
| `_Part<N>_bem` | Visaton series name, matching `Visatonteile.txt` exactly |
| `_Part<N>_chassisnr` | zero-based driver index |
| `_Part<N>_verpolt` | polarity, -1 meaning inverted |
Other sections: `#General` (nAmps, nChassis), `#AktFilter<n>` (active filters), `#Chassis<n>` (TSP set plus measured frequency/impedance/phase arrays), `#Einbau<n>` (enclosure: Typ 1 closed with `Vb` in m³, etc.), `#Schallwand<n>` (baffle geometry in m), `#GemGeh` (shared enclosure).
Other sections: `#General`, `#AktFilter<n>` and `#Aktfilter<n>` (active filters, note the differing case), `#Chassis<n>` (name plus Thiele-Small parameters and measured response arrays), `#Einbau<n>` (enclosure, `Vb` in m³), `#Schallwand<n>` (baffle geometry in m), `#GemGeh` (shared enclosure).
## Tool
## How the parts are resolved
`Weichenbauteile/Visatonteile.txt` is the authoritative parts list: 95 parts across 11 series, with inductor DC resistance in the fourth column. Its series strings are exactly the `_bem` values Boxsim writes, so resolution is a lookup rather than a guess.
Values are matched numerically against the library rather than formatted into symbol names, because the library's own strings are irregular (`MKP 1.0μF` beside `MKP 10μF`, `SP-Spule 1.0mH (0.6)` beside `LR-Spule 10mH`).
- An empty `_bem` is inferred from a sibling part of the same type and value in the same crossover, else from the widest series for that type. Every inference is logged.
- A `_Rvalue` that differs from the catalogue is reported, never corrected: it is a legitimate edit in Boxsim.
- A value Visaton does not stock is a hard error naming the nearest stocked values. Nothing is substituted silently.
The library contains one symbol the catalogue does not list, `MKT 22μF`. It is a real Visaton part (radial film capacitor, 250 V) that Boxsim does not offer in its picker, so no project can reference it.
## Verification
ERC only proves a schematic is well formed; it would happily pass one wired to the wrong cells. So `boxsim build --verify` compares the netlist KiCad extracts from the generated sheet against the netlist derived independently from the Boxsim grid, and both derivations must agree.
The codec is held to the same standard. `tools/difftest.py` decodes every container in the Boxsim distribution twice, once with the pure Python codec and once by emulating Boxsim's own `ExpandA2F` under Unicorn, and requires byte-identical output:
```bash
.venv/bin/python bpj2ini.py <input.bpj|.bch> [output.ini]
.venv/bin/python tools/difftest.py --self-test
# 134 byte-identical, 270 rejected by both as pre-2.0 containers
```
Needs `unicorn` and `pefile` (see `.venv/`) and `Boxsim.exe` (path via `BOXSIM_EXE` env, default: the bundled download under `boxsim_dl/`). Works for `.bpj` and `.bch` alike.
Both gates are checked against a deliberately broken input so they are known to fail when they should: `--self-test` corrupts a digest, and `test_gate_has_teeth` mis-wires a component.
The oracle is development-only. It needs `unicorn`, `pefile` and a copy of `Boxsim.exe`:
```bash
pip install -e '.[oracle,dev]'
export BOXSIM_EXE="/path/to/Boxsim.exe"
```
## Tests
```bash
pytest # fast, no Boxsim.exe needed
ruff check boxsim tools tests
```
Tests needing KiCad or the Boxsim distribution skip themselves when those are absent.
## Licence
MIT.