{"content":"# aux4/test\n\naux4 testing tool\n\naux4/test provides a lightweight markdown-driven test runner for aux4 packages. Tests are written as Markdown files with special fenced blocks (execute, expect, error, file, timeout, hooks, etc.). The package interprets these files and runs commands, capturing stdout/stderr and validating output against the provided expectations. It's designed for authoring reproducible command-line tests, verifying CLI behavior, file artefacts, timeouts, and test hooks.\n\nThis README documents all features exercised by the official test suite. Each section shows real test-file content (copied from the test/ folder) and explains the behavior and usage. Examples are taken verbatim from the repository tests so you can copy and run them.\n\n## Installation\n\n``bash\naux4 aux4 pkger install aux4/test\n`\n\n## System Dependencies\n\nThis package requires system dependencies. You need to have one of the following system installers:\n\n- [brew](/r/public/packages/aux4/system-installer-brew)\n- [apt](/r/public/packages/aux4/system-installer-apt)\n- [npm](/r/public/packages/aux4/system-installer-npm)\n\nFor more details, see [system-installer](/r/public/packages/aux4/pkger/commands/aux4/pkger/system).\n\n## Quick Start\n\nRun the test runner in the current directory:\n\n`bash\naux4 test run .\n`\n\nThis scans for .test.md files in the directory structure and runs them using the aux4 test runner. By default the runner uses the directory \".\"; you may pass another directory as the positional dir variable.\n\nThe package provides these primary commands:\n\n- aux4 test run — run tests (link: ./commands/test/run)\n- aux4 test coverage — run tests with coverage report (link: ./commands/test/coverage)\n- aux4 test report — render coverage report from JSON file (link: ./commands/test/report)\n- aux4 test add — add test entries to a test file (link: ./commands/test/add)\n\nSee command docs: [aux4 test run](./commands/test/run), [aux4 test coverage](./commands/test/coverage), [aux4 test report](./commands/test/report), and [aux4 test add](./commands/test/add).\n\n---\n\n## Execute and Expect Basic Blocks\n\nOverview:\n\n- execute blocks contain shell commands to run.\n- expect blocks validate stdout (exact match by default).\n- error blocks validate stderr.\n- A test typically pairs a single execute block with one or more expect/error blocks.\n\nExample (from test/execute-expect.test.md):\n\n``markdown\n# Execute/Expect Basic Functionality\n\nTests for basic execute and expect blocks to verify command execution and output validation.\n\n## Simple Echo Test\n\n`execute\necho \"Hello World\"\n`\n\n`expect\nHello World\n`\n``\n\nHow it works:\n\n- The runner executes the command(s) inside the `execute block.\n- It captures stdout and stderr separately.\n- The expect block is matched against stdout; error is matched against stderr.\n- Exact matching: the default expect requires the output lines to match exactly (including order and line breaks).\n\nOther examples in the same file illustrate multi-line output, combined stdout/stderr, and environment variable usage.\n\n---\n\n## Expect Modifiers — partial, ignoreCase, regex\n\nThe runner supports modifiers appended to the expect/error block token to change matching semantics.\n\nGeneral forms:\n\n- `expect:partial — substring / wildcard matching\n\n `\n\n- `expect:ignoreCase — case-insensitive exact matching\n\n `\n\n- `expect:regex — regular-expression matching\n\n `\n\n- `expect:json — JSON formatting (pretty-print compact JSON before comparing)\n\n `\n\n- Combinations like `expect:regex:ignoreCase are supported. The :json modifier can also be combined with others: `expect:json:partial, `expect:json:ignoreCase, `expect:json:regex.\n\nBelow are examples directly taken from the tests for each modifier.\n\nExpect:partial (substring and wildcard matching)\n\n``markdown\n# Expect Partial Modifier\n\nTests for expect:partial modifier to verify substring matching and wildcard pattern matching.\n\n## Simple Substring Match\n\n`execute\necho \"This is a long output with many words\"\n`\n\n`expect:partial\nlong output\n`\n``\n\nKey points:\n\n- partial checks that the expected text occurs somewhere in stdout.\n- Wildcards:\n - ? acts like a \"match anything\" placeholder in partial patterns. For example Start ? end will match Start middle end.\n - * acts like a greedy \"match anything\" placeholder.\n - acts like a multiline \"match anything\" placeholder that can span across newlines. For example Startend will match Start\\nMiddle content\\nend.\n- partial also applies to error:partial for stderr.\n\nExpect:ignoreCase (case-insensitive matching)\n\n``markdown\n# Expect IgnoreCase Modifier\n\nTests for expect:ignoreCase modifier to verify case-insensitive output matching.\n\n## Basic Case Insensitive Match\n\n`execute\necho \"Hello World\"\n`\n\n`expect:ignoreCase\nhello world\n`\n``\n\nKey points:\n\n- ignoreCase makes the comparison case-insensitive.\n- It works for both expect (stdout) and error (stderr).\n- Can combine with regex so expect:regex:ignoreCase runs a case-insensitive regex.\n\nExpect:regex (regular expressions)\n\n``markdown\n# Expect Regex Modifier\n\nTests for expect:regex modifier to verify regular expression matching in output validation.\n\n## Simple Pattern Matching\n\n`execute\necho \"Hello World 123\"\n`\n\n`expect:regex\n^Hello World \\d+$\n`\n``\n\nKey points:\n\n- Regexes are applied line-wise unless the pattern covers multiple lines explicitly.\n- Use typical regex escapes (e.g., \\d, \\b) as in the test examples.\n- error:regex validates stderr with a regex.\n\nExpect:json (JSON formatting)\n\nThe :json modifier pretty-prints compact JSON output before comparison. When a command outputs inline/compact JSON like {\"name\":\"John\",\"age\":30}, the modifier parses it and reformats it with JSON.stringify(JSON.parse(value), null, 2) so the expect block can be written in a human-readable, indented format.\n\nThe JSON formatting is applied to the **actual output** (stdout or stderr) before any other modifier logic runs. This means :json can be combined with other modifiers:\n\n- expect:json — exact match against pretty-printed JSON\n- expect:json:partial — substring/wildcard match against pretty-printed JSON\n- expect:json:ignoreCase — case-insensitive match against pretty-printed JSON\n- expect:json:regex — regex match against pretty-printed JSON\n- error:json — same behavior for stderr\n\nIf the output is not valid JSON, it is left as-is and the test will fail with a meaningful diff showing what was expected vs. the raw output.\n\n``markdown\n# Expect JSON Modifier\n\n## Basic JSON Object\n\n`execute\necho '{\"name\":\"John\",\"age\":30}'\n`\n\n`expect:json\n{\n \"name\": \"John\",\n \"age\": 30\n}\n`\n\n## JSON Combined with Partial\n\n`execute\necho '{\"status\":\"ok\",\"data\":{\"count\":42,\"items\":[\"x\",\"y\"]}}'\n`\n\n`expect:json:partial\n\"count\": 42\n`\n\n## Error JSON\n\n`execute\necho '{\"error\":\"not found\",\"code\":404}' >&2\n`\n\n`error:json\n{\n \"error\": \"not found\",\n \"code\": 404\n}\n`\n``\n\n---\n\n## Multiple Expects for a Single Execute\n\nOverview:\n\n- A single execute block may be followed by many expect blocks.\n- All expectations are evaluated against that execute's captured stdout/stderr.\n- This is useful to assert multiple aspects of multi-line output.\n\nExample (from test/multiple-expects-test.test.md):\n\n``markdown\n# Multiple Expects Test\n\nTest that a single execute command can have multiple expect blocks with different modifiers.\n\n## Test Multiple Expects for Single Execute\n\n`execute\necho \"Line 1: Hello World\nLine 2: Testing 123\nLine 3: Final result\"\n`\n\n`expect:partial\nHello World\n`\n\n`expect:partial\nTesting 123\n`\n\n`expect:partial\nFinal result\n`\n\n`expect:regex\nLine \\d+: Hello World\n`\n``\n\nHow it works:\n\n- Each expect (or error) block is evaluated independently.\n- If any one expectation fails, the test fails (the runner reports which expectation failed).\n\n---\n\n## File Blocks and File Scope\n\nOverview:\n\n- file:<filename> blocks create files with the provided content during the test.\n- Files created by file blocks are scoped: they are available to nested tests, but siblings do not share file state.\n- File blocks are commonly used to set up fixtures (config files, sample inputs, etc.).\n\nExample (from test/file-scope.test.md):\n\n``markdown\n# File Block Scope Testing\n\nTests to verify that files created in `file blocks are available for nested tests but not for tests at the same level.\n\n## Parent Scenario with File\n\n`file:parent-file.txt\nThis file is created at parent level\n`\n\n### Nested Test 1 - File Should Be Available\n\n`execute\ncat parent-file.txt\n`\n\n`expect\nThis file is created at parent level\n`\n``\n\nImportant scope rules illustrated by the test:\n\n- A file declared at a parent level is available to nested scenarios/tests beneath it.\n- Files created in one sibling scenario are not visible in other sibling scenarios (each sibling has its own scope).\n- When writing tests, place file blocks at the scenario level that should share those files with nested tests.\n\nExample showing sibling isolation (from same test file):\n\n``markdown\nThis scenario is at the same level as \"Parent Scenario with File\", so the parent-file.txt should not be available here.\n\n`execute\nls parent-file.txt 2>/dev/null || echo \"File not found\"\n`\n\n`expect\nFile not found\n`\n``\n\n---\n\n## Hooks: beforeAll, afterAll, beforeEach, afterEach\n\nOverview:\n\n- Hooks are multi-command blocks that run at certain points in the test lifecycle:\n - beforeAll — run once before the tests in the current scenario.\n - afterAll — run once after all tests in the current scenario.\n - beforeEach — run before each test in the current scenario.\n - afterEach — run after each test in the current scenario.\n- Tests in the repository exercise beforeAll and afterAll. The runner supports the other hooks as well; they follow the same scoping rules as file blocks.\n\nExample with beforeAll/afterAll (from test/hooks.test.md):\n\n``markdown\n# Hooks Functionality\n\nTests for beforeAll, afterAll, beforeEach, and afterEach hooks to verify test setup and cleanup.\n\n## Setup and Cleanup Hooks\n\n`beforeAll\nmkdir -p test-dir\necho \"Setup complete\" > test-dir/setup.log\n`\n\n`afterAll\nrm -rf test-dir\n`\n\n### Test 1 - Verify Setup\n\n`execute\ncat test-dir/setup.log\n`\n\n`expect\nSetup complete\n`\n``\n\nHow it works:\n\n- beforeAll created the directory and file; nested tests verify its presence.\n- afterAll removes the created artifacts.\n- beforeEach and afterEach (not used in the example) would run around each test in the scenario, enabling per-test setup/cleanup (e.g., resetting state or deleting temporary files).\n- Hook blocks may contain multiple shell commands; they are executed in the same shell environment as the tests (so side effects are available).\n\n---\n\n## Timeout Block\n\nOverview:\n\n- Use a timeout block containing a number (milliseconds) to override the default test timeout for the following execute block.\n- The test suite demonstrates short and long timeouts to cover long-running commands.\n\nExamples (from test/timeout.test.md):\n\n``markdown\n# Timeout Functionality\n\nTests for timeout configuration to verify that long-running tests can be properly configured.\n\n## Fast Command Test\n\n`timeout\n1000\n`\n\n`execute\necho \"Quick command\"\n`\n\n`expect\nQuick command\n`\n``\n\nMore examples in the same file show 5000 and 10000 millisecond timeouts. If no timeout block is present, the runner uses the default timeout.\n\nPractical notes:\n\n- Place the timeout block directly before the execute it applies to.\n- The timeout must be a numeric value in milliseconds.\n- If the command exceeds the timeout, the test fails with a timeout error.\n\n---\n\n## Error Blocks (stderr) and Combined Output\n\n- error blocks validate stderr content (supports the same modifiers as expect, e.g., error:partial, error:regex, error:ignoreCase).\n- Tests show combined stdout and stderr where both expect and error blocks are used for the same execute.\n\nExample snippet (from execute-expect.test.md):\n\n``markdown\n## Combined Output and Error\n\n`execute\necho \"Success output\" && echo \"Error output\" >&2\n`\n\n`expect\nSuccess output\n`\n\n`error\nError output\n`\n``\n\nKey points:\n\n- stdout and stderr are captured separately and matched to expect and error blocks respectively.\n- If both appear, you can assert on each independently.\n\n---\n\n## Adding/Authoring Tests: aux4 test add\n\nThe package includes aux4 test add to programmatically append tests to a test markdown file. The test suite includes tests that exercise adding tests with files and different heading levels.\n\nExcerpt (from test/add.test.md):\n\n```markdown\n# Aux4 Test Add Command\n\nTests for the aux4 test add command functionality to verify test creation and file generation.\n\n`beforeAll\necho \"# Test Suite\" > my-test-suite.test.md\n`\n\n`afterAll\nrm -f my-test-suite.test.md sample-file.txt another-file.js\n`\n\n## Basic Test Addition\n\n### Add Simple Test with Execute\n\n`execute\necho \"# Test Suite\" > my-test-suite.test.md && aux4 test add my-test-suite.test.md --level 2 --name \"Simple Echo Test\" --execute \"echo Hello World\"\n`\n\n`execute\ncat my-test-suite.test.md\n`\n\n``expect\n# Test Suite\n\n## Simple Echo Test\n\n`execute\necho Hello World\n`\n\n`expect\nHello World\n`\n``\n```\n\nUsage:\n\n- aux4 test add <testFile> --level <n> --name \"<name>\" --execute \"<command>\" appends a test at the specified Markdown heading level.\n- --file <path> may be included multiple times to embed file blocks into the test being added.\n- The add command is useful for generating or updating test files programmatically as part of toolchains.\n\nThe add.test.md file includes multiple examples: adding tests with single or multiple files, different heading levels, and execute-only tests. The test verifies that the add command correctly writes headers, file blocks, execute blocks, and expect blocks into the test file.\n\n---\n\n## Putting It All Together — Examples\n\nBelow are runnable, real examples taken directly from the tests. Each example includes the test file heading and the important sections so you can see the full test structure.\n\nExample: Basic execute/expect test file fragment\n\n``markdown\n# Execute/Expect Basic Functionality\n\n## Simple Echo Test\n\n`execute\necho \"Hello World\"\n`\n\n`expect\nHello World\n`\n``\n\nExample: Partial matching with wildcards\n\n``markdown\n# Expect Partial Modifier\n\n## Wildcard Pattern with \\*?\n\n`execute\necho \"Start middle end\"\n`\n\n`expect:partial\nStart *? end\n`\n``\n\nExample: Case-insensitive expect and combining with regex\n\n``markdown\n# Expect IgnoreCase Modifier\n\n## Combined Modifiers - Regex and IgnoreCase\n\n`execute\necho \"Error Code: ABC123\"\n`\n\n`expect:regex:ignoreCase\nerror code: [a-z]+\\d+\n`\n``\n\nExample: File scoping (parent creates a file, nested tests use it)\n\n``markdown\n# File Block Scope Testing\n\n## Parent Scenario with File\n\n`file:parent-file.txt\nThis file is created at parent level\n`\n\n### Nested Test 1 - File Should Be Available\n\n`execute\ncat parent-file.txt\n`\n\n`expect\nThis file is created at parent level\n`\n``\n\nExample: Hooks setup/teardown\n\n``markdown\n# Hooks Functionality\n\n## Setup and Cleanup Hooks\n\n`beforeAll\nmkdir -p test-dir\necho \"Setup complete\" > test-dir/setup.log\n`\n\n`afterAll\nrm -rf test-dir\n`\n\n### Test 1 - Verify Setup\n\n`execute\ncat test-dir/setup.log\n`\n\n`expect\nSetup complete\n`\n``\n\nExample: Timeout usage\n\n``markdown\n# Timeout Functionality\n\n## Long Timeout Test\n\n`timeout\n10000\n`\n\n`execute\nsleep 2 && echo \"Command completed after 2 seconds\"\n`\n\n`expect\nCommand completed after 2 seconds\n`\n``\n\nExample: Adding a test using aux4 test add and verifying the produced test file (excerpt)\n\n```markdown\n# Aux4 Test Add Command\n\n`execute\necho \"# Test Suite\" > my-test-suite.test.md && aux4 test add my-test-suite.test.md --level 2 --name \"Simple Echo Test\" --execute \"echo Hello World\"\n`\n\n`execute\ncat my-test-suite.test.md\n`\n\n``expect\n# Test Suite\n\n## Simple Echo Test\n\n`execute\necho Hello World\n`\n\n`expect\nHello World\n`\n``\n```\n\n---\n\n## Expect:similar — Deterministic Text Similarity\n\nThe :similar modifier compares the command output against a reference using text similarity metrics. No LLM calls — fully deterministic with zero token cost.\n\nTwo forms for providing the reference:\n\n**Inline reference** (content below --- separator):\n\n``markdown\n`execute\necho \"Hello World\"\n`\n\n`expect:similar\nmetric: fuzzy\npass: 0.8\n---\nHello World\n`\n``\n\n**File reference:**\n\n``markdown\n`file:expected/output.txt\nHello World\n`\n\n`execute\necho \"Hello World!\"\n`\n\n`expect:similar\npass: 0.9\nfile: expected/output.txt\n`\n``\n\nBlock fields:\n\n- name — optional label for the score (appears in JSON output)\n- metric — similarity algorithm: fuzzy (default), cosine, jaccard\n- pass — minimum similarity score 0-1 (default: 0.8)\n- file — path to reference file (alternative to inline content)\n\nAvailable metrics:\n\n- fuzzy — Levenshtein distance ratio (0-1). Character-level similarity. Good for \"almost the same\" text.\n- cosine — word-level cosine similarity (0-1). Good for meaning overlap regardless of word order.\n- jaccard — word set intersection/union (0-1). Good for \"same words present.\"\n\nCan be combined with :ignoreCase:\n\n``markdown\n`expect:similar:ignoreCase\npass: 0.9\n---\nhello world\n`\n``\n\nOutput:\n\n`\n similarity: 0.92 (fuzzy) ✓ PASS (>= 0.8)\n`\n\n---\n\n## Expect:ai:score — LLM-as-a-Judge Scoring\n\nThe :ai:score modifier uses an LLM to score the command output on a criterion, returning a numeric score instead of pass/fail. Requires --aiConfig and --configFile like expect:ai.\n\n``markdown\n`execute\ncat config.yaml\n`\n\n`expect:ai:score\nname: correctness\neval: Does the output match expected/config.yaml?\nrange: 1-5\npass: 3\n`\n\n`expect:ai:score\nname: completeness\neval: Are all required fields present?\n`\n``\n\nBlock fields:\n\n- name — optional label for the score\n- eval — what to evaluate (required, the judge prompt)\n- range — score range as min-max (default: 1-5)\n- pass — minimum passing score (default: 3)\n\nThe judge agent has readFile tool access, so the eval text can reference files created by file: blocks (e.g., \"Does the output match expected/config.yaml?\").\n\nOutput:\n\n`\n correctness: 5/5 Values match the ground truth\n completeness: 4/5 Missing optional headers\n ✓ PASS (all criteria met)\n`\n\nRun with:\n\n`bash\naux4 test run test.md --aiConfig agent --configFile config.yaml\n`\n\n---\n\n## JSON Output — --output flag\n\nWrite structured test results to a JSON file:\n\n`bash\naux4 test run test/ --output results.json\n`\n\nThe JSON includes all test results with scores and assertion outcomes:\n\n`json\n{\n \"timestamp\": \"2026-04-25T10:30:00Z\",\n \"tests\": [\n {\n \"title\": \"should create valid config\",\n \"passed\": true,\n \"duration\": 8400,\n \"results\": [\n { \"type\": \"exact\", \"passed\": true },\n { \"type\": \"similar\", \"name\": \"structure\", \"metric\": \"fuzzy\", \"score\": 0.92, \"pass\": 0.8 },\n { \"type\": \"ai:score\", \"name\": \"correctness\", \"eval\": \"Does it match?\", \"score\": 5, \"max\": 5, \"pass\": 3, \"reason\": \"Values match\" }\n ]\n }\n ],\n \"summary\": {\n \"total\": 1,\n \"passed\": 1,\n \"failed\": 0\n }\n}\n`\n\n---\n\n## Coverage — Step-Level Instrumentation\n\naux4/test includes a coverage system that tracks which execute steps, branches, and loop iterations were exercised during test runs or regular command usage.\n\n### Running Tests with Coverage\n\n`bash\naux4 test coverage test/\n`\n\nThis runs all tests normally, then prints a coverage report showing which commands and execute steps were hit:\n\n`text\nCoverage Report\n======================================================================\n\n Package: my-app@1.0.0\n\n main/build 3/3 steps ████████████ 100% 1.2s\n main/deploy 2/5 steps █████░░░░░░░ 40% 0.3s\n ✗ [2] when:${env}=prod:nout:backup-db\n ✗ [3] nout:notify-slack ${env}\n ✗ [4] log:deployed to ${env}\n\n----------------------------------------------------------------------\n\n Summary:\n Commands: 2/3 (67%)\n Steps: 5/8 (63%)\n Branches: 1/2 (50%)\n\n Slowest commands:\n main/build 1.2s\n main/deploy 0.3s\n\n======================================================================\n`\n\n### Coverage Metrics\n\n| Metric | What it tracks |\n|--------|---------------|\n| **Step coverage** | Which execute items in each command were hit |\n| **Command coverage** | Which commands were invoked at least once |\n| **Branch coverage** | Which when: conditions were evaluated as both true and false |\n| **Iteration tracking** | How many times each: loops ran and per-iteration durations |\n| **Duration** | Time per step, command, and iteration |\n\n### Standalone Usage with AUX4_COVERAGE_FILE\n\nCoverage instrumentation is built into the aux4 core. Set the AUX4_COVERAGE_FILE environment variable to record coverage from any aux4 command:\n\n`bash\n# Record coverage from regular commands\nAUX4_COVERAGE_FILE=cov.json aux4 build\nAUX4_COVERAGE_FILE=cov.json aux4 deploy --env staging\n\n# Multiple runs merge into the same file\nAUX4_COVERAGE_FILE=cov.json aux4 deploy --env prod\n\n# View the report\naux4 test report cov.json\n`\n\nWhen AUX4_COVERAGE_FILE is not set, coverage is completely disabled with zero performance overhead.\n\n### Enforcing a Coverage Threshold\n\nUse --threshold to fail with exit code 1 if step coverage is below the specified percentage:\n\n`bash\n# Fail CI if coverage drops below 80%\naux4 test coverage test/ --threshold 80\n\n# Same for standalone report\naux4 test report cov.json --threshold 80\n`\n\nWhen below threshold:\n\n`text\nCoverage 50% is below threshold 80%\n`\n\n### Viewing a Coverage Report\n\n`bash\naux4 test report <coverage-file> [--dir <path>] [--threshold <n>]\n`\n\nThe report command reads a coverage JSON file and scans the specified directory for .aux4 files to build the full universe of commands, then renders which were covered and which were missed.\n\nSee command docs: [aux4 test coverage](./commands/test/coverage) and [aux4 test report](./commands/test/report).\n\n---\n\n## Datasets — Data-Driven Tests\n\nThe dataset block runs an entire scenario (including nested children) once per entry in a JSON array, similar to Jest's it.each. Variables from each dataset entry are substituted into execute, expect, error, file, and hook blocks using {{variable}} syntax.\n\nTwo syntax forms are supported:\n\n**Config block** (with options):\n\n``markdown\n## should add numbers\n\n`dataset\nfile: dataset/math.json\nroot: $.data.items\nkey: id\n`\n\n`execute\necho $(({{a}} + {{b}}))\n`\n\n`expect\n{{result}}\n`\n``\n\n**Shorthand** (file path in the language tag):\n\n``markdown\n## should add numbers\n\n`dataset:dataset/math.json\n`\n\n`execute\necho $(({{a}} + {{b}}))\n`\n\n`expect\n{{result}}\n`\n``\n\nWhere dataset/math.json contains:\n\n`json\n[\n {\"a\": 1, \"b\": 2, \"result\": \"3\"},\n {\"a\": 10, \"b\": 20, \"result\": \"30\"}\n]\n`\n\n### Dataset Block Fields\n\n- file — path to a JSON file (resolved relative to the test file's directory)\n- root — JSONPath expression to extract an array from the JSON structure (default: $, the root). Use when the array is nested inside the JSON, e.g., $.data.items\n- key — field name from each entry to use as the label in test output. When set, entries appear as [keyValue] instead of [#0], [#1], etc.\n\n### How It Works\n\n- The dataset block is placed at a scenario heading level (e.g., ##). The entire describe for that scenario — including all its tests and nested children — is repeated once per dataset entry.\n- Dataset variables are merged with config params (--config). **Dataset values override config values** when keys collide.\n- Object and array values in dataset entries are automatically JSON.stringify'd for substitution.\n- null/undefined values become empty strings.\n- Empty datasets skip the scenario with a warning.\n- Non-array data (after root resolution) produces an error.\n\n### Nested Datasets (Cartesian Product)\n\nWhen a child scenario also has a dataset block, the result is a cartesian product — every combination runs:\n\n``markdown\n## should combine greetings\n\n`dataset\nfile: dataset/prefixes.json\n`\n\n### should greet\n\n`dataset\nfile: dataset/names.json\n`\n\n`execute\necho \"{{prefix}}, {{name}}!\"\n`\n\n`expect:partial\n{{prefix}}, {{name}}!\n`\n``\n\nWith prefixes.json = [{\"prefix\":\"Hello\"},{\"prefix\":\"Hi\"}] and names.json = [{\"name\":\"Alice\"},{\"name\":\"Bob\"}], this runs 4 tests (2 x 2).\n\nThe child dataset entries are merged on top of the parent's, so inner variables can override outer ones.\n\n### Test Output\n\nEach dataset entry appears as a separate describe block in test output:\n\n`text\n1.1. should add numbers [#0]\n ✓ 1. should print output\n1.1. should add numbers [#1]\n ✓ 1. should print output\n`\n\nWith key: id:\n\n`text\n1.1. should add numbers [addition]\n ✓ 1. should print output\n1.1. should add numbers [subtraction-like]\n ✓ 1. should print output\n`\n\n### Hooks with Datasets\n\nbeforeAll, afterAll, beforeEach, and afterEach hooks inside a dataset scenario run per entry — each entry gets its own describe block, so beforeAll runs once per entry, not once globally.\n\n---\n\n## Nested File Paths\n\nfile: blocks automatically create parent directories if they don't exist, and clean up only the directories that were created:\n\n``markdown\n`file:nested/deep/config.yaml\nhost: localhost\nport: 3000\n`\n\n`execute\ncat nested/deep/config.yaml\n`\n\n`expect\nhost: localhost\nport: 3000\n`\n``\n\n---\n\n## Test File Conventions and Best Practices\n\n- Each test file is Markdown and SHOULD begin with a top-level heading (# Title). The test runner uses the document structure to group scenarios and nested tests.\n- Use file:<filename> blocks to create fixture files. Place them in the scenario where they should be visible (parent level for nested tests to access).\n- Prefer small, focused execute blocks and multiple expect blocks to validate different aspects of output.\n- Use expect:partial for flexible substring tests, expect:regex for pattern matching, and expect:ignoreCase for case-insensitive comparisons.\n- Use timeout only when necessary for long-running commands to avoid hanging test suites.\n- Use beforeAll and afterAll for expensive setup/cleanup that applies to many tests; use beforeEach/afterEach for per-test isolation.\n- Avoid sharing state between sibling scenarios; rely on scoped file blocks and hooks to control visibility and cleanup.\n\n---\n\n## Examples Summary\n\nThe provided test files in this package demonstrate all primary features:\n\n- test/execute-expect.test.md — execute, expect, error, multi-line output\n- test/expect-partial.test.md — partial matching and wildcard usage\n- test/expect-ignorecase.test.md — case-insensitive matching\n- test/expect-regex.test.md — regex matching\n- test/expect-json.test.md — JSON formatting modifier\n- test/multiple-expects-test.test.md — multiple expectations per execute\n- test/file-scope.test.md — file blocks and scoping rules\n- test/hooks.test.md — beforeAll/afterAll (and hooks in general)\n- test/timeout.test.md — timeout block usage\n- test/add.test.md — programmatic test creation via aux4 test add\n- test/expect-similar.test.md — text similarity matching (fuzzy, cosine, jaccard)\n- test/expect-ai-score.test.md — LLM-as-a-judge scoring\n- test/output-json.test.md — JSON output flag\n- test/coverage.test.md — coverage report rendering and JSON format\n- test/file-nested-path.test.md — nested directory creation for file blocks\n- test/dataset-basic.test.md — basic dataset-driven tests\n- test/dataset-root.test.md — dataset with JSONPath root\n- test/dataset-nested.test.md — nested datasets (cartesian product)\n- test/dataset-key.test.md — dataset with key field for labels\n- test/dataset-objects.test.md — dataset with object values\n- test/dataset-file-syntax.test.md — dataset:file shorthand syntax\n\nYou can open each file to see complete, runnable test fragments. The README examples above copy the real tests so you can replicate them.\n\n---\n\n## Troubleshooting & Tips\n\n- If an expect fails, check which block failed and compare the captured stdout/stderr to the expected content.\n- Use expect:partial for tests where output contains timestamps, order-insensitive fragments, or variable content.\n- For complex output, prefer expect:regex with anchors and groups to precisely target the necessary parts.\n- If files are unexpectedly missing in a scenario, verify where the file: block is declared and that scoping matches the intended nested tests.\n- To programmatically add tests, use aux4 test add and then inspect the generated .test.md` file.\n\n---\n\n## License\n\nThis package is licensed under the Apache-2.0 License.\n\nSee LICENSE for details.\n"}