ESpec (Encoding Specification) Documentation

Overview

ESpec is a domain-specific language used throughout NGDP for specifying BLTE encoding instructions. It defines how content blocks are compressed, encrypted, and structured within BLTE containers. ESpec appears in patch configurations, encoding files, and BLTE block headers.

Grammar Components

ESpec uses single-character identifiers for encoding operations:

Basic Encodings

n: Plain/uncompressed data
z: Zlib compression
e: Encryption
b: Block-based encoding
c: BCPack compression
g: GDeflate compression

Encoding Combinations

ESpec supports nested and sequential encoding operations through composition.

Block Syntax

Size Specifications

Block sizes support unit suffixes:

K: Kilobytes (1024 bytes)
M: Megabytes (1024 * 1024 bytes)
No suffix: Bytes

Count Specifications

Block counts can be:

Exact number: Specific block count (e.g., 3)
Variable: Asterisk (*) for variable block count
Dynamic sizing: Block count of zero with an average block size. Block boundaries are determined dynamically based on content. Distinct from variable (*) block count.

Block Format

b:{size[*count]=encoding}

Components:

size: Block size with optional unit suffix
count: Block count (optional, defaults to 1)
encoding: Encoding specification for blocks

Grammar Reference

Simple Encodings

plain := "n"
zlib := "z" [ ":" ( level | "{" zlib_params "}" ) ]
zlib_params := ( level | variant ) [ "," ( variant | window_bits ) ] [ "," window_bits ]
encryption := "e" ":" "{" key "," iv "," content_encoding "}"

Zlib supports multiple syntax forms: z, z:9, z:{9}, z:{9,mpq}, z:{9,15}, z:{9,mpq,15}, z:{mpq}, z:{mpq,15}. The second parameter can be either a variant name or a numeric window_bits value.

Block Encoding

block := "b" ":" ( "{" block_spec { "," block_spec } "}" | encoding )
block_spec := size [ "*" count ] "=" encoding
size := number [ unit ]
unit := "K" | "M"
count := number | "*"

A block table can omit braces when it contains a single encoding with no size specification: b:z is equivalent to a single block with no explicit size.

Complex Encodings

encoding := plain | zlib | encryption | block | bcpack | gdeflate
bcpack := "c" [ ":" "{" bcn "}" ]
gdeflate := "g" [ ":" "{" level "}" ]

Examples

Simple Block Encoding

b:{495=z,9673=n}

This specifies:

First block: 495 bytes, zlib compressed
Second block: 9673 bytes, uncompressed

Variable Block Sizes

b:{16K*=z}

This specifies:

Variable number of 16KB blocks
All blocks use zlib compression

Encrypted Blocks

b:{256K*=e:{key,iv,z}}

This specifies:

Variable number of 256KB blocks
Each block is encrypted with specified key and IV
Content is zlib compressed before encryption

Compression Levels

b:{16K*=z:{6,mpq}}

This specifies:

Variable number of 16KB blocks
Zlib compression level 6
MPQ-compatible compression settings

Mixed Block Types

b:{1K=n,4K*=z,2K=n}

This specifies:

First block: 1KB uncompressed
Variable number of 4KB zlib-compressed blocks
Final block: 2KB uncompressed

Zlib Compression Levels

Level Specification

Zlib compression supports level, variant, and window bits parameters:

z:{level}
z:{level,window_bits}
z:{level,variant}
z:{level,variant,window_bits}

Standard Levels

Valid levels are 1-9:

1: Fastest compression
6: Default compression (balance of speed/size)
9: Maximum compression

Level 0 is not accepted.

Variant Specifications

mpq: MPQ-compatible compression settings
zlib: Standard zlib settings
lz4hc: LZ4HC-compatible compression settings

Window Bits

Zlib window bit count can be specified in range [8, 15]. Two values can be provided (must match). Default is 15.

Compression Examples

z:{1}           # Fast compression
z:{9}           # Maximum compression
z:{6,mpq}       # MPQ-compatible level 6
z:{6,zlib,15}   # Zlib variant with explicit window bits

Encryption Specification

Format

e:{key,iv,content_encoding}

Components

key: Encryption key identifier or value
iv: Initialization vector
content_encoding: Encoding applied before encryption

Key Format

Keys must be exactly 16 hex characters (8 bytes):

e:{0123456789abcdef,fedcba98,z}

This specifies:

Encryption key: 0123456789abcdef (16 hex chars, 8 bytes)
IV: fedcba98 (8 hex chars, 4 bytes)
Content: zlib compressed before encryption

The parser rejects keys that are not exactly 16 hex characters. The IV must be exactly 8 hex characters (4 bytes).

BCPack Compression

BCPack Usage

c
c:{3}

BCPack compression uses a proprietary algorithm optimized for specific content types. An optional BCn (block compression number) parameter selects the mode, in range [1, 7]:

bcpack := "c" [ ":" "{" bcn "}" ]

Block-Based BCPack

b:{64K*=c}
b:{64K*=c:{5}}

Variable 64KB blocks using BCPack compression.

GDeflate Compression

GDeflate Usage

g
g:{6}

GDeflate is a GPU-accelerated deflate variant designed for DirectStorage. An optional compression level parameter can be specified in range [1, 12]:

gdeflate := "g" [ ":" "{" level "}" ]

Block-Based GDeflate

b:{32K*=g}
b:{32K*=g:{8}}

Variable 32KB blocks using GDeflate compression.

Usage Contexts

PatchConfig Files

ESpec appears in patch-entry lines:

patch-entry = source_hash target_hash size espec

Example:

patch-entry = 1234567890abcdef abcdef1234567890 524288 b:{16K*=z}

Encoding Files

Encoding files use ESpec for content encoding specifications:

content_key encoded_key size espec

BLTE Data Blocks

BLTE headers contain ESpec for block processing instructions:

graph TD
    A[BLTE Header] --> B[Block Count]
    A --> C[ESpec]
    C --> D[Block 1 Processing]
    C --> E[Block 2 Processing]
    C --> F[Block N Processing]

Parser Implementation

Tokenization

ESpec parsing requires tokenization of:

Identifiers: Single characters (n, z, e, b, c, g)
Numbers: Decimal integers
Units: Size suffixes (K, M)
Delimiters: Braces, colons, commas, equals, asterisks

Grammar Rules

#![allow(unused)]
fn main() {
// Example parser structure
enum ESpec {
    Plain,
    Zlib { level: Option<u8>, variant: Option<String> },
    Encryption { key: String, iv: String, content: Box<ESpec> },
    Block { specs: Vec<BlockSpec> },
    BCPack,
    GDeflate,
}

struct BlockSpec {
    size: u64,
    count: BlockCount,
    encoding: ESpec,
}

enum BlockCount {
    Exact(u32),
    Variable,
}
}

Error Handling

Common parsing errors:

Invalid identifier characters
Malformed block specifications
Missing required parameters
Invalid size or count values
Unbalanced braces or parentheses

Validation Rules

Size Constraints

Block sizes must be positive integers
Maximum block size typically limited to several MB
Minimum block size typically 1 byte

Count Constraints

Block counts must be positive integers when specified
Variable count (*) requires size specification
Total content size must be consistent

Encoding Constraints

Encryption requires valid key and IV lengths
Compression levels must be within algorithm-specific ranges
Nested encodings must be logically valid

Performance Considerations

Block Size Selection

Block sizes depend on usage:

Small blocks (1-4KB): Better for streaming, higher overhead
Medium blocks (16-64KB): Balanced performance
Large blocks (256KB+): Better compression ratios, higher memory usage

Compression Algorithm Selection

Algorithm characteristics:

zlib: Universal compatibility, good compression
BCPack: Optimized for specific content types
GDeflate: Fast compression with good ratios
None (n): Maximum speed, no space savings

Memory Usage

#![allow(unused)]
fn main() {
// Example memory-efficient processing
fn process_blocks(espec: &ESpec, data: &[u8]) -> Result<Vec<u8>> {
    match espec {
        ESpec::Block { specs } => {
            let mut output = Vec::new();
            let mut offset = 0;

            for spec in specs {
                let block_data = &data[offset..offset + spec.size as usize];
                let processed = process_encoding(&spec.encoding, block_data)?;
                output.extend(processed);
                offset += spec.size as usize;
            }

            Ok(output)
        }
        // Other encoding types...
    }
}
}

Common Patterns

Streaming-Optimized

b:{16K*=z}

Small, consistent block sizes for streaming applications.

Storage-Optimized

b:{1M*=z:{9}}

Large blocks with maximum compression for storage efficiency.

Mixed Content

b:{4K=n,64K*=z,4K=n}

Headers and footers uncompressed, bulk content compressed.

Encrypted Streaming

b:{32K*=e:{key,iv,z:{6}}}

Moderate block sizes with encryption and balanced compression.

Debugging and Validation

ESpec Validation

#![allow(unused)]
fn main() {
fn validate_espec(espec: &str) -> Result<ESpec, ESpecError> {
    let parsed = parse_espec(espec)?;
    validate_constraints(&parsed)?;
    Ok(parsed)
}

fn validate_constraints(espec: &ESpec) -> Result<(), ESpecError> {
    match espec {
        ESpec::Zlib { level: Some(level), .. } if *level > 9 => {
            Err(ESpecError::InvalidCompressionLevel(*level))
        }
        ESpec::Block { specs } if specs.is_empty() => {
            Err(ESpecError::EmptyBlockSpec)
        }
        // Additional validation rules...
        _ => Ok(())
    }
}
}

Round-Trip Testing

#![allow(unused)]
fn main() {
#[test]
fn test_espec_round_trip() {
    let original = "b:{16K*=z:{6}}";
    let parsed = parse_espec(original).unwrap();
    let serialized = serialize_espec(&parsed);
    assert_eq!(original, serialized);
}
}

Integration Examples

BLTE Block Processing

#![allow(unused)]
fn main() {
fn process_blte_block(espec: &ESpec, input: &[u8]) -> Result<Vec<u8>> {
    match espec {
        ESpec::Plain => Ok(input.to_vec()),
        ESpec::Zlib { level, .. } => decompress_zlib(input),
        ESpec::Encryption { key, iv, content } => {
            let decrypted = decrypt(input, key, iv)?;
            process_blte_block(content, &decrypted)
        }
        ESpec::Block { specs } => process_block_specs(specs, input),
    }
}
}

Patch Application

#![allow(unused)]
fn main() {
fn apply_patch_with_espec(
    source: &[u8],
    patch: &[u8],
    espec: &ESpec
) -> Result<Vec<u8>> {
    let processed_patch = process_blte_block(espec, patch)?;
    apply_binary_patch(source, &processed_patch)
}
}

Reference Implementation

Complete Parser

#![allow(unused)]
fn main() {
use nom::{
    branch::alt,
    bytes::complete::tag,
    character::complete::{alphanumeric1, char, digit1},
    combinator::{map, opt},
    multi::separated_list0,
    sequence::{delimited, preceded, separated_pair, tuple},
    IResult,
};

pub fn parse_espec(input: &str) -> IResult<&str, ESpec> {
    alt((
        parse_plain,
        parse_zlib,
        parse_encryption,
        parse_block,
        parse_bcpack,
        parse_gdeflate,
    ))(input)
}

fn parse_plain(input: &str) -> IResult<&str, ESpec> {
    map(char('n'), |_| ESpec::Plain)(input)
}

fn parse_zlib(input: &str) -> IResult<&str, ESpec> {
    map(
        tuple((
            char('z'),
            opt(preceded(
                char(':'),
                delimited(
                    char('{'),
                    separated_pair(
                        digit1,
                        opt(char(',')),
                        opt(alphanumeric1)
                    ),
                    char('}')
                )
            ))
        )),
        |(_, params)| match params {
            Some((level, variant)) => ESpec::Zlib {
                level: level.parse().ok(),
                variant: variant.map(|s| s.to_string()),
            },
            None => ESpec::Zlib { level: None, variant: None },
        }
    )(input)
}
}

Implementation Status

Rust Implementation (cascette-formats)

ESpec parser:

Plain (n) - Uncompressed content
ZLib compression (z) - Level [1,9], variant (mpq/zlib/lz4hc), window bits [8,15]; all optional, 3-param syntax supported
Encryption (e) - Key, IV, and nested content encoding
Block-based (b) - Variable and fixed block specifications
BCPack (c) - Optional BCn version [1,7]; bare c accepted
GDeflate (g) - Optional level [1,12]; bare g accepted

Parser Features:

Safe integer casting with try_from to prevent truncation
Display trait implementation for round-trip string conversion
Test suite covering production ESpec patterns and edge cases
Integration with BLTE and Encoding file processing

Analysis and Validation

ESpec patterns are validated across all CASC formats to ensure correct parsing and processing of compression and encryption specifications.

Keyboard shortcuts

cascette-rs