PC MISE

WIP: Database Docs

Terminology

Term	Definition
Leftover	Pieces from the bag from the previous PC. Ex: TSZ to solve PCO and have the leftover ILJO on 2nd
Build	Pieces within the setup
Cover Pattern	Extended pieces stating the when to build the setup. Ex: OQB may need S before Z for next step
Solve Pattern	Extended pieces stating what pieces to solve. May be related to cover pattern if cover pattern depends on pieces not in build

Setup ID

Goals

Effective order
Representative of setup
Uniqueness

Specification

The ID is a packed 48 bit hex string. Ex: 25b2eefaccff

The order of the packed data determines the order and grouping of the setups when sorted. For this database, the order of TILJSZO is used for pieces.

Section	Description
PC Number	1-9 value corresponding to 1st to 9th PC
OQB	Flag whether the setup is OQB setup
Duplicate Piece	Maps duplicate leftover piece to 0-7. No duplicate -> `0`, `T` -> 1, …, and `O` -> 7
Leftover Pieces	Inverted bitmap of existence of each piece in leftover in `TILJSZO` order
Solve Length	Number of pieces needed to solve. Equivalent to 10 - build length
4x	Flag whether there is 4 duplicate pieces in build
Build Pieces	Inverted counts of each piece in build in `TILJSZO` order with two bits per piece
Fumen Hash	Hash of the fumen into four bits
Cover Hash	Hash of the cover pattern into two bits
Unique ID	Decrement from 255 in case of collision for uniqueness

Hashing Functions

def fumen_hash(fumen: str) -> int:
    fumen_encoded = fumen[5:-8].encode() # take reasonably random section
    h = len(fumen_encoded) 
    for byte in fumen_encoded:
        h = ((h << 3) | (h >> 4)) & 0x7F # more mixing
        h ^= byte
    return h & 0b1111

def cover_pattern_hash(cover_pattern: str) -> int:
    cover_pattern_encoded = cover_pattern.encode() 
    h = len(cover_pattern_encoded)
    for byte in cover_pattern_encoded:
        h ^= byte
    return h & 0b11

These hash functions were based on xor of the characters in the strings. Modifications were made based on the entropy of the output leading to the above functions.

The cover hash is intended to distribute setups with same values except cover pattern for use case such as QB setups, so entropy is calculated on sets of these kinds of setups. However, there is likely bias as arbitrary modifications were made to increase specifically the entropies of these sets of setups as few sets in original dataset. The entropies were normalized by dividing by the maximium value, which is same as number of bits in hash: 4 for fumen hash and 2 for cover hash.

Group	Count	Entropy (Normalized)
All Original Fumens	4109	0.9979379436338721
All One Piece Fumens	162	0.9806935374824224
Setups starting with 50f2eff6f[0-3]	7	0.9751060324573734
Setups starting with 50f2eff6f[4-7]	7	0.9751060324573734

Example

PC Number	Leftover	Build	Cover Pattern	Fumen	OQB
2nd	TTIO	TIO	T,[TIO]!	v115@9gwhIewhIewhEewwAeRpwhDeywRpJeAgH	false

Detailed Conversion

Section	Description
PC Number	2nd -> 2 = `0010₂`
OQB	Not OQB -> `0₂`
Duplicate Piece	TTIO has T as duplicate piece. T -> 1 = `001₂`
Leftover Pieces	TTIO has the pieces TIO -> 1st, 2nd and 7th bit unset = `0011110₂`
Solve Length	Build TIO has 3 pieces. 10 - 3 = 7 = `0111₂`
4x	No 4 duplicate piece in build -> `0₂`
Build Pieces	Build TIO has 1 T, 1 I, and 1 O. Computing bitwise NOT or decrement from 3 gives `10₂` for TIO subsection and `11₂` for LJSZ subsection -> `10101111111110₂`
Fumen Hash	Run `fumen_hash` on the fumen to get 4 = `0100₂`
Cover Hash	Run `cover_pattern_hash` on the cover pattern to get 1 = `01₂`
Unique ID	No other existing setups in original dataset conflict for first 40 bits -> 255 = `11111111₂`

Convert to hex: 001000010011110011101010111111111001000111111111₂ -> 213ceaff91ff₁₆

OQB

The oqb_path and oqb_depth help with handling OQB setups. If a setup is not OQB, oqb_path and oqb_depth are NULL

On insert, the oqb_path may be set to the same value as setup_id to state setup is OQB. This is only necessary when setting solve_pattern to be NULL for setups that aren't intended to be solve but rather has a next setup.

The oqb_path is automatically populated as a materialized/label path from the setup_oqb_links table that contain the parent/previous setup for each setup. Similarly, oqb_depth is automatically populated for depth of the setup, equivalently the number of ancestor setups/nodes.

Cover Pattern

The cover pattern is in extended pieces notation and used to restrict what queues that the setup covers.

The cover pattern does not need to be exactly the correct coverage. The cover_data in statistics is a bitstring for which queue the setup is covered.

Setup Variants

Some setups, especially one-solve or QB setups, have effectively the same setup but some pieces may be placed before completing the base setup.

In the following image, the first page, page with least number of pieces, is the base setup. To increase the coverage of this setup, the S or O piece can be placed before finishing the base setup. These base setups with extraneous pieces are call variants of the setup.

Setups with S and O being extraneous to the base setup

Statistics Table

Since the values of a setup depends on the kicktable or have 180, this table separates the statistics for each kicktable.

Kicktable	Definition
srs	Super Rotation System from The Tetris Company
srs_plus	Tetrio's kicktable: srs with symmetric I spins and 180
srs180	srs with 180 spins
none	No kicks or 180 spins
srsx	Tetrio's kicktable: srs with more powerful 180 spins
ars	Arika Rotation System
arc	Kicktable made for Ascension supported in Tetrio
tetrax	Kicktable made for Tetris Legends supported in Tetrio

Path File

The path_file is a flag whether the path.csv file is stored on the server.

The file is stored with the filename <setup-id>-<kicktable>.csvd.xz, which is a compressed version of the file.

The basic idea on how to compress the file is to have the fumens keyed by a number then store the mapping at the beginning. In addition, remove columns that can be computed from the other columns, which the queues can be compressed into the extended pieces notation that generated it at the beginning of the file. Then apply xz or the underlying lzma for compression of the file. This converts the 1-3 MB file into 5-150 KB file.

The following function is the actual implementation.

/**
 * Compress path file
 */
export async function compressPath(filename: string, pieces: string): Result<Buffer> {
  const inputCsv = fs.readFileSync(filename, 'utf-8');
  const records: Record<string, string>[] = parse(inputCsv, {
    columns: true,
    skip_empty_lines: true
  });

  records.sort((a, b) => queueKey(a['ツモ']) - queueKey(b['ツモ']));

  const fumens: Record<string, number> = {};

  for (const row of records) {
    const rowFumens = row['テト譜'].trim().split(';');
    for (let i = 0; i < rowFumens.length; i++) {
      const f = rowFumens[i];
      if (!(f in fumens)) {
        fumens[f] = Object.keys(fumens).length;
      }
      rowFumens[i] = fumens[f].toString();
    }
    row['テト譜'] = rowFumens.join(';');
    row['未使用ミノ'] = mapSaves(row['未使用ミノ']).toString();

    delete row['ツモ'];
    delete row['対応地形数'];
    delete row['使用ミノ'];
  }

  const fumensKeys = Object.keys(fumens);
  const outputLines = [
    pieces,
    '',
    ...fumensKeys,
    '',
    stringify(records, {
      header: true,
      columns: Object.keys(records[0])
    }).trim()
  ];

  try {
    const compressed = await lzma.compress(Buffer.from(outputLines.join('\n')), { preset: 9 });

    // Validate output (though lzma-native should always return Buffer)
    if (!Buffer.isBuffer(compressed)) {
      return { data: null, error: new Error('Compression returned invalid data') };
    }

    return { data: compressed, error: null };
  } catch (err) {
    const error = err instanceof Error ? err : new Error(String(err));
    error.message = `Compression failed: ${error.message}`;

    return { data: null, error };
  }
}

Saves Table

This table is to store save data following the notation used in PC-Saves-Get stored in save column.

Priority save percent/fraction is values in an array of the priority of that save. Ex: T,I,O on 2nd and wanted to know percent of getting T then remaining percent for I then O. This corresponds with the --best-save flag on PC-Saves-Get