Tiny options for competitions

1. Introduction

A RKC (=competition) file is limited to 20480 bytes. It consists of a header with basic infos (80 bytes) and a YAZ0 compressed U8-archive with the track modifications. This is very small for self made competitions.

If a RKC becomes larger than 20480 bytes, it must be reduced. There are two ways to reduce the space of the YAZ0/U8 archive without removing objects:

Use a better YAZ0 compression.
Rearrange and change the uncompressed data to make it better compressible.

The SZS tools support both ways. And this guide explains how to reduce the size.

2. Concepts

Beginning with version 1.54, the SZS tools will support RKC files and will also provide some methods to reduce the size of compressed RKC files.

2.1 Optimize YAZ0 compression

YAZ0 compression searches the context for repeated data sequence. YAZ0 allow a back reference up to 4096 bytes before the current position. The maximum copied chunk size is 273 bytes.

The old compression levels 1 to 9 works all with the same method. For level 9 it is: From any point, compare the coming sequence of bytes with all 4096 possible places before. Use the longest sequence found. For sequences <3, make a byte copy. Then continue with the next position. The result is a good compression, but not so good as Nintendo did it.

Level 1 to 8 works with the same method, but reduce the number of compared sequences in steps to at least 256 bytes before the current position. So level 1 is about 16 times faster than level 9.

In 2011 I developed compression level 10, and now I have optimized it. If a back reference is found, try a single byte copy as alternative to the found back reference. Then compare both variants and use the better one. With this level 10, the SZS tools are able to compress tracks better than Nintendo did it.

Level 10 looks 1 step into the future. But what is, if an algorithm looks N steps into the future and decides then to use a single byte copy or the back reference? I realized this recursive algorithm with levels 100 to 150. For time optimization, I have implemented it as iterative variant with a stack. The least significant digits (00 to 50) defines the recursion depth. Levels 100 to 101 are insignificant and exist only to compare them with other levels. Level 120 is already time intensive and only manageable with small RKC files.

Theoretical, 1 level more means double time. But because of some optimizations 5 level more need usually 10 to 20 times as much time. So level 140 needs years for a standard (large) track.

And so the first step to minimize the compressed file is done!

2.2 Re-order objects

The next 2 optimizations are to change the data before compression to get smaller compressed data. This can be done by reordering the data (lossless) and by rounding values to get more repeated data sequences (see below).

There are 2 good candidates for reordering data without side effect. One is the order of KCL vertices and normals, and the other one the order of global objects. The SZS tools support both. The effect of reordering is to have more repeated sequences within the back reference window of 4096 bytes.

2.3 Round floating point values

The next optimization is to round floating point values to get more repeated data sequences instead of similar floats with only some changed bits. Good candiates are KMP positions for routes, where exact positions are not needed. So rounding to the nearest integer is possibe without any disadvantages.

Also KCL vertices (positions) and normals can be rounded. And sometimes they must be rounded (and Nintendo did it too) because the number of different values is limited. If rounding in lager steps, the compressed data becomes smaller. Here we talk about rouding some digits behind the comma.

3. Tiny levels

The SZS tools support tiny levels from 0 (off) to 7 (dangerous). Option --tiny defines the general level. With options --kcl and --kmp you can control the tiny level for KCL or KMP only.

Tiny levels
Keyword	Description
0 OFF
1 LOSSLESS
2 RECREATE
3 MINIMAL
4 GOOD
5 MEDIUM
6 EXTREME
7 DANGEROUS

4. Options

4.1 -C level | --compr level

Define a compression level between 0..9: 0 (or NOCHUNKS) disables finding chunks, 1 (or FAST) for the fastest standard compression and 9 (or BEST) for the best (=default) compression. 10 (or ULTRA) and 100-150 are special time-consuming compression modes. They are dedicated to competitions with size limitations. The special value UNCOMPRESSED acts like option --no-compress. Because of many repeated data, the best bz-compression mode varies. Therefor the levels TRY2..TRY5 (or short T2..T5) are defined to find the best compression mode with testing the first N modes of 9, 1, 8, 2, 5. TRY2 is used for normalizing. For non bz-compression, TRY* are the same as BEST. Do not use compression level >6 for LZMA if the file is intended for Mario Kart Wii, as too much memory is required for decoding. Option --norm takes precedence over --compr and sets the compression level for bzip2 to TRY2 (BEST) and for LZMA to 6. For more modes and details type wszst -C list. To force colorized output type wszst -C clist.

4.2 --tiny mode

Set tiny mode to a level between 0 and 7.

4.3 --kcl list

Set global options for KCL processing. Parameter 'list' is a comma separated list of keywords. A minus sign before a keyword disables a setting. Each occurrence of the option will only change entered settings and all other settings are untouched. Keyword DEFAULT resets the default settings and CLEAR disables all. Keywords SMALL, MEDIUM and CHARY select the default parameters for the octree creation. The other allowed keywords are: FAST, NEW, CENTER, ROUND, NORMALS, MTL, WIIMM, TRIANGLES, OUT-SWAP, G, USEMTL, CLIP, IN-SWAP, AUTO, HEX4, HEX23, HEX, DROP-UNUSED, DROP-FIXED, DROP-INVALID, DROP, RM-FACEDOWN, RM-FACEUP, FIX-ALL, TINY-0 ... TINY-7, CONV-FACEUP, WEAK-WALLS, SORT, INPLACE, SILENT and LOG.

4.4 --kmp list

Set global options for KMP processing. Parameter 'list' is a comma separated list of keywords. A minus sign before a keyword disables a setting. Each occurrence of the option will only change entered settings and all other settings are untouched. Keyword DEFAULT resets the default settings and CLEAR disables all. The other allowed keywords are: FORCE, NEW, RM-SPCITEM, LEFT, RIGHT, WIDE, NARROW, FIX-CKPH, FIX-ENPH, FIX-ITPH, FIX-PH, FIX-CKNEXT, FIX-CKJGPT, FIX-CK, FIX-ALL, MASK-PFLAGS, RM-LECODE, PURGE-GOBJ, FULL-DEFOBJ, DUMP-CLASS, DUMP-ONEWAY, DUMP-ALL, 1LAP .. 9LAPS, MAX-LAPS, RM-EMPTY, TINY-0 .. TINY-7, INPLACE, SILENT and LOG.