#474 Chunk-based updater reverse engineering

Size      Description
4         ?
1         file 1: name length -> "fnlen"
fnlen     file 1: name (relative to the game root)
1         ? 0x12
1         ?
1         ? 0x0A
1         file 1 chunk 1: meta length -> "cmlen1", generally = 0x31
cmlen1    file 1 chunk 1: meta name
             Format: "<hash of length 17>_<hash of length 32>"
1         ? 0x12
1         file 1 chunk 1: hash length -> "chlen1", generally = 0x20
chlen1    file 1 chunk 1: hash (?)
1         ? 0x18 or 0x20
21        ?

1         file 1 chunk 2: meta length
    ...

?         file 2: name length

## Context The devs are currently A/B testing a new method to update the game more efficiently by using chunks. This is noticeably more complex than the conventional method, hence this issue. ## Motivation The current install script is far from perfect but does serve its purpose for those who would like to use it. If this chunk-based method becomes mandatory, other 3rd party installers/launchers could benefit from any open-source implementation that is available. ## API documentation (WIP) Tree view of the data structure: * json * Similar to the known `resource` * manifest file * File names and patch offsets (?) * chunk data * Also zstd-compressed * Unknown relation with the files generated by the launcher **Main JSON** * see `updater/sophon/updater.py` **Manifest file** * see `updater/sophon/updater.py` * (old) Analysis of the Protobuf data: (in bytes, starting from offset 0) ``` Size Description 4 ? 1 file 1: name length -> "fnlen" fnlen file 1: name (relative to the game root) 1 ? 0x12 1 ? 1 ? 0x0A 1 file 1 chunk 1: meta length -> "cmlen1", generally = 0x31 cmlen1 file 1 chunk 1: meta name Format: "<hash of length 17>_<hash of length 32>" 1 ? 0x12 1 file 1 chunk 1: hash length -> "chlen1", generally = 0x20 chlen1 file 1 chunk 1: hash (?) 1 ? 0x18 or 0x20 21 ? 1 file 1 chunk 2: meta length ... ? file 2: name length ``` **EDIT 2024-04-25:** Received a proto definition and further explanation. Currently clarifying licensing matters. **EDIT 2024-04-26:** First script prototype uploaded **Chunk data** (segments?) * Source: *Main JSON*. URL: * `".data .manifests[] .chunk_download .url_prefix"` + `"/"` + `<hash of length 17>_<hash of length 32>` * File format: zstd-compressed * `zstd -d FILENAME -o output` **Chunk data (launcher)** * Source: unknown. Possibly assembled or renamed from the section above * Destination: GAMEDIR/chunk/ * File name: (32 bytes), hash * `_tmp` is used during download. ---- If you have any relevant information or hints - feel free to leave a comment below or send me an email.