Shattered Memories / History of Violence

Following is an compilation from numerous posts i made @redump.org concerning PSX dating from 2008 till 2011, which i tried, to best of my abilities, dress in somewhat coherent narration. The thing is, i did some interesting stuff during last few years, but it's all scattered across this electronic wasteland. I kinda do regreat it now, should have noted those things in one place. So, what i'll do now, i will try to bring it over here, what i still remember, and write guides and shit. Reason behind this is - currently i'm working on some PSX JPN->ENG translations, a project of a sort. You'll see. I just wanted to give a kind of a background to that - how things got started and what could be interesting to other people relating to those translations.

I was messing around Saturn's protection at that time but weren't really interested that much in this system itself. And so, as time passes, i realize, that i actually know little about PSX, which on the other hand i used and liked a lot. Even though this was the system that startet redump.org and we worked with it for quite some time there still were many questions left without answers. So i started looking for information on it and eventually stumbled into this thread, where some of redump.org members were dumping and cataloguing PSX BIOSes, which was neat, i though. But i didn't have hardware to do this, and was low on cash then. But thing is, i was really into CRCs, since i was doing algorithm reversing on LiteOn's firmware (i'll probably write a guide on CRC reversing too, sometime, when i'll take a break from those translations). So, even though i haven't previously done any programming for consoles, i was programming in assembler and thought maybe i could write a program that would just calculate CRCs on the flight, without a necessity to physically transfer BIOS to PC. And so i did. I found SDK and some guides and i did. But what i also found was this, this and this. CDINFO.TXT described undocumented service commands one can access in PSX CD unit. And that was so interesting i hand to try it, so i expanded this PSX program to have ability to send those commands.

3rd page (Japanese) was for relatively new PSX emulator XEBRA, where it's author Dr.Hell would document results of his research. Now, with this program i could test PSX CD unit on quite low level and compare results between hardware & emulators. And what i noticed was, how XEBRA was different from rest of the emulators. It acts really close to hardware, whereas usually emulators would go for simplest solution, e.g. just return some constants and such - skipping a lot of processing. And i thought at first this was issue with plugins. So i decide to make my own CD plugin. And i did. Though, i had quite some plans for it, but as i made it, i came to understanding how limited those plugins really are. Issue was with emulator architecture. They are hackish - made of shortcuts - layers upon layers of shortcuts. And with this realization i lost interest in developing it further. I did quite a lot of fine-tuning for it, though, and it might be, it's the best for certain situations: it's the only one that works with physical drives connected to my motherboard (IDE2USB); overall catching strategy is good, comparable to cdrsapu and it gave best results reading scratched CDs on PCs i tested. So, you could give it a try, maybe, if you use plugin-based emulators. What i would suggest, though, is: stick with XEBRA. It's the best PSX emulator there is, hands down.

What i did with this plugin, though, since i could recompile it as i liked, i made it so it would ease examination of LibCrypt protected titles. I disassembled / modified them and ran through this plugin to log certain routines. And what i found was, how this protection really works. It turns out actual subchannel content doesn't matter at all - it can't even be read fully. There just have to be error in frame, so it would get replaced with previous frame - that's how this system is built. Always same 16 pairs of addresses are read. From those pairs 16bit key is formed. If both frames from pair are moded, bit is set.

Reading sector 13955... original sector \ bitF = 0
Reading sector 13960... original sector /
Reading sector 14081... original sector \ bitE = 0
Reading sector 14086... original sector /
Reading sector 14335... LibCrypt, LC1 sector \ bitD = 1
Reading sector 14340... LibCrypt, LC1 sector /
Reading sector 14429... LibCrypt, LC1 sector \ bitC = 1
Reading sector 14434... LibCrypt, LC1 sector /
Reading sector 14499... LibCrypt, LC1 sector \ bitB = 1
Reading sector 14504... LibCrypt, LC1 sector /
Reading sector 14749... original sector \ bitA = 0
Reading sector 14754... original sector /
Reading sector 14906... LibCrypt, LC1 sector \ 1
Reading sector 14911... LibCrypt, LC1 sector /
Reading sector 14980... original sector \ 0
Reading sector 14985... original sector /
Reading sector 15092... original sector \ 0
Reading sector 15097... original sector /
Reading sector 15162... LibCrypt, LC1 sector \ 1
Reading sector 15167... LibCrypt, LC1 sector /
Reading sector 15228... LibCrypt, LC1 sector \ 1
Reading sector 15233... LibCrypt, LC1 sector /
Reading sector 15478... original sector \ 0
Reading sector 15483... unknown         /
Reading sector 15769... LibCrypt, LC1 sector \ 1
Reading sector 15774... LibCrypt, LC1 sector /
Reading sector 15881... LibCrypt, LC1 sector \ 1
Reading sector 15886... LibCrypt, LC1 sector /
Reading sector 15951... original sector \ 0
Reading sector 15956... original sector /
Reading sector 16017... original sector \ bit0 = 0
Reading sector 16022... original sector /

So for given example key would be: 0011 1010 0110 1100 = 0x3a6c (it's PAL 'Dino Crisis' btw). Also it's pretty easy to derive those values from redump.org DB: those kept would be ones and missing sectors would be 0, e.g. it's cec1 for 'Ape Escape', 89ea for 'Mulan', 096f for 'Speed Freaks', e728 - 'Crash Bash', 90af - 'Italian Job', and so on.

Also it became apperant that there is a problem with .sbi files. They work but this format, it's clumsy - not thought out well. They store only part of Q channel frame. So it's not good for preservation. Data gets lost in DB to .sbi file transfer.

Another interesting thing to note is that some anti-mod chip protections would branch depending on console region. For example this routine checks letter @ BIOS offset 0x7ff52 and bails out from following checks if it's 'E' (Europe).

It's curious. Haven't explored it further though.

Extraction of PSX CD images from PSP EBOOT.PBP container.

Definition of problem: We need to extract one or several PSX CD images that are encapsulated inside of PSP EBOOT.PBP file.

It's straightforward with files containing single image. You'll need latest version of PSX2PSP [i'm working with v.1.4.2.] Launch this application with 'PSX2PSP.exe' and you should see dialog similar to this:

So, what you'd do, you'd click on ellipsis following 'ISO/PBP File' combobox and select .pbp file. [You might receive 'I/O error 998.' warning now. If so, some of textboxes might not fill properly, but, despite this, extraction should carry out just fine afterwards.] Then click 'Extract ISO', fill in output file name and click 'Save'. You should see extraction progress reported at the bottom of dialog now. Wait until it reaches 100% and you're done.

For extraction of multiple images you'll need hex editor as well [For actual editing i use MadEdit, as it supports SHIFT-JIS encoding, which is useful for translations. Otherwise it's quite basic editor. But most of the time it's enought with just FAR manager's built-in viewer. So, just stick with your favorite one.]. And some means to binary manipulate (split/join) files [being more comfortable with command-line than GUIs, i use my own programs for this purpose but your hex editor probably has this functionality].

Example 1: Strider Hiryuu 1 & 2 (SLPS-02620/SLPS-02621) Size: 284448402 CRC32: e75392d3

Extract 1st image just the same as single image. Search for 'PSTITLEIMG' in .pbp container and cut this file here (before this string), to separate header [remove --size=$10000 --direction=left hdr EBOOT.PBP]. Search and cut before each 'PSISOIMG' [remove --size=$8000 --direction=left crap EBOOT.PBP; remove --size=$1cf0000 --direction=left cd1 EBOOT.PBP]. What's left in .pbp now is CD2 and footer. Join it with header [copy /b hdr+EBOOT.PBP CD2.PBP] and extract as single image container.

Example 2: Dancing Blade Katte ni Momotenshi II: Tears of Eden (SLPM-86210/SLPM-86211/SLPM-86212) Size: 1343468306 CRC32: 94125c2e

Extract 1st image. Separate header [remove --size=$10000 --direction=left hdr EBOOT.PBP]. Cut at each 'PSISOIMG' [remove --size=$8000 --direction=left crap EBOOT.PBP; remove --size=$1eec0000 --direction=left cd1 EBOOT.PBP; remove --size=$19ba8000 --direction=left cd2 EBOOT.PBP]. Prepend header to remaining .pbp file (last CD + footer) [copy /b hdr+EBOOT.PBP CD3.PBP] and extract it. Search EBOOT.PBP file for 'STARTDAT' and separate footer from there [sextract --offset=$176baac0 EBOOT.PBP]. Add header + cd2 + footer [copy /b hdr + cd2 + EBOOT.xtr CD2.PBP] and extract it.

Would there be some difficulties extracting middle CDs this way, try removing trailing null characters (0x00) from data fragments containing images so, that last meaningful byte would be followed by just enought null characters to reach first round hex offset (one, that ends with 0 - divides with 16 without remainder) and hex-edit those null characters to zeroes (0x30) [for an example take a look at last .pbp fragment, how footer is attached to last CD image.].

Restoration of CD Images by Checksums and Size.
(Zem Asfalta Ir Pludmale)

v0.2

#0.0 - Foreword
#1.0 - Importance of multiple samples
#2.0 - Audio tracks
#2.1 - Fixing by comparison
#2.2 - Pattern regeneration
#3.0 - Data tracks
#3.1 - System area
#3.2 - Primary Volume Descriptor
#3.3 - Root Directory Record
#3.4 - Patches
#3.5 - EDC/noEDC
#3.6 - Damaged files
#3.7 - Different versions
#3.8 - Other things to try
#4.0 - Version history


#0.0 - Foreword

It's surprising how little information sometimes could be sufficient to reverse changes in files with certain defined structure, assuming you know this structure well and can make judgment where these changes occured. We'll take a look at CD images here, particularly  PSX images, but similar principles can be applied to more general cases or different data  altogeather.
Definition of problem: there are given sizes and checksums (CRC32) for CD image divided into tracks; one or more tracks of our image do not match to etalon by those given values.


#1.0 - Importance of multiple samples

Very important part of described process is in obtaining multiple samples. While it's not always a necessity, it would ease next steps tremendously, opening up more possibilities, and, could be, this new sample is good right away, thus eliminating a need for further processing at all. Possibly you could just get a CD. Judge time/money. If you restore some images, but it takes a lot of time, maybe wiser choice might have been to part with some cash. Your time has a value, do not forget it. However, if purchasing is not an option, often something can be found online: forums (do not ignore foreign forums, such smaller and isolated communities could actually be most interesting ones), file hosting services (use specific search engines, like filestube.com, filesearch.ru, etc., for those), usenet, irc, p2p (torrents, emule, dc, share, winny,  etc.) - you know the drill.
[Specific source for PSX images is PSP converted data (EBOOT.PBP). You can look for them too -  i'll try to cover process of extraction of such embedded images later in a separate guide.]


#2.0 - Audio tracks

Now, regarding audio tracks. There is little you can do with audio tracks if you do not have multiple samples. This data is just raw audio signal so correction is done by comparison of two samples and elimination of erronous data. Also you should understand what audio offset is and how to compensate it (read guides and forum posts at redump.org; however, do not use psxt001z 'track' command - it's slow and broken; you could use fff in it's stead) and be familiar with CD image basics - how to convert from one format to another, split tracks apart, recreate gaps - such things. Usually there shouldn't be any data in first gap (data->audio), if you see sector headers here or some garbage, try removing it so it would be all clean up until audio kicks in. Most of the time those are just software/hardware artifacts. Also worth mentioning is that usually audio data on PSX CDs should start at hex offset 0x56220 (if you keep tracks with gaps at the beginning, that is - a la redump.org), i.e. right after gap (352800 / 2352 = 150). So, if for some reason you are unable to determine audio offset of image, try just aligning data this way.


#2.1 - Fixing by comparison

So i'll assume now that you do have at least 2 different samples (when offset corrected / aligned similary) of audio tracks, that do not match to given checksum values. Do a binary compare on them to find first difference: 'fc /b "Track 02.001" "Track 02.002" |more', e.g.:

Comparing files Track 27.bin and TRACK 27.BAD
001A72B0: C2 62
001A72B2: 9B 61
001A72DE: 37 C0
001A72DF: 05 04
001A72E6: 90 F8
001E0B44: 1F CF
001E0B45: F1 F0
001E0B4C: 5E 45
001E0B4D: F7 F6
...

Locate 1st returned offset in hex editor. What you'll see most of the time is something similar to this:

Audio data in 2nd image was interpolated (red). Interpolation is most common strategy employed by CD readers to mask audio reading errors. You can tell it took place if values (int16) are close to ones calculated by adding previous and next audio samples (green) and dividing sum by two.
($1cea + $19da)/2 = $1b62
($12e2 + $11e0)/2 = $1261
($0725 + $025d)/2 = $04c1
($025d + $fd94)/2 = $fff8
So you'll cut this part out of affected file and replace it with good part from the other one. To do this i use my own commandline programs reMove and sExtract, but you're free to choose your own. Ther's rarely more than 3 sectors in a row affected, so you can either go for that or determine exact size by offsets, e.g.: it's 001A72B0..001A72E6 for case illustrated above. Afterwards you'll proceed to next difference and so on.


Other cases might be: mute, noise and either positive or negative desync. It's easy to spot mute - part of audio is replaced with silence. Noise looks somewhat similar to interpolation - few bytes would differ here and there, except that those values make no sense: they break audio wave pattern. Desync is when reader skips a little back, thus  adding extra data, or skips ahead, cutting some data out. You can tell, when all data starting from reported offset looks different. Search for this data from 1st file in itself and 2nd file, and vice versa. This way you should be able to determine nature of those changes, e.g.: if you'd find same patter that is in one of files a little back, then drive skipped back here, hence data gets repeated; if you'd find pattern from one of files in other, but little further ahead, then drive skipped forth, jumping over some of data. Seldom you might run into data from older CD readers where similar patterns can be observed, but, not on audio sample level - on sector level. Just the same - apply neccessary corrections to corrupted track fragment and proceed forth.

If you're unable to determine which file has erroneus data but there aren't too many differences, brute-force approach could be applied. Commandline program reCombine would just run through all possible combinations of two files, looking for certain CRC value.

If you could not obtain alternative image to your own, having different tracks to those in need of corrections, still, there is a hope. Sometimes there are common tracks, that could be found in other games / other versions of same game (even demos and such or, if game has multiple CDs, often some of data repeats there) / releases of same game for different regions - check for that - e.g., if you're using redump.org db for reference, get latest .dat and look for track's CRC in it with text editor's search function. Same game might be released for different system, e.g. Saturn, Dreamcast, NeoGeo, etc, check there. When checking different systems / regions, remember that game's title might have changed. Also offset can be different - compare track sizes - if they're somewhat close, get this version anyway and try to locate some data from your track there with hex editor (i use FAR's internal viewer F3 or, to locate larger fragments of data, when few bytes yield too many hits, my own program - fff.). Sometimes fragments of audio would repeat within track itself [restored SLPS-00131's, SLPS-00187's, SLPS-00391's & SLPS-01782's last tracks using parts of themselves as alternate samples]. Sometimes a part of audio data or even whole track is mirrored in other track - search other tracks for hex string from damaged track [in image of SLPS-00340 i had, Track 14 could be deconstructed to fragments of Track 04 and Track 13, except 4 bytes, that would be odd. When Track 14 were recreated again from those fragments, omitting spare bytes, it matched checksum; End of SLPS-00572's Track 14  could be restored from Track 11 ]. It also might be that audio track is mirrored on data track as dummy file [restored last track of SLPS-01441 this way].


#2.2 - Pattern regeneration

Sometimes whole track consist of certain pattern. Most frequent subset of such cases is dummy tracks (all 0x00). They can be generated with psxt001z command 'gen'. It could also be a different pattern, e.g.: all 0xff's. Most likely you'll need to do some programming to recreate such tracks. Sometimes ther's certain pattern only in a specific part of the track, very end most of the time. With some programming it could be restored too. For example, if audio at the very end of last track is cut off because of offset, but last bytes that remain form a constant 0xff pattern, you would first align this track right, filling missing data with 0x00 and then write a program that would expand 0xff further, replacing one byte of 0x00 at a time, checking CRC on each step. It's because data seldom lasts until very end. Few tens of last bytes are usually 0x00.

#3.0 - Data tracks

More dependencies and interconnections in data tracks (read ECMA-130/ECMA-119). This allows flexibility we couldn't have with audio and often running ECC check/fix is all it would take to reverse unwanted changes. You can use CDMage for this, but you'll notice it's slow and not very practical, so, i use program of my own. If you do not have CD programming experience, good place to start is Kris Kaspersky book 'CD Cracking Uncovered', which also includes several code examples for data regeneration from ECC. Same as with audio, though, the more samples you have, the more prepared you are - the better: other CDs from multi CD games, other versions (demos too), other regions - it's all good. Also please note, that i'll be talking about RAW (2352 byte) sectors here, unless specified otherwise.

#3.1 - System area

First 16 sectors of CD's user data area (volume space) are reserved - unused on PCs but filled with some specific data on consoles. On PSX ther's license and logo stored in there. There are about 3 - 4 variations of this data for each region. If actual data doesn't match to any of common patterns, it's modified most likely. Such modifications are usually deliberate replacement of logo or removal of license. psxt001z will check system area automatically, though, it has only most common values build in and hence can report false modifications on some occasions. Another program is reLicense, which you can configure and add as many system area checksums as you like, and it would then calculate CRC32 of specified image as if it would have those defined system areas. Replace modified license with good one from some other image.

#3.2 - Primary Volume Descriptor

PVD is located right after system area. As this structure contains some text fields, modifications here are usually made for branding. Most common ones i encountered were by Extremegames and R18; once in a while some odd ones, but usually it's one of these two. AFAIK both of those groups mostly operated for Japan region. Modified fields could include: subheader 00 00 09 00 -> 00 00 08 00; System Identifier (@0x20) removed (replaced with spaces); Volume Identifier (@0x40) removed, added or replaced; Volume Creation Date (@0x345) zeroed; '$' symbol (0x24) after Volume Creation Date (@0x355) set to 0x00; text string inserted in some of text fields. Extremegames usually have more fields modified for otherwise good images. So what you'd do is: you'd write a program that tries all combinations with possible modifications removed. Also, don't forget you have to regenerate EDC/ECC on each pass. Volume ID and Date would be the most difficult ones to reverse. Still, if date were removed, you can guess it pretty close and brute force from there. For Volume ID you could try some forms of serial, or look for hints in related CDs (close related games, other games by same developers, etc.). [Images were such modifications were removed include: SLPS-01098, SLPS-01541, SLPS-01697, SLPS-02556, SLPS-03219] R18 images usually have RHP patch applied and Root Directory Record modified, so you'd brute force through PVD combinations in conjunction with those ones [examples: SLPM-86494, SLPM-86505, SLPM-86835, SLPM-86990, SLPM-87287, SLPS-02075,  SLPS-02730, SLPS-02886].


#3.3 - Root Directory Record

Only instances i saw this record moded were R18 releases. They'd usually set File Flags of some or all entries to 0x4. It's never 0x4 on original CDs, so you can easy tell. Reset them back to 0x0 for files and 0x2 for directories. Correct EDC/ECC afterwards.


#3.4 - Patches

For Japan region patches are simple - one/two assembler instructions. There are severa recurring patterns. Interesting to note, however, is that usually they'd differ from ones you'd find in tables posted online (when looking for RHP) and from those in XPS patch files. So, it looks like there was some competition going on and multiple persons cracked same protections at the same time. So, you'd write a program that looks for certain byte pattern (patched), replaces it with other (unpatched), corrects EDC/ECC and checks checksum on whole file. Mostly ther's only one patch applied per image, but in some cases could be 2 or more, so, if you get no hits, it wouldn't hurt to check for such cases too. [some examples, where patches were reversed: SCPS-10140, SLPM-87012, SLPM-87089, SLPM-87216, SLPS-01830, SLPS-01831, SLPS-02561]


#3.5 - EDC/noEDC

This is specific to PSX CDs. Early CDs pressed for this system had EDC fields missing from Form2 sectors (no error control). AFAIR it was mentioned somewhere in SDK as mastering software's error and was corrected later on. Some CDs were released both ways: with those fields missing and added. Also some early CD drives could insert EDC fields by themselves. So it might be that image needs those fields removed or added. Ther's a special command set for this in psxt001z: --zektor / --antizektor.


#3.6 - Damaged files

Streaming media files on PSX lack error correction information and sometimes EDC too, as explained above. So it's not uncommon for them to get damaged on reading. In such cases you could try to replace those files with same files from other sources. CDMage has 'Import File' command for such cases. Though, what i would actually do, is: extract all files from images (with IsoBuster) and compare now stripped down files, lacking any container wrapping, to easy spot any differences, and based on results determine further actions.


#3.7 - Different versions

It could be possible to convert one version of game into another, if sole difference is updated serial number. First try renaming exe in file system and SYSTEM.CNF. If CRC doesn't match then, try renaming onther references to serial. As a last resort you could try to brute force Volume Creation Date. Keep in mind that EDC/ECC fields needs updating after each change.


#3.8 - Other things to try

For CDs having audio tracks, quite often very end of data track gets damaged. Use psxt001z --fix command then (it will regenerate damaged empty data track sectors) and Injector afterwards (--fix command might remove some specific sectors, this program will recreate them).
If you have multiple images but can't tell which part is good in which one, you could try reCombine program. It would just run through all possible combinations.


#4.0 - Version history

v0.1 First draft (audio only) @20120310
v0.2 Sketch on data tracks @20120311



themabus[at]inbox[dot]lv

her's modded nullDC's GD-ROM plugin 'GDR_Win32.dll' (based on Aug 20 2008)
it redefines .gdi syntax parsing for file names
with it file names with whitespaces within them can be used enclosed in quotation marks
e.g.

19
01 0 0 0 "none" 0
02 0 0 0 "none" 0
03 45000 4 2352 "Track 03.bin" 0
04 355602 0 2352 "Track 04.bin" 0
05 363680 0 2352 "Track 05.bin" 0
06 377124 0 2352 "Track 06.bin" 0
07 379975 0 2352 "Track 07.bin" 0
08 386064 0 2352 "Track 08.bin" 0
09 393416 0 2352 "Track 09.bin" 0
10 395488 0 2352 "Track 10.bin" 0
11 399325 0 2352 "Track 11.bin" 0
12 408596 0 2352 "Track 12.bin" 0
13 433134 0 2352 "Track 13.bin" 0
14 458158 0 2352 "Track 14.bin" 0
15 465757 0 2352 "Track 15.bin" 0
16 493667 0 2352 "Track 16.bin" 0
17 503104 0 2352 "Track 17.bin" 0
18 505295 0 2352 "Track 18.bin" 0
19 505819 4 2352 "Track 19.bin" 0

place it in 'plugins' sub-directory and it should show up on plugin selection list
to use .gdis without quotation marks switch back to unmoded plugin

http://www.mediafire.com/?ym2acwgr3dn