|
|
Post by mariokart64n on Jan 13, 2015 19:26:24 GMT -5
I'm gonna try to post something everyday even if its small just to keep myself focused and motivated.
After christmas I ended up blankly starring at my script code getting lost then confused then frustrated.
so gonna hammer below the structures I know, aswell write some notes so that others can follow along.
at the end of it we'll all have an idea of how the files work, then I hope I can finally give you guys a stable model importer.
|
|
|
|
Post by mariokart64n on Jan 13, 2015 19:26:53 GMT -5
Format Study of Blue Castle’s BIGs engine
Article By: mariokart64n (January 2015)
Intro
Hi I’ve been doing amateur scripting/reversing for 6 years now. I seem to work on model extraction the most, just from its popularity.
All of what I figure out is from pure visual observation, no debuggers/listeners/monitors/disaasemblers were used in my findings.
Please feel welcome to correct and build upon any of my observations
Terms/Def
INT8: 1byte signed integer value
UINT8: 1byte unsigned integer value
INT16: 2byte signed integer value
UINT16: 2byte unsigned integer value
INT32: 4byte signed integer value
UINT32: 4byte unsigned integer value
FLOAT16: nvidia’s 16bit encoded half-float
FLOAT32: typical 32bit encoded float
STRING: 1 dimensional array of characters
Overview
Assets are stored in archive files with the extension (.BIG) some other extensions are used but still follow the same archive structure.
|
|
|
|
Post by mariokart64n on Jan 13, 2015 19:27:05 GMT -5
The Archive "BIG" files are stored in the archive and can be compressed using zlib(inflate?), based on platform of the game the archive may adopt that platforms native compression library. Files are kept with there full name including there extension. However there seems to be no folder structure. An unknown ID is specified for each file, which could be interpreted as a data type ID or perhaps a predetermined folder name. As the archive stores a large range of game binaries, sound, animation, geometry, textures, etc An odd thing is some archives contain valid header data as to retrieve sub files, but then the archive ends short. It’s thought that the archive may be used in memory and can be used to address raw packages in RAM. However that is a guess, what is known is that some archives do not contain any readable files. Such a thing causes issues with parsing as specific data files are found within the structure of the archive but then that data simply does not exists All files in the archive are given a 32bit SDBM hash code, used for file linking and likely other data lookups by the game. my implementation of SDBM was derived from the following article: www.cse.yorku.ca/~oz/hash.htmlData Structure *all address offsets are absolute from the starting of the file, unless otherwise noted HEADER {
UINT32:fileid // PC DR3, always seems to be 0x03040506
UINT32:data_offset // jumps to file raw data
UINT32:filesize // entire archive size, including header
UINT32:filecount // number of files in archive
UINT32:filetable_offset // jumps to table that describes archive files
UINT32:stringtable_offset // jumps to table that contains strings used by files
}
STRINGTABLE{
STRING:filename //dynamic size, delimited with white space 0x00
}
FILETABLE{
UINT32:filename_offset // jumps to the string contained in the stringtable
UINT32:hash // SDBM hash generated from filename
UINT32:csize // compressed size
UINT32:usize // uncompressed size
UINT32:offset // jumps to file data
UINT32:type // can be used to sort archive files?
UINT32:cflag // compression flag, if not 0 then file is compressed, 1=zlib
}
CFLAG_ENUM{
0x00: raw
0x01: zlib
}
TYPE_ ENUM {
0x0004: headerinfo
0x0010: texturedata // also includes HavokData
0x0020: modeldata
0x0100: bigfile
0x0200: audiodata
0x0800: compressed
}
|
|
|
|
Post by solidcal on Jan 14, 2015 6:58:48 GMT -5
Good to see you are still doing it. We could really use it.  Also gonna sticky this as it is important and we don't want it buried under other topics later. |
|
|
|
Post by mariokart64n on Jan 15, 2015 2:32:38 GMT -5
The Model Package
Geometry is stored obviously within BIG containers, which were explained above...
But BIG's are not a standalone model file, they are just for storing files, among whatever else they're used for...
The BIG's I see that work as models contain a breakup of multiple sub file model components.
so we have to pick apart and understand each of these sub files within the big in order to successfully read and write new model BIG's
First I'll list known subfiles, then I'll start to document them. bare in mind I do not know the hierarchy of each file.
What that means is some files are read first, or last some are required or included by earlier files etc.. but without knowing that its messy guess work :\
Last thing to note, there is a unknown ID given to each subfile that works as a category identifier of sorts. 0x04 seems to caontain most of the files,
so I'll be covering those first, before the other ones
List of Discovered (0x04)BIG SubFiles *(List subject to change)
_attribute_ // single integer, not known what it does
_animlib_skeleton_ // skeleton
_boundingbox_ // model bound box
_collisionlist2instances_ //
_collisionlist2_ //
convexhull //
convexhullplanes //
convexhullvertices //
commandbuffer // array of commands needed to parse geometry
desc2 //
ibheader //
vbheader //
vdheader //
maskrenderstrip //
combined_ibheader //
combined_vbheader //
matarray //
matparameterarray //
matsamplerinfoarray //
mattextureinfoarray //
scenedescription //
sceneheader //
sharedopaquecommandbuffer //
sharedtransparentcommandbuffer //
sharedzpasscommandbuffer //
sharedzpassfirstcommandbuffer //
sharedzpasssecondcommandbuffer //
texturenames // array of texture names
totalibsize // total size of index buffer in bytes
totalvbsize // total size of vertex buffer in bytes
|
|
|
|
Post by mariokart64n on Jan 21, 2015 22:59:34 GMT -5
_attribute_
yup no idea what this does
attribute{
UINT:unknown
}
|
|
|
|
Post by mariokart64n on Jan 21, 2015 23:00:16 GMT -5
_animlib_skeleton_
header{
UINT32:unknown1 // version?
UINT16:count //
UINT16:unknown2 // flag?
}
position_table{
FLOAT32:RotationX // quat rotation
FLOAT32:RotationY
FLOAT32:RotationZ
FLOAT32:RotationW
FLOAT32:PositionX // position
FLOAT32:PositionY
FLOAT32:PositionZ
FLOAT32:UNKNOWN1 // ?position scalar value?
UINT16:UNKNOWN2
UINT16:string_offset
}
hierarchy_table{
UINT8:index
}
name_table{
STRING:bone_name
}
|
|
|
|
Post by mariokart64n on Jan 21, 2015 23:01:52 GMT -5
_boundingbox_
bounding box for the model, donno whats this is used for, camera, or mesh scaling
basically you get two 3D points that create a cage around the model.
boundingbox{
FLOAT32:MinPositionX
FLOAT32:MinPositionY
FLOAT32:MinPositionZ
FLOAT32:MaxPositionX
FLOAT32:MaxPositionY
FLOAT32:MaxPositionZ
}
|
|
|
|
Post by mariokart64n on Jan 21, 2015 23:03:09 GMT -5
_collisionlist2instances_
PLACEHOLDER
not documented yet..found in a stage file
|
|
|
|
Post by mariokart64n on Jan 21, 2015 23:03:23 GMT -5
_collisionlist2_
PLACEHOLDER
not documented yet..found in a stage file
|
|
|
|
Post by mariokart64n on Jan 21, 2015 23:04:13 GMT -5
convexhull
PLACEHOLDER
not documented yet..found in a stage file
|
|
|
|
Post by mariokart64n on Jan 21, 2015 23:04:33 GMT -5
convexhullplanes
PLACEHOLDER
not documented yet..found in a stage file
|
|
|
|
Post by mariokart64n on Jan 21, 2015 23:04:47 GMT -5
convexhullvertices
PLACEHOLDER
not documented yet..found in a stage file
|
|
|
|
Post by mariokart64n on Jan 21, 2015 23:18:29 GMT -5
commandbuffer
credits to SirKane for sharing the field names
command{
UINT8:magic
}
commandcodes{
0x01:CMD_RET
0x08:CMD_DRAW_INDEXED
0x09:SET_VERTEX_DECL
0x0C:SET_TEXTURE
0x0D:SET_TEXTURE_STATE
0x0E:SET_DRAW_STATE
0x25:SET_OFFSET_VERTEX_BUFFER
0x35:SET_SHADER_RUNTIME_PASS
0x41:SET_OFFSET_INDEX_BUFFER
0x73:SET_TEXTURE_UNRESOLVED
0x74:SET_TEXTURE_STATE_UNRESOLVED
}
... need to complete structs for each code
|
|
djlarryt
Modder
Ridin the dolphin!
Posts: 171
|
Post by djlarryt on Jan 22, 2015 7:47:42 GMT -5
Thanks MarioKart.... I almost gave up hope on this |
|
|
|
Post by mariokart64n on Feb 10, 2015 14:13:19 GMT -5
sorry I should probably update:
started writing my script from scratch in JAN but since this FEB I've had bad luck for concentration.
I said before that I felt my implementation was wrong, and I think that was because I was reading the
command buffer in a strange way, I only parsed for one instance of certain data stored in it and then read the entire
vertex buffer and used the stored crap from the command buffer to build faces.
I know that i did that entirely wrong, I basically have to store the entire command buffer in a multi dimensional array
then only read what the command tells me to and once the commit or RET(0x01) command is supplied
..anyway I feel REAL bad about the time frame, usually I'm up and going with a script within a month or two. :\
but its been like half a year! thats unforgivable in any situation, I'm sorry about that
I want to pay you back since I've been mostly absent in accomplishing what I said I would do.
please let me know if your interested in something on steam and I'll gift you a game, as you did for me with dr3
I'm not giving up on creating a tool for DR3, its just not ontop of my list of top priorities right now
cause right now I've got the flu with splitting headache and a comfy bed that I'm going to be hibernating in for the next week >_< lol
|
|
|
|
Post by solidcal on Feb 12, 2015 16:19:16 GMT -5
It's fine. A shame you have had such a set back and people will need to wait a while longer, but don't rush and burn out and vanish again. |
|
|
|
Post by solidcal on Feb 16, 2015 7:49:43 GMT -5
Mario I could kiss you right now. Thank you. We have something to test. Shall we PM any bugs to you or have you already seen most of them?
Btw mario do you have like a beta textures converter? As the old one converts all except the bump maps sadly.
|
|
tommah
Veteran
Where's My Wiskey
Posts: 1,270
|
Post by tommah on Feb 16, 2015 12:56:12 GMT -5
very cool, congrats
|
|
|
|
Post by mariokart64n on Feb 21, 2015 3:56:47 GMT -5
oh man this format is such a headache, they dynamically jump around files and use different vertex definitions. but each time they do that they must actively store a pointer and address for each different vertex type. my code is quickly getting very messy, and I havent found a very efficient way to do it. .. I tell yah, you think you seen it all then some silly format tosses you on your head and you feel like an idiot all over again  anyway here's the latest "Beta Script" and no I haven't bothered to add UV's yet Download: pastebin.com/YHwDmg57 |
|
|
|
Post by solidcal on Feb 21, 2015 5:36:29 GMT -5
oh man this format is such a headache, they dynamically jump around files and use different vertex definitions. but each time they do that they must actively store a pointer and address for each different vertex type. my code is quickly getting very messy, and I havent found a very efficient way to do it. .. I tell yah, you think you seen it all then some silly format tosses you on your head and you feel like an idiot all over again anyway here's the latest "Beta Script" and no I haven't bothered to add UV's yet Download: pastebin.com/YHwDmg57Thank you mario. Also yeah the UVs are the only thing missing. I mean we could try to fix the uvs ourselves in like 3ds max. But the UVs are that messed up we can't do that. Also don't over rush yourself and burn out. Just relax and do what you do. Like stipo said we have patience and the day this it is finished will be a glorious day. So yeah thanks for even doing this mario. |
|
|
|
Post by mariokart64n on Feb 22, 2015 14:00:35 GMT -5
A dozen NPC's continue to defy me! still working towards swiftly delivering them to their miserable demise before revealing the true colours of their texture maps!!!!!!! Lastest Maxscript: pastebin.com |
|
|
|
Post by solidcal on Feb 22, 2015 16:01:33 GMT -5
Great work mario. A lot more of the npcs are being imported right now. A few clothing items like the chuck dlc outfit still come out in a mess and yeah just the uvs need to be added when and if possible. Some good progress here. |
|
|
|
Post by mariokart64n on Feb 23, 2015 21:34:56 GMT -5
alright like 99% of the NPC models appear to import correctly (uvs aside) so I'm considering just moving forward with what I got. ? any thoughts, I need input and testers if I move forward I'll dig out more of the other files so that I can start to re-write data into the game latest maxscript: pastebin.com/uCfRTU1h |
|
|
|
Post by solidcal on Feb 24, 2015 6:12:30 GMT -5
alright like 99% of the NPC models appear to import correctly (uvs aside) so I'm considering just moving forward with what I got. ? any thoughts, I need input and testers if I move forward I'll dig out more of the other files so that I can start to re-write data into the game latest maxscript: pastebin.com/uCfRTU1hRight just tested the new script. All seems to work well. Tested it on a lot of meshes that came out all Fed up and they are all fine now. I think it's safe to say you can move on to the uving and other finaliszing things and such. That's just me though. Great work Mario. |
|
djlarryt
Modder
Ridin the dolphin!
Posts: 171
|
Post by djlarryt on Feb 24, 2015 6:51:39 GMT -5
Corey, you are amazing! Thanks so much for all your hard work- importing to 3DsMax is halfway there, so stop beating yourself up., it's closer to succes. I was right when I said if you couldn't crack it, it's above what anyone without inside info could accomplish  ? I'd be happy to test anything you need when I get back from holidays next week (I'm in Brazil ☀️). I have worked in both game/app development and I can see some techniques used here (eg roaming data containers, this speeds up gameplay and rendering which is why there are buckets that don't seem to contain anything) and know there is always some definition in-program that tells the game engine what to do... So nothing is really that obscured, we just need to figure it out ? Again, thanks for all your diligence in hacking away at this, we'll chat soon. |
|
|
|
Post by mariokart64n on Feb 25, 2015 2:46:42 GMT -5
yeah there is alot of hashing, the model's texture list contains more then what is in the actual texture container. I reckon shared textures such as eyes, mouth etc are linked somehow with the hashes they have written everywhere. anyway UV's work, gonna have to test because my D3DVertexDesc isn't very well flushed out yet. comp (
0x00: #position
0x01: #weight
0x02: #boneid
0x03: #normal
0x04: #binormal -- not confirmed
0x05: #texture
0x06: #tangent -- not confirmed
)
datatype (
0x06: #point3 -- 3 32bit floats
0x0D: #pack4 -- packed normal (4bytes)
0x1E: #point4b -- 4bytes, for bone ids and weights
0x25: #point2f -- 2 shorts (4bytes)
) anyway enjoy your time in brazil man |
|
|
|
Post by solidcal on Feb 25, 2015 5:30:42 GMT -5
yeah there is alot of hashing, the model's texture list contains more then what is in the actual texture container. I reckon shared textures such as eyes, mouth etc are linked somehow with the hashes they have written everywhere. anyway UV's work, gonna have to test because my D3DVertexDesc isn't very well flushed out yet. comp (
0x00: #position
0x01: #weight
0x02: #boneid
0x03: #normal
0x04: #binormal -- not confirmed
0x05: #texture
0x06: #tangent -- not confirmed
)
datatype (
0x06: #point3 -- 3 32bit floats
0x0D: #pack4 -- packed normal (4bytes)
0x1E: #point4b -- 4bytes, for bone ids and weights
0x25: #point2f -- 2 shorts (4bytes)
)anyway enjoy your time in brazil man Yeah as I recall on npcs and player models the mouth, tongue and teeth seem to always be shared as unpacking the tex files show they are named the same. Also any plans to add eapon model importing or are you just gonna leave that and do a all in one dr importer script you talked about last year? |
|
|
|
Post by mariokart64n on Feb 25, 2015 22:29:42 GMT -5
where are the weapons located?
|
|
|
|
Post by solidcal on Feb 26, 2015 4:35:44 GMT -5
where are the weapons located? dynamicprops_dr3 and inside the dlc folder inside it. Seems to include the normal weapons along with the dlc ones like Frank's slugger bat and chuck's paddlesaw. |
|
?