Hacking Sm4shExplorer (Wii U only for now)

[admin]

Please keep in mind that this is an early version, made purposely to mess around with the game files and see what can be done with it (model swapping, undubbing, etc...). It probably contains bugs.

Current Sm4shExplorer version: 0.07.1
Current Sm4shMusic plugin version: 0.7.1
Current Smash Bros 4 Wii U version: 288 (aka 1.1.6)
Binary download: See attachment >>at the end of this post<<
Source: https://github.com/Deinonychus71/Sm4shExplorer

FOR 0.06 YOU NEED TO DELETE YOUR CONFIG FILE (sm4shmod.xml) AND LET SM4SHEXPLORER GENERATE A NEW ONE.

Also, appart from critical bugs that could still happen for this version, Im gonna take a break for a while.
First because too busy irl to the point where it's not fun anymore to spend most of the free time on it.
Second because I want to fix some issues with the code itself, make it cleaner, optimize building time and most of all, add support for multi workspace, and I want to take my time to do it.

However, if a dev wants help to make their own plugin, or use python with plugins, I am more than happy to help and add some hooks if needed (or take some good advices cause I need some too )
We really need more people willing to help figuring all the missing stuff.

Sm4shExplorer, what does it do?

• It will let you see the whole filesystem of Smash (main data + regions) in a treeview using DTLSExtractor from Sammi-Husky. The treeview will show the updated version of the game (core files in black + patch files in blue)
• Give you a few informations about any file (name, path, size, flags, source...)
• Extract any file/folder to a specific "extract" folder
• Let you open any file with a hex editor (needs to be configured first in the option panel)
• Drag & Drop any files into a "workspace" folder. If a file is modified/added in a packed file, the whole packed file will be rebuilt automatically during export. Alternatively, you can also copy your new files in the "workspace" folder directly and refresh the treeview.
• For region folders, "unlocalize" a folder or a file so that the game loads the unlocalized (english) file instead.
• Let you remove some original resources (experimental, test with stage models to remove background)
• Repack files found in the "workspace" folder and rebuild resources and patchlist into a specific "export" folder.
• Plugin system to add more features in the future

Sm4shMusic



Please make sure to BACKUP your SAVE before attempting to add new songs. We didn't get reports of save corrupted yet but better be safe than sorry.

Sm4shMusic is the first plugin for Sm4shExplorer that enables support for song customization.

• See the list of all the sound IDs referenced by the game, and add/edit them
• Assign up to 40 musics per stage
• Control a bunch of properties like the frequency of a music, description, source, what character appears in the background of Sound Test and the order of the list...
• Save your changes in xml format
• Compile the changes and add them within your workspace directory

Please note: A lot of variables are still unknown! Also we don't know what is the real limit of musics you can reference in the game, or even if all the stages can support up to 40 songs. When such limits will be found, the plugin will be updated accordingly.
Thank you to Soneek, who figured out most of the stuff! Also thank you to kode54 for libvgmstream!



Sm4shParamEditor

Sm4shParamEditor is a very small and simple plugin that will determine if a file can be read by ParamEditor from Sammi-Husky and will open it directly by double clicking on the resource in the treeview. Upon saving your changes, Sm4shExplorer will ask you if you want to push it in your workspace.
Thank you to Sammi-Husky for this very useful tool, which also happened to serve as PoC to test the plugin system xD



What do I need?

This mini tutorial assume you already know how to dump a game, download a patch, use loadiine or cafiine. If you don't please get used to those tools before considering using Sm4shExplorer.

You need to set up a folder with the full game dump + latest patch on your computer. Ideally, try to avoid long paths as there is a nasty limitation with Windows that could create issues.
This folder will have to remain there, untouched, as it will serve as source for Sm4shExplorer:

Step 1 - Dump your game with ddd (there are tutorials): You should end up with code/content/meta folders. Code and Meta are not mandatory but recommended. Within content you have to find at least ls/dt00/dt01 and the sound folder if you wanna use sm4shMusic. Keep that on your computer or nas or whatever, this is NOT going in your SD Card!
Step 2 - Patch download (Again, there are tutorials with UWizard or U_Tility): Get the code/content/meta out of it and copy it in your game dump, replace any existing file. You will now have a "patch" folder in content, more songs in sound/bgm too.

That's all with environment setting, now you don't need to touch this folder ever again, do not edit it, replace anything manually or anything, unless of course a new patch get released (then repeat step 2)

Step 3 - Start sm4shexplorer from fresh (no config sm4shmod.xml file), point to the folder from Step 1 and 2. If the game reads the meta as it should, a popup will spawn telling you the region you're using and your current version (288), save, you're all set.

After step 3 you don't have to worry after the game folder anymore, just leave it be. Everytime you'll start sm4shexplorer it will catch the right folder and right info. What you want on your SD is the result of the mod built, that's what you're gonna find in the export folder after building a new mod with Sm4shexplorer.
IF a new patch ever comes out and you don't want to redo Step 3 (and losing your config), just edit the config file with notepad, find the version (288) and change it to the new version of the new patch.

What you should see on your computer running Windows:

• A copy of your game (extracted) on your computer with v288 patch and at least the following folders/files:
  • content
    • patch
      • data
      • data(region_language)
      • resource
      • resource(region_language)
    • ls
    • dt00
    • dt01
  • meta
    • meta.xml
• If dt00 and dt01 are not present, you will not be able to extract any file from them.
• Please make sure that your core game and your patch are from the same region. During the initialization of sm4shexplorer, a popup will spawn and will ask you to specify the game region and version.
• All these files must remained "unmodified" at all time. Sm4shExplorer will not override any of them and instead will use its own folders.

On your SD/WiiU

• Backup your 'content/patch' folder on your SD before doing ANY CHANGE.
• Loadiine/Cafiine to try your changes.

Testing and reporting bugs

• Before doing any test editing/adding files, try rebuilding the resources & patchlist once and see if the game launches. If it does, the odds are your environment is set up properly.
• Not every single resource is figured out, so don't expect to be able to do everything yet. Also adding/removing resources is often not enough to properly integrate them.

Okay, this is the most important part (to me). Remember when I say it's not stable and is mainly made for developers. Well it means if you ask questions without providing a way to help you (basically, taking 5 seconds to ask something and expecting us to guide you through everystep of the process to find out you didnt set up your environment properly), I'm personally not going to bother answering. Feel free to help if you want, though.

The must-know in order to help properly:
Label: Kind of title for the bug you're having
Sm4shexplorer version used: (0.04? 0.06?) Very important. Also it's nice to mention if something that previously worked on a version is broken with a new one.
Game version: 208/244/288? Custom?
Game region: EUR/USA/JPN?
Language used: yes, it can make a huge difference
Environment setting: Do you only have the patch or do you have the full dump + patch. This is REALLY important as a LOT of problems have to do with people not setting up their environment as intended.

Now for the description of the issue:
Description of the problem: What exactly did you do, what files did you add that caused the issue, etc. Saying "it doesnt work" or "it doesnt do anything in game" is not gonna help.
Did it ever work at all: Yes or no, again, helps to understand if your issue is due to a specific action or not.
Screens: Do you have screen that specifically show the problem? If you get an exception, please include the details, an exception saying there's an exception is useless!
Logs: There are now file logs since 0.06, please copy past the part you think has to do with the error (before, error, after) or upload the full log.
With that i can assure you everyone will save time.


Send to SD

This feature is made to help newbs copying their mods to the SD without having to worry much about what they're doing. Well on paper it looks like it at least xD
- The program will try to find the path of your game on your SD:
* Loadiine: X://wiiu/games/whateverthatcontainsthegameID/
* SDCafiine: X://fullgameID/
The SD card must be formatted in FAT32 and be a removable drive (that should be the standard...)

- Next, the program will initialize a cache file for your mod folder (export), for your SD content folder, and for the official game content folder. It will ignore the dt files, ls, and movies folder. This operation will take a while, because it's basically calculating a CRC32 of all the existing files to know what will need to be copied where. It also uses the last written date to know when to recalculate the CRC32 or when to just trust the cache.

- If you're on Loadiine, the program will now try to see if any official file exists on the SD. If a previous modded file was found, it will replace it by the original file (except if found in the export folder, cause no need to replace twice)
- If you're on SDCafiine, the program will delete existing modded file on the SD that arent going to be part of the new mod.
- Finally, the program will copy the new mod (export) to the SD, again, only if needed.

So basically, a good way to make sure it works is to try it once, wait for the whole operation to complete, and then try resend immediately. If it works correctly, no file will be copied and the operation will only take a few seconds.
And then if you just modify one file it will only copy the changed files, making Send to SD way faster.

Beware though, it will NOT remove files that doesn't need to removed to make the game works. So if you're using loadiine and you keep some extra old mod files (or csv, texts or anything) in your SD for example, they won't be removed, but they won't be referenced either so it should be fine.



Troubleshooting and known issues

• Game doesnt start/softlock during loading screens: It usually means that a resource can't be found. Using the loging options of loadiine/cafiine might help finding what file failed to load.
• DSI Error on Loadiine: It usually means that some files in the patch folder don't match the information found in the resource/patchlist files. It can happens when the mod is built over already modified files. Also, don't try to simply replace a file by another one, unless its size is exactly the same, it simply won't work as the resource files need to be rebuilt.
• LS/DT file issues. This is either because you don't have the DT files in the current folder (or not at all) or because it doesn't match the region setting in your config file. Your patch and dumped game should be of the same region as well (can't use a US dt00 with a EUR patch), it will not work and I'm not supporting it.
• VGMStream failed to load (Sm4shMusic): Welp, it might happen to some people... I honestly never had the problem and I haven't put much thoughts into it. I'm also not familiar with the lib itself. Running the soft in admin mode or simply restarting your computer seem to have helped randomly fixing the issue. Note that this problem won't prevent Sm4shMusic from working at all, but you will need another way of retrieving the sample loop information (using vgmstream plugin for winamp or foobar, for example). Should be fixed with 0.06.4.
• My new songs won't appear in Sm4sh! Make sure you have "In SoundTest"/ Region US/EUR checked. Generally, try to study the properties of the song by looking at the working songs.
• The current "state" of my custom songs are not saved! This seem to be a problem with how Smash save them, some reports suggest it only saves the first 20 songs. This will need some investigation.

Plugins, how do they work? For devs only

The objective with plugins is to let anyone add their own piece of code to help building mods. I'm fairly new with building a plugin architecture, I can't guarantee there wont be any change for now, but here you go:

To create a plugin, you create a new VS project, add sm4shexplorer.exe as a reference, create a new class that will extends "Sm4shBasePlugin". This is an abstract class.
The built project (dll) will have to be in the plugins folder.
Sm4shexplorer will automatically detect it.
I tried to describe every hook and methods as much as possible.

The best current way to know what you can do and what hooks exist in this abstract class is to check it on my github, but basically:

• Name, Author (GUI/Research), Description of a plugin: The classic.
• ShowInPluginList: Boolean to add the plugin in the menu or not.
• Icons: List of icons that can be associated with the compatible files.

A few overrides are available.

• OpenMenuPlugin: Method called when clicking the plugin in the menu.
• CanBeLoaded: Extra checks performed by your plugin to know if it can be used (basically, for the music I check that the bgm folder exists and warn the user that it needs to be there)
• OnLoad: First hook called right after loading the plugin, can be used to instanciate an object one time only for example
• NewModBuilding: Hook happening right before creating a new build, the exportfolder is used as parameter, can be cancelled
• NewModBuilt: Hook happening right after creating a new build, the export folde ris used as parameter
• CanResourceBeLoaded: Check while browsing the treeview to override the default icon if a resource is compatible with the plugin
• ResourceSelected: Happens when double clicking on a resource. If it returns true it will not load the resource hex. Also parameters are the ResourceItem object and the path to the extracted resource (extract folder)
• ResourceAddingToWorkspace: Happens before adding a new file to the workspace, can be cancelled
• ResourceAddedToWorkspace: Happens after adding a new file to the workspace
• ResourceRemovingFromWorkspace: Happens before removing a new file from the workspace, can be cancelled
• ResourceRemovedFromWorkspace: Happens after removing a new file from the workspace
• GridViewPopulated: In order to add more key/value to the gridview when a resource is selected.

An object is available to extract resources and add resources to the workspace as well as accessing the parsed config file.
I'm open to add more features to this class if needed.



Future plans

• Multiple workspace, to make build testing easier
• Rebuild without repacking (use of flag 0x4000)
• RFManager working for the new DTLS
• Incremental build, to make the building process much faster
• 3DS support
• One-click mod installation support

Example: Undubbing Smash Bros for Wii U

• With Sm4shExplorer, locate your JPN version of the game (v208), use Sm4shExplorer to extract the japanese partition (or find the files somewhere).
• Delete the sm4shmod.xml config file, and locate the US version of the game (v208). Copy JPN fighter nus3bank files, JPN sound nus3banks files, JPN stage/melee/wiifit nus3bank files and JPN item/pokemon item/assist sounds files into the data partition of the US version.
• Rebuild (Click on Project > Build the mod)
• Make a copy of the patch folder in your SD card
• Copy the whole content of the "export" folder to your Smash game patch folder on your SD card
• Have fun

Changelog

0.07.1 (Sm4shMusic 0.7):

• Quick & Dirty fix of a crash that can happen when repacking certain stages (flag 0x4000 deactivated)
• Prompt to copy to SD should be deactivated if no SD card is found after building
• Added little plugin for msbt, it requires MSBT Reloaded and its dll to be put in the tools folder.

0.07 (Sm4shMusic 0.7):

• Some work on deleting folders to avoid some lock related crashs.
• Fixed the stupid error with export/extract folder, please reset your extract folder in option so that it doesnt go to export anymore. (also clean the export folder)
• Fix illegal extension crash that could happen when keeping some compressed files (or trash files) in the workspace
• Fix resource removal bug
• Sm4shMusic only: Unk3 figured out by BoomShakaLarka, thanks nhb for reporting it.
• Send to SD feature (experimental, tested only with US, see below)
• Open SD folder (if found)
• Pack modded folders: This is mainly to be able to pack new skins (like c07, c08) so that they appear as "packed" files, which will make the game load them much faster. Again, not much testing was done.
• Loading/Build mod/SendToSD are now in separate threads, which means the app shouldn't freeze anymore.

0.06.4b (Sm4shMusic 0.6b):

• Fix for libvgmstream: Some people claim they didnt have the bug after this version, so hopefully that'll work for everyone.
• Fix for sound desync that some people who removed a song (or blank) were having. Sadly I don't really see a way around starting from scratch, unless you know how to edit the saved xml, but that's seriously gonna take longer.
• Unsigned values have been changed to signed (4294967295 will be shown as -1 now), please understand that it breaks the deserialization of any saved xml config.
• Rarity changed to 100 by default
• Unk5 is now known as Play Delay thanks to @Pib
• BLANK files are now automatically removed when loading the list. This is to avoid a lot of confusion. Instead when you click Add, Sm4shMusic will now automatically fill the gaps in IDs (like in a config from scratch, the first ID to be created will be 15)
• MSBT variables names are now sync to the ID of the song, base 36 (so like ID 10 will be SOUNDA, ID 11 will be SOUNDB, etc). It means that the xml doesn't have to hold the information about the label anymore, making it smaller.

0.06.3 (Sm4shMusic 0.5):

• Added stacktrace log in debug for plugins
• Sm4shMusic: XML Loading -should- be fixed, edition 2
• Fixed a crashed that prevented adding new songs in "stages" with no SoundDB.

0.06.2 (Sm4shMusic 0.4):

• Add a popup before building the mod if the release or debug folder is not empty, letting you choose if you want to keep going or not. It will now clean the export/release (or debug) folder before beginning to export.
• Swap Region EUR/US and Region JAP fixed [TESTED]
• Original/Remix and other labels fixed [TESTED]
• MyMusic Index + 1 fixed [TESTED]
• EUR game region selection -should- be fixed. My bad EUR code is 4 and not 3 (have no idea what 3 is then)
• Whenever you select a pre existing blank file, it will ask you if you want to change the properties to make it visible in Sound Test. [TESTED]
• One rare crash fixed while launching Sm4shMusic after a compilation

0.06.1 (Sm4shMusic 0.2):

• Sm4shMusic Fix bug during the deserialization of the saved music config file
• Added Plugin version in plugin abstract class

0.06:

• Metaknight victory 3 crash error fixed
• Added a file log system to help debugging
• Added a form during initialization to specify game region and game version. This information is now saved in the config file.
• Added a plugin system to let anyone add their own features.
• Added a form to change the priority of the plugins
• Added original resource removal (experimental)
• The loading of the treeview is now much faster, thanks for Sammi's latest DTLS and treeview lazy-loading (there's still room for improvement)
• Added a few shortcuts to open the different folders used by Sm4shexplorer from the GUI.
• meta.xml not needed anymore, however the game is going to assume that you are using the latest version (288) and region US. Please use the form at initialization to change that.
• Added a lot of foolproof checks to try to make sure your environment is set properly.
• Added Sm4shMusic
• Added Sm4shParamEditor

0.04:

• Initial commit

Thanks:

Soneek: His research on music and general knowledge about everything
Sammi-Husky: DTLS, good advices on coding !
DemonSlayerX8: Guinea pig and beta tester
The whole team of Sm4sh Research, just cause they're very cool and motivated people