Introduction

Hotloading was for a while, well a hot, topic in the game development industry. As games progressed from being very simple programs to more of world simulators today the issue of tweaking and how assets are handled became more of an issue. When your data fits on a standard floppy disk it's not that bad, but when you have your system administrator breathing down your neck because your using up a Gigabyte of disk each night things become complicated.

The sheer amount of data often manifest itself in long loading times during development. This can become a real problem for the artists or programmers who need real time previewing at certain points in the game. The patch for this much larger issue seems to be hotloading, which indeed can be very useful. Hotloading basically means that for example if we press save inside our image editor we want the texture we worked on to automatically be loaded into the running game and show up.

Fig1: Hotloading in action. To the right I have a shader that renders the dice, I've made a modification to output a static color (red). The changewatchdog in the console window below builds the shader as it picks up the change.

Detect, build and load

It's pretty easy to detect inside a plain old win32 application that files have changed on disk, windows provide native support for this in the function ReadDirectoryChangesW. Once we've detected the changed file we can simply call load on this one and replace the asset in the game.

Chances are though that you are not editing the same file that you are loading up into the game, typically one have preprocessing tools that packages these files into neat little chunks that doesn't need very smart routines to load them, e.g. for textures we might provide a header that tells the loader the number of mipmaps, the format and the top level size. So we need to invoke the build system to drag the new asset through the pipeline (or a fastpath), but most likely we're not going to support loading of raw assets inside the game (it's certainly feasible for plain old textures, but consider the case of models, I'm not going to load Maya source files into my game, or even something more complex like your level data). Hm, this could be solved by a little watchdog monitoring changes and triggering a build script.

Again, our little friend python comes to the rescue. A small script can easily detect the changes in the filesystem and in response spawn a build process. This is a longer listing than what I usually show, but it's mainly to show a real live example of code that I run right now and that works.

 
import os,re
import win32file
import win32con

VERBOSE = False
UPDATE_ACTION = 3
FILE_LIST_DIRECTORY = 0x0001
BUFFER_SIZE = 2048
DEFAULT_COMMAND = 'nant 2>&1'
EXTENSIONS = [
    ".cpp",
    ".h",
    ".tga",
    ".jpg",
    ".fx",
    ".tmx",
    ".wav"
]


def findErrorMessage( lines ):
    pattern = re.compile( "BUILD FAILED" )
    for line in lines:
        if pattern.search(line):
            return True
    return False

def executeTask( command ):
    if VERBOSE: 
        print 'Now running the command: "%s"' % command
    pipe = os.popen( command )
    lines = pipe.readlines()
    returnCode = pipe.close()
    
    if findErrorMessage(lines):
        returnCode = 1
    
    if not returnCode:
        if VERBOSE: 
            print "".join(lines)
        return True
    
    print "".join(lines)
    return False
    
def watchDirectory( path, commandLine ):
    hDir = win32file.CreateFile (
        path,
        FILE_LIST_DIRECTORY,
        win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE,
        None,
        win32con.OPEN_EXISTING,
        win32con.FILE_FLAG_BACKUP_SEMANTICS,
        None
    )
    
    changeCount = 0
    while 1:
        results = win32file.ReadDirectoryChangesW (
            hDir,
            BUFFER_SIZE,
            True,
            win32con.FILE_NOTIFY_CHANGE_LAST_WRITE,
            None,
            None
        )
        
        for action, file in results:
            fullFilename = os.path.join (path, file)
            name, ext = os.path.splitext(fullFilename)
            
            actionName = "Unknown"
            if action == UPDATE_ACTION:
                actionName = "Changed"
            
            if ext.lower() not in EXTENSIONS:
                if VERBOSE: 
                    print "%s %s ignored." % (fullFilename, actionName)
                continue
            
            if VERBOSE: 
                print "Change %i: %s %s " % (changeCount, 
                                                    fullFilename, actionName)
            if executeTask( commandLine ):
                print "Ran build %i successfully." % changeCount
            else:
                print "Build failed!"
            changeCount = changeCount + 1

if __name__ == "__main__":
    watchDirectory( ".", DEFAULT_COMMAND )
		

The script starts to listen to changes in the current directory. Whenever it catches one, it checks against a known set of file extensions. If a match is generated, it calls the build task and let that one figure out how to rebuild whatever needs to be rebuilt.

All the voodoo magic happens in the two calls win32file.CreateFile and win32file.ReadDirectoryChangesW. These are the standard win32 functions for detecting changes in the filesystem. The parameters to the functions are of course standard win32, that means a lot of them and half of them redundant. MSDN have the full documentation on the functions for the curious.

Yes, I use nant to build the data. Nant got its issues, but for my little hobby project it's certainly sufficient (even a little overkill). How to mold nant into something that you can actually use is enough material for another article however. Sufficient to say, you could plug in any build system here, simply change the variable DEFAULT_COMMAND.

In closing

So, what are the benefits of this little script? The first thing is that it offloads logic from the game itself. We now have a chance to do our preprocessing in smaller tools before we load assets into the game. At which point we can assume that they don't contain any errors, since the small buildscript have caught them and rejected the new resources. This let us keep to the philosophy that the runtime should not need to handle bad assets, because it's impossible for them to make it that far.

In my little project at home, hotloading is certainly overkill. There are certainly some benefits to hotloading, in server/client architectures it's almost a requirement for any development speed. Since the data path I have for most of the assets at home is really short, there is only one path. One might consider to have a quick and dirty path alongside the regular path for previewing and hotloading for games with larger data sets, or more pre-calculating going on.

Comments