Introduction

Bringing data into your program is always a source of problems. Problems like filesystems, paths and unreliable system calls sound familiar? These problems are not the ones you want to battle with in addition to your unit tests. There is light at the end of the tunnel though. Static data in the executable. We've all done it, quick hardcoded tables inline with the code. Taking it to the next step is simply integrating this into the buildstep and creating these tables from ordinary data, bypassing the filesystem at runtime completely.

bin2c.py

bin2c is short for binary to c array. The input to this little script is any file and the output is a plain textfile that is suitable for inclusion in a standard C array. Listing 1 shows the script. If you are stuck on a windows box, there are a couple of caveats to using this script. The first one is that the pipes on win32 are horribly broken so you can't redirect input any way you want and have it interact with python scripts. Secondly, there is a wonderful idea of binary v/s text files in windows, which this script handles.

Interesting python point is the fact that you can import modules anywhere in the script, not that I'm importing a windows specific module after I've checked that I'm in fact on a windows box. 

#!/usr/bin/env python
import sys
OCTETS_PER_LINE = 16

def convert( outstream, instream ):
    writtenBytes = 0
    while True:
        byte = instream.read(1)
        if len(byte) != 1:
            break
        outstream.write( "0x%.2x, " % ord(byte) )
        writtenBytes += 1
        if writtenBytes % OCTETS_PER_LINE == 0:
            outstream.write( "\n" )
    message = "\n\n/* Size = %i bytes */\n" % writtenBytes
    outstream.write( message )

if __name__ == '__main__':
    if len(sys.argv) == 1:
        instream = sys.stdin
        outstream = sys.stdout
        if sys.platform == "win32":
            import os,msvcrt
            msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
    elif len(sys.argv) == 3:
        instream = file(sys.argv[1],'rb')
        outstream = file(sys.argv[2], 'wt')
    else:
        print "Usage: bin2c [infile outfile]"
        print "\tIf no filenames are given, read from stdin and out"
        print "\tOn windows, invoke through python.exe if redirecting"
        print "\tinput and output since pipes are broken on win32."
        sys.exit(0)
    convert(outstream, instream)
    sys.exit(0)
Listing 1: The bin2c.py script in all it's glory. 
0x98, 0xab, 0xad, 0x16, 0xd5, 0x5c, 0xcd, 0xd6, 
0xd6, 0xd6, 0xcc, 0xd4, 0xd8, 0xfd, 0xa2, 0x2b, 
0x10, 0x8e, 0x19, 0xfc, 0xb9, 0x6d, 0xa0, 0x15, 
0xfe, 0x78, 0xe9, 0xff, 0x58, 0xfb, 0x43, 0x7d, 
0x62, 0x62, 0x1a, 0xaf, 0x5f, 0xab, 0x4d, 0x4b, 
0x19, 0xd8, 0x1f, 0x72, 0xa5, 0xbc, 0x08, 0xe9, 
0x02, 0x5b, 0xcf, 0x3e, 0x39, 0xb4, 0x5b, 0xae, 
0x77, 0x11, 0x0e, 0xa1, 0x3e, 0xa0, 0xbf, 0xb1, 
0x00, 0x49, 0x45, 0x4e, 0x44,

/* Size = 5049 bytes */
Listing 2: Sample output from bin2c.py (snipped for brevity)

Sample

The following shows how to glue everything together. This small program will print the first four bytes from an image in the executable, blue.png, as well as list its' original size in bytes. The makefiles is very simple and glosses over the how to find the compiler and script. I hate to do this to you, but how to encode that is left as an exercise for the reader (hint: look at GNU make standard library). 

#include <cstdio>

static const unsigned char image[] =
{
    #include "blue.inl"
};

int main( int, char** )
{
    std::printf( "Image length is %d bytes\n", sizeof(image) );
    std::printf( "First four octets: %02x%02x%02x%02x", 
                    image[0], 
                    image[1], 
                    image[2], 
                    image[3] );
    return 0;
}
Listing 3: Code that uses the static data pulled in from the .inl file. 
# Windows specific make file
PYTHON        = python
RM            = rm -f
CXX           = cl
CXXFLAGS      = -nologo

SRCDATA       = blue.png
INLDATA       = $(subst .png,.inl, $(SRCDATA))
SOURCE        = test.cpp
EXE           = test.exe

.PHONY: clean all

all: $(INLDATA) $(EXE)

clean:
    @$(RM) $(INLDATA) $(EXE) $(subst .cpp,.obj, $(SOURCE))

%.inl: %.png
    @echo $<
    @$(PYTHON) bin2c.py > $@ < $<

%.exe: %.cpp
    @$(CXX) $(CXXFLAGS) $<
Listing 4: Makefile for gluing everything together.

In closing

The makefile that I've shown is just for demonstration purposes and very limited. In real life you might want to have something more advanced, e.g. dependency checking and automatic generation. There is nothing that prevents inclusion of this technique to visual studio projects, through the custom build step in visual studio one can easily provide automatic generation of the .inl file.

Comments