Let me tell you about my build iteration optimization projects I built this week!
And by you I mean me, I had a lot of fun creating this but at the same time I need to motivate having spent time on this project instead of working on the project it is intended for.
Let’s get right into the pitch
I’ve frequently relied on .bat files on windows and bash scripts on linux/mac for my retro game dev projects. It is safe and traditional, but somewhat error prone and tedious when the project scales up.
So here is my solution: something in between a batch file and a makefile.
If you haven’t used makefiles they are basically predating your IDE or CMake files, using mostly characters to define rules for building a project.
The drawback of makefiles is that the syntax is hard to remember, and wildly ordering commands like you can in a script is a little trickier. What makefiles handle for you is to skip redundant compilation.
You can find the download here: Initial release of bath
And the code is here: bath repo (there is a fun naming story in the readme.md)
Here is my solution
A small tool named bath that rearranges each command line into a tool, an expected output and an expected input and arguments.
This allows the builder to check if the inputs are older than the outputs, and only then run the command. In addition it also builds multiple commands in parallel with simple sync points when data needs to be complete before continuing.
I call this tool “bath”. It is literally a combination of ‘bat’ and ‘sh’.
My Project Status
I have a Commodore 64 project named Island Adventure. This is the amount of scripted commands I run to build data and code:
- Number of files that gets merged into the cartridge: 560
- Number of commands executed to build the files: 529 / 529
- Total input files: 598
- Total output files: 774
So let’s compare timing
Full rebuild single threaded, the closest to a .bat or .sh file:
Total execution time: 37.828682800 seconds
Full rebuild parallel:
Total execution time: 10.343019500 seconds (update: around 7 seconds after some build script fixes I found)
No changes, just check input and output last modified time:
Total execution time: 1.955738900 seconds (update: around 1 second after build script fixes)
Making a change to a single code file will process in a fraction of a second, meaning that I’ve now turned my iteration time from nearly 40 seconds down to around 1 second.
This is not strictly true since I would previously build only the active code files when programming so it would not be 40 seconds for just code iteration but closer to 10 seconds. But the parallelization really helps with that sweet iteration time!
Having multiple bat/bash files also meant I had to remember which types of data I had changed so I could run all the relevant ones. Usually this meant just running each one after pulling latest on a computer I hadn’t work on for a while.
A former colleague used to argue if a developer ever had time to pull out their phone while the build was running you’d lost their focus. I feel that time is around 5 seconds so this goal is looking like it is made!
Highlights
- Skips commands if output is newer than input
- Parallelized processing when desired
- Sync points when outputs must finish before continuing
- Full unicode support for files and script
- Missing output folders are created as needed
- Input files are validated before running any command
- Change directory / make directory commands
- Sets the current directory to the .bath file path, restores on exit.
- Stops on error
- Clean / Rebuild if file dates are off or tools change
- Can execute regular command lines in raw mode
Current Status
I’ve finally converted a smaller project and now this project. It has been a bit tedious to go over all the conversions again but it has really cleaned up some minor mistakes that I didn’t spot before.
I’ve got it running consistently on two projects on both my Linux and Windows computers. My Raspberry Pi 400 memory card ran out of space in the middle of a system update so probably need to get a bigger card and start over…
Getting Sidetracked
While working on this I realized that I had another process obstruction. I make the art for the game and I use an art tool that has its own custom format (Pixcen, .gpx). So I used to save as .png with every change I made to the .gpx file and each time I would do that the tool would ask for confirmation for overwriting the new file. This is another way to break developer focus, too many dialog boxes to click through!
So I pulled up my sleeves and did a custom loader, it was quite intricate but then at the core it turned out to be stored in the Vic II hardware format (Commodore 64 graphics chip).
I can now convert the graphics data directly from the work file and that will make it much easier for me to move forward!
C64Gfx can be downloaded from: C64Gfx releases
The code is available (including the .gpx loader) in the repo: C64Gfx repo
Building notes
I’ve switched from visual studio solution files to cmake to keep the build consistent. It you want to open the project in visual studio you can just open the project folder and visual studio will locate the cmake file. CMake can also generate a visual studio solution.
In Visual Studio Code on either platform it usually works to just press F7 to build cmake projects. Look at the output pane next to the terminal to see which folder the executable ended up in.
To help avoid lengthy dependency installations in Linux for clang and cmake you can just run ./build.sh to build either project with GNU C/C++.
If I had a mac I’d support it, but there is a chance both of these projects will just work as is.
Summary
I’m looking forward to staying more focused when I have a little time to code or do art for my game. These changes have been my focus this week and I feel they have wrapped up nicely so I can move on to actually using them 🙂